Merge pull request #312857 from msftadam/patch-579405

Revise upgrade failure behavior documentation

Author: denrea

articles/operator-service-manager/safe-upgrades-nf-level-rollback.md

@@ -3,55 +3,89 @@ title: Control upgrade failure behavior with Azure Operator Service Manager
 description: Learn about recovery behaviors including pause on failure and rollback on failure.
 author: msftadam
 ms.author: adamdor
-ms.date: 08/30/2024
+ms.date: 03/06/2026
 ms.topic: upgrade-and-migration-article
 ms.service: azure-operator-service-manager
 ---
 
 # Control upgrade failure behavior
 
-## Overview
-This guide describes the Azure Operator Service Manager (AOSM) upgrade failure behavior features for container network functions (CNFs). These features, as part of the AOSM safe upgrade practices initiative, offer a choice between faster retries, with pause on failure, versus return to starting point, with rollback on failure.
+This guide describes the Azure Operator Service Manager (AOSM) upgrade failure behavior features for container network functions (CNFs). For faster retries, use pause on failure. To return to the starting point, use rollback on failure.
 
 ## Pause on failure
 Any upgrade using AOSM starts with a site network service (SNS) reput operation. The reput operation processes the network function applications (nfApps) found in the network function design version (NFDV). The reput operation implements the following default logic:
+* A user initiates an SNS reput operation with pause-on-failure enabled.
 * nfApps are processed following either `updateDependsOn` ordering, or in the sequential order they appear.
-* nfApps with parameter `applicationEnabled` set to disable are skipped.
-* nfApps present, but not referenced by the new NFDV are deleted.
-* The execution sequence is paused if any of the nfApp upgrades fail and an atomic rollback is considered.
-* The failure leaves the NF resource in a failed state.
+* If an nfApp install or upgrade operation fails, the atomic rollback setting for that operation and nfApp is honored.
+* No prior completed NfApps are further operated upon.
+* The task terminates and leaves the SNS resource in a failed state.
 
-With pause on failure, AOSM rolls back only the failed nfApp, via the `testOptions`, `installOptions`, or `upgradeOptions` parameters. No action is taken on any nfApps which proceed the failed nfApp.  This method allows the end user to troubleshoot the failed nfApp and then restart the upgrade from that point forward. As the default behavior, this method is the most efficient method, but may cause network function (NF) inconsistencies while in a mixed version state. 
+With pause on failure, AOSM rolls back only the failed nfApp, via the `testOptions`, `installOptions`, or `upgradeOptions` operation parameters. No action is taken on any nfApps proceeding the failed nfApp. This method allows the end user to troubleshoot the failed nfApp and then restart the upgrade from that point forward. As the default behavior, this method is the most efficient method, but may cause network function (NF) inconsistencies while in a mixed version state. 
+
+### Upgrade successful
+An upgrade is considered successful if all nfApps reach the desired target state without generating helm install or helm upgrade failures. In such conditions, Azure Operator Service Manager returns the following operational status and message:
+
+```
+  - Upgrade Succeeded
+    - Provisioning State: Succeeded
+    - Message: <empty>
+```
+
+### Upgrade unsuccessful
+An upgrade is considered unsuccessful if any nfApp generates a helm install or helm upgrade failure. In such conditions, Azure Operator Service Manager returns the following operational status and message:
+
+```
+  - Upgrade Failed
+    - Provisioning State: Succeeded
+    - Message: Application(<ComponentName>) : <Failure Reason>
+```
 
 ## Rollback on failure
-To address risk of mismatched nfApp versions, AOSM now supports NF level rollback on failure. With this option enabled, if an nfApp operation fails, both the failed nfApp, and all prior completed nfApps, can be rolled back to initial version state. This method minimizes, or eliminates, the amount of time the NF is exposed to nfApp version mismatches. The optional rollback on failure feature works as follows:
-* A user initiates an SNS reput operation and enables rollback on failure.
+To address risk of mismatched nfApp versions, Azure Operator Service Manager supports NF level rollback on failure. With this option enabled, if an nfApp operation fails, both the failed nfApp, and all prior completed nfApps, can be rolled back to initial version state. This method minimizes, or eliminates, the amount of time the NF is exposed to nfApp version mismatches. The optional rollback on failure feature works as follows:
+* A user initiates an SNS reput operation with rollback on failure enabled.
+* nfApps are processed following either `updateDependsOn` ordering, or in the sequential order they appear.
+* Atomic state for all NfApps is forced to true, any operator provided values are ignored.
 * A snapshot of the current nfApp versions is captured and stored.
 * The snapshot is used to determine the individual nfApp actions taken to reverse actions that completed successfully.
   - `helm install` action on deleted components,
   - `helm rollback` action on upgraded components,
   - `helm delete` action on newly installed components
-* nfApp failure occurs, AOSM restores the nfApps to the snapshot version state before the upgrade, with most recent actions reverted first.
+* If an nfApp install or upgrade operation fails, an atomic rollback of the failed nfApp is executed first.
+* After the atomic rollback, the prior completed NfApps are restored to original snapshot version, with most recent actions reverted first.
+* The task terminates and leaves the SNS resource in a failed state.
 
 > [!NOTE]
 > * AOSM doesn't create a snapshot if a user doesn't enable rollback on failure.
 > * A rollback on failure only applies to the successfully completed nfApps.
->   - Use the `testOptions`, `installOptions`, or `upgradeOptions` parameters to control rollback of the failed nfApp.
+> * An error with the atomic rollback isn't treated as a rollback failure.
+
+### Upgrade successful
+An upgrade is considered successful if all nfApps reach the desired target state without generating helm install or helm upgrade failures. In such conditions, Azure Operator Service Manager returns the following operational status and message:
 
-AOSM returns the following operational status and messages, given the respective results:
 ```
   - Upgrade Succeeded
     - Provisioning State: Succeeded
     - Message: <empty>
+```
 
+### Rollback successful
+A rollback is considered successful if all prior completed NfApps reached the original snapshot state without generating a helm rollback failure. In such conditions, Azure Operator Service Manager returns the following operational status and message:
+
+```
   - Upgrade Failed, Rollback Succeeded
     - Provisioning State: Failed
     - Message: Application(<ComponentName>) : <Failure Reason>; Rollback succeeded
+```
 
+### Rollback unsuccessful
+A rollback is considered unsuccessful if any prior completed nfApps fail to reach the original snapshot state, instead generating a helm rollback failure. In such conditions, Azure Operator Service Manager stops processing any further rollback-eligible nfApps and terminates with the following operational status and message:
+
+```
   - Upgrade Failed, Rollback Failed
     - Provisioning State: Failed
     - Message: Application(<ComponentName>) : <Failure reason>; Rollback Failed (<RollbackComponentName>) : <Rollback Failure reason>
 ```
+
 ## How to configure rollback on failure
 The most flexible method to control failure behavior is to extend a new configuration group schema (CGS) parameter, `rollbackEnabled`, to allow for configuration group value (CGV) control via `roleOverrideValues` in the NF payload. First, define the CGS parameter: 
 ```
@@ -96,6 +130,14 @@ example:
 > * Each `roleOverrideValues` entry overrides the default behavior of the NfAapps.
 > * If multiple entries of `nfConfiguration` are found in the `roleOverrideValues`, then the NF reput is returned as a bad request.
 
+## Manage nfApps that don't support rollback
+Almost all publishers report some nfApps that aren't compatible with helm rollback operations. These nfApps maybe sourced from third-parties who don't common support such strict resiliency requirements. These nfApps maybe related to database applications with complicated schema management requirements. In these cases, special consideration should be taken to deal with nfApps that don't support rollback.
+
+* The strong preference is to push vendors to support helm rollback.
+* nfApps that don't support rollback can't be skipped.
+* nfApp rollback order can't change.
+* Incremental-NFDV approach must be used in these situations.
+
 ## How to troubleshoot rollback on failure
 ### Understand pod states
 Understanding the different pod states is crucial for effective troubleshooting. The following are the most common pod states: