Merge pull request #312927 from msftadam/patch-291559

Revise upgrade failure behavior documentation

Author: prmerger-automator[bot]

articles/operator-service-manager/safe-upgrades-nf-level-rollback.md

@@ -9,10 +9,9 @@ ms.service: azure-operator-service-manager
 ---
 
 # Control upgrade failure behavior
-
 This guide describes the Azure Operator Service Manager (AOSM) upgrade failure behavior features for container network functions (CNFs). For faster retries, use pause on failure. To return to the starting point, use rollback on failure.
 
-## Pause on failure
+## Pause on failure overview
 Any upgrade using AOSM starts with a site network service (SNS) reput operation. The reput operation processes the network function applications (nfApps) found in the network function design version (NFDV). The reput operation implements the following default logic:
 * A user initiates an SNS reput operation with pause-on-failure enabled.
 * nfApps are processed following either `updateDependsOn` ordering, or in the sequential order they appear.
@@ -40,7 +39,7 @@ An upgrade is considered unsuccessful if any nfApp generates a helm install or h
     - Message: Application(<ComponentName>) : <Failure Reason>
 ```
 
-## Rollback on failure
+## Rollback on failure overview
 To address risk of mismatched nfApp versions, Azure Operator Service Manager supports NF level rollback on failure. With this option enabled, if an nfApp operation fails, both the failed nfApp, and all prior completed nfApps, can be rolled back to initial version state. This method minimizes, or eliminates, the amount of time the NF is exposed to nfApp version mismatches. The optional rollback on failure feature works as follows:
 * A user initiates an SNS reput operation with rollback on failure enabled.
 * nfApps are processed following either `updateDependsOn` ordering, or in the sequential order they appear.
@@ -86,7 +85,7 @@ A rollback is considered unsuccessful if any prior completed nfApps fail to reac
     - Message: Application(<ComponentName>) : <Failure reason>; Rollback Failed (<RollbackComponentName>) : <Rollback Failure reason>
 ```
 
-## How to configure rollback on failure
+## Configure rollback on failure
 The most flexible method to control failure behavior is to extend a new configuration group schema (CGS) parameter, `rollbackEnabled`, to allow for configuration group value (CGV) control via `roleOverrideValues` in the NF payload. First, define the CGS parameter: 
 ```
 {
@@ -133,11 +132,48 @@ example:
 ## Manage nfApps that don't support rollback
 Almost all publishers report some nfApps that aren't compatible with helm rollback operations. These nfApps maybe sourced from third-parties who don't common support such strict resiliency requirements. These nfApps maybe related to database applications with complicated schema management requirements. In these cases, special consideration should be taken to deal with nfApps that don't support rollback.
 
-* The strong preference is to push vendors to support helm rollback.
+* The strong preference is to push publishers to support helm rollback for all nfApps.
 * nfApps that don't support rollback can't be skipped.
 * nfApp rollback order can't change.
 * Incremental-NFDV approach must be used in these situations.
 
+### Selective rollback using incremental NFDVs
+A network function’s composition often includes one, or more, nfApplications that can't support a helm rollback operation, such as Elastic or VoltDb. If a rollback is attempted on one of these nfApplications, the resulting nfApplication is broken. Pursuing publisher enhancements, to make these nfApplications rollback complaint is the best solution. Recognizing the potential for long publisher enhancement lead times, a method to prevent execution of rollback on selective nfApplications is needed. Selectively skipping rollback requires thorough testing with the network function owners as it resulting in transiet condition where multiple version permutation exist. 
+
+#### Problem Statement
+At the network function level, when nfRollbackEnabled is true, and a failure occurs during an upgrade or install, a rollback is executed across all nfApps which proceed the failure. This may include those which are rollback noncompliant. A selective rollback parameter is not supported. It introduces risk of an operational state that doesn't correspond to a defined NFDV. This state mismatch results in nondeterministic behavior, increases the testing surface significantly, and undermines the reliability guarantees of deployment processes. Instead we rely on NFDVs to ensure deterministic workload states that map to well-defined and tested deployment configurations.
+
+#### Proposed Solution 
+AOSM proposes that publishers should use a combination of skipUpgrade and nfRollbackEnabled configurations in CGVs, along with multiple NFDVs, to logically segment nfApplications based on rollback compatibility. This multi-NFDV strategy allows customers to bypass rollback for select charts while preserving safety for the rest. This approach is production-safe and aligns with existing AOSM mechanisms. This staged approach effectively simulates per-chart rollback behavior using NFDV-level constructs. Consider the following example where a network function is composed of 20 nfApps with five nfApps that don't support rollback.
+
+* NFDV1
+  * Performs initial install of all 20 charts with version v1.0.
+  * In CGV1: rollbackEnabled: irrelevant (fresh install).
+* NFDV2:
+  * Contains all 20 charts but the five Helm charts without rollback support, upgraded to v2.0.
+  * In CGV2:
+    * Use skipUpgrade: true for the remaining 15 charts.
+    * Set nfRollbackEnabled: false.
+  * Result: 
+    * Success: Only five charts upgrade
+    * Failure:
+      * No rollback if upgrade fails.
+      * Due to chart limitations, the workload is left in a nondeterministic state. No rollback is possible. To recover, there are two options:
+        * Upgrade with a working NFDV2
+        * Upgrade with NFDV1 and skipUpgrade disabled for every nfApplication
+* NFDV3:
+  * Contains all charts but the 15 rollback-compatible charts upgraded to v2.0.
+  * In CGV3:
+    * Use skipUpgrade: true for the 5 charts already handled in NFDV2.
+    * Set nfRollbackEnabled: true.
+  * Result: Remaining 15 charts upgrade; rollback occurs on failure.
+
+> [!NOTE]
+> * The five rollback-incompatible charts must not have runtime upgrade dependencies on charts in NFDV3.
+> * AOSM's rollback design assumes that rollback restores the workload state to the previous NFDV state.
+
+This approach providers cleaner separation and manageability of applications not supporting standard helm operations. Maintains the operation’s idempotency and state on the cluster reflected by the last operation. NFDV 2/3 can directly be used for install operations as well (installation of previous version not needed) with any difference in goal state. Overall upgrade time and deployment reliability remain the same. 
+
 ## How to troubleshoot rollback on failure
 ### Understand pod states
 Understanding the different pod states is crucial for effective troubleshooting. The following are the most common pod states: