Add upgrade known issue

Author: dominicbetts

articles/iot-operations/troubleshoot/known-issues.md

@@ -5,7 +5,7 @@ author: dominicbetts
 ms.author: dobett
 ms.topic: troubleshooting-known-issue
 ms.custom: sfi-ropc-nochange
-ms.date: 11/21/2025
+ms.date: 04/17/2026
 ---
 
 # Known issues for Azure IoT Operations
@@ -14,6 +14,67 @@ This article lists the current known issues you might encounter when using Azure
 
 For general troubleshooting guidance, see [Troubleshoot Azure IoT Operations](troubleshoot.md).
 
+## Deployment and upgrade issues
+
+This section lists current known issues with deploying and upgrading Azure IoT Operations.
+
+### Upgrade to Azure IoT Operations 2603 can silently fail
+
+---
+
+Log signature: N/A
+
+---
+
+When you run `az iot ops upgrade` to upgrade to Azure IoT Operations 2603, the upgrade can silently fail to reach the cluster. You then observe the following symptoms:
+ 
+- `provisioningState: Failed` on the Azure IoT Operations extension.
+- All on-cluster workloads remain healthy (no upgrade activity occurs).
+- `az iot ops upgrade` might report nothing to upgrade on subsequent attempts.
+
+#### Root cause
+ 
+During the upgrade, if a dependent system extension, such as `microsoft.extensiondiagnostics` experiences a transient Helm timeout, Azure Resource Manager marks it as **Failed**. Even if the extension eventually succeeds on-cluster, the cloud-side state remains **Failed**. This blocks the dependency chain — Azure Resource Manager never delivers the updated Azure IoT Operations or secret-store extension config to the cluster's config agent.
+ 
+Symptoms include:
+ 
+- Config agent PostStatus returns `400: "Configuration spec has been modified"`
+- `getPendingConfigs` returns empty results
+- Extension manager never receives Helm upgrade instructions
+
+#### Workaround
+ 
+The workaround is to force Azure Resource Manager to re-submit the extension specs by running a no-op update on both the Azure IoT Operations and secret-store extensions, then retrying the upgrade:
+ 
+```azurecli
+az k8s-extension update --name <aio-extension-name> \
+    --cluster-name <cluster-name> \
+    --resource-group <resource-group> \
+    --cluster-type connectedClusters \
+    --configuration-settings AgentOperationTimeoutInMinutes=120
+
+az k8s-extension update --name azure-secret-store \
+    --cluster-name <cluster-name> \
+    --resource-group <resource-group> \
+    --cluster-type connectedClusters \
+    --configuration-settings AgentOperationTimeoutInMinutes=120
+
+az iot ops upgrade
+```
+
+To identify the Azure IoT Operations extension name, which includes a random suffix (for example, `azure-iot-operations-cym7h`), find your specific extension name by running:
+
+```azurecli
+az k8s-extension list \
+    --cluster-name <cluster-name> \
+    --resource-group <resource-group> \
+    --cluster-type connectedClusters \
+    --query "[?extensionType=='microsoft.iotoperations'].name" -o tsv
+```
+
+> [!IMPORTANT]
+> After the upgrade completes, reset `AgentOperationTimeoutInMinutes` back to a lower value like five minutes to avoid long wait times on future operations if something else fails.
+
 ## Azure Device Registry issues
 
 This section lists current known issues for the Azure Device Registry.