Skip to content

Commit 01c8e86

Browse files
fix: private connectivity doc updates - max sessions warning, test connection link, remove bad screenshot
1 parent b6e9858 commit 01c8e86

3 files changed

Lines changed: 35 additions & 32 deletions

File tree

articles/iot-operations/end-to-end-tutorials/tutorial-layered-network-private-connectivity.md

Lines changed: 15 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -31,12 +31,13 @@ In this article, you:
3131

3232
- An [Azure subscription](/azure/cost-management-billing/manage/create-subscription). If you don't have one, [create a free account](https://azure.microsoft.com/free/) before you begin.
3333
- Your [tenant ID](/azure/azure-portal/get-subscription-tenant-id).
34-
- Sufficient permissions to create Private Endpoints, Private DNS Zones, and role assignments (typically **Owner** or **Contributor** + **User Access Administrator**). This tutorial uses custom roles defined in the [Appendix](#appendix): **ACX–Secrets Store Extension Owner** and **AdaptiveCloud_AIO–Contributors**.
34+
- Sufficient permissions to create Private Endpoints, Private DNS Zones, and role assignments (typically **Owner** or **Contributor** + **User Access Administrator**). This tutorial uses custom roles defined in the [Appendix](#appendix).
3535
- A [Kubernetes cluster](/azure/iot-operations/deploy-iot-ops/howto-prepare-cluster) deployed at each network layer (Level 2, Level 3, and Level 4), with devices or VMs assigned static IPs.
36-
- Network segmentation between layers (for example, firewalls allowing only L2 ↔ L3 ↔ L4 communication) and DNS resolution across layers using CoreDNS.
36+
- An existing network segmentation between layers (for example, firewalls allowing only L2 ↔ L3 ↔ L4 communication) with DNS resolution across layers using CoreDNS.
3737
- [Azure Private Endpoints](/azure/private-link/private-endpoint-overview) for Event Grid, Storage (for Schema Registry), and Key Vault, assigned private IPs and accessible via [ExpressRoute](/azure/expressroute/expressroute-introduction) or equivalent private routing.
3838
- [Azure Firewall Explicit Proxy](/azure/firewall/explicit-proxy) at Level 4 (ports 8080/8443), reachable from Level 4 over ExpressRoute. All outbound HTTP/HTTPS traffic from Level 4 flows through this proxy.
39-
- [Azure CLI](/cli/azure/install-azure-cli), [kubectl](https://kubernetes.io/docs/tasks/tools/), and [Docker](https://docs.docker.com/get-docker/) installed on your admin or jump machine.
39+
- [Azure CLI](/cli/azure/install-azure-cli), [kubectl](https://kubernetes.io/docs/tasks/tools/), and [Docker](https://docs.docker.com/get-docker/) installed on the machine you use to deploy resources and manage Kubernetes clusters.
40+
- (Optional) Familiarity with the [Purdue Model](https://www.isa.org/isa95/), which defines levels of industrial control systems and is commonly used in manufacturing environments.
4041

4142
> [!NOTE]
4243
> In the validated telemetry flow, only HTTPS (port 8443) was used. In customer environments, Level 4 may route through your own enterprise proxy instead.
@@ -45,7 +46,8 @@ In this article, you:
4546

4647
This deployment aligns with the Purdue Model, implementing a physically segmented, multi-level architecture spanning Levels 2 through 4.
4748

48-
Each level is separated by network firewalls that restrict communication to adjacent layers only (for example, L2 ↔ L3 ↔ L4), ensuring tight segmentation. Outbound traffic to Azure is proxied through an Arc Gateway and routed to Azure Event Grid over Private Link, ensuring no internet-exposed endpoints are used at any layer.
49+
Each level is separated by network firewalls that restrict communication to adjacent layers only (for example, L2 ↔ L3 ↔ L4), ensuring tight segmentation.
50+
Outbound traffic to Azure is routed through an explicit proxy and Private Link (optionally via Arc Gateway), ensuring no internet-exposed endpoints are used at any layer.
4951

5052
:::image type="content" source="media/layered-network-private-connectivity-architecture.png" alt-text="Diagram showing Layered Network Guidance for Azure IoT Operations in segmented industrial-style network environments, with a Purdue model pyramid spanning Levels 2 through 5 on the left and an Azure Arc architecture on the right showing CoreDNS, Envoy, and Azure IoT Operations deployed across Levels 3 and 4.":::
5153

@@ -350,7 +352,7 @@ Verify that:
350352
> [!NOTE]
351353
> Arc requires working DNS resolution (via CoreDNS) to complete onboarding.
352354
353-
## Arc-enable clusters with Arc Gateway
355+
## Arc-enable clusters with Arc Gateway (optional)
354356

355357
With DNS resolution and private connectivity in place, Arc-enable your Kubernetes clusters behind the explicit proxy. This step connects each cluster to Azure Arc and associates it with the Arc Gateway resource created in [Step 1](#step-1-create-azure-resources).
356358

@@ -404,26 +406,19 @@ az connectedk8s connect \
404406

405407
## Deploy Azure IoT Operations
406408

407-
With Arc-enabled clusters at L2 and L3, deploy Azure IoT Operations on each layer. Repeat the deployment for each Arc-enabled cluster. This creates the system-assigned managed identity needed for RBAC assignments in the next section.
408-
409-
For full instructions, see [Deploy Azure IoT Operations](/azure/iot-operations/deploy-iot-ops/howto-deploy-iot-operations).
409+
With Arc-enabled clusters at L2 and L3, deploy Azure IoT Operations on each layer. This creates the system-assigned managed identity needed for RBAC assignments in the next section.
410410

411411
> [!IMPORTANT]
412412
> Use the `--skip-ra` flag when creating the Schema Registry. This prevents the CLI from attempting role assignments that require Owner rights. See [Known limitations](#known-limitations) for details.
413413
414-
Deploy on each layer:
415-
416-
1. **Level 2:** Switch to the L2 cluster context and deploy AIO:
417-
418-
```bash
419-
kubectl config use-context <L2-cluster>
420-
```
421-
422-
1. **Level 3:** Switch to the L3 cluster context and deploy AIO:
414+
For each Arc-enabled cluster (L2 and L3), switch to the cluster context and follow the steps in [Deploy Azure IoT Operations](/azure/iot-operations/deploy-iot-ops/howto-deploy-iot-operations):
423415

424-
```bash
425-
kubectl config use-context <L3-cluster>
426-
```
416+
```bash
417+
# For Level 2
418+
kubectl config use-context <L2-cluster>
419+
# Follow deploy instructions, then repeat for Level 3
420+
kubectl config use-context <L3-cluster>
421+
```
427422

428423
After each deployment, verify that all pods are running:
429424

articles/iot-operations/manage-layered-network/howto-private-connectivity.md

Lines changed: 20 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ This article describes how to configure private connectivity for Azure IoT Opera
1717

1818
| Step | Section | What it does |
1919
|------|---------|-------------|
20-
| 1 | [Connect your cluster via Arc Gateway](#connect-your-cluster-via-arc-gateway) | Create the Arc Gateway resource and retrieve the custom locations OID |
20+
| 1 | [Set up Arc Gateway](#set-up-arc-gateway) | Create the Arc Gateway resource and retrieve the custom locations OID |
2121
| 2 | [Create private endpoints and DNS zones](#create-private-endpoints-and-dns-zones) | Create Private Endpoints and DNS zones for the dataplane resources (Storage, Key Vault, Event Grid) you created in the prerequisites |
2222
| 3 | [Connect the cluster to Azure Arc](#connect-the-cluster-to-azure-arc) | Arc-enable the cluster. Choose between **Arc Gateway only** or **Arc Gateway + Explicit Proxy** |
2323
| 4 | [Deploy Azure IoT Operations](#deploy-azure-iot-operations) | Deploy Azure IoT Operations. Traffic already routes privately via DNS |
@@ -32,15 +32,12 @@ These scenarios apply to environments with a single Arc-enabled Kubernetes clust
3232
- [Azure CLI](/cli/azure/install-azure-cli) and [kubectl](https://kubernetes.io/docs/tasks/tools/) installed on your admin or jump machine.
3333
- A Kubernetes cluster deployed and ready to Arc-enable. See [Prepare your cluster](/azure/iot-operations/deploy-iot-ops/howto-prepare-cluster) for supported configurations and setup steps.
3434
- An Azure VNet with network connectivity from your cluster ([ExpressRoute](/azure/expressroute/expressroute-introduction), [VPN Gateway](/azure/vpn-gateway/vpn-gateway-about-vpngateways), VNet peering, or other private routing). If your cluster runs on Azure VMs within the same VNet or a peered VNet, this connectivity is already in place.
35-
36-
Azure IoT Operations depends on the following **dataplane resources** at runtime. Create them now — you set up Private Endpoints for each one in [Create private endpoints and DNS zones](#create-private-endpoints-and-dns-zones):
37-
3835
- An [Azure Storage account](/azure/storage/common/storage-account-create) in the same resource group — used by Schema Registry for schema storage. If you encounter a **RequestDisallowedByPolicy** error during creation, add `--allow-shared-key-access false` to the `az storage account create` command.
3936
- An [Azure Key Vault](/azure/key-vault/general/quick-create-cli) in the same resource group — used for secret synchronization with your cluster.
40-
- (Optional) An [Azure Event Grid namespace](/azure/event-grid/create-view-manage-namespaces) with MQTT enabled — needed only if you route dataflow traffic to Event Grid in [Configure dataflow destinations with private endpoints](#configure-dataflow-destinations-with-private-endpoints).
37+
- (Optional) An [Azure Event Grid namespace](/azure/event-grid/create-view-manage-namespaces) with MQTT enabled. Needed only if you route dataflow traffic to Event Grid in [Configure dataflow destinations with private endpoints](#configure-dataflow-destinations-with-private-endpoints).
4138
- (Optional) An [Azure Firewall](/azure/firewall/overview) with [explicit proxy](/azure/azure-arc/azure-firewall-explicit-proxy) enabled in your VNet, reachable from your cluster. Required only if you follow the **Arc Gateway + Explicit Proxy** tab for fully private connectivity with no public internet exposure.
4239

43-
## Connect your cluster via Arc Gateway
40+
## Set up Arc Gateway
4441

4542
[Azure Arc Gateway](/azure/azure-arc/kubernetes/arc-gateway-simplify-networking) consolidates the ~200+ Azure endpoints that Arc agents and extensions require into a single gateway URL. This significantly simplifies your firewall allowlist, instead of allowing 200+ individual FQDNs, you allow approximately 9.
4643

@@ -53,22 +50,35 @@ If you don't already have an Arc Gateway resource, create one. You need the gate
5350
> [!NOTE]
5451
> A maximum of five Arc Gateway resources are supported per subscription.
5552
53+
For the list of FQDNs that must be allowed through your firewall when using Arc Gateway, see [Allowed endpoints with Arc Gateway](/azure/azure-arc/kubernetes/arc-gateway-simplify-networking#allowed-endpoints-with-arc-gateway).
54+
5655
### Step 2: Retrieve the custom locations Object ID
5756

5857
The `--custom-locations-oid` parameter used when connecting the cluster requires the Object ID (OID) of the Azure Arc Custom Locations service principal.
5958

6059
To find it:
6160

61+
# [Azure portal](#tab/portal-oid)
62+
6263
1. Go to **[Microsoft Entra ID](https://entra.microsoft.com)** in the Azure portal.
6364
1. Select **Enterprise applications**.
6465
1. Search for **Azure Arc Kubernetes Custom Locations**.
6566
1. Open the application, go to **Properties**, and copy the **Object ID**.
6667

67-
For the list of FQDNs that must be allowed through your firewall when using Arc Gateway, see [Allowed endpoints with Arc Gateway](/azure/azure-arc/kubernetes/arc-gateway-simplify-networking#allowed-endpoints-with-arc-gateway).
68+
# [Azure CLI](#tab/cli-oid)
69+
70+
```azurecli
71+
az ad sp show --id bc313c14-388c-4e7d-a58e-70017303ee3b --query id -o tsv
72+
```
73+
74+
> [!IMPORTANT]
75+
> Don't replace the `--id` value. The GUID `bc313c14-388c-4e7d-a58e-70017303ee3b` is the predefined App ID for the Custom Locations service principal.
76+
77+
---
6878

6979
## Create private endpoints and DNS zones
7080

71-
The storage account, Key Vault, and Event Grid namespace you created in the [prerequisites](#prerequisites) are the dataplane resources that Azure IoT Operations uses at runtime. Create Private Endpoints and DNS zones for these resources now so all traffic routes privately from the start before you connect the cluster or deploy Azure IoT Operations.
81+
The storage account, Key Vault, and Event Grid namespace you created in the [prerequisites](#prerequisites) are the dataplane resources that Azure IoT Operations uses at runtime. Create Private Endpoints and DNS zones for these resources now so all traffic routes privately from the start, before you connect the cluster or deploy Azure IoT Operations.
7282

7383
### Step 1: Create Private Endpoints
7484

@@ -195,7 +205,7 @@ With Private Endpoints and DNS in place, connect your cluster to Azure Arc. Choo
195205
- **Arc Gateway only** — The cluster connects through Arc Gateway with a simplified firewall allowlist (~9 FQDNs), but outbound traffic still uses public internet paths.
196206
- **Arc Gateway + Explicit Proxy** — All outbound traffic routes through [Azure Firewall Explicit Proxy](/azure/azure-arc/azure-firewall-explicit-proxy) over your private network with no public internet exposure.
197207

198-
Both tabs build on [Connect your cluster via Arc Gateway](#connect-your-cluster-via-arc-gateway). Complete that section first to create the Arc Gateway resource and retrieve the custom locations OID.
208+
Both tabs build on [Set up Arc Gateway](#set-up-arc-gateway). Complete that section first to create the Arc Gateway resource and retrieve the custom locations OID.
199209

200210
# [Arc Gateway only](#tab/arc-gateway-only)
201211

@@ -315,7 +325,7 @@ az connectedk8s connect \
315325
This command configures all Arc traffic to route through the Azure Firewall Explicit Proxy and the Arc Gateway, consolidating ~200+ endpoints to ~9 allowed FQDNs with no public internet exposure.
316326

317327
> [!IMPORTANT]
318-
> Arc agent trafficincluding extension installs and container image pulls from MCR (`mcr.microsoft.com`) routes through the proxy automatically because `az connectedk8s connect` injects the proxy environment variables into the Arc agent pods. However, if your container runtime (containerd or CRI-O) pulls images outside of the Arc agent (for example, during node-level kubelet pulls), you may also need to configure proxy settings at the node level. On Ubuntu with systemd, create `/etc/systemd/system/containerd.service.d/http-proxy.conf` with your proxy values, then run `sudo systemctl daemon-reload && sudo systemctl restart containerd`.
328+
> Arc agent traffic, including extension installs and container image pulls from MCR (`mcr.microsoft.com`), routes through the proxy automatically because `az connectedk8s connect` injects the proxy environment variables into the Arc agent pods. However, if your container runtime (containerd or CRI-O) pulls images outside of the Arc agent (for example, during node-level kubelet pulls), you may also need to configure proxy settings at the node level. On Ubuntu with systemd, create `/etc/systemd/system/containerd.service.d/http-proxy.conf` with your proxy values, then run `sudo systemctl daemon-reload && sudo systemctl restart containerd`.
319329
320330
> [!TIP]
321331
> **For existing Arc-enabled clusters:** If your cluster is already connected to Azure Arc, use `az connectedk8s update` instead of `az connectedk8s connect`:
@@ -348,8 +358,6 @@ This command configures all Arc traffic to route through the Azure Firewall Expl
348358

349359
Each result should return an IP in your private address range, not a public IP.
350360

351-
:::image type="content" source="media/howto-private-connectivity/nslookup-private-dns.jpg" alt-text="Screenshot of nslookup commands showing DNS resolution for Event Grid, Storage, and Key Vault FQDNs.":::
352-
353361
1. Verify the cluster appears as **Connected** in the Azure portal under **Azure Arc > Kubernetes clusters**.
354362

355363
If any FQDN resolves to a public IP, see [DNS resolves to a public IP instead of a private IP](howto-troubleshoot-private-connectivity.md#dns-resolves-to-a-public-ip-instead-of-a-private-ip).

0 commit comments

Comments
 (0)