You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/iot-operations/end-to-end-tutorials/tutorial-layered-network-private-connectivity.md
+15-20Lines changed: 15 additions & 20 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -31,12 +31,13 @@ In this article, you:
31
31
32
32
- An [Azure subscription](/azure/cost-management-billing/manage/create-subscription). If you don't have one, [create a free account](https://azure.microsoft.com/free/) before you begin.
33
33
- Your [tenant ID](/azure/azure-portal/get-subscription-tenant-id).
34
-
- Sufficient permissions to create Private Endpoints, Private DNS Zones, and role assignments (typically **Owner** or **Contributor** + **User Access Administrator**). This tutorial uses custom roles defined in the [Appendix](#appendix): **ACX–Secrets Store Extension Owner** and **AdaptiveCloud_AIO–Contributors**.
34
+
- Sufficient permissions to create Private Endpoints, Private DNS Zones, and role assignments (typically **Owner** or **Contributor** + **User Access Administrator**). This tutorial uses custom roles defined in the [Appendix](#appendix).
35
35
- A [Kubernetes cluster](/azure/iot-operations/deploy-iot-ops/howto-prepare-cluster) deployed at each network layer (Level 2, Level 3, and Level 4), with devices or VMs assigned static IPs.
36
-
-Network segmentation between layers (for example, firewalls allowing only L2 ↔ L3 ↔ L4 communication) and DNS resolution across layers using CoreDNS.
36
+
-An existing network segmentation between layers (for example, firewalls allowing only L2 ↔ L3 ↔ L4 communication) with DNS resolution across layers using CoreDNS.
37
37
-[Azure Private Endpoints](/azure/private-link/private-endpoint-overview) for Event Grid, Storage (for Schema Registry), and Key Vault, assigned private IPs and accessible via [ExpressRoute](/azure/expressroute/expressroute-introduction) or equivalent private routing.
38
38
-[Azure Firewall Explicit Proxy](/azure/firewall/explicit-proxy) at Level 4 (ports 8080/8443), reachable from Level 4 over ExpressRoute. All outbound HTTP/HTTPS traffic from Level 4 flows through this proxy.
39
-
-[Azure CLI](/cli/azure/install-azure-cli), [kubectl](https://kubernetes.io/docs/tasks/tools/), and [Docker](https://docs.docker.com/get-docker/) installed on your admin or jump machine.
39
+
-[Azure CLI](/cli/azure/install-azure-cli), [kubectl](https://kubernetes.io/docs/tasks/tools/), and [Docker](https://docs.docker.com/get-docker/) installed on the machine you use to deploy resources and manage Kubernetes clusters.
40
+
- (Optional) Familiarity with the [Purdue Model](https://www.isa.org/isa95/), which defines levels of industrial control systems and is commonly used in manufacturing environments.
40
41
41
42
> [!NOTE]
42
43
> In the validated telemetry flow, only HTTPS (port 8443) was used. In customer environments, Level 4 may route through your own enterprise proxy instead.
@@ -45,7 +46,8 @@ In this article, you:
45
46
46
47
This deployment aligns with the Purdue Model, implementing a physically segmented, multi-level architecture spanning Levels 2 through 4.
47
48
48
-
Each level is separated by network firewalls that restrict communication to adjacent layers only (for example, L2 ↔ L3 ↔ L4), ensuring tight segmentation. Outbound traffic to Azure is proxied through an Arc Gateway and routed to Azure Event Grid over Private Link, ensuring no internet-exposed endpoints are used at any layer.
49
+
Each level is separated by network firewalls that restrict communication to adjacent layers only (for example, L2 ↔ L3 ↔ L4), ensuring tight segmentation.
50
+
Outbound traffic to Azure is routed through an explicit proxy and Private Link (optionally via Arc Gateway), ensuring no internet-exposed endpoints are used at any layer.
49
51
50
52
:::image type="content" source="media/layered-network-private-connectivity-architecture.png" alt-text="Diagram showing Layered Network Guidance for Azure IoT Operations in segmented industrial-style network environments, with a Purdue model pyramid spanning Levels 2 through 5 on the left and an Azure Arc architecture on the right showing CoreDNS, Envoy, and Azure IoT Operations deployed across Levels 3 and 4.":::
51
53
@@ -350,7 +352,7 @@ Verify that:
350
352
> [!NOTE]
351
353
> Arc requires working DNS resolution (via CoreDNS) to complete onboarding.
352
354
353
-
## Arc-enable clusters with Arc Gateway
355
+
## Arc-enable clusters with Arc Gateway (optional)
354
356
355
357
With DNS resolution and private connectivity in place, Arc-enable your Kubernetes clusters behind the explicit proxy. This step connects each cluster to Azure Arc and associates it with the Arc Gateway resource created in [Step 1](#step-1-create-azure-resources).
356
358
@@ -404,26 +406,19 @@ az connectedk8s connect \
404
406
405
407
## Deploy Azure IoT Operations
406
408
407
-
With Arc-enabled clusters at L2 and L3, deploy Azure IoT Operations on each layer. Repeat the deployment for each Arc-enabled cluster. This creates the system-assigned managed identity needed for RBAC assignments in the next section.
408
-
409
-
For full instructions, see [Deploy Azure IoT Operations](/azure/iot-operations/deploy-iot-ops/howto-deploy-iot-operations).
409
+
With Arc-enabled clusters at L2 and L3, deploy Azure IoT Operations on each layer. This creates the system-assigned managed identity needed for RBAC assignments in the next section.
410
410
411
411
> [!IMPORTANT]
412
412
> Use the `--skip-ra` flag when creating the Schema Registry. This prevents the CLI from attempting role assignments that require Owner rights. See [Known limitations](#known-limitations) for details.
413
413
414
-
Deploy on each layer:
415
-
416
-
1.**Level 2:** Switch to the L2 cluster context and deploy AIO:
417
-
418
-
```bash
419
-
kubectl config use-context <L2-cluster>
420
-
```
421
-
422
-
1.**Level 3:** Switch to the L3 cluster context and deploy AIO:
414
+
For each Arc-enabled cluster (L2 and L3), switch to the cluster context and follow the steps in [Deploy Azure IoT Operations](/azure/iot-operations/deploy-iot-ops/howto-deploy-iot-operations):
423
415
424
-
```bash
425
-
kubectl config use-context <L3-cluster>
426
-
```
416
+
```bash
417
+
# For Level 2
418
+
kubectl config use-context <L2-cluster>
419
+
# Follow deploy instructions, then repeat for Level 3
420
+
kubectl config use-context <L3-cluster>
421
+
```
427
422
428
423
After each deployment, verify that all pods are running:
Copy file name to clipboardExpand all lines: articles/iot-operations/manage-layered-network/howto-private-connectivity.md
+20-12Lines changed: 20 additions & 12 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -17,7 +17,7 @@ This article describes how to configure private connectivity for Azure IoT Opera
17
17
18
18
| Step | Section | What it does |
19
19
|------|---------|-------------|
20
-
| 1 |[Connect your cluster via Arc Gateway](#connect-your-cluster-via-arc-gateway)| Create the Arc Gateway resource and retrieve the custom locations OID |
20
+
| 1 |[Set up Arc Gateway](#set-up-arc-gateway)| Create the Arc Gateway resource and retrieve the custom locations OID |
21
21
| 2 |[Create private endpoints and DNS zones](#create-private-endpoints-and-dns-zones)| Create Private Endpoints and DNS zones for the dataplane resources (Storage, Key Vault, Event Grid) you created in the prerequisites |
22
22
| 3 |[Connect the cluster to Azure Arc](#connect-the-cluster-to-azure-arc)| Arc-enable the cluster. Choose between **Arc Gateway only** or **Arc Gateway + Explicit Proxy**|
23
23
| 4 |[Deploy Azure IoT Operations](#deploy-azure-iot-operations)| Deploy Azure IoT Operations. Traffic already routes privately via DNS |
@@ -32,15 +32,12 @@ These scenarios apply to environments with a single Arc-enabled Kubernetes clust
32
32
-[Azure CLI](/cli/azure/install-azure-cli) and [kubectl](https://kubernetes.io/docs/tasks/tools/) installed on your admin or jump machine.
33
33
- A Kubernetes cluster deployed and ready to Arc-enable. See [Prepare your cluster](/azure/iot-operations/deploy-iot-ops/howto-prepare-cluster) for supported configurations and setup steps.
34
34
- An Azure VNet with network connectivity from your cluster ([ExpressRoute](/azure/expressroute/expressroute-introduction), [VPN Gateway](/azure/vpn-gateway/vpn-gateway-about-vpngateways), VNet peering, or other private routing). If your cluster runs on Azure VMs within the same VNet or a peered VNet, this connectivity is already in place.
35
-
36
-
Azure IoT Operations depends on the following **dataplane resources** at runtime. Create them now — you set up Private Endpoints for each one in [Create private endpoints and DNS zones](#create-private-endpoints-and-dns-zones):
37
-
38
35
- An [Azure Storage account](/azure/storage/common/storage-account-create) in the same resource group — used by Schema Registry for schema storage. If you encounter a **RequestDisallowedByPolicy** error during creation, add `--allow-shared-key-access false` to the `az storage account create` command.
39
36
- An [Azure Key Vault](/azure/key-vault/general/quick-create-cli) in the same resource group — used for secret synchronization with your cluster.
40
-
- (Optional) An [Azure Event Grid namespace](/azure/event-grid/create-view-manage-namespaces) with MQTT enabled — needed only if you route dataflow traffic to Event Grid in [Configure dataflow destinations with private endpoints](#configure-dataflow-destinations-with-private-endpoints).
37
+
- (Optional) An [Azure Event Grid namespace](/azure/event-grid/create-view-manage-namespaces) with MQTT enabled. Needed only if you route dataflow traffic to Event Grid in [Configure dataflow destinations with private endpoints](#configure-dataflow-destinations-with-private-endpoints).
41
38
- (Optional) An [Azure Firewall](/azure/firewall/overview) with [explicit proxy](/azure/azure-arc/azure-firewall-explicit-proxy) enabled in your VNet, reachable from your cluster. Required only if you follow the **Arc Gateway + Explicit Proxy** tab for fully private connectivity with no public internet exposure.
42
39
43
-
## Connect your cluster via Arc Gateway
40
+
## Set up Arc Gateway
44
41
45
42
[Azure Arc Gateway](/azure/azure-arc/kubernetes/arc-gateway-simplify-networking) consolidates the ~200+ Azure endpoints that Arc agents and extensions require into a single gateway URL. This significantly simplifies your firewall allowlist, instead of allowing 200+ individual FQDNs, you allow approximately 9.
46
43
@@ -53,22 +50,35 @@ If you don't already have an Arc Gateway resource, create one. You need the gate
53
50
> [!NOTE]
54
51
> A maximum of five Arc Gateway resources are supported per subscription.
55
52
53
+
For the list of FQDNs that must be allowed through your firewall when using Arc Gateway, see [Allowed endpoints with Arc Gateway](/azure/azure-arc/kubernetes/arc-gateway-simplify-networking#allowed-endpoints-with-arc-gateway).
54
+
56
55
### Step 2: Retrieve the custom locations Object ID
57
56
58
57
The `--custom-locations-oid` parameter used when connecting the cluster requires the Object ID (OID) of the Azure Arc Custom Locations service principal.
59
58
60
59
To find it:
61
60
61
+
# [Azure portal](#tab/portal-oid)
62
+
62
63
1. Go to **[Microsoft Entra ID](https://entra.microsoft.com)** in the Azure portal.
63
64
1. Select **Enterprise applications**.
64
65
1. Search for **Azure Arc Kubernetes Custom Locations**.
65
66
1. Open the application, go to **Properties**, and copy the **Object ID**.
66
67
67
-
For the list of FQDNs that must be allowed through your firewall when using Arc Gateway, see [Allowed endpoints with Arc Gateway](/azure/azure-arc/kubernetes/arc-gateway-simplify-networking#allowed-endpoints-with-arc-gateway).
68
+
# [Azure CLI](#tab/cli-oid)
69
+
70
+
```azurecli
71
+
az ad sp show --id bc313c14-388c-4e7d-a58e-70017303ee3b --query id -o tsv
72
+
```
73
+
74
+
> [!IMPORTANT]
75
+
> Don't replace the `--id` value. The GUID `bc313c14-388c-4e7d-a58e-70017303ee3b` is the predefined App ID for the Custom Locations service principal.
76
+
77
+
---
68
78
69
79
## Create private endpoints and DNS zones
70
80
71
-
The storage account, Key Vault, and Event Grid namespace you created in the [prerequisites](#prerequisites) are the dataplane resources that Azure IoT Operations uses at runtime. Create Private Endpoints and DNS zones for these resources now so all traffic routes privately from the start — before you connect the cluster or deploy Azure IoT Operations.
81
+
The storage account, Key Vault, and Event Grid namespace you created in the [prerequisites](#prerequisites) are the dataplane resources that Azure IoT Operations uses at runtime. Create Private Endpoints and DNS zones for these resources now so all traffic routes privately from the start, before you connect the cluster or deploy Azure IoT Operations.
72
82
73
83
### Step 1: Create Private Endpoints
74
84
@@ -195,7 +205,7 @@ With Private Endpoints and DNS in place, connect your cluster to Azure Arc. Choo
195
205
-**Arc Gateway only** — The cluster connects through Arc Gateway with a simplified firewall allowlist (~9 FQDNs), but outbound traffic still uses public internet paths.
196
206
-**Arc Gateway + Explicit Proxy** — All outbound traffic routes through [Azure Firewall Explicit Proxy](/azure/azure-arc/azure-firewall-explicit-proxy) over your private network with no public internet exposure.
197
207
198
-
Both tabs build on [Connect your cluster via Arc Gateway](#connect-your-cluster-via-arc-gateway). Complete that section first to create the Arc Gateway resource and retrieve the custom locations OID.
208
+
Both tabs build on [Set up Arc Gateway](#set-up-arc-gateway). Complete that section first to create the Arc Gateway resource and retrieve the custom locations OID.
199
209
200
210
# [Arc Gateway only](#tab/arc-gateway-only)
201
211
@@ -315,7 +325,7 @@ az connectedk8s connect \
315
325
This command configures all Arc traffic to route through the Azure Firewall Explicit Proxy and the Arc Gateway, consolidating ~200+ endpoints to ~9 allowed FQDNs with no public internet exposure.
316
326
317
327
> [!IMPORTANT]
318
-
> Arc agent traffic — including extension installs and container image pulls from MCR (`mcr.microsoft.com`) — routes through the proxy automatically because `az connectedk8s connect` injects the proxy environment variables into the Arc agent pods. However, if your container runtime (containerd or CRI-O) pulls images outside of the Arc agent (for example, during node-level kubelet pulls), you may also need to configure proxy settings at the node level. On Ubuntu with systemd, create `/etc/systemd/system/containerd.service.d/http-proxy.conf` with your proxy values, then run `sudo systemctl daemon-reload && sudo systemctl restart containerd`.
328
+
> Arc agent traffic, including extension installs and container image pulls from MCR (`mcr.microsoft.com`), routes through the proxy automatically because `az connectedk8s connect` injects the proxy environment variables into the Arc agent pods. However, if your container runtime (containerd or CRI-O) pulls images outside of the Arc agent (for example, during node-level kubelet pulls), you may also need to configure proxy settings at the node level. On Ubuntu with systemd, create `/etc/systemd/system/containerd.service.d/http-proxy.conf` with your proxy values, then run `sudo systemctl daemon-reload && sudo systemctl restart containerd`.
319
329
320
330
> [!TIP]
321
331
> **For existing Arc-enabled clusters:** If your cluster is already connected to Azure Arc, use `az connectedk8s update` instead of `az connectedk8s connect`:
@@ -348,8 +358,6 @@ This command configures all Arc traffic to route through the Azure Firewall Expl
348
358
349
359
Each result should return an IP in your private address range, not a public IP.
350
360
351
-
:::image type="content" source="media/howto-private-connectivity/nslookup-private-dns.jpg" alt-text="Screenshot of nslookup commands showing DNS resolution for Event Grid, Storage, and Key Vault FQDNs.":::
352
-
353
361
1. Verify the cluster appears as **Connected** in the Azure portal under **Azure Arc > Kubernetes clusters**.
354
362
355
363
If any FQDN resolves to a public IP, see [DNS resolves to a public IP instead of a private IP](howto-troubleshoot-private-connectivity.md#dns-resolves-to-a-public-ip-instead-of-a-private-ip).
0 commit comments