You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/iot-operations/connect-to-cloud/howto-configure-dataflow-endpoint.md
+3Lines changed: 3 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -268,6 +268,9 @@ Similar to the MQTT example, you can create multiple data flows that use the sam
268
268
269
269
## Next steps
270
270
271
+
> [!TIP]
272
+
> To route dataflow traffic to cloud destinations through Private Link instead of public endpoints, see [Configure dataflow destinations with private endpoints](../manage-layered-network/howto-private-connectivity.md#configure-dataflow-destinations-with-private-endpoints).
273
+
271
274
Create a data flow endpoint:
272
275
273
276
- [MQTT or Event Grid](howto-configure-mqtt-endpoint.md)
Copy file name to clipboardExpand all lines: articles/iot-operations/end-to-end-tutorials/tutorial-layered-network-private-connectivity.md
+22-20Lines changed: 22 additions & 20 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -8,13 +8,14 @@ ms.date: 03/24/2026
8
8
9
9
#CustomerIntent: As an operator in an industrial environment with Purdue-style network segmentation, I want to deploy Azure IoT Operations with private Azure connectivity so that no endpoints are exposed to the public internet.
10
10
ms.service: azure-iot-operations
11
+
ms.subservice: layered-network-management
11
12
---
12
13
13
14
# Tutorial: Deploy Azure IoT Operations in a layered network with private connectivity
14
15
15
-
This tutorial describes how to deploy Azure IoT Operations in a physically layered network topology with explicit proxy routing and private connectivity to Azure services via ExpressRoute. This deployment uses Private Link to reach services like Event Grid, and exposes no public endpoints at any network layer.
16
+
This tutorial describes how to deploy Azure IoT Operations in a physically layered network topology with explicit proxy routing and private connectivity to Azure services through ExpressRoute. This deployment uses Private Link to reach services like Event Grid, and exposes no public endpoints at any network layer.
16
17
17
-
This scenario was validated using physical machines in a Purdue/ISA-95 segmented network spanning Levels 2 through 4. The Azure Firewall Explicit Proxy is deployed in an Azure VNet, with connectivity provided via ExpressRoute.
18
+
This scenario was validated by using physical machines in a Purdue/ISA-95 segmented network spanning Levels 2 through 4. The Azure Firewall Explicit Proxy is deployed in an Azure VNet, with connectivity provided through ExpressRoute.
18
19
19
20
In this article, you:
20
21
@@ -34,20 +35,20 @@ In this article, you:
34
35
- Sufficient permissions to create Private Endpoints, Private DNS Zones, and role assignments (typically **Owner** or **Contributor** + **User Access Administrator**). This tutorial uses custom roles defined in the [Appendix](#appendix).
35
36
- A [Kubernetes cluster](/azure/iot-operations/deploy-iot-ops/howto-prepare-cluster) deployed at each network layer (Level 2, Level 3, and Level 4), with devices or VMs assigned static IPs.
36
37
- An existing network segmentation between layers (for example, firewalls allowing only L2 ↔ L3 ↔ L4 communication) with DNS resolution across layers using CoreDNS.
37
-
-[Azure Private Endpoints](/azure/private-link/private-endpoint-overview) for Event Grid, Storage (for Schema Registry), and Key Vault, assigned private IPs and accessible via[ExpressRoute](/azure/expressroute/expressroute-introduction) or equivalent private routing.
38
+
-[Azure Private Endpoints](/azure/private-link/private-endpoint-overview) for Event Grid, Storage (for Schema Registry), and Key Vault, assigned private IPs and accessible through[ExpressRoute](/azure/expressroute/expressroute-introduction) or equivalent private routing.
38
39
-[Azure Firewall Explicit Proxy](/azure/firewall/explicit-proxy) at Level 4 (ports 8080/8443), reachable from Level 4 over ExpressRoute. All outbound HTTP/HTTPS traffic from Level 4 flows through this proxy.
39
40
-[Azure CLI](/cli/azure/install-azure-cli), [kubectl](https://kubernetes.io/docs/tasks/tools/), and [Docker](https://docs.docker.com/get-docker/) installed on the machine you use to deploy resources and manage Kubernetes clusters.
40
41
- (Optional) Familiarity with the [Purdue Model](https://www.isa.org/isa95/), which defines levels of industrial control systems and is commonly used in manufacturing environments.
41
42
42
43
> [!NOTE]
43
-
> In the validated telemetry flow, only HTTPS (port 8443) was used. In customer environments, Level 4 may route through your own enterprise proxy instead.
44
+
> In the validated telemetry flow, only HTTPS (port 8443) is used. In customer environments, Level 4 might route through your own enterprise proxy instead.
44
45
45
46
## Architecture summary
46
47
47
48
This deployment aligns with the Purdue Model, implementing a physically segmented, multi-level architecture spanning Levels 2 through 4.
48
49
49
50
Each level is separated by network firewalls that restrict communication to adjacent layers only (for example, L2 ↔ L3 ↔ L4), ensuring tight segmentation.
50
-
Outbound traffic to Azure is routed through an explicit proxy and Private Link (optionally via Arc Gateway), ensuring no internet-exposed endpoints are used at any layer.
51
+
Outbound traffic to Azure is routed through an explicit proxy and Private Link (optionally through Arc Gateway), ensuring no internet-exposed endpoints are used at any layer.
51
52
52
53
:::image type="content" source="media/layered-network-private-connectivity-architecture.png" alt-text="Diagram showing Layered Network Guidance for Azure IoT Operations in segmented industrial-style network environments, with a Purdue model pyramid spanning Levels 2 through 5 on the left and an Azure Arc architecture on the right showing CoreDNS, Envoy, and Azure IoT Operations deployed across Levels 3 and 4.":::
53
54
@@ -57,7 +58,7 @@ Outbound traffic to Azure is routed through an explicit proxy and Private Link (
57
58
| ----- | ---------- | ------- |
58
59
| L2 | CoreDNS, Azure IoT Operations Dataflows, Azure IoT Operations MQTT Broker | Ingests telemetry from OPC UA sources, applies initial enrichment, and forwards data upward |
59
60
| L3 | CoreDNS, Envoy Proxy, Azure IoT Operations Dataflows, Azure IoT Operations MQTT Broker | Aggregates and transforms data, resolves DNS to reach L4, and securely forwards telemetry |
60
-
| L4 | Envoy Proxy | Forwards enriched telemetry to Event Grid via Azure Firewall Explicit Proxy and Private Endpoint over ExpressRoute |
61
+
| L4 | Envoy Proxy | Forwards enriched telemetry to Event Grid through Azure Firewall Explicit Proxy and Private Endpoint over ExpressRoute |
| L4 | Envoy Proxy (egress only, outbound access) | p3tiny-03 | 172.22.232.Z | Not Arc-enabled, handles egress via Azure Firewall Explicit Proxy over ExpressRoute |
120
+
| L4 | Envoy Proxy (egress only, outbound access) | p3tiny-03 | 172.22.232.Z | Not Arc-enabled, handles egress through Azure Firewall Explicit Proxy over ExpressRoute |
120
121
121
122
### Step 3: Enforce network isolation between layers
122
123
@@ -128,7 +129,7 @@ Use firewalls or host-level policies (for example, [UFW](https://help.ubuntu.com
128
129
| L3 ↔ L4 | Allow |
129
130
| L2 ↔ L4 | Block |
130
131
| L2/L3 → Internet | Block |
131
-
| L4 → Azure | Allow via Azure Firewall Explicit Proxy over ExpressRoute |
132
+
| L4 → Azure | Allow through Azure Firewall Explicit Proxy over ExpressRoute |
132
133
133
134
Example UFW rules for the L2 host (allow L3 only, deny everything else):
134
135
@@ -146,7 +147,7 @@ Repeat with appropriate rules on each host. See [UFW documentation](https://help
146
147
147
148
### Step 4: Route Azure-bound traffic through L4 only
148
149
149
-
Only the Level 4 node may initiate outbound traffic, forwarding it to the [Azure Firewall Explicit Proxy](/azure/firewall/explicit-proxy) over ExpressRoute, which then routes it to Azure services via Private Link.
150
+
Only the Level 4 node might initiate outbound traffic, forwarding it to the [Azure Firewall Explicit Proxy](/azure/firewall/explicit-proxy) over ExpressRoute, which then routes it to Azure services through Private Link.
150
151
151
152
1. Deploy Envoy Proxy on the L4 machine. For sample configurations, see [Configure infrastructure](https://github.com/Azure-Samples/explore-iot-operations/blob/main/samples/layered-networking/configure-infrastructure.md).
152
153
1. Forward HTTPS traffic to the Azure Firewall Explicit Proxy:
@@ -261,7 +262,7 @@ For the full list of private DNS zone names, see [Azure Private DNS Zone values]
261
262
Deploy [CoreDNS](https://coredns.io/) on L2 and L3 to forward private Azure domain queries to `168.63.129.16`, Azure's internal DNS resolver for Private Endpoint domains. For deployment instructions, see [Configure infrastructure](https://github.com/Azure-Samples/explore-iot-operations/blob/main/samples/layered-networking/configure-infrastructure.md).
262
263
263
264
> [!NOTE]
264
-
> `168.63.129.16` is Azure's internal wire server DNS and is only reachable from inside an Azure VNet. In this lab, L4 is a VNet-hosted VM, so CoreDNS queries from L2/L3 are forwarded through Envoy at L3 and L4, ultimately reaching `168.63.129.16`via the L4 VNet.
265
+
> `168.63.129.16` is Azure's internal wire server DNS and is only reachable from inside an Azure VNet. In this lab, L4 is a VNet-hosted VM, so CoreDNS queries from L2/L3 are forwarded through Envoy at L3 and L4, ultimately reaching `168.63.129.16`through the L4 VNet.
@@ -518,12 +519,12 @@ The end-to-end telemetry flow follows this path:
518
519
1.**DNS resolution (L3):** Azure IoT Operations Dataflows and CoreDNS at L3 resolve private service names to reach L4.
519
520
1.**Proxy forwarding (L3 to L4):** Envoy Proxy on L3 forwards MQTT traffic to Envoy Proxy on L4.
520
521
1.**Egress (L4):** Envoy Proxy on L4 sends traffic to the Azure Firewall Explicit Proxy on port 8443 over ExpressRoute.
521
-
1.**Private routing:** The proxy routes requests to Azure services via Private Endpoints.
522
+
1.**Private routing:** The proxy routes requests to Azure services through Private Endpoints.
522
523
1.**Cloud integration:** Services such as Event Grid Topic Spaces, Azure Storage, and Azure Key Vault are accessed privately using Azure Private Link. Public network access is disabled for all Azure services in the deployment.
523
524
524
525
### Event Grid topic spaces
525
526
526
-
Data is published to Event Grid via MQTT over WebSocket (`/mqtt` path suffix). L3's Azure IoT Operations Dataflow authenticates using its system-assigned managed identity (which has EventGrid TopicSpaces Publisher and Subscriber roles) and sends traffic through Envoy Proxy on L3, which forwards to Envoy Proxy on L4. Outbound traffic from Level 4 is routed through the Azure Firewall Explicit Proxy (port 8443), which then forwards to the Event Grid private endpoint over ExpressRoute. Level 4 acts as a pure pass-through and doesn't authenticate.
527
+
Data is published to Event Grid through MQTT over WebSocket (`/mqtt` path suffix). L3's Azure IoT Operations Dataflow authenticates using its system-assigned managed identity (which has EventGrid TopicSpaces Publisher and Subscriber roles) and sends traffic through Envoy Proxy on L3, which forwards to Envoy Proxy on L4. Outbound traffic from Level 4 is routed through the Azure Firewall Explicit Proxy (port 8443), which then forwards to the Event Grid private endpoint over ExpressRoute. Level 4 acts as a pure pass-through and doesn't authenticate.
527
528
528
529
Telemetry is validated against schemas defined in Blob Storage, enforced by the Azure IoT Operations Dataflows running at L2 and L3.
529
530
@@ -636,9 +637,10 @@ If any query returns a public IP, check your CoreDNS forwarding rules and Privat
636
637
637
638
## Known limitations
638
639
639
-
-**Platform validation:** This scenario was validated on K3s running on Ubuntu Server 24.04 only. Other Kubernetes distributions or operating systems haven't been validated.
640
-
-**Proxy support:** Only Azure Firewall Explicit Proxy was validated. Third-party proxies (for example, Palo Alto) or transparent proxies aren't supported in this validated scenario.
641
-
-**Schema Registry public access:** Schema Registry may require public access enabled at creation time. After creation, you can disable public access. Use the `--skip-ra` flag when creating the Schema Registry to avoid requiring Owner-level permissions.
640
+
For common limitations related to platform validation, proxy support, and Schema Registry, see [Known limitations in Deploy Azure IoT Operations with private connectivity](../manage-layered-network/howto-private-connectivity.md#known-limitations).
641
+
642
+
The following limitations are specific to the layered network tutorial:
643
+
642
644
-**Level 1:** The L1 device layer is unused in this deployment flow.
643
645
-**Level 4 Arc:** Level 4 is not Arc-enabled; only Envoy Proxy is deployed at this layer.
644
646
-**Sovereign clouds:** This scenario was validated in Azure public cloud only. Sovereign cloud environments (for example, Azure Government, Azure China 21Vianet) use different endpoints and Private DNS Zone names and haven't been validated.
> In environments using Azure Policy automation, these manual role definitions may not be required for OT teams. Policies can pre-assign Contributor or Storage Blob Data Contributor roles as needed.
725
+
> In environments using Azure Policy automation, these manual role definitions might not be required for OT teams. Policies can pre-assign Contributor or Storage Blob Data Contributor roles as needed.
Copy file name to clipboardExpand all lines: articles/iot-operations/manage-layered-network/howto-private-connectivity.md
+4-4Lines changed: 4 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -45,7 +45,7 @@ These scenarios apply to environments with a single Arc-enabled Kubernetes clust
45
45
46
46
If you don't already have an Arc Gateway resource, create one. You need the gateway resource ID when you connect the cluster in the next section. For creation steps, see [Create the Arc Gateway resource](/azure/azure-arc/kubernetes/arc-gateway-simplify-networking#create-the-arc-gateway-resource).
47
47
48
-
:::image type="content" source="media/howto-private-connectivity/arc-gateway-portal.jpg" alt-text="Screenshot of the Azure portal showing an Arc Gateway resource with its Gateway URL and resource properties.":::
48
+
:::image type="content" source="media/howto-private-connectivity/arc-gateway-portal.png" alt-text="Screenshot of the Azure portal showing an Arc Gateway resource with its Gateway URL and resource properties.":::
49
49
50
50
> [!NOTE]
51
51
> A maximum of five Arc Gateway resources are supported per subscription.
@@ -325,7 +325,7 @@ az connectedk8s connect \
325
325
This command configures all Arc traffic to route through the Azure Firewall Explicit Proxy and the Arc Gateway, consolidating ~200+ endpoints to ~9 allowed FQDNs with no public internet exposure.
326
326
327
327
> [!IMPORTANT]
328
-
> Arc agent traffic, including extension installs and container image pulls from MCR (`mcr.microsoft.com`), routes through the proxy automatically because `az connectedk8s connect` injects the proxy environment variables into the Arc agent pods. However, if your container runtime (containerd or CRI-O) pulls images outside of the Arc agent (for example, during node-level kubelet pulls), you may also need to configure proxy settings at the node level. On Ubuntu with systemd, create `/etc/systemd/system/containerd.service.d/http-proxy.conf` with your proxy values, then run `sudo systemctl daemon-reload && sudo systemctl restart containerd`.
328
+
> Arc agent traffic, including extension installs and container image pulls from MCR (`mcr.microsoft.com`), routes through the proxy automatically because `az connectedk8s connect` injects the proxy environment variables into the Arc agent pods. However, if your container runtime (containerd or CRI-O) pulls images outside of the Arc agent (for example, during node-level kubelet pulls), you might also need to configure proxy settings at the node level. On Ubuntu with systemd, create `/etc/systemd/system/containerd.service.d/http-proxy.conf` with your proxy values, then run `sudo systemctl daemon-reload && sudo systemctl restart containerd`.
329
329
330
330
> [!TIP]
331
331
> **For existing Arc-enabled clusters:** If your cluster is already connected to Azure Arc, use `az connectedk8s update` instead of `az connectedk8s connect`:
@@ -685,7 +685,7 @@ Then check the dataflow is working:
685
685
1. Navigate to your Event Grid namespace in the Azure portal.
686
686
1. Check **Metrics** for incoming MQTT messages.
687
687
688
-
:::image type="content" source="media/howto-private-connectivity/event-grid-messaging-metrics.jpg" alt-text="Screenshot of Event Grid namespace metrics showing successful MQTT published messages.":::
688
+
:::image type="content" source="media/howto-private-connectivity/event-grid-messaging-metrics.png" alt-text="Screenshot of Event Grid namespace metrics showing successful MQTT published messages.":::
689
689
690
690
1. Verify the dataflow pod logs show successful message delivery:
691
691
@@ -700,7 +700,7 @@ After disabling public access on any Azure resource, verify Azure IoT Operations
700
700
## Known limitations
701
701
702
702
-**Platform validation:** The private connectivity patterns described here are based on validated K3s on Ubuntu Server 24.04 scenarios. Other Kubernetes distributions or operating systems haven't been independently validated.
703
-
-**Schema Registry creation:** Schema Registry may require public access enabled at creation time. After creation, you can disable public access and rely on Private Endpoints plus trusted service bypass. Use the `--skip-ra` flag when creating the Schema Registry to avoid requiring Owner-level permissions.
703
+
-**Schema Registry creation:** Schema Registry might require public access enabled at creation time. After creation, you can disable public access and rely on Private Endpoints plus trusted service bypass. Use the `--skip-ra` flag when creating the Schema Registry to avoid requiring Owner-level permissions.
704
704
-**TLS inspection:** Arc Gateway doesn't support TLS termination or inspection. If your firewall performs TLS inspection, you must exclude the Arc Gateway endpoint from inspection. See [Arc Gateway and TLS inspection](/azure/azure-arc/kubernetes/arc-gateway-simplify-networking#azure-arc-gateway-and-tls-inspection).
705
705
-**Arc Gateway limits:** A maximum of five Arc Gateway resources are supported per subscription.
706
706
-**Explicit Proxy:** Only Azure Firewall Explicit Proxy has been validated. Third-party proxies (for example, Palo Alto) or transparent proxies aren't supported in validated scenarios. Azure IoT Operations doesn't support proxy servers that require a trusted certificate.
0 commit comments