You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
title: Layered networking for Azure IoT Operations
3
+
description: Learn about deploying Azure IoT Operations across layered (Purdue/ISA-95) industrial networks using Envoy proxy chaining, CoreDNS, and Kubernetes-based configuration.
4
+
author: dominicbetts
5
+
ms.subservice: layered-network-management
6
+
ms.author: dobett
7
+
ms.topic: concept-article
8
+
ms.custom:
9
+
- ignite-2023
10
+
ms.date: 03/24/2026
11
+
12
+
#CustomerIntent: As an operator, I want to understand how Azure IoT Operations works in a layered network so I can deploy across segmented industrial environments.
13
+
ms.service: azure-iot-operations
14
+
---
15
+
16
+
# Layered networking for Azure IoT Operations
17
+
18
+
In many industrial environments, segmented networking architectures (such as the [Purdue Network Architecture](https://en.wikipedia.org/wiki/Purdue_Enterprise_Reference_Architecture)) separate assets, control systems, and business applications into distinct network layers. Lower layers typically can't connect directly to the internet and can only communicate with adjacent layers. Azure IoT Operations supports deploying across these layered networks, using Envoy proxy chaining, CoreDNS, and Kubernetes-based configuration to route traffic between layers.
19
+
20
+
The following diagram shows Azure IoT Operations deployed to multiple clusters in different network segments. In the Purdue Network architecture, level 4 is the enterprise network, level 3 is the operation and control layer, and level 2 is the controller system layer. Only level 4 has direct internet access, and the other levels can only communicate with their adjacent levels.
21
+
22
+
:::image type="content" source="media/layered-network-architecture.png" alt-text="Diagram that shows layered networking architecture for industrial layered networks.":::
23
+
24
+
In this example, Azure IoT Operations is deployed to levels 2 through 4. Each layer uses the following components to route traffic upward to the next layer:
25
+
26
+
-**Envoy Proxy** (levels 3 and 4) — acts as a reverse proxy that forwards traffic from child layers toward Azure.
27
+
-**CoreDNS** (levels 2 and 3) — resolves approved URIs to the parent cluster's Envoy Proxy, so lower layers can reach Azure services through adjacent layers.
28
+
29
+
This setup lets you Arc-enable clusters at every layer and keep them connected without direct internet access.
30
+
31
+
You can deploy Azure IoT Operations components across layers based on your architecture and data flow needs:
32
+
33
+
-**Connector for OPC UA** — place at the lower layer, closer to your assets and OPC UA servers.
34
+
-**MQTT broker** — deploy at each layer to transfer data upward toward the cloud.
35
+
-**Data Flows** — place on nodes with enough compute resources, as this component typically uses more compute. With extra configuration, Data Flows can also route traffic east-west between components at the same or upper levels.
36
+
37
+
## How telemetry flows through layers
38
+
39
+
In a layered network, asset telemetry doesn't pass straight through from the lowest layer to the cloud. Instead, telemetry *terminates and re-originates* at each layer. This hop-by-hop pattern is a key difference from a flat network deployment.
40
+
41
+
The flow works as follows:
42
+
43
+
1.**Level 2 (asset layer):** The Connector for OPC UA reads data from OPC UA servers and publishes it to the local MQTT broker (for example, to a topic like `clusterl2/data/oven1`). A Data Flow picks up those messages, optionally transforms or enriches them (for example, adding product context), and forwards them to the MQTT broker at level 3.
44
+
1.**Level 3 (operations layer):** The level 3 MQTT broker receives the messages from level 2. A Data Flow at this layer can add further context (for example, production line details) and forwards the enriched messages to the MQTT broker at level 4.
45
+
1.**Level 4 (enterprise layer):** The level 4 MQTT broker receives the messages from level 3. A final Data Flow adds any remaining context (for example, factory identifiers) and sends the messages to a cloud destination such as Azure Event Hubs or Azure Event Grid through the private or public connectivity path.
46
+
47
+
At every layer, the data is fully terminated on the local MQTT broker, it isn't tunneled or pass-through. This gives you the ability to:
48
+
49
+
-**Enrich data at each level** — add contextual metadata (product, line, factory) that lower-layer assets don't know about.
50
+
-**Filter or aggregate** — reduce data volume or drop irrelevant messages before forwarding upward.
51
+
-**Consume locally** — other applications at the same layer can subscribe to the local MQTT broker for departmental workloads without waiting for data to round-trip to the cloud.
52
+
53
+
For a hands-on walkthrough of this pattern, see step 5 in the [sample walkthrough](#sample-walkthrough).
54
+
55
+
## Sample walkthrough
56
+
57
+
A [layered networking guidance sample](https://github.com/Azure-Samples/explore-iot-operations/tree/main/samples/layered-networking) is available in the Azure IoT Operations samples repository. The sample repository contains the infrastructure configuration files (Envoy proxy configs, CoreDNS Corefiles, and Kubernetes manifests) used at each network layer. The [tutorial](../end-to-end-tutorials/tutorial-layered-network-private-connectivity.md) provides the complete end-to-end deployment guide, including Azure resource setup, Private Link and DNS configuration, Arc enablement, RBAC assignments, and post-deployment audit, and references the sample for layer-specific configuration files.
58
+
59
+
The sample and guidance show how to:
60
+
61
+
- Use Kubernetes-based configuration and networking primitives for layered environments.
62
+
- Connect devices in isolated networks at scale to [Azure Arc](/azure/azure-arc/) for application lifecycle management and remote configuration.
63
+
- Enforce security and governance across network levels with URL/IP allow lists and connection auditing.
64
+
- Ensure compatibility with all Azure IoT Operations services.
65
+
66
+
> [!NOTE]
67
+
> The guidance doesn't recommend specific practices or provide production-ready implementation, configuration, or operations details. The guidance doesn't make recommendations about production networking architecture.
68
+
69
+
To learn more about how to prepare for a production-ready deployment of Azure IoT Operations, see the [Azure IoT Operations production guidelines](../deploy-iot-ops/concept-production-guidelines.md).
70
+
71
+
To try the sample in a test environment, follow the step-by-step walkthrough:
72
+
73
+
1.[How Azure IoT Operations works in a layered network](https://github.com/Azure-Samples/explore-iot-operations/blob/main/samples/layered-networking/aio-layered-network.md)
74
+
2.[Configure the infrastructure](https://github.com/Azure-Samples/explore-iot-operations/blob/main/samples/layered-networking/configure-infrastructure.md)
75
+
3.[Arc-enable the K3s clusters](https://github.com/Azure-Samples/explore-iot-operations/blob/main/samples/layered-networking/arc-enable-clusters.md)
> If you want to deploy Azure IoT Operations end-to-end in a layered network with private connectivity, see [Tutorial: Deploy Azure IoT Operations in a layered network with private connectivity](../end-to-end-tutorials/tutorial-layered-network-private-connectivity.md).
81
+
82
+
## Platform compatibility
83
+
84
+
The layered networking approach described here applies to **Arc-enabled Kubernetes clusters**. If you use other Arc-enabled platforms in your layered environment, keep the following in mind:
85
+
86
+
-**[Azure Arc-enabled servers](/azure/azure-arc/servers/overview):** The Envoy proxy chain has only been validated with Arc-enabled Kubernetes clusters. Arc-enabled servers use a different agent ([Azure Connected Machine agent](/azure/azure-arc/servers/agent-overview)) with different endpoint requirements. If you need to Arc-enable servers in a layered network, verify that the required endpoints are included in your Envoy proxy and CoreDNS configuration. Azure IoT Operations components (MQTT broker, Data Flows, Connector for OPC UA) require Kubernetes and can't run on Arc-enabled servers directly.
87
+
-**[Azure Local](/azure/azure-local/overview) (formerly Azure Stack HCI):** Azure Local nodes that host Kubernetes clusters (such as [AKS enabled by Azure Arc](/azure/aks/aksarc/overview)) are fully compatible with this layered networking approach. The Kubernetes cluster running on Azure Local follows the same Arc enablement and Envoy proxy chain described above.
-[Deploy Azure IoT Operations with private connectivity](howto-private-connectivity.md)
93
+
-[Tutorial: Deploy Azure IoT Operations in a layered network with private connectivity](../end-to-end-tutorials/tutorial-layered-network-private-connectivity.md)
Copy file name to clipboardExpand all lines: articles/iot-operations/manage-layered-network/overview-layered-network.md
+1-74Lines changed: 1 addition & 74 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -22,80 +22,7 @@ Networking is a foundational aspect of deploying and managing distributed system
22
22
Before deploying, determine which networking approach fits your scenario:
23
23
24
24
-**Private connectivity:** If you have a single cluster that needs private connectivity to Azure without network segmentation between layers, use this approach. You can use Private Link for storage and data flow destinations, Arc Gateway to reduce firewall endpoints, and Azure Firewall Explicit Proxy to keep all traffic on private networks. See [Deploy Azure IoT Operations with private connectivity](howto-private-connectivity.md).
25
-
-**Layered network:** If you have a Purdue/ISA-95 segmented topology with multiple network layers (L2/L3/L4) and adjacent-only communication, use a layered network deployment. This approach adds Envoy proxy chaining, CoreDNS at each layer, and multi-cluster Azure IoT Operations deployments across layers. **If you have a layered topology, this is the recommended approach.** See [Tutorial: Deploy Azure IoT Operations in a layered network with private connectivity](../end-to-end-tutorials/tutorial-layered-network-private-connectivity.md).
26
-
27
-
## Layered networking
28
-
29
-
In many industrial environments, segmented networking architectures (such as the [Purdue Network Architecture](https://en.wikipedia.org/wiki/Purdue_Enterprise_Reference_Architecture)) separate assets, control systems, and business applications into distinct network layers. Lower layers typically can't connect directly to the internet and can only communicate with adjacent layers. Azure IoT Operations supports deploying across these layered networks, using Envoy proxy chaining, CoreDNS, and Kubernetes-based configuration to route traffic between layers.
30
-
31
-
The following diagram shows Azure IoT Operations deployed to multiple clusters in different network segments. In the Purdue Network architecture, level 4 is the enterprise network, level 3 is the operation and control layer, and level 2 is the controller system layer. Only level 4 has direct internet access, and the other levels can only communicate with their adjacent levels.
32
-
33
-
:::image type="content" source="media/layered-network-architecture.png" alt-text="Diagram that shows layered networking architecture for industrial layered networks.":::
34
-
35
-
In this example, Azure IoT Operations is deployed to levels 2 through 4. Each layer uses the following components to route traffic upward to the next layer:
36
-
37
-
-**Envoy Proxy** (levels 3 and 4) — acts as a reverse proxy that forwards traffic from child layers toward Azure.
38
-
-**CoreDNS** (levels 2 and 3) — resolves approved URIs to the parent cluster's Envoy Proxy, so lower layers can reach Azure services through adjacent layers.
39
-
40
-
This setup lets you Arc-enable clusters at every layer and keep them connected without direct internet access.
41
-
42
-
You can deploy Azure IoT Operations components across layers based on your architecture and data flow needs:
43
-
44
-
-**Connector for OPC UA** — place at the lower layer, closer to your assets and OPC UA servers.
45
-
-**MQTT broker** — deploy at each layer to transfer data upward toward the cloud.
46
-
-**Data Flows** — place on nodes with enough compute resources, as this component typically uses more compute. With extra configuration, Data Flows can also route traffic east-west between components at the same or upper levels.
47
-
48
-
### How telemetry flows through layers
49
-
50
-
In a layered network, asset telemetry doesn't pass straight through from the lowest layer to the cloud. Instead, telemetry *terminates and re-originates* at each layer. This hop-by-hop pattern is a key difference from a flat network deployment.
51
-
52
-
The flow works as follows:
53
-
54
-
1.**Level 2 (asset layer):** The Connector for OPC UA reads data from OPC UA servers and publishes it to the local MQTT broker (for example, to a topic like `clusterl2/data/oven1`). A Data Flow picks up those messages, optionally transforms or enriches them (for example, adding product context), and forwards them to the MQTT broker at level 3.
55
-
1.**Level 3 (operations layer):** The level 3 MQTT broker receives the messages from level 2. A Data Flow at this layer can add further context (for example, production line details) and forwards the enriched messages to the MQTT broker at level 4.
56
-
1.**Level 4 (enterprise layer):** The level 4 MQTT broker receives the messages from level 3. A final Data Flow adds any remaining context (for example, factory identifiers) and sends the messages to a cloud destination such as Azure Event Hubs or Azure Event Grid through the private or public connectivity path.
57
-
58
-
At every layer, the data is fully terminated on the local MQTT broker, it isn't tunneled or pass-through. This gives you the ability to:
59
-
60
-
-**Enrich data at each level** — add contextual metadata (product, line, factory) that lower-layer assets don't know about.
61
-
-**Filter or aggregate** — reduce data volume or drop irrelevant messages before forwarding upward.
62
-
-**Consume locally** — other applications at the same layer can subscribe to the local MQTT broker for departmental workloads without waiting for data to round-trip to the cloud.
63
-
64
-
For a hands-on walkthrough of this pattern, see step 5 in the [sample walkthrough](#sample-walkthrough).
65
-
66
-
### Sample walkthrough
67
-
68
-
A [layered networking guidance sample](https://github.com/Azure-Samples/explore-iot-operations/tree/main/samples/layered-networking) is available in the Azure IoT Operations samples repository. The sample repository contains the infrastructure configuration files (Envoy proxy configs, CoreDNS Corefiles, and Kubernetes manifests) used at each network layer. The [tutorial](../end-to-end-tutorials/tutorial-layered-network-private-connectivity.md) provides the complete end-to-end deployment guide, including Azure resource setup, Private Link and DNS configuration, Arc enablement, RBAC assignments, and post-deployment audit, and references the sample for layer-specific configuration files.
69
-
70
-
The sample and guidance show how to:
71
-
72
-
- Use Kubernetes-based configuration and networking primitives for layered environments.
73
-
- Connect devices in isolated networks at scale to [Azure Arc](/azure/azure-arc/) for application lifecycle management and remote configuration.
74
-
- Enforce security and governance across network levels with URL/IP allow lists and connection auditing.
75
-
- Ensure compatibility with all Azure IoT Operations services.
76
-
77
-
> [!NOTE]
78
-
> The guidance doesn't recommend specific practices or provide production-ready implementation, configuration, or operations details. The guidance doesn't make recommendations about production networking architecture.
79
-
80
-
To learn more about how to prepare for a production-ready deployment of Azure IoT Operations, see the [Azure IoT Operations production guidelines](../deploy-iot-ops/concept-production-guidelines.md).
81
-
82
-
To try the sample in a test environment, follow the step-by-step walkthrough:
83
-
84
-
1.[How Azure IoT Operations works in a layered network](https://github.com/Azure-Samples/explore-iot-operations/blob/main/samples/layered-networking/aio-layered-network.md)
85
-
2.[Configure the infrastructure](https://github.com/Azure-Samples/explore-iot-operations/blob/main/samples/layered-networking/configure-infrastructure.md)
86
-
3.[Arc-enable the K3s clusters](https://github.com/Azure-Samples/explore-iot-operations/blob/main/samples/layered-networking/arc-enable-clusters.md)
> If you want to deploy Azure IoT Operations end-to-end in a layered network with private connectivity, see [Tutorial: Deploy Azure IoT Operations in a layered network with private connectivity](../end-to-end-tutorials/tutorial-layered-network-private-connectivity.md).
92
-
93
-
### Platform compatibility
94
-
95
-
The layered networking approach described here applies to **Arc-enabled Kubernetes clusters**. If you use other Arc-enabled platforms in your layered environment, keep the following in mind:
96
-
97
-
-**[Azure Arc-enabled servers](/azure/azure-arc/servers/overview):** The Envoy proxy chain has only been validated with Arc-enabled Kubernetes clusters. Arc-enabled servers use a different agent ([Azure Connected Machine agent](/azure/azure-arc/servers/agent-overview)) with different endpoint requirements. If you need to Arc-enable servers in a layered network, verify that the required endpoints are included in your Envoy proxy and CoreDNS configuration. Azure IoT Operations components (MQTT broker, Data Flows, Connector for OPC UA) require Kubernetes and can't run on Arc-enabled servers directly.
98
-
-**[Azure Local](/azure/azure-local/overview) (formerly Azure Stack HCI):** Azure Local nodes that host Kubernetes clusters (such as [AKS enabled by Azure Arc](/azure/aks/aksarc/overview)) are fully compatible with this layered networking approach. The Kubernetes cluster running on Azure Local follows the same Arc enablement and Envoy proxy chain described above.
25
+
-**Layered network:** If you have a Purdue/ISA-95 segmented topology with multiple network layers (L2/L3/L4) and adjacent-only communication, use a layered network deployment. This approach adds Envoy proxy chaining, CoreDNS at each layer, and multi-cluster Azure IoT Operations deployments across layers. **If you have a layered topology, this is the recommended approach.** See [Layered networking for Azure IoT Operations](concept-layered-network.md) for architecture details, or go straight to the [Tutorial: Deploy Azure IoT Operations in a layered network with private connectivity](../end-to-end-tutorials/tutorial-layered-network-private-connectivity.md).
0 commit comments