Skip to content

howto-configure-mqtt-endpoint.md

Latest commit

 

History

History
1520 lines (1095 loc) · 57.8 KB

howto-configure-mqtt-endpoint.md

File metadata and controls

1520 lines (1095 loc) · 57.8 KB
title Configure MQTT data flow endpoints in Azure IoT Operations
description Learn how to configure data flow endpoints for MQTT sources and destinations.
author sethmanheim
ms.author sethm
ms.service azure-iot-operations
ms.subservice azure-data-flows
ms.topic how-to
ms.date 06/13/2025
ai-usage ai-assisted
ms.custom sfi-image-nochange

Configure MQTT data flow endpoints

MQTT data flow endpoints are used for MQTT sources and destinations. You can configure the endpoint settings, Transport Layer Security (TLS), authentication, and other settings.

Prerequisites

Azure IoT Operations local MQTT broker

Azure IoT Operations provides a built-in local MQTT broker that you can use with data flows. You can use the MQTT broker as a source to receive messages from other systems or as a destination to send messages to other systems.

Default endpoint

When you deploy Azure IoT Operations, an MQTT broker data flow endpoint named "default" is created with default settings. You can use this endpoint as a source or destination for data flows.

Important

You must use the default endpoint, or one with the same settings, in every data flow. It can be the source, the destination, or both. For more details, see Data flows must use local MQTT broker endpoint.

The default endpoint uses the following settings:

Caution

Don't delete the default endpoint. If you delete the default endpoint, you must recreate it with the same settings.

To view or edit the default MQTT broker endpoint settings:

Operations experience

  1. In the operations experience, select the Data flow endpoints.

  2. Select the default endpoint to view or edit the settings.

    :::image type="content" source="media/howto-configure-mqtt-endpoint/default-mqtt-endpoint.png" alt-text="Screenshot using operations experience to view the default MQTT data flow endpoint.":::

Azure CLI

Use the az iot ops dataflow endpoint apply command to create or change the default MQTT broker data flow endpoint.

az iot ops dataflow endpoint apply --resource-group <ResourceGroupName> --instance <AioInstanceName> --name default --config-file <ConfigFilePathAndName>

The --config-file parameter is the path and file name of a JSON configuration file containing the resource properties.

In this example, assume a configuration file named default-mqtt-endpoint.json with the following content stored in the user's home directory:

{
    "endpointType": "Mqtt",
    "mqttSettings": {
        "host": "aio-broker:18883",
        "authentication": {
            "method": "ServiceAccountToken",
            "serviceAccountTokenSettings": {
                "audience": "aio-internal"
            }
        },
        "tls": {
            "mode": "Enabled",
            "trustedCaCertificateConfigMapRef": "azure-iot-operations-aio-ca-trust-bundle"
        }
    }
}

Here's an example command to create or update the default MQTT broker data flow endpoint:

az iot ops dataflow endpoint apply --resource-group myResourceGroup --instance myAioInstance --name default --config-file ~/default-mqtt-endpoint.json

Bicep

To edit the default endpoint, create a Bicep .bicep file with the following content. Update the settings as needed, and replace the placeholder values like <AIO_INSTANCE_NAME> with your own.

param aioInstanceName string = '<AIO_INSTANCE_NAME>'
param customLocationName string = '<CUSTOM_LOCATION_NAME>'

resource aioInstance 'Microsoft.IoTOperations/instances@2024-11-01' existing = {
  name: aioInstanceName
}
resource customLocation 'Microsoft.ExtendedLocation/customLocations@2021-08-31-preview' existing = {
  name: customLocationName
}
resource defaultMqttBrokerDataflowEndpoint 'Microsoft.IoTOperations/instances/dataflowEndpoints@2024-11-01' = {
  parent: aioInstance
  name: 'default'
  extendedLocation: {
    name: customLocation.id
    type: 'CustomLocation'
  }
  properties: {
    endpointType: 'Mqtt'
    mqttSettings: {
      host: 'aio-broker:18883'
      authentication: {
        method: 'ServiceAccountToken'
        serviceAccountTokenSettings: {
          audience: 'aio-internal'
        }
      }
      tls: {
        mode: 'Enabled'
        trustedCaCertificateConfigMapRef: 'azure-iot-operations-aio-ca-trust-bundle'
      }
    }
  }
}

Then, deploy via Azure CLI.

az deployment group create --resource-group <RESOURCE_GROUP> --template-file <FILE>.bicep

Kubernetes (debug only)

[!INCLUDE kubernetes-debug-only-note]

You can view the default MQTT broker endpoint settings in the Kubernetes cluster. To view the settings, use the following command:

kubectl get dataflowendpoint default -n azure-iot-operations -o yaml

Create new endpoint

You can also create new local MQTT broker endpoints with custom settings. For example, you can create a new MQTT broker endpoint using a different port, authentication, or authorization settings. However, you must still always use the default endpoint as either the source or destination in every data flow, even if you create new endpoints.

Operations experience

  1. In the operations experience, select the Data flow endpoints.

  2. Under Create new data flow endpoint, select Azure IoT Operations Local MQTT > New.

    :::image type="content" source="media/howto-configure-mqtt-endpoint/local-mqtt-endpoint.png" alt-text="Screenshot using operations experience to create a new local MQTT data flow endpoint.":::

    Enter the following settings for the endpoint:

    Setting Description
    Name The name of the data flow endpoint.
    Host The hostname and port of the MQTT broker. Use the format <hostname>:<port>
    Authentication method The method used for authentication. Choose Service account token, or X509 certificate
    Service audience The audience for the service account token. Required if using Service account token.
    X509 client certificate The X.509 client certificate used for authentication. Required if using X509 certificate. You can upload industry-standard multi-line X509 certificates improving device authentication management, security, and flexibility.
    X509 client key The private key corresponding to the X.509 client certificate. Required if using X509 certificate.
    X509 intermediate certificates The intermediate certificates for the X.509 client certificate chain. Required if using X509 certificate.

Azure CLI

Create or replace

Use the az iot ops dataflow endpoint create fabric-onelake command to create or replace a local MQTT broker data flow endpoint.

az iot ops dataflow endpoint create local-mqtt --resource-group <ResourceGroupName> --instance <AioInstanceName> --name <EndpointName> --port <Port> --host <Host>

This command creates a local MQTT broker endpoint with default settings. You can specify additional options as needed.

Here's an example command to create or replace a local MQTT broker data flow endpoint named local-mqtt-endpoint:

az iot ops dataflow endpoint create local-mqtt --resource-group myResourceGroup --instance myAioInstance --name local-mqtt-endpoint --port 1883 --host aio-broker --auth-type ServiceAccountToken --audience aio-internal

Create or change

Use the az iot ops dataflow endpoint apply command to create or change a local MQTT broker data flow endpoint.

az iot ops dataflow endpoint apply --resource-group <ResourceGroupName> --instance <AioInstanceName> --name <EndpointName> --config-file <ConfigFilePathAndName>

The --config-file parameter is the path and file name of a JSON configuration file containing the resource properties.

In this example, assume a configuration file named local-mqtt-endpoint.json with the following content stored in the user's home directory:

{
    "endpointType": "Mqtt",
    "mqttSettings": {
        "host": "aio-broker:1883",
        "authentication": {
            "method": "ServiceAccountToken",
            "serviceAccountTokenSettings": {
                "audience": "aio-internal"
            }
        },
        "tls": {
            "mode": "Enabled",
            "trustedCaCertificateConfigMapRef": "azure-iot-operations-aio-ca-trust-bundle"
        }
    }
}

Here's an example command to create or update a local MQTT broker data flow endpoint named local-mqtt-endpoint:

az iot ops dataflow endpoint apply --resource-group myResourceGroup --instance myAioInstance --name local-mqtt-endpoint --config-file ~/local-mqtt-endpoint.json

Bicep

param aioInstanceName string = '<AIO_INSTANCE_NAME>'
param customLocationName string = '<CUSTOM_LOCATION_NAME>'
param endpointName string = '<ENDPOINT_NAME>'
param mqttBrokerHostname string = '<HOSTNAME>:<PORT>'
param trustedCA string = '<TRUST_BUNDLE>'
param serviceAccountAudience string = '<SA_AUDIENCE>'

resource aioInstance 'Microsoft.IoTOperations/instances@2024-11-01' existing = {
  name: aioInstanceName
}
resource customLocation 'Microsoft.ExtendedLocation/customLocations@2021-08-31-preview' existing = {
  name: customLocationName
}
resource MqttBrokerDataflowEndpoint 'Microsoft.IoTOperations/instances/dataflowEndpoints@2024-11-01' = {
  parent: aioInstance
  name: endpointName
  extendedLocation: {
    name: customLocation.id
    type: 'CustomLocation'
  }
  properties: {
    endpointType: 'Mqtt'
    mqttSettings: {
      host: mqttBrokerHostname
      authentication: {
        method: 'ServiceAccountToken'
        serviceAccountTokenSettings: {
          audience: serviceAccountAudience
        }
      }
      tls: {
        mode: 'Enabled'
        trustedCaCertificateConfigMapRef: trustedCA
      }
    }
  }
}

Kubernetes (debug only)

[!INCLUDE kubernetes-debug-only-note]

apiVersion: connectivity.iotoperations.azure.com/v1
kind: DataflowEndpoint
metadata:
  name: <ENDPOINT_NAME>
  namespace: azure-iot-operations
spec:
  endpointType: Mqtt
  mqttSettings:
    host: "<HOSTNAME>:<PORT>"
    tls:
      mode: Enabled
      trustedCaCertificateConfigMapRef: <TRUST_BUNDLE>
    authentication:
      method: ServiceAccountToken
      serviceAccountTokenSettings:
        audience: <SA_AUDIENCE>

Azure Event Grid

Azure Event Grid provides a fully managed MQTT broker that works with Azure IoT Operations data flows. To configure an Azure Event Grid MQTT broker endpoint, we recommend that you use managed identity for authentication.

Configure Event Grid namespace

If you haven't done so already, create Event Grid namespace first.

Enable MQTT

Once you have an Event Grid namespace, go to Configuration and check:

  • Enable MQTT: Select the checkbox.
  • Maximum client sessions per authentication name: Set to 3 or more.

The max client sessions option is important so that data flows can scale up and still be able to connect. To learn more, see Event Grid MQTT multi-session support.

Create a topic space

In order for data flows to send or receive messages to Event Grid MQTT broker, you need to create at least one topic space in the Event Grid namespace. You can create a topic space in the Event Grid namespace by selecting Topic spaces > New topic space.

To quickly get started and for testing, you can create a topic space with the wildcard topic # as the topic template.

Assign permission to managed identity

To configure a data flow endpoint for Event Grid MQTT broker, we recommend using either a user-assigned or system-assigned managed identity. This approach is secure and eliminates the need for managing credentials manually.

After the topic space is created, you need to assign a role to the Azure IoT Operations managed identity that grants permission to send or receive messages to the Event Grid MQTT broker.

If using system-assigned managed identity, in Azure portal, go to your Azure IoT Operations instance and select Overview. Copy the name of the extension listed after Azure IoT Operations Arc extension. For example, azure-iot-operations-xxxx7. Your system-assigned managed identity can be found using the same name of the Azure IoT Operations Arc extension.

Then, go to the Event Grid namespace > Access control (IAM) > Add role assignment.

  1. On the Role tab select an appropriate role like EventGrid TopicSpaces Publisher or EventGrid TopicSpaces Subscriber. This gives the managed identity the necessary permissions to send or receive messages for all topic spaces in the namespace. To learn more, see Microsoft Entra JWT authentication and Azure RBAC authorization to publish or subscribe MQTT messages.
  2. On the Members tab:
    1. If using system-assigned managed identity, for Assign access to, select User, group, or service principal option, then select + Select members and search for the name of the Azure IoT Operations Arc extension.
    2. If using user-assigned managed identity, for Assign access to, select Managed identity option, then select + Select members and search for your user-assigned managed identity set up for cloud connections.

Alternatively, you can assign the role at the topic space level. Go to the topic space > Access control (IAM) > Add role assignment. Assign the managed identity with an appropriate role like EventGrid TopicSpaces Publisher or EventGrid TopicSpaces Subscriber. This gives the managed identity the necessary permissions to send or receive messages for the specific topic space.

Note

A second deployment with the same data flow configuration will not be able to connect and will result in an authorization error. To address this issue, change the data flow name in the second configuration. This issue arises only if the second deployment occurs simultaneously with the first deployment or shortly after, until the MQTT session expires.

Create data flow endpoint for Event Grid MQTT broker

Once the Event Grid namespace is configured, you can create a data flow endpoint for the Event Grid MQTT broker.

Operations experience

  1. In the operations experience, select the Data flow endpoints tab.

  2. Under Create new data flow endpoint, select Azure Event Grid MQTT > New.

    :::image type="content" source="media/howto-configure-mqtt-endpoint/event-grid-endpoint.png" alt-text="Screenshot using operations experience to create an Azure Event Grid endpoint.":::

    Enter the following settings for the endpoint:

    Setting Description
    Name The name of the data flow endpoint.
    Host The hostname and port of the Event Grid MQTT broker. Use the format <NAMESPACE>.<REGION>-1.ts.eventgrid.azure.net:8883
    Authentication method The method used for authentication. We recommend that you choose System assigned managed identity or User assigned managed identity.

  3. Select Apply to provision the endpoint.

Azure CLI

Create or replace

Use the az iot ops dataflow endpoint create eventgrid command to create or replace an Azure Event Grid MQTT data flow endpoint.

az iot ops dataflow endpoint create eventgrid --resource-group <ResourceGroupName> --instance <AioInstanceName> --name <EndpointName> --host <Namespace>.<Region>-1.ts.eventgrid.azure.net --port 9092

This command creates an Event Grid MQTT broker endpoint with default settings and system-assigned managed identity authentication. You can specify additional options as needed.

Here's an example command to create or replace an Event Grid MQTT broker data flow endpoint named event-grid-endpoint:

az iot ops dataflow endpoint create eventgrid --resource-group myResourceGroup --instance myAioInstance --name event-grid-endpoint --host mynamespace.eastus-1.ts.eventgrid.azure.net --port 9092

Create or change

Use the az iot ops dataflow endpoint apply command to create or change an Azure Event Grid MQTT broker data flow endpoint.

az iot ops dataflow endpoint apply --resource-group <ResourceGroupName> --instance <AioInstanceName> --name <EndpointName> --config-file <ConfigFilePathAndName>

The --config-file parameter is the path and file name of a JSON configuration file containing the resource properties.

In this example, assume a configuration file named event-grid-endpoint.json with the following content stored in the user's home directory:

{
    "endpointType": "Mqtt",
    "mqttSettings": {
        "host": "mynamespace.eastus-1.ts.eventgrid.azure.net:9092",
        "authentication": {
            "method": "SystemAssignedManagedIdentity",
            "systemAssignedManagedIdentitySettings": {}
        },
        "tls": {
            "mode": "Enabled"
        }
    }
}

Here's an example command to create or update an Event Grid MQTT broker data flow endpoint named event-grid-endpoint:

az iot ops dataflow endpoint apply --resource-group myResourceGroup --instance myAioInstance --name event-grid-endpoint --config-file ~/event-grid-endpoint.json

Bicep

Create a Bicep .bicep file with the following content.

param aioInstanceName string = '<AIO_INSTANCE_NAME>'
param customLocationName string = '<CUSTOM_LOCATION_NAME>'
param endpointName string = '<ENDPOINT_NAME>'
param eventGridHostName string = '<NAMESPACE>.<REGION>-1.ts.eventgrid.azure.net:8883'

resource aioInstance 'Microsoft.IoTOperations/instances@2024-11-01' existing = {
  name: aioInstanceName
}
resource customLocation 'Microsoft.ExtendedLocation/customLocations@2021-08-31-preview' existing = {
  name: customLocationName
}
resource remoteMqttBrokerDataflowEndpoint 'Microsoft.IoTOperations/instances/dataflowEndpoints@2024-11-01' = {
  parent: aioInstance
  name: endpointName
  extendedLocation: {
    name: customLocation.id
    type: 'CustomLocation'
  }
  properties: {
    endpointType: 'Mqtt'
    mqttSettings: {
      host: eventGridHostName
      authentication: {
        // See available authentication methods section for method types
        // method: <METHOD_TYPE>
      }
      tls: {
        mode: 'Enabled'
      }
    }
  }
}

Then, deploy via Azure CLI.

az deployment group create --resource-group <RESOURCE_GROUP> --template-file <FILE>.bicep

Kubernetes (debug only)

[!INCLUDE kubernetes-debug-only-note]

Create a Kubernetes manifest .yaml file with the following content.

apiVersion: connectivity.iotoperations.azure.com/v1
kind: DataflowEndpoint
metadata:
  name: <ENDPOINT_NAME>
  namespace: azure-iot-operations
spec:
  endpointType: Mqtt
  mqttSettings:
    host: <NAMESPACE>.<REGION>-1.ts.eventgrid.azure.net:8883
    authentication:
      # See available authentication methods section for method types
      # method: <METHOD_TYPE>
    tls:
      mode: Enabled

Then apply the manifest file to the Kubernetes cluster.

kubectl apply -f <FILE>.yaml

Once the endpoint is created, you can use it in a data flow to connect to the Event Grid MQTT broker as a source or destination. The MQTT topics are configured in the data flow.

Use X.509 certificate authentication with Event Grid

When you use X.509 authentication with an Event Grid MQTT broker, go to the Event Grid namespace > Configuration and check these settings:

  • Enable MQTT: Select the checkbox.
  • Enable alternative client authentication name sources: Select the checkbox.
  • Certificate Subject Name: Select this option in the dropdown list.
  • Maximum client sessions per authentication name: Set to 3 or more.

The alternative client authentication and maximum client sessions options allow data flows to use client certificate subject name for authentication instead of MQTT CONNECT Username. This capability is important so that data flows can spawn multiple instances and still be able to connect. To learn more, see Event Grid MQTT client certificate authentication and Multi-session support.

Then, follow the steps in X.509 certificate to configure the endpoint with the X.509 certificate settings.

Event Grid shared subscription limitation

Azure Event Grid MQTT broker doesn't support shared subscriptions, which means that you can't set the instanceCount to more than 1 in the data flow profile if Event Grid is used as a source (where the data flow subscribes to messages) for a data flow. In this case, if you set instanceCount greater than 1, the data flow fails to start.

Custom MQTT brokers

For other MQTT brokers, you can configure the endpoint, TLS, authentication, and other settings as needed.

Operations experience

  1. In the operations experience, select the Data flow endpoints tab.

  2. Under Create new data flow endpoint, select Custom MQTT Broker > New.

    :::image type="content" source="media/howto-configure-mqtt-endpoint/custom-mqtt-broker.png" alt-text="Screenshot using operations experience to create a custom MQTT broker endpoint.":::

  3. Enter the following settings for the endpoint:

    Setting Description
    Name The name of the data flow endpoint
    Host The hostname of the MQTT broker endpoint in the format <hostname>.<port>.
    Authentication method The method used for authentication. Choose Service account token, or X509 certificate.
    Service audience The audience for the service account token. Required if using Service account token.
    X509 client certificate The X.509 client certificate used for authentication. Required if using X509 certificate. You can upload industry-standard multi-line X509 certificates improving device authentication management, security, and flexibility.
    X509 client key The private key corresponding to the X.509 client certificate. Required if using X509 certificate.
    X509 intermediate certificates The intermediate certificates for the X.509 client certificate chain. Required if using X509 certificate.

  4. Select Apply to provision the endpoint.

Azure CLI

Create or replace

Use the az iot ops dataflow endpoint create custom-mqtt command to create or replace a custom MQTT broker data flow endpoint.

az iot ops dataflow endpoint create custom-mqtt --resource-group <ResourceGroupName> --instance <AioInstanceName>  --name <EndpointName> --host <Host> --port <Port>

This command creates a custom MQTT broker endpoint with default settings and system-assigned managed identity authentication. You can specify additional options as needed.

Here's an example command to create or replace a custom MQTT broker data flow endpoint named custom-mqtt-endpoint:

az iot ops dataflow endpoint create custom-mqtt --resource-group myResourceGroup --instance myAioInstance --name custom-mqtt-endpoint --host mycustombroker.contoso.com --port 8883

Create or change

Use the az iot ops dataflow endpoint apply command to create or change a custom MQTT broker data flow endpoint.

az iot ops dataflow endpoint apply --resource-group <ResourceGroupName> --instance <AioInstanceName> --name <EndpointName> --config-file <ConfigFilePathAndName>

The --config-file parameter is the path and file name of a JSON configuration file containing the resource properties.

In this example, assume a configuration file named custom-mqtt-endpoint.json with the following content stored in the user's home directory:

{
    "endpointType": "Mqtt",
    "mqttSettings": {
        "host": "mycustombroker.contoso.com:8883",
        "authentication": {
          "method": "SystemAssignedManagedIdentity",
          "systemAssignedManagedIdentitySettings": {}
        },
        "tls": {
            "mode": "Enabled"
        }
    }
}

Here's an example command to create or update a custom MQTT broker data flow endpoint named custom-mqtt-endpoint:

az iot ops dataflow endpoint apply --resource-group myResourceGroup --instance myAioInstance --name custom-mqtt-endpoint --config-file ~/custom-mqtt-endpoint.json

Bicep

mqttSettings: {
  authentication: {
    // See available authentication methods section for method types
    // method: <METHOD_TYPE>
  }
  host: '<HOST>:<PORT>'
  tls: {
    mode: 'Enabled' // or 'Disabled'
    trustedCaCertificateConfigMapRef: '<TRUSTED_CA_CERT_CONFIGMAP>'
  }
}

Kubernetes (debug only)

[!INCLUDE kubernetes-debug-only-note]

spec:
  endpointType: Mqtt
  mqttSettings:
    host: <HOST>:<PORT>
    authentication:
      # See available authentication methods section for method types
      # method: <METHOD_TYPE>
    tls:
      mode: Enabled # or Disabled
      trustedCaCertificateConfigMapRef: <TRUSTED_CA_CERT_CONFIGMAP>

To customize the MQTT endpoint settings, see the following sections for more information.

Available authentication methods

The following authentication methods are available for MQTT broker data flow endpoints.

System-assigned managed identity

Before you configure the data flow endpoint, assign a role to the Azure IoT Operations managed identity that grants permission to connect to the MQTT broker:

  1. In Azure portal, go to your Azure IoT Operations instance and select Overview.
  2. Copy the name of the extension listed after Azure IoT Operations Arc extension. For example, azure-iot-operations-xxxx7.
  3. Go to the cloud resource you need to grant permissions. For example, go to the Event Grid namespace > Access control (IAM) > Add role assignment.
  4. On the Role tab select an appropriate role.
  5. On the Members tab, for Assign access to, select User, group, or service principal option, then select + Select members and search for the Azure IoT Operations managed identity. For example, azure-iot-operations-xxxx7.

Then, configure the data flow endpoint with system-assigned managed identity settings.

Operations experience

In the operations experience data flow endpoint settings page, select the Basic tab then choose Authentication method > System assigned managed identity.

Azure CLI

Create or replace

Use the az iot ops dataflow endpoint create command with the --auth-type parameter set to SystemAssignedManagedIdentity for system-assigned managed identity authentication.

az iot ops dataflow endpoint create <Command> --auth-type SystemAssignedManagedIdentity --audience <Audience> --resource-group <ResourceGroupName> --instance <AioInstanceName> --name <EndpointName>

Create or change

Use the az iot ops dataflow endpoint apply command with the --config-file parameter.

In this example, assume a configuration file with the following content:

{
    "endpointType": "Mqtt",
    "mqttSettings": {
        "host": "mycustombroker.contoso.com:8883",
        "authentication": {
          "method": "SystemAssignedManagedIdentity",
          "systemAssignedManagedIdentitySettings": {}
        },
        "tls": {
            "mode": "Enabled"
        }
    }
}

Bicep

mqttSettings: {
  authentication: {
    method: 'SystemAssignedManagedIdentity'
    systemAssignedManagedIdentitySettings: {}
  }
}

Kubernetes (debug only)

[!INCLUDE kubernetes-debug-only-note]

mqttSettings:
  authentication:
    method: SystemAssignedManagedIdentity
    systemAssignedManagedIdentitySettings:
      {}

User-assigned managed identity

To use user-assigned managed identity for authentication, you must first deploy Azure IoT Operations with secure settings enabled. Then you need to set up a user-assigned managed identity for cloud connections. To learn more, see Enable secure settings in Azure IoT Operations deployment.

Before you configure the data flow endpoint, assign a role to the user-assigned managed identity that grants permission to connect to the MQTT broker:

  1. In Azure portal, go to the cloud resource you need to grant permissions. For example, go to the Event Grid namespace > Access control (IAM) > Add role assignment.
  2. On the Role tab select an appropriate role.
  3. On the Members tab, for Assign access to, select Managed identity option, then select + Select members and search for your user-assigned managed identity.

Then, configure the data flow endpoint with user-assigned managed identity settings.

Operations experience

In the operations experience data flow endpoint settings page, select the Basic tab then choose Authentication method > User assigned managed identity.

Azure CLI

Create or replace

Use the az iot ops dataflow endpoint create command with the --auth-type parameter set to UserAssignedManagedIdentity for user-assigned managed identity authentication.

az iot ops dataflow endpoint create <Command> --auth-type UserAssignedManagedIdentity --client-id <ClientId> --tenant-id <TenantId> --resource-group <ResourceGroupName> --instance <AioInstanceName> --name <EndpointName>

Create or change

Use the az iot ops dataflow endpoint apply with the --config-file parameter

In this example, assume a configuration file with the following content:

{
    "endpointType": "Mqtt",
    "mqttSettings": {
        "authentication": {
          "method": "UserAssignedManagedIdentity",
          "userAssignedManagedIdentitySettings": {
            "clientId": "<ID>",
            "tenantId": "<ID>",
            // Optional
            "scope": "https://<Scope_Url>"
          }
        }
    }
}

Bicep

mqttSettings: {
  authentication: {
    method: 'UserAssignedManagedIdentity'
    userAssignedManagedIdentitySettings: {
      clientId: '<ID>'
      tenantId: '<ID>'
      // Optional, defaults to 'https://eventgrid.azure.net/.default'
      // scope: 'https://<SCOPE_URL>' 
    }
  }
}

Kubernetes (debug only)

[!INCLUDE kubernetes-debug-only-note]

mqttSettings:
  authentication:
    method: UserAssignedManagedIdentity
    userAssignedManagedIdentitySettings:
      clientId: <ID>
      tenantId: <ID>
      # Optional, defaults to 'https://eventgrid.azure.net/.default'
      # scope: https://<SCOPE_URL>

Kubernetes service account token (SAT)

To use Kubernetes service account token (SAT) for authentication, you don't need to create a secret. The SAT is used to authenticate with the MQTT broker by matching the audience.

Operations experience

In the operations experience data flow endpoint settings page, select the Basic tab then choose Authentication method > Service account token.

Enter the service audience.

Azure CLI

Create or replace

Use the az iot ops dataflow endpoint create command with the --auth-type parameter set to ServiceAccountToken for Kubernetes service account token authentication.

az iot ops dataflow endpoint create <Command> --auth-type ServiceAccountToken --audience <Audience> --resource-group <ResourceGroupName> --instance <AioInstanceName> --name <EndpointName>

Create or change

Use the az iot ops dataflow endpoint apply with the --config-file parameter.

In this example, assume a configuration file with the following content:

{
    "endpointType": "Mqtt",
    "mqttSettings": {
        "authentication": {
          "method": "ServiceAccountToken",
          "serviceAccountTokenSettings": {
            "audience": "<ServiceAccountAudience>"
          }
        }
    }
}

Bicep

mqttSettings: {
  authentication: {
    method: 'ServiceAccountToken'
    serviceAccountTokenSettings: {
      audience: '<YOUR_SERVICE_ACCOUNT_AUDIENCE>'
    }
  }
}

Kubernetes (debug only)

[!INCLUDE kubernetes-debug-only-note]

mqttSettings:
  authentication:
    method: ServiceAccountToken
    serviceAccountTokenSettings:
      audience: <YOUR_SERVICE_ACCOUNT_AUDIENCE>

X.509 certificate

Many MQTT brokers, like Event Grid, support X.509 authentication. Data flows can present a client X.509 certificate and negotiate the TLS communication.

The certificate and private key must be in PEM format and not password protected.

Tip

PEM format is a common format for certificates and keys. Certificates and keys in PEM format are base64-encoded ASCII files with headers that look like -----BEGIN CERTIFICATE----- and -----BEGIN EC PRIVATE KEY----

If you have a certificate in another format, you can convert it to PEM format using OpenSSL. To learn more, see How to convert a certificate into the appropriate format.

Before configuring the data flow endpoint, create a secret with the certificate and private key.

  • If you use the operations portal, the secret is automatically formatted and synced to the Kubernetes cluster.

  • If you use Bicep or Kubernetes, manually create the secret with the certificate and private key in the same namespace as the MQTT data flow endpoint.

    kubectl create secret generic <X509_SECRET_NAME> -n azure-iot-operations --from-file=client_cert.pem=<CLIENT_CERT_FILE>.pem --from-file=client_key.pem=<PRIVATE_KEY_FILE>.pem --from-file=client_intermediate_certs.pem=<INTERMEDIATE_CERT_FILE>.pem

    Here, the secret must have client_cert.pem and client_key.pem as the key names for the certificate and private key. Optionally, the secret can also have client_intermediate_certs.pem as the key name for the intermediate certificates.

Operations experience

Important

To use the operations experience web UI to manage secrets, Azure IoT Operations must first be enabled with secure settings by configuring an Azure Key Vault and enabling workload identities. To learn more, see Enable secure settings in Azure IoT Operations deployment.

Important

The operations experience web UI currently has a known issue where creating an X.509 secret results in a secret with incorrectly encoded data. To learn more and the workaround, see known issues.

In the operations experience data flow endpoint settings page, select the Basic tab then choose Authentication method > X509 certificate.

Here, under Synced secret name, enter a name for the secret. This name is used to reference the secret in the data flow endpoint settings and is the name of the secret as stored in the Kubernetes cluster.

Then, under X509 client certificate, X509 client key, and X509 intermediate certificates, select Add reference to add the certificate, private key, and intermediate certificates. On the next page, select the secret from Azure Key Vault with Add from Azure Key Vault or Create new secret.

If you select Create new, enter the following settings:

Setting Description
Secret name The name of the secret in Azure Key Vault. Pick a name that is easy to remember to select the secret later from the list.
Secret value The certificate, private key, or intermediate certificates in PEM format. You can upload industry-standard multi-line X509 certificates improving device authentication management, security, and flexibility.
Set activation date If turned on, the date when the secret becomes active.
Set expiration date If turned on, the date when the secret expires.

To learn more about secrets, see Create and manage secrets in Azure IoT Operations.

Azure CLI

Create or replace

Use the az iot ops dataflow endpoint create command with the --auth-type parameter set to X509Certificate for X.509 certificate authentication.

az iot ops dataflow endpoint create <Command> --auth-type X509Certificate --secret-name <X509SecretName> --resource-group <ResourceGroupName> --instance <AioInstanceName> --name <EndpointName>

Create or change

Use the az iot ops dataflow endpoint apply with the --config-file parameter.

In this example, assume a configuration file with the following content:

{
    "endpointType": "Mqtt",
    "mqttSettings": {
        "authentication": {
          "method": "X509Certificate",
          "x509CertificateSettings": {
            "secretRef": "<X509SecretName>"
          }
        }
    }
}

Bicep

mqttSettings: {
  authentication: {
    method: 'X509Certificate'
      x509CertificateSettings:
        secretRef: '<X509_SECRET_NAME>'
  }
}

Kubernetes (debug only)

[!INCLUDE kubernetes-debug-only-note]

mqttSettings:
  authentication:
    method: X509Certificate
    x509CertificateSettings:
      secretRef: <X509_SECRET_NAME>

Anonymous

To use anonymous authentication, set the authentication method to Anonymous.

Operations experience

In the operations experience data flow endpoint settings page, select the Basic tab then choose Authentication method > None.

Azure CLI

Create or replace

Use the az iot ops dataflow endpoint create command with the --no-auth parameter for anonymous authentication.

az iot ops dataflow endpoint create <Command> --no-auth --resource-group <ResourceGroupName> --instance <AioInstanceName> --name <EndpointName>

Create or change

Use the az iot ops dataflow endpoint apply with the --config-file parameter.

In this example, assume a configuration file with the following content:

{
    "endpointType": "Mqtt",
    "mqttSettings": {
        "authentication": {
          "method": "Anonymous"
        }
    }
}

Bicep

mqttSettings: {
  authentication: {
    method: 'Anonymous'
  }
}

Kubernetes (debug only)

[!INCLUDE kubernetes-debug-only-note]

mqttSettings:
  authentication:
    method: Anonymous
    anonymousSettings:
      {}

Advanced settings

You can set advanced settings for the MQTT broker data flow endpoint such as TLS, trusted CA certificate, MQTT messaging settings, and CloudEvents. You can set these settings in the data flow endpoint Advanced portal tab, within the data flow endpoint custom resource.

Operations experience

In the operations experience, select the Advanced tab for the data flow endpoint.

Azure CLI

{
    "mqttSettings": {
        "qos": 1,
        "retain": "Keep",
        "sessionExpirySeconds": 3600,
        "keepAliveSeconds": 60,
        "maxInflightMessages": 100,
        "protocol": "WebSockets",
        "clientIdPrefix": "dataflow",
        "cloudEventAttributes": "Propagate",
        "tls": {
          "mode": "Enabled"
        }
    }
}

Bicep

// See sections below for explanation of each setting
mqttSettings: {
  qos: 1
  retain: 'Keep'
  sessionExpirySeconds: 3600
  keepAliveSeconds: 60
  maxInflightMessages: 100
  protocol: WebSockets
  clientIdPrefix: 'dataflow'
  cloudEventAttributes : 'Propagate' // or 'CreateOrRemap'
}

Kubernetes (debug only)

[!INCLUDE kubernetes-debug-only-note]

# See sections below for explanation of each setting
mqttSettings:
  qos: 1
  retain: Keep
  sessionExpirySeconds: 3600
  keepAliveSeconds: 60
  maxInflightMessages: 100
  protocol: WebSockets
  clientIdPrefix: dataflow
  cloudEventAttributes : Propagate # or CreateOrRemap

TLS settings

TLS mode

To enable or disable TLS for the MQTT endpoint, update the mode setting in the TLS settings.

Operations experience

In the operations experience data flow endpoint settings page, select the Advanced tab then use the checkbox next to TLS mode enabled.

Azure CLI

{
    "mqttSettings": {
        "tls": {
          "mode": "Enabled"
        }
    }
}

Bicep

mqttSettings: {
  tls: {
    mode: 'Enabled' // or 'Disabled'
  }
}

Kubernetes (debug only)

[!INCLUDE kubernetes-debug-only-note]

mqttSettings:
  tls:
    mode: Enabled # or Disabled

The TLS mode can be set to Enabled or Disabled. If the mode is set to Enabled, the data flow uses a secure connection to the MQTT broker. If the mode is set to Disabled, the data flow uses an insecure connection to the MQTT broker.

Trusted CA certificate

Configure the trusted CA certificate for the MQTT endpoint to establish a secure connection to the MQTT broker. This setting is important if the MQTT broker uses a self-signed certificate or a certificate signed by a custom CA that isn't trusted by default.

Operations experience

In the operations experience data flow endpoint settings page, select the Advanced tab then use the Trusted CA certificate config map field to specify the ConfigMap containing the trusted CA certificate.

Bicep

mqttSettings: {
  tls: {
    trustedCaCertificateConfigMapRef: '<YOUR_CA_CERTIFICATE>'
  }
}

Azure CLI

{
    "mqttSettings": {
        "tls": {
          "trustedCaCertificateConfigMapRef": "<YOUR_CA_CERTIFICATE>"
        }
    }
}

Kubernetes (debug only)

[!INCLUDE kubernetes-debug-only-note]

mqttSettings:
  tls:
    trustedCaCertificateConfigMapRef: <YOUR_CA_CERTIFICATE>

This ConfigMap should contain the CA certificate in PEM format. The ConfigMap must be in the same namespace as the MQTT data flow resource. For example:

kubectl create configmap client-ca-configmap --from-file root_ca.crt -n azure-iot-operations

Tip

When connecting to Event Grid MQTT broker, the CA certificate isn't required because the Event Hubs service uses a certificate signed by a public CA that is trusted by default.

Client ID prefix

You can set a client ID prefix for the MQTT client. The client ID is generated by appending the data flow instance name to the prefix.

Caution

Most applications should not modify the client ID prefix. Don't modify this after an initial IoT Operations deployment. Changing the client ID prefix after deployment might result in data loss.

Operations experience

In the operations experience data flow endpoint settings page, select the Advanced tab then use the Client ID prefix field to specify the prefix.

Azure CLI

{
    "mqttSettings": {
        "clientIdPrefix": "<YOUR_PREFIX>"
    }
}

Bicep

mqttSettings: {
  clientIdPrefix: '<YOUR_PREFIX>'
}

Kubernetes (debug only)

[!INCLUDE kubernetes-debug-only-note]

mqttSettings:
  clientIdPrefix: <YOUR_PREFIX>

QoS

You can set the Quality of Service (QoS) level for the MQTT messages to either 1 or 0. The default is 1.

Operations experience

In the operations experience data flow endpoint settings page, select the Advanced tab then use the Quality of service (QoS) field to specify the QoS level.

Azure CLI

{
    "mqttSettings": {
        "qos": 1
    }
}

Bicep

mqttSettings: {
  qos: 1 // Or 0
}

Kubernetes (debug only)

[!INCLUDE kubernetes-debug-only-note]

mqttSettings:
  qos: 1 # Or 0

Retain

Use the retain setting to specify whether the data flow should keep the retain flag on MQTT messages. The default is Keep.

Setting this field to Keep is useful to ensure that the remote broker has the same messages retained as the local broker, which can be important for Unified Namespace (UNS) scenarios.

If set to Never, the retain flag is removed from the MQTT messages. This can be useful when you don't want the remote broker to retain any messages or if the remote broker doesn't support retain.

To configure retain settings:

Operations experience

In the operations experience data flow endpoint settings page, select the Advanced tab then use the Retain field to specify the retain setting.

Azure CLI

{
    "mqttSettings": {
        "retain": "Keep"
    }
}

Bicep

mqttSettings: {
  retain: Keep // or Never
}

Kubernetes (debug only)

[!INCLUDE kubernetes-debug-only-note]

mqttSettings:
  retain: Keep # or Never

The retain setting only takes effect if the data flow uses MQTT endpoint as both source and destination. For example, in an MQTT bridge scenario.

Important

Azure Event Grid MQTT broker currently doesn't support the retain flag. This means if you set the retain flag to Keep for an Event Grid MQTT broker endpoint and it's being used as a destination, the messages are rejected. To avoid this, set the retain flag to Never when using Event Grid MQTT broker as a destination.

Session expiry

You can set the session expiry interval for the data flow MQTT client. The session expiry interval is the maximum time that an MQTT session is maintained if the data flow client disconnects. The default is 600 seconds. To configure the session expiry interval:

Operations experience

In the operations experience data flow endpoint settings page, select the Advanced tab then use the Session expiry field to specify the session expiry interval.

Azure CLI

{
    "mqttSettings": {
        "sessionExpirySeconds": 600
    }
}

Bicep

mqttSettings: {
  sessionExpirySeconds: 600
}

Kubernetes (debug only)

[!INCLUDE kubernetes-debug-only-note]

mqttSettings:
  sessionExpirySeconds: 600

MQTT or WebSockets protocol

By default, WebSockets isn't enabled. To use MQTT over WebSockets, set the protocol field to WebSockets.

Important

When using MQTT over WebSockets, Event Grid requires /mqtt appended to the end of the host name value. For example, my-event-grid.eventgrid.azure.net:443/mqtt or my-event-grid/mqtt.

Operations experience

In the operations experience data flow endpoint settings page, select the Advanced tab then use the Protocol field to specify the protocol.

Azure CLI

{
    "mqttSettings": {
        "protocol": "WebSockets"
    }
}

Bicep

mqttSettings: {
  protocol: 'WebSockets'
}

Kubernetes (debug only)

[!INCLUDE kubernetes-debug-only-note]

mqttSettings:
  protocol: WebSockets

Max inflight messages

You can set the maximum number of inflight messages that the data flow MQTT client can have. The default is 100.

Operations experience

In the operations experience data flow endpoint settings page, select the Advanced tab then use the Maximum in-flight messages field to specify the maximum number of inflight messages.

Azure CLI

{
    "mqttSettings": {
        "maxInflightMessages": 100
    }
}

Bicep

mqttSettings: {
  maxInflightMessages: 100
}

Kubernetes (debug only)

[!INCLUDE kubernetes-debug-only-note]

mqttSettings:
  maxInflightMessages: 100

For subscribe when the MQTT endpoint is used as a source, this is the receive maximum. For publish when the MQTT endpoint is used as a destination, this is the maximum number of messages to send before waiting for an acknowledgment.

Keep alive

You can set the keep alive interval for the data flow MQTT client. The keep alive interval is the maximum time that the data flow client can be idle before sending a PINGREQ message to the broker. The default is 60 seconds.

Operations experience

In the operations experience data flow endpoint settings page, select the Advanced tab then use the Keep alive field to specify the keep alive interval.

Azure CLI

{
    "mqttSettings": {
        "keepAliveSeconds": 60
    }
}

Bicep

mqttSettings: {
  keepAliveSeconds: 60
}

Kubernetes (debug only)

[!INCLUDE kubernetes-debug-only-note]

mqttSettings:
  keepAliveSeconds: 60

CloudEvents

CloudEvents are a way to describe event data in a common way. The CloudEvents settings are used to send or receive messages in the CloudEvents format. You can use CloudEvents for event-driven architectures where different services need to communicate with each other in the same or different cloud providers.

The cloudEventAttributes options are Propagate orCreateOrRemap. To configure CloudEvents settings:

Operations experience

In the operations experience data flow endpoint settings page, select the Advanced tab then use the Cloud event attributes field to specify the CloudEvents setting.

Azure CLI

{
    "mqttSettings": {
        "cloudEventAttributes": "Propagate"
    }
}

Bicep

mqttSettings: {
  cloudEventAttributes : 'Propagate' // or 'CreateOrRemap'
}

Kubernetes (debug only)

[!INCLUDE kubernetes-debug-only-note]

mqttSettings:
  cloudEventAttributes : Propagate # or CreateOrRemap

The following sections provide more information about the CloudEvents settings.

Propagate setting

CloudEvent properties are passed through for messages that contain the required properties. If the message doesn't contain the required properties, the message is passed through as is.

Name Required Sample value Output value
specversion Yes 1.0 Passed through as is
type Yes ms.aio.telemetry Passed through as is
source Yes aio://mycluster/myoven Passed through as is
id Yes A234-1234-1234 Passed through as is
subject No aio/myoven/sensor/temperature Passed through as is
time No 2018-04-05T17:31:00Z Passed through as is. It's not restamped.
datacontenttype No application/json Changed to the output data content type after the optional transform stage.
dataschema No sr://fabrikam-schemas/123123123234234234234234#1.0.0 If an output data transformation schema is given in the transformation configuration, dataschema is changed to the output schema.

CreateOrRemap setting

CloudEvent properties are passed through for messages that contain the required properties. If the message doesn't contain the required properties, the properties are generated.

Name Required Generated value if missing
specversion Yes 1.0
type Yes ms.aio-dataflow.telemetry
source Yes aio://<target-name>
id Yes Generated UUID in the target client
subject No The output topic where the message is sent
time No Generated as RFC 3339 in the target client
datacontenttype No Changed to the output data content type after the optional transform stage
dataschema No Schema defined in the schema registry

Next steps

To learn more about data flows, see Create a data flow.