Skip to content

howto-configure-local-storage-endpoint.md

Latest commit

 

History

History
188 lines (130 loc) · 7.8 KB

howto-configure-local-storage-endpoint.md

File metadata and controls

188 lines (130 loc) · 7.8 KB
title Configure local storage data flow endpoint in Azure IoT Operations
description Learn how to configure a local storage data flow endpoint in Azure IoT Operations.
author sethmanheim
ms.author sethm
ms.service azure-iot-operations
ms.subservice azure-data-flows
ms.topic how-to
ms.date 06/13/2025
ai-usage ai-assisted

Configure data flow endpoints for local storage

To send data to local storage in Azure IoT Operations, you can configure a data flow endpoint. This configuration allows you to specify the endpoint, authentication, table, and other settings.

Prerequisites

Create a local storage data flow endpoint

Use the local storage option to send data to a locally available persistent volume, through which you can upload data via Azure Container Storage enabled by Azure Arc edge volumes.

Operations experience

  1. In the operations experience, select the Data flow endpoints tab.

  2. Under Create new data flow endpoint, select Local Storage > New.

    :::image type="content" source="media/howto-configure-local-storage-endpoint/create-local-storage-endpoint.png" alt-text="Screenshot using operations experience to create a Local Storage data flow endpoint.":::

  3. Enter the following settings for the endpoint:

    Setting Description
    Name The name of the data flow endpoint.
    Persistent volume claim name The name of the PersistentVolumeClaim (PVC) to use for local storage.

  4. Select Apply to provision the endpoint.

Azure CLI

Create or replace

Use the az iot ops dataflow endpoint create fabric-onelake command to create or replace a local storage data flow endpoint.

az iot ops dataflow endpoint create local-storage --resource-group <ResourceGroupName> --instance <AioInstanceName> --name <EndpointName> --pvc-ref <PersistentVolumeClaimName>

The --pvc-ref parameter is the name of the PersistentVolumeClaim (PVC) to use for local storage. The PVC must be in the same namespace as the data flow endpoint.

Here's an example command to create or replace a local storage data flow endpoint named local-storage-endpoint:

az iot ops dataflow endpoint create local-storage --resource-group myResourceGroup --instance myAioInstance --name local-storage-endpoint --pvc-ref mypvc

Create or change

Use the az iot ops dataflow endpoint apply command to create or change a local storage data flow endpoint.

az iot ops dataflow endpoint apply --resource-group <ResourceGroupName> --instance <AioInstanceName> --name <EndpointName> --config-file <ConfigFilePathAndName>

The --config-file parameter is the path and file name of a JSON configuration file containing the resource properties.

In this example, assume a configuration file named local-storage-endpoint.json with the following content stored in the user's home directory:

{
    "endpointType": "LocalStorage",
    "localStorageSettings": {
        "persistentVolumeClaimRef": "<PersistentVolumeClaimName>"
    }
}

Here's an example command to create a new local storage data flow endpoint named local-storage-endpoint:

az iot ops dataflow endpoint apply --resource-group myResourceGroupName --instance myAioInstanceName --name local-storage-endpoint --config-file ~/local-storage-endpoint.json

Bicep

Create a Bicep .bicep file with the following content.

param aioInstanceName string = '<AIO_INSTANCE_NAME>'
param customLocationName string = '<CUSTOM_LOCATION_NAME>'
param endpointName string = '<ENDPOINT_NAME>'
param persistentVCName string = '<PERSISTENT_VC_NAME>'

resource aioInstance 'Microsoft.IoTOperations/instances@2024-11-01' existing = {
  name: aioInstanceName
}
resource customLocation 'Microsoft.ExtendedLocation/customLocations@2021-08-31-preview' existing = {
  name: customLocationName
}
resource localStorageDataflowEndpoint 'Microsoft.IoTOperations/instances/dataflowEndpoints@2024-11-01' = {
  parent: aioInstance
  name: endpointName
  extendedLocation: {
    name: customLocation.id
    type: 'CustomLocation'
  }
  properties: {
    endpointType: 'LocalStorage'
    localStorageSettings: {
      persistentVolumeClaimRef: persistentVCName
    }
  }
}

Then, deploy via Azure CLI.

az deployment group create --resource-group <RESOURCE_GROUP> --template-file <FILE>.bicep

Kubernetes (debug only)

[!INCLUDE kubernetes-debug-only-note]

Create a Kubernetes manifest .yaml file with the following content.

apiVersion: connectivity.iotoperations.azure.com/v1
kind: DataflowEndpoint
metadata:
  name: <ENDPOINT_NAME>
  namespace: azure-iot-operations
spec:
  endpointType: LocalStorage
  localStorageSettings:
    persistentVolumeClaimRef: <PVC_NAME>

Then apply the manifest file to the Kubernetes cluster.

kubectl apply -f <FILE>.yaml

The PersistentVolumeClaim (PVC) must be in the same namespace as the DataflowEndpoint.

Supported serialization formats

The only supported serialization format is Parquet.

Use Azure Container Storage enabled by Azure Arc (ACSA)

You can use the local storage data flow endpoint together with Azure Container Storage enabled by Azure Arc to store data locally or send data to a cloud destination.

Important

You must install Azure Container Storage enabled by Azure Arc (ACSA) before using it with a local storage data flow endpoint.

Local shared volume

To write to a local shared volume, first create a PersistentVolumeClaim (PVC) according to the instructions from Local Shared Edge Volumes.

Then, when configuring your local storage data flow endpoint, input the PVC name under persistentVolumeClaimRef.

Cloud ingest

To write your data to the cloud, follow the instructions in Cloud Ingest Edge Volumes configuration to create a PVC and attach a subvolume for your desired cloud destination.

Important

To configure cloud ingest, your cluster must have secure settings enabled. The cloud ingest feature relies on workload identity federation.

Important

Don't forget to create the subvolume after creating the PVC, or else the data flow fails to start and the logs show a "read-only file system" error.

Then, when configuring your local storage data flow endpoint, input the PVC name under persistentVolumeClaimRef.

Finally, when you create the data flow, the data destination parameter must match the spec.path parameter you created for your subvolume during configuration.

Next steps

To learn more about data flows, see Create a data flow.