Skip to content

monitor-azure-stream-analytics.md

Latest commit

 

History

History
132 lines (93 loc) · 8.5 KB

monitor-azure-stream-analytics.md

File metadata and controls

132 lines (93 loc) · 8.5 KB
title Monitor Azure Stream Analytics
description Start here to learn how to monitor Azure Stream Analytics.
ms.date 03/21/2024
ms.custom horz-monitor
ms.topic how-to
author spelluru
ms.author spelluru
ms.service azure-stream-analytics

Monitor Azure Stream Analytics

[!INCLUDE horz-monitor-intro]

For instructions on how to monitor and manage Azure Stream Analytics resources with Azure PowerShell cmdlets and PowerShell scripting, see Monitor and manage Stream Analytics jobs with Azure PowerShell cmdlets.

[!INCLUDE horz-monitor-resource-types] For more information about the resource types for Azure Stream Analytics, see Azure Stream Analytics monitoring data reference.

[!INCLUDE horz-monitor-data-storage]

[!INCLUDE horz-monitor-platform-metrics]

Azure Stream Analytics metrics

For a description of how to monitor metrics in the Azure portal, see Monitor Stream Analytics job with Azure portal.

Note

Azure Stream Analytics jobs that are created via REST APIs, Azure SDK, or PowerShell don't have monitoring enabled by default. To enable monitoring, follow the steps in Programmatically create a Stream Analytics job monitor. The monitoring data then appears in the Metrics area of the Azure portal page for your Stream Analytics job.

The following table lists conditions and corrective actions for some commonly monitored Azure Stream Analytics metrics.

[!INCLUDE metrics-scenarios]

For a list and descriptions of all available metrics for Azure Stream Analytics, see Azure Stream Analytics monitoring data reference.

[!INCLUDE horz-monitor-resource-logs]

Azure Stream Analytics logs

[!INCLUDE resource-logs]

For the available resource log categories, their associated Log Analytics tables, and the log schemas for Azure Stream Analytics, see Azure Stream Analytics monitoring data reference.

For a detailed walkthrough of how to troubleshoot Azure Stream Analytics job failures by using resource logs, see Troubleshoot Azure Stream Analytics by using resource logs.

[!INCLUDE horz-monitor-activity-log]

For a detailed walkthrough of how to troubleshoot Azure Stream Analytics job failures by using the activity log, see Debugging using activity logs.

[!INCLUDE horz-monitor-analyze-data]

[!INCLUDE horz-monitor-external-tools]

[!INCLUDE horz-monitor-kusto-queries]

Sample queries

Following are sample queries that you can use to help monitor your Azure Stream Analytics resources:

  • List all input data errors. The following query shows all errors that occurred while processing the data from inputs.

    AzureDiagnostics 
| where ResourceProvider == "MICROSOFT.STREAMANALYTICS" and parse_json(properties_s).Type == "DataError" 
| project TimeGenerated, Resource, Region_s, OperationName, properties_s, Level, _ResourceId

  • Events that arrived late. The following query shows errors due to events where difference between application time and arrival time is greater than the late arrival policy.

    AzureDiagnostics
| where ResourceProvider == "MICROSOFT.STREAMANALYTICS" and  parse_json(properties_s).DataErrorType == "LateInputEvent"
| project TimeGenerated, Resource, Region_s, OperationName, properties_s, Level, _ResourceId

  • Events that arrived early. The following query shows errors due to events where difference between Application time and Arrival time is greater than 5 minutes.

    AzureDiagnostics
| where ResourceProvider == "MICROSOFT.STREAMANALYTICS" and parse_json(properties_s).DataErrorType == "EarlyInputEvent"
| project TimeGenerated, Resource, Region_s, OperationName, properties_s, Level, _ResourceId

  • Events that arrived out of order. The following query shows errors due to events that arrive out of order according to the out-of-order policy.

    // To create an alert for this query, click '+ New alert rule'
AzureDiagnostics
| where ResourceProvider == "MICROSOFT.STREAMANALYTICS" and parse_json(properties_s).DataErrorType == "OutOfOrderEvent"
| project TimeGenerated, Resource, Region_s, OperationName, properties_s, Level, _ResourceId

  • All output data errors. The following query shows all errors that occurred while writing the results of the query to the outputs in your job.

    AzureDiagnostics
| where ResourceProvider == "MICROSOFT.STREAMANALYTICS" and parse_json(properties_s).DataErrorType in ("OutputDataConversionError.RequiredColumnMissing", "OutputDataConversionError.ColumnNameInvalid", "OutputDataConversionError.TypeConversionError", "OutputDataConversionError.RecordExceededSizeLimit", "OutputDataConversionError.DuplicateKey")
| project TimeGenerated, Resource, Region_s, OperationName, properties_s, Level, _ResourceId

  • The following query shows the summary of failed operations in the last seven days.

    AzureDiagnostics
| where TimeGenerated > ago(7d) //last 7 days
| where ResourceProvider == "MICROSOFT.STREAMANALYTICS" and status_s == "Failed" 
| summarize Count=count(), sampleEvent=any(properties_s) by JobName=Resource

[!INCLUDE horz-monitor-alerts]

Azure Stream Analytics alert rules

The following table lists some suggested alert rules for Azure Stream Analytics. These alerts are just examples. You can set alerts for any metric, log entry, or activity log entry listed in the Azure Stream Analytics monitoring data reference.

Alert type Condition Description
Platform metrics Streaming unit (SU) Memory Utilization Whenever average SU (Memory) % Utilization is greater than 80%
Activity log Failed operations Whenever the activity log has an event with Category='Administrative', Signal name='All Administrative operations', Status='Failed'

For detailed instructions on how to set up an alert for Azure Stream Analytics, see Set up alerts for Azure Stream Analytics jobs.

[!INCLUDE horz-monitor-advisor-recommendations]

Related content