Skip to content

monitor-synapse-analytics-reference.md

Latest commit

 

History

History
184 lines (133 loc) · 15.2 KB

monitor-synapse-analytics-reference.md

File metadata and controls

184 lines (133 loc) · 15.2 KB
title Monitoring data reference for Azure Synapse Analytics
description This article contains important reference material you need when you monitor Azure Synapse Analytics.
ms.date 03/25/2024
ms.custom horz-monitor
ms.topic reference
author ajagadish-24
ms.author ajagadish
ms.service azure-synapse-analytics

Azure Synapse Analytics monitoring data reference

[!INCLUDE horz-monitor-ref-intro]

See Monitor Azure Synapse Analytics for details on the data you can collect for Azure Synapse Analytics and how to use it.

[!INCLUDE horz-monitor-ref-metrics-intro]

Supported metrics for Microsoft.Synapse/workspaces

The following table lists the metrics available for the Microsoft.Synapse/workspaces resource type. [!INCLUDE horz-monitor-ref-metrics-tableheader] [!INCLUDE microsoft-synapse-workspaces-metrics-include]

Azure Synapse Link metrics

Azure Synapse Link emits the following metrics to Azure Monitor:

Metric Aggregation types Description
Link connection events Sum Number of Synapse Link connection events, including start, stop, and failure
Link latency in seconds Max, Min, Avg Synapse Link data processing latency in seconds
Link processed data volume (bytes) Sum Data volume in bytes processed by Synapse Link
Link processed rows Sum Row counts processed by Synapse Link
Link table events Sum Number of Synapse Link table events, including snapshot, removal, and failure

Supported metrics for Microsoft.Synapse/workspaces/bigDataPools

The following table lists the metrics available for the Microsoft.Synapse/workspaces/bigDataPools resource type. [!INCLUDE horz-monitor-ref-metrics-tableheader] [!INCLUDE microsoft-synapse-workspaces-bigdatapools-metrics-include]

Supported metrics for Microsoft.Synapse/workspaces/kustoPools

The following table lists the metrics available for the Microsoft.Synapse/workspaces/kustoPools resource type. [!INCLUDE horz-monitor-ref-metrics-tableheader] [!INCLUDE microsoft-synapse-workspaces-kustopools-metrics-include]

Supported metrics for Microsoft.Synapse/workspaces/scopePools

The following table lists the metrics available for the Microsoft.Synapse/workspaces/scopePools resource type. [!INCLUDE horz-monitor-ref-metrics-tableheader] [!INCLUDE microsoft-synapse-workspaces-scopepools-metrics-include]

Supported metrics for Microsoft.Synapse/workspaces/sqlPools

The following table lists the metrics available for the Microsoft.Synapse/workspaces/sqlPools resource type. [!INCLUDE horz-monitor-ref-metrics-tableheader] [!INCLUDE microsoft-synapse-workspaces-sqlpools-metrics-include]

Details

  • Dedicated SQL pool measures performance in compute data warehouse units (DWUs). Rather than surfacing details of individual nodes such as memory per node or number of CPUs per node, metrics such as MemoryUsedPercent and CPUPercent show general usage trend over a period of time. These trends help administrators understand how a dedicated SQL pool instance is utilized. Changes in memory or CPU footprint could be a trigger for actions such as scale-up or scale-down of DWUs, or investigating queries that might require optimization.

  • DWUUsed represents only high-level usage across the SQL pool and isn't a comprehensive indicator of utilization. To determine whether to scale up or down, consider all factors that DWU can impact, such as concurrency, memory, tempdb size, and adaptive cache capacity. Run your workload at different DWU settings to determine what works best to meet your business objectives.

  • MemoryUsedPercent reflects utilization even if the data warehouse is idle, not active workload memory consumption. Track this metric along with tempdb size and Gen2 cache to decide whether you need to scale for more cache capacity to increase workload performance.

  • Failed and successful connections are reported for a particular data warehouse, not for the server itself.

[!INCLUDE horz-monitor-ref-metrics-dimensions-intro]

[!INCLUDE horz-monitor-ref-metrics-dimensions]

Microsoft.Synapse/workspaces

Result, FailureType, Activity, ActivityType, Pipeline, Trigger, EventType, TableName, LinkTableStatus, LinkConnectionName, SQLPoolName, SQLDatabaseName, JobName, LogicalName, PartitionId, ProcessorInstance

Use the Result dimension of the IntegrationActivityRunsEnded, IntegrationPipelineRunsEnded, IntegrationTriggerRunsEnded, and BuiltinSqlPoolDataRequestsEnded metrics to filter by Succeeded, Failed, or Canceled final state.

Microsoft.Synapse/workspaces/bigDataPools

SubmitterId, JobState, JobType, JobResult

Microsoft.Synapse/workspaces/kustoPools

Database, SealReason, ComponentType, ComponentName, ContinuousExportName, Result, EventStatus, State, RoleInstance, IngestionResultDetails, FailureKind, MaterializedViewName, Kind, Result, QueryStatus, ComponentType, CommandType

Microsoft.Synapse/workspaces/scopePools

JobType, JobResult

Microsoft.Synapse/workspaces/sqlPools

IsUserDefined, Result

[!INCLUDE horz-monitor-ref-resource-logs]

Supported resource logs for Microsoft.Synapse/workspaces

[!INCLUDE microsoft-synapse-workspaces-logs-include]

Note

The event SynapseBuiltinSqlPoolRequestsEnded is emitted only for queries that read data from storage. It's not emitted for queries that process only metadata.

Supported resource logs for Microsoft.Synapse/workspaces/bigDataPools

[!INCLUDE microsoft-synapse-workspaces-bigdatapools-logs-include]

Supported resource logs for Microsoft.Synapse/workspaces/kustoPools

[!INCLUDE microsoft-synapse-workspaces-kustopools-logs-include]

Supported resource logs for Microsoft.Synapse/workspaces/scopePools

[!INCLUDE microsoft-synapse-workspaces-scopepools-logs-include]

Supported resource logs for Microsoft.Synapse/workspaces/sqlPools

[!INCLUDE microsoft-synapse-workspaces-sqlpools-logs-include]

Dynamic Management Views (DMVs)

For more information on these logs, see the following information:

To view the list of DMVs that apply to Synapse SQL, see System views supported in Synapse SQL.

[!INCLUDE horz-monitor-ref-logs-tables]

Synapse Workspaces

Microsoft.Synapse/workspaces

Available Apache Spark configurations

Configuration name Default value Description
spark.synapse.logAnalytics.enabled false To enable the Log Analytics sink for the Spark applications, true. Otherwise, false.
spark.synapse.logAnalytics.workspaceId - The destination Log Analytics workspace ID.
spark.synapse.logAnalytics.secret - The destination Log Analytics workspace secret.
spark.synapse.logAnalytics.keyVault.linkedServiceName - The Key Vault linked service name for the Log Analytics workspace ID and key.
spark.synapse.logAnalytics.keyVault.name - The Key Vault name for the Log Analytics ID and key.
spark.synapse.logAnalytics.keyVault.key.workspaceId SparkLogAnalyticsWorkspaceId The Key Vault secret name for the Log Analytics workspace ID.
spark.synapse.logAnalytics.keyVault.key.secret SparkLogAnalyticsSecret The Key Vault secret name for the Log Analytics workspace
spark.synapse.logAnalytics.uriSuffix ods.opinsights.azure.com The destination Log Analytics workspace URI suffix. If your workspace isn't in Azure global, you need to update the URI suffix according to the respective cloud.
spark.synapse.logAnalytics.filter.eventName.match - Optional. The comma-separated spark event names, you can specify which events to collect. For example: SparkListenerJobStart,SparkListenerJobEnd
spark.synapse.logAnalytics.filter.loggerName.match - Optional. The comma-separated log4j logger names, you can specify which logs to collect. For example: org.apache.spark.SparkContext,org.example.Logger
spark.synapse.logAnalytics.filter.metricName.match - Optional. The comma-separated spark metric name suffixes, you can specify which metrics to collect. For example: jvm.heap.used

Note

  • For Microsoft Azure operated by 21Vianet, the spark.synapse.logAnalytics.uriSuffix parameter should be ods.opinsights.azure.cn.
  • For Azure Government, the spark.synapse.logAnalytics.uriSuffix parameter should be ods.opinsights.azure.us.
  • For any cloud except Azure, the spark.synapse.logAnalytics.keyVault.name parameter should be the fully qualified domain name (FQDN) of the Key Vault. For example, AZURE_KEY_VAULT_NAME.vault.usgovcloudapi.net for AzureUSGovernment.

[!INCLUDE horz-monitor-ref-activity-log]

Related content