You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
# Troubleshoot high data ingestion in Application Insights
10
10
11
-
An increase in billing charges for Application Insights or Log Analytics often occur due to high data ingestion. This article helps you troubleshoot this issue and provides methods to reduce data ingestion costs.
11
+
An increase in billing charges for Application Insights or Log Analytics often occurs due to high data ingestion. This article helps you troubleshoot this issue and provides methods to reduce data ingestion costs.
12
12
13
13
## General troubleshooting steps
14
14
15
15
### Step 1: Identify resources presenting high data ingestion
16
16
17
17
In the Azure portal, navigate to your subscription and select **Cost Management** > **Cost analysis**. This blade offers cost analysis views to chart costs per resource, as follows:
:::image type="content" source="media/troubleshoot-high-data-ingestion/cost-analysis.png" alt-text="Screenshot that shows the 'cost analysis' blade.":::
20
20
21
21
22
22
### Step 2: Identify costly tables with high data ingestion
@@ -52,19 +52,19 @@ Once you've identified an Application Insights resource or a Log Analytics works
52
52
| project-away BillingTelemetrySizeInBytes
53
53
```
54
54
55
-
Similar to the record count queries, these queries above can assist in identifying the most active tables, allowing you to pinpoint specific tables for further investigation.
55
+
Similar to the record count queries, the preceding queries can help identify the most active tables, allowing you to pinpoint specific tables for further investigation.
56
56
57
57
- Using Log Analytics workspace Workbooks
58
58
59
59
In the Azure portal, navigate to your Log Analytics workspace, select **Monitoring** > **Workbooks**, and then select **Usage** under **Log Analytics Workspace Insights**.
:::image type="content" source="media/troubleshoot-high-data-ingestion/log-analytics-usage-workbook.png" alt-text="Screenshot that shows the Log Analytics workbook pane." lightbox="media/troubleshoot-high-data-ingestion/log-analytics-usage-workbook.png":::
62
62
63
63
This workbook provides valuable insights, such as the percentage of data ingestion for each table and detailed ingestion statistics for each resource reporting to the same workspace.
64
64
65
65
### Step 3: Determine factors contributing to high data ingestion
66
66
67
-
After identifying the tables with high data ingestion, focus on the table with the highest activity and determine factors contributing to high data ingestion. This could be a specific application that generates more data than the others, an exception message that gets logged too frequently, or a new logger category that emits too information.
67
+
After identifying the tables with high data ingestion, focus on the table with the highest activity and determine factors contributing to high data ingestion. This might be a specific application that generates more data than the others, an exception message that is logged too frequently, or a new logger category that emits too much information.
68
68
69
69
Here are some sample queries you can use for this identification:
70
70
@@ -111,7 +111,7 @@ exceptions
111
111
```
112
112
113
113
114
-
You can try out different telemetry fields. For example, perhaps you first run the following query and observe there is no obvious cause for the excessive telemetry:
114
+
You can try different telemetry fields. For example, perhaps you first run the following query and observe there's no obvious cause for the excessive telemetry:
115
115
116
116
```Kusto
117
117
dependencies
@@ -147,11 +147,11 @@ exceptions
147
147
| sort by count_ desc
148
148
```
149
149
150
-
### Step 4: Investigate evolution of ingestion over time
150
+
### Step 4: Investigate the evolution of ingestion over time
151
151
152
152
Examine the evolution of ingestion over time based on the factors identified previously. This way can determine whether this behavior has been consistent or if changes occurred at a specific point. By analyzing data in this way, you can pinpoint when the change happened and provide a clearer understanding of the causes behind the high data ingestion. This insight will be important for addressing the issue and implementing effective solutions.
153
153
154
-
In the following queries, the [bin()](/kusto/query/bin-function) Kusto Query Language (KQL) scalar function is used to segment data into 1-day intervals. This approach facilitates trend analysis as you can see how data has changed or not changed over time.
154
+
In the following queries, the [bin()](/kusto/query/bin-function) Kusto Query Language (KQL) scalar function is used to segment data into ome-day intervals. This approach facilitates trend analysis as you can see how data has changed or not changed over time.
155
155
156
156
```Kusto
157
157
dependencies
@@ -174,7 +174,7 @@ dependencies
174
174
175
175
### Scenario 1: High data ingestion in Log Analytics
176
176
177
-
1. Query all tables within a Log Analytics workspace.
177
+
1. Query all tables within a Log Analytics workspace:
178
178
179
179
```Kusto
180
180
search *
@@ -186,9 +186,9 @@ dependencies
186
186
| project-away TotalBilledSize
187
187
```
188
188
189
-
You can get what table is the biggest contributor to costs. Here's an exmaple of `AppTraces`:
189
+
You can know which table is the biggest contributor to costs. Here's an example of `AppTraces`:
190
190
191
-
:::image type="content" source="media/troubleshoot-high-data-ingestion/apptraces-table.png" alt-text="A screenshot thst shows that the AppTraces table is the biggest contributor to costs.":::
191
+
:::image type="content" source="media/troubleshoot-high-data-ingestion/apptraces-table.png" alt-text="Screenshot that shows that the AppTraces table is the biggest contributor to costs.":::
192
192
193
193
2. Query the specific application driving the costs for traces:
194
194
@@ -202,7 +202,7 @@ dependencies
202
202
| project-away TotalBilledSize
203
203
```
204
204
205
-
:::image type="content" source="media/troubleshoot-high-data-ingestion/specific-application-driving-costs-for-traces.png" alt-text="A screenshot thst shows the specific application driving the costs for traces.":::
205
+
:::image type="content" source="media/troubleshoot-high-data-ingestion/specific-application-driving-costs-for-traces.png" alt-text="Screenshot that shows the specific application driving the costs for traces.":::
206
206
207
207
3. Run the following query specific to that application and look further into the specific logger categories sending telemetry to the `AppTraces` table:
208
208
@@ -220,7 +220,7 @@ dependencies
220
220
221
221
The result shows two main categories responsible for the costs:
222
222
223
-
:::image type="content" source="media/troubleshoot-high-data-ingestion/logger-categories-sending-telemetry-to-apptraces.png" alt-text="A screenshot thst shows the specific logger categories sending telemetry to the AppTraces table.":::
223
+
:::image type="content" source="media/troubleshoot-high-data-ingestion/logger-categories-sending-telemetry-to-apptraces.png" alt-text="Screenshot that shows the specific logger categories sending telemetry to the AppTraces table.":::
224
224
225
225
### Scenario 2: High data ingestion in Application Insight
226
226
@@ -235,9 +235,9 @@ To determine the factors contributing to the costs, follow these steps:
235
235
| sort by count_ desc
236
236
```
237
237
238
-
Here's an exmaple that shows Azure Functions is generating lots of trace and exception telemetry:
238
+
The following exmaple shows Azure Functions is generating lots of trace and exception telemetry:
239
239
240
-
:::image type="content" source="media/troubleshoot-high-data-ingestion/table-sdkversion-count.png" alt-text="A screenshot thst shows what table and SDK is generating most Trace and Exception telemetry.":::
240
+
:::image type="content" source="media/troubleshoot-high-data-ingestion/table-sdkversion-count.png" alt-text="Screenshot that shows which table and SDK is generating the most Trace and Exception telemetry.":::
241
241
242
242
243
243
2. Run the following query to get the specific app generating more traces than the others:
@@ -251,7 +251,7 @@ To determine the factors contributing to the costs, follow these steps:
251
251
```
252
252
253
253
254
-
:::image type="content" source="media/troubleshoot-high-data-ingestion/app-generating-more-traces.png" alt-text="A screenshot thst shows what app is generating most traces.":::
254
+
:::image type="content" source="media/troubleshoot-high-data-ingestion/app-generating-more-traces.png" alt-text="Screenshot that shows which app is generating the most traces.":::
255
255
256
256
3. Refine the query to include that specific app and generate a count of records per each individual message:
257
257
@@ -266,45 +266,45 @@ To determine the factors contributing to the costs, follow these steps:
266
266
267
267
The result can show the specific message increasing ingestion costs:
268
268
269
-
:::image type="content" source="media/troubleshoot-high-data-ingestion/app-message-count.png" alt-text="A screenshot thst shows a count of records per each individual message.":::
269
+
:::image type="content" source="media/troubleshoot-high-data-ingestion/app-message-count.png" alt-text="Screenshot that shows a count of records per each message.":::
270
270
271
271
### Scenario 3: Reach daily cap unexpectedly
272
272
273
-
Assume you reached daily cap unexpected on September 4th. Use the following query to obtain a count of custom events and identify the most recent timestamp associated with each event:
273
+
Assume you reached the daily cap unexpectedly on September 4th. Use the following query to obtain a count of custom events and identify the most recent timestamp associated with each event:
274
274
275
275
```Kusto
276
276
customEvents
277
277
| where timestamp between(datetime(8/25/2024) .. 15d)
278
278
| summarize count(), min(timestamp) by name
279
279
```
280
280
281
-
This analysis indicates that certain events started ingested on September 4th and subsequently became noisy very quickly.
281
+
This analysis indicates that certain events started being ingested on September 4th and subsequently became noisy very quickly.
282
282
283
-
:::image type="content" source="media/troubleshoot-high-data-ingestion/custom-events.png" alt-text="A screenshot thst shows a count of custom events.":::
283
+
:::image type="content" source="media/troubleshoot-high-data-ingestion/custom-events.png" alt-text="Screenshot that shows a count of custom events.":::
284
284
285
285
## Reduce data ingestion costs
286
286
287
287
After identifying the factors in the Azure Monitor tables responsible for unexpected data ingestion, reduce data ingestion costs using the following methods per your scenarios:
288
288
289
-
### Method 1: Update daily cap configuration
289
+
### Method 1: Update the daily cap configuration
290
290
291
-
Adjust the daily cap to prevent excess telemetry ingestion.
291
+
Adjust the daily cap to prevent excessive telemetry ingestion.
292
292
293
-
### Method 2: Switch table plan
293
+
### Method 2: Switch the table plan
294
294
295
-
Switch to another supported table plan for Application Insights. Billing for data ingestion depends on the table plan and the region of the Log Analytics workspace. See [Table plans](/azure/azure-monitor/logs/data-platform-logs) and [Tables that support the Basic table plan in Azure Monitor Logs](/azure/azure-monitor/logs/basic-logs-azure-tables).
295
+
Switch to another supported table plan for Application Insights. Billing for data ingestion depends on the table plan and the region of the Log Analytics workspace. See [Table plans](/azure/azure-monitor/logs/data-platform-logs#table-plans) and [Tables that support the Basic table plan in Azure Monitor Logs](/azure/azure-monitor/logs/basic-logs-azure-tables).
296
296
297
-
### Method 3: Use telemetry SDK features for Java agent
297
+
### Method 3: Use telemetry SDK features for the Java agent
298
298
299
299
The default recommended solution is using [sampling overrides](/azure/azure-monitor/app/java-standalone-sampling-overrides). The Application Insights Java agent provides [two types of sampling](/azure/azure-monitor/app/java-standalone-config#sampling). A common use case is [suppressing collecting telemetry for health checks](/azure/azure-monitor/app/java-standalone-sampling-overrides#suppress-collecting-telemetry-for-health-checks).
300
300
301
-
There are some supplemental methods to sampling overrides:
301
+
There are some supplemental methods for sampling overrides:
302
302
303
-
- Reduce cost from the `traces` table:
303
+
- Reduce the cost from the `traces` table:
304
304
305
-
-[Reduce the telemetry log level](/azure/azure-monitor/app/java-standalone-config#autocollected-logging)
306
-
-[Remove application (not frameworks/libs) logs with MDC attribute and sampling override](/azure/azure-monitor/app/java-standalone-sampling-overrides#suppress-collecting-telemetry-for-log)
307
-
- Disable log instrumentation by updating the *applicationinsights.json* file:
305
+
-[Reduce the telemetry log level](/azure/azure-monitor/app/java-standalone-config#autocollected-logging).
306
+
-[Remove application (not frameworks/libs) logs with the MDC attribute and sampling override](/azure/azure-monitor/app/java-standalone-sampling-overrides#suppress-collecting-telemetry-for-log).
307
+
- Disable log instrumentation by updating the **applicationinsights.json** file:
308
308
309
309
```JSON
310
310
{
@@ -316,36 +316,36 @@ There are some supplemental methods to sampling overrides:
316
316
}
317
317
```
318
318
319
-
- Reduce cost from the `dependencies` table:
319
+
- Reduce the cost from the `dependencies` table:
320
320
321
321
- [Suppress collecting telemetry for the Java method producing the dependency telemetry](/azure/azure-monitor/app/java-standalone-sampling-overrides#suppress-collecting-telemetry-for-a-java-method).
322
322
- [Disable the instrumentation](/azure/azure-monitor/app/java-standalone-config#suppress-specific-autocollected-telemetry) producing the dependency telemetry data.
323
323
324
-
If the dependency is a database call, you then won't see the database on the application map. If you remove the dependency instrumentation of an HTTP call or a message (for example a Kafka message), all the downstream telemetry data are dropped.
324
+
If the dependency is a database call, you won't see the database on the application map. If you remove the dependency instrumentation of an HTTP call or a message (for example, a Kafka message), all the downstream telemetry data are dropped.
325
325
326
-
- Reduce cost from the `customMetrics` table:
326
+
- Reduce the cost from the `customMetrics` table:
327
327
328
-
- [Increase the metrics interval](/azure/azure-monitor/app/java-standalone-config#metric-interval)
329
-
- [Exclude a metric with a telemetry processor](/azure/azure-monitor/app/java-standalone-telemetry-processors#metric-filter)
330
-
- [Increase the heartbeat interval](/azure/azure-monitor/app/java-standalone-config#heartbeat)
328
+
- [Increase the metrics interval](/azure/azure-monitor/app/java-standalone-config#metric-interval).
329
+
- [Exclude a metric with a telemetry processor](/azure/azure-monitor/app/java-standalone-telemetry-processors#metric-filter).
330
+
- [Increase the heartbeat interval](/azure/azure-monitor/app/java-standalone-config#heartbeat).
331
331
332
-
- Reduce OpenTelemetry attributes cost:
332
+
- Reduce the cost of OpenTelemetry attributes:
333
333
334
-
OpenTelemetry attributes are added to the **customDimensions** column. They are represented as properties in Application Insights. You can remove attributes by using [an attribute telemetry processor](/azure/azure-monitor/app/java-standalone-telemetry-processors#attribute-processor). For more information, see [Telemetry processor examples - Delete](/azure/azure-monitor/app/java-standalone-telemetry-processors-examples#delete).
334
+
OpenTelemetry attributes are added to the **customDimensions** column. They're represented as properties in Application Insights. You can remove attributes by using [an attribute telemetry processor](/azure/azure-monitor/app/java-standalone-telemetry-processors#attribute-processor). For more information, see [Telemetry processor examples - Delete](/azure/azure-monitor/app/java-standalone-telemetry-processors-examples#delete).
335
335
336
-
### Method 4: Update application code (log levels or exceptions)
336
+
### Method 4: Update the application code (log levels or exceptions)
337
337
338
-
In some scenarios, updating the application code directly might help reduce the amount of telemetry being generated and consumed by the Application Insights backend service. A common example might be a noisy exception surfaced by the application.
338
+
In some scenarios, updating the application code directly might help reduce the amount of telemetry generated and consumed by the Application Insights backend service. A common example might be a noisy exception surfaced by the application.
0 commit comments