Skip to content

Commit 5c72355

Browse files
b-ahibbardb-ahibbard
committed
merge main
2 parents a45d072 + 2052ffd commit 5c72355

4 files changed

Lines changed: 98 additions & 5 deletions

File tree

‎articles/sentinel/TOC.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,9 @@
1616
href: microsoft-sentinel-defender-portal.md
1717
- name: Data lake exploration
1818
items:
19+
- name: Compare KQL jobs, summary rules, and search jobs
20+
href: datalake/kql-jobs-summary-rules-search-jobs.md
21+
displayName: data lake
1922
- name: KQL for data lake exploration
2023
items:
2124
- name: Overview

‎articles/sentinel/datalake/kql-jobs-summary-rules-search-jobs.md‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
---
2+
title: Compare KQL jobs, summary rules, and search jobs
3+
titleSuffix: Microsoft Security
4+
description: Compare KQL jobs, summary rules, and search jobs in Microsoft Sentinel to choose the best tool for querying and analyzing security data.
5+
author: EdB-MSFT
6+
ms.service: microsoft-sentinel
7+
ms.topic: how-to
8+
ms.subservice: sentinel-graph
9+
ms.date: 08/19/2025
10+
ms.author: edbaynash
11+
12+
ms.collection: ms-security
13+
14+
# Customer intent: As a security analyst, I need to choose the right tool for querying and analyzing data in Microsoft Sentinel.
15+
16+
---
17+
18+
# Compare KQL jobs, summary rules, and search jobs
19+
20+
This article compares KQL jobs, summary rules, and search jobs in Microsoft Sentinel. These features let you query and analyze data in Microsoft Sentinel, and each serves different purposes and use cases.
21+
22+
+ **KQL jobs**: Run one-time or scheduled asynchronous queries on data stored in the Microsoft Sentinel data lake. They're best for incident investigations using historical logs, enrichment using low-fidelity logs, and scenarios that need queries with joins or unions across multiple tables. For more information, see [KQL jobs](kql-jobs.md).
23+
24+
+ **Summary rules**: Run scheduled queries that aggregate and store insights from large sets of log data. They're ideal for frequent summarization tasks, like aggregating high-volume logs such as network insights. These rules run in the background and populate custom tables in Log Analytics. For more information, see [Summary rules](../summary-rules.md).
25+
26+
+ **Search jobs**: Run one-time, long-running asynchronous queries across large datasets. Search jobs are useful when you need to hydrate large volumes of data from a single table into a new custom table within the Analytics tier as a one-time operation for further investigation or forensic analysis. For more information, see [Search jobs](../search-jobs.md).
27+
28+
## Feature comparison
29+
30+
| Feature | KQL Jobs | Summary Rules | Search jobs |
31+
|----------------------|----------------------------------------|------------------------------------|----------------------------------|
32+
| **Purpose** | Run ad-hoc or scheduled queries for investigation and enrichment | Aggregate and store insights from high-volume logs | Run async queries on large datasets to store results in the analytics tier |
33+
| **Data tier** | Microsoft Sentinel data lake tier | Analytics, auxiliary, basic, data lake (except for tables in the default workspace) | Analytics, auxiliary, basic, data lake (except for tables in the default workspace) |
34+
| **Workspace scope** | Any Microsoft Sentinel workspace connected to Microsoft Defender | Any Microsoft Sentinel workspace connected to Microsoft Defender | Any Microsoft Sentinel workspace |
35+
| **Table scope** | Multiple tables | Multiple tables | Single table |
36+
| **Query language** | [KQL jobs supported operators](kql-jobs.md#considerations-and-limitations) | KQL <br>All KQL operators supported except for:<br>• [Cross-resource queries](/azure/azure-monitor/logs/cross-workspace-query) queries, which use the workspaces(), app(), and resource() expressions, and [cross-service queries](/azure/azure-monitor/logs/azure-monitor-data-explorer-proxy), which use the `ADX()` and `ARG()` expressions<br>• Plugins that reshape the data schema, including [bag](/kusto/query/bag-unpack-plugin) [unpack](/azure/data-explorer/kusto/query/bag-unpack-plugin), [narrow](/azure/data-explorer/kusto/query/narrow-plugin), and [pivot](/azure/data-explorer/kusto/query/pivot-plugin).<br>• User-defined functions aren't supported | [Limited KQL operators](/azure/azure-monitor/logs/search-jobs#kql-query-considerations) |
37+
| **Join support** | Supported | Analytics tier: supported<br>Basic: join up to five Analytics tables using [lookup](/azure/data-explorer/kusto/query/lookup-operator) operator | Not supported |
38+
| **Scheduling frequency** | On-demand<br>Daily, weekly, monthly | 20 minutes to 24 hours | On-demand (long-running searches support up to a 24-hour timeout) |
39+
| **Lookback period** | Up to 12 years | Up to 1 day | Up to 12 years |
40+
| **Timespan** | - | - | Up to 1 year |
41+
| **Timeout** | 1 hour | 10 minutes | 24 hours |
42+
| **Maximum number of results**| Dependent on query timeout |500,000 records |1 million records
43+
| **Pricing model** | GB of data analyzed | Analytics tier: free<br>Basic and auxiliary tier: Data scan Log Analytics pricing model | GB of data analyzed |
44+
45+
46+
47+
## Usage scenarios and feature choice
48+
49+
The following section helps you decide which feature is best for your needs.
50+
51+
If you have any of the following requirements, use KQL jobs:
52+
53+
+ You're onboarded to the Microsoft Sentinel data lake.
54+
+ You require lookback greater than 24 hours.
55+
+ You want to query historical data of up to 12 years.
56+
+ You need to run complex queries involving full KQL operators including joins or unions.
57+
+ You need ad-hoc investigation capabilities.
58+
+ Data is in the default workspace.
59+
60+
61+
Use summary rules if you have any of the following requirements:
62+
63+
+ Your tenant isn't onboarded to Microsoft Sentinel data lake, and your data may still reside in Auxiliary or Basic tiers.
64+
+ You need lookback within 24 hours.
65+
+ You need frequent summarization (for example, every 20 minutes).
66+
+ You want to use out-of-the-box templates.
67+
68+
If you have any of the following requirements, use search jobs:
69+
70+
+ Your Microsoft Sentinel workspace isn't connected to Defender portal and your data resides in Analytics or basic tiers.
71+
+ You have data in archive tier. If you're onboarded to Microsoft Sentinel data lake, to access data older than your onboarding date, use search jobs. For data from your onboarding date onward, use KQL jobs.
72+
+ You need to hydrate large volumes of data from a single table.
73+
+ Your use case involves targeted extraction rather than frequent summarization or complex multi-table joins.
74+
+ You want to analyze up to one year of historical data within a table from any data tier.
75+
76+
## Related articles
77+
78+
- [KQL and the Microsoft Sentinel data lake (preview)](kql-overview.md)
79+
- [Jupyter notebooks and the Microsoft Sentinel data lake (preview)](notebooks-overview.md)
80+
- [Aggregate Microsoft Sentinel data with summary rules](../summary-rules.md)
81+
- [Search for specific events across large datasets in Microsoft Sentinel](../search-jobs.md)

‎articles/sentinel/datalake/kql-jobs.md‎

Lines changed: 11 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -17,20 +17,26 @@ ms.collection: ms-security
1717

1818

1919
# Create KQL jobs in the Microsoft Sentinel data lake (preview)
20-
2120

22-
A job is a one-time or repeatedly scheduled task that runs a KQL (Kusto Query Language) query against the data in the data lake tier to promote the results to the analytics tier. Once in the analytics tier, use the advanced hunting KQL editor to query the data. Promoting data to the analytics tier has the following benefits:
21+
KQL jobs are one-time or scheduled asynchronous KQL queries on data in the Microsoft Sentinel data lake. Jobs are useful for investigative and analytical scenarios for example;
22+
+ Long-running one-time queries for incident investigations and incident response (IR)
23+
+ Data aggregation tasks that support enrichment workflows using low-fidelity logs
24+
+ Historical threat intelligence (TI) matching scans for retrospective analysis
25+
+ Anomaly detection scans that identify unusual patterns across multiple tables
2326

24-
+ Combine current and historical data in the analytics tier to run advanced analytics and machine learning models on your data.
27+
KQL jobs are especially effective when queries use joins or unions across different datasets.
2528

29+
Jobs are also used to promote the data from the data lake tier to the analytics tier. Once in the analytics tier, use the advanced hunting KQL editor to query the data. Promoting data to the analytics tier has the following benefits:
30+
31+
+ Combine current and historical data in the analytics tier to run advanced analytics and machine learning models on your data.
2632
+ Reduce query costs by running queries in the analytics tier.
2733
+ Combine data from multiple workspaces to a single workspace in the analytics tier.
2834
+ Combine Microsoft Entra ID, Microsoft 365, and Microsoft Resource Graph data in the analytics tier to run advanced analytics across data sources.
2935

3036
> [!NOTE]
3137
> Storage in the analytics tier incurs higher billing rates than in the data lake tier. To reduce costs, only promote data that you need to analyze further. Use the KQL in your query to project only the columns you need, and filter the data to reduce the amount of data promoted to the analytics tier.
3238
33-
When promoting data to the analytics tier, make sure that the destination workspace is visible in the advanced hunting query editor. You can only query connected workspaces in the advanced hunting query editor. You will not be able to see data promoted to workspaces that aren't connected or to the default workspace in advance hunting. For more information on connected workspaces, see [Connect a workspace](/defender-xdr/advanced-hunting-microsoft-defender#connect-a-workspace). You can promote data to a new table or append the results to an existing table in the analytics tier. When creating a new table, the table name is suffixed with *_KQL_CL* to indicate that the table was created by a KQL job.
39+
When promoting data to the analytics tier, make sure that the destination workspace is visible in the advanced hunting query editor. You can only query connected workspaces in the advanced hunting query editor. You won't be able to see data promoted to workspaces that aren't connected or to the default workspace in advance hunting. For more information on connected workspaces, see [Connect a workspace](/defender-xdr/advanced-hunting-microsoft-defender#connect-a-workspace). You can promote data to a new table or append the results to an existing table in the analytics tier. When creating a new table, the table name is suffixed with *_KQL_CL* to indicate that the table was created by a KQL job.
3440

3541
## Prerequisites
3642

@@ -151,7 +157,7 @@ The following standard columns aren't supported for export. These columns are ov
151157
+ _IsBillable
152158
+ _WorkspaceId
153159

154-
+ `TimeGenerated` will be overwritten if it's older that 2 days. To preserve the original event time, we recommend writing the source timestamp to a separate column.
160+
+ `TimeGenerated` is overwritten if it's older that 2 days. To preserve the original event time, we recommend writing the source timestamp to a separate column.
155161

156162
For service limits, see [Microsoft Sentinel data lake (preview) service limits](sentinel-lake-service-limits.md#service-parameters-and-limits-for-kql-jobs).
157163

‎articles/sentinel/datalake/sentinel-lake-connectors.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,9 @@ ms.date: 07/09/2025
1010
ms.author: edbaynash
1111

1212
ms.collection: ms-security
13+
14+
# Customer intent: As a security admin, I want to set up connectors for Microsoft Sentinel data lake so that I can mirror and retain security data for long-term analysis.
15+
1316
---
1417

1518
# Set up connectors for the Microsoft Sentinel data lake (preview)

0 commit comments

Comments
 (0)