|
1 | 1 | --- |
2 | | -title: What to ingest into the data lake |
| 2 | +title: Which logs should you ingest into the data lake |
3 | 3 | description: How to choose which log sources to ingest into your Microsoft Sentinel data lake. |
4 | 4 | ms.topic: conceptual |
5 | 5 | ms.date: 01/29/2026 |
6 | 6 | author: EdB-MSFT |
7 | 7 | ms.author: edbaynash |
8 | 8 | ms.service: microsoft-sentinel |
9 | 9 | ms.subservice: sentinel-graph |
| 10 | + |
| 11 | +# Customer intent: As a security architect, I want to decide which log sources belong in the Sentinel data lake so that I balance cost and visibility. |
| 12 | + |
10 | 13 | --- |
11 | 14 |
|
12 | 15 |
|
13 | 16 |
|
14 | | -# What to ingest into the data lake |
| 17 | +# Which logs should you ingest into the data lake? |
15 | 18 |
|
16 | | -After onboarding to Microsoft Sentinel data lake, you can decide which logs to ingest into the data lake. |
| 19 | +After onboarding to Microsoft Sentinel data lake, decide which logs to ingest into the data lake. |
17 | 20 |
|
18 | | -The analytics tier in Sentinel provides real-time analysis and alerting capabilities using log data ingested into Sentinel workspaces. The analytics tier supports the following use cases: |
19 | | -+ **Real-time detection and correlation**: Immediate alerting on critical events, such as endpoints, identity, cloud security, perimeter. |
| 21 | +The analytics tier in Sentinel provides real-time analysis and alerting capabilities by using log data ingested into Sentinel workspaces. The analytics tier supports the following use cases: |
| 22 | ++ **Real-time detection and correlation**: Immediate alerting on critical events such as endpoints, identity, cloud security, perimeter. |
20 | 23 | + **Rapid investigation**: Live searches for active incidents and threat responses. |
21 | 24 | + **High-fidelity, actionable logs**: Focus on sources with direct security value, such as EDR signals, privileged access, authentication, threat alerts. |
22 | 25 |
|
23 | | -The data lake tier in Sentinel provides large-scale, long term storage and advanced analytics capabilities. The data lake supports the following use cases: |
24 | | -+ **High-volume, lower-priority logs**: Sources that are valuable for deep forensics, analysis of past incidents to understand attack vectors and impacts, or periodic hunts but costly to keep the analytics tier. |
| 26 | +The data lake tier in Sentinel provides large-scale, long-term storage and advanced analytics capabilities. The data lake supports the following use cases: |
| 27 | ++ **High-volume, lower-priority logs**: Sources that are valuable for deep forensics, analysis of past incidents to understand attack vectors and impacts, or periodic hunts but are costly to keep in the analytics tier. |
25 | 28 | + **Analytics and threat hunting**: Cross-log searching, long-term trend analysis, and proactive exploration of historical data to identify hidden threats and patterns. |
26 | | -+ **Batch analytics and summarization**: Use Spark, KQL, or similar tools to enrich, correlate, or summarize data before forwarding only the high-risk signals to the analytics tier for active monitoring. |
| 29 | ++ **Batch analytics and summarization**: Use Spark, KQL, or similar tools to enrich, correlate, or summarize data before forwarding only high-risk signals to the analytics tier. |
27 | 30 | + **Advanced analytics and machine learning**: Use big data techniques to uncover complex relationships and trends. |
28 | 31 |
|
29 | | -Depending on your organization's security needs, you may choose to ingest different log sources into the data |
30 | | -lake. Store high volume logs that are less critical for real-time detection but valuable for deep analysis and forensics in the lake and retain only high-value logs in the analytics tier. |
| 32 | +Depending on your organization's security needs, you might choose to ingest different log sources into the data |
| 33 | +lake. Store high-volume logs that are less critical for real-time detection but valuable for deep analysis and forensics in the lake, and retain only high-value logs in the analytics tier. |
31 | 34 |
|
32 | | -The following table provides guidance on common log source types, their typical log volume, and their value for different security use cases. Use this information to help determine which log sources to ingest into your data lake based on your organization's specific needs and priorities. |
| 35 | +Use the following table to prioritize which sources belong in the data lake versus the analytics tier. |
33 | 36 |
|
34 | | -| Log source type | Typical Log Volume |Value for real-time threat detection/alerting | Value for threat hunting | Value for incident investigation/forensics | Ingest to data lake | |
| 37 | +| Log source type | Typical log volume | Value for real-time threat detection and alerting | Value for threat hunting | Value for incident investigation and forensics | Ingest to data lake | |
35 | 38 | |-------------------------------------------------|--------------------|-------------------------------------|----------------|-----------------------------------|-----------------------| |
36 | 39 | | AAA (TACACS/Radius) | Medium | High | High | High | Yes | |
37 | 40 | | Active Directory (on-premises) | High | High | High | High | No | |
@@ -92,4 +95,11 @@ The following table provides guidance on common log source types, their typical |
92 | 95 | | Web Application Firewall (WAF) Logs | Medium | High | High | High | Yes | |
93 | 96 | | Windows Server Events | High | High | High | High | No | |
94 | 97 | | XDR Source Logs (Defender: Office, Identity, Endpoint, CloudApp) | Medium | High | High | High | No | |
95 | | -| Zoom Meeting Logs | Low-Medium | Low | Low | Medium | Yes | |
| 98 | +| Zoom Meeting Logs | Low-Medium | Low | Low | Medium | Yes | |
| 99 | + |
| 100 | + |
| 101 | +## Related articles |
| 102 | + |
| 103 | +- [Manage data tiers and retention in Microsoft Sentinel](../manage-data-overview.md) |
| 104 | +- [Configure table settings in Microsoft Sentinel](../manage-table-tiers-retention.md) |
| 105 | +- [Set up connectors for the Microsoft Sentinel data lake](../datalake/sentinel-lake-connectors.md) |
0 commit comments