Merge pull request #302361 from MicrosoftDocs/release-preview-sentinel-lake

Release preview sentinel lake

Author: Albertyang0

articles/sentinel/TOC.yml

@@ -5,22 +5,58 @@
   items:
   - name: What is Microsoft Sentinel?
     href: overview.md
+  - name: Microsoft Sentinel data lake overview
+    href: datalake/sentinel-lake-overview.md
+    displayName: data lake
   - name: What's new
     href: whats-new.md
   - name: Best practices
     href: best-practices.md
   - name: Experience in Defender portal
     href: microsoft-sentinel-defender-portal.md
+- name: Data lake exploration
+  items:  
+  - name: KQL for data lake exploration
+    items:
+    - name: Overview 
+      href: datalake/kql-overview.md
+      displayName: data lake
+    - name: Run KQL queries
+      href: datalake/kql-queries.md
+      displayName: data lake
+    - name: Create KQL jobs
+      href: datalake/kql-jobs.md 
+      displayName: data lake 
+    - name: Manage KQL jobs
+      href: datalake/kql-manage-jobs.md
+      displayName: data lake
+    - name: Troubleshoot KQL for the lake
+      href: datalake/kql-troubleshoot.md
+      displayName: data lake  
+  - name: Notebooks for data lake exploration
+    items:
+    - name: Overview
+      href: datalake/notebooks-overview.md
+      displayName: data lake
+    - name: Run notebooks
+      href: datalake/notebooks.md
+      displayName: data lake
+    - name: Microsoft Sentinel provider class reference
+      href: datalake/sentinel-provider-class-reference.md
+      displayName: data lake
+    - name: Create and manage notebook jobs
+      href: datalake/notebook-jobs.md
+      displayName: data lake
+    - name: Notebook examples for data lake exploration
+      href: datalake/notebook-examples.md
 - name: Plan
   items:
   - name: Deployment planning guide  
     href: deploy-overview.md
   - name: Prerequisites      
     href: prerequisites.md
   - name: Workspace architecture
-    items:
-    - name: Design workspace architecture
-      href: /azure/azure-monitor/logs/workspace-design?toc=/azure/sentinel/TOC.json&bc=/azure/sentinel/breadcrumb/toc.json        
+    items:       
     - name: Review sample workspace designs
       href: sample-workspace-designs.md
     - name: Prepare for multiple workspaces
@@ -29,6 +65,7 @@
     href: prioritize-data-connectors.md 
   - name: Plan roles and permissions
     href: roles.md
+    displayName: data lake
   - name: Plan interactive and long-term data retention
     href: log-plans.md
   - name: Plan costs
@@ -55,6 +92,12 @@
     href: quickstart-onboard.md
   - name: Connect Microsoft Sentinel to the Defender portal
     href: /unified-secops-platform/microsoft-sentinel-onboard?toc=/azure/sentinel/TOC.json&bc=/azure/sentinel/breadcrumb/toc.json
+  - name: Onboard to Microsoft Sentinel data lake
+    href: datalake/sentinel-lake-onboarding.md
+    displayName: data lake
+  - name: Set up connectors for the Microsoft Sentinel data lake
+    href: datalake/sentinel-lake-connectors.md
+    displayName: data lake    
   - name: Configure content
     href: configure-content.md
   - name: Set up multiple workspaces
@@ -356,6 +399,14 @@
       href: summary-rules.md
     - name: Aggregate insights from raw data into an Auxiliary table
       href: summary-rules-tutorial.md
+- name: Manage data
+  items:      
+  - name: Data management overview
+    href: manage-data-overview.md
+    displayName: table management, tiers, retention, manage data, tables
+  - name: Manage tables, tiers, and retention
+    href: manage-table-tiers-retention.md
+    displayName: table management, tiers, retention, tables  
 - name: Integrate threat intelligence
   items:
   - name: Overview
@@ -665,6 +716,7 @@
     href: soc-optimization/soc-optimization-reference.md
 - name: Manage Microsoft Sentinel
   items:
+
   - name: Manage costs and billing
     items:      
     - name: Monitor costs
@@ -675,9 +727,7 @@
       href: enroll-simplified-pricing-tier.md
     - name: Optimize costs with pre-purchase plan
       href: billing-pre-purchase-plan.md
-    - name: Manage data retention
-      href: /azure/azure-monitor/logs/data-retention-configure?toc=/azure/sentinel/TOC.json&bc=/azure/sentinel/breadcrumb/toc.json
-    - name: Auxiliary logs use cases
+    - name: Data lake use cases
       href: basic-logs-use-cases.md
   - name: Manage multiple workspaces 
     items:
@@ -695,7 +745,7 @@
       href: multiple-workspace-view.md
     - name: Manage your intellectual property in Microsoft Sentinel
       href: mssp-protect-intellectual-property.md
-  - name: Manage workspace access
+  - name: Manage workspace access with resource-context RBAC
     href: resource-context-rbac.md
   - name: Set up customer-managed keys
     href: customer-managed-keys.md
@@ -717,6 +767,9 @@
       href: monitor-analytics-rule-integrity.md
   - name: Auditing Microsoft Sentinel with Azure Activity Logs
     href: audit-sentinel-data.md
+  - name: Audit log for Microsoft Sentinel data lake
+    href: datalake/auditing-lake-activities.md
+    displayName: data lake
   - name: Remove Microsoft Sentinel from your workspaces
     href: offboard.md
 - name: Build and publish Microsoft Sentinel solutions
@@ -757,8 +810,11 @@
     href: aws-s3-troubleshoot.md
 - name: Reference
   items:
-  - name: Service limits
+  - name: Microsoft Sentinel service limits
     href: sentinel-service-limits.md
+  - name: Microsoft Sentinel data lake service limits
+    href: datalake/sentinel-lake-service-limits.md  
+    displayName: data lake
   - name: Microsoft Sentinel REST-API
     href: /rest/api/securityinsights/
   - name: OOTB content centralization changes

articles/sentinel/basic-logs-use-cases.md

@@ -1,10 +1,10 @@
 ---
-title: When to use Auxiliary Logs in Microsoft Sentinel
-description: Learn what log sources might be appropriate for Auxiliary Log or Basic Log ingestion and what are the attributes to look for to decide about other sources.
-author: cwatson-cat
-ms.author: cwatson
+title: When to use the Microsoft Sentinel data lake
+description: Learn what log sources might be appropriate for the Microsoft Sentinel data lake and what attributes to look for, to decide about other sources.
+author: EdB-MSFT
+ms.author: edbaynash
 ms.topic: conceptual
-ms.date: 03/31/2025
+ms.date: 07/15/2025
 appliesto:
     - Microsoft Sentinel in the Microsoft Defender portal
     - Microsoft Sentinel in the Azure portal
@@ -14,12 +14,14 @@ ms.collection: usx-security
 #Customer intent: As a security analyst, I want to ingest high-volume, verbose logs into a cost-effective storage solution so that I can enhance my threat hunting and incident investigation capabilities.
 
 ---
-# Log sources to use for Auxiliary Logs ingestion
+# Log sources to use for the Microsoft Sentinel data lake
 
-This article highlights log sources to consider configuring as Auxiliary Logs (or Basic Logs) when they're stored in Log Analytics tables. Before choosing a log type for which to configure a given table, do the research to see which is most appropriate. For more information about data categories and log data plans, see [Log retention plans in Microsoft Sentinel](log-plans.md).
+This article highlights log sources to consider configuring as data lake tier only when enabling a connector. Before choosing a tier for which to configure a given table, check which tier is most appropriate for your use case. For more information about data categories and data tiers, see [Log retention plans in Microsoft Sentinel](log-plans.md).
 
 [!INCLUDE [unified-soc-preview](includes/unified-soc-preview.md)]
 
+[!INCLUDE [sentinel-lake-preview](includes/sentinel-lake-preview.md)]
+
 ## Storage access logs for cloud providers
 
 Storage access logs can provide a secondary source of information for investigations that involve exposure of sensitive data to unauthorized parties. These logs can help you identify issues with system or user permissions granted to the data.
@@ -64,7 +66,8 @@ A new and growing source of log data is Internet of Things (IoT)-connected devic
 
 ## Next steps
 
-- [Select a table plan based on data usage in a Log Analytics workspace](/azure/azure-monitor/logs/logs-table-plans)
-- [Set up a table with the Auxiliary plan in your Log Analytics workspace](/azure/azure-monitor/logs/create-custom-table-auxiliary)
-- [Manage data retention in a Log Analytics workspace](/azure/azure-monitor/logs/data-retention-configure)
-- [Start an investigation by searching for events in large datasets (preview)](investigate-large-datasets.md)
+- [What is the Microsoft Sentinel data lake? (preview)](datalake/sentinel-lake-overview.md)
+- [Manage data tiers and retention in Microsoft Defender Portal (preview)](manage-data-overview.md)
+- [KQL and the Microsoft Sentinel data lake (preview)](datalake/kql-overview.md)
+- [Jupyter notebooks in the Microsoft Sentinel data lake (preview)](datalake/notebooks-overview.md)
+

articles/sentinel/best-practices.md

@@ -1,8 +1,8 @@
 ---
 title: Best practices for Microsoft Sentinel
 description: Learn about best practices to employ when managing your Log Analytics workspace for Microsoft Sentinel.
-author: cwatson-cat
-ms.author: cwatson
+author: EdB-MSFT
+ms.author: edbaynash
 ms.topic: conceptual
 ms.date: 07/16/2025
 
@@ -21,13 +21,17 @@ Best practice guidance is provided throughout the technical documentation for Mi
 
 Start with the [deployment guide for Microsoft Sentinel](deploy-overview.md). The deployment guide covers the high level steps to plan, deploy, and fine-tune your Microsoft Sentinel deployment. From that guide, select the provided links to find detailed guidance for each stage in your deployment.
 
+## Adopt a single-platform architecture
+
+Microsoft Sentinel is integrated with a modern data lake that offers affordable, long-term storage enabling teams to simplify data management, optimize costs, and accelerate the adoption of AI. The Microsoft Sentinel data lake (preview) enables a single-platform architecture for security data and empowers analysts with a unified query experience while leveraging Microsoft Sentinel’s rich connector ecosystem. For more information, see [Microsoft Sentinel data lake (preview)](datalake/sentinel-lake-overview.md).
+
 ## Microsoft security service integrations
 
 Microsoft Sentinel is empowered by the components that send data to your workspace, and is made stronger through integrations with other Microsoft services. Any logs ingested into products, such as Microsoft Defender for Cloud Apps, Microsoft Defender for Endpoint, and Microsoft Defender for Identity, allow these services to create detections, and in turn provide those detections to Microsoft Sentinel. Logs can also be ingested directly into Microsoft Sentinel to provide a fuller picture for events and incidents.
 
-For example, the following image shows how Microsoft Sentinel ingests data from other Microsoft services and multicloud and partner platforms to provide coverage for your environment:
+For example, the following image shows how Microsoft Sentinel ingests data from other Microsoft services, multicloud, and partner platforms to provide coverage for your environment:
 
-:::image type="content" source="media/best-practices/azure-sentinel-and-other-services.png" alt-text="Microsoft Sentinel integrating with other Microsoft and partner services":::
+:::image type="content" source="media/best-practices/azure-sentinel-and-other-services.png"  lightbox="media/best-practices/azure-sentinel-and-other-services.png" alt-text="A diagram showing the Microsoft Sentinel integrating with other Microsoft and partner services.":::
 
 More than ingesting alerts and logs from other sources, Microsoft Sentinel also:
 
@@ -49,7 +53,7 @@ If you're using Microsoft Sentinel in the Azure portal, consider onboarding Micr
 
 The following image shows recommended steps in an incident management and response process.
 
-:::image type="content" source="media/best-practices/incident-handling.png" alt-text="Diagram of incident management process: Triage. Preparation. Remediation. Eradication. Post incident activities.":::
+:::image type="content" source="media/best-practices/incident-handling.png" alt-text="Diagram showing incident management process: Triage. Preparation. Remediation. Eradication. Post incident activities.":::
 
 The following table provides high-level descriptions for how to use Microsoft Sentinel features for incident management and response. For more information, see [Investigate incidents with Microsoft Sentinel](investigate-cases.md).
 
@@ -67,6 +71,7 @@ The following table provides high-level descriptions for how to use Microsoft Se
 
 - [Microsoft Sentinel operational guide](ops-guide.md)
 - [On-board Microsoft Sentinel](quickstart-onboard.md)
+- [On-board Microsoft Sentinel data lake](datalake/sentinel-lake-onboarding.md)
 - [Deployment guide for Microsoft Sentinel](deploy-overview.md)
 - [Protecting MSSP intellectual property in Microsoft Sentinel](mssp-protect-intellectual-property.md)