You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: learn-pr/wwl-sci/design-solutions-security-operations/2-design-security-operations-capabilities-hybrid-multicloud-environments.yml
Copy file name to clipboardExpand all lines: learn-pr/wwl-sci/design-solutions-security-operations/6-design-solution-security-orchestration-automation-response.yml
Copy file name to clipboardExpand all lines: learn-pr/wwl-sci/design-solutions-security-operations/includes/2-design-security-operations-capabilities-hybrid-multicloud-environments.md
+3Lines changed: 3 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -95,6 +95,9 @@ The Microsoft Defender portal provides a unified view of security monitoring dat
95
95
96
96
This centralized visibility eliminates the need for security teams to switch between multiple consoles when monitoring hybrid environments, reducing mean time to detect (MTTD) threats.
97
97
98
+
> [!NOTE]
99
+
> This unit focuses on infrastructure monitoring across hybrid and multicloud environments. Microsoft 365 productivity workloads (email, files, identity, collaboration) have their own monitoring capabilities through Microsoft Defender XDR services, and is covered in [Design solutions for securing Microsoft 365](/training/modules/design-solutions-secure-microsoft-365/). The Defender portal serves as the convergence point where infrastructure and productivity monitoring combine for unified incident correlation and response.
Copy file name to clipboardExpand all lines: learn-pr/wwl-sci/design-solutions-security-operations/includes/3-design-centralized-logging-auditing.md
+29-5Lines changed: 29 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,6 +2,24 @@ Centralized logging and auditing are fundamental to security operations, providi
2
2
3
3
**Logging** focuses on recording events for historical analysis, compliance evidence, and forensic investigations. It answers the questions "What happened?" and "Can we prove compliance?" **Auditing** specifically tracks user and administrator activities to establish accountability and meet regulatory requirements. While monitoring (covered in the previous unit) provides real-time visibility, logging ensures you have a durable record of events—sometimes retained for years—to support investigations and demonstrate compliance.
4
4
5
+
## Understanding security data domains
6
+
7
+
As a security architect, it's important to understand that security log data comes from two distinct domains that converge in Microsoft Sentinel for unified analysis:
8
+
9
+
| Domain | What it covers | Primary tools | Log characteristics |
Both domains are essential for comprehensive security operations:
15
+
16
+
-**Infrastructure logs** help you detect attacks on workloads, investigate lateral movement, and understand resource-level threats
17
+
-**Productivity logs** help you detect compromised accounts, investigate data exfiltration, and establish user accountability
18
+
19
+
Microsoft Sentinel serves as the convergence point, ingesting logs from both domains into a unified Log Analytics workspace. This enables security analysts to correlate an identity-based attack (detected in Entra ID or M365 logs) with subsequent infrastructure activity (detected in Azure or multicloud logs)—providing the complete attack story.
20
+
21
+
The rest of this unit covers logging solutions for both domains: Log Analytics workspaces and Microsoft Sentinel for infrastructure data, and Microsoft Purview Audit for productivity data.
22
+
5
23
## Design guidance for centralized logging
6
24
7
25
The Microsoft Cloud Security Benchmark (MCSB) provides guidance on designing logging capabilities:
@@ -48,26 +66,29 @@ The Microsoft Sentinel data lake is a fully managed, cloud-native data lake purp
48
66
-**Open format Parquet data files** for interoperability and extensibility
49
67
-**Single copy of data** mirrored from analytics tier for efficient storage
50
68
-**Separation of storage and compute** for flexibility and cost optimization
51
-
-**Multiple analytics engines** including KQL jobs, Spark notebooks, and graph analytics
69
+
-**Multiple analytics engines** including KQL and Jupyter notebooks
52
70
-**Up to 12 years of retention** for compliance requirements
71
+
-**Activity auditing** tracks data lake activities including data access, job management, and query events
53
72
54
73
Data in the analytics tier is automatically mirrored to the data lake tier at no extra cost when retention periods match. Organizations can choose to ingest data exclusively into the data lake tier for high-volume, lower-security-value logs.
55
74
75
+
The data lake's built-in activity audit provides accountability for security operations activities—you can monitor who accessed data, ran notebooks, or created and modified jobs. This auditing is enabled by default and supports compliance requirements for tracking access to security data.
76
+
56
77
### Data lake analytics capabilities
57
78
58
79
The data lake provides multiple ways to analyze historical log data:
59
80
60
81
| Capability | Purpose | Best for |
61
82
|------------|---------|----------|
62
83
|**KQL jobs**| Run one-time or scheduled asynchronous queries against data lake data with full KQL support including joins and unions | Incident investigations using historical logs, threat intelligence matching, anomaly detection across months of data |
63
-
|**Summary rules**| Run frequent summarization jobs (every 20 minutes to 24 hours) to aggregate high-volume data | Aggregating network and firewall logs, creating baseline tables for detection |
64
-
|**Search jobs**| Run long-running asynchronous queries to hydrate large volumes of data from a single table | Forensic analysis requiring restoration of archived data|
65
-
|**Spark notebooks**| Use Python-based advanced analytics with Jupyter notebooks| Machine learning models, complex statistical analysis, custom visualizations |
84
+
|**Summary rules**| Run scheduled aggregation jobs (bin sizes from 20 minutes to 24 hours) to precompute data into custom log tables | Aggregating network and firewall logs, creating baseline tables for detection, cost optimization for verbose logs|
85
+
|**Search jobs**| Run long-running searches through up to a year of data in a table, sending results to a new Analytics table | Forensic analysis when query timeout is insufficient, searching large datasets for specific events|
86
+
|**Jupyter notebooks**| Use Python-based advanced analytics with machine learning libraries| Machine learning models, complex statistical analysis, custom visualizations |
66
87
67
88
KQL jobs can promote data from the data lake tier to the analytics tier, enabling investigation of historical events alongside current incidents. This is particularly valuable for zero-day threat detection and retrospective threat hunting.
68
89
69
90
> [!TIP]
70
-
> Use KQL jobs for complex queries involving joins across multiple tables. Use summary rules for recurring aggregations. Use search jobs when you need to hydrate large volumes from a single archived table.
91
+
> Use KQL jobs for incident investigations, threat intelligence matching, and promoting data from data lake to analytics tier. Use summary rules for recurring aggregations that support detection rules. Use search jobs when you need to scan large datasets for specific events.
71
92
72
93
### Data connectors and data flow
73
94
@@ -77,6 +98,9 @@ When you onboard to Microsoft Sentinel data lake, your existing data connectors
77
98
-**Analytics tier with data lake mirroring** - Default configuration for most security data
78
99
-**Data lake tier only** - For high-volume logs with limited real-time security value
79
100
101
+
102
+
:::image type="content" source="../media/data-lake-tiers-data-flow.png" lightbox="../media/data-lake-tiers-data-flow.png" alt-text="A block diagram that depicts the mirroring of data from analytics tier to the data lake tier.":::
103
+
80
104
## Microsoft Purview Audit for compliance
81
105
82
106
Microsoft Purview Audit provides an integrated auditing solution to help organizations respond to security events, forensic investigations, and compliance obligations.
Copy file name to clipboardExpand all lines: learn-pr/wwl-sci/design-solutions-security-operations/includes/5-design-solutions-detection-response.md
+76-8Lines changed: 76 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,6 +4,8 @@
4
4
5
5
**SIEM** solutions collect, aggregate, normalize, and analyze large volumes of data from organization-wide applications, devices, servers, and users in near real time. By consolidating telemetry into a unified platform, SIEM solutions provide a comprehensive view of an organization's security posture.
6
6
7
+
Together, XDR and SIEM cover both security data domains discussed in the previous unit: XDR provides deep detection across the productivity domain (email, identity, collaboration) while SIEM correlates data from both productivity and infrastructure domains for comprehensive threat visibility.
8
+
7
9
## Design guidance for detection and response
8
10
9
11
When designing detection and response solutions, consider these best practices:
@@ -13,6 +15,7 @@ When designing detection and response solutions, consider these best practices:
13
15
-**Enable automatic attack disruption** to contain threats at machine speed
14
16
-**Design for unified incident management** to correlate alerts across all security tools
15
17
-**Plan for threat intelligence integration** to enrich detections with context
18
+
-**Leverage AI-assisted detection** to identify sophisticated threats that rule-based detection might miss
16
19
17
20
## Microsoft Defender XDR
18
21
@@ -26,25 +29,67 @@ Microsoft Defender XDR provides XDR capabilities across the Microsoft 365 enviro
26
29
| Protect operational technology (OT) and IT resources | Microsoft Defender for IoT |
27
30
| Identify assets and assess device security posture | Microsoft Defender Vulnerability Management |
28
31
| Protect and control access to SaaS cloud apps | Microsoft Defender for Cloud Apps |
32
+
| Detect threats across Azure, AWS, and GCP workloads | Microsoft Defender for Cloud |
29
33
30
34
Microsoft Defender XDR automatically correlates alerts into incidents, providing end-to-end attack chain visibility and AI-powered automation for faster detection and response.
31
35
32
36
## Microsoft Sentinel as SIEM and security platform
33
37
34
-
Microsoft Sentinel is a cloud-native SIEM that delivers AI-driven security across multicloud and multiplatform environments with capabilities for threat detection, investigation, hunting, response, and automated attack disruption.
38
+
Microsoft Sentinel is a cloud-native SIEM and unified security platform for agentic defense. It has evolved beyond traditional SIEM to provide an AI-ready, data-first foundation that transforms telemetry into a security graph, standardizes access for AI agents, and coordinates autonomous actions while keeping humans in command of strategy and high-impact investigations.
39
+
40
+
As a SIEM, Microsoft Sentinel delivers AI-driven security across multicloud and multiplatform environments with capabilities for threat detection, investigation, hunting, response, and automated attack disruption. As a platform, it provides a foundation built on a modern data lake, graph capabilities for contextual analysis, and a hosted Model Context Protocol (MCP) server for agent-ready tooling.
41
+
42
+
:::image type="content" source="../media/microsoft-sentinel-overview.png" alt-text="A diagram that depicts the Microsoft Sentinel AI-first, end-to-end SIEM and security platform." lightbox="../media/microsoft-sentinel-overview.png":::
35
43
36
-
### Key capabilities
44
+
### SIEM capabilities
37
45
38
46
-**Comprehensive data collection** across users, devices, identities, applications, infrastructure, and operational technology via 350+ connectors
39
47
-**Advanced threat detection** with built-in analytics rules, machine learning models, and threat intelligence enrichment
40
48
-**SOC efficiency tools** including unified incidents, entity pages, investigation graph, hunting notebooks, and Security Copilot integration
41
-
-**Automatic attack disruption** signals that integrate with Microsoft Defender XDR for machine-speed response
42
49
43
-
### Platform components
50
+
### Core platform components
51
+
52
+
| Component | Description | Design relevance |
53
+
|-----------|-------------|------------------|
54
+
|**Microsoft Sentinel data lake**| Fully managed, cloud-native data lake that unifies, retains, and analyzes security data at scale with support for up to 12 years retention | Enables multi-modal analytics including KQL queries, graph analysis, and AI-powered notebooks on a single copy of open-format data |
55
+
|**Microsoft Sentinel graph**| Unified graph analytics that models relationships across assets, identities, activities, and threat intelligence | Enables reasoning over interconnected data to answer complex questions like attack paths from compromised entities to critical assets |
56
+
|**Microsoft Sentinel MCP server**| Hosted interface enabling natural language interaction with security data and building intelligent security agents | Accelerates agent creation and enables security teams to query data without schema knowledge or coding |
57
+
|**Content Hub**| Solutions, analytics rules, playbooks, and workbooks from Microsoft and partners | Provides pre-built detection and response content to accelerate deployment |
58
+
59
+
### Data tiering for detection and retention
60
+
61
+
Microsoft Sentinel's tiered data architecture supports both real-time detection and long-term investigation:
62
+
63
+
-**Analytics tier** - High-performance storage for real-time detection, alerting, and interactive investigation
64
+
-**Data lake tier** - Cost-efficient storage for historical analysis, behavioral baselines, and compliance retention
65
+
-**Automatic mirroring** - Data ingested into the analytics tier is automatically mirrored to the data lake tier, ensuring a unified repository
66
+
67
+
This architecture allows security architects to design solutions that balance detection latency, investigation depth, and cost optimization.
68
+
69
+
## AI-powered detection and investigation
70
+
71
+
AI and machine learning are fundamental to modern detection and response, enabling capabilities that traditional rule-based detection cannot achieve:
72
+
73
+
### Detection capabilities
44
74
45
-
-**Microsoft Sentinel data lake**: Fully managed data lake for security data retention up to 12 years
46
-
-**Content Hub**: Solutions, analytics rules, playbooks, and workbooks from Microsoft and partners
47
-
-**Security Copilot integration**: AI-powered investigation assistance and natural language queries
75
+
| AI capability | How it helps detection |
76
+
|---------------|----------------------|
77
+
|**Behavioral analytics (UEBA)**| Establishes baseline behavior for users and entities, detecting anomalies that indicate compromised accounts or insider threats |
78
+
|**Machine learning anomaly detection**| Identifies unusual patterns in network traffic, authentication, and resource access without predefined rules |
79
+
|**Threat intelligence correlation**| Automatically matches indicators of compromise (IOCs) against incoming telemetry at scale |
80
+
|**Attack chain detection**| Correlates low-fidelity signals across multiple data sources to identify sophisticated multi-stage attacks |
81
+
82
+
### Security Copilot for SOC efficiency
83
+
84
+
Microsoft Security Copilot transforms how analysts investigate and respond to threats:
85
+
86
+
-**Incident summarization** - Automatically generates plain-language summaries of complex incidents, reducing triage time from hours to minutes
87
+
-**Guided investigation** - Suggests next investigation steps based on incident context and attack patterns
88
+
-**Natural language queries** - Enables analysts to query security data using conversational language instead of KQL
89
+
-**Script analysis** - Analyzes malicious scripts and explains their behavior in plain language
90
+
-**Report generation** - Creates investigation reports and executive summaries automatically
91
+
92
+
For security architects, designing for Security Copilot means ensuring comprehensive data collection (Copilot can only analyze data that's available in your workspace) and training SOC teams to effectively prompt the AI for investigation assistance.
48
93
49
94
## Unified security operations in the Defender portal
50
95
@@ -68,7 +113,28 @@ When Microsoft Sentinel is connected with Microsoft Defender XDR in the unified
68
113
-**Enriched investigation paths** combine XDR deep insights with SIEM breadth
69
114
70
115
> [!IMPORTANT]
71
-
> Starting July 2025, new customers are automatically onboarded to the Defender portal. After March 31, 2027, Microsoft Sentinel will be available only in the Microsoft Defender portal.
116
+
> New customers are automatically onboarded to the unified Defender portal. After March 31, 2027, Microsoft Sentinel will be available only in the Microsoft Defender portal. Plan your security operations architecture with this unified experience in mind.
117
+
118
+
## Detection engineering considerations
119
+
120
+
As a security architect, design your detection strategy to balance coverage, quality, and maintainability:
121
+
122
+
### Analytics rule design
123
+
124
+
| Rule type | Use case | Design consideration |
125
+
|-----------|----------|---------------------|
126
+
|**Scheduled rules**| Custom detection logic using KQL queries | Run frequency vs. cost; lookback period vs. detection latency |
127
+
|**Near-real-time (NRT) rules**| Time-sensitive detections requiring sub-minute response | Limited to specific data sources; higher resource consumption |
128
+
|**Microsoft security rules**| Automatic incident creation from Defender alerts | Configure filtering to avoid duplicate incidents |
129
+
|**ML behavior analytics**| Anomaly detection without predefined thresholds | Requires sufficient baseline data; tune sensitivity over time |
130
+
|**Threat intelligence rules**| IOC matching against threat feeds | Ensure TI feeds are current and relevant to your threat landscape |
131
+
132
+
### Detection coverage strategy
133
+
134
+
-**Map detections to MITRE ATT&CK** - Use the coverage matrix in Sentinel to identify gaps across tactics and techniques
135
+
-**Layer detection types** - Combine rule-based, ML-based, and threat intelligence detections for defense in depth
136
+
-**Prioritize high-impact techniques** - Focus detection engineering on techniques most commonly used against your industry
137
+
-**Plan for detection maintenance** - Allocate ongoing effort to tune rules, reduce false positives, and adapt to evolving threats
72
138
73
139
## Design considerations for detection and response
74
140
@@ -79,4 +145,6 @@ When Microsoft Sentinel is connected with Microsoft Defender XDR in the unified
79
145
|**Automation**| Enable automatic attack disruption for high-confidence threats to contain attacks at machine speed |
80
146
|**Integration**| Design for unified incident management across XDR and SIEM |
81
147
|**Threat intelligence**| Integrate threat intelligence feeds to enrich detections and provide investigation context |
148
+
|**AI enablement**| Ensure comprehensive data collection to maximize Security Copilot effectiveness |
149
+
|**Detection engineering**| Plan for ongoing rule development, tuning, and MITRE ATT&CK coverage assessment |
82
150
|**Compliance**| Consider data residency and retention requirements when designing the solution |
0 commit comments