Skip to content

Commit ac823c4

Browse files
committed
Fix blocking review feedback for module 8
1 parent 097476c commit ac823c4

9 files changed

Lines changed: 97 additions & 161 deletions

learn-pr/wwl/analyze-monitor-tune-ai-powered-business-solutions/includes/2-recommend-process-tools-monitoring-agents.md

Lines changed: 26 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -4,9 +4,9 @@ This unit equips solution architects with the expertise to define, recommend, an
44

55
You will explore monitoring processes, recommended tools, observability patterns, dashboards, alerting approaches, and analytical insights that support continuous improvement of agent behavior.
66

7-
## 1. Understanding Monitoring Requirements for AI Agents
7+
## Understanding Monitoring Requirements for AI Agents
88

9-
### Monitoring AI agents requires a multilayered approach. Solution architects must consider:
9+
### Monitoring AI agents requires a multilayered approach. Solution architects must consider
1010

1111
**Operational Health**<br>Uptime, availability, error frequency, throttling conditions, processing delays.
1212

@@ -18,19 +18,15 @@ You will explore monitoring processes, recommended tools, observability patterns
1818

1919
**Risk, Compliance, and Security**<br>Guardrail violations, sensitivedata handling, suspicious activity spikes, adherence to organizational policies.
2020

21-
### AI Agent Monitoring Layers
22-
23-
:::image type="content" source="../media/ai-agent-monitoring-layers.png" alt-text="AI agent monitoring layers":::
24-
25-
## 2. Recommended Processes for Monitoring AI Agents
21+
## Recommended Processes for Monitoring AI Agents
2622

2723
Solution architects should recommend the processes for monitoring AI Agents across an organization. When an existing framework is in place, the architect should look for missing components or improvements.
2824

29-
### 2.1 Establish a Monitoring Operating Model
25+
### Establish a Monitoring Operating Model
3026

3127
* A strong operational model ensures consistency, ownership, and accountability.
3228

33-
#### Key components:
29+
#### Key components
3430

3531
* Defined roles (Ops team, product owners, data engineers, architects)
3632

@@ -44,15 +40,15 @@ Solution architects should recommend the processes for monitoring AI Agents acro
4440

4541
* Documentation of expected agent behaviors and constraints
4642

47-
### 2.2 Configure Guardrails and Threshold Alerts
43+
### Configure Guardrails and Threshold Alerts
4844

4945
* Set thresholds for latency, exception volume, and unusual activity.
5046

5147
* Create automated alerts for guardrail triggers or tool invocation failures.
5248

5349
* Monitor for unexpected spikes in prompts indicating potential misuse.
5450

55-
### 2.3 Conduct Regular Quality Evaluations
51+
### Conduct Regular Quality Evaluations
5652

5753
* Humanintheloop spot checks
5854

@@ -62,7 +58,7 @@ Solution architects should recommend the processes for monitoring AI Agents acro
6258

6359
* Validate alignment with business rules or compliance requirements
6460

65-
### 2.4 Continuously Improve Based on Insights
61+
### Continuously Improve Based on Insights
6662

6763
* Analyze logs and telemetry to find failure patterns.
6864

@@ -72,13 +68,13 @@ Solution architects should recommend the processes for monitoring AI Agents acro
7268

7369
* Propose workflow adjustments or retraining of custom models (if applicable).
7470

75-
## 3. Recommended Tools for Monitoring AI Agents
71+
## Recommended Tools for Monitoring AI Agents
7672

7773
Solution architects should recommend the toolset that covers **observability**, **analytics**, and **administrative insights**.
7874

79-
### 3.1 Azure Monitor (Core Telemetry + Alerts)
75+
### Azure Monitor (Core Telemetry + Alerts)
8076

81-
#### Azure Monitor provides:
77+
#### Azure Monitor provides
8278

8379
* Application and agent telemetry
8480

@@ -88,17 +84,17 @@ Solution architects should recommend the toolset that covers **observability**,
8884

8985
* Integration with Log Analytics Workspaces
9086

91-
#### Use cases:
87+
#### Use cases
9288

9389
* Monitor agent workflows built with Power Platform or custom services.
9490

9591
* Track errors, latency, throughput, connector failures.
9692

9793
* Build KQL-based queries for deep diagnostics.
9894

99-
### 3.2 Microsoft 365 Admin Analytics (Usage & Adoption Trends)
95+
### Microsoft 365 Admin Analytics (Usage & Adoption Trends)
10096

101-
#### Useful for:
97+
#### Useful for
10298

10399
* Understanding agent usage volume
104100

@@ -108,9 +104,9 @@ Solution architects should recommend the toolset that covers **observability**,
108104

109105
* Measuring improvements week-over-week
110106

111-
### 3.3 Copilot & Agent Analytics Dashboards
107+
### Copilot & Agent Analytics Dashboards
112108

113-
#### When available in an organization's tenant, Copilot analytics can provide:
109+
#### When available in an organization's tenant, Copilot analytics can provide
114110

115111
* Agent invocation frequency
116112

@@ -122,9 +118,9 @@ Solution architects should recommend the toolset that covers **observability**,
122118

123119
* Error or guardrail-trigger events
124120

125-
### 3.4 Power Platform Admin Center (Environment-Level Monitoring)
121+
### Power Platform Admin Center (Environment-Level Monitoring)
126122

127-
#### Provides:
123+
#### Provides
128124

129125
* Environment health
130126

@@ -134,9 +130,9 @@ Solution architects should recommend the toolset that covers **observability**,
134130

135131
* DLP rule impact visibility
136132

137-
### 3.5 Foundry or Organizational Observability Platforms
133+
### Foundry or Organizational Observability Platforms
138134

139-
#### Enterprises may adopt centralized observability platforms (example: Foundry-like solutions, if present in the environment) to unify:
135+
#### Enterprises may adopt centralized observability platforms (example: Foundry-like solutions, if present in the environment) to unify
140136

141137
* Multisystem logs
142138

@@ -148,9 +144,9 @@ Solution architects should recommend the toolset that covers **observability**,
148144

149145
* These platforms reduce fragmentation and provide a single-pane-of-glass view for complex agent ecosystems.
150146

151-
### 3.6 Custom Dashboards for Enterprise AI Agents
147+
### Custom Dashboards for Enterprise AI Agents
152148

153-
#### Solution architects often design:
149+
#### Solution architects often design
154150

155151
* KPI dashboards in Power BI
156152

@@ -160,19 +156,14 @@ Solution architects should recommend the toolset that covers **observability**,
160156

161157
* Compliance trend reports
162158

163-
#### Example: Agent Health Summary
159+
#### Example Agent Health Summary
164160

165161
| Agent Name | Success Rate | Avg. Response Time | Errors Today | Usage Trend |
166162
| --- | --- | --- | --- | --- |
167163
| Sales Helper | 98% | 1.8 sec | 3 | ↑ Increasing |
168164
| Ops Agent | 92% | 2.5 sec | 17 | → Steady |
169165
| Finance Advisor | 86% | 3.2 sec | 28 | ↓ Decreasing |
170166

171-
### 4. Designing an Enterprise Monitoring Architecture for Agents
172-
173-
:::image type="content" source="../media/designing-enterprise-monitoring-architecture-for-agents.png" alt-text="Designing enterprise monitoring architecture for AI agents":::
174-
175-
176167
#### Best Practices
177168

178169
* Always centralize logs.
@@ -187,10 +178,10 @@ Solution architects should recommend the toolset that covers **observability**,
187178

188179
## References
189180

190-
[https://learn.microsoft.com/en-us/training/modules/describe-monitoring-tools-azure/4-describe-azure-monitor](/training/modules/describe-monitoring-tools-azure/4-describe-azure-monitor)
181+
[https://learn.microsoft.com/training/modules/describe-monitoring-tools-azure/4-describe-azure-monitor](/training/modules/describe-monitoring-tools-azure/4-describe-azure-monitor)
191182

192-
[https://learn.microsoft.com/en-us/training/modules/perform-admin-tasks-microsoft-365-copilot/](/training/modules/perform-admin-tasks-microsoft-365-copilot/)
183+
[https://learn.microsoft.com/training/modules/perform-admin-tasks-microsoft-365-copilot/](/training/modules/perform-admin-tasks-microsoft-365-copilot/)
193184

194-
[https://learn.microsoft.com/en-us/azure/ai-foundry/observability/how-to/how-to-monitor-agents-dashboard?view=foundry](/azure/ai-foundry/observability/how-to/how-to-monitor-agents-dashboard)
185+
[https://learn.microsoft.com/azure/ai-foundry/observability/how-to/how-to-monitor-agents-dashboard?view=foundry](/azure/ai-foundry/observability/how-to/how-to-monitor-agents-dashboard)
195186

196187
[https://learn.microsoft.com/power-platform/admin/analytics-copilot](/power-platform/admin/analytics-copilot)

learn-pr/wwl/analyze-monitor-tune-ai-powered-business-solutions/includes/3-analyze-backlog-user-feedback-ai-agent-usage.md

Lines changed: 21 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -4,9 +4,9 @@ This unit teaches solution architects how to evaluate, interpret, and act on bac
44

55
Solution architects play a critical role in translating insights from telemetry, backlog queues, and conversation transcripts into architectural decisions, iterative improvements, and governance recommendations. This unit provides a repeatable framework for analyzing data and driving continuous improvement cycles for AIpowered agents.
66

7-
## 1. Understanding AI Backlogs and User Feedback Loops
7+
## Understanding AI Backlogs and User Feedback Loops
88

9-
### A backlog in AI and agent operations typically contains:
9+
### A backlog in AI and agent operations typically contains
1010

1111
* Enhancement requests
1212

@@ -20,7 +20,7 @@ Solution architects play a critical role in translating insights from telemetry,
2020

2121
* Governance or policy misalignment concerns
2222

23-
### User feedback may originate from:
23+
### User feedback may originate from
2424

2525
* Conversation transcripts
2626

@@ -34,7 +34,7 @@ Solution architects play a critical role in translating insights from telemetry,
3434

3535
* Observability dashboards
3636

37-
### Effective backlog analysis helps solution architects:
37+
### Effective backlog analysis helps solution architects
3838

3939
* Prioritize improvements based on impact
4040

@@ -46,13 +46,13 @@ Solution architects play a critical role in translating insights from telemetry,
4646

4747
* Identify opportunities for automation and process redesign
4848

49-
## 2. Framework for Backlog Analysis
49+
## Framework for Backlog Analysis
5050

5151
Use a structured, repeatable model to elevate analysis to a solutionarchitecture level.
5252

53-
### 2.1 Categorize the Backlog by Domain
53+
### Categorize the Backlog by Domain
5454

55-
#### Organize items into categories such as:
55+
#### Organize items into categories such as
5656

5757
* **Accuracy and Reasoning**: Incorrect, missing, or lowconfidence responses
5858

@@ -77,15 +77,15 @@ Use a structured, repeatable model to elevate analysis to a solutionarchitecture
7777
| Integration | API failures; broken actions |
7878
| Governance | Data access blocked; DLP alerts |
7979

80-
### 2.2 Prioritize by Impact and Effort
80+
### Prioritize by Impact and Effort
8181

8282
Use a matrix to classify items based on business impact and required effort.
8383

8484
:::image type="content" source="../media/impact-effort-prioritization-matrix.png" alt-text="Impact effort prioritization matrix":::
8585

86-
### 2.3 Leverage User Feedback Signals
86+
### Leverage User Feedback Signals
8787

88-
#### Solution architects should analyze:
88+
#### Solution architects should analyze
8989

9090
* Frequency of similar feedback (volume signals)
9191

@@ -102,9 +102,9 @@ Use a matrix to classify items based on business impact and required effort.
102102
:::image type="content" source="../media/user-feedback-funnel.png" alt-text="User feedback funnel":::
103103

104104

105-
## 3. Using Conversation Transcripts to Identify Patterns
105+
## Using Conversation Transcripts to Identify Patterns
106106

107-
### Conversation transcripts reveal:
107+
### Conversation transcripts reveal
108108

109109
* Where the agent misunderstood intent
110110

@@ -116,7 +116,7 @@ Use a matrix to classify items based on business impact and required effort.
116116

117117
* Workflows requiring human intervention
118118

119-
### Solution architects should:
119+
### Solution architects should
120120

121121
* Extract common failure paths
122122

@@ -128,11 +128,11 @@ Use a matrix to classify items based on business impact and required effort.
128128

129129
* Agent Improvement Plan
130130

131-
## 4. Monitoring Agent Usage and Behavior
131+
## Monitoring Agent Usage and Behavior
132132

133133
Monitoring ensures agents operate as intended and scale properly.
134134

135-
### Key items for architects to monitor:
135+
### Key items for architects to monitor
136136

137137
* Usage trends and adoption
138138

@@ -158,11 +158,11 @@ Monitoring ensures agents operate as intended and scale properly.
158158
| Supply Chain Helper | 91% | 2.8 sec | 14 |
159159
| Analytics Coach | 88% | 3.2 sec | 21 |
160160

161-
## 5. Closing the Loop: Turning Insights into Action
161+
## Closing the Loop Turning Insights into Action
162162

163163
Solution architects drive improvement cycles by:
164164

165-
### 5.1 Designing a FeedbacktoImprovement Pipeline
165+
### Designing a FeedbacktoImprovement Pipeline
166166

167167
* Consolidate backlog and user feedback
168168

@@ -176,7 +176,7 @@ Solution architects drive improvement cycles by:
176176

177177
* Monitor drift and regression
178178

179-
### 5.2 Embedding Continuous Improvement
179+
### Embedding Continuous Improvement
180180

181181
* Refresh knowledge sources regularly
182182

@@ -188,9 +188,9 @@ Solution architects drive improvement cycles by:
188188

189189
* Implement new guardrails based on risk patterns
190190

191-
### 5.3 Reporting and Stakeholder Communication
191+
### Reporting and Stakeholder Communication
192192

193-
#### Communicate:
193+
#### Communicate
194194

195195
* Top backlog themes
196196

@@ -204,7 +204,7 @@ Solution architects drive improvement cycles by:
204204

205205
## References
206206

207-
[https://learn.microsoft.com/en-us/microsoft-copilot-studio/guidance/kit-agent-inventory](/microsoft-copilot-studio/guidance/kit-agent-inventory)
207+
[https://learn.microsoft.com/microsoft-copilot-studio/guidance/kit-agent-inventory](/microsoft-copilot-studio/guidance/kit-agent-inventory)
208208

209209
[https://learn.microsoft.com/power-platform/admin/analytics-copilot](/power-platform/admin/analytics-copilot)
210210

0 commit comments

Comments
 (0)