You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: learn-pr/wwl/analyze-monitor-tune-ai-powered-business-solutions/includes/2-recommend-process-tools-monitoring-agents.md
+26-35Lines changed: 26 additions & 35 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,9 +4,9 @@ This unit equips solution architects with the expertise to define, recommend, an
4
4
5
5
You will explore monitoring processes, recommended tools, observability patterns, dashboards, alerting approaches, and analytical insights that support continuous improvement of agent behavior.
6
6
7
-
## 1. Understanding Monitoring Requirements for AI Agents
7
+
## Understanding Monitoring Requirements for AI Agents
8
8
9
-
### Monitoring AI agents requires a multilayered approach. Solution architects must consider:
9
+
### Monitoring AI agents requires a multilayered approach. Solution architects must consider
## 2. Recommended Processes for Monitoring AI Agents
21
+
## Recommended Processes for Monitoring AI Agents
26
22
27
23
Solution architects should recommend the processes for monitoring AI Agents across an organization. When an existing framework is in place, the architect should look for missing components or improvements.
28
24
29
-
### 2.1 Establish a Monitoring Operating Model
25
+
### Establish a Monitoring Operating Model
30
26
31
27
* A strong operational model ensures consistency, ownership, and accountability.
32
28
33
-
#### Key components:
29
+
#### Key components
34
30
35
31
* Defined roles (Ops team, product owners, data engineers, architects)
36
32
@@ -44,15 +40,15 @@ Solution architects should recommend the processes for monitoring AI Agents acro
44
40
45
41
* Documentation of expected agent behaviors and constraints
46
42
47
-
### 2.2 Configure Guardrails and Threshold Alerts
43
+
### Configure Guardrails and Threshold Alerts
48
44
49
45
* Set thresholds for latency, exception volume, and unusual activity.
50
46
51
47
* Create automated alerts for guardrail triggers or tool invocation failures.
52
48
53
49
* Monitor for unexpected spikes in prompts indicating potential misuse.
54
50
55
-
### 2.3 Conduct Regular Quality Evaluations
51
+
### Conduct Regular Quality Evaluations
56
52
57
53
* Humanintheloop spot checks
58
54
@@ -62,7 +58,7 @@ Solution architects should recommend the processes for monitoring AI Agents acro
62
58
63
59
* Validate alignment with business rules or compliance requirements
64
60
65
-
### 2.4 Continuously Improve Based on Insights
61
+
### Continuously Improve Based on Insights
66
62
67
63
* Analyze logs and telemetry to find failure patterns.
68
64
@@ -72,13 +68,13 @@ Solution architects should recommend the processes for monitoring AI Agents acro
72
68
73
69
* Propose workflow adjustments or retraining of custom models (if applicable).
74
70
75
-
## 3. Recommended Tools for Monitoring AI Agents
71
+
## Recommended Tools for Monitoring AI Agents
76
72
77
73
Solution architects should recommend the toolset that covers **observability**, **analytics**, and **administrative insights**.
78
74
79
-
### 3.1 Azure Monitor (Core Telemetry + Alerts)
75
+
### Azure Monitor (Core Telemetry + Alerts)
80
76
81
-
#### Azure Monitor provides:
77
+
#### Azure Monitor provides
82
78
83
79
* Application and agent telemetry
84
80
@@ -88,17 +84,17 @@ Solution architects should recommend the toolset that covers **observability**,
88
84
89
85
* Integration with Log Analytics Workspaces
90
86
91
-
#### Use cases:
87
+
#### Use cases
92
88
93
89
* Monitor agent workflows built with Power Platform or custom services.
### 4. Designing an Enterprise Monitoring Architecture for Agents
172
-
173
-
:::image type="content" source="../media/designing-enterprise-monitoring-architecture-for-agents.png" alt-text="Designing enterprise monitoring architecture for AI agents":::
174
-
175
-
176
167
#### Best Practices
177
168
178
169
* Always centralize logs.
@@ -187,10 +178,10 @@ Solution architects should recommend the toolset that covers **observability**,
Copy file name to clipboardExpand all lines: learn-pr/wwl/analyze-monitor-tune-ai-powered-business-solutions/includes/3-analyze-backlog-user-feedback-ai-agent-usage.md
+21-21Lines changed: 21 additions & 21 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,9 +4,9 @@ This unit teaches solution architects how to evaluate, interpret, and act on bac
4
4
5
5
Solution architects play a critical role in translating insights from telemetry, backlog queues, and conversation transcripts into architectural decisions, iterative improvements, and governance recommendations. This unit provides a repeatable framework for analyzing data and driving continuous improvement cycles for AIpowered agents.
6
6
7
-
## 1. Understanding AI Backlogs and User Feedback Loops
7
+
## Understanding AI Backlogs and User Feedback Loops
8
8
9
-
### A backlog in AI and agent operations typically contains:
9
+
### A backlog in AI and agent operations typically contains
10
10
11
11
* Enhancement requests
12
12
@@ -20,7 +20,7 @@ Solution architects play a critical role in translating insights from telemetry,
20
20
21
21
* Governance or policy misalignment concerns
22
22
23
-
### User feedback may originate from:
23
+
### User feedback may originate from
24
24
25
25
* Conversation transcripts
26
26
@@ -34,7 +34,7 @@ Solution architects play a critical role in translating insights from telemetry,
0 commit comments