Skip to content

Commit 9c8775f

Browse files
committed
Adding bullets
1 parent 6fd7f3f commit 9c8775f

1 file changed

Lines changed: 44 additions & 44 deletions

File tree

learn-pr/wwl/analyze-monitor-tune-ai-powered-business-solutions/includes/5-monitor-agent-performance-metrics.md

Lines changed: 44 additions & 44 deletions
Original file line numberDiff line numberDiff line change
@@ -10,47 +10,47 @@ Solution architects are responsible for ensuring that agents perform reliably at
1010

1111
#### Effective monitoring ensures:
1212

13-
Agents behave predictably in production.
13+
* Agents behave predictably in production.
1414

15-
Performance degradation is detected early.
15+
* Performance degradation is detected early.
1616

17-
Quality of reasoning, grounding content, and actions remain consistent.
17+
* Quality of reasoning, grounding content, and actions remain consistent.
1818

19-
Usage insights inform iteration, adoption strategy, and capability refinement.
19+
* Usage insights inform iteration, adoption strategy, and capability refinement.
2020

21-
Governance and compliance requirements are continuously met.
21+
* Governance and compliance requirements are continuously met.
2222

23-
Monitoring focuses on **metrics**, **logs**, **telemetry**, and **user behavior signals** to help architects make informed decisions.
23+
* Monitoring focuses on **metrics**, **logs**, **telemetry**, and **user behavior signals** to help architects make informed decisions.
2424

2525
## 2. Core Metrics for AI Agent Performance
2626

2727
Solution architects should define and track a baseline set of performance indicators across operational, behavioral, and qualitative dimensions.
2828

2929
### 2.1 Operational Metrics
3030

31-
**Latency** - Time taken to process agent requests.
31+
* **Latency** - Time taken to process agent requests.
3232

33-
**Throughput** - Volume of completed runs over a period.
33+
* **Throughput** - Volume of completed runs over a period.
3434

35-
**Error Rate** - Proportion of failed or incomplete tasks.
35+
* **Error Rate** - Proportion of failed or incomplete tasks.
3636

37-
**Resource Utilization** - Compute, memory, and token consumption.
37+
* **Resource Utilization** - Compute, memory, and token consumption.
3838

3939
### 2.2 Quality and Reasoning Metrics
4040

41-
**Response Accuracy** - Alignment with expected or validated outputs.
41+
* **Response Accuracy** - Alignment with expected or validated outputs.
4242

43-
**Knowledge Coverage** - Ability to surface correct domain content.
43+
* **Knowledge Coverage** - Ability to surface correct domain content.
4444

45-
**Action Effectiveness** - Completion of multistep tasks as intended.
45+
* **Action Effectiveness** - Completion of multistep tasks as intended.
4646

4747
### 2.3 UserCentered Metrics
4848

49-
**Satisfaction Indicators** - User feedback trends and sentiment.
49+
* **Satisfaction Indicators** - User feedback trends and sentiment.
5050

51-
**Abandonment Rate** - Dropoff during agent workflows.
51+
* **Abandonment Rate** - Dropoff during agent workflows.
5252

53-
**Task Completion Rate** - Whether users achieve intended outcomes.
53+
* **Task Completion Rate** - Whether users achieve intended outcomes.
5454

5555
## 3. Tools and Processes Used to Monitor AI Agents
5656

@@ -70,67 +70,67 @@ Solution architects can design monitoring patterns using platform observability
7070

7171
#### Operational Telemetry
7272

73-
System logs
73+
* System logs
7474

75-
Execution traces
75+
* Execution traces
7676

77-
Trigger based run logs
77+
* Trigger based run logs
7878

79-
Exception events
79+
* Exception events
8080

81-
Performance counters
81+
* Performance counters
8282

8383
#### Behavioral Telemetry
8484

85-
User interaction logs
85+
* User interaction logs
8686

87-
Conversation transcripts
87+
* Conversation transcripts
8888

89-
Intent recognition patterns
89+
* Intent recognition patterns
9090

91-
Feature usage signals
91+
* Feature usage signals
9292

9393
#### Analytics Dashboards
9494

9595
##### Provide trend views for:
9696

97-
Top user tasks
97+
* Top user tasks
9898

99-
Success/failure distribution
99+
* Success/failure distribution
100100

101-
Conversation or run volumes
101+
* Conversation or run volumes
102102

103-
Peak usage intervals
103+
* Peak usage intervals
104104

105-
Quality indicators
105+
* Quality indicators
106106

107107
## 4. Monitoring Model Performance for Generative AI
108108

109109
Even when agent logic is stable, model-driven behavior can shift over time. Architects should monitor:
110110

111111
### Model Drift
112112

113-
Shifts in response patterns
113+
* Shifts in response patterns
114114

115-
Declining accuracy in recurring tasks
115+
* Declining accuracy in recurring tasks
116116

117-
Increased hallucination or off topic responses
117+
* Increased hallucination or off topic responses
118118

119119
### Token Consumption
120120

121-
Costtoperformance ratio
121+
* Costtoperformance ratio
122122

123-
Efficiency of prompting patterns
123+
* Efficiency of prompting patterns
124124

125-
Thick vs. thin prompt behavior
125+
* Thick vs. thin prompt behavior
126126

127127
### Reliability Indicators
128128

129-
Sudden increases in latency
129+
* Sudden increases in latency
130130

131-
Changes in model selection effectiveness
131+
* Changes in model selection effectiveness
132132

133-
Errors related to external dependencies
133+
* Errors related to external dependencies
134134

135135
## 5. Diagnosing Issues and Applying Tuning
136136

@@ -146,15 +146,15 @@ Errors related to external dependencies
146146

147147
### 5.2 Tuning Techniques
148148

149-
Refine agent instructions, prompts, and behavior patterns.
149+
* Refine agent instructions, prompts, and behavior patterns.
150150

151-
Update or reorganize knowledge assets.
151+
* Update or reorganize knowledge assets.
152152

153-
Adjust action sequences to reduce bottlenecks.
153+
* Adjust action sequences to reduce bottlenecks.
154154

155-
Reconfigure environment or connector settings.
155+
* Reconfigure environment or connector settings.
156156

157-
Apply versioning and rollback strategies for safety.
157+
* Apply versioning and rollback strategies for safety.
158158

159159
## 6. Architecture
160160

0 commit comments

Comments
 (0)