Skip to content

Commit 097476c

Browse files
committed
last fix
1 parent 9c8775f commit 097476c

2 files changed

Lines changed: 53 additions & 53 deletions

File tree

learn-pr/wwl/analyze-monitor-tune-ai-powered-business-solutions/includes/6-interpret-telemetry-data-performance-model-tuning.md

Lines changed: 30 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -10,47 +10,47 @@ Telemetry provides data about how the system behaves in real time. It is essenti
1010

1111
#### Operational Telemetry
1212

13-
Latency and throughput
13+
* Latency and throughput
1414

15-
Error rates and failure modes
15+
* Error rates and failure modes
1616

17-
Resource consumption and throttling
17+
* Resource consumption and throttling
1818

1919
#### ModelLevel Telemetry
2020

21-
Token usage and cost patterns
21+
* Token usage and cost patterns
2222

23-
Response consistency
23+
* Response consistency
2424

25-
Drift indicators and degradation trends
25+
* Drift indicators and degradation trends
2626

2727
#### Behavioral Telemetry
2828

29-
User satisfaction and completion rates
29+
* User satisfaction and completion rates
3030

31-
Prompt patterns and abandonment rates
31+
* Prompt patterns and abandonment rates
3232

33-
Model alignment to intended tasks
33+
* Model alignment to intended tasks
3434

3535
#### Governance and Compliance Signals
3636

37-
Guardrail interventions
37+
* Guardrail interventions
3838

39-
Blocked actions or restricted data access
39+
* Blocked actions or restricted data access
4040

41-
Policy or sensitivity label conflicts
41+
* Policy or sensitivity label conflicts
4242

4343
## 2. Performance Signals and Interpretation
4444

4545
Solution architects should focus on **patterns**, not isolated events.
4646

4747
### Performance Indicators
4848

49-
**Increased latency**<br>Indicates heavy workloads, inefficient prompt structures, or connector delays.
49+
* **Increased latency**<br>Indicates heavy workloads, inefficient prompt structures, or connector delays.
5050

51-
**Spikes in error rates**<br>Often point to broken integrations, incorrect environment configuration, or model instability.
51+
* **Spikes in error rates**<br>Often point to broken integrations, incorrect environment configuration, or model instability.
5252

53-
**High token usage**<br>Suggests verbose outputs, unclear prompts, or an overly complex workflow.
53+
* **High token usage**<br>Suggests verbose outputs, unclear prompts, or an overly complex workflow.
5454

5555
### Performance Signal Map
5656

@@ -68,31 +68,31 @@ Model tuning focuses on improving the quality and reliability of responses.
6868

6969
### 3.1 Tuning Opportunities
7070

71-
**Prompt Refinement**<br>Improving instructions, constraints, and expectations for predictable results.
71+
* **Prompt Refinement**<br>Improving instructions, constraints, and expectations for predictable results.
7272

73-
**Knowledge Updates**<br>Adding, removing, or restructuring knowledge sources for better grounding.
73+
* **Knowledge Updates**<br>Adding, removing, or restructuring knowledge sources for better grounding.
7474

75-
**Behavioral Adjustments**<br>Introducing fallback logic, clarifying actions, or refining orchestration flow.
75+
* **Behavioral Adjustments**<br>Introducing fallback logic, clarifying actions, or refining orchestration flow.
7676

77-
**Cost Optimization**<br>Reducing unnecessary token usage and optimizing invocation structure.
77+
* **Cost Optimization**<br>Reducing unnecessary token usage and optimizing invocation structure.
7878

7979
## 4. TelemetryDriven Diagnosis Workflow
8080

8181
A consistent workflow helps isolate issues quickly.
8282

8383
### StepbyStep Diagnostic Flow
8484

85-
**Monitor Key Metrics**<br>Gather baseline information across latency, throughput, quality, and satisfaction.
85+
* **Monitor Key Metrics**<br>Gather baseline information across latency, throughput, quality, and satisfaction.
8686

87-
**Identify Anomalies**<br>Look for deviations from expected patterns.
87+
* **Identify Anomalies**<br>Look for deviations from expected patterns.
8888

89-
**Correlate Related Signals**<br>Combine user behavior, failures, and performance metrics.
89+
* **Correlate Related Signals**<br>Combine user behavior, failures, and performance metrics.
9090

91-
**Determine Root Cause**<br>Validate if the issue is modelbased, integrationbased, or promptbased.
91+
* **Determine Root Cause**<br>Validate if the issue is modelbased, integrationbased, or promptbased.
9292

93-
**Apply Targeted Tuning**<br>Update prompts, improve workloads, adjust knowledge, or change configuration.
93+
* **Apply Targeted Tuning**<br>Update prompts, improve workloads, adjust knowledge, or change configuration.
9494

95-
**Validate Improvements**<br>Compare beforeandafter telemetry patterns to ensure successful tuning.
95+
* **Validate Improvements**<br>Compare beforeandafter telemetry patterns to ensure successful tuning.
9696

9797
## 5. Architecture Flow of Telemetry Analysis
9898

@@ -103,15 +103,15 @@ A consistent workflow helps isolate issues quickly.
103103

104104
Solution architects should define clear KPIs aligned to business goals:
105105

106-
**Responsiveness**<br>Median response time within acceptable limits.
106+
* **Responsiveness**<br>Median response time within acceptable limits.
107107

108-
**Accuracy & Relevance**<br>Model outputs aligned to task expectations.
108+
* **Accuracy & Relevance**<br>Model outputs aligned to task expectations.
109109

110-
**Reliability**<br>Low failure rate across workflows.
110+
* **Reliability**<br>Low failure rate across workflows.
111111

112-
**CostEffectiveness**<br>Balanced token usage and model selection.
112+
* **CostEffectiveness**<br>Balanced token usage and model selection.
113113

114-
**User Outcome Completion**<br>Ability for users to complete tasks without manual intervention.
114+
* **User Outcome Completion**<br>Ability for users to complete tasks without manual intervention.
115115

116116
## References
117117

learn-pr/wwl/analyze-monitor-tune-ai-powered-business-solutions/includes/8-module-summary.md

Lines changed: 23 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -6,61 +6,61 @@ The module emphasized data driven decision making, structured analysis methods,
66

77
### 1. Monitoring Is Foundational to Reliable AI Operations
88

9-
Effective monitoring spans multiple layers: operational health, performance, quality, usage, and risk.
9+
* Effective monitoring spans multiple layers: operational health, performance, quality, usage, and risk.
1010

11-
A strong monitoring operating model includes defined roles, standardized metrics, log review cadence, and clear expectations for agent behavior.
11+
* A strong monitoring operating model includes defined roles, standardized metrics, log review cadence, and clear expectations for agent behavior.
1212

13-
Guardrails and threshold based alerts help detect anomalies, prevent misuse, and maintain governance alignment.
13+
* Guardrails and threshold based alerts help detect anomalies, prevent misuse, and maintain governance alignment.
1414

1515
### 2. Enterprise Tools Enable Deep Observability
1616

17-
Solution architects rely on telemetry tools, analytics dashboards, and observability platforms to gain insight into agent performance.
17+
* Solution architects rely on telemetry tools, analytics dashboards, and observability platforms to gain insight into agent performance.
1818

19-
Monitoring solutions help track errors, latency, usage trends, connector performance, and model behavior.
19+
* Monitoring solutions help track errors, latency, usage trends, connector performance, and model behavior.
2020

21-
Centralized dashboards improve visibility across complex AI environments.
21+
* Centralized dashboards improve visibility across complex AI environments.
2222

2323
### 3. Backlogs and User Feedback Drive Continuous Improvements
2424

25-
Backlogs capture enhancements, recurring issues, feature gaps, and governance concerns.
25+
* Backlogs capture enhancements, recurring issues, feature gaps, and governance concerns.
2626

27-
Categorizing backlog items into domains like accuracy, knowledge, performance, UX, integration, and compliance enables structured prioritization.
27+
* Categorizing backlog items into domains like accuracy, knowledge, performance, UX, integration, and compliance enables structured prioritization.
2828

29-
User feedback—volume, severity, sentiment, and friction points—provides critical signals for improvement planning.
29+
* User feedback—volume, severity, sentiment, and friction points—provides critical signals for improvement planning.
3030

31-
Conversation transcripts reveal root causes that aren't always visible in raw telemetry.
31+
* Conversation transcripts reveal root causes that aren't always visible in raw telemetry.
3232

3333
### 4. Telemetry and Diagnostics Reveal Root Causes
3434

35-
Key telemetry signals include latency spikes, error patterns, token consumption, drift patterns, and guardrail triggers.
35+
* Key telemetry signals include latency spikes, error patterns, token consumption, drift patterns, and guardrail triggers.
3636

37-
A consistent diagnostic workflow helps isolate issues: monitor → detect anomalies → correlate patterns → identify root causes → tune → validate.
37+
* A consistent diagnostic workflow helps isolate issues: monitor → detect anomalies → correlate patterns → identify root causes → tune → validate.
3838

3939
### 5. Tuning Improves Accuracy, Performance, and User Experience
4040

41-
Knowledge tuning: update, restructure, or remove outdated content.
41+
* Knowledge tuning: update, restructure, or remove outdated content.
4242

43-
Behavioral tuning: clarify orchestration steps, refine prompts, and introduce fallback logic.
43+
* Behavioral tuning: clarify orchestration steps, refine prompts, and introduce fallback logic.
4444

45-
Performance tuning: optimize workflows, reduce unnecessary actions, fix connector, or dependency bottlenecks.
45+
* Performance tuning: optimize workflows, reduce unnecessary actions, fix connector, or dependency bottlenecks.
4646

47-
Governance aligned tuning ensures that DLP, permissions, and sensitivity rules remain intact.
47+
* Governance aligned tuning ensures that DLP, permissions, and sensitivity rules remain intact.
4848

4949
### 6. Performance and User Centered Metrics Track Effectiveness
5050

51-
Core operational metrics: latency, throughput, error rate, and resource utilization.
51+
* Core operational metrics: latency, throughput, error rate, and resource utilization.
5252

53-
Quality metrics: accuracy, knowledge coverage, and action effectiveness.
53+
* Quality metrics: accuracy, knowledge coverage, and action effectiveness.
5454

55-
User centered metrics: satisfaction, abandonment rate, and task completion rate (the best indicator of user success).
55+
* User centered metrics: satisfaction, abandonment rate, and task completion rate (the best indicator of user success).
5656

57-
Monitoring model and drift indicators ensure models remain reliable over time.
57+
* Monitoring model and drift indicators ensure models remain reliable over time.
5858

5959
### 7. Continuous Improvement Is an Ongoing Lifecycle
6060

61-
AI agents require an iterative approach: monitor → analyze → tune → validate → release → monitor again.
61+
* AI agents require an iterative approach: monitor → analyze → tune → validate → release → monitor again.
6262

63-
Regular updates to knowledge content, workflows, guardrails, and orchestration refine agent behavior.
63+
* Regular updates to knowledge content, workflows, guardrails, and orchestration refine agent behavior.
6464

6565

66-
Insights from telemetry and transcripts feed into improvement cycles and stakeholder communication.
66+
* Insights from telemetry and transcripts feed into improvement cycles and stakeholder communication.

0 commit comments

Comments
 (0)