Merge pull request #54049 from MicrosoftDocs/NEW-module08-analyze-monitor-tune-ai-powered-business-solutions

New module08 analyze monitor tune ai powered business solutions

Author: v-dirichards

learn-pr/wwl/analyze-monitor-tune-ai-powered-business-solutions/1-introduction.yml

@@ -0,0 +1,13 @@
+### YamlMime:ModuleUnit
+uid: learn.wwl.analyze-monitor-tune-ai-powered-business-solutions.introduction
+title: "Introduction"
+metadata:
+  title: "Introduction"
+  description: "Learn the essentials of monitoring, analyzing, and tuning AI-powered agents to ensure reliability, effectiveness, and continuous improvement."
+  ms.date: 02/13/2026
+  author: msdavidram
+  ms.author: taeldin
+  ms.topic: unit
+durationInMinutes: 3
+content: |
+  [!include[](includes/1-introduction.md)]

learn-pr/wwl/analyze-monitor-tune-ai-powered-business-solutions/2-recommend-process-tools-monitoring-agents.yml

@@ -0,0 +1,13 @@
+### YamlMime:ModuleUnit
+uid: learn.wwl.analyze-monitor-tune-ai-powered-business-solutions.recommend-process-tools-monitoring-agents
+title: "Recommend process tools for monitoring agents"
+metadata:
+  title: "Recommend Process Tools for Monitoring Agents"
+  description: "Learn how to recommend processes and tools for monitoring AI agents, ensuring observability, compliance, and continuous improvement."
+  ms.date: 02/13/2026
+  author: msdavidram
+  ms.author: taeldin
+  ms.topic: unit
+durationInMinutes: 6
+content: |
+  [!include[](includes/2-recommend-process-tools-monitoring-agents.md)]

learn-pr/wwl/analyze-monitor-tune-ai-powered-business-solutions/3-analyze-backlog-user-feedback-ai-agent-usage.yml

@@ -0,0 +1,13 @@
+### YamlMime:ModuleUnit
+uid: learn.wwl.analyze-monitor-tune-ai-powered-business-solutions.analyze-backlog-user-feedback-ai-agent-usage
+title: "Analyze backlog and user feedback for AI agent usage"
+metadata:
+  title: "Analyze Backlog and User Feedback for AI Agent Usage"
+  description: "Learn how to analyze backlog data and user feedback to improve AI agent performance, prioritize enhancements, and address operational issues."
+  ms.date: 02/13/2026
+  author: msdavidram
+  ms.author: taeldin
+  ms.topic: unit
+durationInMinutes: 6
+content: |
+  [!include[](includes/3-analyze-backlog-user-feedback-ai-agent-usage.md)]

learn-pr/wwl/analyze-monitor-tune-ai-powered-business-solutions/4-apply-ai-based-tools-analyze-identify-issues-perform-tuning.yml

@@ -0,0 +1,13 @@
+### YamlMime:ModuleUnit
+uid: learn.wwl.analyze-monitor-tune-ai-powered-business-solutions.apply-ai-based-tools-analyze-identify-issues-perform-tuning
+title: "Apply AI-based tools to analyze, identify issues, and perform tuning"
+metadata:
+  title: "Apply AI-Based Tools to Analyze, Identify Issues, and Perform Tuning"
+  description: "Learn how to analyze AI agent behavior, diagnose issues, and implement tuning strategies to improve reliability, performance, and user satisfaction."
+  ms.date: 02/13/2026
+  author: msdavidram
+  ms.author: taeldin
+  ms.topic: unit
+durationInMinutes: 5
+content: |
+  [!include[](includes/4-apply-ai-based-tools-analyze-identify-issues-perform-tuning.md)]

learn-pr/wwl/analyze-monitor-tune-ai-powered-business-solutions/5-monitor-agent-performance-metrics.yml

@@ -0,0 +1,13 @@
+### YamlMime:ModuleUnit
+uid: learn.wwl.analyze-monitor-tune-ai-powered-business-solutions.monitor-agent-performance-metrics
+title: "Monitor AI agent performance metrics"
+metadata:
+  title: "Monitor AI Agent Performance Metrics"
+  description: "Learn how to monitor AI agent performance metrics, evaluate operational health, and optimize reliability using structured observability practices."
+  ms.date: 02/13/2026
+  author: msdavidram
+  ms.author: taeldin
+  ms.topic: unit
+durationInMinutes: 5
+content: |
+  [!include[](includes/5-monitor-agent-performance-metrics.md)]

learn-pr/wwl/analyze-monitor-tune-ai-powered-business-solutions/6-interpret-telemetry-data-performance-model-tuning.yml

@@ -0,0 +1,13 @@
+### YamlMime:ModuleUnit
+uid: learn.wwl.analyze-monitor-tune-ai-powered-business-solutions.interpret-telemetry-data-performance-model-tuning
+title: "Interpret telemetry data to tune AI performance"
+metadata:
+  title: "Interpret Telemetry Data to Tune AI Performance"
+  description: "Learn to analyze telemetry data from AI systems to diagnose issues, optimize performance, and guide continuous tuning for better outcomes."
+  ms.date: 02/13/2026
+  author: msdavidram
+  ms.author: taeldin
+  ms.topic: unit
+durationInMinutes: 4
+content: |
+  [!include[](includes/6-interpret-telemetry-data-performance-model-tuning.md)]

learn-pr/wwl/analyze-monitor-tune-ai-powered-business-solutions/7-knowledge-check.yml

@@ -0,0 +1,57 @@
+### YamlMime:ModuleUnit
+uid: learn.wwl.analyze-monitor-tune-ai-powered-business-solutions.knowledge-check
+title: "Module assessment"
+metadata:
+  title: "Knowledge check"
+  description: "Knowledge check"
+  ms.date: 02/13/2026
+  author: msdavidram
+  ms.author: taeldin
+  ms.topic: unit
+  module_assessment: false
+durationInMinutes: 3
+content: "Choose the best response for each of the following questions."
+quiz:
+  questions:
+  - content: "Which of the following is a key component when establishing a monitoring operating model for AI agents?"
+    choices:
+    - content: "Ignoring agent guardrail triggers"
+      isCorrect: false
+      explanation: "Incorrect. Ignoring guardrail triggers can compromise agent reliability and compliance."
+    - content: "Standardizing metric definitions and log review cadence"
+      isCorrect: true
+      explanation: "Correct. Establishing a monitoring operating model requires clear and consistent processes, including standardized metric definitions and a regular log review cadence. This ensures accountability, consistency, and the ability to detect and respond to issues effectively."
+    - content: "Disabling all alerts to reduce noise"
+      isCorrect: false
+      explanation: "Incorrect. Disabling alerts can prevent the detection of critical issues and compromise agent reliability."
+    - content: "Allowing unrestricted agent configuration"
+      isCorrect: false
+      explanation: "Incorrect. Allowing unrestricted configuration can lead to inconsistencies and potential compliance issues."
+  - content: "When analyzing backlog items for AI and agent usage, what is the best first step?"
+    choices:
+    - content: "Immediately redesign agent prompts"
+      isCorrect: false
+      explanation: "Incorrect. Immediate redesign does not provide the necessary insight for targeted improvement."
+    - content: "Categorize backlog items into meaningful domains"
+      isCorrect: true
+      explanation: "Correct. The first step in effective backlog analysis is to categorize items into meaningful domains such as accuracy, knowledge, performance, user experience, integration, and governance. This structured approach helps solution architects prioritize improvements, detect patterns, and address root causes systematically."
+    - content: "Archive past feedback to avoid noise"
+      isCorrect: false
+      explanation: "Incorrect. Archiving feedback can result in the loss of valuable insights needed for improvement."
+    - content: "Disable the agent until issues are fixed"
+      isCorrect: false
+      explanation: "Incorrect. Disabling the agent does not address the root causes or provide insights for improvement."
+  - content: "Which metric best indicates whether users are achieving the intended outcome of an agent workflow?"
+    choices:
+    - content: "Token usage"
+      isCorrect: false
+      explanation: "Incorrect. Token usage is an operational metric and does not directly indicate user success in achieving intended outcomes."
+    - content: "Task completion rate"
+      isCorrect: true
+      explanation: "Correct. Task completion rate directly measures whether users are able to successfully complete the workflows for which the agent was designed. It reflects both the agent's effectiveness and the user's satisfaction with the process."
+    - content: "Connector quota"
+      isCorrect: false
+      explanation: "Incorrect. Connector quota is an operational metric and does not directly indicate user success in achieving intended outcomes."
+    - content: "Storage utilization"
+      isCorrect: false
+      explanation: "Incorrect. Storage utilization is an operational metric and does not directly indicate user success in achieving intended outcomes."

learn-pr/wwl/analyze-monitor-tune-ai-powered-business-solutions/8-module-summary.yml

@@ -0,0 +1,13 @@
+### YamlMime:ModuleUnit
+uid: learn.wwl.analyze-monitor-tune-ai-powered-business-solutions.module-summary
+title: "Ensure reliable AI agent operations"
+metadata:
+  title: "Ensure Reliable AI Agent Operations"
+  description: "Learn how to monitor, analyze, and tune AI agents to ensure reliability, optimize performance, and align with enterprise governance goals."
+  ms.date: 02/13/2026
+  author: msdavidram
+  ms.author: taeldin
+  ms.topic: unit
+durationInMinutes: 4
+content: |
+  [!include[](includes/8-module-summary.md)]

learn-pr/wwl/analyze-monitor-tune-ai-powered-business-solutions/includes/1-introduction.md

@@ -0,0 +1,15 @@
+This module is designed to empower solution architects with the foundational knowledge and practical techniques required to ensure the reliability, effectiveness, and continuous improvement of AI-driven agents within enterprise environments.
+
+AI-powered agents are transforming business operations by automating workflows, enhancing decision-making, and enabling new forms of user interaction. However, their success depends on robust monitoring, structured analysis, and systematic tuning practices. Solution architects play a pivotal role in defining strategies that guarantee agents operate predictably, deliver high-quality outcomes, and comply with organizational governance standards.
+
+## Throughout this module, you will learn about:
+
+- **Establish Monitoring Frameworks**: Understand multi-layered monitoring requirements, including operational health, performance metrics, quality assurance, usage insights, and risk management.
+- **Leverage Industry Tools and Processes**: Explore recommended monitoring tools such as Azure Monitor, Microsoft 365 Admin Analytics, Copilot analytics dashboards, Power Platform Admin Center, and enterprise observability platforms. Learn how to design resilient monitoring models, configure guardrails, set alerts, and conduct regular quality evaluations.
+- **Analyze Backlogs and User Feedback:** Develop repeatable frameworks for interpreting backlog data and user feedback. Learn to categorize issues, prioritize enhancements, and translate insights into actionable improvements.
+- **Apply AI-Based Diagnostic and Tuning Methods:** Master the use of telemetry, conversation transcripts, and performance scorecards to diagnose agent issues and implement targeted tuning strategies.
+- **Monitor Performance and Metrics:** Define and track operational, qualitative, and user-centered metrics. Understand how to interpret telemetry data to identify anomalies, assess model drift, and optimize agent workflows.
+
+This module will equip you with the expertise to design and operationalize monitoring and tuning strategies that align with business objectives, drive continuous improvement, and ensure compliance with IT and governance requirements. By mastering these principles, you will be prepared to deliver high-confidence AI solutions that scale reliably and adapt to evolving enterprise needs.
+
+Let us begin our journey into the essential practices of analyzing, monitoring, and tuning AI-powered business solutions.

learn-pr/wwl/analyze-monitor-tune-ai-powered-business-solutions/includes/2-recommend-process-tools-monitoring-agents.md

@@ -0,0 +1,187 @@
+## Overview
+
+This unit equips solution architects with the expertise to define, recommend, and operationalize a monitoring strategy for AI agents across the Microsoft ecosystem. The focus is on designing a resilient, governed, and observable monitoring model that enables organizations to measure agent effectiveness, detect operational risks, and ensure compliance with IT and business requirements.
+
+You will explore monitoring processes, recommended tools, observability patterns, dashboards, alerting approaches, and analytical insights that support continuous improvement of agent behavior.
+
+## Understanding Monitoring Requirements for AI Agents
+
+### Monitoring AI agents requires a multilayered approach. Solution architects must consider
+
+**Operational Health**<br>Uptime, availability, error frequency, throttling conditions, processing delays.
+
+**Performance Metrics**<br>Response times, success rates of actions, tool invocation reliability, workflow completion metrics.
+
+**Quality and Output Accuracy**<br>Appropriateness of generated actions or responses, alignment with business rules, deviation from expected behavior.
+
+**Usage Insights**<br>Volume trends, active user adoption, agent feature utilization, behavioral patterns over time.
+
+**Risk, Compliance, and Security**<br>Guardrail violations, sensitivedata handling, suspicious activity spikes, adherence to organizational policies.
+
+## Recommended Processes for Monitoring AI Agents
+
+Solution architects should recommend the processes for monitoring AI Agents across an organization. When an existing framework is in place, the architect should look for missing components or improvements. 
+
+### Establish a Monitoring Operating Model
+
+* A strong operational model ensures consistency, ownership, and accountability.
+
+#### Key components
+
+* Defined roles (Ops team, product owners, data engineers, architects)
+
+* Process workflows for incident response
+
+* Standardized metric definitions (creating a baseline with trends)
+
+* Log review cadence (daily/weekly/monthly)
+
+* Change management and version tracking
+
+* Documentation of expected agent behaviors and constraints
+
+### Configure Guardrails and Threshold Alerts
+
+* Set thresholds for latency, exception volume, and unusual activity.
+
+* Create automated alerts for guardrail triggers or tool invocation failures.
+
+* Monitor for unexpected spikes in prompts indicating potential misuse.
+
+### Conduct Regular Quality Evaluations
+
+* Humanintheloop spot checks
+
+* Scenariobased evaluations
+
+* Review lowconfidence outputs
+
+* Validate alignment with business rules or compliance requirements
+
+### Continuously Improve Based on Insights
+
+* Analyze logs and telemetry to find failure patterns.
+
+* Identify training needs for users.
+
+* Recommend prompt engineering improvements.
+
+* Propose workflow adjustments or retraining of custom models (if applicable).
+
+## Recommended Tools for Monitoring AI Agents
+
+Solution architects should recommend the toolset that covers **observability**, **analytics**, and **administrative insights**.
+
+### Azure Monitor (Core Telemetry + Alerts)
+
+#### Azure Monitor provides
+
+* Application and agent telemetry
+
+* *Dashboards for real-time* metrics
+
+* Alert rules for anomalies
+
+* Integration with Log Analytics Workspaces
+
+#### Use cases
+
+* Monitor agent workflows built with Power Platform or custom services.
+
+* Track errors, latency, throughput, connector failures.
+
+* Build KQL-based queries for deep diagnostics.
+
+### Microsoft 365 Admin Analytics (Usage & Adoption Trends)
+
+#### Useful for
+
+* Understanding agent usage volume
+
+* Tracking adoption and engagement
+
+* Identifying departments with low usage or operational barriers
+
+* Measuring improvements week-over-week
+
+### Copilot & Agent Analytics Dashboards
+
+#### When available in an organization's tenant, Copilot analytics can provide
+
+* Agent invocation frequency
+
+* Task completion trends
+
+* Common user queries
+
+* Productivity pattern insights
+
+* Error or guardrail-trigger events
+
+### Power Platform Admin Center (Environment-Level Monitoring)
+
+#### Provides
+
+* Environment health
+
+* Connector usage and limits
+
+* Flow telemetry (for agents using workflows)
+
+* DLP rule impact visibility
+
+### Foundry or Organizational Observability Platforms
+
+#### Enterprises may adopt centralized observability platforms (example: Foundry-like solutions, if present in the environment) to unify
+
+* Multisystem logs
+
+* Event traces
+
+* Cross-environment dashboards
+
+* AI model execution insights
+
+* These platforms reduce fragmentation and provide a single-pane-of-glass view for complex agent ecosystems.
+
+### Custom Dashboards for Enterprise AI Agents
+
+#### Solution architects often design
+
+* KPI dashboards in Power BI
+
+* Heatmaps of usage
+
+* Drift detection visualizations
+
+* Compliance trend reports
+
+#### Example Agent Health Summary
+
+| Agent Name | Success Rate | Avg. Response Time | Errors Today | Usage Trend |
+| --- | --- | --- | --- | --- |
+| Sales Helper | 98% | 1.8 sec | 3 | ↑ Increasing |
+| Ops Agent | 92% | 2.5 sec | 17 | → Steady |
+| Finance Advisor | 86% | 3.2 sec | 28 | ↓ Decreasing |
+
+#### Best Practices
+
+* Always centralize logs.
+
+* Standardize naming conventions.
+
+* Define clear SLAs for agent responsiveness.
+
+* Automate alerting for critical business workflows.
+
+* Integrate monitoring outputs into monthly operational reviews.
+
+## References
+
+[https://learn.microsoft.com/training/modules/describe-monitoring-tools-azure/4-describe-azure-monitor](/training/modules/describe-monitoring-tools-azure/4-describe-azure-monitor)
+
+[https://learn.microsoft.com/training/modules/perform-admin-tasks-microsoft-365-copilot/](/training/modules/perform-admin-tasks-microsoft-365-copilot/)
+
+[https://learn.microsoft.com/azure/ai-foundry/observability/how-to/how-to-monitor-agents-dashboard?view=foundry](/azure/ai-foundry/observability/how-to/how-to-monitor-agents-dashboard)
+
+[https://learn.microsoft.com/power-platform/admin/analytics-copilot](/power-platform/admin/analytics-copilot)