Merge pull request #53615 from madiepev/update-auto-eval

New module auto eval

Author: Carolyn135

learn-pr/paths/operationalize-gen-ai-apps/index.yml

@@ -2,8 +2,8 @@
 uid: learn.wwl.operationalize-gen-ai-apps
 metadata:
   title: Operationalize generative AI applications (GenAIOps)
-  description: Learn how to develop, evaluate, optimize, and deploy generative AI applications (GenAIOps)
-  ms.date: 08/06/2025
+  description: Learn the full GenAIOps lifecycle for generative AI applications, from planning and prompt management to evaluation, automated testing, monitoring, and tracing in production.
+  ms.date: 02/23/2026
   author: wwlpublish
   ms.author: madiepev
   ms.topic: learning-path
@@ -13,23 +13,28 @@ title: Operationalize generative AI applications (GenAIOps)
 prerequisites: |
   Before starting this learning path, you should be familiar with fundamental generative AI concepts and services in Azure. Consider completing the [Microsoft Azure AI Fundamentals: Generative AI](/training/paths/introduction-generative-ai/?azure-portal=true) learning path first.
 summary: |
-  To effectively scale generative Artificial Intelligence (AI) applications, you need to manage, deploy, and maintain GenAI apps to ensure their performance, reliability, and continuous improvement in real-world applications.
+  Learn how to operationalize generative AI applications using the complete GenAIOps lifecycle. This learning path covers planning and preparing GenAIOps solutions, managing prompts for agents with version control, evaluating and optimizing agents through structured experiments, automating evaluations with Microsoft Foundry and GitHub Actions, monitoring application performance and costs, and implementing distributed tracing to debug complex AI workflows.
 iconUrl: /training/achievements/generic-badge.svg
 levels:
 - intermediate
 roles:
 - data-scientist
 - ai-engineer
+- devops-engineer
 products:
 - ai-services
+- azure-ai-foundry
+- github
 subjects:
 - artificial-intelligence
 - machine-learning
 - natural-language-processing
+- devops
 modules:
 - learn.wwl.plan-prepare-genaiops
 - learn.wwl.prompt-versioning-genaiops
-- learn.evaluate-generative-ai-apps
+- learn.wwl.evaluate-optimize-agents
+- learn.wwl.automated-evaluation-genaiops
 - learn.wwl.monitor-generative-ai-app
 - learn.wwl.tracing-generative-ai-app
 trophy:

learn-pr/wwl-data-ai/automated-evaluation-genaiops/1-introduction.yml

@@ -0,0 +1,15 @@
+### YamlMime:ModuleUnit
+uid: learn.wwl.automated-evaluation-genaiops.introduction
+title: Introduction
+metadata:
+  title: Introduction
+  description: "Introduction to automated evaluations with Microsoft Foundry and GitHub Actions"
+  ms.date: 02/22/2026
+  author: madiepev
+  ms.author: madiepev
+  ms.topic: unit
+  ms.custom:
+  - N/A
+durationInMinutes: 3
+content: |
+  [!include[](includes/1-introduction.md)]

learn-pr/wwl-data-ai/automated-evaluation-genaiops/2-why-automated-evaluations.yml

@@ -0,0 +1,15 @@
+### YamlMime:ModuleUnit
+uid: learn.wwl.automated-evaluation-genaiops.why-automated-evaluations
+title: Understand why automated evaluations matter
+metadata:
+  title: Understand why automated evaluations matter
+  description: "Understand the trade-offs between human and automated evaluation, and learn how human-in-the-loop approaches combine both strategically"
+  ms.date: 02/22/2026
+  author: madiepev
+  ms.author: madiepev
+  ms.topic: unit
+  ms.custom:
+  - N/A
+durationInMinutes: 6
+content: |
+  [!include[](includes/2-why-automated-evaluations.md)]

learn-pr/wwl-data-ai/automated-evaluation-genaiops/3-align-evaluators-human-criteria.yml

@@ -0,0 +1,15 @@
+### YamlMime:ModuleUnit
+uid: learn.wwl.automated-evaluation-genaiops.align-evaluators-human-criteria
+title: Align evaluators with human criteria
+metadata:
+  title: Align evaluators with human criteria
+  description: "Follow a workflow to select evaluators, run shadow rating, monitor alignment, and refine with custom evaluators"
+  ms.date: 02/22/2026
+  author: madiepev
+  ms.author: madiepev
+  ms.topic: unit
+  ms.custom:
+  - N/A
+durationInMinutes: 10
+content: |
+  [!include[](includes/3-align-evaluators-human-criteria.md)]

learn-pr/wwl-data-ai/automated-evaluation-genaiops/4-create-evaluation-data.yml

@@ -0,0 +1,15 @@
+### YamlMime:ModuleUnit
+uid: learn.wwl.automated-evaluation-genaiops.create-evaluation-data
+title: Create evaluation datasets
+metadata:
+  title: Create evaluation datasets
+  description: "Create comprehensive evaluation datasets from production data and synthetic generation with proper composition across common scenarios, variations, edge cases, and adversarial examples"
+  ms.date: 02/22/2026
+  author: madiepev
+  ms.author: madiepev
+  ms.topic: unit
+  ms.custom:
+  - N/A
+durationInMinutes: 10
+content: |
+  [!include[](includes/4-create-evaluation-data.md)]

learn-pr/wwl-data-ai/automated-evaluation-genaiops/5-batch-evaluations-python.yml

@@ -0,0 +1,15 @@
+### YamlMime:ModuleUnit
+uid: learn.wwl.automated-evaluation-genaiops.batch-evaluations-python
+title: Implement batch evaluations with Python
+metadata:
+  title: Implement batch evaluations with Python
+  description: "Learn how to run batch evaluations using Python scripts with Microsoft Foundry"
+  ms.date: 02/22/2026
+  author: madiepev
+  ms.author: madiepev
+  ms.topic: unit
+  ms.custom:
+  - N/A
+durationInMinutes: 10
+content: |
+  [!include[](includes/5-batch-evaluations-python.md)]

learn-pr/wwl-data-ai/automated-evaluation-genaiops/6-github-actions-workflow.yml

@@ -0,0 +1,15 @@
+### YamlMime:ModuleUnit
+uid: learn.wwl.automated-evaluation-genaiops.github-actions-workflow
+title: Integrate evaluations into GitHub Actions
+metadata:
+  title: Integrate evaluations into GitHub Actions
+  description: "Learn how to automate Python evaluation scripts in GitHub Actions workflows triggered by pull requests"
+  ms.date: 02/22/2026
+  author: madiepev
+  ms.author: madiepev
+  ms.topic: unit
+  ms.custom:
+  - N/A
+durationInMinutes: 10
+content: |
+  [!include[](includes/6-github-actions-workflow.md)]

learn-pr/wwl-data-ai/automated-evaluation-genaiops/7-exercise.yml

@@ -0,0 +1,15 @@
+### YamlMime:ModuleUnit
+uid: learn.wwl.automated-evaluation-genaiops.exercise
+title: Exercise - Set up automated evaluations
+metadata:
+  title: Exercise - Set up automated evaluations
+  description: "Implement automated evaluations with Microsoft Foundry and GitHub Actions"
+  ms.date: 02/22/2026
+  author: madiepev
+  ms.author: madiepev
+  ms.topic: unit
+  ms.custom:
+  - N/A
+durationInMinutes: 20
+content: |
+  [!include[](includes/7-exercise.md)]

learn-pr/wwl-data-ai/automated-evaluation-genaiops/8-knowledge-check.yml

@@ -0,0 +1,50 @@
+### YamlMime:ModuleUnit
+uid: learn.wwl.automated-evaluation-genaiops.knowledge-check
+title: Knowledge check
+metadata:
+  title: Knowledge check
+  description: "Knowledge check for automated evaluation with Microsoft Foundry and GitHub Actions"
+  ms.date: 02/22/2026
+  author: madiepev
+  ms.author: madiepev
+  ms.topic: unit
+  ms.custom:
+  - N/A
+durationInMinutes: 3
+content: |
+quiz:
+  title: "Check your knowledge"
+  questions:
+  - content: "What is the primary benefit of using shadow ratings during the transition from human to automated evaluations?"
+    choices:
+    - content: "Shadow ratings eliminate the need for human evaluators completely."
+      isCorrect: false
+      explanation: "Incorrect. Shadow ratings don't eliminate human evaluators; they run alongside them to validate automated evaluators."
+    - content: "Shadow ratings allow you to compare automated evaluator scores with human ratings to measure alignment before fully trusting automation."
+      isCorrect: true
+      explanation: "Correct. Shadow ratings run automated evaluations alongside human evaluations to validate that automated scores align with human judgment before relying on them exclusively."
+    - content: "Shadow ratings reduce the cost of evaluations by replacing expensive cloud computing with local processing."
+      isCorrect: false
+      explanation: "Incorrect. Shadow ratings don't focus on cost reduction; they focus on validating automated evaluators against human judgment."
+  - content: "When creating a synthetic test dataset, what percentage should typically represent edge cases?"
+    choices:
+    - content: "5-10% to ensure the system handles unusual scenarios without over-optimizing for rare cases."
+      isCorrect: true
+      explanation: "Correct. Edge cases should represent 5-10% of your test dataset to validate handling of unusual scenarios while maintaining focus on common use cases."
+    - content: "50% to ensure comprehensive coverage of all possible scenarios."
+      isCorrect: false
+      explanation: "Incorrect. 50% edge cases would over-represent unusual scenarios and lead to systems over-optimized for rare situations."
+    - content: "Less than 1% since edge cases rarely occur in production."
+      isCorrect: false
+      explanation: "Incorrect. While edge cases are rare, less than 1% provides insufficient coverage to validate system behavior in unusual scenarios."
+  - content: "What triggers a GitHub Actions workflow for automated evaluation in a pull request-based workflow?"
+    choices:
+    - content: "Manual approval from a senior team member after code review."
+      isCorrect: false
+      explanation: "Incorrect. GitHub Actions workflows are triggered automatically by events like pull request creation, not manual approval."
+    - content: "Creating or updating a pull request that modifies prompt files."
+      isCorrect: true
+      explanation: "Correct. GitHub Actions workflows use triggers like 'pull_request' events on specific paths to automatically run evaluations when prompt changes are proposed."
+    - content: "Deploying code to production after merging to the main branch."
+      isCorrect: false
+      explanation: "Incorrect. While you can trigger workflows on merge, the primary evaluation workflow runs before merge during the pull request phase to catch issues early."

learn-pr/wwl-data-ai/automated-evaluation-genaiops/9-summary.yml

@@ -0,0 +1,15 @@
+### YamlMime:ModuleUnit
+uid: learn.wwl.automated-evaluation-genaiops.summary
+title: Summary
+metadata:
+  title: Summary
+  description: "Summary of automated evaluation with Microsoft Foundry and GitHub Actions"
+  ms.date: 02/22/2026
+  author: madiepev
+  ms.author: madiepev
+  ms.topic: unit
+  ms.custom:
+  - N/A
+durationInMinutes: 2
+content: |
+  [!include[](includes/9-summary.md)]