You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: learn-pr/wwl-data-ai/automated-evaluation-genaiops/includes/5-batch-evaluations-python.md
+5-5Lines changed: 5 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,4 +1,4 @@
1
-
Cloud evaluations enable systematic quality assessment by running multiple evaluators across entire test datasets in Microsoft Foundry. This eliminates the need to manage local compute infrastructure and supports large-scale automated testing workflows.
1
+
Cloud evaluations enable systematic quality assessment by running multiple evaluators across entire test datasets in Microsoft Foundry. These automated evaluations eliminate the need to manage local compute infrastructure and support large-scale automated testing workflows.
2
2
3
3
Adventure Works needs to evaluate 500 test examples against multiple quality criteria to validate their prompt update before deployment. Cloud evaluation with the Foundry SDK completes this work efficiently, running evaluators in parallel and storing results for analysis.
4
4
@@ -7,11 +7,11 @@ Adventure Works needs to evaluate 500 test examples against multiple quality cri
7
7
8
8
## Define data schema and evaluators
9
9
10
-
Cloud evaluation needs to understand your data structure before running evaluators. You define this through a **data source config** that describes the fields in your JSONL dataset and specifies which evaluators to run.
10
+
Cloud evaluation needs to understand your data structure before running evaluators. You define this structure through a **data source config** that describes the fields in your JSONL dataset and specifies which evaluators to run.
11
11
12
12
**Why you need a data schema:**
13
13
14
-
The data schema tells the evaluation service what fields exist in your dataset and which ones are required. This enables validation before execution and helps the service allocate the right resources. Think of it as a contract between your data and the evaluation service.
14
+
The data schema tells the evaluation service what fields exist in your dataset and which ones are required. The data schema enables validation before execution and helps the service allocate the right resources. Think of it as a contract between your data and the evaluation service.
15
15
16
16
```python
17
17
from openai.types.eval_create_params import DataSourceConfigCustom
All evaluators return a standardized schema for each evaluated item:
185
185
186
-
-**Label**: Binary "pass" or "fail" label, similar to a unit test's output—use this for quick comparisons across evaluators
186
+
-**Label**: Binary "pass" or "fail" label, similar to a unit test's output—use the label for quick comparisons across evaluators
187
187
-**Score**: Score from the evaluator's natural scale (1-5 for quality evaluators, 0-7 for safety evaluators, 0-1 for similarity metrics)
188
188
-**Threshold**: Default threshold that determines pass/fail from the score (you can override this)
189
189
-**Reason**: Explanation for the score (for LLM-judge evaluators only)
@@ -194,4 +194,4 @@ For aggregate results across your entire dataset, access `run.result_counts` for
194
194
> [!TIP]
195
195
> Use the `report_url` to view results in the Foundry portal with filtering, sorting, and visualization tools. For CI/CD workflows, parse `output_items` programmatically to enforce quality gates.
196
196
197
-
Now that you understand how to run cloud evaluations with the Foundry SDK, you're ready to learn how to integrate these evaluation workflows into GitHub Actions for automated quality assurance on every code change.
197
+
In the next unit, youlearn how to integrate these evaluation workflows into GitHub Actions to automate quality assurance on every code change.
title: Design a solution for cloud infrastructure entitlement management
4
+
metadata:
5
+
title: Design a solution for cloud infrastructure entitlement management
6
+
description: "Learn to design cloud infrastructure entitlement management (CIEM) solutions using Permissions Management in Microsoft Defender for Cloud for multicloud environments."
0 commit comments