Skip to content

Commit b45752d

Browse files
committed
model catalog evaluation module
1 parent 4f84c3b commit b45752d

21 files changed

Lines changed: 654 additions & 0 deletions
Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
### YamlMime:ModuleUnit
2+
uid: learn.wwl.model-catalog-evaluate.introduction
3+
title: Introduction
4+
metadata:
5+
title: Introduction
6+
description: Learn why model selection and evaluation are essential for building effective generative AI applications.
7+
ms.date: 02/13/2026
8+
author: ivorb
9+
ms.author: berryivor
10+
ms.topic: unit
11+
ai-usage: ai-assisted
12+
durationInMinutes: 3
13+
content: |
14+
[!include[](includes/1-introduction.md)]
Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
### YamlMime:ModuleUnit
2+
uid: learn.wwl.model-catalog-evaluate.explore-model-catalog
3+
title: Explore the model catalog
4+
metadata:
5+
title: Explore the model catalog
6+
description: Learn how to browse and filter models in the Microsoft Foundry portal model catalog.
7+
ms.date: 02/13/2026
8+
author: ivorb
9+
ms.author: berryivor
10+
ms.topic: unit
11+
ai-usage: ai-assisted
12+
durationInMinutes: 7
13+
content: |
14+
[!include[](includes/2-explore-model-catalog.md)]
Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
### YamlMime:ModuleUnit
2+
uid: learn.wwl.model-catalog-evaluate.select-models-benchmarks
3+
title: Select models using benchmarks
4+
metadata:
5+
title: Select models using benchmarks
6+
description: Understand how to use quality, safety, cost, and performance benchmarks to compare and select models.
7+
ms.date: 02/13/2026
8+
author: ivorb
9+
ms.author: berryivor
10+
ms.topic: unit
11+
ai-usage: ai-assisted
12+
durationInMinutes: 9
13+
content: |
14+
[!include[](includes/3-select-models-benchmarks.md)]
Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
### YamlMime:ModuleUnit
2+
uid: learn.wwl.model-catalog-evaluate.deploy-models
3+
title: Deploy models to endpoints
4+
metadata:
5+
title: Deploy models to endpoints
6+
description: Learn how to deploy models to different endpoint types and test them in the playground.
7+
ms.date: 02/13/2026
8+
author: ivorb
9+
ms.author: berryivor
10+
ms.topic: unit
11+
ai-usage: ai-assisted
12+
durationInMinutes: 8
13+
content: |
14+
[!include[](includes/4-deploy-models.md)]
Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
### YamlMime:ModuleUnit
2+
uid: learn.wwl.model-catalog-evaluate.evaluate-performance
3+
title: Evaluate model performance
4+
metadata:
5+
title: Evaluate model performance
6+
description: Discover how to evaluate models using manual testing, AI-assisted metrics, and NLP metrics.
7+
ms.date: 02/13/2026
8+
author: ivorb
9+
ms.author: berryivor
10+
ms.topic: unit
11+
ai-usage: ai-assisted
12+
durationInMinutes: 10
13+
content: |
14+
[!include[](includes/5-evaluate-performance.md)]
Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
### YamlMime:ModuleUnit
2+
uid: learn.wwl.model-catalog-evaluate.exercise
3+
title: Exercise - Select, deploy, and evaluate models
4+
metadata:
5+
title: Exercise - Select, deploy, and evaluate models
6+
description: Practice selecting, deploying, and evaluating models in Microsoft Foundry portal.
7+
ms.date: 02/13/2026
8+
author: ivorb
9+
ms.author: berryivor
10+
ms.topic: unit
11+
ai-usage: ai-assisted
12+
durationInMinutes: 20
13+
content: |
14+
[!include[](includes/6-exercise.md)]
Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
### YamlMime:ModuleUnit
2+
uid: learn.wwl.model-catalog-evaluate.knowledge-check
3+
title: Knowledge check
4+
metadata:
5+
title: Knowledge check
6+
description: Check your understanding of model selection, deployment, and evaluation concepts.
7+
ms.date: 02/13/2026
8+
author: ivorb
9+
ms.author: berryivor
10+
ms.topic: unit
11+
ai-usage: ai-assisted
12+
durationInMinutes: 3
13+
quiz:
14+
title: Check your knowledge
15+
questions:
16+
- content: "Which deployment type in Microsoft Foundry portal requires an Azure Marketplace subscription for models from partners and the community?"
17+
choices:
18+
- content: "Managed compute"
19+
isCorrect: false
20+
explanation: "Incorrect. Managed compute doesn't necessarily require Azure Marketplace subscriptions. It uses Azure virtual machines and is billed for hosting and inference costs."
21+
- content: "Serverless API"
22+
isCorrect: true
23+
explanation: "Correct. Serverless API deployments for models from partners and community require Azure Marketplace subscriptions, while models sold directly by Azure don't require this subscription."
24+
- content: "Provisioned"
25+
isCorrect: false
26+
explanation: "Incorrect. Provisioned deployments reserve dedicated capacity but don't specifically require Azure Marketplace subscriptions."
27+
- content: "Which evaluation metric measures whether model responses are based on provided context rather than speculation?"
28+
choices:
29+
- content: "Fluency"
30+
isCorrect: false
31+
explanation: "Incorrect. Fluency evaluates linguistic correctness and natural language quality, not whether responses are grounded in context."
32+
- content: "Groundedness"
33+
isCorrect: true
34+
explanation: "Correct. Groundedness determines whether responses are based on provided context rather than speculation or the model's general knowledge."
35+
- content: "Coherence"
36+
isCorrect: false
37+
explanation: "Incorrect. Coherence assesses whether responses flow logically and maintain consistent ideas, not whether they're based on provided context."
38+
- content: "What type of model should you select if your application needs to process both text and images?"
39+
choices:
40+
- content: "Small Language Model (SLM)"
41+
isCorrect: false
42+
explanation: "Incorrect. SLMs are efficient text-focused models but don't inherently process images alongside text."
43+
- content: "Embedding model"
44+
isCorrect: false
45+
explanation: "Incorrect. Embedding models convert text into numerical representations for semantic search and similarity tasks, not for processing images."
46+
- content: "Multimodal model"
47+
isCorrect: true
48+
explanation: "Correct. Multimodal models like GPT-4o and Phi-3-vision can process multiple data types including both text and images."
Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
### YamlMime:ModuleUnit
2+
uid: learn.wwl.model-catalog-evaluate.summary
3+
title: Summary
4+
metadata:
5+
title: Summary
6+
description: Review key concepts for selecting, deploying, and evaluating models in Microsoft Foundry portal.
7+
ms.date: 02/13/2026
8+
author: ivorb
9+
ms.author: berryivor
10+
ms.topic: unit
11+
ai-usage: ai-assisted
12+
durationInMinutes: 3
13+
content: |
14+
[!include[](includes/8-summary.md)]
Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
Building effective generative AI applications requires selecting the right foundation model for your specific use case. With thousands of models available, you need a structured approach to discover, compare, deploy, and validate that a model meets your requirements.
2+
3+
Consider a scenario where you're building an AI-powered customer support chatbot for a retail company. You need to select a language model that can understand customer questions, provide accurate responses, and maintain appropriate tone and safety standards. But how do you choose from the vast catalog of available models? How do you know if a model performs well for your specific needs? And once deployed, how do you measure and improve its performance?
4+
5+
The Microsoft Foundry portal provides a comprehensive platform for this entire workflow. You can explore over 1,900 models from providers like Microsoft, Anthropic, OpenAI, Meta, and Hugging Face. You can compare models using industry-standard benchmarks for quality, safety, cost, and performance. After selecting a model, you deploy it to an endpoint where your application can consume it. Finally, you evaluate the model's performance using both automated metrics and manual testing to ensure it meets your quality and safety requirements.
6+
7+
In this module, you explore how to use the Microsoft Foundry portal to select, deploy, and evaluate models from the model catalog. You learn how to make informed decisions about model selection, understand different deployment options, and assess model performance using various evaluation approaches.
8+
9+
By the end of this module, you'll be able to:
10+
11+
- Explore and filter models in the model catalog
12+
- Compare models using benchmark metrics for quality, safety, cost, and performance
13+
- Deploy a model to an endpoint and test it in the playground
14+
- Evaluate model performance using manual and automated approaches
15+
- Understand different evaluation metrics and when to use them
Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
The model catalog in Microsoft Foundry portal serves as your central hub for discovering and comparing AI models. With over 1,900 models available from various providers, you need effective ways to filter and find models that match your specific requirements.
2+
3+
## Access the model catalog
4+
5+
You access the model catalog from the Microsoft Foundry portal at [ai.azure.com](https://ai.azure.com). After signing in and selecting your project, choose **Model catalog** from the left navigation pane. The catalog displays model cards showing key information about each model, including the provider, capabilities, and deployment options.
6+
7+
:::image type="content" source="../media/model-catalog.png" alt-text="Screenshot of the model catalog in Microsoft Foundry portal.":::
8+
9+
## Filter models by key attributes
10+
11+
The model catalog provides several filters to help you narrow your search:
12+
13+
**Collection** filters let you browse models by provider, such as Azure OpenAI, Meta, Mistral, Cohere, or Hugging Face. This helps when you have preferences or requirements for specific model families.
14+
15+
**Industry** filters show models trained on industry-specific datasets. These specialized models often outperform general-purpose models in their respective domains.
16+
17+
**Capabilities** filters highlight unique model features. You can filter for reasoning capabilities (complex problem-solving), tool calling (API and function integration), or multimodal processing (text, images, audio).
18+
19+
**Deployment options** filters help you find models that support your preferred deployment type:
20+
- Serverless API for pay-per-call flexibility
21+
- Provisioned deployment for consistent, high-volume workloads
22+
- Managed compute for virtual machine-based hosting
23+
- Batch processing for cost-optimized, non-latency-sensitive jobs
24+
25+
**Inference tasks** and **Fine-tune tasks** filters let you find models suited for specific activities like text generation, summarization, translation, or entity extraction.
26+
27+
**License** filters help you identify models that align with your organization's licensing requirements and usage policies.
28+
29+
## Understand model types
30+
31+
As you explore the catalog, you encounter different categories of models designed for various use cases.
32+
33+
### Large Language Models and Small Language Models
34+
35+
**Large Language Models (LLMs)** like GPT-4, Mistral Large, and Llama 3 70B are powerful models designed for tasks requiring deep reasoning, complex content generation, and extensive context understanding. These models excel at sophisticated applications but require more computational resources.
36+
37+
**Small Language Models (SLMs)** like Phi-3, Mistral OSS models, and Llama 3 8B offer efficiency and cost-effectiveness while handling common natural language processing tasks. They're ideal for scenarios where speed and cost matter more than handling the most complex reasoning tasks. SLMs can run on lower-end hardware or edge devices.
38+
39+
### Chat completion and reasoning models
40+
41+
Most language models in the catalog are **chat completion** models designed to generate coherent, contextually appropriate text responses. These models power conversational interfaces and content generation applications.
42+
43+
For scenarios requiring higher performance in complex tasks like mathematics, coding, science, strategy, and logistics, **reasoning models** like DeepSeek-R1 provide enhanced problem-solving capabilities. These models can break down complex problems and show their reasoning process.
44+
45+
### Multimodal models
46+
47+
Beyond text-only processing, **multimodal models** like GPT-4o and Phi-3-vision can handle multiple data types including images, audio, and text. Use these models when your application needs to analyze visual content, such as document understanding, image description, or chart explanation.
48+
49+
### Specialized models
50+
51+
The catalog also includes task-specific models:
52+
53+
**Image generation models** like DALL·E 3 create visual content from text descriptions. Use these for generating marketing materials, illustrations, or design mockups.
54+
55+
**Embedding models** like Ada and Cohere convert text into numerical representations. These models enable semantic search, recommendation systems, and Retrieval Augmented Generation (RAG) scenarios where you need to find relevant information based on meaning rather than exact keyword matches.
56+
57+
### Regional and domain-specific models
58+
59+
Some models are optimized for specific languages, regions, or industries. When you need specialized performance in a particular domain or language, these models often outperform general-purpose alternatives. Examples include models trained on medical literature, legal documents, or specific language corpora.
60+
61+
## Use the search and compare features
62+
63+
Beyond filters, the model catalog offers search functionality to find models by name or keywords. You can open multiple model cards to compare their specifications, benchmarks, and capabilities side by side. This comparison helps you make informed decisions about which model best fits your use case, budget, and performance requirements.
64+
65+
When you identify promising candidates, you can view detailed benchmark results, test models in the playground, or proceed directly to deployment. The structured approach of filtering, comparing, and testing helps ensure you select the right model for your generative AI application.

0 commit comments

Comments
 (0)