Skip to content

docs: AI Fabric API documentation update #245

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
karkarl wants to merge 2 commits into
base: docs
Choose a base branch
from
Draft

docs: AI Fabric API documentation update #245

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b650363
docs: update App Content Search for AppContentIndexer API changes
karkarl Mar 2, 2026
a7e3bdf
Update AppContentIndex docs for Preview 1 API changes
karkarl Mar 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
130 changes: 122 additions & 8 deletions docs/apis/app-content-search-tutorial.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: Get Started with App Content Search in the Windows App SDK
description: Tutorial showing how to use the Windows AI AppContentIndexer API in the Windows App SDK to add AI-enhanced search capabilities based on semantic meaning and intent to your Windows app.
ms.topic: article
ms.date: 11/17/2025
ms.date: 03/03/2026
---

# Get Started with App Content Search
Expand All @@ -17,7 +17,10 @@ Specifically, you will learn how to use the [AppContentIndexer](/windows/windows
> - Add text strings to the index and then run a query
> - Manage long text string complexity
> - Index image data and then search for relevant images
> - Wait for indexing to complete
> - Use content regions for multi-region items
> - Enable RAG (Retrieval-Augmented Generation) scenarios
> - Use query sessions for interactive search
> - Use AppContentIndexer on a background thread
> - Close AppContentIndexer when no longer in use to release resources

Expand Down Expand Up @@ -97,21 +100,53 @@ This sample demonstrates how to add some text strings to the index created for y
foreach (var match in textMatches)
{
Console.WriteLine(match.ContentId);
if (match.ContentKind == QueryMatchContentKind.AppManagedText)
if (match is AppManagedTextQueryMatch textResult)
{
AppManagedTextQueryMatch textResult = (AppManagedTextQueryMatch)match;
// Only part of the original string may match the query. So we can use TextOffset and TextLength to extract the match.
// In this example, we might imagine that the substring "Cats are cute and fluffy" from "item1" is the top match for the query.
string matchingData = simpleTextData[match.ContentId];
string matchingString = matchingData.Substring(textResult.TextOffset, textResult.TextLength);
Console.WriteLine(matchingString);
}
else if (match is AppManagedOcrTextQueryMatch ocrResult)
{
// OCR text was found in an indexed image matching the query.
Console.WriteLine($"OCR match in image '{match.ContentId}': '{ocrResult.Fragment}'");
Console.WriteLine($" Image subregion: {ocrResult.Subregion}");
}
}
}
```

`QueryMatch` includes only `ContentId` and `TextOffset`/`TextLength`, not the matching text itself. It is your responsibility as the app developer to reference the original text. Query results are sorted by relevancy, with the top result being most relevant. Indexing occurs asynchronously, so queries may run on partial data. You can check the indexing status as outlined below.

## Wait for indexing to complete

Indexing happens asynchronously after you call `AddOrUpdate`. If you need to ensure all content has been indexed before querying, use `WaitForIndexingIdleAsync`:

```csharp
public async Task IndexAndWaitSample()
{
AppContentIndexer indexer = GetIndexerForApp();

// Add content to the index
foreach (var item in simpleTextData)
{
var textContent = AppManagedIndexableAppContent.CreateFromString(item.Key, item.Value);
indexer.AddOrUpdate(textContent);
}

// Wait for indexing to finish (with a 30-second timeout)
await indexer.WaitForIndexingIdleAsync(TimeSpan.FromSeconds(30));

// Now all content is indexed and queries will return complete results
AppIndexTextQuery query = indexer.CreateTextQuery("Facts about kittens.");
IReadOnlyList<TextQueryMatch> matches = query.GetNextMatches(5);
}
```

You can also monitor the indexing status of individual content items using `ContentItemStatusResult` and `ContentRegionStatusResult` to track whether specific items have been fully indexed.

## Manage long text string complexity

The sample demonstrates that it is not necessary for the app developer to divide the text content into smaller sections for model processing. The **AppContentIndexer** manages this aspect of complexity.
Expand Down Expand Up @@ -209,20 +244,45 @@ This sample demonstrates how to index image data as `SoftwareBitmaps` and then s
    foreach (var match in imageMatches)
    {
        Console.WriteLine(match.ContentId);
        if (match.ContentKind == QueryMatchContentKind.AppManagedImage)
        if (match is AppManagedImageQueryMatch imageResult)
        {
            AppManagedImageQueryMatch imageResult = (AppManagedImageQueryMatch)match;
            var matchingFileName = imageFilesToIndex[match.ContentId];

            // It might be that the match is at a particular region in the image. The result includes
            // the subregion of the image that includes the match.
            // It might be that the match is at a particular region in the image. The result may include
            // a rect of a region of interest that was the source of the match.

            Console.WriteLine($"Matching file: '{matchingFileName}' at location {imageResult.Subregion}");
            Console.WriteLine($"Matching file: '{matchingFileName}' at location {imageResult.RegionOfInterest}");
        }
    }
}
```

## Use content regions for multi-region items

If a content item contains multiple regions of different types (for example, a document with both text paragraphs and embedded images), you can use `AppIndexContentRegion` to index them together under a single content identifier.

```csharp
public void ContentRegionSample()
{
AppContentIndexer indexer = GetIndexerForApp();

// Create regions for a document that has both text and an image
var textRegion = AppIndexContentRegion.CreateTextRegion(
"This document describes the lifecycle of butterflies.");

var softwareBitmap = Helpers.GetSoftwareBitmapFromFile("Butterfly.jpg");
var imageRegion = AppIndexContentRegion.CreateImageRegion(softwareBitmap);

// Index the item with multiple regions under one content ID
IndexableAppContent content = AppManagedIndexableAppContent.CreateFromContentRegions(
"doc1",
new List<AppIndexContentRegion> { textRegion, imageRegion });
indexer.AddOrUpdate(content);
}
```

When you query this index, matches may come from any region within the content item. Use the match type to determine which region produced the result.

## Enable RAG (Retrieval-Augmented Generation) scenarios

RAG (Retrieval-Augmented Generation) involves augmenting user queries to language models with additional relevant data that the model can use for generating responses. The user's query serves as input for semantic search, which identifies pertinent information in an index. The resulting data from the semantic search is then incorporated into the prompt given to the language model so that it can generate more accurate and context-aware responses.
Expand Down Expand Up @@ -270,6 +330,60 @@ To enable RAG scenarios with the **AppContentIndexer** API, you can follow this
}
```

## Query Sessions for interactive search

In many cases you want to submit a single query to the index and get results. But there are cases where the query string may get updated before the query has completed, such as an interactive UI where the user is typing into a search box and results are updated as the user types.

For these cases, use a query session (`AppIndexTextQuerySession` or `AppIndexImageQuerySession`). A query session executes a query for a given query string that can be updated at any point. Any new results reflect the updated query string. It is not necessary to explicitly cancel outstanding queries—the session handles that automatically.

```csharp
AppIndexTextQuerySession querySession;

public void StartQuerySession()
{
AppContentIndexer indexer = GetIndexerForApp();

querySession = indexer.CreateTextQuerySession();
querySession.DesiredMatchesPerResult = 8;
querySession.ResultChanged += QuerySession_ResultChanged;
querySession.Start();

SearchTextBox.TextChanged += SearchBoxTextChanged;
SearchTextBox.QuerySubmitted += SearchBox_QuerySubmitted;
}

public void SearchBoxTextChanged(AutoSuggestBox sender, AutoSuggestBoxTextChangedEventArgs args)
{
querySession.UpdateQueryPhrase(sender.Text);
}

public void QuerySession_ResultChanged(AppIndexTextQuerySession sender, Object args)
{
// This event is raised on a background thread, so dispatch UI work to the UI thread.
this.DispatcherQueue.TryEnqueue(() =>
{
if (querySession != null)
{
TextQuerySessionResult result = querySession.GetResult();
IReadOnlyList<TextQueryMatch> matches = result.Matches;
DisplaySearchResultsInUI(matches);
}
});
}

private void SearchBox_QuerySubmitted(AutoSuggestBox sender, AutoSuggestBoxQuerySubmittedEventArgs args)
{
var chosenMatch = args.ChosenSuggestion as TextQueryMatch;

// Optionally provide the chosen match so the system can improve future results.
querySession.Stop(chosenMatch);
querySession.Dispose();
querySession = null;
}
```

You can also create an image query session following the same pattern using `indexer.CreateImageQuerySession()`.

## Use AppContentIndexer on a background thread

An **AppContentIndexer** instance is not associated with a particular thread; it is an agile object that can operate across threads. Certain methods of **AppContentIndexer** and its related types may require considerable processing time. Therefore, it is advisable to avoid invoking **AppContentIndexer** APIs directly from the application's UI thread and instead use a background thread.
Expand Down
24 changes: 21 additions & 3 deletions docs/apis/app-content-search.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: App Content Search Overview
description: Learn how App Content Search and the Windows AI AppContentIndexer API can enhance your Windows app search capabilities using AI to search based on semantic meaning and intent.
ms.topic: article
ms.date: 11/17/2025
ms.date: 03/03/2026
---

# App Content Search Overview
Expand All @@ -15,13 +15,30 @@ Use this API to:

- Support Retrieval-Augmented Generation (RAG) by enabling local knowledge retrieval. When paired with a Large Language Model (LLM), this allows you to retrieve the most relevant content from your app's knowledge base and generate more accurate, context-aware responses.

The ApplicationContentIndexer API is currently only available in Windows App SDK release 2.0 Experimental 4.
The ApplicationContentIndexer API is available starting in Windows App SDK 2.0 Preview 1.

> [!div class="nextstepaction"]
> [Open AI Dev Gallery to try App Content Search](aidevgallery://apis/f8465a45-8e23-4485-8c16-9909e96eacf6)

The AI Dev Gallery app offers an interactive sample of the AppContentIndexer API enabling you to experiment with the App Content Search feature. [Learn more about the AI Dev Gallery](../ai-dev-gallery/index.md), including how to install from the Microsoft Store or from the source code on GitHub.

## What's new in Preview 1

The AppContentIndexer API has been updated since the experimental release with several improvements and new features:

- **Namespace change**: `Microsoft.Windows.AI.Search.Experimental.AppContentIndex` has moved to `Microsoft.Windows.Search.AppContentIndex`.
- **Separate query types**: `AppIndexQuery` has been split into `AppIndexTextQuery` and `AppIndexImageQuery` for type safety.
- **Unified result retrieval**: `GetNextTextMatches()` and `GetNextImageMatches()` have been replaced with a single `GetNextMatches()` method.
- **Pattern matching**: Use `match is AppManagedTextQueryMatch` instead of checking `match.ContentKind`.
- **OCR text matching**: Text queries can now return `AppManagedOcrTextQueryMatch` results when indexed images contain matching text, including the `Fragment` and `Subregion` of the match.
- **Wait for indexing**: Use `WaitForIndexingIdleAsync(timeout)` to wait until all content has been indexed before querying.
- **Query sessions**: `CreateTextQuerySession()` and `CreateImageQuerySession()` enable live search-as-you-type experiences.
- **Content regions**: `AppIndexContentRegion` allows you to combine text and image data in a single content item.
- **Index capabilities**: Use `IndexCapabilitiesOfCurrentSystem` and `IndexCapabilities` to check whether semantic search is available on the current device.
- **Status monitoring**: `ContentItemStatusResult` and `ContentRegionStatusResult` let you track the indexing status of individual items.

For code examples demonstrating these features, see the [App Content Search tutorial](app-content-search-tutorial.md).

## What is the AppContentIndexer API?

The **AppContentIndexer API** allows apps to make their text and image content searchable using both keyword-based (lexical) and meaning-based (semantic) search—without requiring developers to understand the underlying complexity.
Expand Down Expand Up @@ -52,12 +69,13 @@ ApplicationContentIndexer supports adding the following types of content:

- **Text** – plain or structured text content.
- **Images** – including screenshots, photos, or image files that contain text or recognizable visual elements.
- **OCR text from images** – text recognized within indexed images can be returned as `AppManagedOcrTextQueryMatch` results, including the matching text fragment and image subregion.

### App-defined content identifiers

**AppContentIndexer** supports app-managed content by allowing apps to index items using app-defined content identifiers. Queries return these identifiers, which the app uses to retrieve the actual content from its own data store.

Text queries return AppManagedTextQueryMatch objects, and image queries return AppManagedImageQueryMatch objects—both include only the ContentId, not the content itself.
Text queries return `AppManagedTextQueryMatch` objects, and image queries return `AppManagedImageQueryMatch` objects—both include only the `ContentId`, not the content itself. Text queries can also return `AppManagedOcrTextQueryMatch` objects when indexed images contain matching text.

For guidance on how to integrate this feature into your app and use the ApplicationContentIndexer API, see: [Quickstart: App Content Search](app-content-search-tutorial.md)

Expand Down