Skip to content

multimodal-overview.md

Latest commit

 

History

History
733 lines (519 loc) · 22.7 KB

multimodal-overview.md

File metadata and controls

733 lines (519 loc) · 22.7 KB
title Use multimodal input with AI functions
description Learn how to use file-based multimodal input, such as images, PDFs, and text files, with AI functions in Microsoft Fabric.
ms.reviewer vimeland
ms.topic how-to
ms.date 02/18/2026
ms.search.form AI functions
ai-usage ai-assisted

Use multimodal input with AI functions (Preview)

[!INCLUDE preview-note]

AI functions can process files—images, PDFs, and text files—in addition to text. With multimodal input, you can classify documents, summarize PDFs, extract information from images, and more, all with the same AI function interface you already use for text.

Multimodal input works with these existing AI functions:

Multimodal input also introduces three new functions:

Supported file types

Multimodal AI functions support the following file types:

  • Images: jpg, jpeg, png, static gif, webp
  • Documents: pdf
  • Text files: md, txt, csv, tsv, json, xml, py, and other text files

Note

  • Office file formats (such as .docx, .pptx, and .xlsx) aren't currently supported.
  • You can convert .docx and .pptx files to PDF and .xlsx files to CSV before using them with multimodal AI functions.
  • Each input file is limited to 50 MB in size.

Supported URL protocols

Multimodal inputs are provided as string with one of the following URL protocols:

  • local file paths
  • http(s)
  • wasbs
  • abfs(s)

Prerequisites

Multimodal AI functions share the same prerequisites as text-based AI functions. For the full list, see Prerequisites.

Set up your files

Organize your files in a folder that can be referenced by a path or a glob-style string.

Tip

Starter notebooks are available in Fabric that demonstrate end-to-end multimodal scenarios, including image, PDF, and text ingestion, schema inference, and extraction. Look for the AI functions starter notebooks in your Fabric workspace to accelerate setup and usage.

Example

You can store files in a Lakehouse attached to your notebook.

pandas

folder_path = "/lakehouse/default/Files"

PySpark

folder_path = "abfss://path_to_your_lakehouse"
# or
folder_path = "Files"

Load your files

To use AI functions with multimodal input, you may either load the file contents into a structured table or reference the file paths directly in your DataFrame. The following examples show both approaches.

Load files into a table

Use the aifunc.load function to read files from a folder and generate a structured table. The function can infer the table structure on its own, or you can provide a prompt to guide the extraction, or a schema for consistent structure. This approach is useful when you want the AI to extract specific information from the files and present it in a structured format.

pandas

df, schema = aifunc.load(folder_path)
# or
df, schema = aifunc.load(folder_path, prompt="Give me candidate's name and the most recent company they worked for.")
display(df)

PySpark

df, schema = aifunc.load(folder_path)
# or
df, schema = aifunc.load(folder_path, prompt="Give me candidate's name and the most recent company they worked for.")
display(df)

Load file paths into a column

Alternatively, you can use aifunc.list_file_paths to get a list of file paths from a folder and load them into a DataFrame column. This approach is useful when you want to run AI functions across each file.

Note

Most AI functions accept file-path inputs via column_type="path" (pandas) or input_col_type/col_types="path" (PySpark). This applies uniformly across all the multimodal functions listed in this article.

pandas

file_path_series = aifunc.list_file_paths(folder_path)
df = pd.DataFrame({"file_path": file_path_series}).reset_index(drop=True)
display(df)

PySpark

df = aifunc.list_file_paths(folder_path)
display(df)

Important

When your file paths are stored as string URLs in a DataFrame column, you must explicitly tell the AI function to treat the values as file paths rather than plain text.

pandas

For Series-level AI functions (operating on a single column), set the column_type parameter:

df["result"] = df["file_path"].ai.analyze_sentiment(column_type="path")

For DataFrame-level AI functions (operating on multiple columns), use the column_type_dict parameter:

df["result"] = df.ai.generate_response(
    prompt="Describe the content.",
    column_type_dict={"file_path": "path"},
)

Note

If you use aifunc.list_file_paths() to create your file path column, the returned yarl.URL objects are automatically detected as file paths. You only need to specify column_type="path" when your column contains plain string URLs.

PySpark

For single-column AI functions, set the input_col_type parameter:

results = df.ai.analyze_sentiment(input_col="file_path", input_col_type="path", output_col="sentiment")

For DataFrame-level AI functions (operating on multiple columns), use the col_types parameter:

results = df.ai.generate_response(
    prompt="Describe the content.",
    col_types={"file_path": "path"},
    output_col="response",
)

New multimodal functions

Load files into a table with aifunc.load

The aifunc.load function reads all files from a folder path and generates a structured table from their contents. You can optionally provide a prompt to guide the extraction, or a schema for consistent structure.

Syntax

pandas

df, schema = aifunc.load(folder_path, prompt=None, schema=None)

PySpark

df, schema = aifunc.load(folder_path, prompt=None, schema=None)

Parameters

Name Description
folder_path
Required		 A string path to a folder or a glob-style pattern matching files.
prompt
Optional		 A string that guides the table generation process. Use it to specify which fields to extract from the files.
schema
Optional		 A schema object (returned by a previous load call) that defines the table structure. When provided, the function uses this schema directly.

Returns

A tuple of (DataFrame, schema). The DataFrame contains the structured data extracted from the files. The schema can be reused in subsequent load calls for consistent results.

Example

pandas

# This code uses AI. Always review output for mistakes.

# Basic load – let the AI infer the table structure
df, schema = aifunc.load(folder_path)
display(df)
# This code uses AI. Always review output for mistakes.

# Guided load – provide a prompt to specify what to extract
guided_df, guided_schema = aifunc.load(
    folder_path,
    prompt="Give me candidate's name and the most recent company they worked for.",
)
display(guided_df)

PySpark

# This code uses AI. Always review output for mistakes.

# Basic load – let the AI infer the table structure
df, schema = aifunc.load(folder_path)
display(df)
# This code uses AI. Always review output for mistakes.

# Guided load – provide a prompt to specify what to extract
guided_df, guided_schema = aifunc.load(
    folder_path,
    prompt="Give me candidate's name and the most recent company they worked for.",
)
display(guided_df)

List files with aifunc.list_file_paths

The aifunc.list_file_paths function fetches all valid file paths from a specified folder. You can use the returned file paths as input to any multimodal AI function. The function also supports glob-style patterns.

Syntax

pandas

file_path_series = aifunc.list_file_paths(folder_path)

PySpark

file_path_df = aifunc.list_file_paths(folder_path)

Parameters

Name Description
folder_path
Required		 A string path to a folder or a glob-style pattern matching files.

Returns

pandas

A pandas Series of yarl.URL objects, indexed by their string representations. These yarl.URL objects are automatically treated as file paths by AI functions, so you don't need to specify column_type="path".

PySpark

A Spark DataFrame with a file_path column containing the URLs of the matched files.

Example

pandas

# This code uses AI. Always review output for mistakes.

file_path_series = aifunc.list_file_paths(folder_path)
custom_df = pd.DataFrame({"file_path": file_path_series}).reset_index(drop=True)
display(custom_df)

PySpark

# This code uses AI. Always review output for mistakes.

custom_df = aifunc.list_file_paths(folder_path)
display(custom_df)

Infer schema from files with ai.infer_schema

The ai.infer_schema function infers a common schema from file contents. The inferred schema is represented as a list of aifunc.ExtractLabel objects that you can pass directly to ai.extract for structured data extraction.

Syntax

pandas

schema = df["file_path"].ai.infer_schema(column_type="path")

PySpark

schema = df.ai.infer_schema(input_col="file_path", input_col_type="path")

Parameters

pandas

Name Description
prompt
Optional		 A string to guide schema inference. If not provided, the function infers the schema from the file contents alone.
n_samples
Optional		 An integer specifying how many items to sample for inference. Default is 3.
column_type
Optional		 Set to "path" to treat column values as file paths.

PySpark

Name Description
input_col
Required		 A string that contains the name of the column with file path values.
input_col_type
Optional		 Set to "path" to treat column values as file paths.
prompt
Optional		 A string to guide schema inference. If not provided, the function infers the schema from the file contents alone.
n_samples
Optional		 An integer specifying how many items to sample for inference. Default is 3.

Returns

A list of aifunc.ExtractLabel objects that describe the inferred schema. You can pass this list to ai.extract to extract structured data from files.

Example

pandas

# This code uses AI. Always review output for mistakes.

# Infer a schema from file contents
schema = df["file_path"].ai.infer_schema(column_type="path")
for label in schema:
    print(label)

# Use the inferred schema with ai.extract
extracted_df = df["file_path"].ai.extract(*schema, column_type="path")
display(extracted_df)

PySpark

# This code uses AI. Always review output for mistakes.

# Infer a schema from file contents
schema = df.ai.infer_schema(input_col="file_path", input_col_type="path")
for label in schema:
    print(label)

# Use the inferred schema with ai.extract
extracted_df = df.ai.extract(labels=schema, input_col="file_path", input_col_type="path")
display(extracted_df)

Use multimodal input with existing AI functions

The following examples show how to use multimodal input with each of the supported AI functions.

Detect sentiment from files with ai.analyze_sentiment

For more information about ai.analyze_sentiment, see the detailed documentation for pandas and PySpark.

pandas

# This code uses AI. Always review output for mistakes.

animal_urls = [
    "<image-url-golden-retriever>",  # Replace with URL to an image of a golden retriever
    "<image-url-giant-panda>",  # Replace with URL to an image of a giant panda
    "<image-url-bald-eagle>",  # Replace with URL to an image of a bald eagle
]
animal_df = pd.DataFrame({"file_path": animal_urls})

animal_df["sentiment"] = animal_df["file_path"].ai.analyze_sentiment(column_type="path")
display(animal_df)

PySpark

# This code uses AI. Always review output for mistakes.

animal_urls = [
    "<image-url-golden-retriever>",  # Replace with URL to an image of a golden retriever
    "<image-url-giant-panda>",  # Replace with URL to an image of a giant panda
    "<image-url-bald-eagle>",  # Replace with URL to an image of a bald eagle
]
animal_df = spark.createDataFrame([(u,) for u in animal_urls], ["file_path"])

results = animal_df.ai.analyze_sentiment(
    input_col="file_path",
    input_col_type="path",
    output_col="sentiment",
)
display(results)

Classify files with ai.classify

For more information about ai.classify, see the detailed documentation for pandas and PySpark.

pandas

# This code uses AI. Always review output for mistakes.

custom_df["highest_degree"] = custom_df["file_path"].ai.classify(
    "Master", "PhD", "Bachelor", "Other",
    column_type="path",
)
display(custom_df)

PySpark

# This code uses AI. Always review output for mistakes.

results = custom_df.ai.classify(
    labels=["Master", "PhD", "Bachelor", "Other"],
    input_col="file_path",
    input_col_type="path",
    output_col="highest_degree",
)
display(results)

Extract entities from files with ai.extract

For more information about ai.extract, see the detailed documentation for pandas and PySpark.

pandas

# This code uses AI. Always review output for mistakes.

extracted = custom_df["file_path"].ai.extract(
    aifunc.ExtractLabel(
        "name",
        description="The full name of the candidate, first letter capitalized.",
        max_items=1,
    ),
    "companies_worked_for",
    aifunc.ExtractLabel(
        "year_of_experience",
        description="The total years of professional work experience the candidate has, excluding internships.",
        type="integer",
        max_items=1,
    ),
    column_type="path",
)
display(extracted)

PySpark

# This code uses AI. Always review output for mistakes.

extracted = custom_df.ai.extract(
    labels=[
        aifunc.ExtractLabel(
            "name",
            description="The full name of the candidate, first letter capitalized.",
            max_items=1,
        ),
        "companies_worked_for",
        aifunc.ExtractLabel(
            "year_of_experience",
            description="The total years of professional work experience the candidate has, excluding internships.",
            type="integer",
            max_items=1,
        ),
    ],
    input_col="file_path",
    input_col_type="path",
)
display(extracted)

Fix grammar in files with ai.fix_grammar

For more information about ai.fix_grammar, see the detailed documentation for pandas and PySpark.

pandas

# This code uses AI. Always review output for mistakes.

custom_df["corrections"] = custom_df["file_path"].ai.fix_grammar(column_type="path")
display(custom_df)

PySpark

# This code uses AI. Always review output for mistakes.

results = custom_df.ai.fix_grammar(
    input_col="file_path",
    input_col_type="path",
    output_col="corrections",
)
display(results)

Answer custom prompts from files with ai.generate_response

For more information about ai.generate_response, see the detailed documentation for pandas and PySpark.

pandas

# This code uses AI. Always review output for mistakes.

# Series-level: generate a response from each file
animal_df["animal_name"] = animal_df["file_path"].ai.generate_response(
    prompt="What type of animal is in this image? Give me only the animal's common name.",
    column_type="path",
)
display(animal_df)
# This code uses AI. Always review output for mistakes.

# DataFrame-level: use all columns as context
animal_df["description"] = animal_df.ai.generate_response(
    prompt="Describe this animal's natural habitat and one interesting fact about it.",
    column_type_dict={"file_path": "path"},
)
display(animal_df)

PySpark

# This code uses AI. Always review output for mistakes.

# Generate a response from each file
results = animal_df.ai.generate_response(
    prompt="What type of animal is in this image? Give me only the animal's common name.",
    col_types={"file_path": "path"},
    output_col="animal_name",
)
display(results)
# This code uses AI. Always review output for mistakes.

# DataFrame-level: use all columns as context
results = animal_df.ai.generate_response(
    prompt="Describe this animal's natural habitat and one interesting fact about it.",
    col_types={"file_path": "path"},
    output_col="description",
)
display(results)

Summarize files with ai.summarize

For more information about ai.summarize, see the detailed documentation for pandas and PySpark.

pandas

# This code uses AI. Always review output for mistakes.

# Summarize file content from a single column
custom_df["summary"] = custom_df["file_path"].ai.summarize(
    instructions="Talk like a pirate! You only have one minute",
    column_type="path",
)
display(custom_df)

PySpark

# This code uses AI. Always review output for mistakes.

# Summarize file content from a single column
results = custom_df.ai.summarize(
    instructions="Talk like a pirate! You only have one minute",
    input_col="file_path",
    input_col_type="path",
    output_col="summary",
)
display(results)

You may summarize values across all columns in a DataFrame by omitting the input column and specifying file path columns with column_type_dict (pandas) or col_types (PySpark):

pandas

# This code uses AI. Always review output for mistakes.

custom_df["summary"] = custom_df.ai.summarize(
    column_type_dict={"file_path": "path"},
)
display(custom_df)

PySpark

# This code uses AI. Always review output for mistakes.

results = custom_df.ai.summarize(
    col_types={"file_path": "path"},
    output_col="summary",
)
display(results)

Translate files with ai.translate

For more information about ai.translate, see the detailed documentation for pandas and PySpark.

pandas

# This code uses AI. Always review output for mistakes.

custom_df["chinese_version"] = custom_df["file_path"].ai.translate(
    "Chinese",
    column_type="path",
)
display(custom_df)

PySpark

# This code uses AI. Always review output for mistakes.

results = custom_df.ai.translate(
    to_lang="Chinese",
    input_col="file_path",
    input_col_type="path",
    output_col="chinese_version",
)
display(results)

Evaluate output quality

Evaluation notebooks (Eval) are available for AI functions. These notebooks provide structured workflows that use LLM-as-a-Judge to assess multimodal outputs and compute metrics such as accuracy, precision, recall, F1, coherence, consistency, and relevance. You can use these workflows to validate the quality of classification, extraction, summarization, and other AI function results before moving to production.

Monitor cost and capacity usage

AI functions include a configurable progress bar cost calculator that displays real-time token estimates and Fabric capacity units while operations run in notebooks. You can configure the cost calculator in three modes:

  • basic: Shows a summary of estimated tokens and capacity units consumed.
  • stats: Shows detailed per-call statistics, including input and output token counts.
  • disable: Turns off the progress bar cost display.

For details on configuring these modes, see Configure AI functions.

The Fabric Capacity Metrics App also includes a dedicated AI Functions operation that separates AI functions usage from Spark and Dataflows Gen2. You can use this view to track multimodal AI functions consumption and identify capacity impact. For more information, see What is the Microsoft Fabric Capacity Metrics app?.

Related content