Skip to content

Promote TwelveLabsVideoParser and MarengoEmbedder from the Video RAG template into pathway.xpacks.llm #255

Description

@zxqfd555

Background

pathwaycom/llm-app#129 added a Video RAG with TwelveLabs template containing two reusable building blocks:

  • TwelveLabsVideoParser (pw.UDF) — uploads video bytes as a TwelveLabs asset and turns them into a rich text description via the Pegasus video-understanding model. Output is the standard list[(text, metadata)], so it chunks/embeds/indexes exactly like the built-in PDF parsers.
  • MarengoEmbedder (BaseEmbedder) — a retriever embedder backed by the Marengo multimodal model, returning 512-dim vectors in a shared text/image/audio/video embedding space.

Both currently live in a per-template pathway_twelvelabs package. This issue proposes promoting them into pathway.xpacks.llm as first-class, supported components.

Motivation

  • Fills a real gap in modality coverage. xpacks/llm/parsers.py already ships ImageParser and AudioParser but has no video parser. MarengoEmbedder would be the first xpacks embedder in a shared multimodal space. New modality, zero new infrastructure.
  • Already conforms to xpacks conventions. MarengoEmbedder subclasses BaseEmbedder like OpenAIEmbedder/GeminiEmbedder/BedrockEmbedder and reuses udfs.async_executor, ExponentialBackoffRetryStrategy, CacheStrategy, max_batch_size. TwelveLabsVideoParser mirrors the existing pw.UDF parsers. Minimal adaptation required.
  • Generic, not template-specific. Prompt, model, API key, retries, and caching are all parameterized; no template-specific assumptions in the classes themselves.

Proposed scope

  • Add the two components under pathway/xpacks/llm/ fold into embedders.py / parsers.py with exports in the respective __init__.
  • Keep twelvelabs an optional dependency via the existing lazy-import + ImportError guidance pattern, so plain import pathway is unaffected. Create a twelvelabs section in pyproject.toml so that the dependency can be installed via pip install pathway[twelvelabs].
  • Port the unit tests (stubbed SDK, no network) into python/pathway/xpacks/llm/tests/; keep the live smoke test gated behind TWELVELABS_API_KEY.
  • Ensure all introduced classes are properly documented.
  • Add a CHANGELOG entry (user-visible Python API).

Shortly after the components are ported into the Pathway framework and the changes are released, it makes sense to refactor the llm-app template so that it imports components directly from Pathway.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions