[Feature] GALLEY — model-agnostic local media generation engine

[isaacsight]

Problem

Generative media workflows are usually locked behind provider-specific SDKs, opaque pricing, cloud-only artifacts, and tools that assume one particular model or agent.

kernel.chat needs a model-agnostic way for any capable agent to:

• discover available image, video, and speech models;
• estimate cost before generating;
• require a human decision before paid work;
• submit and monitor jobs through a stable local interface;
• retain completed media on the user's own disk; and
• hand the resulting assets to an editor without locking the workflow to one LLM vendor.

Proposed Solution

Build GALLEY, a local media-generation engine exposed as plain HTTP on loopback.

Any LLM, terminal agent, MCP client, or script that can make a web request can operate it. Provider credentials remain server-side, while callers interact with a small, documented API.

Architecture

Any agent or script
        |
        | HTTP on 127.0.0.1
        v
GALLEY Creative Canvas
        |
        +-- :5411 local mflux image generation (free)
        |
        +-- :5412 fal proxy
              +-- curated model registry and normalized prices
              +-- full model catalog
              +-- video, premium image, and speech generation
              +-- queue polling and local artifact download
              +-- configurable daily spend cap

The implementation is currently developed on feat/galley-video-engine.

Key files:

• docs/ENGINE.md — cross-model operator contract
• tools/video-models.mjs — curated registry, pricing, and result extraction
• tools/local-video-server.mjs — fal queue proxy and artifact ownership
• tools/local-image-server.mjs — local mflux image generation
• tools/spend-tracker.mjs — daily spend accounting
• tools/engine.mjs — one-command launcher
• src/pages/CreativeCanvasPage.tsx — confirmation UI, catalog, and refinement flow

Safety Contract

Paid routes must follow this sequence:

1. Fetch an estimate or published catalog price.
2. Present the per-item price, quantity, and batch total.
3. Wait for explicit human approval of that total.
4. Submit only the approved work.
5. Require new approval for retries or expanded batches.

The server also enforces FAL_DAILY_SPEND_LIMIT (default: $10 per local day) and keeps FAL_KEY out of the browser and caller process.

The daily cap is a backstop, not a substitute for approval. It tracks estimated submission cost and may differ from the provider's final invoice.

Example Usage

Start the engine:

npm run engine

Inspect and estimate without spending:

curl -sS http://127.0.0.1:5412/v1/models

curl -sS -X POST http://127.0.0.1:5412/v1/videos/estimate \
  -H 'Content-Type: application/json' \
  -d '{"model":"seedance-lite","durationSeconds":5}'

After explicit approval, submit and poll:

curl -sS -X POST http://127.0.0.1:5412/v1/videos/generations \
  -H 'Content-Type: application/json' \
  -d '{"prompt":"A paper boat drifting downstream","model":"seedance-lite","durationSeconds":5}'

curl -sS http://127.0.0.1:5412/v1/videos/jobs/JOB_ID

Completed files are downloaded to output/videos/, output/images/, or output/audio/.

Acceptance Criteria

• npm run engine starts both services and stops them together.
• Both services bind only to 127.0.0.1.
• Provider credentials never reach the browser or API caller.
• Agents can inspect health, curated models, and catalog entries for free.
• Curated video and speech estimates are deterministic and tested.
• Unknown catalog pricing is surfaced as unknown rather than fabricated.
• The Creative Canvas shows a cost-confirmation gate before paid work.
• The HTTP layer enforces a configurable daily spend cap.
• Paid routes return job IDs that can be polled to completion or error.
• Completed artifacts are downloaded to locally owned files.
• Image-to-video chaining can use the provider-side source URL while retaining a local copy.
• A cold-start agent can operate the full workflow from docs/ENGINE.md.
• Model and pricing logic remains covered by automated tests.

Follow-up Work

• Persist job metadata across server restarts.
• Reconcile estimated spend with final provider billing where the provider exposes it.
• Add a first-class premium-image estimate route.
• Make approval evidence enforceable at the HTTP layer rather than relying only on caller discipline.
• Define a provider-adapter interface beyond fal without changing the public API.
• Add structured batch submission with one approved total and bounded retries.
• Export editing timelines through Palmier/FCPXML as a documented downstream workflow.

Alternatives Considered

• Provider SDKs in each client: leaks vendor concerns into every agent and risks exposing credentials.
• Browser-to-provider calls: makes secret handling and spend control substantially weaker.
• MCP-only generation: useful for some clients, but plain HTTP is a smaller interoperability baseline.
• Cloud-hosted artifacts: convenient, but violates the local-ownership contract and creates avoidable lock-in.

Additional Context

The complete pipeline has already produced a 59-second film using premium stills, image-to-video motion, ElevenLabs narration, and a Palmier edit. The production record is in docs/video/2026-07-18-what-we-made.md; generated media remains untracked under output/.

This issue should serve as the public project tracker and architectural contract for bringing GALLEY into kernel.chat.

Palmier Pro VFX Toolkit

GALLEY should extend beyond generation into a non-destructive VFX layer for Palmier Pro. Palmier remains the timeline and render authority; GALLEY supplies higher-level, model-readable recipes that compile into Palmier's native MCP editing tools.

V1: Live, Editable VFX Presets

A versioned preset schema should compose Palmier's existing primitives:

• Looks: grain, glow, vignette, clarity, dehaze, LUTs, curves, and color wheels.
• Cleanup: sharpening, noise reduction, exposure correction, and reference-based color matching.
• Compositing: chroma key, overlays, picture-in-picture, split screens, and track stacking.
• Motion treatments: animated opacity, transform, crop, blur, and authored text.
• Verification: composited-frame inspection plus numerical color-scope comparison.

Example:

{
  "name": "Warm 16mm Dream",
  "effects": [
    {
      "type": "stylize.grain",
      "params": { "amount": 0.22, "size": 1.8 }
    },
    {
      "type": "stylize.glow",
      "params": { "intensity": 0.16, "radius": 24, "warmth": 0.3 }
    },
    {
      "type": "stylize.vignette",
      "params": { "amount": -0.18, "feather": 0.75 }
    }
  ],
  "color": {
    "temperature": 6900,
    "contrast": 0.94,
    "saturation": 0.88
  }
}

VFX Workflow

1. Duplicate the active timeline as a safe VFX version.
2. Apply a named preset as live, editable effects.
3. Animate supported properties when the treatment calls for motion.
4. Inspect representative composited frames and color scopes.
5. Refine, bypass, or remove effects non-destructively.
6. Export through Palmier's normal video/FCPXML pipeline.

The preset compiler should map semantic recipes onto Palmier MCP operations such as apply_effect, apply_color, set_keyframes, apply_layout, add_texts, inspect_timeline, and inspect_color.

VFX V1 Acceptance Criteria

• Define a documented, versioned JSON preset schema.
• Ship 12–20 polished presets covering looks, cleanup, compositing, and motion treatments.
• Support a global intensity control without destroying the underlying preset values.
• Apply presets as live Palmier effects rather than baked replacement media.
• Preserve undoability and allow individual effects to be bypassed or removed.
• Duplicate the timeline before broad or destructive-looking treatments.
• Verify results using representative rendered frames and color scopes.
• Copy a treatment consistently across multiple selected clips.
• Keep preset compilation deterministic and covered by tests.

Advanced VFX Boundary

Palmier's current MCP surface does not expose masks, motion tracking, rotoscoping, blend modes, track mattes, displacement, particles, or optical-flow controls. Those capabilities require one of two explicit paths:

1. add native Palmier primitives and expose them through MCP; or
2. render the effect through an external compositor and import the baked result.

Prefer native, editable effects. Use a baked external-render path only when the requested result cannot be represented by Palmier's live effect stack.

Future advanced work:

• masks with feathering and keyframes;
• point/planar motion tracking;
• subject-aware rotoscoping;
• blend modes and track mattes;
• displacement and distortion;
• particle and procedural generators;
• optical-flow retiming and motion interpolation;
• preset preview thumbnails and before/after comparison UI.