Skip to content

project-vision.md

Latest commit

 

History

History
119 lines (65 loc) · 6.05 KB

project-vision.md

File metadata and controls

119 lines (65 loc) · 6.05 KB
Raw

Capability Schema — Vision v0.1

Layer: L1 (Vision Statement) SOP: SOP-001 §2 Layer 1 Status: Draft · Date: 2026-07-01

North Star

Capability Schema does not replace world model benchmarks. It gives them a shared language.

What This Project Is

Capability Schema is a semantic standard for describing, observing, and reporting the capabilities of generative world models.

It is:

  1. A shared vocabulary. Different world models (Dreamer4, Cosmos, Genie) produce results that are not mutually comparable today. Capability Schema defines Capabilities in model-independent terms so that any World Model can be evaluated against the same semantic dimensions.

  2. A conformance standard. It defines not just what a Capability means, but what evidence is required to report it, and what tests must pass to claim "Capability Schema Conformant."

  3. An observability layer. It separates the definition of a Capability (what is being measured) from its observation (how it is measured). The same Capability can be observed through Metrics (when ground-truth is available) or estimated through Predictors (when it is not).

  4. A platform for continuous reliability monitoring. Unlike a benchmark that is run once and produces a number, Capability Schema supports ongoing evaluation — tracking how a model's Capabilities evolve across versions, domains, and deployment conditions.

What This Project Is Not

  1. Not a benchmark. It does not rank models, maintain leaderboards, or define dataset-specific evaluation protocols. Benchmarks can adopt Capability Schema; they do not compete with it.

  2. Not a training framework. It does not train models, provide training recipes, or offer model cards. It evaluates pretrained checkpoints only.

  3. Not an SDK that binds to one model architecture. The WorldModel ABC has exactly three methods (encode, decode, step). Any architecture that satisfies these three methods is a first-class citizen.

  4. Not a product company. It does not offer finetuning services, managed evaluation, or SaaS dashboards. Commercial services may be built on top of Capability Schema; the Schema itself is a public standard.

  5. Not an AI Safety framework (by default). Safety assessment may exist as a separate Capability Domain, but never in the World Model Core.

Architecture Principles

Principle 1 — Capability is the Only Stable Abstraction

All code changes. Capability definitions stay.

The Capability layer is the invariant. Adapters for new models, new Metrics, and new Predictors are all consumers of the Capability Schema — they do not define it. A Capability is defined once, in one YAML file, and all implementations converge toward it.

Verifiable by: grep -r "dreamer4\|cosmos\|genie" capabilities/ returns zero matches.

Principle 2 — Metrics Witness; They Do Not Define

A Capability exists independently of how it is measured.

temporal_stability is a semantic property of any world model. ΔPSNR is one way to observe it. Flow instability is another. Adding a new observation method never changes the Capability definition.

Verifiable by: observed_by is a list of metric/predictor names — never a computation formula. No Capability YAML contains a compute or score = field.

Principle 3 — Models are Replaceable

Any WorldModel ABC passes the same CapabilityQuery.

The WorldModel ABC defines exactly three methods: encode(observation) → latent, decode(latent) → observation, step(latent, action) → latent. Any model satisfying this contract — regardless of architecture, tokenizer design, or parameter count — can be evaluated through the same Capability pipeline.

Verifiable by: MockWorldModel (64×64 resolution, latent_dim=16) passes through the entire CapabilityQuery pipeline without crashing.

Principle 4 — Spec and Implementation are Physically Separate

The standard lives in its own repository, under its own license, with its own governance.

The Capability Schema Specification (capability-schema-spec, CC BY 4.0) defines what a Capability is. The Reference Implementation (capability-schema-reference, MIT) proves that the Spec can be implemented. No .py file in the Spec repo. No Spec authority in the Implementation repo.

Verifiable by: find capability-schema-spec/ -name "*.py" returns only conformance tests. Spec repo README does not import or require the Reference Implementation.

Principle 5 — Conformance is Testable, Not Declarative

Anyone can say "we use Capability Schema." Only those who pass the Conformance Test can say "Capability Schema Conformant."

A claim of conformance is backed by an executable test suite. The test is owned by the Spec, not the Implementation. The Reference Implementation must also pass the Conformance Test — it has no special exemption.

Verifiable by: test_conformance.py exists in the Spec repo. Running it against the Reference Implementation produces a CAPABILITY_SCHEMA_CONFORMANCE.json report.

Project Identity

Dimension Value
Formal name Capability Schema Specification
Daily name Capability Schema
In papers "We report results using the Capability Schema Specification."
Short citation "following the Capability Schema"
Repos capability-schema-spec (standard) + capability-schema-reference (implementation)
License Spec: CC BY 4.0 · Reference: MIT · Conformance Tests: MIT

Success Definition

A benchmark paper by a team unaffiliated with this project writes: "We report temporal_stability following the Capability Schema Specification."

That sentence, appearing in any peer-reviewed venue, is the moment Capability Schema transitions from "a project" to "a standard." The code serves that sentence. The governance protects that sentence. The conformance tests validate that sentence.

Capability Schema does not belong to any single model or any single paper. It defines a shared language for discussing world model reliability.