Wave

Wave is a versioned, extensible dataset compiler for LLM training.

Wave is not a dataset. Wave is not an exporter. Wave is the source compiler that transforms a rich, human-editable dataset into multiple machine-readable training formats.

YAML  ->  Parser  ->  AST  ->  Normalizer  ->  Validator  ->  Compiler  ->  Exporters
                                                                              |-- OpenAI JSONL
                                                                              |-- ChatML
                                                                              |-- Code-only
                                                                              |-- Instruction (Alpaca-style)
                                                                              `-- ... more to come

Philosophy

Training frameworks evolve. Models evolve. Prompt formats evolve. The dataset should not. Wave stores each task once, as a single source of truth, and compiles it deterministically into whatever format the next model or framework needs.

• Human-readable: every task is a plain YAML file.
• Git-friendly: one task per file, so history and diffs stay clean.
• Schema validated: every task is checked against a versioned Pydantic schema before it can be compiled.
• Framework agnostic: the source format never encodes a chat template, a tokenizer, or a training library's conventions.
• Deterministic: the same dataset always compiles to byte-identical output.

Installation

git clone https://github.com/sammwyy/wave.git
cd wave
pip install -e .

# Optional: exact token counts via tiktoken (falls back to an approximation otherwise)
pip install -e ".[tokenizers]"

Requires Python 3.10+.

Quick start

# Scaffold a new project (tasks/, exports/, wave.toml, and one example task)
wave init my-dataset
cd my-dataset

# Validate the dataset against the schema and dataset-wide rules
wave validate

# Get a full repository health report
wave doctor

# Compile and export to a training format
wave export openai --output exports/openai.jsonl
wave export chatml --output exports/chatml.jsonl

The task format

Each task is one YAML file. Wave derives the task id from its file path if id is omitted, so tasks/rust/hashmap/insert.yaml becomes the id rust/hashmap/insert.

version: 1
id: rust/hashmap/insert

metadata:
  language: rust
  difficulty: easy
  category: collections
  subcategory: hashmap
  tags:
    - hashmap
    - std
  license: MIT
  author: sammwyy
  reviewed: true
  verified: true

input:
  prompt: Write a Rust function that inserts a key-value pair into a HashMap and returns the previous value if it existed.
  context: Use std::collections::HashMap.

output:
  summary: Implements HashMap::insert semantics via a thin wrapper function.
  explanation: |
    HashMap::insert already returns Option<V> with the previous value, so
    the wrapper simply forwards to it.
  files:
    - path: src/lib.rs
      content: |
        use std::collections::HashMap;

        pub fn insert(map: &mut HashMap<String, i32>, key: String, value: i32) -> Option<i32> {
            map.insert(key, value)
        }
  tests:
    - path: tests/basic.rs
      content: |
        use std::collections::HashMap;

        #[test]
        fn inserts_and_returns_previous_value() {
            let mut map = HashMap::new();
            assert_eq!(map.insert("a".to_string(), 1), None);
            assert_eq!(map.insert("a".to_string(), 2), Some(1));
        }

validation:
  compile: cargo check
  test: cargo test
  lint: cargo clippy
  format: cargo fmt --check

See schemas/task.v1.schema.json for the full JSON Schema, or run wave schema to print it directly from the code.

CLI reference

Command	Description
wave init [PATH]	Scaffold a new project: tasks/, exports/, wave.toml, and an example task.
wave validate	Schema validation plus dataset-wide checks (duplicate ids, empty files, etc). Exits non-zero on error.
wave lint	Same pipeline as validate, framed around quality warnings (missing tests, duplicate prompts, unusual difficulty labels).
wave doctor	Full repository health report: task counts, language distribution, missing tests, duplicates, token stats.
wave stats	Language / difficulty / category distributions and a token histogram.
wave export <format>	Compile and export. Formats: openai, chatml, code, instruction.
wave tokenize	Per-task token counts and the largest tasks in the dataset.
wave split	Deterministic train/validation/test split by id, with configurable ratios and seed.
wave search <query...>	Filter tasks with language=, tag=, difficulty=, category=, subcategory=, plus free text.
wave schema	Print the JSON Schema, or --task <path> to validate a single file.
wave migrate	Migrate task files to the latest schema version (extensible migration registry; a no-op today since only v1 exists).

Run wave --help or wave <command> --help for full option lists.

Export formats

Format	Shape	Notes
openai	{"messages": [system, user, assistant]}	OpenAI fine-tuning chat format.
chatml	{"text": "<|im_start|>..."}	Raw ChatML text, used by many open chat models.
code	{"prompt": ..., "completion": ...}	Code artifacts only, no prose.
instruction	{"instruction": ..., "input": ..., "output": ...}	Alpaca-style instruction tuning.

New formats are added by writing a class in src/wavecompiler/formats/ and registering it with @register -- the parser, validator, and compiler never need to change.

Design principles

• The parser never exports. Exporters never parse.
• The compiler never reads files directly; it only receives an already parsed and normalized AST.
• The CLI contains no business logic -- every command's logic lives in src/wavecompiler/commands/ and is independently testable.
• Every module has a single, explicit responsibility.
• Compilation is deterministic: identical input always produces identical output, byte for byte.

What's implemented in this MVP

This release covers the full compiler pipeline end to end plus a usable CLI, matching the original specification's foundational goals:

• Versioned Pydantic task schema (version, id, metadata, input, output, validation), with schema-level validation (schemas/).
• Parser (YAML -> AST), never touched directly by exporters.
• Normalizer for deterministic ordering, whitespace, and casing.
• Dataset-wide validator: duplicate ids, duplicate prompts, duplicate output file sets, empty file contents, unconventional difficulty labels, missing tests.
• Deterministic compiler pipeline and exporter registry.
• Four working exporters: OpenAI, ChatML, code-only, instruction-tuning.
• CLI commands: init, validate, lint, doctor, stats, export, tokenize, split, search, schema, migrate.
• Token counting via tiktoken, with a dependency-free fallback.
• Project discovery via wave.toml, mirroring how git/npm find their root.
• Extensible migration framework, ready for schema v2+.
• A full pytest test suite covering models, parser, normalizer, validator, compiler, exporters, statistics, and the CLI.

Not yet implemented (see ROADMAP.md): running validation commands (compile/test/lint/format) against real toolchains, dataset quality scoring, automatic benchmark generation, and additional export targets (Llama, Qwen, Unsloth/Axolotl configs, agent/tool-calling/RAG datasets).

Contributing

See CONTRIBUTING.md.

License

MIT. See LICENSE.