Policy Climb

Policy Climb is a deterministic benchmark for comparing bounded policy-search strategies with a frozen evaluator and held-out task splits.

Current Status

Surface	Status	Evidence
Fixture benchmark	Implemented with explicit train, validation, and test splits	data/policy_tasks.json
Policy evaluator	Deterministic local evaluator with split-separated reporting	python3 -m pytest -q
Mutation operators	Named, testable policy mutations with immutable rollback behavior	tests/test_policy_climb.py
E001 smoke	Local smoke artifacts generated from a fixed budget and seed	results/e001/summary.json
Canonical E001	Not run	experiments/e001/configs/canonical.json is intentionally disabled

The checked-in fixture is small and public. It is useful for testing optimizer mechanics and budget accounting, not for claiming general policy-search quality.

Question

Under the same evaluator-call budget, does a diversity-aware search policy find a better held-out policy than fixed coordinate mutation, random mutation, beam search, or no optimization?

Method

The repository keeps only a compact local benchmark:

• A JSON task suite with train, validation, and test split labels.
• A small editable policy in policies/baseline.json.
• Named mutation operators that return new policies instead of mutating files.
• A budgeted train evaluator that reports evaluator calls, token proxy, and wall time.
• Final validation/test scoring that is separate from the optimizer loop.

DSPy and BFCL/Gorilla are pinned in UPSTREAM.md as intended future substrates. The current smoke does not call either upstream package.

Reproduce

Smoke artifact:

scripts/reproduce_e001.sh experiments/e001/configs/smoke.json

Canonical E001:

scripts/reproduce_e001.sh experiments/e001/configs/canonical.json

The canonical command stops while disabled. Enable it only after the DSPy or BFCL task adapter, hidden task source, and matched run budget are implemented.

Local verification:

python3 -m pytest -q
python3 -m compileall -q src tests
uv run --with ruff ruff check .

Repository Map

Path	Why inspect it
UPSTREAM.md	DSPy and BFCL/Gorilla pins and integration boundary
data/policy_tasks.json	Small public fixture with split labels
policies/baseline.json	Editable policy surface for the smoke benchmark
src/policy_climb/	Task loading, evaluator, mutation operators, and optimizers
experiments/e001/experiment.md	Hypothesis, baselines, controls, metrics, and promotion rule
experiments/e001/configs/	Smoke and disabled canonical configs
results/e001/	Generated smoke manifest, summary, CSV, trace, and figure
docs/benchmark-schema.md	Fixture and policy schema
docs/limitations.md	Evidence boundaries and unsupported claims

Evidence Boundary

• Measured by tests: split integrity, immutable policy mutations, rollback behavior, budget accounting, deterministic evaluation, and CLI smoke paths.
• Generated by command: E001 local smoke summary, trace, and figure.
• Inferred: the small fixture can exercise optimizer mechanics before an upstream task adapter exists.
• Not tested: hidden benchmark generalization, DSPy program optimization, BFCL/CallBench official task metrics, live model calls, or cloud execution.

Limitations

The fixture is public, tiny, deterministic, and hand-authored. It should be read as a unit-testable optimizer scaffold, not as a hidden benchmark or policy quality result. See docs/limitations.md for the full boundary.

License and Upstream Attribution

License: MIT.

DSPy and BFCL/Gorilla upstream pins are recorded in UPSTREAM.md.