OpenRepro-Agent v0.4.0

OpenRepro-Agent v0.4.0

OpenRepro-Agent v0.4.0 adds opt-in provider readiness and paper-to-code guardrails while keeping benchmark output limited to workflow-compliance evidence.

Highlights

• Bumped package, config, docs, manifests, and benchmark schema to 0.4.0.
• Added openrepro configure-provider for mock/OpenAI-compatible provider settings without storing secrets.
• Added an OpenAI-compatible provider path that requires explicit enable_real_api opt-in and environment-backed API keys.
• Added openrepro scaffold-experiment for human-gated experiment folders generated from candidate formulas and parameters.
• Added openrepro repair-plan to turn diagnosis output into advisory repair artifacts.
• Added openrepro compare-runs for manifest and metric comparisons across project runs.
• Added openrepro benchmark-suite and benchmarks/sample_suite.json.
• Updated smoke tests and regression coverage for provider configuration, experiment scaffolds, repair plans, run comparison, and benchmark suites.

Policy

• Mock provider remains the default.
• Real provider calls are disabled unless explicitly enabled in project config.
• API keys must remain in environment variables; OpenRepro-Agent does not store secret values.
• Experiment scaffolds are generated from candidate_unverified evidence and require human review.
• Benchmark and suite outputs are workflow-compliance evidence only, not scientific reproduction scores.

Validation

Validated locally with:

python -m pytest -q --basetemp .codex_tmp\pytest-basetemp
.\scripts\smoke_test.ps1

OpenRepro-Agent v0.3.1

OpenRepro-Agent v0.3.1

OpenRepro-Agent v0.3.1 is a stability and observability release for the v0.3 provider-and-benchmark layer.

Highlights

• Added openrepro inspect <project_name> with a Rich table and workspace/inspect_summary.json.
• Added openrepro validate <project_name> --all to validate every run under outputs/.
• Added benchmark index generation after each benchmark run.
• Added openrepro benchmark-index [--runs-dir PATH] to rebuild benchmark_index.json and benchmark_index.md.
• Hardened benchmark task schemas while preserving v0.3.0 compatibility.
• Added required/optional artifact and metric checks plus workflow and pass-criteria controls.
• Updated tests and smoke workflows for inspection, all-run validation, benchmark indexing, and schema compatibility.

Benchmark Schema

v0.3.1 keeps expected_artifacts and evaluation_metrics working, and adds:

• artifacts.required
• artifacts.optional
• metrics.required
• metrics.optional
• workflow.run_demo
• workflow.run_sweep
• pass_criteria.require_manifest_valid

Optional artifacts and metrics are reported as evidence but do not fail benchmark status.

Policy

• v0.3.1 does not implement real LLM providers.
• Benchmark output is workflow-compliance evidence only.
• No benchmark score, paper reproduction success, user count, token usage, or cost is fabricated.

Validation

Validated locally with:

python -m pytest -q --basetemp .codex_tmp\pytest-basetemp
.\scripts\smoke_test.ps1

OpenRepro-Agent v0.3.0

OpenRepro-Agent v0.3.0

OpenRepro-Agent v0.3.0 adds the provider-and-benchmark layer while keeping real API calls disabled by default.

Highlights

• Added provider abstractions: ProviderRequest, ProviderResponse, BaseProvider, MockProvider, and get_provider.
• Added request-hash cache support for deterministic mock provider calls.
• Extended API usage summaries with cache hits, cache misses, cached events, and request hashes.
• Added openrepro benchmark --task <task.json> [--project <project>].
• Updated benchmarks/benchmark_schema.json and benchmarks/sample_task.json to schema version 0.3.0.
• Added openrepro diagnose for failure classification and repair suggestions.
• Added benchmark, provider, cache, and diagnostics regression tests.

Policy

• v0.3.0 does not implement real LLM providers.
• Benchmark output is workflow-compliance evidence only.
• No benchmark score, paper reproduction success, user count, token usage, or cost is fabricated.

Validation

Validated locally with:

python -m pytest -q --basetemp .codex_tmp\pytest-basetemp
.\scripts\smoke_test.ps1

OpenRepro-Agent v0.2.0

OpenRepro-Agent v0.2.0

OpenRepro-Agent v0.2.0 upgrades the project from a runnable scaffold into a more auditable reproduction workflow.

Highlights

• Added run-level manifest.json files with artifact paths, categories, file sizes, and SHA-256 hashes.
• Added openrepro validate to verify manifests and required run artifacts.
• Added PDF text extraction with pdfplumber, including page-level provenance files.
• Added rule-based formula candidates, parameter candidates, and structured model_ledger.json.
• Added experiment_plan_validation.json during openrepro plan.
• Added openrepro run-sweep for noise/seed parameter sweeps.
• Updated README, roadmap, smoke tests, and developer docs for the v0.2.0 workflow.

Notes

• v0.2.0 still does not call real LLM APIs by default.
• Formula, parameter, and model outputs are candidates and remain candidate_unverified.
• The built-in BOC-like demo and sweep do not claim benchmark scores or complete paper reproduction.

Validation

Validated locally with:

python -m pytest -q --basetemp .codex_tmp\pytest-basetemp
.\scripts\smoke_test.ps1