Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .claude-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "agent-estimate",
"description": "Effort estimation for AI coding agents — PERT three-point estimation with METR reliability thresholds and wave planning",
"version": "0.7.2",
"version": "0.7.3",
"skills": ["./skills/estimate/claude"],
"author": {
"name": "Kiloloop"
Expand Down
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/bug_report.yml
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ body:
id: version
attributes:
label: agent-estimate version
placeholder: "0.7.2"
placeholder: "0.7.3"
validations:
required: true

Expand Down
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

## [0.7.3] - 2026-07-10

### Fixed
- `agent-estimate validate` now reports a clean user-facing error instead of a traceback when calibration DB observation fields have invalid types, with end-to-end tests covering the calibration write path.
- Refreshed shipped example outputs for current engine numbers and expanded docs freshness tests to cover every example capture.

## [0.7.2] - 2026-06-14

### Fixed
Expand Down Expand Up @@ -110,6 +116,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Modifier flags: `--warm-context`, `--spec-clarity`, `--issues`
- PyPI package: `pip install agent-estimate`

[0.7.3]: https://github.com/kiloloop/agent-estimate/releases/tag/v0.7.3
[0.7.2]: https://github.com/kiloloop/agent-estimate/releases/tag/v0.7.2
[0.7.1]: https://github.com/kiloloop/agent-estimate/releases/tag/v0.7.1
[0.7.0]: https://github.com/kiloloop/agent-estimate/releases/tag/v0.7.0
Expand Down
2 changes: 1 addition & 1 deletion action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ inputs:
required: false
default: '3.12'
version:
description: 'agent-estimate version to install (e.g. "0.7.2"). Omit for latest.'
description: 'agent-estimate version to install (e.g. "0.7.3"). Omit for latest.'
required: false
token:
description: 'GitHub token for issue fetching and PR comments'
Expand Down
4 changes: 2 additions & 2 deletions examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ Real input/output examples from production agent dispatches. Every command below
| Example | Task type | Tier | Compression | File |
|---------|-----------|------|-------------|------|
| Fix pyproject URLs after org rename | Coding | XS | 2.00x | [coding-s.md](./coding-s.md) |
| Implement CLI command with code generation | Coding | M | 2.53x | [coding-m.md](./coding-m.md) |
| Implement CLI command with code generation | Coding | M | 3.00x | [coding-m.md](./coding-m.md) |
| Audit cloud infrastructure providers | Research | S | 2.61x | [research.md](./research.md) |
| Write quickstart guide + README | Documentation | S | 2.58x | [documentation.md](./documentation.md) |
| Write quickstart guide + README | Documentation | S | 3.87x | [documentation.md](./documentation.md) |
| 3-agent parallel session (3 features) | Multi-agent | M×3 | 8.07x | [multi-agent.md](./multi-agent.md) |

## How to read the output
Expand Down
36 changes: 32 additions & 4 deletions examples/coding-m.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,13 @@ agent-estimate estimate "Implement add-agent CLI command with SPEC.md generation

| Task | Model | Tier | Agent | Base PERT (O/M/P) | Modifiers | Effective Duration | Human Equivalent |
| --- | --- | --- | --- | --- | --- | --- | --- |
| **Implement add-agent CLI command with SPEC.md generation** | coding | M | Claude | 25m / 50m / 90m (E=52.5m) | spec 1.00 x warm 1.00 x fit 1.00 = 1.00 | 52.5m | 190.9m |
| **Implement add-agent CLI command with SPEC.md generation** | coding | M | Claude | 25m / 50m / 90m (E=52.5m) | spec 1.00 x warm 1.00 x fit 1.00 = 1.00 | 52.5m | 226.4m |

### Wave Plan

| Wave | Tasks | Duration | Agent Assignments (amortized review) |
| --- | --- | --- | --- |
| 0 | Implement add-agent CLI command with SPEC.md generation | 75.4m | Claude: Implement add-agent CLI command with SPEC.md generation +15m review |

### Timeline Summary

Expand All @@ -23,15 +29,37 @@ agent-estimate estimate "Implement add-agent CLI command with SPEC.md generation
| Best case | 44.7m |
| Expected case | 75.4m |
| Worst case | 117.2m |
| Human-speed equivalent | 190.9m |
| Compression ratio | 2.53x |
| Human-speed equivalent | 226.4m |
| Compression ratio | 3.00x |
| Review overhead (per-task, pre-amortization) | 15m |

### Review Overhead

Review is amortized per agent per wave: one review cycle covers all PRs from that
agent in the wave. Per-task values below are the naive (pre-amortization) figures.

| Task | Review Overhead |
| --- | --- |
| Implement add-agent CLI command with SPEC.md generation | 15m |
| **Total (naive)** | **15m** |

### Agent Load Summary

| Agent | Task Count | Total Work | Estimated Cost |
| --- | --- | --- | --- |
| Claude | 1 | 60.4m | $1.45 |
| Codex | 0 | 0m | $0.00 |
| Gemini | 0 | 0m | $0.00 |

### Critical Path

**Implement add-agent CLI command with SPEC.md generation**

### Tier Corrections

No tier corrections.

### METR Warnings

No METR threshold warnings.

Expand Down Expand Up @@ -66,4 +94,4 @@ The task ran slightly over the expected case due to a second review round. This

## Key takeaway

M-tier tasks are the most common dispatch target — complex enough to justify agent overhead, predictable enough for reliable estimation. The 2.53x compression means a feature that takes a human developer ~3 hours lands in ~75 minutes of wall clock time.
M-tier tasks are the most common dispatch target — complex enough to justify agent overhead, predictable enough for reliable estimation. The 3.00x compression means a feature that takes a human developer ~3.8 hours lands in ~75 minutes of wall clock time.
31 changes: 30 additions & 1 deletion examples/coding-s.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,12 @@ agent-estimate estimate "Fix pyproject.toml URLs after org rename"
| --- | --- | --- | --- | --- | --- | --- | --- |
| **Fix pyproject.toml URLs after org rename** | coding | XS | Claude | 5m / 10m / 20m (E=10.8m) | spec 1.00 x warm 1.00 x fit 1.00 = 1.00 | 10.8m | 54.8m |

### Wave Plan

| Wave | Tasks | Duration | Agent Assignments (amortized review) |
| --- | --- | --- | --- |
| 0 | Fix pyproject.toml URLs after org rename | 27.5m | Claude: Fix pyproject.toml URLs after org rename +15m review |

### Timeline Summary

| Metric | Value |
Expand All @@ -27,14 +33,37 @@ agent-estimate estimate "Fix pyproject.toml URLs after org rename"
| Compression ratio | 2.00x |
| Review overhead (per-task, pre-amortization) | 15m |

### Review Overhead

Review is amortized per agent per wave: one review cycle covers all PRs from that
agent in the wave. Per-task values below are the naive (pre-amortization) figures.

| Task | Review Overhead |
| --- | --- |
| Fix pyproject.toml URLs after org rename | 15m |
| **Total (naive)** | **15m** |

### Agent Load Summary

| Agent | Task Count | Total Work | Estimated Cost |
| --- | --- | --- | --- |
| Claude | 1 | 12.5m | $0.30 |
| Codex | 0 | 0m | $0.00 |
| Gemini | 0 | 0m | $0.00 |

### Critical Path

**Fix pyproject.toml URLs after org rename**

### Tier Corrections

No tier corrections.

### METR Warnings

No METR threshold warnings.


## What actually happened

| Metric | Estimated | Actual |
Expand All @@ -49,4 +78,4 @@ The task was dispatched to Codex (better fit for mechanical find-and-replace acr

## Key takeaway

XS tasks have a lower compression ratio (2.00x) because the fixed 15m review overhead dominates. For self-merged work where you trust the agent, use `--review-mode none` to see the raw speed advantage — the effective duration drops to ~11m vs ~55m human time (5x compression).
XS tasks have a lower compression ratio (2.00x) because the fixed 15m review overhead dominates. For self-merged work where you trust the agent, use `--review-mode none` to remove the review overhead: an executed none-mode run keeps effective duration at 10.8m, drops expected case to 12.5m, and scales human-equivalent time down to 23m human time, for 1.84x compression. Human-equivalent time scales with the agent total, so removing review lowers both sides of the comparison and keeps roughly the same ~2x compression ratio instead of creating a 5x raw-speed claim.
37 changes: 33 additions & 4 deletions examples/documentation.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,13 @@ agent-estimate estimate \

| Task | Model | Tier | Agent | Base PERT (O/M/P) | Modifiers | Effective Duration | Human Equivalent |
| --- | --- | --- | --- | --- | --- | --- | --- |
| **Write quickstart guide and README with protocol comparison table** | documentation | S | Claude | 10m / 25m / 45m (E=25.8m) | spec 1.00 x warm 1.00 x fit 1.00 = 1.00 | 25.8m | 115.5m |
| **Write quickstart guide and README with protocol comparison …** | documentation | S | Claude | 10m / 25m / 45m (E=25.8m) | spec 1.00 x warm 1.00 x fit 1.00 = 1.00 | 25.8m | 173.2m |

### Wave Plan

| Wave | Tasks | Duration | Agent Assignments (amortized review) |
| --- | --- | --- | --- |
| 0 | Write quickstart guide and README with protocol comparison … | 44.7m | Claude: Write quickstart guide and README with protocol comparison … +15m review |

### Timeline Summary

Expand All @@ -25,18 +31,41 @@ agent-estimate estimate \
| Best case | 27.4m |
| Expected case | 44.7m |
| Worst case | 65.7m |
| Human-speed equivalent | 115.5m |
| Compression ratio | 2.58x |
| Human-speed equivalent | 173.2m |
| Compression ratio | 3.87x |
| Review overhead (per-task, pre-amortization) | 15m |

### Review Overhead

Review is amortized per agent per wave: one review cycle covers all PRs from that
agent in the wave. Per-task values below are the naive (pre-amortization) figures.

| Task | Review Overhead |
| --- | --- |
| Write quickstart guide and README with protocol comparison … | 15m |
| **Total (naive)** | **15m** |

### Agent Load Summary

| Agent | Task Count | Total Work | Estimated Cost |
| --- | --- | --- | --- |
| Claude | 1 | 29.7m | $0.71 |
| Codex | 0 | 0m | $0.00 |
| Gemini | 0 | 0m | $0.00 |

### Critical Path

**Write quickstart guide and README with protocol comparison …**

### Tier Corrections

No tier corrections.

### METR Warnings

No METR threshold warnings.


## What actually happened

| Metric | Estimated | Actual |
Expand All @@ -51,4 +80,4 @@ Almost exactly on estimate. Documentation tasks are the most predictable categor

## Key takeaway

Documentation has the highest human-multiplier range (3.0-6.0x) because humans find technical writing tedious and slow. Agents produce consistent, structured output quickly. The 2.58x wall-clock compression accounts for the review cycle — without review (`--review-mode none`), raw agent time is ~26m vs ~116m human time, a 4.5x compression.
Documentation has the highest human-multiplier range (3.0-6.0x) because humans find technical writing tedious and slow. Agents produce consistent, structured output quickly. The 3.87x wall-clock compression accounts for the review cycle — without review (`--review-mode none`), an executed none-mode run reports 25.8m effective duration, 29.7m expected case, 109.6m human-equivalent time, and 3.69x compression.
19 changes: 18 additions & 1 deletion examples/multi-agent.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ agent-estimate estimate --file tasks.txt

| Wave | Tasks | Duration | Agent Assignments (amortized review) |
| --- | --- | --- | --- |
| 0 | All 3 tasks | 75.4m | Claude: add-agent CLI +15m review; Codex: known_debt.md +15m review; Gemini: quickstart guide +15m review |
| 0 | Implement add-agent CLI command with SPEC.md generation, Add known_debt.md as standard protocol memory file, Write quickstart guide with protocol comparison table | 75.4m | Claude: Implement add-agent CLI command with SPEC.md generation +15m review; Codex: Add known_debt.md as standard protocol memory file +15m review; Gemini: Write quickstart guide with protocol comparison table +15m review |

### Timeline Summary

Expand All @@ -45,6 +45,18 @@ agent-estimate estimate --file tasks.txt
| Compression ratio | 8.07x |
| Review overhead (per-task, pre-amortization) | 45m |

### Review Overhead

Review is amortized per agent per wave: one review cycle covers all PRs from that
agent in the wave. Per-task values below are the naive (pre-amortization) figures.

| Task | Review Overhead |
| --- | --- |
| Implement add-agent CLI command with SPEC.md generation | 15m |
| Add known_debt.md as standard protocol memory file | 15m |
| Write quickstart guide with protocol comparison table | 15m |
| **Total (naive)** | **45m** |

### Agent Load Summary

| Agent | Task Count | Total Work | Estimated Cost |
Expand All @@ -57,11 +69,16 @@ agent-estimate estimate --file tasks.txt

**Implement add-agent CLI command with SPEC.md generation**

### Tier Corrections

No tier corrections.

### METR Warnings

- **Add known_debt.md as standard protocol memory file**: Estimate (75m) exceeds gpt_5_4 p80 threshold (60m). Consider splitting the task.
- **Write quickstart guide with protocol comparison table**: Estimate (75m) exceeds gemini_3_1_pro p80 threshold (45m). Consider splitting the task.


## What actually happened

These three tasks were dispatched to real agents in a production multi-agent workflow:
Expand Down
33 changes: 31 additions & 2 deletions examples/research.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,13 @@ agent-estimate estimate \

| Task | Model | Tier | Agent | Base PERT (O/M/P) | Modifiers | Effective Duration | Human Equivalent |
| --- | --- | --- | --- | --- | --- | --- | --- |
| **Audit cloud infrastructure providers for production deployment** | research | S | Claude | 10m / 20m / 30m (E=20m) | spec 1.00 x warm 1.00 x fit 1.00 = 1.00 | 20m | 99m |
| **Audit cloud infrastructure providers for production deploym…** | research | S | Claude | 10m / 20m / 30m (E=20m) | spec 1.00 x warm 1.00 x fit 1.00 = 1.00 | 20m | 99m |

### Wave Plan

| Wave | Tasks | Duration | Agent Assignments (amortized review) |
| --- | --- | --- | --- |
| 0 | Audit cloud infrastructure providers for production deploym… | 38m | Claude: Audit cloud infrastructure providers for production deploym… +15m review |

### Timeline Summary

Expand All @@ -29,14 +35,37 @@ agent-estimate estimate \
| Compression ratio | 2.61x |
| Review overhead (per-task, pre-amortization) | 15m |

### Review Overhead

Review is amortized per agent per wave: one review cycle covers all PRs from that
agent in the wave. Per-task values below are the naive (pre-amortization) figures.

| Task | Review Overhead |
| --- | --- |
| Audit cloud infrastructure providers for production deploym… | 15m |
| **Total (naive)** | **15m** |

### Agent Load Summary

| Agent | Task Count | Total Work | Estimated Cost |
| --- | --- | --- | --- |
| Claude | 1 | 23m | $0.55 |
| Codex | 0 | 0m | $0.00 |
| Gemini | 0 | 0m | $0.00 |

### Critical Path

**Audit cloud infrastructure providers for production deploym…**

### Tier Corrections

No tier corrections.

### METR Warnings

No METR threshold warnings.


## What actually happened

| Metric | Estimated | Actual |
Expand All @@ -50,4 +79,4 @@ The agent finished well under the estimate — research tasks with web access te

## Key takeaway

Research tasks have the highest human-multiplier range (3.0-6.0x) because pattern matching across large datasets is tedious for humans but routine for agents. Scanning 7 cloud providers' pricing, compliance, and feature matrices would take a human most of a workday. Use `--type research` to select the research estimation model — it uses a flat PERT curve with depth scaling instead of the tier-based coding model.
Research tasks use the generic human-multiplier range (2.0-4.0x) while the research model handles the agent-side time box. Pattern matching across large datasets is still tedious for humans but routine for agents: scanning 7 cloud providers' pricing, compliance, and feature matrices would take a human most of a workday. Use `--type research` to select the research estimation model — it uses a flat PERT curve with depth scaling instead of the tier-based coding model.
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "hatchling.build"

[project]
name = "agent-estimate"
version = "0.7.2"
version = "0.7.3"
description = "Know what an AI task will cost before you run it"
readme = "README.md"
license = "Apache-2.0"
Expand Down
2 changes: 1 addition & 1 deletion src/agent_estimate/cli/commands/validate.py
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ def run(
if db is not None:
try:
obs = _build_observation(raw, estimated, actual_work, actual_total, error_ratio, verdict)
except ValueError as exc:
except (TypeError, ValueError) as exc:
typer.echo(f"Error: Invalid observation field: {exc}", err=True)
raise typer.Exit(code=2)

Expand Down
2 changes: 1 addition & 1 deletion src/agent_estimate/version.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
"""Version constants for agent-estimate."""

__version__ = "0.7.2"
__version__ = "0.7.3"
Loading
Loading