Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .claude-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
{
"name": "agent-estimate",
"description": "Effort estimation for AI coding agents — PERT three-point estimation with METR reliability thresholds and wave planning",
"version": "0.7.0",
"version": "0.7.1",
"skills": ["./skills/estimate/claude"],
"author": {
"name": "Kiloloop"
},
Expand Down
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/bug_report.yml
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ body:
id: version
attributes:
label: agent-estimate version
placeholder: "0.7.0"
placeholder: "0.7.1"
validations:
required: true

Expand Down
11 changes: 11 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

## [0.7.1] - 2026-06-11

### Fixed
- Fixed six engine correctness gaps: early `--format` validation before side effects, explicit `--warm-context 1.0` handling, friction-aware assigned-agent METR re-checks, Python 3.10-compatible Z timestamp parsing, UTC-normalized calibration cohorts/order, and `plan_waves` duplicate/negative input validation (#31, #33, #36, #37, #38, #39).
- README hero example now uses the real CLI surface with real captured output — the previous block invoked a nonexistent `--model` flag, passed an unsupported second positional task, and showed fabricated numbers (#28).
- README Action workflow example now grants `issues: read`, required for issue fetching on private repos where an explicit `permissions:` block zeroes unlisted scopes (#40).
- Claude Code plugin exposes the `/estimate` skill again: `plugin.json` now points skill discovery at `skills/estimate/claude/`, which the v0.7.0 multi-runtime restructure had moved out of the default `skills/<name>/SKILL.md` scan path (#29).
- Dropped the phantom `/validate-estimate` and `/calibrate` slash commands (documented but never installable); validation and calibration now route through the installed skill as `/estimate validate` and `/estimate calibrate` (#29).
- Refreshed stale METR model keys in `examples/multi-agent.md` output (`gpt_5_3` → `gpt_5_4`, `gemini_3_pro` → `gemini_3_1_pro`) to match current output (#28).

## [0.7.0] - 2026-05-20

### Added
Expand Down Expand Up @@ -94,6 +104,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Modifier flags: `--warm-context`, `--spec-clarity`, `--issues`
- PyPI package: `pip install agent-estimate`

[0.7.1]: https://github.com/kiloloop/agent-estimate/releases/tag/v0.7.1
[0.7.0]: https://github.com/kiloloop/agent-estimate/releases/tag/v0.7.0
[0.6.1]: https://github.com/kiloloop/agent-estimate/releases/tag/v0.6.1
[0.6.0]: https://github.com/kiloloop/agent-estimate/releases/tag/v0.6.0
Expand Down
51 changes: 23 additions & 28 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,40 +102,29 @@ Real estimates from production use — including the misses.

**An honest over-estimate.** We pre-registered a UI mockup build at ~95 minutes with no prior app-dev data. Two agents did it in parallel in 12 and 25 minutes — a 4–8x over-estimate. agent-estimate now ships an `app_dev` prior shaped by that result. The miss stays in the README because calibration means showing where you were wrong.

**Two tasks, one model** — what the tool prints, including the METR reliability flag:
**Three tasks, three agents, in parallel** — what the tool prints, including the METR reliability flags. Input is the three-task `tasks.txt` from [`examples/multi-agent.md`](./examples/multi-agent.md); the output below is captured from a real run, trimmed to the timeline and warnings (the full report — per-task PERT table, wave plan, agent loads — is in that example):

```text
$ agent-estimate estimate "Implement auth" "Add tests" --model opus

Task Tier PERT (O/M/P) Expected Human-eq
───────────────────────────────────────────────────────────
Implement auth M 25/50/90m 57.8m 160m
Add tests S 12/23/40m 24.0m 75m

Timeline ──────────────────────────────
best 37m · expected 81.8m · worst 130m
human-equivalent: 235m → 2.87× compression
$ agent-estimate estimate --file tasks.txt

⚠ METR warning: "Implement auth" exceeds Opus p80
```
## Timeline Summary

~82 minutes expected versus ~4 hours by hand — plus a flag that the auth task runs past Opus's p80 reliability horizon, so you split it or add a checkpoint before dispatching.
| Metric | Value |
| --- | --- |
| Best case | 44.7m |
| Expected case | 75.4m |
| Worst case | 117.2m |
| Human-speed equivalent | 572.8m |
| Compression ratio | 7.60x |
| Review overhead (per-task, pre-amortization) | 45m |

**Three tasks, three agents, in parallel:**
## METR Warnings

```bash
$ agent-estimate estimate --file tasks.txt
- **Add known_debt.md as standard protocol memory file**: Estimate (68m) exceeds gpt_5_4 p80 threshold (60m). Consider splitting the task.
- **Write quickstart guide with protocol comparison table**: Estimate (68m) exceeds gemini_3_1_pro p80 threshold (45m). Consider splitting the task.
```

| Metric | Value |
| --- | --- |
| Wave 0 | All 3 tasks in parallel (Claude + Codex + Gemini) |
| Expected case | 131m |
| Human-speed equivalent | 709.5m |
| **Compression ratio** | **5.42x** |
| Estimated cost | $4.84 |

~2 hours wall-clock versus ~12 hours sequential. You see the compression before you commit the compute. More in [`examples/`](./examples/) — coding S/M, research, documentation, multi-agent.
~75 minutes wall-clock versus ~9.5 hours of sequential human work, at an estimated $3.51 fleet cost — plus two flags that the Codex- and Gemini-assigned tasks run past their models' p80 reliability horizons, so you split them or add a checkpoint before dispatching. The same three tasks were later run by real agents; the retro is in the example file. More in [`examples/`](./examples/) — coding S/M, research, documentation, multi-agent.

## Integrations

Expand All @@ -150,8 +139,8 @@ $ agent-estimate estimate --file tasks.txt
/estimate Add a login page with OAuth
/estimate --file spec.md
/estimate --issues 1,2,3 --repo myorg/myrepo
/validate-estimate observation.yaml
/calibrate
/estimate validate observation.yaml
/estimate calibrate
```

### GitHub Action
Expand All @@ -173,6 +162,7 @@ on:

permissions:
contents: read
issues: read
pull-requests: write

jobs:
Expand Down Expand Up @@ -265,8 +255,13 @@ agent-estimate estimate "Ship packaging flow" --config ./my_agents.yaml
agent-estimate estimate "Refactor auth pipeline" --format json # machine-readable
agent-estimate estimate --repo myorg/myrepo --issues 11,12,14 # from GitHub issues
agent-estimate estimate --file tasks.txt # from file
agent-estimate estimate "Follow-up fix" --history-file data.json # auto warm-context
```

When `--warm-context` is omitted, the CLI can auto-infer it from `--history-file`;
if no history file is passed and `./data.json` exists, that file is used as the
default dispatch history source.

### Calibration

Validate estimates against observed outcomes and build a calibration database:
Expand Down
2 changes: 1 addition & 1 deletion action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ inputs:
required: false
default: '3.12'
version:
description: 'agent-estimate version to install (e.g. "0.7.0"). Omit for latest.'
description: 'agent-estimate version to install (e.g. "0.7.1"). Omit for latest.'
required: false
token:
description: 'GitHub token for issue fetching and PR comments'
Expand Down
4 changes: 2 additions & 2 deletions examples/multi-agent.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,8 +59,8 @@ agent-estimate estimate --file tasks.txt

### METR Warnings

- **Add known_debt.md as standard protocol memory file**: Estimate (68m) exceeds gpt_5_3 p80 threshold (60m). Consider splitting the task.
- **Write quickstart guide with protocol comparison table**: Estimate (68m) exceeds gemini_3_pro p80 threshold (45m). Consider splitting the task.
- **Add known_debt.md as standard protocol memory file**: Estimate (68m) exceeds gpt_5_4 p80 threshold (60m). Consider splitting the task.
- **Write quickstart guide with protocol comparison table**: Estimate (68m) exceeds gemini_3_1_pro p80 threshold (45m). Consider splitting the task.

## What actually happened

Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "hatchling.build"

[project]
name = "agent-estimate"
version = "0.7.0"
version = "0.7.1"
description = "Know what an AI task will cost before you run it"
readme = "README.md"
license = "Apache-2.0"
Expand Down
4 changes: 2 additions & 2 deletions skills/estimate/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,8 @@ cp skills/estimate/codex/SKILL.md .codex/skills/estimate/SKILL.md
/estimate Add a login page with OAuth
/estimate --file tasks.md
/estimate --issues 1,2,3 --repo myorg/myrepo
/validate-estimate observation.yaml
/calibrate
/estimate validate observation.yaml
/estimate calibrate

# Codex
# Invoke via AGENTS.md task dispatch or direct skill reference
Expand Down
32 changes: 17 additions & 15 deletions skills/estimate/claude/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@ Run PERT three-point estimation with METR reliability thresholds and wave planni
/estimate --format json <task>
/estimate --review-mode none <task>
/estimate --title "My Report" <task>
/validate-estimate <observation.yaml>
/calibrate
/estimate validate <observation.yaml>
/estimate calibrate
```

## Instructions
Expand All @@ -28,13 +28,15 @@ When the user invokes this skill, follow these steps:

### 1. Parse the invocation

Determine which subcommand to run based on context:
Determine which subcommand to run from the shape of the arguments after `/estimate`:

| Invocation pattern | Subcommand |
| ----------------------------- | -------------------------- |
| `/estimate ...` with any args | `agent-estimate estimate` |
| `/validate-estimate <file>` | `agent-estimate validate` |
| `/calibrate` | `agent-estimate calibrate` |
| Invocation pattern | Subcommand |
| ------------------------------- | -------------------------- |
| `validate <path>` — first token is `validate`, rest is one `.yaml`/`.yml` path plus optional `--db <path>` | `agent-estimate validate` |
| `calibrate` — first token is `calibrate`, rest is empty or only `--db <path>` | `agent-estimate calibrate` |
| anything else | `agent-estimate estimate` |

Shape-match strictly: if the arguments start with `validate` or `calibrate` but the rest does not fit the shapes above, treat the whole argument string as a task description for `agent-estimate estimate` — e.g. `/estimate validate the login flow` is an estimation request, not a malformed validate call.

### 2. Build the CLI command

Expand Down Expand Up @@ -75,11 +77,11 @@ If none of `--file`, `--issues`, or a task description is provided, prompt the u
- `frontend` — UI/page work (content patches use 15/25/40; page builds use 40/60/90).
- `app_dev` — app shells and desktop/mobile builds (cold generic L-style prior; use modifiers for warm or highly specified work).

#### For `/validate-estimate`
#### For `/estimate validate`

The argument after `/validate-estimate` is the observation YAML file path. Optionally pass `--db <path>` if provided.
The argument after `validate` is the observation YAML file path. Optionally pass `--db <path>` if provided.

#### For `/calibrate`
#### For `/estimate calibrate`

Optionally pass `--db <path>` if provided. Default: `~/.agent-estimate/calibration.db`.

Expand Down Expand Up @@ -120,17 +122,17 @@ Display the CLI output directly to the user. No post-processing — the CLI hand
/estimate --config agents.yaml --format json "Refactor auth module"
→ agent-estimate estimate --config agents.yaml --format json "Refactor auth module"

/validate-estimate results/sprint1.yaml
/estimate validate results/sprint1.yaml
→ agent-estimate validate results/sprint1.yaml

/validate-estimate results/sprint1.yaml --db ~/.agent-estimate/calibration.db
/estimate validate results/sprint1.yaml --db ~/.agent-estimate/calibration.db
→ agent-estimate validate results/sprint1.yaml --db ~/.agent-estimate/calibration.db

/calibrate
/estimate calibrate
→ agent-estimate calibrate
```

## Observation YAML format (for `/validate-estimate`)
## Observation YAML format (for `/estimate validate`)

```yaml
task_type: feature
Expand Down
17 changes: 13 additions & 4 deletions src/agent_estimate/adapters/sqlite_store.py
Original file line number Diff line number Diff line change
Expand Up @@ -406,17 +406,26 @@ def _validate_observation(observation: ObservationInput) -> None:

def _normalize_timestamp(value: str | None) -> str:
if value is None:
return datetime.now(timezone.utc).isoformat(timespec="seconds")
return value
dt = datetime.now(timezone.utc)
else:
dt = _parse_iso_timestamp(value)
return dt.isoformat(timespec="seconds")


def _week_start(timestamp: str) -> str:
normalized = timestamp.replace("Z", "+00:00")
dt = datetime.fromisoformat(normalized)
dt = _parse_iso_timestamp(timestamp)
monday = (dt - timedelta(days=dt.weekday())).date()
return monday.isoformat()


def _parse_iso_timestamp(value: str) -> datetime:
normalized = value[:-1] + "+00:00" if value.endswith("Z") else value
dt = datetime.fromisoformat(normalized)
if dt.tzinfo is None:
return dt.replace(tzinfo=timezone.utc)
return dt.astimezone(timezone.utc)


def _percentile(values: list[float], percent: float) -> float:
if not values:
raise ValueError("percentile requires at least one value")
Expand Down
6 changes: 4 additions & 2 deletions src/agent_estimate/cli/commands/_pipeline.py
Original file line number Diff line number Diff line change
Expand Up @@ -330,10 +330,12 @@ def _build_report(
assigned_agent = assignment_map.get(str(i), default_agent)
model_tier = agent_model_tier.get(assigned_agent, default_tier)

# Re-check METR threshold with the assigned agent's model tier
# Re-check METR threshold with the assigned agent's model tier and the
# same frictioned work duration the wave planner schedules.
metr_minutes = (est.pert.expected * config.settings.friction_multiplier) + est.review_minutes
corrected_warning = check_metr_threshold(
model_tier,
est.total_expected_minutes,
metr_minutes,
thresholds=thresholds,
fallback_threshold=fallback,
agent_name=assigned_agent,
Expand Down
12 changes: 12 additions & 0 deletions src/agent_estimate/cli/commands/_utils.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
"""Shared CLI command helpers."""

from __future__ import annotations

import typer


def validate_output_format(value: str) -> None:
"""Validate a command's output format before side-effectful work runs."""
if value not in {"markdown", "json"}:
typer.echo(f"Error: Unknown format: {value!r}. Use markdown or json.", err=True)
raise typer.Exit(code=2)
21 changes: 12 additions & 9 deletions src/agent_estimate/cli/commands/estimate.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
from agent_estimate.adapters.github_ghcli import GitHubGhCliAdapter
from agent_estimate.audit import emit_audit_event
from agent_estimate.cli.commands._pipeline import run_estimate_pipeline
from agent_estimate.cli.commands._utils import validate_output_format
from agent_estimate.cli.commands.github import parse_issue_selection
from agent_estimate.core import EstimationCategory, EstimationConfig, ReviewMode
from agent_estimate.core.history import infer_warm_context
Expand Down Expand Up @@ -58,10 +59,13 @@ def run(
"--spec-clarity",
help="Spec clarity modifier (range: 0.3 to 1.3; lower means clearer spec).",
),
warm_context: float = typer.Option(
1.0,
warm_context: Optional[float] = typer.Option(
None,
"--warm-context",
help="Warm context modifier (range: 0.3 to 1.15; lower means warmer context).",
help=(
"Warm context modifier (range: 0.3 to 1.15; lower means warmer context). "
"When omitted, auto-infers from --history-file or ./data.json when present."
),
),
agent_fit: float = typer.Option(
1.0,
Expand Down Expand Up @@ -124,6 +128,7 @@ def run(
_error("Provide a task description, --file, or --issues.", 2)
if sources > 1:
_error("Provide only one input source: task argument, --file, or --issues.", 2)
validate_output_format(format)

descriptions: list[str] = []

Expand Down Expand Up @@ -197,10 +202,10 @@ def run(
warm_ctx = infer_warm_context(
history_path, agent=history_agent, project=history_project
)
# Auto-inferred warm_context applies when --warm-context wasn't explicitly set
effective_warm_context = warm_context
# Auto-inferred warm_context applies only when --warm-context was omitted.
effective_warm_context = 1.0 if warm_context is None else warm_context
effective_detail: str | None = None
if warm_ctx.value != 1.0 and warm_context == 1.0:
if warm_ctx.value != 1.0 and warm_context is None:
effective_warm_context = warm_ctx.value
effective_detail = warm_ctx.detail
logger.info(
Expand Down Expand Up @@ -263,10 +268,8 @@ def run(
# --- Output ---
if format == "markdown":
typer.echo(render_markdown_report(report))
elif format == "json":
typer.echo(render_json_report(report), nl=False)
else:
_error(f"Unknown format: {format!r}. Use markdown or json.", 2)
typer.echo(render_json_report(report), nl=False)


def _error(message: str, exit_code: int) -> NoReturn:
Expand Down
7 changes: 4 additions & 3 deletions src/agent_estimate/cli/commands/session.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
import typer

from agent_estimate.audit import emit_audit_event
from agent_estimate.cli.commands._utils import validate_output_format
from agent_estimate.core.session import (
DEFAULT_COORDINATION_OVERHEAD_MINUTES,
SESSION_TYPE_DURATIONS,
Expand Down Expand Up @@ -62,6 +63,7 @@ def run(
) -> None:
"""Estimate wall-clock and agent-minutes for a multi-agent session."""
started_at = time.perf_counter()
validate_output_format(format)
overhead = (
coordination_overhead
if coordination_overhead is not None
Expand Down Expand Up @@ -110,10 +112,8 @@ def run(
"rounds_breakdown": list(result.rounds_breakdown),
}
typer.echo(json.dumps(data, indent=2), nl=False)
elif format == "markdown":
_render_markdown(result)
else:
_error(f"Unknown format: {format!r}. Use markdown or json.", 2)
_render_markdown(result)


# ---------------------------------------------------------------------------
Expand Down Expand Up @@ -156,3 +156,4 @@ def _render_markdown(result: SessionEstimate) -> None:
def _error(message: str, exit_code: int) -> NoReturn:
typer.echo(f"Error: {message}", err=True)
raise typer.Exit(code=exit_code)

Loading
Loading