Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .claude-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "agent-estimate",
"description": "Effort estimation for AI coding agents — PERT three-point estimation with METR reliability thresholds and wave planning",
"version": "0.7.1",
"version": "0.7.2",
"skills": ["./skills/estimate/claude"],
"author": {
"name": "Kiloloop"
Expand Down
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/bug_report.yml
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ body:
id: version
attributes:
label: agent-estimate version
placeholder: "0.7.1"
placeholder: "0.7.2"
validations:
required: true

Expand Down
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

## [0.7.2] - 2026-06-14

### Fixed
- Fixed engine and documentation correctness gaps: documentation estimates now use the docs human multiplier, CLI file/config and GitHub adapter failures stay user-facing, REST issue ingestion is reachable in token-only environments, renderer warning/detail fields are escaped and serialized, sizing heuristics prefer explicit test/docs matches, store/audit/validation edge cases are hardened, and stale skill/session/release docs are refreshed (#30, #32, #34, #35, #41, #42, #43, #44, #45).
- `AGENT_ESTIMATE_AUDIT_DESTINATION=stdout` now emits a deprecation warning and routes audit events to stderr so report stdout remains parseable for JSON consumers (#43).

## [0.7.1] - 2026-06-11

### Fixed
Expand Down Expand Up @@ -104,6 +110,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Modifier flags: `--warm-context`, `--spec-clarity`, `--issues`
- PyPI package: `pip install agent-estimate`

[0.7.2]: https://github.com/kiloloop/agent-estimate/releases/tag/v0.7.2
[0.7.1]: https://github.com/kiloloop/agent-estimate/releases/tag/v0.7.1
[0.7.0]: https://github.com/kiloloop/agent-estimate/releases/tag/v0.7.0
[0.6.1]: https://github.com/kiloloop/agent-estimate/releases/tag/v0.6.1
Expand Down
8 changes: 8 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,14 @@ If your change touches CLI behavior, include at least one integration-style test
4. Include a short test plan and validation output.
5. Update docs/changelog when behavior changes.

## Release Checklist

1. Bump all versioned artifacts together: `pyproject.toml`, `src/agent_estimate/version.py`, `.claude-plugin/plugin.json`, `action.yml`, issue templates, and version tests.
2. Run `ruff check .`, `pytest -q`, `python -m build`, and any release-specific smoke tests.
3. Publish the GitHub release and PyPI artifacts.
4. From the `kiloloop/agent-estimate` public clone, move the floating public Action tag, for example `git tag -f v0 vX.Y.Z && git push origin v0 --force`, so `uses: kiloloop/agent-estimate@v0` tracks the latest compatible release. Do not perform this step from the private staging clone.
5. Confirm README, examples, and skill docs match the released CLI output and model keys.

## Reporting Bugs

Use the bug report issue template and include:
Expand Down
24 changes: 19 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ No config required — sensible defaults for a 3-agent fleet (Claude, Codex, Gem
```bash
agent-estimate estimate --file tasks.txt
agent-estimate estimate --repo myorg/myrepo --issues 11,12,14
agent-estimate session --agents 3 --rounds 2 --type review
```

## How It Works
Expand Down Expand Up @@ -114,17 +115,17 @@ $ agent-estimate estimate --file tasks.txt
| Best case | 44.7m |
| Expected case | 75.4m |
| Worst case | 117.2m |
| Human-speed equivalent | 572.8m |
| Compression ratio | 7.60x |
| Human-speed equivalent | 608.2m |
| Compression ratio | 8.07x |
| Review overhead (per-task, pre-amortization) | 45m |

## METR Warnings

- **Add known_debt.md as standard protocol memory file**: Estimate (68m) exceeds gpt_5_4 p80 threshold (60m). Consider splitting the task.
- **Write quickstart guide with protocol comparison table**: Estimate (68m) exceeds gemini_3_1_pro p80 threshold (45m). Consider splitting the task.
- **Add known_debt.md as standard protocol memory file**: Estimate (75m) exceeds gpt_5_4 p80 threshold (60m). Consider splitting the task.
- **Write quickstart guide with protocol comparison table**: Estimate (75m) exceeds gemini_3_1_pro p80 threshold (45m). Consider splitting the task.
```

~75 minutes wall-clock versus ~9.5 hours of sequential human work, at an estimated $3.51 fleet cost — plus two flags that the Codex- and Gemini-assigned tasks run past their models' p80 reliability horizons, so you split them or add a checkpoint before dispatching. The same three tasks were later run by real agents; the retro is in the example file. More in [`examples/`](./examples/) — coding S/M, research, documentation, multi-agent.
~75 minutes wall-clock versus ~10.1 hours of sequential human work, at an estimated $3.51 fleet cost — plus two flags that the Codex- and Gemini-assigned tasks run past their models' p80 reliability horizons, so you split them or add a checkpoint before dispatching. The same three tasks were later run by real agents; the retro is in the example file. More in [`examples/`](./examples/) — coding S/M, research, documentation, multi-agent.

## Integrations

Expand Down Expand Up @@ -262,6 +263,19 @@ When `--warm-context` is omitted, the CLI can auto-infer it from `--history-file
if no history file is passed and `./data.json` exists, that file is used as the
default dispatch history source.

### Session estimates

Use `agent-estimate session` for coordinated workflows where multiple agents run
rounds of brainstorm, review, research, documentation, config, or coding work:

```bash
agent-estimate session --agents 3 --rounds 2 --type review
agent-estimate session --agents 4 --rounds 1 --per-round-minutes 25 --format json
```

The command reports wall-clock time, total agent-minutes, coordination overhead,
and per-round breakdowns.

### Calibration

Validate estimates against observed outcomes and build a calibration database:
Expand Down
2 changes: 1 addition & 1 deletion action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ inputs:
required: false
default: '3.12'
version:
description: 'agent-estimate version to install (e.g. "0.7.1"). Omit for latest.'
description: 'agent-estimate version to install (e.g. "0.7.2"). Omit for latest.'
required: false
token:
description: 'GitHub token for issue fetching and PR comments'
Expand Down
2 changes: 1 addition & 1 deletion examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Real input/output examples from production agent dispatches. Every command below
| Implement CLI command with code generation | Coding | M | 2.53x | [coding-m.md](./coding-m.md) |
| Audit cloud infrastructure providers | Research | S | 2.61x | [research.md](./research.md) |
| Write quickstart guide + README | Documentation | S | 2.58x | [documentation.md](./documentation.md) |
| 3-agent parallel session (3 features) | Multi-agent | M×3 | 7.60x | [multi-agent.md](./multi-agent.md) |
| 3-agent parallel session (3 features) | Multi-agent | M×3 | 8.07x | [multi-agent.md](./multi-agent.md) |

## How to read the output

Expand Down
16 changes: 8 additions & 8 deletions examples/multi-agent.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ agent-estimate estimate --file tasks.txt

| Task | Model | Tier | Agent | Base PERT (O/M/P) | Modifiers | Effective Duration | Human Equivalent |
| --- | --- | --- | --- | --- | --- | --- | --- |
| **Implement add-agent CLI command with SPEC.md generation** | coding | M | Claude | 25m / 50m / 90m (E=52.5m) | spec 1.00 x warm 1.00 x fit 1.00 = 1.00 | 52.5m | 190.9m |
| **Implement add-agent CLI command with SPEC.md generation** | coding | M | Claude | 25m / 50m / 90m (E=52.5m) | spec 1.00 x warm 1.00 x fit 1.00 = 1.00 | 52.5m | 226.4m |
| Add known_debt.md as standard protocol memory file | coding | M | Codex | 25m / 50m / 90m (E=52.5m) | spec 1.00 x warm 1.00 x fit 1.00 = 1.00 | 52.5m | 190.9m |
| Write quickstart guide with protocol comparison table | coding | M | Gemini | 25m / 50m / 90m (E=52.5m) | spec 1.00 x warm 1.00 x fit 1.00 = 1.00 | 52.5m | 190.9m |

Expand All @@ -41,8 +41,8 @@ agent-estimate estimate --file tasks.txt
| Best case | 44.7m |
| Expected case | 75.4m |
| Worst case | 117.2m |
| Human-speed equivalent | 572.8m |
| Compression ratio | 7.60x |
| Human-speed equivalent | 608.2m |
| Compression ratio | 8.07x |
| Review overhead (per-task, pre-amortization) | 45m |

### Agent Load Summary
Expand All @@ -59,8 +59,8 @@ agent-estimate estimate --file tasks.txt

### METR Warnings

- **Add known_debt.md as standard protocol memory file**: Estimate (68m) exceeds gpt_5_4 p80 threshold (60m). Consider splitting the task.
- **Write quickstart guide with protocol comparison table**: Estimate (68m) exceeds gemini_3_1_pro p80 threshold (45m). Consider splitting the task.
- **Add known_debt.md as standard protocol memory file**: Estimate (75m) exceeds gpt_5_4 p80 threshold (60m). Consider splitting the task.
- **Write quickstart guide with protocol comparison table**: Estimate (75m) exceeds gemini_3_1_pro p80 threshold (45m). Consider splitting the task.

## What actually happened

Expand All @@ -72,12 +72,12 @@ These three tasks were dispatched to real agents in a production multi-agent wor
| Protocol memory file | Codex | 75.4m | ~20m | Clean merge | Merged |
| Quickstart + README | Claude | 75.4m | ~45m | 2 rounds | Merged |

**Wall clock**: All three ran in parallel. Total elapsed ~90m (bounded by the slowest task). Sequential human equivalent: ~573m (~9.5 hours).
**Wall clock**: All three ran in parallel. Total elapsed ~90m (bounded by the slowest task). Sequential human equivalent: ~608m (~10.1 hours).

**Actual compression**: 6.4x (90m wall clock / 573m human equivalent). The estimate predicted 7.6x — close, because one task (add-agent CLI) needed an extra review round that pushed it past the expected case.
**Actual compression**: 6.8x (90m wall clock / 608m human equivalent). The estimate predicted 8.1x — close, because one task (add-agent CLI) needed an extra review round that pushed it past the expected case.

The METR warnings were useful: Codex's task landed well under the threshold (20m actual vs 60m warning), but the warning correctly flagged that the Gemini-assigned task was near its reliability limit.

## Key takeaway

Multi-agent sessions are where `agent-estimate` delivers the most value. Three agents working in parallel produce **7.6x compression** — ~75 minutes wall clock vs ~9.5 hours of sequential human work. The wave planner automatically assigns tasks to agents, schedules them in parallel, and flags reliability risks via METR warnings. You see the total cost ($3.51), parallelism benefit, and risk before committing compute.
Multi-agent sessions are where `agent-estimate` delivers the most value. Three agents working in parallel produce **8.1x compression** — ~75 minutes wall clock vs ~10.1 hours of sequential human work. The wave planner automatically assigns tasks to agents, schedules them in parallel, and flags reliability risks via METR warnings. You see the total cost ($3.51), parallelism benefit, and risk before committing compute.
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "hatchling.build"

[project]
name = "agent-estimate"
version = "0.7.1"
version = "0.7.2"
description = "Know what an AI task will cost before you run it"
readme = "README.md"
license = "Apache-2.0"
Expand Down
16 changes: 15 additions & 1 deletion skills/estimate/shared/INTENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ Wraps the `agent-estimate` CLI to provide PERT three-point effort estimation wit
| Intent | Command |
|--------|---------|
| Estimate one or more tasks | `agent-estimate estimate ...` |
| Estimate a multi-agent session | `agent-estimate session ...` |
| Validate estimate vs actuals | `agent-estimate validate <observation.yaml> ...` |
| Recompute calibration summary | `agent-estimate calibrate ...` |

Expand All @@ -24,12 +25,25 @@ Accepts exactly one input source:
Optional flags:
- `--config <path>` — path to config YAML with agent definitions
- `--format markdown|json` — output format (default: `markdown`)
- `--review-mode none|standard|complex` — review overhead tier
- `--review-mode none|standard|complex|3-round` — review overhead tier
- `--type coding|brainstorm|research|config|documentation|frontend|app_dev` — task category; omit to auto-detect
- `--spec-clarity <0.3..1.3>` — spec clarity modifier
- `--warm-context <0.3..1.15>` — warm context modifier
- `--agent-fit <0.9..1.2>` — agent fit modifier
- `--title <text>` — report title
- `--verbose` — enable debug logging

If no input source is provided, prompt the user.

### `session`

- `--agents <n>` — number of parallel agents
- `--rounds <n>` — number of sequential rounds
- `--type brainstorm|review|research|documentation|config|coding` — session task type
- `--coordination-overhead <minutes>` — per-round coordination overhead
- `--per-round-minutes <minutes>` — explicit per-agent round duration
- `--format markdown|json` — output format

### `validate`

- Required: observation YAML path
Expand Down
2 changes: 1 addition & 1 deletion skills/estimate/skill.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,6 @@ compatible_runtimes:
- claude-code
- codex
requires_oacp: ">=0.1.0"
public_status: staging
public_status: public
owners:
- kiloloop
11 changes: 10 additions & 1 deletion src/agent_estimate/adapters/config_loader.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
from importlib.resources import as_file, files
from pathlib import Path
from typing import Any
import warnings

import yaml
from pydantic import ValidationError
Expand Down Expand Up @@ -72,7 +73,15 @@ def discover_plugin_profiles() -> list[AgentProfile]:
"""Discover and validate agent profiles from entry points."""
discovered: dict[str, AgentProfile] = {}
for ep in _iter_agent_entry_points():
profile = _load_entry_point_profile(ep)
try:
profile = _load_entry_point_profile(ep)
except ValueError as exc:
warnings.warn(
f"Skipping agent profile entry point {ep.name!r}: {exc}",
RuntimeWarning,
stacklevel=2,
)
continue
discovered[profile.name] = profile
return list(discovered.values())

Expand Down
44 changes: 40 additions & 4 deletions src/agent_estimate/adapters/github_ghcli.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
import json
import subprocess
import time
import warnings
from typing import Callable, Sequence

from agent_estimate.adapters.github_adapter import (
Expand All @@ -14,6 +15,8 @@
)
from agent_estimate.audit import emit_audit_event

_ISSUE_LIST_LIMIT = 1000


class GitHubGhCliAdapter:
"""Fetch GitHub issues through gh CLI commands."""
Expand Down Expand Up @@ -60,7 +63,9 @@ def fetch_issues_by_numbers(self, repo: str, issue_numbers: Sequence[int]) -> li
repo=repo,
issue_number=issue_number,
)
payload = json.loads(output)
payload = _load_json(output, "gh issue view")
if not isinstance(payload, dict):
raise GitHubAdapterError(f"Unexpected gh issue view output: {payload!r}")
issues.append(_parse_issue(payload))
return issues

Expand All @@ -83,7 +88,7 @@ def fetch_issues_by_label(
"--state",
state,
"--limit",
"1000",
str(_ISSUE_LIST_LIMIT),
"--json",
"number,title,body",
]
Expand Down Expand Up @@ -114,9 +119,28 @@ def fetch_issues_by_label(
label=label,
state=state,
)
payload = json.loads(output)
payload = _load_json(output, "gh issue list")
if not isinstance(payload, list):
raise GitHubAdapterError(f"Unexpected gh issue list output: {payload!r}")
if len(payload) >= _ISSUE_LIST_LIMIT:
warnings.warn(
"gh issue list returned the 1000-item limit; results may be truncated",
RuntimeWarning,
stacklevel=2,
)
emit_audit_event(
"api_call",
action="github_issue_list",
outcome="warning",
level="WARNING",
client="gh",
endpoint="gh issue list",
repo=repo,
label=label,
state=state,
result_count=len(payload),
limit=_ISSUE_LIST_LIMIT,
)
return [_parse_issue(raw_issue) for raw_issue in payload]

def fetch_task_descriptions_by_numbers(
Expand All @@ -139,14 +163,26 @@ def fetch_task_descriptions_by_label(


def _run_gh(args: list[str]) -> str:
result = subprocess.run(args, check=False, capture_output=True, text=True)
try:
result = subprocess.run(args, check=False, capture_output=True, text=True)
except OSError as exc:
raise GitHubAdapterError(
"gh CLI not found; install gh or set GITHUB_TOKEN",
) from exc
if result.returncode != 0:
raise GitHubAdapterError(
f"gh command failed ({' '.join(args)}): {result.stderr.strip() or result.stdout.strip()}",
)
return result.stdout


def _load_json(output: str, command: str) -> object:
try:
return json.loads(output)
except json.JSONDecodeError as exc:
raise GitHubAdapterError(f"{command} returned invalid JSON: {exc}") from exc


def _parse_issue(payload: dict[str, object]) -> GitHubIssue:
number = int(payload["number"])
title = str(payload.get("title", ""))
Expand Down
Loading
Loading