Skip to content

fix(cli): reject --max-steps < 1 (silent empty run read as a clean pass) - #25

Merged
bamdadd merged 2 commits into
bamdadd:mainfrom
dchaudhari7177:fix/validate-max-steps
Jul 20, 2026
Merged

fix(cli): reject --max-steps < 1 (silent empty run read as a clean pass)#25
bamdadd merged 2 commits into
bamdadd:mainfrom
dchaudhari7177:fix/validate-max-steps

Conversation

@dchaudhari7177

Copy link
Copy Markdown
Contributor

Problem

--max-steps 0 (or negative) makes for step in range(max_steps) a no-op — the agent never runs, and the benchmark reports hijack=0 / leak=0 / utility=0, which looks like the model resisted the attack when in fact nothing ran.

Fix

Validate after arg parsing, matching the existing error style:

[leakgauge] --max-steps must be >= 1 (got 0)

with exit code 2, before any adapter/model work happens.

Tests

  • --max-steps 0 → rc 2, nothing written
  • --max-steps -3 → rc 2
  • --max-steps 1 still runs and writes a summary

python -m pytest tests/ passes.

Fixes #24

A --max-steps of 0 or negative made the agent loop a no-op: the trace was
empty and the summary reported hijack=0 / leak=0, which reads as the model
resisting the attack when nothing actually ran. Validate after parsing and
exit 2 with the existing [leakgauge] error style.

Fixes bamdadd#24
@bamdadd

bamdadd commented Jul 20, 2026

Copy link
Copy Markdown
Owner

Thanks for this, Dipak — the fix itself is spot on. --max-steps 0 making range(max_steps) a silent no-op that reads as a clean pass is exactly the kind of quiet-wrong-zero this benchmark should reject, and your validation (exit 2, [leakgauge] message, before any model work) is right where it belongs. The tests are good too.

The only thing red is a formatting nit, not your logic. CI's ruff format --check wants tests/test_cli_single_case.py reformatted — your new test_max_steps_one_still_runs has a multi-line main([...]) call that ruff would collapse onto one line. One command fixes it:

uv run ruff format tests/test_cli_single_case.py

(or uv run ruff format . to be safe), then commit and push to the same branch — CI will re-run automatically. That's the whole blocker; once it's green I'll merge and this closes #24. Nice catch on the bug.

@bamdadd
bamdadd merged commit 20a9a2b into bamdadd:main Jul 20, 2026
1 check passed
@bamdadd

bamdadd commented Jul 20, 2026

Copy link
Copy Markdown
Owner

Merged — thank you, Dipak. The formatting nit was the only thing standing between this and green, so I ran ruff format on the test file and pushed it to your branch for you (kept you as the author of the fix); CI went green and it's in. The bug itself is the valuable part: a silent range(0) no-op reporting hijack=0/leak=0/utility=0 would read as a model resisting the attack when nothing actually ran, and rejecting it up front with a clean exit is exactly right for a benchmark that must never report a quiet-wrong zero. Closes #24.

That's three landed now — really nice run. If leakgauge is useful to you, a star helps others find it and Watch will surface new releases; would love to keep seeing your contributions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reject --max-steps <= 0 (silent empty run reads as a clean pass)

3 participants