Make coverage mean coverage - #417
Merged
Merged
Conversation
kensa coverage meant two unrelated things depending on whether --framework was present: without it, a deprecated alias for the mechanism listing; with it, the framework coverage report. It also warned that the name would change meaning "in v0.2", which shipped on 2026-05-28 and was still being announced at v0.10.0 as though it were ahead of us. The name now selects one operation. Running coverage without --framework is a usage error naming the flag, not a silent fallback that returns an unrelated document with exit 0 to a script that asked for coverage. mechanisms is the only name for the mechanism listing. --rules-dir stays conditional and both directions are pinned by tests. A framework whose controls kensa reads from the corpus needs it; one that ships an embedded objective catalog does not, because kensa can already enumerate its controls. Today that is nist_800_171. The asymmetry is surprising enough that a later tidying could plausibly remove the working path, so it is contract rather than incident. The warning and its KENSA_NO_REPURPOSE_WARNINGS suppression knob are gone from the runtime, help, completion, guide, and manpage. KENSA_NO_DEPRECATION_WARNINGS is a separate knob and is untouched. Two corrections fall out of removing the alias. Coverage help now shows --quiet, --full and --from-scan: the report always accepted them and the alias help was what hid them. The generated manpage now has a coverage section, which it lacked because the alias would have duplicated one flag table under two headings.
Review found four assertions weaker than the criteria they were mapped to, false spec traceability, comments that still described the alias, and an uncounted command failure in the smoke script. Traceability. Every test in coverage_report_test.go carried a sequentially numbered annotation against the deprecated rename spec and a shifted one against the framework draft. The numbering was mechanical, not evidence: the bad-format test claimed the missing-rules-dir criterion, the JSON test claimed the mechanisms rejection, and so on. Each annotation now names the criterion its test actually proves, or is gone. Two tests survived that asserted a warning nobody emits any more and passed only because the new error text happens to lack the searched words; both are deleted. The deprecated spec keeps the two criteria kensa still satisfies and declares its measured 25% rather than borrowing credit from unrelated tests. Its threshold has to be that number: zero reads as unset. Assertions. Help now checks exit codes, not just stdout equality. The control list is compared in order, because the approved document is ordered and a map accepted either arrangement. The objective digest must be 64 lowercase hex, not any hex. The environment baseline genuinely unsets the variable instead of setting it empty, which code using LookupEnv can tell apart. Top-level help compares whole command rows, so a swapped description fails. A new test reads the manpage, guide, changelog and spec lifecycle, comparing whole flag names: a substring check accepted --fullx for --full. Comments. The orphaned migrate line under the coverage synopsis, the function comment describing dispatch fall-through, the unconditional --rules-dir description, and the package comment denying any objective catalog all still contradicted the shipped behavior. The smoke script contained echoecho. With set -uo pipefail and no -e the shell reported command not found, continued, and still printed 227 passed.
Two review findings, both of them mine. The traceability correction reported in the previous commit was not in it. While reverting a mutation I ran git checkout on mechanisms_test.go, which restores from HEAD and discarded the real edit along with the mutation. I then reported a measurement taken before that wipe, so the deprecated spec was described at 25% covering two criteria while it actually stood at 50% covering four, and the stale warning-suppression test was still present. The lesson is narrow and mechanical: restore a mutated file from the copy made before mutating it, never from the index, when the file also carries work in progress. That test is now deleted. It asserted that an environment variable silences a warning; neither the variable nor the warning exists, and it passed only because the current error text happens to lack the words it searched for. TestRunMechanisms_NoWarning moves to AC-04, the criterion it proves, and TestRunMechanisms_HelpExitsZero loses its annotation because AC-07 requires both subcommands and it exercises one. The remaining references to deleted tests in the deprecated spec now read as supersession statements rather than dangling attributions. The flag assertions compared help, completion and the manpage against each other, so they agreed on a set that nobody approved: adding --rogue to all three passed every criterion and manpage-check. Each now compares against one approved literal set, in both directions, so an extra flag fails as loudly as a missing one.
AC-02's expected output names the exact seven-flag set, but its test only compared the three help invocations against each other. An extra flag appeared in all three identically, so it agreed with itself and passed; only the completion criterion caught it. Both criteria now call one shared assertion against the approved literal set, so each fails on its own. The rest is comments that still described the removed alias as current. The direct-handler missing-framework test claimed the dispatch criterion while calling the handler directly, and its comment said dispatch falls through to the mechanism alias. It keeps its value as a unit regression and loses the mapping: the criterion is about dispatch, and a runCLI test proves that. The heading for a deleted test was left behind. The hasFrameworkFlag comment still described routing coverage between report and alias modes, when the helper now exists so the mechanisms rejection agrees with what the coverage flagset would accept. The mechanisms-rejection test names both criteria it satisfies and why they overlap. mechanisms_test.go called itself tests for the rename and the alias. The one-sided help test no longer claims the historical criterion that requires both commands. quiet_test.go pointed at an alias test deleted with the alias, and now points at the coverage case, which has the flag.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
kensa coveragemeant two unrelated things depending on whether--frameworkwas present: without it, a deprecated alias for the mechanism listing; with it,
the framework coverage report. It also warned that the name would change meaning
"in v0.2" — a version tagged 2026-05-28, still being announced as future at
v0.10.0, describing a flip that had already partly happened behind a flag.
The name now selects one operation and flags only refine it.
kensa coverage--framework, exit 2kensa coverage --helpkensa coverage --framework X --rules-dir Dkensa coverage --framework nist_800_171--rules-dirkensa mechanismskensa mechanisms --framework XBare coverage never falls back. A script relying on it for the mechanism list
must call
kensa mechanisms; it previously exited 0 and now exits 2.--rules-dirstays conditional, and both directions are pinned by tests: aframework whose controls kensa reads from the corpus requires it, and one that
ships an embedded objective catalog does not, because kensa can already
enumerate its controls. Today that is
nist_800_171. The asymmetry issurprising enough that a later tidying could plausibly break the working path.
The warning and
KENSA_NO_REPURPOSE_WARNINGSare gone from the runtime, help,completion, guide and manpage. Setting the variable now does nothing.
KENSA_NO_DEPRECATION_WARNINGSis a separate knob and is untouched.Two corrections that fall out of removing the alias
--quiet,--fulland--from-scan. The reportalways accepted them; the alias help listed only the mechanism flags, which is
why a spec and two tests concluded coverage had no
--quiet.alias, it would have duplicated one flag table under two headings.
Verification
go test ./...go test -race ./cmd/kensa ./internal/coveragemake lintspecter check --strictmake spec-sync/spec-coverage-strictmake manpage-checkmake cli-smokemake docs-check/docs-style-syncmake comment-lint,gofmt, both diff checksPre-fix, the flip-specific criteria failed on the base for their stated reasons:
bare coverage exited 0 printing mechanisms (AC-01), its help was the alias help
(AC-02), the warning function and env read were live (AC-06), completion omitted
four real flags (AC-07), the manpage had no coverage section (AC-08), and the
missing-framework cases entered alias mode (AC-09).
AC-03, AC-04 and AC-05 are regression contracts, not flip criteria: they pin
behavior that already worked on the base and must survive the change. They
passed before and after, which is the point of having them.
Fifteen mutations were applied across two rounds and each killed its target,
including a renamed JSON key caught by a raw key-set comparison rather than a
typed decode, and a renamed manpage flag that a substring check had accepted.
Scope
CLI dispatch, help, completion, docs and specs. No change to
api/, the engine,handlers, coverage or NIST calculations, report schemas, mappings, rule loading,
corpus,
VERSION, or packaging. The coverage document and its key sets areunchanged, locked by a test.
The old rename spec is marked
deprecatedwith a dated note pointing at the newapproved spec.
cli-framework-coverage,cli-quietandcli-manpagekeepstatus: draft— their result semantics are not promoted; only clauses thatcontradicted the shipped runtime carry dated supersession notes.
Out of scope, recorded
The manpage generator omits the registered
recoverandcompletioncommandswhile its draft spec says it covers every registered subcommand. Pre-existing and
left for a separate documentation slice.