v4.9.0 — non-Bash error detection, verified scrubbing, restored diagnostics - #31
Merged
Conversation
…etection, by-design file exclusions, workflow trigram filter One real session produced 190 proposals of which 189 were junk. Three root causes, all fixed: 1. observe_v3.py flagged is_error=True whenever the output text contained "error"/"failed" keywords — including inside the CONTENT of successful Reads (e.g. source code with 'throw new Error'). Proof: err_msg was identical to sample_output on the junk proposals. Now detection is (a) structural harness failure shapes for every tool (dict error field, tool_use_error marker, start-of-output Error/InputValidationError) and (b) hard failure markers (Permission denied, command not found, Traceback, npm ERR!, ...) only for Bash, whose output is a run result rather than arbitrary file content. 2. user_correction counted any file edited 3+ times, but journals and control panels (hot.md, brief.md, working-memory.md, MEMORY.md, CLAUDE.md, daily summaries, wiki dirs) are re-edited BY DESIGN during end-of-day flows. They are now excluded via BYDESIGN_RE. 3. workflow_chain proposed every tool trigram repeated 2+ times, which is dominated by generic coding activity (Bash>Bash>Bash x380 in one session, 154 junk proposals). Trigrams now require 5+ repeats, >= 2 distinct tools, and at least one non-generic (MCP/custom) tool. The learner also re-validates the legacy is_error flag against hard markers (isRealError) because observations written by older observers remain contaminated, and PATTERN 5 (agent errors) uses the same validation instead of keyword sniffing. A/B on a real 8000-line observation window: 134 proposals with the old code -> 17 with this fix, and the 17 survivors are all legitimate signals (3 genuine tool failures, 7 genuinely iterated source files, 7 repeated MCP workflow sequences). Co-Authored-By: Claude Fable 5 <[email protected]>
…e diagnostics Rounds out the two contributed PRs (#29 juanparisma, #22 Sergio-LPA) and clears the debt they surfaced. Non-Bash tool failures were flagged by nobody. #29 correctly stopped scanning file CONTENT for error words, but restricting hard markers to Bash left real Edit/Read failures ("String to replace not found", "File does not exist") undetected — they are not harness-shaped either, so the structural branch also missed them. Detection now keys on shape: a failure returns a bare verdict string, a success returns a payload serialising from { or [, and payloads are never scanned — that scan is the bug #29 exists to fix. Bounded by length as well. Zero false positives over 7,931 real non-Bash observations. The learner discarded its own diagnostics. The node block ended in 2>/dev/null, swallowing lock-contention and failed-upsert messages and overriding the SINAPSIS_DEBUG redirect, so the debug switch produced nothing for the 95% of the script inside that block. Diagnostics append to _session-learner.log. Secret scrubbing had never been tested. Test Group 4 imported a script that cannot be imported, then looked for a function that does not exist; every assertion returned LOAD_FAIL and printed as SKIP, so "11/11 passed" covered nothing. It now feeds the hook a real secret and asserts what reached disk, across five formats, with a control proving the scrubber is not simply destroying all output. The fixtures are synthetic — nothing here was ever a live credential, and two of them are the vendors' own published examples — but each prefix is assembled at runtime so no complete token-shaped literal lands in the tree. A scanner cannot tell a fixture from a leak: written out whole, these block the push and scare anyone who greps the repo. What reaches the observer is byte-identical. New tests/test-error-detection.sh (10 tests) pins both halves of the fix. Documented for future test authors: the observer resolves its config dir via expanduser("~"), which on Windows reads USERPROFILE, not HOME — redirecting HOME alone leaks synthetic observations into real learning data. Docs: CHANGELOG v4.9.0, README header/badges/What's New, installer banners, and FEATURES notes on the two accepted limits of the cross-OS key (posix collapses macOS with Linux; the name key can fuse unrelated same-named projects, which is why it now stamps and logs). Suite: 217 tests across 16 files, all registered in CI. Co-Authored-By: Claude Opus 5 <[email protected]>
commands/dashboard-sinapsis.md told every user to run a script under a home directory that is not theirs, via a path that exists on no installation at all: the installer lays the generator down as ~/.claude/skills/_generate-dashboard.py and the template beside it as _dashboard-template.html, not under any hooks/ folder. The command now matches the installer and the convention the other commands already follow. The Windows-path fixture in test-eod-gather.sh loses a real username it never needed; the assertion only ever read the basename. Co-Authored-By: Claude Opus 5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release v4.9.0. Two contributed PRs plus the debt they surfaced.
Contributions
@juanparisma — #29, learner false positives. The learner scanned file content for error words, so every file that merely contained the string "error" became a proposal. Verified over a real corpus: 1,074 flags down to 22. His commit rides in this branch, so #29 closes as merged when this lands.
@Sergio-LPA — #22, cross-OS registry (already on
main, c072380). The same project seen from macOS and from Windows produced two registry entries. His fix collapses them by remote, with a name-key fallback guarded so two same-named projects on one machine stay apart, plus a migration pass that heals registries written before the feature. Two months of dogfooding across Mac and PC before he opened it, and he took two rounds of review with weeks in between without complaint.The debt those two surfaced
Non-Bash tool failures were flagged by nobody. #29 was right to stop scanning payloads, but restricting hard markers to Bash left real
Edit/Readfailures undetected. Detection now keys on shape: a failure returns a bare verdict string, a success returns a payload serialising from{or[, and payloads are never scanned — that scan is the bug #29 exists to fix. Zero false positives over 7,931 real non-Bash observations.The learner threw away its own diagnostics. The
node -eblock ended in2>/dev/null, which swallowed lock-contention and failed-upsert messages and overrode theSINAPSIS_DEBUGredirect — so the debug switch produced nothing at all for the 95% of the script living inside that block. Diagnostics now append to_session-learner.log. @Sergio-LPA independently found this while working on #22 and offered to send a PR; it was already in flight here.Secret scrubbing had never actually been tested. Test Group 4 imported a script that cannot be imported, then looked for a function that does not exist. Every assertion returned
LOAD_FAILand printed as SKIP, so the headline "11/11 passed" covered nothing. It now drives the hook through the front door — feed it a payload carrying a secret, assert what reached disk — across five formats, with a control proving the scrubber is not simply destroying all output.Housekeeping
Token-shaped test fixtures are assembled at runtime instead of written out whole. None of them was ever a live credential and two are the vendors' own published examples, but a scanner cannot tell a fixture from a leak: written out whole they block the push and alarm anyone who greps the tree. What reaches the observer is byte-identical.
Two hardcoded personal paths went with them.
commands/dashboard-sinapsis.mdtold every user to run a script under a home directory that is not theirs, via a path that exists on no installation at all — the installer lays the generator down as~/.claude/skills/_generate-dashboard.py.Tests
217 across 16 files, all registered in CI.