fix(observability): stop reporting degraded and failed states as fine - #216
Conversation
Three things that were quietly invisible. doctor reported `semantic search: off` as severity "ok" — a green tick. On the 784-note reference vault, turning the semantic leg on moved recall@1 from 40% to 60% and MRR from 0.42 to 0.64, so the tick was calling "materially worse recall than you could have" healthy. Now a warning that names the fix. compliance.py swallowed every OSError with a bare `return`, so "the log is being written" and "every write has failed for a week" looked identical from outside. That is how the Windows rotation bug (#202) hid for a full release. Failures now leave a labelled breadcrumb in the hook failure log doctor already reads. A missing log on a fresh machine stays silent — that is not a failure. The retrieval eval set goes from 5 labelled cases to 30. Five sounds adequate until you notice each case is worth 20 percentage points: a real 10% gain could not register, and did not — #193 scored identically on five cases while individual ranks moved underneath it. Queries are phrased the way an agent asks, avoid echoing their target's title words, and were authored from note contents before anything was measured. Baseline: recall@1 60.0%, recall@5 83.3%, MRR 0.704 — recall@5 far above what the 5-case set implied, which is the underpowering made visible. Two known misses are kept rather than relabelled: a near-duplicate note outranking its target (a vault problem for consolidate/frontier), and a genuine gap where the note says "production-grade, hardened" and the query says "quality bar". Co-Authored-By: Claude Opus 5 <[email protected]>
|
Warning Review limit reached
Next review available in: 13 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (5)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| place `omind doctor` looks. Imported lazily: hooks imports this module. | ||
| """ | ||
| try: | ||
| from omind.hooks import _record_failure |
2 participants
Items 3, 4 and 6 of 6 — the last of the follow-up batch. Ships in v8.1.0.
All three are the same bug in different clothes: a real problem that doesn't read as one.
1.
doctorgave a green tick to degraded searchSeverity
ok. But on the 784-note reference vault, turning that leg on moved recall@1 40% → 60% and MRR 0.42 → 0.64. The tick was reporting "your recall is materially worse than it could be" as healthy — the same shape as an index that silently stopped updating (#210). Now[!], naming the fix.2.
compliance.pyswallowed errors with nothing to readEvery failure path returned silently so the guard hook can never raise into the agent. Correct — and it made "the compliance log is being written" and "every write has failed for a week" indistinguishable from outside.
That is precisely how #202 hid for a full release: a
PermissionErrorabsorbed by a bareexcept OSError: return. Failures now leave a labelled breadcrumb in the size-capped hook failure logomind doctoralready reads.FileNotFoundErroron a fresh machine stays silent, because that isn't a failure.3. The eval set was too small to detect its own blind spot
Five labelled cases means each is worth 20 percentage points. A change genuinely improving recall by 10% could not register — and didn't: #193 scored identically on five cases while individual ranks moved several places underneath. Five cases cannot tell "no effect" from "an effect this instrument cannot see."
Now 30 cases. Method, since an eval set written by the same author who benefits from it is worth scrutinising:
Baseline: recall@1 60.0%, recall@5 83.3%, MRR 0.704.
Note recall@5 is far higher than the 5-case set implied (60%). The old number wasn't measuring quality so much as sampling noise — the underpowering, made visible.
Two misses kept on purpose
consolidateandgraph frontierexist to findRelabelling either would make the number prettier and the instrument worse.
Gates
ruff check .·mypy src(strict) ·pytest(891 passed) ·pip-audit— green locally.🤖 Generated with Claude Code