Add usage-driven knowledge-base curation loop#2
Open
ryanshepps wants to merge 2 commits into
Open
Conversation
Close the retrieval loop for the shared code/write knowledge bases: record trajectories and misses, then turn them into reviewed curation edits with an automatic prompt so nothing depends on remembering to run a review. - Enrich fetch telemetry (log-knowledge-fetch.sh) with session_id + cwd so fetches group into trajectories and co-fetch pairs are detectable. - kb_gap.py: code/write skills record a gap when KB traversal finds no fit, appended to <domain>/.stats/gaps.jsonl. - kb_reflect.py: mine fetches + gaps into candidate edits (new leaves from gaps, missing related: links from co-fetches, priority signals, cold leaves) against a last_reviewed watermark; --mark-reviewed advances it. - knowledge-reflect skill + /knowledge-reflect command: present candidates, apply only on approval, regen+validate, advance watermark. Human approves every write; fetch count treated as a weak proxy; foundational cold leaves protected from demotion. - kb-reflect-nudge.sh (SessionStart): prompt automatically when pending gaps cross a threshold or a review is stale; silent otherwise. Dual-agent + KB validators pass; settings template renders valid JSON in both .work branches.
brainstorming independently traverses the code KB with the same protocol as the code skill, so genuine misses during a brainstorm were going unrecorded. The code-* spec/review skills don't fetch the KB directly (they rely on the auto-triggering code skill) and are intentionally left unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Turns the shared knowledge base from a static, hand-curated library into a
self-improving loop, inspired by the ACE (Agentic Context Engineering) pattern
in Lilian Weng's harness engineering
post: context as an evolving playbook maintained by a Generator → Reflector → Curator
loop, rather than a corpus you remember to prune by hand.
Today the KB has the Generator (fetches happen) and raw logging, but the loop is
open — nothing distills usage into curation. This closes it, and makes the
prompt to curate automatic so it never depends on memory. Every write stays
behind human approval.
What changes
Record trajectories (Reflector needs episodes, not events)
log-knowledge-fetch.shnow stamps each fetch withsession_id+cwd, sofetches group into per-task trajectories and co-fetch pairs become detectable.
Backward-compatible with existing
{ts,path,domain}lines.Record misses (weakness mining / negative results)
kb_gap.py— thecode/writeskills call it when KB traversal finds noadequate leaf, appending to
<domain>/.stats/gaps.jsonl. The miss is thesingle most valuable signal for growing the corpus, and today it's invisible.
Reflect + curate (the loop)
kb_reflect.pymines fetches + gaps against alast_reviewedwatermark intofive candidate buckets: new-leaf (from gaps), missing
related:links (fromco-fetches), priority signals, cold demote/delete, and cold-but-foundational
(keep).
--mark-reviewedadvances the watermark. Pure reporting — never mutates.knowledge-reflectskill +/knowledge-reflectcommand present the buckets,apply only what you approve, regen + validate, then advance the watermark.
Prompt automatically (the trigger)
kb-reflect-nudge.shruns onSessionStart: if pending gaps cross a threshold(≥3) or a review is stale (≥14d), it prints a one-line nudge into context;
silent otherwise. This is what replaces "remembering to run a review."
Guardrails (straight from the article's Future Challenges)
the human disposes. No unattended writes, no auto priority changes.
"keep", never as demotion candidates. Cold ≠ useless.
is substantial (≥30 events), so early noise can't drive deletions.
Producer/consumer note
I consolidated the originally-discussed Stop+SessionStart hook pair: the
producer is the
code/writeskills writing gaps live during a session, andthe consumer is the
SessionStartnudge. A separate Stop hook would onlyre-derive signals the skill already wrote — YAGNI. All heavy reasoning lives in
kb_reflect.py/knowledge-reflect, keeping session start fast.Dual-agent
Shared assets (
kb_gap.py,kb_reflect.py,knowledge-reflectskill, gaplogging in
code/write) work for both Claude and Codex. TheSessionStartnudge is Claude-only (Codex's hook set is narrower); on Codex the loop degrades
gracefully — gaps still log,
/knowledge-reflectstill runs on demand.Verification
validate_dual_agent_repo.py✓ ·validate_kb.pycode + write ✓settings.json.tmplrenders valid JSON in both.workbranches (1 vs 2SessionStart entries)
kb_reflect.py/kb_gap.py/ both hooks tested against synthetic telemetry:gap watermarking, co-fetch detection on unlinked pairs, thin-log suppression,
nudge threshold + staleness + silent-when-empty, enriched-JSON logging, and
non-knowledge reads correctly ignored.