Skip to content

Add usage-driven knowledge-base curation loop#2

Open
ryanshepps wants to merge 2 commits into
masterfrom
knowledge-reflect-loop
Open

Add usage-driven knowledge-base curation loop#2
ryanshepps wants to merge 2 commits into
masterfrom
knowledge-reflect-loop

Conversation

@ryanshepps

Copy link
Copy Markdown
Owner

Turns the shared knowledge base from a static, hand-curated library into a
self-improving loop, inspired by the ACE (Agentic Context Engineering) pattern
in Lilian Weng's harness engineering
post: context as an evolving playbook maintained by a Generator → Reflector → Curator
loop, rather than a corpus you remember to prune by hand.

Today the KB has the Generator (fetches happen) and raw logging, but the loop is
open — nothing distills usage into curation. This closes it, and makes the
prompt to curate automatic so it never depends on memory. Every write stays
behind human approval.

What changes

Record trajectories (Reflector needs episodes, not events)

  • log-knowledge-fetch.sh now stamps each fetch with session_id + cwd, so
    fetches group into per-task trajectories and co-fetch pairs become detectable.
    Backward-compatible with existing {ts,path,domain} lines.

Record misses (weakness mining / negative results)

  • kb_gap.py — the code/write skills call it when KB traversal finds no
    adequate leaf, appending to <domain>/.stats/gaps.jsonl. The miss is the
    single most valuable signal for growing the corpus, and today it's invisible.

Reflect + curate (the loop)

  • kb_reflect.py mines fetches + gaps against a last_reviewed watermark into
    five candidate buckets: new-leaf (from gaps), missing related: links (from
    co-fetches), priority signals, cold demote/delete, and cold-but-foundational
    (keep). --mark-reviewed advances the watermark. Pure reporting — never mutates.
  • knowledge-reflect skill + /knowledge-reflect command present the buckets,
    apply only what you approve, regen + validate, then advance the watermark.

Prompt automatically (the trigger)

  • kb-reflect-nudge.sh runs on SessionStart: if pending gaps cross a threshold
    (≥3) or a review is stale (≥14d), it prints a one-line nudge into context;
    silent otherwise. This is what replaces "remembering to run a review."

Guardrails (straight from the article's Future Challenges)

  • Reward hacking — fetch count is a weak proxy (Goodhart); the loop proposes,
    the human disposes. No unattended writes, no auto priority changes.
  • Diversity collapse — foundational (p1-2) cold leaves are surfaced as
    "keep", never as demotion candidates. Cold ≠ useless.
  • Thin-data honesty — cold demote/delete is suppressed until the fetch log
    is substantial (≥30 events), so early noise can't drive deletions.

Producer/consumer note

I consolidated the originally-discussed Stop+SessionStart hook pair: the
producer is the code/write skills writing gaps live during a session, and
the consumer is the SessionStart nudge. A separate Stop hook would only
re-derive signals the skill already wrote — YAGNI. All heavy reasoning lives in
kb_reflect.py/knowledge-reflect, keeping session start fast.

Dual-agent

Shared assets (kb_gap.py, kb_reflect.py, knowledge-reflect skill, gap
logging in code/write) work for both Claude and Codex. The SessionStart
nudge is Claude-only (Codex's hook set is narrower); on Codex the loop degrades
gracefully — gaps still log, /knowledge-reflect still runs on demand.

Verification

  • validate_dual_agent_repo.py ✓ · validate_kb.py code + write ✓
  • settings.json.tmpl renders valid JSON in both .work branches (1 vs 2
    SessionStart entries)
  • kb_reflect.py / kb_gap.py / both hooks tested against synthetic telemetry:
    gap watermarking, co-fetch detection on unlinked pairs, thin-log suppression,
    nudge threshold + staleness + silent-when-empty, enriched-JSON logging, and
    non-knowledge reads correctly ignored.
  • No inline comments; module docstrings only (matches existing scripts).

Close the retrieval loop for the shared code/write knowledge bases: record
trajectories and misses, then turn them into reviewed curation edits with an
automatic prompt so nothing depends on remembering to run a review.

- Enrich fetch telemetry (log-knowledge-fetch.sh) with session_id + cwd so
  fetches group into trajectories and co-fetch pairs are detectable.
- kb_gap.py: code/write skills record a gap when KB traversal finds no fit,
  appended to <domain>/.stats/gaps.jsonl.
- kb_reflect.py: mine fetches + gaps into candidate edits (new leaves from
  gaps, missing related: links from co-fetches, priority signals, cold leaves)
  against a last_reviewed watermark; --mark-reviewed advances it.
- knowledge-reflect skill + /knowledge-reflect command: present candidates,
  apply only on approval, regen+validate, advance watermark. Human approves
  every write; fetch count treated as a weak proxy; foundational cold leaves
  protected from demotion.
- kb-reflect-nudge.sh (SessionStart): prompt automatically when pending gaps
  cross a threshold or a review is stale; silent otherwise.

Dual-agent + KB validators pass; settings template renders valid JSON in both
.work branches.
brainstorming independently traverses the code KB with the same protocol as
the code skill, so genuine misses during a brainstorm were going unrecorded.
The code-* spec/review skills don't fetch the KB directly (they rely on the
auto-triggering code skill) and are intentionally left unchanged.
@connor1219

Copy link
Copy Markdown
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants