feat(memory): event-sourced associative recall + importance tool + arch cleanup#188
Merged
Merged
Conversation
…ch cleanup
Why: memory_recall previously had no reinforcement signal (cold-start every
session) and no way for the LLM to mark salient nodes — limiting it to pure
graph traversal. This change makes recall behavior accumulate across sessions
via an append-only event log, and exposes user/LLM intent through a new
importance tool. Verified per-query lift of MRR +7.4% / R@10 +16.1% on the
expanded 58-query eval fixture.
Core additions
- event_log/: per-session JSONL append-only log (writer / fold / gc / schema /
recall_stats), git-mergeable by design; folds into in-memory RecallStats on
session start
- graph::recall now emits QueryEvent + RecallHit; scoring layer adds
recall_reinforcement_bonus (frequency) and importance_bonus (user intent)
- MemoryImportanceTool: production write path so LLM/user can mark nodes
important; previously the score path read a field that had no writer
- MemorySubsystem::bootstrap: memory crate owns its own orchestration
(canonicalize → gc → fold → install_stats → set_event_log → scan → watch)
instead of agent-server stitching primitives
- MemoryConfig: gc_compress_after_days (90) + gc_archive_after_days (365)
replace hardcoded constants
- ensure_gitignore: auto-managed gitignore rules for memory-events/
Bug fixes found via multi-angle sweep (8 angles × verify)
- compress_file orphan recovery (.jsonl + .jsonl.gz coexistence after crash
between rename and remove caused 2× recall_count inflation)
- compress_file concurrent GC race (per-process pid+nanos tmp suffix
replaces fixed .tmp that two sessions could truncate)
- fold_one_file partial-batch recovery (single IO error no longer drops the
entire file's already-parsed events)
- run_gc read_dir error now counts toward stats.errors instead of silent
zero-return
- compress_file streams via io::copy instead of read_to_end (was OOM-prone
on multi-GB session files)
- ensure_gitignore handles non-UTF8 existing files via lossy decode (was
appending rules indefinitely on every session start)
Architecture cleanup (arch_check)
- EmotionalTag → ImportanceTag rename (13 symbols + serde wire tag) — the
field was always literally i8 importance with no emotion model
- install_recall_stats now &mut self (compile-time prevents post-Arc race)
- gitignore moved out of store/ (store layer should not reach project root)
- MemoryGraph::record_event made pub (delete record_emotional_tag wrapper —
inconsistent with recall.rs use-site construction pattern)
- store/event_log_* + recall_stats moved to event_log/ submodule (separate
concerns: SQLite vs event sourcing)
- recall.rs (311 lines) split into recall/{mod,anchors,emit}.rs
- queries_node.rs (222 lines) split into queries_node/{mod,read,write}.rs
- Dead code removed: CoRecall emit (no fold consumer), NearMiss schema
(never emitted), runner::run() (replaced by run_with_warmup(0))
Testing
- 47 new fixture .md files + 48 new ground-truth queries (4 domain clusters:
postgres ops, ci/cd, auth/security, observability)
- per-query warmup A/B framework in eval (cold vs warm 5x/20x vs Imp +5/+10
vs Per-Q 5x)
- 8 GC recovery tests (orphan / concurrent tmp / streaming / read_dir error
/ archive collision / fold partial recovery / non-UTF8 gitignore)
- 6 importance tool tests (metadata / validation / e2e write+fold / repeated
tagging / silent drop without event log)
- 2 associative recall integration tests (reinforcement persists across
sessions; importance lifts ranking)
Verification
- bazel test //... — all passing (207 memory crate + 6 other targets)
- bazel build //... --config=clippy --config=rustfmt — clean
- 200-line file limit — 0 violations
- eval gates: cold R@5=0.579, MRR=0.753; per-Q lift +7.4% MRR, +16.1% R@10
Artifact from initial fixture-generation script where an empty slug produced a 0-byte file named literally ".md" in the fixtures directory. Not referenced by fixture.rs registry; harmless but noise.
gc_propagates_read_dir_permission_error uses std::os::unix::fs::PermissionsExt to chmod 000 a directory and verify run_gc surfaces the read_dir error. The unix API is gated out on Windows, breaking the Windows CI build. Skip the test on non-unix targets; the production code path (run_gc → fs::read_dir → io::ErrorKind::PermissionDenied → stats.errors += 1) is platform-agnostic and covered by unix runs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
MemoryImportanceToollets LLM/user mark salient nodes. Previouslyemotional_bonusread a field that had no writer — the path was dead.EmotionalTag → ImportanceTagrename (no emotion model, only i8 importance),event_log/extracted fromstore/,MemorySubsystem::bootstrapowns its own lifecycle.Changes
Core additions
loopal-memory/src/event_log/: writer / fold / gc / schema / recall_stats — per-session append-only JSONL, git-mergeable, transparent .gz readgraph::recall::emit_events: writes QueryEvent + RecallHit per callgraph::score::recall_reinforcement_bonus+importance_bonus: ranking signal from accumulated statstools::importance::MemoryImportanceTool: production write path (1..=10 salience scale)MemorySubsystem::bootstrap(memory_dir, session_dir, events_dir, sid, gc_compress_days, gc_archive_days): lifecycle moved out ofagent-server::build_subsystemMemoryConfig:gc_compress_after_days(90),gc_archive_after_days(365)init::ensure_gitignore: auto-managed gitignore rules formemory-events/Bug fixes (8-angle workflow sweep, 13 confirmed → 6 unique)
compress_fileorphan recovery:.jsonl+.jsonl.gzcoexistence after crash → fold double-count (2× recall_count inflation)compress_fileconcurrent GC race: per-process.tmp.<pid>.<nanos>suffix replaces fixed.tmpfold_one_filepartial-batch recovery: single IO error no longer drops already-parsed eventsrun_gcread_direrror: now counts stats.errors instead of silent zero-returncompress_file:io::copystream replacesread_to_end(was OOM on multi-GB files)ensure_gitignore:from_utf8_lossyhandles non-UTF8 existing files (was appending indefinitely)Architecture cleanup (arch_check)
EmotionalTag → ImportanceTagrename (13 symbols + serde wire tag)install_recall_stats(&mut self)(was&self, allowed post-Arc race)gitignoremoved out ofstore/(storage shouldn't reach project root)MemoryGraph::record_eventmadepub+ deleterecord_emotional_tagwrapperrecall.rs(311 lines) →recall/{mod,anchors,emit}.rsqueries_node.rs(222 lines) →queries_node/{mod,read,write}.rsEval framework
Tests
Test plan