Migration safety: rebuild FTS after incompatible-index wipe, fail fast on embedder feature gap#210
Merged
Merged
Conversation
Opening a store over a foreign/incompatible tantivy index (e.g. the retired Python impl's index with a different schema) makes open_or_recreate wipe it to a fresh empty index. rebuild_fts had no call sites, so all indexed history was silently lost and never repopulated. open_dir now surfaces whether it took the recreate path; HistoryStore::open repopulates any index that was recreated or is empty while its source table still holds rows, from the DB, synchronously (small corpus), logging counts. Co-Authored-By: Claude Opus 4.8 <[email protected]>
create_embedder's error (e.g. fastembed without the local-embed extra) was swallowed by .unwrap_or(None) at the async_main call site, so startup proceeded to open and wipe the FTS and only died later in create_projectors under a misleading Discord-token hint. Extract a testable resolve_embedder seam and call it in run_inner BEFORE Familiar::load_from_disk opens the store: on error, log the real message (which names the fix) and exit 1. The embedder is threaded into async_main. Co-Authored-By: Claude Opus 4.8 <[email protected]>
…o 30s recv_wake's 1s timeout is only a hang-guard (every caller .expects Some; none asserts None), but the runner does several spawn_blocking DB round-trips before publishing the wake, which can outlast 1s on a loaded current-thread test runtime — turning the guard into a false failure (~67% under CPU load). Raise the bound to 30s; a genuinely-stuck test still fails, the race is gone. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Co-Authored-By: Claude Opus 4.8 <[email protected]>
CassidyPrather
approved these changes
Jul 17, 2026
CassidyPrather
pushed a commit
that referenced
this pull request
Jul 18, 2026
Union of #210's migration-safety work with this branch's issue fixes. Only textual conflict was the run.rs test-import list; full suite, clippy -D warnings, and fmt verified on the merged tree. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_017hGpXhasxotWvHbXVY3VC3
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three defects found while verifying the migration of a live instance (Sapphire, Python-era data) onto the Rust port. All are general port fixes — the migration just exposed them.
fix(history): rebuild FTS from DB when the on-disk index was wiped or empty.TantivyFts::open_or_recreatedeletes any incompatible index dir (real case: every Python-written index — the on-disk schema hasrow_id.fieldnorms=false, the Rust schema buildstrue) andrebuild_ftshad zero call sites, so a wipe left search permanently empty with no error.HistoryStore::opennow repopulates any index that was recreated or is empty while its source table has rows (facts previously had no rebuild path at all). One-time, idempotent on reopen, INFO-logged.fix(run): fail fast on embedder feature-gap before opening the store.create_embedder(...).unwrap_or(None)swallowedFastembedMissing; afastembedconfig on a binary built withoutlocal-embedmutated the store/FTS and then died increate_projectorswith a message recommendingbackend="hash"— wrapped in the "token may be invalid" hint.resolve_embeddernow runs beforeFamiliar::load_from_diskand surfaces the real error (naminglocal-embed), honoring the factory doc's "refuses to start" contract.fix(activities): de-flakec1_state_cleared_before_wake_publish. Pre-existing since the Layer-3 port: the invariant holds in the implementation (state cleared strictly before wake publish, same task); the flake wasrecv_wake's 1s hang-guard racing 6–8spawn_blockingDB round-trips under load (~5% idle, ~67% under CPU saturation). Raised to 30s — all six call sites.expect(Some), so no assertion weakened.Plus a
style:commit (rustfmt).Verification
--features discord; clippy-D warningsclean under both;fmt --all --checkclean.familiar_enanalyzer chain re-applied over the DB) — exact result-set match on every query; the original index was itself stale (33 orphaned turn docs, 13 live facts missing), so the rebuild is strictly more correct.Notes
store.rsinit path andrun.rsregions as open Streaming leak detection, familiars root migration, and guard audit logging #209 (which removesmigrate()per Migration audit #202) — no semantic conflict, but whichever lands second takes a small rebase. Streaming leak detection, familiars root migration, and guard audit logging #209's docs referencerebuild_ftspicking up skipped docs; this PR is what wires that behavior up.🤖 Generated with Claude Code