Skip to content

feat(plugin): dual-backend fallback (remote primary + local fallback)#7

Open
lightzt99 wants to merge 5 commits into
mainfrom
feat/dual-backend-fallback
Open

feat(plugin): dual-backend fallback (remote primary + local fallback)#7
lightzt99 wants to merge 5 commits into
mainfrom
feat/dual-backend-fallback

Conversation

@lightzt99

Copy link
Copy Markdown
Owner

Summary

  • Add optional local fallback server so the Claude Code hook keeps working when the remote PowerMem backend is unreachable. Default read/write goes to remote; on net error / timeout / 5xx, the hook retries on the local fallback.
  • When POWERMEM_FALLBACK_BASE_URL is unset, behavior is byte-identical to single-backend mode (fast path).
  • Circuit-breaker state persisted to ~/.powermem/fallback-state.json with up/down TTLs to avoid probing primary on every hook event.
  • POWERMEM_FALLBACK_DISABLED=1 is a runtime kill switch.
  • Replay/sync deferred — needs server-side metadata filter support; will follow up separately.

Env vars

Var Purpose Default
POWERMEM_FALLBACK_BASE_URL fallback URL; empty = single-backend empty
POWERMEM_FALLBACK_API_KEY fallback API key empty
POWERMEM_FALLBACK_DISABLED runtime kill switch 0
POWERMEM_FALLBACK_DOWN_TTL_SECONDS "primary down" cache TTL (5..300) 30
POWERMEM_FALLBACK_UP_TTL_SECONDS "primary up" cache TTL (5..300) 30
POWERMEM_FALLBACK_TRIGGER_5XX treat 5xx as fallback trigger 1
POWERMEM_FALLBACK_LOG_FILE fallback event log path $DATA_DIR/powermem-hook.log
POWERMEM_INIT_FALLBACK_BASE_URL init.sh non-interactive fallback URL empty

Fallback triggers

Failure mode Triggers fallback?
client timeout (context.DeadlineExceeded) yes
dial refused / DNS / TLS / conn-reset (net.Error) yes
HTTP 5xx with TRIGGER_5XX=1 (default) yes
HTTP 5xx with TRIGGER_5XX=0 no
HTTP 4xx no
HTTP 2xx no (success)

Test plan

  • go test ./... in apps/claude-code-plugin/cmd/powermem-hook (14 new tests covering helpers, state, routing, 5xx, kill switch, single-backend fast path)
  • python3 apps/claude-code-plugin/tests/test_runtime_dual.py (5 cases including shell-metachar quoting)
  • Existing make test-claude-hook-docker regression unaffected (single-backend fast path preserved)
  • Docker dual-backend smoke test + make test-claude-hook-dual target — follow-up
  • Manual: set POWERMEM_BASE_URL=http://remote:8848 POWERMEM_FALLBACK_BASE_URL=http://localhost:8849, run claude -p, stop remote, run again → should still succeed; check ~/.powermem/fallback-state.json down→up transition

lightzt99 added 4 commits July 1, 2026 17:30
Add optional local fallback server so the hook keeps working when the
remote PowerMem backend is unreachable. When POWERMEM_FALLBACK_BASE_URL
is unset, behavior is byte-identical to single-backend mode.

- Go: doRequestWithFallback routes search/write to primary, retries on
  fallback for net errors / timeouts / 5xx (5xx opt-in via
  POWERMEM_FALLBACK_TRIGGER_5XX, default on). Circuit-breaker state
  persisted to ~/.powermem/fallback-state.json with up/down TTLs to
  avoid probing primary on every hook event.
- Shell: write_runtime_dual in common.sh + dual-mode branch in init.sh
  (POWERMEM_INIT_FALLBACK_BASE_URL or interactive prompt).
- POWERMEM_FALLBACK_DISABLED=1 is a runtime kill switch.
- Replay/sync deferred: needs server-side metadata filter support.
…preload-model.sh

Removed sections that duplicate script behavior or conflict with SKILL.md:
- PRE-CHECK & PREREQUISITES (uv/Python detection lives in common.sh)
- STEP BY STEP PROVEN PATH (Method A conflicts with SKILL.md's "do not
  run source/developer flow"; Method B uses outdated `claude mcp add --
  powermem-mcp stdio` — init.sh now uses `--scope user --transport http`;
  Method C duplicates the E0xx error guide)
- FINAL VALIDATION STEPS (status.sh already covers health; write/search
  round-trip belongs in status.sh, not setup docs)
- SUMMARY (placeholder text, no information value)

E006 now points to `scripts/preload-model.sh` instead of inline
ModelScope/HuggingFace commands — the script already auto-detects region
and bridges into the HF hub cache.

Net: 1076 → 947 lines. Remaining sections are either canonical reference
(masking rules, dev-mode, dual-backend env vars, E001-E014 troubleshooting,
systemd unit) or the SKILL.md entry point.
Add "Remote + local fallback" as a fourth Question 0 option and a dual
flow that:
- Asks 4 questions in one round (Server URL, API Key, Connection,
  Fallback URL). Connection drops the MCP option in dual mode — fallback
  only applies to the hook's REST calls, not the MCP transport.
- Reuses Question 1-3 (Storage / LLM / Embedding) verbatim to configure
  the fallback server, so the agent runs the same question flow as a
  standalone local server.
- Defaults the fallback URL to http://localhost:8848 (same port as a
  standalone local server; port is the user's decision).
- Detects port conflicts by probing health on the fallback URL: if a
  healthy server is already there, state the conflict plainly and let
  the user decide (reuse vs. pick a different URL). Never silently
  restart or kill the existing server.
- Runs two init.sh invocations: (1) start the fallback server, (2) write
  the dual runtime.env with POWERMEM_INIT_FALLBACK_BASE_URL.
- Notes v1 limitations (no replication, stale reads, no conflict
  resolution, MCP not covered) and the kill switch
  (POWERMEM_FALLBACK_DISABLED=1).
- Documents the removal path: re-run init picking "Remote" (single
  backend) to overwrite runtime.env and clear the fallback URL.
Adds an isolated regression test that runs the hook binary against two
in-process fake PowerMem HTTP servers (primary + fallback) and verifies
the circuit-breaker routing end-to-end. No LLM, no network (Docker
--network none).

Coverage:
- both up → request routes to primary, state file primary_down=false
- primary down → request routes to fallback, state primary_down=true
- cached down within TTL → skip primary probe, go straight to fallback
- recovery → state seeded with stale last_probe_at, primary back up →
  probe hits primary, state marks primary_down=false
- primary 5xx (TRIGGER_5XX=1 default) → fallback triggered, state down
- primary 5xx (TRIGGER_5XX=0) → no trigger, error surfaces, no state
- POWERMEM_FALLBACK_DISABLED=1 → kill switch, no fallback, no state
- POWERMEM_FALLBACK_BASE_URL empty → single-backend fast path, no state

The fake server subclasses RecordingHTTPServer so the handler's
self.server resolves to the controllable instance (with label /
fail_search / fail_health), and supports stop()/restart() on the same
port for down→up transitions.

Adds docker/Dockerfile.claude-hook-dual (same uv+go multi-stage pattern
as the no-LLM regression image) and the make target test-claude-hook-dual.
@lightzt99
lightzt99 force-pushed the feat/dual-backend-fallback branch from a018690 to ecc680d Compare July 1, 2026 09:33
init.sh: POWERMEM_INIT_RECONFIGURE=1 stops the healthy managed server,
removes .env, recreates it from current POWERMEM_INIT_* env, and starts a
fresh server. Without the flag, init prints a hint and exits 0.

SKILL.md: when status.sh reports the server healthy (local or remote), ask
via AskUserQuestion whether to keep current setup or reconfigure. Local
managed mode runs init.sh with RECONFIGURE=1; remote mode re-asks the
Server URL / API key / Connection / Fallback questions and re-runs init.sh
(remote server is not managed by the plugin, so no RECONFIGURE flag needed).

Also rebuilds the 5 platform hook binaries to include the dual-backend code.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant