Skip to content

Scale changed-path ledger for large repositories#2

Merged
forhappy merged 147 commits into
mainfrom
perf/trusted-git-handoff
Jul 16, 2026
Merged

Scale changed-path ledger for large repositories#2
forhappy merged 147 commits into
mainfrom
perf/trusted-git-handoff

Conversation

@forhappy

@forhappy forhappy commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Summary

  • activate the changed-path ledger across observers, recovery, status, diff, record, patch application, API, and MCP paths
  • automatically reconcile filesystem state and start or replace per-workspace daemons on status, diff, and record
  • avoid repository-wide scans by consuming authenticated changed-path candidates and controlled projection intervals
  • add fail-closed workspace activation and clear reinitialization guidance instead of migration or backward-compatibility behavior
  • add native Linux/macOS coverage, strict scale thresholds, benchmark tooling, CI/release gates, and production diagnostics

Why

Large repositories such as the Linux kernel can contain 100,000 or more files. Recording and applying lane changes previously risked repeated full-tree traversal, stale daemon ownership, and WAL/checkpoint visibility races. Those costs made small changes scale with repository size instead of changed-path count.

This change makes candidate discovery proportional to actual changes, binds observer data to authenticated durable publication, and performs an automatic full reconciliation when continuity cannot be proven.

Impact

Status, diff, record, and structured patch workflows remain fast for sparse changes in large workspaces. Materialized and transparent COW lanes use the same changed-path authority, while stale or incompatible workspaces are rejected with instructions to reinitialize.

Strict 100,000-file runs passed on native Linux and macOS. On the Linux run, sparse materialized records completed in approximately 0.18-0.57 seconds and patches in approximately 0.21-1.21 seconds. The macOS run passed the same candidate-count and no-full-scan assertions.

Validation

  • cargo fmt --all -- --check
  • cargo check -p trail --all-targets
  • 36 benchmark/checker Python tests
  • bash -n scripts/cli-scale-bench.sh
  • changed-path producer inventory test
  • activation, API, Git, recovery, materialized-lane, observer, and daemon-focused suites during implementation
  • strict native Linux and macOS 100,000-file scale runs

Branch ancestry

The local repository already contained an unpublished integration stack. Consequently, this head is 147 commits ahead of GitHub's current main; this PR includes that inherited ancestry as well as commit 9e10f7c. Land the ancestor stack first or rebase/retarget this branch before merging if the PR must contain only the changed-path-ledger delta.

Follow-up

This is intentionally a draft. A later hardening pass can complete the full release matrix and refine the materialized-lane REST/daemon lifecycle edge case before final release promotion.

forhappy added 30 commits July 11, 2026 18:24
Implement layered multi-ecosystem lane environments, adapter SDK and runtime lifecycle support, including opaque service secrets and macOS/Linux copy-on-write verification. Add native agent hook and ACP capture with durable evidence, provenance, CLI, HTTP, MCP, and OpenAPI surfaces.
BREAKING CHANGE: human output now uses the unified terminal renderer.\nUse --color never instead of --no-color, and use JSON or NDJSON\nfor automation.
@forhappy
forhappy marked this pull request as ready for review July 16, 2026 21:33
@forhappy
forhappy merged commit fa13ebd into main Jul 16, 2026
9 of 41 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant