feat: verify-and-escalate shadow capture at the edit seam (phase 1, gated off)#921
Open
anandgupta42 wants to merge 9 commits into
Open
feat: verify-and-escalate shadow capture at the edit seam (phase 1, gated off)#921anandgupta42 wants to merge 9 commits into
anandgupta42 wants to merge 9 commits into
Conversation
…ated off)
First step of integrating the altimate-router verifier platform (verify-and-escalate) into
altimate-code. To measure the verifier's real false-ship φ + savings on LIVE traffic we need
real before/after edits WITH full file context (better than mining transcript snippets).
- altimate/verify-shadow.ts (new): VerifyShadow.captureEdit appends {ts, file, language,
before, after, old/new string} to a local JSONL. OFF unless ALTIMATE_VERIFY_SHADOW is set;
fail-safe (never throws — capture must never break an edit). An offline pass in
altimate-router (eval/phi) reconstructs runnable units and runs the verifier.
- tool/edit.ts: one marker-wrapped, fail-safe capture call after a successful edit + the
marker-wrapped import. SHADOW ONLY — no behaviour change, no escalation/acting yet.
Phase 2 (later): call the verifier live (verify-daemon / napi addon) and act on the Decision,
gated behind the same flag, after shadow data confirms φ=0 on real traffic.
Verified: upstream marker check clean; `bun run typecheck` green (0 errors); runtime smoke —
OFF writes nothing, ON captures before/after with language detected.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
📝 WalkthroughWalkthroughThis PR enhances the dbt PR review engine with deterministic quality improvements and test infrastructure. It adds dialect-aware equivalence checking, tightens YAML classification to avoid false positives, refactors schema pattern detection to run catalog rules, improves structural and PII finding severity/deduplication, hardens DuckDB connection handling, normalizes CLI flag defaults, introduces edit capture for verification, and provides comprehensive test coverage across all changes. ChangesDeterministic Review Improvements
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
packages/drivers/src/duckdb.ts (1)
83-87:⚠️ Potential issue | 🟠 Major | ⚡ Quick winClose the DB instance on timeout to prevent lock/resource leaks.
At Line 83-87, timeout rejection does not close
instance. In the Bun scenario noted by your own comment (open callback never fires), that leaves the native handle open and can keep the file locked in-process.Suggested fix
const tryConnect = (accessMode?: string): Promise<any> => new Promise<any>((resolve, reject) => { let resolved = false let timeout: ReturnType<typeof setTimeout> | undefined let instance: any + const safeClose = () => { + try { + if (instance && typeof instance.close === "function") { + instance.close(() => {}) + } + } catch { + // fail-safe cleanup + } + } const opts = accessMode ? { access_mode: accessMode } : undefined const onOpen = (err: Error | null) => { if (resolved) { - if (instance && typeof instance.close === "function") instance.close() + safeClose() return } resolved = true if (timeout) clearTimeout(timeout) if (err) { const msg = err.message || String(err) if (msg.toLowerCase().includes("locked") || msg.includes("SQLITE_BUSY") || msg.includes("DUCKDB_LOCKED")) { reject(new Error("DUCKDB_LOCKED")) } else { reject(err) } } else { resolve(instance) } } instance = opts ? new duckdb.Database(dbPath, opts, onOpen) : new duckdb.Database(dbPath, onOpen) // Bun: native callback may not fire; fall back after 2s timeout = setTimeout(() => { if (!resolved) { resolved = true + safeClose() reject(new Error(`Timed out opening DuckDB database "${dbPath}"`)) } }, 2000) })🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/drivers/src/duckdb.ts` around lines 83 - 87, The timeout rejection path in the open callback does not close the DuckDB native handle `instance`, which can leave the DB file locked; update the timeout branch to call `instance.close()` (or the appropriate close/cleanup method on the DuckDB client object) before rejecting, guarding that `instance` is defined and swallowing/logging any error from close to avoid throwing from the timeout handler; ensure this change is made alongside the existing `resolved` check in the same asynchronous open routine so the native handle is always released if the open callback never fires.packages/opencode/src/tool/edit.ts (1)
63-87:⚠️ Potential issue | 🟠 Major | ⚡ Quick winMove shadow capture outside the lock and run it for all successful edits.
Line 86 returns early for
oldString === "", so those edits are never captured. Also, Lines 126-134 execute capture whileFileTime.withLockis held, increasing lock contention for non-critical shadow work.Suggested restructuring
- // altimate_change start — verify shadow capture (gated off by default; fail-safe, no behavior change) - VerifyShadow.captureEdit({ - file: filePath, - before: contentOld, - after: contentNew, - oldString: params.oldString, - newString: params.newString, - }) - // altimate_change end FileTime.read(ctx.sessionID, filePath) }) + + VerifyShadow.captureEdit({ + file: filePath, + before: contentOld, + after: contentNew, + oldString: params.oldString, + newString: params.newString, + })Also applies to: 126-134
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/opencode/src/tool/edit.ts` around lines 63 - 87, The shadow-capture logic is currently executed inside FileTime.withLock and skipped for the early-return branch when params.oldString === "" (and also increases lock duration for the other path); refactor edit handling so that FileTime.withLock only wraps the minimal critical section (checks, ctx.ask permission, Filesystem.write and Bus.publish calls that must be atomic) and move the shadow capture call(s) (the FileTime.read / shadow capture logic currently at lines ~126-134) outside and after the withLock block so they run for all successful edits (both the early-return case and the normal path); ensure you still call FileTime.read(ctx.sessionID, filePath) once after successful write/publishes and after releasing the lock, and keep references to FileTime.withLock, ctx.ask, Filesystem.write, Bus.publish and FileWatcher.Event to locate the spots to change.
🧹 Nitpick comments (1)
packages/opencode/src/altimate/verify-shadow.ts (1)
13-66: 🏗️ Heavy liftRefactor this module to Effect services/tracing conventions.
This new service bypasses the project’s Effect patterns (
Effect.fn/Effect.gen) and direct-effect service access, which will make tracing/composition inconsistent with surrounding Effectified services.As per coding guidelines,
packages/opencode/**/*.{ts,tsx}should “UseEffect.gen(function* () { ... })”, “UseEffect.fn("Domain.method")”, and “PreferFileSystem.FileSysteminstead of rawfs/promisesfor effectful file I/O`.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/opencode/src/altimate/verify-shadow.ts` around lines 13 - 66, The module should be converted into an Effect-style service: replace the top-level imperative captureEdit function with an Effect.fn/Effect.gen-backed service method and stop using synchronous node fs calls; implement VerifyShadow as an Effect service that exposes captureEdit (e.g., VerifyShadow.captureEdit) implemented with Effect.fn("VerifyShadow.captureEdit") or Effect.gen(function* () { ... }), use the project's FileSystem.FileSystem abstractions for mkdir/append (instead of mkdirSync/appendFileSync) and use the effectful way to read env/homedir or a small Env config service (replace logPath with an effectful resolver), keep EXT_LANG and the record shape but construct and persist the JSONL inside the Effect while catching/logging errors via the existing Log service inside the effect so the operation never throws.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/opencode/src/altimate/verify-shadow.ts`:
- Around line 59-61: The shadow log is being created with default permissions
and may be world-readable; update the log creation to enforce owner-only access
by making the directory with mode 0o700 (replace mkdirSync(dirname(path), ...)
to include { recursive: true, mode: 0o700 }) and write the file with owner-only
mode 0o600 instead of relying on default umask — e.g., open the path with
fs.openSync(path, "a", 0o600) then write the JSON + "\n" via fs.writeSync and
fs.closeSync (or call appendFileSync with an explicit { mode: 0o600 } option) so
logPath(), mkdirSync and appendFileSync usage is hardened to owner-only
permissions.
---
Outside diff comments:
In `@packages/drivers/src/duckdb.ts`:
- Around line 83-87: The timeout rejection path in the open callback does not
close the DuckDB native handle `instance`, which can leave the DB file locked;
update the timeout branch to call `instance.close()` (or the appropriate
close/cleanup method on the DuckDB client object) before rejecting, guarding
that `instance` is defined and swallowing/logging any error from close to avoid
throwing from the timeout handler; ensure this change is made alongside the
existing `resolved` check in the same asynchronous open routine so the native
handle is always released if the open callback never fires.
In `@packages/opencode/src/tool/edit.ts`:
- Around line 63-87: The shadow-capture logic is currently executed inside
FileTime.withLock and skipped for the early-return branch when params.oldString
=== "" (and also increases lock duration for the other path); refactor edit
handling so that FileTime.withLock only wraps the minimal critical section
(checks, ctx.ask permission, Filesystem.write and Bus.publish calls that must be
atomic) and move the shadow capture call(s) (the FileTime.read / shadow capture
logic currently at lines ~126-134) outside and after the withLock block so they
run for all successful edits (both the early-return case and the normal path);
ensure you still call FileTime.read(ctx.sessionID, filePath) once after
successful write/publishes and after releasing the lock, and keep references to
FileTime.withLock, ctx.ask, Filesystem.write, Bus.publish and FileWatcher.Event
to locate the spots to change.
---
Nitpick comments:
In `@packages/opencode/src/altimate/verify-shadow.ts`:
- Around line 13-66: The module should be converted into an Effect-style
service: replace the top-level imperative captureEdit function with an
Effect.fn/Effect.gen-backed service method and stop using synchronous node fs
calls; implement VerifyShadow as an Effect service that exposes captureEdit
(e.g., VerifyShadow.captureEdit) implemented with
Effect.fn("VerifyShadow.captureEdit") or Effect.gen(function* () { ... }), use
the project's FileSystem.FileSystem abstractions for mkdir/append (instead of
mkdirSync/appendFileSync) and use the effectful way to read env/homedir or a
small Env config service (replace logPath with an effectful resolver), keep
EXT_LANG and the record shape but construct and persist the JSONL inside the
Effect while catching/logging errors via the existing Log service inside the
effect so the operation never throws.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: a6a4f458-2d3f-4a48-8685-a19934e1bbdf
📒 Files selected for processing (18)
docs/docs/usage/dbt-pr-review.mddocs/internal/2026-06-08-dbt-pr-review-self-improvement-loop.mddocs/internal/dbt-pr-review-demo-scenario-corpus.mdpackages/drivers/src/duckdb.tspackages/opencode/src/altimate/native/altimate-core.tspackages/opencode/src/altimate/native/types.tspackages/opencode/src/altimate/review/dbt-patterns.tspackages/opencode/src/altimate/review/diff-filter.tspackages/opencode/src/altimate/review/orchestrate.tspackages/opencode/src/altimate/review/runner.tspackages/opencode/src/altimate/verify-shadow.tspackages/opencode/src/cli/cmd/review.tspackages/opencode/src/tool/edit.tspackages/opencode/test/altimate/data-diff-duckdb-e2e.test.tspackages/opencode/test/altimate/review-ci.test.tspackages/opencode/test/altimate/review-dbt-patterns.test.tspackages/opencode/test/altimate/review-runner.test.tspackages/opencode/test/altimate/review.test.ts
Comment on lines
+59
to
+61
| const path = logPath() | ||
| mkdirSync(dirname(path), { recursive: true }) | ||
| appendFileSync(path, JSON.stringify(record) + "\n") |
There was a problem hiding this comment.
Create the shadow log with owner-only permissions.
Line 61 appends full-file before/after payloads, but default create mode can leave this JSONL readable by other local users (umask-dependent). That’s a sensitive-data exposure risk.
Suggested hardening
- appendFileSync(path, JSON.stringify(record) + "\n")
+ appendFileSync(path, JSON.stringify(record) + "\n", { mode: 0o600 })📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| const path = logPath() | |
| mkdirSync(dirname(path), { recursive: true }) | |
| appendFileSync(path, JSON.stringify(record) + "\n") | |
| const path = logPath() | |
| mkdirSync(dirname(path), { recursive: true }) | |
| appendFileSync(path, JSON.stringify(record) + "\n", { mode: 0o600 }) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/opencode/src/altimate/verify-shadow.ts` around lines 59 - 61, The
shadow log is being created with default permissions and may be world-readable;
update the log creation to enforce owner-only access by making the directory
with mode 0o700 (replace mkdirSync(dirname(path), ...) to include { recursive:
true, mode: 0o700 }) and write the file with owner-only mode 0o600 instead of
relying on default umask — e.g., open the path with fs.openSync(path, "a",
0o600) then write the JSON + "\n" via fs.writeSync and fs.closeSync (or call
appendFileSync with an explicit { mode: 0o600 } option) so logPath(), mkdirSync
and appendFileSync usage is hardened to owner-only permissions.
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
6 issues found across 18 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="docs/internal/dbt-pr-review-demo-scenario-corpus.md">
<violation number="1" location="docs/internal/dbt-pr-review-demo-scenario-corpus.md:48">
P3: Hardcoded developer-specific absolute paths make the validation command non-portable for other contributors and CI.</violation>
</file>
<file name="packages/opencode/src/tool/edit.ts">
<violation number="1" location="packages/opencode/src/tool/edit.ts:127">
P2: Shadow capture is branch-limited; successful `oldString === ""` edits skip capture entirely. This undercounts edit traffic and can skew verifier φ/savings measurement.</violation>
</file>
<file name="packages/opencode/src/altimate/verify-shadow.ts">
<violation number="1" location="packages/opencode/src/altimate/verify-shadow.ts:61">
P2: Synchronous disk writes in `captureEdit` block the edit path when shadow mode is enabled. Use async fs writes to keep edit latency non-blocking.</violation>
</file>
<file name="packages/opencode/src/altimate/review/dbt-patterns.ts">
<violation number="1" location="packages/opencode/src/altimate/review/dbt-patterns.ts:822">
P2: This adds duplicate reporting for `relationships` test removals in schema.yml. Duplicate findings inflate noise and can over-weight a single issue in review output.</violation>
</file>
<file name="packages/opencode/src/altimate/review/orchestrate.ts">
<violation number="1" location="packages/opencode/src/altimate/review/orchestrate.ts:1081">
P1: PII fallback is effectively disabled whenever classifier APIs exist, causing silent misses when diff-scoped classification returns empty.</violation>
<violation number="2" location="packages/opencode/src/altimate/review/orchestrate.ts:1181">
P2: Regex PII findings are suppressed based on capability presence, not actual classifier coverage, creating false negatives.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| if (lanes.has("test_coverage")) tasks.push(missingGrainTestLane(ctx, input.runner)) | ||
| if (lanes.has("pii_exposure")) all.push(piiLane(ctx.file, ctx.pii)) | ||
| const hasDiffScopedPii = !!input.runner.classifyPii && !!input.runner.columnLineage | ||
| if (lanes.has("pii_exposure") && !hasDiffScopedPii) all.push(piiLane(ctx.file, ctx.pii)) |
There was a problem hiding this comment.
P1: PII fallback is effectively disabled whenever classifier APIs exist, causing silent misses when diff-scoped classification returns empty.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/opencode/src/altimate/review/orchestrate.ts, line 1081:
<comment>PII fallback is effectively disabled whenever classifier APIs exist, causing silent misses when diff-scoped classification returns empty.</comment>
<file context>
@@ -1070,7 +1077,8 @@ export async function runReview(input: OrchestrateInput): Promise<VerdictEnvelop
if (lanes.has("test_coverage")) tasks.push(missingGrainTestLane(ctx, input.runner))
- if (lanes.has("pii_exposure")) all.push(piiLane(ctx.file, ctx.pii))
+ const hasDiffScopedPii = !!input.runner.classifyPii && !!input.runner.columnLineage
+ if (lanes.has("pii_exposure") && !hasDiffScopedPii) all.push(piiLane(ctx.file, ctx.pii))
// PII classification always runs (cheap name-pattern check, diff-scoped to
// newly-introduced columns) — exposing PII is a hard-floor concern that
</file context>
| createTwoFilesPatch(filePath, filePath, normalizeLineEndings(contentOld), normalizeLineEndings(contentNew)), | ||
| ) | ||
| // altimate_change start — verify shadow capture (gated off by default; fail-safe, no behavior change) | ||
| VerifyShadow.captureEdit({ |
There was a problem hiding this comment.
P2: Shadow capture is branch-limited; successful oldString === "" edits skip capture entirely. This undercounts edit traffic and can skew verifier φ/savings measurement.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/opencode/src/tool/edit.ts, line 127:
<comment>Shadow capture is branch-limited; successful `oldString === ""` edits skip capture entirely. This undercounts edit traffic and can skew verifier φ/savings measurement.</comment>
<file context>
@@ -120,6 +123,15 @@ export const EditTool = Tool.define("edit", {
createTwoFilesPatch(filePath, filePath, normalizeLineEndings(contentOld), normalizeLineEndings(contentNew)),
)
+ // altimate_change start — verify shadow capture (gated off by default; fail-safe, no behavior change)
+ VerifyShadow.captureEdit({
+ file: filePath,
+ before: contentOld,
</file context>
| } | ||
| const path = logPath() | ||
| mkdirSync(dirname(path), { recursive: true }) | ||
| appendFileSync(path, JSON.stringify(record) + "\n") |
There was a problem hiding this comment.
P2: Synchronous disk writes in captureEdit block the edit path when shadow mode is enabled. Use async fs writes to keep edit latency non-blocking.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/opencode/src/altimate/verify-shadow.ts, line 61:
<comment>Synchronous disk writes in `captureEdit` block the edit path when shadow mode is enabled. Use async fs writes to keep edit latency non-blocking.</comment>
<file context>
@@ -0,0 +1,66 @@
+ }
+ const path = logPath()
+ mkdirSync(dirname(path), { recursive: true })
+ appendFileSync(path, JSON.stringify(record) + "\n")
+ } catch (e) {
+ log.error("edit capture failed", { error: e instanceof Error ? e.message : String(e) })
</file context>
| }) | ||
| out.push(f) | ||
| } | ||
| out.push(...evaluateCatalog(file, "", added, removed, rubric)) |
There was a problem hiding this comment.
P2: This adds duplicate reporting for relationships test removals in schema.yml. Duplicate findings inflate noise and can over-weight a single issue in review output.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/opencode/src/altimate/review/dbt-patterns.ts, line 822:
<comment>This adds duplicate reporting for `relationships` test removals in schema.yml. Duplicate findings inflate noise and can over-weight a single issue in review output.</comment>
<file context>
@@ -794,29 +794,33 @@ export function detectSchemaYmlPatterns(file: ChangedFile, rubric: Rubric): Find
+ })
+ out.push(f)
+ }
+ out.push(...evaluateCatalog(file, "", added, removed, rubric))
+ return out
+ .map((f) => ({ ...f, severity: clampSeverity(f.category, f.severity, f.confidence) }))
</file context>
Suggested change
| out.push(...evaluateCatalog(file, "", added, removed, rubric)) | |
| out.push( | |
| ...evaluateCatalog(file, "", added, removed, rubric).filter( | |
| (x) => !(x.ruleKey === "test_coverage:relationship-test-removed-yml" && genuinelyRemoved.some((l) => /\brelationships\b/i.test(l))), | |
| ), | |
| ) |
| const fallbackRule = String((f.evidence?.result as any)?.rule ?? "").replace(/_/g, "-") | ||
| if ( | ||
| (tool === "dbt-patterns" || tool === "rule-catalog") && | ||
| diffScopedPiiFiles.has(f.file) && |
There was a problem hiding this comment.
P2: Regex PII findings are suppressed based on capability presence, not actual classifier coverage, creating false negatives.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/opencode/src/altimate/review/orchestrate.ts, line 1181:
<comment>Regex PII findings are suppressed based on capability presence, not actual classifier coverage, creating false negatives.</comment>
<file context>
@@ -1160,13 +1174,22 @@ export async function runReview(input: OrchestrateInput): Promise<VerdictEnvelop
+ const fallbackRule = String((f.evidence?.result as any)?.rule ?? "").replace(/_/g, "-")
+ if (
+ (tool === "dbt-patterns" || tool === "rule-catalog") &&
+ diffScopedPiiFiles.has(f.file) &&
+ f.category === "pii_exposure" &&
+ (fallbackRule === "pii-into-mart" || fallbackRule === "select-pii-columns")
</file context>
Suggested change
| diffScopedPiiFiles.has(f.file) && | |
| diffScopedPiiFiles.has(f.file) && flat.some((x) => x.file === f.file && x.evidence?.tool === "altimate_core.classify_pii") && |
| Default focused validation command: | ||
|
|
||
| ```bash | ||
| ALTIMATE_LOCAL_CORE=/Users/anandgupta/codebase/altimate-core-internal/crates/altimate-core-node \ |
There was a problem hiding this comment.
P3: Hardcoded developer-specific absolute paths make the validation command non-portable for other contributors and CI.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At docs/internal/dbt-pr-review-demo-scenario-corpus.md, line 48:
<comment>Hardcoded developer-specific absolute paths make the validation command non-portable for other contributors and CI.</comment>
<file context>
@@ -0,0 +1,217 @@
+Default focused validation command:
+
+```bash
+ALTIMATE_LOCAL_CORE=/Users/anandgupta/codebase/altimate-core-internal/crates/altimate-core-node \
+ bun --conditions=browser /Users/anandgupta/codebase/altimate-code/packages/opencode/src/index.ts review \
+ --cwd /Users/anandgupta/codebase/altimate-code/demo/dbt-pr-review-demo \
</file context>
dev-punia-altimate
left a comment
dev-punia-altimate
left a comment
Contributor
There was a problem hiding this comment.
Multi-Persona Review — Verdict: skipped
Multi-persona review completed.
0/0 agents completed · 1s · 0 findings (0 critical, 0 high, 0 medium)
Multi-Persona Review · vllm:qwen3-next-80b (waves) + vllm-fallback (synth) ·
Contributor
❌ Tests — Failures DetectedTypeScript — 15 failure(s)
Next StepPlease address the failing cases above and re-run verification. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 1: verify-and-escalate shadow capture at the edit seam
First step of wiring the altimate-router verifier platform (verify-and-escalate: run cheap → prove behaviour-equivalence with a sound oracle → escalate the misses) into altimate-code.
Why: the verifier is measured at φ=0 with ~40–50% cost savings on synthetic workloads, and is sound on real edits (0 false-ships), but a real at-scale φ/savings number needs real before/after edits with full file context — which only exist at the edit tool (transcript snippets don't reconstruct into runnable units; see
altimate-router/eval/docs/findings/dogfood-substrate-analysis.md).What this does (SHADOW only):
altimate/verify-shadow.ts(new):VerifyShadow.captureEditappends{ts, file, language, before, after, old/new string}to a local JSONL. OFF unlessALTIMATE_VERIFY_SHADOWis set; fail-safe (never throws — capture must never break an edit).tool/edit.ts: one marker-wrapped, fail-safe capture call after a successful edit (+ the import). No behaviour change, no escalation/acting.An offline pass in altimate-router (
eval/phi) reconstructs runnable units from the captured full-file edits and runs the verifier → the first real-traffic decision distribution + false-ship φ.Phase 2 (separate PR, later): call the verifier live (
verify-daemon/ napi addon) and act on theDecision, behind the same flag, once shadow data confirms φ=0 on real traffic.Verified: upstream marker check clean;
bun run typecheckgreen (0 errors); runtime smoke — OFF writes nothing, ON captures before/after with language detected.🤖 Generated with Claude Code
Summary by cubic
Adds shadow edit capture for the verifier (phase 1). It’s fully gated and never changes edit behavior; captured before/after edits are written to a local JSONL for offline analysis and φ measurement.
New Features
VerifyShadow.captureEditwrites{ts,file,language,before,after,old/new}to~/.altimate-router/edit-capture.jsonl. Off by default; enable withALTIMATE_VERIFY_SHADOW. Hooked once post-edit intool/edit.tsand never throws.join_risk; dialect now threaded into core equivalence; diff-scoped PII classification tightened (e.g., low-confidence Name/Address suppressed) and fallback PII rules suppressed when core runs; schema YAML catalog rules run for test/metadata weakening; added DuckDB data-diff e2e via@altimateai/drivers/duckdb; docs clarify producingtarget/catalog.jsonfor proof-grade equivalence.Bug Fixes
undefinedas open options; uses 1-arg constructor when no options are needed to prevent callback hangs in Bun, with lock detection retained.schema_yml(e.g., GitHub workflows no longer misclassified).--aiis now on by default; use--no-aito disable the advisory lane.Written for commit 7deabd6. Summary will update on new commits.
Summary by CodeRabbit
Release Notes
New Features
--no-aito--aifor clearer advisory reviewer controlBug Fixes
Tests
Documentation