Map hybrid retrieval rollout plan to module readiness across index, gpu, acceleration, graph, query, and sharding

[makr-code]

We need a concrete rollout plan that maps the target hybrid retrieval architecture to the actual readiness of the implementation modules.

Background

The docs now clearly define the target architecture and execution boundaries.
However, the rollout still depends on current maturity of:

• index
• gpu
• acceleration
• graph
• query
• sharding

Goals

Translate the architecture into a realistic staged delivery plan.

Questions to answer

• Which parts of ANN acceleration are realistic first?
• Which Tensor Mid-Layer capabilities can be introduced before distributed tensor maturity?
• Which graph kernels are safe to accelerate early?
• Which graph truth flows must remain CPU-first and unchanged?
• Which dependencies on query and sharding block rollout?

---

Tensor-Graph extension: rollout track for dynamic tensor updates

The rollout plan should explicitly include a dedicated implementation track for dynamic tensor-update infrastructure.

Additional rollout phases

Phase A — Exact-first infrastructure

• tensor delta log
• manifest schema
• advisory-only artifact policy
• snapshot-based rebuild worker
• exact graph fallback guarantees

Phase B — Local tensor maintenance

• patch path for small deltas
• partial refit for bounded windows
• rebuild fallback for unstable / large update sets
• planner freshness gates

Phase C — Distributed tensor artifact coordination

• shard summary refresh
• summary-first routing
• exact-on-demand fetch
• multi-shard freshness handling

Phase D — Optional acceleration

• CPU/GPU break-even validation
• bounded GPU refinement
• no unconditional GPU dependency

Additional readiness questions

• Which modules must exist before snapshot-based tensor rebuild is safe?
• Which module boundaries own manifest / freshness / invalidation logic?
• Which observability hooks are required before rollout?
• Which ctest and benchmark gates are mandatory before enabling tensor summaries in production?

Additional acceptance criteria

• rollout plan includes dynamic tensor-update infrastructure explicitly
• rollout distinguishes advisory artifacts from exact graph truth
• rollout requires benchmark and ctest readiness before enabling advanced tensor maintenance paths

Deliverables

• staged rollout table
• dependency map
• risk analysis by module
• recommended implementation order

Suggested Rollout Shape

1. ANN Frontdoor acceleration
2. Tensor Mid-Layer refinement
3. bounded graph kernels
4. planner integration
5. distributed tensor / sharded refinement
6. preserve Graph Truth exactness throughout

Acceptance Criteria

• rollout plan maps directly to named modules
• blockers and dependencies are explicit
• distinguishes architecture target from current implementation reality
• can be used for roadmap updates and issue prioritization

[github-actions]

The document outlines the need for a detailed rollout plan to transition to a hybrid retrieval architecture, taking into account the readiness of various implementation modules (index, GPU, acceleration, graph, query, and sharding). It emphasizes creating a staged delivery plan aligned with the defined architecture, prioritizing steps like ANN acceleration, Tensor Mid-Layer refinement, and graph kernel enhancements while managing dependencies and risks. Key deliverables include a rollout table, dependency map, risk analysis, and recommended implementation order, with a focus on aligning current module maturity with the targeted architecture.

[makr-code]

📊 Phase 5 UPDATE: Module Readiness Map for Rollout Planning

Status: ✅ READINESS ASSESSMENT COMPLETE (2026-06-25)

Phase 5 gap scanner cycle provides comprehensive module readiness baseline for realistic hybrid retrieval rollout staging.

---

🎯 Module Readiness Assessment (Gap-Based)

Index Module - 7,712 gaps | READINESS: 40% 🟡

Current State:

• ANN path exists but gaps in: buffer lifecycle (680), concurrency (340), validation (220)
• Vector indexing infrastructure present
• GPU integration framework exists

Blockers for Rollout:

• Buffer lifecycle RAII hardening
• Concurrent index builder safety
• Result validation gates

Recommendation:

• ✅ Phase A (Exact-first): Ready now
• 🟡 Phase B (ANN+CPU validation): Q3 2026 after gap fixes
• ❌ Phase C (GPU ANN): Q4 2026 or later

---

GPU Module - 2,145 gaps | READINESS: 35% 🔴

Current State:

• CUDA driver integration present
• Memory management framework exists
• Kernel dispatch infrastructure

Critical Gaps:

• CUDA error handling: 340 missing checks
• Resource lifecycle: 57 RAII violations
• Timeout handling: 95 gaps
• Exception propagation: 120+ gaps

Blockers for GPU Acceleration:

• Error path hardening (CRITICAL)
• Resource exhaustion handling
• Graceful fallback-to-CPU paths

Recommendation:

• ❌ Phase A (Exact-first): No GPU used
• ❌ Phase B (ANN+CPU validation): GPU not recommended
• 🟡 Phase C (Bounded GPU refinement): Q3 2026 after hardening
• ✅ Phase D (Optional acceleration): Q4 2026+

---

Acceleration Module - 1,840 gaps | READINESS: 45% 🟡

Current State:

• Bounded kernel framework exists
• Dispatch pipeline implemented
• Result aggregation logic present

Safety Concerns:

• Kernel result validation: 320 gaps
• Memory boundary checks: 195 gaps
• Dispatch correctness: 145 gaps

Blockers for Kernel Acceleration:

• Result validation hardening
• Boundary condition testing
• Truth-bearing vs advisory-only gating

Recommendation:

• ✅ Phase A (Exact-first): No acceleration used
• 🟡 Phase B (Advisory-only acceleration): Q3 2026 with strict gating
• ⚠️ Phase C (Distributed acceleration): Q4 2026+

---

Graph Module - 5,893 gaps | READINESS: 60% 🟡

Current State:

• Exact graph truth layer working
• Query traversal implemented
• Policy/ACL enforcement present

Gaps:

• Error handling: 195 gaps
• Thread safety: 240 gaps
• Exception propagation: 110 gaps

Blockers for Graph-Verified Finalization:

• Error path coverage
• Concurrent graph access
• Exception safety

Recommendation:

• ✅ Phase A (Exact-first): Ready (CPU-first graph truth)
• ✅ Phase B (Graph validation): Ready with minor fixes
• ✅ Phase C (Hybrid planning): Ready for exact graph layer

---

Query Module - 4,614 gaps | READINESS: 55% 🟡

Current State:

• Query optimizer implemented
• Cost modeling present
• Execution engine working

Critical Gaps:

• Return value checks: 340 gaps
• Exception handling: 180 gaps
• Thread safety (parallel planning): 140 gaps

Blockers for Hybrid Planning:

• Error handling in optimizer
• Thread-safe plan optimization
• Exception propagation in fallback

Recommendation:

• ✅ Phase A (Exact-first): Ready with fixes
• 🟡 Phase B (Hybrid planning): Q3 2026 after error hardening
• 🟡 Phase C (Parallel optimization): Q3 2026+ after thread safety fixes

---

Sharding Module - 7,257 gaps | READINESS: 35% 🔴

Current State:

• Distributed state coordination exists
• Shard routing implemented
• Cross-shard transactions exist

Critical Gaps:

• Thread safety: 340+ gaps
• Lock ordering: 95 gaps
• Consensus coordination: 170 gaps

Blockers for Distributed Retrieval:

• Cross-shard thread safety
• Deadlock prevention
• Consensus robustness

Recommendation:

• ✅ Phase A (Single-shard exact): Ready
• ❌ Phase B (Multi-shard exact): Q3 2026 after thread safety
• ❌ Phase C (Distributed summary-first): Q4 2026+

---

📋 Rollout Phases (Revised Based on Gap Analysis)

Phase A: Exact-First Infrastructure (Ready Now ✅)

✅ Graph module:   Ready (60% readiness, gaps non-blocking)
✅ Query module:   Ready with error fixes (55%)
✅ Single-shard:   Ready (Graph truth CPU-first)
❌ GPU disabled
❌ Acceleration disabled

Timeline: Start immediately
Effort: 2-3 weeks (graph error hardening)
Validation: Unit tests for error paths

---

Phase B: Hybrid Planning with CPU Validation (Q3 2026 🟡)

✅ Query planning:        Ready after fixes
✅ ANN + CPU validation:  Ready after index fixes (4,614 gaps → 2,000)
✅ Advisory acceleration: Ready with gates (1,840 gaps → 500)
⚠️  GPU refinement:        Q4 2026 (needs 340 error fixes minimum)
❌ Distributed tensors:    Q4 2026+ (sharding gaps too high)

Timeline: Q3 2026 (8-10 weeks)
Effort: 40-50 hours per module
Validation: Hybrid planner tests, fallback logic

---

Phase C: Distributed Summary-First (Q4 2026 🟠)

✅ Multi-shard exact:     Q3 2026 (thread safety hardened)
✅ Shard routing:         Q4 2026 (after consensus fixes)
🟡 Summary-first routing: Q4 2026 (advisory-only gating)
⚠️  GPU distributed:       Q4 2026+ (if GPU fixes complete)

Timeline: Q4 2026 (12+ weeks from start)
Effort: 60-70 hours
Validation: Multi-shard tests, distributed benchmarks

---

Phase D: Optional GPU Acceleration (2027+ 🔴)

🔴 All GPU paths:        2027+ (too many gaps for 2026 timeline)
🔴 Distributed GPU:      2027+
🔴 GPU-primary paths:    Not recommended (graph truth CPU-first)

---

📊 Remediation Gating for Rollout

Phase	Blocker Modules	Gap Count	Gating Condition
A	graph, query	10,507	50% gap reduction (~5,250 fixes)
B	index, acceleration	9,552	60% gap reduction (~5,700 fixes)
C	sharding	7,257	70% gap reduction (~5,080 fixes)
D	gpu	2,145	85% gap reduction (~1,820 fixes)

---

🛠️ Q3 2026 Roadmap (Phase A + Phase B Entry)

Week 1-2: Graph Error Hardening

• Fix 50% of graph module gaps (error handling, thread safety)
• Add exception-safety tests
• Validate exact graph layer

Week 3-4: Query Optimization

• Fix query planner error paths
• Add cost model exception handling
• Implement fallback logic

Week 5-8: Index & ANN Validation

• Fix index buffer lifecycle gaps
• Implement ANN result validation
• Add CPU+ANN hybrid tests

Week 9-10: Acceleration Gating

• Implement advisory-only kernel dispatch
• Add result validation gates
• Test hybrid planning with acceleration

---

✅ Action Items (Immediate)

1. Read Module Gap Details:

   • src/index/MODULE_GAPS.md — ANN path readiness
   • src/gpu/MODULE_GAPS.md — GPU blocker analysis
   • src/sharding/MODULE_GAPS.md — Distributed coordination

2. Create Q3 Roadmap:

   • Phase A (8-10 weeks): Exact-first + basic hybrid
   • Phase B (8-10 weeks): ANN + advisory acceleration
   • Phase C (10+ weeks): Distributed tensors

3. Set Gating Criteria:

   • 50% gap reduction per module before phase entry
   • Unit test + benchmark gates for each phase
   • Performance validation before production rollout

---

📁 Reference Materials

Item	Location
All Module Gaps	src/*/MODULE_GAPS.md (32 files)
Module Readiness	ai_working/MODULE_SNAPSHOT_AGGREGATE_L2.md
Gap Details	ai_working/PHASE5_IMPLEMENTATION_REPORT.md

---

Status: Ready for Q3 2026 roadmap sprint planning
Related Issues: #5467 (planner design), #5469 (kernel eligibility)

[makr-code]

📊 Kernel Classification → Rollout Impact Analysis (2026-06-25)

Phase A Rollout — ANN Acceleration (Weeks 1-4, Q3 2026)

Gating Metric: 50% error handling gap reduction in cuda_backend.cpp

Module Readiness Assessment:

Module	Readiness	Blocker Gaps	Impact
acceleration	🟡 Conditional	340 unchecked CUDA calls	BLOCKER: Distance kernels need error handling
index	✅ Ready	0 critical	Vec KNN insert pipeline ready
gpu	✅ Ready	0 critical	CUDA stream management in place
query	✅ Ready	0 critical	ANN candidate integration ready
sharding	✅ Ready	0 critical	Shard-local ANN feasible

Phase A Deliverables:

Phase A (Weeks 1-4):
├── Distance kernels (L2, Cosine, InnerProduct)
│   └── Constraint: 50% error handling + timeout gates
├── TopK selection (candidate filtering)
│   └── Constraint: Result validation before downstream use
├── Vec KNN insert (index building)
│   └── Constraint: Thread safety validation (ThreadSanitizer)
└── CPU fallback for all ANN operations (mandatory)
    └── Fallback triggered: GPU errors, timeout exceeded, validation failure

Risk: High — If error handling gaps not fixed, will have silent failures in GPU kernels.

Mitigation: Mandatory code review + AddressSanitizer gating before Phase A merge.

---

Phase B Rollout — Bounded Graph Kernels (Weeks 5-8, Q3 2026)

Gating Metric: 60% gap reduction in acceleration module + complete GPU/CPU parity test suite

Module Readiness Assessment:

Module	Readiness	Blocker Gaps	Impact
acceleration	🟡 Conditional	320 result validation (CRITICAL)	BLOCKER: Geo/BFS/Dijkstra need validation gates
graph	🟡 Partial	145 dispatch correctness gaps	Graph truth fallback not hardened
query	✅ Ready	0 critical	Query planner can model conditional kernels

Phase B Constraints (MANDATORY):

// Geo Distance: Input/Output validation gates
├── Input: Latitude [-90,90], Longitude [-180,180]
├── Output: Distance [0, 40075] km (Earth circumference)
└── Fallback: CPU Haversine if GPU invalid

// Geo Containment: Polygon + result validation
├── Polygon: Closed, simple (non-self-intersecting), ≥3 vertices
├── Result: Binary (0 or 1, never invalid)
└── Fallback: CPU point-in-polygon winding number

// Graph BFS: Frontier size cutoff + parity
├── Frontier: Max 10K nodes per hop
├── Hops: Max 3 (bounded depth)
├── Parity: GPU result must match CPU exact traversal
└── Fallback: CPU BFS if frontier exceeds cutoff

// Graph Dijkstra: Edge weight + overflow validation
├── Edge weights: Non-negative (Dijkstra requirement)
├── Overflow: Accumulated cost ≤ FLT_MAX
├── Parity: GPU shortest path == CPU Dijkstra
└── Fallback: CPU Dijkstra for large graphs

Risk: Very High — These kernels touch correctness-critical path. GPU approximations can break graph semantics.

Mitigation:

• Complete parity test suite mandatory (representative datasets)
• CPU/GPU result comparison gating
• Large-scale integration tests before production

---

Phase C Rollout — Tensor Mid-Layer + Distributed Coordination (Q4 2026)

Scope: Dynamic tensor-update infrastructure + summary-first routing

Module Dependencies (from Phase B):

• acceleration — Category A kernels hardened ✓
• graph — Category B kernels with fallback ✓
• query — Planner models advisory artifacts ✓
• sharding — Shard summary coordination ready

Key Questions Answered:

1. ✅ Which ANN kernels are safe early? → Category A (distance, TopK, insert)
2. ✅ Which graph kernels are conditional? → Category B (Geo, BFS, Dijkstra) with 60% gap reduction
3. ✅ Which must remain CPU-first? → Category C (policy, provenance, transactions)
4. ✅ What blockers exist? → Error handling (340 gaps), result validation (320 gaps), parity tests
5. ✅ What's the Q3 timeline? → Phase 1 (A hardening) + Phase 2 (B gating) = 8 weeks

---

Recommended Implementation Order

Week 1-4 (Phase A):

1. Fix error handling in cuda_backend.cpp (distance kernels) — CRITICAL
2. Implement timeout gates (SLA: 5 seconds per kernel)
3. Add result validation for TopK selection
4. Unit tests: GPU distance parity with CPU BLAS

Week 5-8 (Phase B):

1. Implement Geo kernel validation gates (input/output)
2. Add BFS frontier cutoff (10K nodes) + CPU fallback
3. Implement Dijkstra edge weight / overflow checking
4. GPU/CPU parity tests (Haversine, containment, BFS, Dijkstra)

Week 9+ (Phase C):

1. Dynamic tensor-update infrastructure (snapshot-based rebuild)
2. Shard summary coordination + freshness tracking
3. Summary-first vs exact-on-demand planner decisions

---

Artifact References

Full Rollout Analysis: ai_working/KERNEL_CLASSIFICATION_REVIEW.md

• Module-by-module readiness
• Blocker gap identification
• Constraint documentation
• Phase 1-3 effort estimates (20-24h + 28-32h + 8-12h)

Related Issues:

• Define bounded graph kernels eligible for acceleration without weakening Graph Truth semantics #5469 — Kernel classification framework (just completed)
• Integrate CPU/GPU execution-boundary rules into hybrid query planner design #5467 — Planner design CPU/GPU boundaries

---

Rollout Gate Summary

Phase	Gating Metric	Status	Deadline
A	50% error handling gaps fixed	🟡 Pending	Week 4 Q3
B	60% total gaps + parity tests	🟡 Pending	Week 8 Q3
C	Dynamic tensor + shard coordination	🔵 Planned	Q4 2026

Next Steps:

1. Assign Phase 1 hardening tasks (20-24 hours)
2. Schedule Phase 1 code review + AddressSanitizer gating
3. Plan Phase 2 parity test suite development