Skip to content

Promote persistent MTP boundary split candidate default#1

Open
guelfoweb wants to merge 1 commit into
mainfrom
mtp-boundary-split-merge-candidate
Open

Promote persistent MTP boundary split candidate default#1
guelfoweb wants to merge 1 commit into
mainfrom
mtp-boundary-split-merge-candidate

Conversation

@guelfoweb

Copy link
Copy Markdown
Owner

Promotes persistent MTP boundary split to default while keeping ORBIT_MTP_BOUNDARY_SPLIT=0 as rollback.

Review note:
This branch was built from a clean integration baseline that did not yet include the native backend/MTP package, so the PR may include native_llama, native_server, and MTP tests as part of the isolated merge candidate. The review-critical behavioral delta is limited to the persistent MTP boundary split default, rollback flag, and documentation.

Reviewer focus:

  • src/orbit/native_llama/vendor/shim/orbit_persistent_mtp.cpp

    • boundary split is now default
    • ORBIT_MTP_BOUNDARY_SPLIT=0 restores previous behavior
    • fallback/guardrails remain in place
  • docs/LLAMA_DEEP_CLIENT_EXPERIMENT.md

    • documents the new default, rollback flag, top1-greedy draft policy, and CPU-affinity benchmark requirement
  • native MTP tests

    • validate persistent MTP and experimental MTP paths

Validation:

  • PYTHONPATH=src python3 -m unittest tests.test_native_persistent_mtp tests.test_native_mtp_experimental -q PASS

  • python3 -m compileall -q src tests scripts PASS

  • git diff --check PASS

  • smoke PASS on:

    • response_short
    • response_medium
    • long_prompt
    • session_reuse_repeat
  • opt-out rollback verified with ORBIT_MTP_BOUNDARY_SPLIT=0

Benchmark:
Controlled with taskset -c 0-5, 10 runs per mode per scenario.

  • response_medium: median -14.3%, p75 -14.5%
  • long_prompt: median -16.3%, p75 -16.5%
  • session_reuse_repeat: median -24.0%, p75 -24.9%
  • response_short: median +0.5%, p75 +0.3%

Safety:

  • zero assert
  • zero unexpected fallback
  • replay does not increase
  • rollback flag works

Risk:
CPU-only latency remains sensitive to scheduler noise. Keep ORBIT_MTP_BOUNDARY_SPLIT=0 available during stabilization and run follow-up benchmarks on another host before removing the rollback flag.

Final decision:
Merge candidate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant