feat: spec_type launch-arg plumbing + ngram-simple benchmark for specialist (#13)#15
Open
aismai wants to merge 1 commit into
Open
feat: spec_type launch-arg plumbing + ngram-simple benchmark for specialist (#13)#15aismai wants to merge 1 commit into
aismai wants to merge 1 commit into
Conversation
… specialist (closes oniwakaa#13) Measured --spec-type ngram-simple on the real TaskClassifier workload: -10% tok/s (draft acceptance 8-23%), 23/25 exact-match outputs. Off by default; enabling is a one-line uncomment in configs/models.yaml. See docs/optimization-log.md for the full writeup.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Resolves #13 — measured, and the honest answer is: ngram-simple is a net slowdown on the specialist workload, so this PR ships the plumbing and the benchmark with the flag off by default, rather than enabling it.
src/lore/models.py:start_model()now forwards an optional per-modelspec_typeconfig key as--spec-type <value>(skipped for the embeddings server). No key set → launch args unchanged.configs/models.yaml:spec_type: ngram-simplepresent on the specialist entry but commented out, with the measured result — enabling it later is a one-line uncomment.scripts/benchmark_ngram_spec.py: A/B benchmark modeled onbenchmark_spec_decode.py. Runs the realTaskClassifierworkload (20 classification queries with the production system prompt, JSON output) + 5 extraction/summarization prompts, temperature 0, and checks output equality between arms as the quality gate.tests/test_models.py: 3 new tests (flag present when configured, absent by default, never passed to embeddings). 27/27 pass.docs/optimization-log.md: full writeup.Measurements
Falcon-H1-1.5B Q4_K_M, arm64 Metal, 16 GB, 8K ctx, temperature 0, 25 prompts/arm.
Why it loses: server logs show draft acceptance mostly 8–23%. The classifier's JSON keys repeat from the prompt, but the values and connective tokens don't, so most drafted n-grams are rejected and verification overhead outweighs accepted tokens. The 2/25 output diffs are batch-size numerics wobble under speculative verification (both arms ran temperature 0).
Setup caveat: measured with mainline llama.cpp (Homebrew b9910) and f16 KV — I couldn't build the turboquant fork in my environment. Both arms share the same binary/KV so the relative delta should carry over, but it's worth a rerun on your M4 with the bundled fork; the script honors env overrides and defaults to the fork binary + turbo4:
If your numbers come out positive, enabling is just uncommenting the
spec_typeline inconfigs/models.yaml— happy to update the default in this PR if so.