Skip to content

feat: spec_type launch-arg plumbing + ngram-simple benchmark for specialist (#13)#15

Open
aismai wants to merge 1 commit into
oniwakaa:mainfrom
aismai:feat/specialist-ngram-spec-decode
Open

feat: spec_type launch-arg plumbing + ngram-simple benchmark for specialist (#13)#15
aismai wants to merge 1 commit into
oniwakaa:mainfrom
aismai:feat/specialist-ngram-spec-decode

Conversation

@aismai

@aismai aismai commented Jul 9, 2026

Copy link
Copy Markdown

Summary

Resolves #13 — measured, and the honest answer is: ngram-simple is a net slowdown on the specialist workload, so this PR ships the plumbing and the benchmark with the flag off by default, rather than enabling it.

  • src/lore/models.py: start_model() now forwards an optional per-model spec_type config key as --spec-type <value> (skipped for the embeddings server). No key set → launch args unchanged.
  • configs/models.yaml: spec_type: ngram-simple present on the specialist entry but commented out, with the measured result — enabling it later is a one-line uncomment.
  • scripts/benchmark_ngram_spec.py: A/B benchmark modeled on benchmark_spec_decode.py. Runs the real TaskClassifier workload (20 classification queries with the production system prompt, JSON output) + 5 extraction/summarization prompts, temperature 0, and checks output equality between arms as the quality gate.
  • tests/test_models.py: 3 new tests (flag present when configured, absent by default, never passed to embeddings). 27/27 pass.
  • docs/optimization-log.md: full writeup.

Measurements

Falcon-H1-1.5B Q4_K_M, arm64 Metal, 16 GB, 8K ctx, temperature 0, 25 prompts/arm.

Metric Baseline ngram-simple
Avg tokens/sec 13.52 12.17 (-10.0%)
Avg latency/request 8.46 s 10.25 s
Exact-match outputs vs baseline 23/25

Why it loses: server logs show draft acceptance mostly 8–23%. The classifier's JSON keys repeat from the prompt, but the values and connective tokens don't, so most drafted n-grams are rejected and verification overhead outweighs accepted tokens. The 2/25 output diffs are batch-size numerics wobble under speculative verification (both arms ran temperature 0).

Setup caveat: measured with mainline llama.cpp (Homebrew b9910) and f16 KV — I couldn't build the turboquant fork in my environment. Both arms share the same binary/KV so the relative delta should carry over, but it's worth a rerun on your M4 with the bundled fork; the script honors env overrides and defaults to the fork binary + turbo4:

.venv/bin/python scripts/benchmark_ngram_spec.py

If your numbers come out positive, enabling is just uncommenting the spec_type line in configs/models.yaml — happy to update the default in this PR if so.

… specialist (closes oniwakaa#13)

Measured --spec-type ngram-simple on the real TaskClassifier workload:
-10% tok/s (draft acceptance 8-23%), 23/25 exact-match outputs. Off by
default; enabling is a one-line uncomment in configs/models.yaml.
See docs/optimization-log.md for the full writeup.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: add ngram-simple speculative decoding to specialist server

1 participant