docs(tokens): correct the note claiming gateway max_output is unenforced#109
Merged
Conversation
added 2 commits
July 20, 2026 17:24
The comment added in 3.35.2 said the gateway's max_output values are 'not enforced: Franklin sends max_tokens 16384 to haiku today and the gateway accepts it.' Accepting the request and honoring it are different things. The gateway clamps with Math.min(request.max_tokens, model.maxOutput) on both handlers and prices the quote off the clamped ceiling — the request is accepted rather than rejected, so the clamp stays invisible until a reply is long enough to hit it. The smoke test that produced that claim returned ten tokens. The conclusion the comment supports is unchanged: the static tables stay authoritative, because the gateway's values are wrong for models we know. Only the reasoning was wrong, and a wrong reason is how a right rule gets reverted later.
Two things the first pass got imprecise, in a comment whose entire job is precision about this mechanism: - It cited 8192 / 64000 in the present tense. Those exact values are being corrected upstream (BlockRunAI/blockrun#266), so the comment was written pre-stale. Reframed as a dated observation, with the durable point stated separately: the catalog can be wrong, and a wrong value here is not cosmetic. - It called the clamp 'invisible' without qualification. The handler does log 'capping to <limit>' — server-side, where no caller sees it (chat/completions/route.ts:1343). Invisible from here, not silent everywhere. Someone who finds that warn should not conclude the comment is wrong again.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #108. Comment-only, no behavior change.
The note I added in 3.35.2 said the gateway's
max_outputvalues are "not enforced: Franklin sends max_tokens 16384 to haiku today and the gateway accepts it."Accepting the request and honoring it are different things. The gateway clamps with
Math.min(request.max_tokens, model.maxOutput)on both the messages and chat/completions handlers, and derives its price quote from the clamped ceiling. The request is accepted rather than rejected, so the clamp stays invisible until a reply is long enough to hit it — the smoke test that produced my claim returned ten tokens, nowhere near either ceiling.The rule the comment supports is unchanged and still correct: static tables stay authoritative, because the gateway's values are wrong for models we already know. Only the stated reason was wrong — and a wrong reason is how a right rule gets reverted by the next person who checks it.
Gateway-side fix for the underlying wrong values: BlockRunAI/blockrun#266.
Local suite 634 pass.