fix(tokens): raise gpt-5-mini to 65536, the one ceiling actually probed#112
Merged
Conversation
openai/gpt-5-mini 16384 -> 65536 Replaces #111, which was closed because its evidence was invalid. That PR also raised gpt-5.5 and gpt-5.4 to 100000, citing "upstream's own stated maximum". There was no upstream response: both SDKs reject max_tokens > 100000 client-side (blockrun_llm validation.py:233, blockrun-llm-ts validation.ts:74), so every probe at 128000 failed before reaching a provider. The "maximum: 100000" I read as an upstream limit is an SDK constant. gpt-5-mini survives because its probe was BELOW that guard — the request actually went out and was accepted at 65536. It is a floor, not a proven ceiling: 65536 is also ESCALATED_MAX_TOKENS, so nothing here could distinguish a higher true limit. The comment says so rather than implying verification. gpt-5.5 and gpt-5.4 stay at 32768. The catalog's 128000 for both is UNREFUTED — I did not disprove it, I mismeasured. Their real ceilings are unknown and raising them needs a probe path that bypasses the SDK guard. Keeps the sentinel rewrite of the precedence test from #111, which was independent of the bad premise. That test hardcoded whichever static-vs-catalog disagreement existed at the time and broke twice as those numbers legitimately changed; the haiku form was worse, silently passing while asserting nothing once both sides agreed. It now primes a sentinel no real model can carry and compares against whatever the static value is, so it tests the rule. Local suite 636 pass.
VickyXAI
added a commit
that referenced
this pull request
Jul 21, 2026
) * fix(tokens): gpt-5.5 and gpt-5.4 to 128000 — now actually measured Both were 32768. The catalog said 128000 and I twice failed to establish whether that was true. #111 raised them to 100000 citing an upstream maximum. That number came from blockrun_llm/validation.py:233 — a client-side guard — so the probe never reached a provider. #112 reverted them to 32768 and recorded the catalog's 128000 as unrefuted but unverified. Re-probed 2026-07-21 with the SDK guard bypassed in-process: both accept 128000, as did all 19 models advertising a ceiling above 100000, including zai/glm-5.2 at 262144. Zero rejections. The catalog was right the whole time; the measurement was broken. Saying so explicitly because the previous conclusion pointed the other way, and this table is exactly where a wrong belief gets frozen. Root cause fixed upstream in blockrun-llm#27 and blockrun-llm-ts#15. Effect: normal turns are unchanged (CAPPED_MAX_TOKENS 16384). On a max_tokens stop the escalation ceiling goes from min(65536, 32768)=32768 to min(65536, 128000)=65536 — recovery doubles instead of clamping back below where it started. Local suite 636 pass. * docs(tokens): record that Franklin does not run the SDK max_tokens guard Review of this PR turned up a wrong dependency claim I had made out loud: that the SDK fixes needed to merge before Franklin could raise these values, or users' local SDK would reject the larger number. Franklin never runs that guard. Both request paths — src/agent/llm.ts and src/proxy/server.ts — use raw fetch and import only payment helpers from @blockrun/llm, never chatCompletion, so validateMaxTokens is not on either path. These values are independent of whether blockrun-llm#27 and blockrun-llm-ts#15 land, and the merge ordering I recommended was unnecessary. Also drops the hardcoded '19 models' count, which would rot as the catalog changes, and marks the referenced SDK PRs as open rather than implying the root cause is already fixed everywhere. --------- Co-authored-by: 1bcMax <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces #111, which I closed because its evidence was invalid.
What changed
openai/gpt-5-miniThat's the whole functional change.
Why #111 was wrong
#111 also raised
gpt-5.5andgpt-5.4to 100,000, citing "upstream's own stated maximum." There was no upstream response. Both SDKs rejectmax_tokens > 100000client-side:blockrun_llm/validation.py:233—raise ValueError("max_tokens too large (maximum: 100000)")blockrun-llm-ts/src/validation.ts:74— identical guardSo every probe I ran at 128,000 failed before reaching a provider. The
maximum: 100000I read as an upstream limit is an SDK constant. I ran 19 of those and read them as 19 catalog errors; they were 19 client-side refusals, and the uniform number across three different providers should have tipped me off immediately.gpt-5-minisurvives the retraction because its probe was below the guard — 65,536 < 100,000, so the request actually went out and came back accepted.What this does NOT claim
gpt-5-mini's 65,536 is a floor, not a proven ceiling. 65,536 is alsoESCALATED_MAX_TOKENS, so nothing measurable here could distinguish a higher true limit. The code comment says that rather than implying verification.gpt-5.5/gpt-5.4stay at 32,768 and the catalog's 128,000 for both is unrefuted — I did not disprove it, I mismeasured. Their real ceilings are unknown. Raising them needs a probe path that bypasses the SDK guard, which is separate work.The gateway bug I reported alongside #111 (catalog 128,000 vs upstream 100,000) does not exist as described. Nothing was filed; nothing needs reverting.
Carried over from #111
The sentinel rewrite of
gateway catalog never overrides a static max-output entry, which was independent of the bad premise. That test hardcoded whichever static-vs-catalog disagreement existed at the time — haiku first, then gpt-5.5 — and broke twice as those numbers legitimately changed. The haiku form was worse than brittle: once the gateway was corrected and both sides read 64,000, it kept passing while asserting nothing. It now primes a sentinel (999_999) no real model can carry and compares against whatever the static value happens to be, so it tests the precedence rule rather than a snapshot of the numbers.Local suite 636 pass, 0 fail.