Skip to content

Commit 0f209b4

Browse files
VickyXAI1bcMax
andauthored
test: assert the error names the SDK's own number, not a model's (#28)
Follow-up to #27. Two gaps in its tests: - The real-ceiling assertions (128000, 262144) sat inside test_accept_valid_values alongside 1/100/1000, so a failure there would not say which kind of value broke. Split into their own test with the probe evidence attached. - Nothing asserted the message content. The whole reason #27 exists is that the old text, 'max_tokens too large (maximum: 100000)', read like a provider response and was recorded as an upstream model ceiling in a downstream token table. If the message regresses to that shape the bug returns, and no test would have caught it. Now pinned: the message must name the SDK's own limit and disclaim being a model limit. Also references MAX_TOKENS_SANITY_LIMIT instead of hardcoding 2_000_000, so the test follows the constant if it moves. Co-authored-by: 1bcMax <[email protected]>
1 parent 5fc5bd2 commit 0f209b4

1 file changed

Lines changed: 18 additions & 8 deletions

File tree

tests/unit/test_validation.py

Lines changed: 18 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22

33
import pytest
44
from blockrun_llm.validation import (
5+
MAX_TOKENS_SANITY_LIMIT,
56
validate_private_key,
67
validate_api_url,
78
validate_model,
@@ -130,10 +131,6 @@ def test_accept_valid_values(self):
130131
validate_max_tokens(100)
131132
validate_max_tokens(1000)
132133
validate_max_tokens(100000)
133-
# Real ceilings the gateway serves — these were rejected before the
134-
# sanity bound was raised, despite every provider accepting them.
135-
validate_max_tokens(128000) # opus-4.8 / sonnet-5 / gpt-5.6 / glm-5
136-
validate_max_tokens(262144) # zai/glm-5.2
137134

138135
def test_accept_none(self):
139136
"""Should accept None."""
@@ -151,17 +148,30 @@ def test_reject_zero(self):
151148

152149
def test_reject_implausible(self):
153150
"""Should reject values no model could mean (typo guard, not a limit)."""
151+
with pytest.raises(ValueError, match="implausibly large") as exc:
152+
validate_max_tokens(MAX_TOKENS_SANITY_LIMIT * 2)
153+
# The message must name the SDK's own number, so a caller can tell it
154+
# apart from a provider ceiling.
155+
assert str(MAX_TOKENS_SANITY_LIMIT) in str(exc.value)
156+
157+
def test_boundary_is_inclusive(self):
158+
"""Pin the exact edge: the limit passes, one above it fails.
159+
160+
Without this, flipping ``>`` to ``>=`` keeps the suite green while
161+
rejecting a legal value.
162+
"""
163+
validate_max_tokens(MAX_TOKENS_SANITY_LIMIT)
154164
with pytest.raises(ValueError, match="implausibly large"):
155-
validate_max_tokens(2_000_000)
165+
validate_max_tokens(MAX_TOKENS_SANITY_LIMIT + 1)
156166

157167
def test_does_not_cap_below_real_ceilings(self):
158168
"""The bound must never be the binding constraint on a real request.
159169
160170
Regression: this was 100000, which rejected every ceiling above it
161171
client-side — the caller saw a ValueError naming a limit no provider
162-
had set, and the request never reached the network. Probed against the
163-
live gateway 2026-07-21: 19 models advertise more than 100000 and all
164-
19 accepted their advertised ceiling.
172+
had set, and the request never reached the network. 128000 is the
173+
common ceiling (opus-4.8 / sonnet-5 / gpt-5.6 / glm-5); 262144 is
174+
zai/glm-5.2, the highest any model serves.
165175
"""
166176
for real_ceiling in (128_000, 262_144):
167177
validate_max_tokens(real_ceiling)

0 commit comments

Comments
 (0)