fix: version HFC security release and credits contact flow#1
Open
MarkDonish wants to merge 161 commits into
Open
fix: version HFC security release and credits contact flow#1MarkDonish wants to merge 161 commits into
MarkDonish wants to merge 161 commits into
Conversation
When multiple OpenAI accounts share similar scores (which is the common case once errors/load are stable), buildOpenAIWeightedSelectionOrder used to fall back to near-uniform random and ignored the configured load_factor entirely (only Concurrency was used as a slot cap). This caused operators with mixed Pro 20x + Plus pools to see Plus accounts taking the same share of traffic as Pro 20x, defeating the whole point of capacity-based pool sizing. See upstream issue Wei-Shaw#979. The fix multiplies each candidate's weight by account.EffectiveLoadFactor() so that, when scores tie, traffic distributes proportionally to capacity. Existing score-based dynamic adjustments (priority / load / queue / error rate / TTFT) still apply on top of this multiplier. Includes two new unit tests covering 15:1 weighting and the LoadFactor=nil fallback to Concurrency. Co-Authored-By: Claude Opus 4.7 <[email protected]>
vite build OOMs at default ~960MB heap when running on a 2GB VPS even with 4GB swap added (cgroup heap limits don't extend to host swap for the buildkit user namespace). Setting NODE_OPTIONS=--max-old-space-size =3072 lets the frontend build succeed on the relay box without changing where we run the build. Co-Authored-By: Claude Opus 4.7 <[email protected]>
Investigation on production showed that /v1/chat/completions and the common OpenAI gateway path go through selectAccountForModelWithExclusions -> selectBestAccount, NOT selectByLoadBalance. The previous patch only fixed the load-balance path; the priority+LRU path still ignored load_factor entirely, so once all accounts shared the same priority an N-account pool ended up distributing traffic by LRU instead of by capacity. Real-world test on a 1×Pro20x + 8×Plus group showed Pro20x receiving 9.8% of traffic instead of the expected ~65%. This commit replaces the inner LRU tiebreaker in selectBestAccount with a weighted reservoir sampling step keyed on account.EffectiveLoadFactor(). Priority remains a hard tier (lower number always wins); compact-tier ordering remains untouched. Within the same (priority, compactTier) tier, each candidate is admitted with probability weight/runningTotal, so the final winner is sampled proportional to LoadFactor. Co-Authored-By: Claude Opus 4.7 <[email protected]>
This is the actual production path when openai_advanced_scheduler_enabled=false
(the default). Previous patches on selectByLoadBalance (advanced scheduler) and
selectBestAccount (legacy fallback) never ran for OpenAI ChatCompletions traffic
in production, which is why the 15:1 load_factor ratio for chong pro20x kept
showing up as ~10-22% uniform-ish distribution.
Selection flow that runs in prod:
ChatCompletions handler
-> SelectAccountWithScheduler
-> selectAccountWithScheduler (advanced=disabled, scheduler==nil)
-> selectAccountWithLoadAwareness Layer 2 (load-aware, LoadBatchEnabled=true)
-> shuffleWithinSortGroups <-- selection randomness lives here
-> for _, item := range selectionOrder { try acquire }
Patch:
- Add weightedShuffleByLoadFactorWithinSortGroups (OpenAI-only helper).
- Group only by (Priority, LoadRate); within each group, weighted Fisher-Yates
by EffectiveLoadFactor so position 0 (the position that gets tried first) is
picked with probability w_i / Sum(w_j).
- When all weights in a segment are equal, fall back to shuffleWithinSortGroups
to preserve LRU/never-used semantics for legacy uniform-LoadFactor pools.
- Revert selectBestAccount to strict priority+LRU; that path is shared with
Gemini/Claude flows and the LoadFactor weighting was breaking their tests
while not running on the prod OpenAI path anyway.
Tests:
- TestWeightedShuffleByLoadFactor_RatioMatchesLoadFactor: 1xPro(LF=15) +
9xPlus(LF=1), 5000 iterations, Pro position-0 ratio within 5% of 15/24,
each Plus within 3% of 1/24.
- TestWeightedShuffleByLoadFactor_PreservesPriorityGrouping: Priority=1
accounts strictly precede Priority=2 across 1000 iterations.
- TestWeightedShuffleByLoadFactor_LoadFactorNilFallsBack: nil LoadFactor
with equal Concurrency yields ~50:50.
- Existing TestOpenAISelectAccountWithLoadAwareness_PreferNeverUsed and
TestOpenAISelectAccountForModelWithExclusions_LeastRecentlyUsed continue
to pass (LRU preserved when LoadFactor uniform).
…resh paths Production observation on loadfactor-v3: with sticky_session_ttl_seconds defaulting to 1h and only one Plus account (Mark4) actively binding sessions, weighted load distribution within the Plus tier collapses to a single account. Over a 12h window, chong pro20x correctly received 63.5% (target 65-75%) but the remaining traffic landed entirely on Mark4 (46/46) while 4 other healthy Plus accounts received 0 chat completions despite being schedulable. Root cause: SelectAccountWithScheduler -> selectAccountWithLoadAwareness writes new sticky bindings via setStickySessionAccountID(..., openaiStickySessionTTL) and refreshes existing ones via refreshStickySessionTTL(..., openaiStickySessionTTL), both of which hardcode the package-level constant time.Hour. The cfg field Gateway.OpenAIWS.StickySessionTTLSeconds is exposed and validated, but the load- aware path never reads it - only BindStickySession (used by Gemini handler) does. Fix: - Replace 5 hardcoded openaiStickySessionTTL call sites in the OpenAI load-aware set/refresh paths with s.openAIWSSessionStickyTTL(), which already implements cfg-aware lookup with the constant as a sane fallback. - Sites changed: - openai_gateway_service.go:1360 selectAccountForModelWithExclusions write - openai_gateway_service.go:1419 tryStickySessionHit refresh - openai_gateway_service.go:1693 selectAccountWithLoadAwareness sticky-hit refresh - openai_gateway_service.go:1770 selectAccountWithLoadAwareness Layer-2 write - openai_gateway_service.go:1849 selectAccountWithLoadAwareness Layer-3 write - BindStickySession (used by Gemini) was already cfg-aware, untouched. Tests: - TestOpenAIWSSessionStickyTTL_ConfigOverridesDefault: cfg=600 -> ttl=10min - TestOpenAIWSSessionStickyTTL_DefaultWhenUnset: cfg=0 -> ttl=1h - All 14 existing Sticky* tests continue to pass. Operational impact: - Default behavior unchanged (still 1h when env unset). - Operators can now set GATEWAY_OPENAI_WS_STICKY_SESSION_TTL_SECONDS=600 to shorten sticky binding to 10min, allowing weighted load balancing within same-priority tier to spread traffic across all healthy accounts. Co-Authored-By: Claude Opus 4.7 <[email protected]>
…patch billing source preserve Captured from /opt/relay/sub2api-fork worktree currently running as hfc/sub2api:agent-platform-adapters-cdf2e2e65593-20260503-155939 (built 15:59 CST 2026-05-03). Commiting so the prod source state is visible on GitHub for reconciliation with MarkDonish/sub2api:feat/loadfactor-weighting. Frontend (no overlap with Marks PR): - UsageTable + DataTable add pageVerticalScroll prop so the usage log viewport stops swallowing wheel events when content fits in the visible area. Backend (overlaps with Marks billing-source commits, needs reconciliation): - applyOpenAIMessagesDispatchBillingSource preserves BillingModelSourceRequested for Claude requests routed through the OpenAI Messages dispatch path. - usage_log_repo defaults the customer-facing model dimension to requested_model (requestedUsageLogModelExpr) so Claude product tiers do not collapse into upstream model labels. Adjusts GetUserModelStats / GetModelStatsWithFilters / resolveModelDimensionExpression default and the corresponding test cases. Co-Authored-By: Claude Opus 4.7 <[email protected]>
… ingress
Some Claude Code CLI versions (notably the multi-turn agentic loop after the
2.1.x auto-update) serialize the model's *in-progress* thinking block as
{"type":"thinking","thinking":"","signature":""} when packaging the previous
turn back into the next request. The native Anthropic /v1/messages API rejects
that block with a 400 schema validation error ("thinking content is required"),
which surfaces to the customer mid-task as an apparent "做到一半就停了" abort.
This adds apicompat.SanitizeAnthropicRequestBody, called as the first step in
both GatewayHandler.Messages/CountTokens (Anthropic native path) and
OpenAIGatewayHandler.Messages (OpenAI translation path), so the same clean
payload is used for ops_error_logs sampling, claude-code-validator, and any
failover that pivots back to a native upstream.
Conservative scrub: only blocks where Type=="thinking" AND Thinking=="" AND
Signature=="" are dropped. Real signed thinking continuations are preserved
untouched, so legitimate extended-thinking multi-turn behavior is unchanged.
The body is parsed via map[string]json.RawMessage so unknown / future top-level
fields (top_k, service_tier, mcp_servers, container, …) round-trip without
loss, and surviving content blocks pass through as raw JSON so block-level
fields not modeled by AnthropicContentBlock are preserved.
Adds Signature field to AnthropicContentBlock and 16 unit tests covering:
empty body, bad JSON, no messages field, string content, no thinking blocks
(idempotent), empty thinking stripped, missing signature key counts as empty,
non-empty thinking preserved, signed thinking preserved, mixed blocks (only
empty thinking dropped), multiple messages, unknown top-level fields, unknown
block fields, system + tools preserved, idempotency, pathological all-empty
content.
… ingress
Some Claude Code CLI versions (notably the multi-turn agentic loop after the
2.1.x auto-update) serialize the model's *in-progress* thinking block as
{"type":"thinking","thinking":"","signature":""} when packaging the previous
turn back into the next request. The native Anthropic /v1/messages API rejects
that block with a 400 schema validation error ("thinking content is required"),
which surfaces to the customer mid-task as an apparent "做到一半就停了" abort.
This adds apicompat.SanitizeAnthropicRequestBody, called as the first step in
both GatewayHandler.Messages/CountTokens (Anthropic native path) and
OpenAIGatewayHandler.Messages (OpenAI translation path), so the same clean
payload is used for ops_error_logs sampling, claude-code-validator, and any
failover that pivots back to a native upstream.
Conservative scrub: only blocks where Type=="thinking" AND Thinking=="" AND
Signature=="" are dropped. Real signed thinking continuations are preserved
untouched, so legitimate extended-thinking multi-turn behavior is unchanged.
The body is parsed via map[string]json.RawMessage so unknown / future top-level
fields (top_k, service_tier, mcp_servers, container, …) round-trip without
loss, and surviving content blocks pass through as raw JSON so block-level
fields not modeled by AnthropicContentBlock are preserved.
Adds Signature field to AnthropicContentBlock and 16 unit tests covering:
empty body, bad JSON, no messages field, string content, no thinking blocks
(idempotent), empty thinking stripped, missing signature key counts as empty,
non-empty thinking preserved, signed thinking preserved, mixed blocks (only
empty thinking dropped), multiple messages, unknown top-level fields, unknown
block fields, system + tools preserved, idempotency, pathological all-empty
content.
Three follow-ups to the empty-thinking sanitize fix shipped in 1eafbba: 1. apicompat/sanitize.go: add a bytes.Contains pre-check on `"thinking"` before the json.Unmarshal pair. ~95% of /v1/messages requests carry no thinking key at all, so the slow path was pure overhead. False positives (user text containing the literal "thinking") fall through to the real parse and are still handled correctly. 2. gateway_handler.go + openai_gateway_handler.go: raise the sanErr branch from Debug to Warn. Production logging is at Info level, so corrupted bodies that the sanitizer cannot parse were being silently forwarded with no visibility — the only signal was an upstream Anthropic 400. 3. sanitize_test.go: rewrite the BadJSON test to include the "thinking" substring so it actually exercises the parse-error path now that the fast path short-circuits non-thinking bodies. Add two new tests: FastPath_NoThinkingSubstring (proves the fast path skips parse) and FastPath_ThinkingSubstringInUserText (proves false positives still yield correct byte-identical output via the slow path). reqLog already carries request_id + api_key_id + group_id from the context-bound logger, so opt3 (more triage context in the log line) is already covered — no code change needed for that one.
Three follow-ups to the empty-thinking sanitize fix shipped in 1eafbba: 1. apicompat/sanitize.go: add a bytes.Contains pre-check on `"thinking"` before the json.Unmarshal pair. ~95% of /v1/messages requests carry no thinking key at all, so the slow path was pure overhead. False positives (user text containing the literal "thinking") fall through to the real parse and are still handled correctly. 2. gateway_handler.go + openai_gateway_handler.go: raise the sanErr branch from Debug to Warn. Production logging is at Info level, so corrupted bodies that the sanitizer cannot parse were being silently forwarded with no visibility — the only signal was an upstream Anthropic 400. 3. sanitize_test.go: rewrite the BadJSON test to include the "thinking" substring so it actually exercises the parse-error path now that the fast path short-circuits non-thinking bodies. Add two new tests: FastPath_NoThinkingSubstring (proves the fast path skips parse) and FastPath_ThinkingSubstringInUserText (proves false positives still yield correct byte-identical output via the slow path). reqLog already carries request_id + api_key_id + group_id from the context-bound logger, so opt3 (more triage context in the log line) is already covered — no code change needed for that one.
gpt-image-2 image_tokens scale primarily with the `quality` parameter (low ≈ 1k, medium ≈ 2k, high ≈ 4k), not pixel size. The billing layer already reads Group.ImagePrice1K/2K/4K but the tier classifier only inspected pixel dimensions and never returned "4K", so the 4K price slot was unreachable. normalizeOpenAIImageSizeTier now takes (size, quality) and prefers quality-based classification, falling back to size dimensions when quality is empty / "auto" / unrecognized to keep older callers (gpt-image-1, dall-e flows) working. Test: existing JSON parse case (size=1024x1024 + quality=high) updated to expect "4K" — the previous "1K" was a billing bug. Verified locally: go build ./... + go test -tags unit ./internal/service/...
…ticky TTL fix merge: openai quota guard + sticky TTL fix + loadfactor weighting (Pro20x dormant)
drift-fix: bring 12 prod-only commits back to main (incl. double-entry billing ledger)
|
Thank you for your contribution! Before we can merge this PR, we need you all to sign our Contributor License Agreement (CLA). To sign, please reply with the following comment:
You only need to sign once — it will be valid for all your future contributions to this project. I have read the CLA Document and I hereby sign the CLA 0 out of 5 committers have signed the CLA. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scope
Fresh verification
This PR is intentionally large because the active production security image was previously built from an uncommitted snapshot. The branch makes that deployed source reviewable and provides an OCI revision for the replacement image.