perf(hscontrol): fixes latency due to loading machines from db - #6
Open
amitsingh21 wants to merge 6 commits into
Open
perf(hscontrol): fixes latency due to loading machines from db#6amitsingh21 wants to merge 6 commits into
amitsingh21 wants to merge 6 commits into
Conversation
amitsingh21
commented
Mar 4, 2025
amitsingh21
commented
- read the CONTRIBUTING guidelines
- raised a GitHub issue or discussed it on the projects chat beforehand
- added unit tests
- added integration tests
- updated documentation if needed
- updated CHANGELOG.md
When expandAlias falls through to getIPsForUser for a non-user alias (e.g. CIDR destinations like "10.81.48.0/24"), filterMachinesByUser returns an empty slice. The subsequent excludeCorrectlyTaggedNodes call then iterates every entry in aclPolicy.TagOwners (1502 in our prod policy) and allocates a fresh tags slice, only to operate on an empty machine set. Short-circuit before that wasteful work. Output is bit-identical: excludeCorrectlyTaggedNodes returns an empty slice when given empty nodes, which already hits the "if len(filteredMachines) == 0" branch below. The new check just avoids the wasted iteration and allocations in between. Measured on io-prod (perf-iter4 -> perf-iter5): - Pod CPU: 4.6 -> 3.0 cores (-33%, -1.5 cores) sustained. - runtime.scanobject (GC): -63%. - gcBgMarkWorker: -46%. Local bench against real prod data (901 machines, 1502-rule policy): - ns/op: 85.1 ms -> 59.0 ms (-31%) - B/op: 30.2 MB -> 7.49 MB (-75%) - allocs/op: 456,485 -> 50,659 (-89%)
Default gRPC limit is 4 MiB. The DB-backed ACL policy and large node lists now exceed it, producing "received message larger than max (4194314 vs. 4194304)" on the REST -> grpc-gateway -> socket path. Raise MaxRecvMsgSize on the local socket server, remote TLS server, grpc-gateway client dial, and CLI client to 64 MiB. Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-07-10 outage: the network path between AKS and Azure Postgres silently dropped all established connections (no RST). Every pool connection sat in ExecPrepared waiting on a dead socket; the pgx default dialer keepalive of 5m left the kernel probing for the better part of an hour, and with the pool exhausted (max 10) every handler and the health check queued behind it until a manual restart. Harden the postgres connection path: - aggressive TCP keepalive (idle 15s, interval 5s, count 4) - TCP_USER_TIMEOUT 30s so writes into dead flows also fail fast (keepalive only covers idle sockets; Linux-only, build-tagged) - 10s connect timeout - statement_timeout / idle_in_transaction_session_timeout 60s - pool 10 -> 25, recycle idle conns after 5m (Azure middleboxes silently expire idle flows) Verified against a real black-hole (docker postgres + iptables DROP mid-query): dead connection now fails in ~30s and the pool self-heals; previously the idle-read case took 461s in the lab and never recovered in production. Co-Authored-By: Claude Fable 5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.