[Bug] TcpTransport connection pool: unbounded growth with threshold mass-GC; stale entries to restarted peers are never evicted

[d3nb]

Environment

mooncake-transfer-engine-cuda13 0.3.11.post1 (PyPI), used as SGLang PD-disaggregation KV transfer backend (MC_FORCE_TCP=1, MC_TCP_ENABLE_CONNECTION_POOL=true), same-host prefill(TP4) → decode(TP4), 8×B300, container network.

Issue 1: pool grows unboundedly, then mass-closes tens of thousands of connections at once

Under sustained load (fixed 10-way concurrency, ~27K-token KV parcels, chunked), /proc/net/tcp on the sender side shows a sawtooth with ~8-minute period:

03:15  ESTABLISHED 29024
03:17  ESTABLISHED 38952
03:18  ESTABLISHED 17724   TIME_WAIT 21089   <- ~21K connections closed within one minute
03:24  ESTABLISHED 44419
03:25  ESTABLISHED 45623   <- 83% of the ephemeral port range
03:26  ESTABLISHED 20738   TIME_WAIT 26831   <- ~27K closed at once

Two problems: (a) between flushes the pool keeps growing toward the ephemeral-port limit (we measured peaks at 83% of a widened 10240-65000 range — with the default ~28K-port range this would hit EADDRNOTAVAIL long before the flush); (b) the flush itself dumps 20K+ sockets into TIME_WAIT within seconds. A bounded pool with per-peer connection reuse would presumably hold this at a few hundred connections (we see ~566 ESTABLISHED steady when idle-ish).

Issue 2: pooled connections to a dead/restarted peer are never evicted

If one side of the PD pair is restarted alone (its listen ports change), the surviving side's pool keeps the stale entries. Small transfers still succeed (presumably via new connections), but large multi-slice transfers stall silently and permanently — the sender reports no error, the receiver times out (KVPoll.WaitingForInput timeout after 300s). It also degrades progressively: after one single-sided peer restart we only measured throughput loss at high concurrency; after a second one, all large transfers hung. Restarting both sides together fully restores normal behavior.

Ask

1. Bound the pool (per-peer cap + LRU reuse) instead of grow-then-mass-GC.
2. Evict/reconnect pool entries on connection failure or peer-identity change, so a peer restart doesn't silently poison large transfers.

Happy to provide more traces or test patches — this is reproducible on demand in our setup.

[github-actions]

Thanks for opening this issue, @d3nb!

Field	Value
Issue	#2930
GitHub user ID	7304667
Reporter	@d3nb

A maintainer will triage this when possible. To help us respond faster, please include:

• Mooncake version or commit SHA
• Environment (OS, CUDA/driver, RDMA stack if relevant)
• Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

[d3nb]

Adding the triage details requested by the bot:

Mooncake version: mooncake-transfer-engine-cuda13 0.3.11.post1 (PyPI wheel, as bundled in the SGLang image lmsysorg/sglang:nightly-dev-cu13-20260713-b94ac87e).

Environment:

• Single node, 8×B300, two containers (prefill / decode) on a docker bridge network
• Container OS: Ubuntu 24.04.3 LTS; host kernel 6.8.0-101-generic
• CUDA 13.0 (V13.0.88), driver 590.48.01
• Transport: TCP (mooncake_tcp, MC_TCP_ENABLE_CONNECTION_POOL=true); the RDMA path is parked on this build due to a "Local segment descriptor not found" regression on long multi-segment transfers ([Bug]: transfer_metadata: Failed to get segment descriptor for segment xxx #2177-shaped), so everything below is pure TcpTransport
• Widened container port range net.ipv4.ip_local_port_range = 10240 65000, tcp_tw_reuse=1

Repro: SGLang PD disaggregation (1 prefill + 1 decode, DeepSeek-V4 class model, KV transfer via TcpTransport), then any benchmark that ramps concurrency (we use aiperf max-concurrency-under-sla over a ~5000-request trace with ~27K-token mean prompts). Watch ss -s / per-state counts on either worker.

Actual:

1. ESTABLISHED count climbs monotonically to ~45.6K (~83% of the widened port range), then roughly every ~8 min a threshold GC flushes 21-27K connections per minute into TIME_WAIT, then the climb restarts. Under default port range this manifests as EADDRNOTAVAIL instead (that's what forced the pool on).
2. After restarting one peer, the surviving peer's pool keeps stale entries for the dead endpoint indefinitely — transfers to the restarted peer keep failing until both sides are restarted. We now operate PD workers under a mandatory paired-restart rule solely because of this.

Expected: bounded pool with idle/LRU eviction and reuse (no sawtooth mass-GC), and eviction of pool entries on connection error so a restarted peer heals without restarting the survivor.

[alogfans]

Thanks for the detailed repro and socket-state timeline.

The pool is currently keyed by host/port and does not have a per-peer or global bound. When all pooled sockets for a peer are in use, getConnection() creates another socket and appends it to the peer queue. Under sustained chunked KV transfers, especially with many concurrent slices, that means the number of ESTABLISHED sockets can grow with workload pressure instead of being capped at a small reuse set.

The sawtooth/TIME_WAIT burst is also consistent with the cleanup behavior. Idle cleanup is triggered lazily from getConnection(), and expired idle sockets can be closed in one cleanup pass. So after a burst of transfers completes, many sockets can become idle around the same time, and a later cleanup pass may close a large batch together, producing the TIME_WAIT spike you measured.

For the restart case, socket->is_open() is not a sufficient liveness check for a pooled TCP connection. A socket can still look open locally after the peer process has exited or after a single-sided restart. The current pool also does not track a peer instance/generation, so it cannot reliably distinguish “same host/port, old peer” from “same logical peer after restart”. That can leave stale pooled entries eligible for reuse until a later async read/write path detects the failure, which is especially painful for large multi-slice transfers.

Planned fixes:

1. Bound the pool

   • Add a configurable per-peer connection cap.
   • Reuse idle connections before creating new sockets.
   • When the cap is reached, evict the oldest idle connection if possible.
   • If all peer connections are busy, use bounded waiting/backpressure or fail fast instead of opening unbounded sockets.

2. Make cleanup gradual

   • Track last-used time and use LRU-style idle eviction.
   • Limit the number of sockets closed in a single cleanup pass.
   • Avoid doing socket close work while holding the pool mutex where possible.

3. Evict bad connections reliably

   • Remove pooled connections on read/write/protocol/timeout failure.
   • Ensure failed sockets are discarded rather than returned to the pool.
   • Add transfer or per-chunk deadlines so large transfers fail visibly instead of hanging forever.

4. Handle peer restart

   • Clear the affected peer’s pool when its advertised TCP endpoint changes.
   • Longer term, add a peer instance ID/generation to segment metadata and include it in the pool key, so connections from a previous peer process cannot be reused after restart.

As this may incur significant changes of the tcp transport, we will perform these fixes seperately.