server: add --max_inflight_bytes to bound concurrent request memory

[mjonss]

Problem

Under a burst of concurrent uploads or downloads, bazel-remote's peak memory scales with client fan-out and can OOM (see #529): each in-flight gRPC bytestream request pins live memory — a per-stream read buffer, a zstd encoder/decoder window, and gRPC send/receive buffers — and there is no limit on how many run concurrently.

A heap profile captured at the peak of a real download burst showed ~5.8 GB resident across ~1000 concurrent Read streams — 48% gRPC send buffers, 29% zstd decoder windows, 20% read buffers — essentially all under grpcServer.Read. The working set fully recovers afterward, so it is live memory proportional to concurrency, not a leak.

Fix

Add a --max_inflight_bytes flag (env BAZEL_REMOTE_MAX_INFLIGHT_BYTES) that bounds the total logical (uncompressed) size of blobs being served or received concurrently, via a weighted semaphore held for the whole request. Once the budget is reached, further requests block (backpressure) instead of exhausting memory. The weight is clamped to [1, budget] so a blob larger than the whole budget is still served (serialized against others) rather than deadlocking. 0 (the default) disables the limit, preserving existing behavior. Client cancellation releases the wait.

This applies the same bounded-concurrency idea that #480 / #556 introduced for disk eviction to request memory, and makes the high memory usage reported in #529 configurable.

Tests

TestGrpcInflightLimitBoundsConcurrentReads verifies the cap deterministically: with a budget of K×blobSize, at most K reads reach the cache concurrently and the rest block until budget frees (verified to fail when the limit is disabled).

This PR also adds a read/decompress benchmark documenting that the read path barely churns (the zstd decoder is pooled) — i.e. the download OOM is driven by live per-stream state × concurrency, not allocation churn, which is why the fix is a concurrency bound rather than buffer pooling.

[mjonss]

This is to make it easier to run bazel-remote with gRPC in low memory setups, which was an issue for me. The argument and env variable name was chosen to be more related to memory usage, instead of directly to concurrency (which then needs to be calculated from how much memory would be used per stream).