DEVEX-1687: raise helm client QPS/burst to stop rate-limiter deploy failures

[pdodgen-revparts]

Problem

helm upgrade --wait (used by every service deploying through helm-deploy-eks.yaml) polls the status of every object in the release on a loop. For large releases this exceeds helm's low default client-side QPS, and client-go's rate limiter aborts status computation with:

failed to compute object status: <deploy>: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline

This fails an otherwise-healthy deploy and, with atomic: true, triggers an auto-rollback of a good release. Observed on webstore prod on 2026-07-09 (~90+ pods + deployments/services/ingress in one release): the paced rollout reached full availability, but helm's --wait never returned and a prior attempt's rollback failed outright with the error above (DEVEX-1687).

Change

Set HELM_QPS=50 and HELM_BURST_LIMIT=300 as env on the deploy step. Helm exposes these exactly for this case (helm --help: "set the Queries Per Second in cases where a high number of calls exceed the operations…", "burst limit in the case the server contains many CRDs").

This only relaxes the client-side limiter. The Kubernetes API server's own API Priority & Fairness still throttles/protects it, so there's no risk of overwhelming the control plane.

Scope / blast radius

Shared workflow — applies to all consumers. That's intended: it's a safe, uniform raise of a client-side limit that's too low for larger releases, and a no-op for small ones (they never approach the default ceiling).

Confidence

The rate-limiter error is a confirmed failure mode (recorded in the webstore rev-400 release record), not inferred. The related --wait hang on the same release is strongly consistent with the same cause (availability was stable at full replicas for minutes while helm never returned). If a future large deploy still hangs after this, it points to a second factor to investigate.

Related

• webstore rollout-pacing fix: encodium/webstore#4666 (DEVEX-1687)

[cursor]

PR Summary

Low Risk
Workflow-only client throttle tuning with no app or cluster config changes; blast radius is shared deploy jobs, with API server protection unchanged.

Overview
Large helm upgrade --wait deploys were failing when Helm’s client-side Kubernetes API rate limiter timed out while polling status for many objects (seen on webstore prod, DEVEX-1687), which could mark healthy rollouts as failed and trigger --atomic rollbacks.

The shared helm-deploy-eks.yaml deploy step now sets HELM_QPS=50 and HELM_BURST_LIMIT=300 so status polling can keep up on big releases. This applies to every service using that workflow; small releases are effectively unchanged. Server-side API Priority & Fairness is unchanged.

^{Reviewed by Cursor Bugbot for commit 5bd5a66. Bugbot is set up for automated code reviews on this repo. Configure here.}