Skip to content

Add retries to calls to the k8s api#338

Open
murphpdx wants to merge 2 commits intoactions:mainfrom
hydrolix:retry-k8s-api
Open

Add retries to calls to the k8s api#338
murphpdx wants to merge 2 commits intoactions:mainfrom
hydrolix:retry-k8s-api

Conversation

@murphpdx
Copy link
Copy Markdown

Summary

Wraps the k8s API clients (CoreV1Api, BatchV1Api, AuthorizationV1Api) in a retry Proxy that handles transient apiserver failures:

  • Retries on HTTP 408/429/500/502/503/504 and network errors (ECONNREFUSED, ECONNRESET, ETIMEDOUT, ENETUNREACH, EAI_AGAIN, ENOTFOUND) with exponential backoff + jitter, capped at 4 attempts total. Honors Retry-After on 429 (capped at 30s).
  • Makes create/delete helpers idempotent: createPod, createJob, createDockerSecret, and createSecretForEnvs handle 409 by reading back the existing resource; deletePod and deleteSecret swallow 404. Covers the lost-response-after-success case the retry wrapper can itself produce.

Why

Under load we see intermittent ECONNRESET / 503 against the kube apiserver during prepareJob, which currently fail the entire workflow. With this change those fail-fast on transient errors become short retry loops. No change to success-path behavior.

Test plan

  • Unit tests: 34 new tests in tests/k8s-retry-test.ts for isRetryableError (status codes, network codes via cause chain, precedence, edge inputs) and retryAfterDelay (header formats, 30s cap, fallbacks). npx jest tests/k8s-retry-test.ts passes.
  • Existing constants-test.ts and k8s-utils-test.ts still pass.
  • npm run build clean; npx tsc --noEmit clean.
  • Cluster test: deploy to a test runner and run workflow.

Known limitations

The createPod 409 handler returns the existing pod unconditionally. In ARC, the pod name is derived from the ephemeral runner pod name, so a stale pod with a mismatched spec is unlikely — 409 is almost always the lost-response case. If the operational signal shows otherwise, we'll add an ownership-label check as a follow-up.

@murphpdx murphpdx requested review from a team and nikola-jokic as code owners April 20, 2026 22:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant