fix(helm): serialise deploys per (repo, env, release) to avoid helm concurrency errors#145
Conversation
…oncurrency errors
Two concurrent `helm upgrade` calls targeting the same release fail
with:
Error: UPGRADE FAILED: another operation (install/upgrade/rollback)
is in progress
Surfaced on 2026-07-08 after the dispatch-refactor decoupled Build
timing from deploy timing. Prior workflow_call nesting effectively
serialised deploys because they shared Build's concurrency group
(`${{ workflow }}-${{ ref }}`); the fire-and-forget dispatch made
Build return immediately, letting a second Build kick off a second
dispatch while the first deploy was still running.
Failing examples:
- encodium/listings-url-service run 28972011991 (stg_deploy failed
with helm 'another operation in progress')
- encodium/returns-api run 28971997409 (same signature)
Fix: add workflow-level concurrency group scoped to
(repository, environment, release_name) on all four shared helm
workflows. Two deploys targeting the same release queue; unrelated
deploys don't block each other.
cancel-in-progress: false because canceling a helm upgrade in flight
risks leaving the release in an inconsistent state. Queueing the next
dispatch is safer than aborting.
Refs DEVEX-1653.
PR SummaryLow Risk Overview This queues overlapping Reviewed by Cursor Bugbot for commit 86cee8e. Bugbot is set up for automated code reviews on this repo. Configure here. |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 86cee8e. Configure here.
| # on helm-deploy-eks.yaml. cancel-in-progress: false because canceling | ||
| # a rollback mid-flight would be worse than queueing. | ||
| concurrency: | ||
| group: helm-deploy-${{ github.repository }}-${{ inputs.environment }}-${{ inputs.deployment }} |
There was a problem hiding this comment.
Rollback concurrency group missing fallback for empty deployment
High Severity
The concurrency group in helm-rollback.yaml uses ${{ inputs.deployment }} without a fallback, but inputs.deployment is optional with no default. When omitted, the group resolves to helm-deploy-<repo>-<env>- (empty trailing segment). The script itself falls back to github.event.repository.name for the actual helm release name. This mismatch means a rollback and a deploy targeting the same release will have different concurrency groups and can still race — the exact "another operation in progress" failure this PR aims to prevent. The three deploy workflows correctly use inputs.release_name || github.event.repository.name; the same || github.event.repository.name fallback is needed here.
Reviewed by Cursor Bugbot for commit 86cee8e. Configure here.


Summary
Adds a workflow-level
concurrency:block to all four shared helm workflows so twohelm upgrade/helm rollbackcalls targeting the same release can't race. Two racing calls fail with:What surfaced this
The dispatch-refactor sweep landed today decoupled Build timing from deploy timing. Under the old
workflow_callnesting, deploys shared Build's concurrency group and were implicitly serialized. Under fire-and-forget dispatch, Build completes immediately after triggering the deploy — so a second Build (e.g., from a common bump ~1 minute later) can start and dispatch a second deploy while the first is still running.Two examples from earlier today, on the same shared helm-deploy-eks.yaml:
Both were second-in-sequence deploys triggered by common bumps that landed within a minute of the first push.
Fix
Workflow-level concurrency on each of the four shared helm workflows:
(repository, environment, release_name). Unrelated deploys don't block each other — a stg deploy for internal_api can run while a stg deploy for catalog_api runs.cancel-in-progress: false: The current deploy runs to completion. Canceling an in-flight helm upgrade risks leaving the release in an inconsistent state; queueing the next dispatch is safer.Applied to all four shared helm workflows:
helm-deploy-eks.yaml(primary EKS deploy — the one actively failing)helm-deploy.yaml(older path, still used by daemon-template, reporting-app, laravel-api-template)helm-deployv2.yaml(used by pub_vehicle_api, prices-api, daemon-template)helm-rollback.yaml(usesdeploymentinput instead ofrelease_name)Blast radius
Zero when there's no concurrency contention — deploys run exactly as before. When there IS contention on the same
(repo, env, release), the second dispatch queues and waits (up to GitHub's default 6-hour timeout, but real deploys complete in minutes) instead of failing.Verification
Once merged, the next scenario where two dispatches land within seconds of each other targeting the same release will exercise the queue. The queued run's Actions UI will show it as "queued behind another workflow run" until the first finishes.
Related