Skip to content

fix(helm): serialise deploys per (repo, env, release) to avoid helm concurrency errors#145

Merged
pdodgen-revparts merged 1 commit into
mainfrom
DEVEX-helm-deploy-concurrency
Jul 8, 2026
Merged

fix(helm): serialise deploys per (repo, env, release) to avoid helm concurrency errors#145
pdodgen-revparts merged 1 commit into
mainfrom
DEVEX-helm-deploy-concurrency

Conversation

@pdodgen-revparts

Copy link
Copy Markdown
Contributor

Summary

Adds a workflow-level concurrency: block to all four shared helm workflows so two helm upgrade/helm rollback calls targeting the same release can't race. Two racing calls fail with:

Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress

What surfaced this

The dispatch-refactor sweep landed today decoupled Build timing from deploy timing. Under the old workflow_call nesting, deploys shared Build's concurrency group and were implicitly serialized. Under fire-and-forget dispatch, Build completes immediately after triggering the deploy — so a second Build (e.g., from a common bump ~1 minute later) can start and dispatch a second deploy while the first is still running.

Two examples from earlier today, on the same shared helm-deploy-eks.yaml:

Run Repo Error
28972011991 listings-url-service UPGRADE FAILED: another operation in progress
28971997409 returns-api same

Both were second-in-sequence deploys triggered by common bumps that landed within a minute of the first push.

Fix

Workflow-level concurrency on each of the four shared helm workflows:

concurrency:
  group: helm-deploy-${{ github.repository }}-${{ inputs.environment }}-${{ inputs.release_name || github.event.repository.name }}
  cancel-in-progress: false
  • Scope: (repository, environment, release_name). Unrelated deploys don't block each other — a stg deploy for internal_api can run while a stg deploy for catalog_api runs.
  • cancel-in-progress: false: The current deploy runs to completion. Canceling an in-flight helm upgrade risks leaving the release in an inconsistent state; queueing the next dispatch is safer.

Applied to all four shared helm workflows:

  • helm-deploy-eks.yaml (primary EKS deploy — the one actively failing)
  • helm-deploy.yaml (older path, still used by daemon-template, reporting-app, laravel-api-template)
  • helm-deployv2.yaml (used by pub_vehicle_api, prices-api, daemon-template)
  • helm-rollback.yaml (uses deployment input instead of release_name)

Blast radius

Zero when there's no concurrency contention — deploys run exactly as before. When there IS contention on the same (repo, env, release), the second dispatch queues and waits (up to GitHub's default 6-hour timeout, but real deploys complete in minutes) instead of failing.

Verification

Once merged, the next scenario where two dispatches land within seconds of each other targeting the same release will exercise the queue. The queued run's Actions UI will show it as "queued behind another workflow run" until the first finishes.

Related

  • Dispatch-refactor sweep (batch#8562 + 17 companion PRs, all merged 2026-07-08)
  • DEVEX-1653 (RT v2)

…oncurrency errors

Two concurrent `helm upgrade` calls targeting the same release fail
with:

  Error: UPGRADE FAILED: another operation (install/upgrade/rollback)
  is in progress

Surfaced on 2026-07-08 after the dispatch-refactor decoupled Build
timing from deploy timing. Prior workflow_call nesting effectively
serialised deploys because they shared Build's concurrency group
(`${{ workflow }}-${{ ref }}`); the fire-and-forget dispatch made
Build return immediately, letting a second Build kick off a second
dispatch while the first deploy was still running.

Failing examples:
  - encodium/listings-url-service run 28972011991 (stg_deploy failed
    with helm 'another operation in progress')
  - encodium/returns-api run 28971997409 (same signature)

Fix: add workflow-level concurrency group scoped to
(repository, environment, release_name) on all four shared helm
workflows. Two deploys targeting the same release queue; unrelated
deploys don't block each other.

cancel-in-progress: false because canceling a helm upgrade in flight
risks leaving the release in an inconsistent state. Queueing the next
dispatch is safer than aborting.

Refs DEVEX-1653.
@cursor

cursor Bot commented Jul 8, 2026

Copy link
Copy Markdown

PR Summary

Low Risk
CI-only workflow metadata; no app or cluster logic changes, and behavior is unchanged when deploys do not overlap.

Overview
Adds workflow-level concurrency to the four shared Helm workflows (helm-deploy-eks.yaml, helm-deploy.yaml, helm-deployv2.yaml, helm-rollback.yaml) so only one Helm operation runs at a time per repository + environment + release (rollback uses deployment instead of release_name).

This queues overlapping helm upgrade / helm rollback dispatches instead of letting them race and fail with another operation (install/upgrade/rollback) is in progress—a failure mode that showed up after deploy timing was decoupled from Build. cancel-in-progress: false keeps the in-flight deploy or rollback running to completion rather than canceling mid-operation.

Reviewed by Cursor Bugbot for commit 86cee8e. Bugbot is set up for automated code reviews on this repo. Configure here.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 86cee8e. Configure here.

# on helm-deploy-eks.yaml. cancel-in-progress: false because canceling
# a rollback mid-flight would be worse than queueing.
concurrency:
group: helm-deploy-${{ github.repository }}-${{ inputs.environment }}-${{ inputs.deployment }}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rollback concurrency group missing fallback for empty deployment

High Severity

The concurrency group in helm-rollback.yaml uses ${{ inputs.deployment }} without a fallback, but inputs.deployment is optional with no default. When omitted, the group resolves to helm-deploy-<repo>-<env>- (empty trailing segment). The script itself falls back to github.event.repository.name for the actual helm release name. This mismatch means a rollback and a deploy targeting the same release will have different concurrency groups and can still race — the exact "another operation in progress" failure this PR aims to prevent. The three deploy workflows correctly use inputs.release_name || github.event.repository.name; the same || github.event.repository.name fallback is needed here.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 86cee8e. Configure here.

@pdodgen-revparts
pdodgen-revparts marked this pull request as ready for review July 8, 2026 20:42
@pdodgen-revparts
pdodgen-revparts requested a review from a team as a code owner July 8, 2026 20:42
@pdodgen-revparts
pdodgen-revparts merged commit d27a4ed into main Jul 8, 2026
2 checks passed
@pdodgen-revparts
pdodgen-revparts deleted the DEVEX-helm-deploy-concurrency branch July 8, 2026 20:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant