Skip to content

fix(helm): serialise deploys per (repo, env, release) to avoid helm concurrency errors #145

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pdodgen-revparts merged 1 commit into from
Jul 8, 2026
Merged

fix(helm): serialise deploys per (repo, env, release) to avoid helm concurrency errors #145

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions .github/workflows/helm-deploy-eks.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,23 @@ on:
aws_access_secret:
required: true

# Serialise deploys per (repository, environment, release_name). Two
# `helm upgrade` calls targeting the same release cannot run concurrently
# — helm rejects the second with "another operation (install/upgrade/
# rollback) is in progress" (surfaced 2026-07-08 after the fire-and-
# forget dispatch refactor decoupled deploy timing from Build timing).
#
# cancel-in-progress: false so the current deploy runs to completion.
# Canceling an in-flight helm upgrade mid-flight risks leaving the
# release in a weird state; queueing the next dispatch is safer.
#
# The group name is scoped to (repo, env, release_name) so unrelated
# deploys don't block each other — a stg deploy for internal_api can
# proceed while a stg deploy for catalog_api runs.
concurrency:
group: helm-deploy-${{ github.repository }}-${{ inputs.environment }}-${{ inputs.release_name || github.event.repository.name }}
cancel-in-progress: false

jobs:
deploy:
runs-on: ubuntu-latest
Expand Down
11 changes: 11 additions & 0 deletions .github/workflows/helm-deploy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,17 @@ on:
k8s_token:
required: true

# Serialise deploys per (repository, environment, release_name). Two
# `helm upgrade` calls targeting the same release cannot run concurrently
# — helm rejects the second with "another operation (install/upgrade/
# rollback) is in progress" (surfaced 2026-07-08 on helm-deploy-eks.yaml).
# Same treatment applied here for consistency across shared helm
# workflows. cancel-in-progress: false so the current deploy runs to
# completion before the next dispatch takes over.
concurrency:
group: helm-deploy-${{ github.repository }}-${{ inputs.environment }}-${{ inputs.release_name || github.event.repository.name }}
cancel-in-progress: false

jobs:
deploy:
runs-on: ubuntu-latest
Expand Down
11 changes: 11 additions & 0 deletions .github/workflows/helm-deployv2.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,17 @@ on:
kubeconfig:
required: true

# Serialise deploys per (repository, environment, release_name). Two
# `helm upgrade` calls targeting the same release cannot run concurrently
# — helm rejects the second with "another operation (install/upgrade/
# rollback) is in progress" (surfaced 2026-07-08 on helm-deploy-eks.yaml).
# Same treatment applied here for consistency across shared helm
# workflows. cancel-in-progress: false so the current deploy runs to
# completion before the next dispatch takes over.
concurrency:
group: helm-deploy-${{ github.repository }}-${{ inputs.environment }}-${{ inputs.release_name || github.event.repository.name }}
cancel-in-progress: false

jobs:
deploy:
runs-on: ubuntu-latest
Expand Down
9 changes: 9 additions & 0 deletions .github/workflows/helm-rollback.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,15 @@ on:
k8s_token:
required: true

# Serialise rollbacks per (repository, environment, deployment) so a
# rollback can't race with a concurrent helm operation on the same
# release — same "another operation in progress" class of failure as
# on helm-deploy-eks.yaml. cancel-in-progress: false because canceling
# a rollback mid-flight would be worse than queueing.
concurrency:
group: helm-deploy-${{ github.repository }}-${{ inputs.environment }}-${{ inputs.deployment }}

@cursor cursor Bot Jul 8, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rollback concurrency group missing fallback for empty deployment

High Severity

The concurrency group in helm-rollback.yaml uses ${{ inputs.deployment }} without a fallback, but inputs.deployment is optional with no default. When omitted, the group resolves to helm-deploy-<repo>-<env>- (empty trailing segment). The script itself falls back to github.event.repository.name for the actual helm release name. This mismatch means a rollback and a deploy targeting the same release will have different concurrency groups and can still race — the exact "another operation in progress" failure this PR aims to prevent. The three deploy workflows correctly use inputs.release_name || github.event.repository.name; the same || github.event.repository.name fallback is needed here.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 86cee8e. Configure here.

cancel-in-progress: false

jobs:
deploy:
runs-on: ubuntu-latest
Expand Down