fix(gpu): unpin AKS kubernetes_version so AKS selects a supported default - #1277
Merged
Conversation
The GPU perf-eval scenarios pinned kubernetes_version = "1.33". AKS has since moved the 1.33 minor into Long-Term Support (LTS) only, so 'az aks create --kubernetes-version 1.33' on a Standard tier cluster now fails with K8sVersionNotSupported (1.33.13 is LTS-only). This broke every scheduled run of the GPU Cluster CRUD / scheduling pipelines. Remove the pin from the Azure inputs of all three GPU scenarios so the aks-cli module omits --kubernetes-version and AKS selects a currently supported default version. Avoids recurrence as versions age out. - k8s-gpu-cluster-crud/terraform-inputs/azure.tfvars - k8s-gpu-scheduling/terraform-inputs/azure.tfvars - k8s-ray-scheduling/terraform-inputs/azure.tfvars (EKS aws.tfvars left on 1.33; EKS still supports it and no GPU pipeline runs on AWS.)
Xu Xue (xuexu6666)
deployed
to
terraform-privileged
July 31, 2026 01:27 — with 
GitHub Actions
Active
GitHub Actions
Active
Xu Xue (xuexu6666)
temporarily deployed
to
terraform-privileged
July 31, 2026 01:28 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Xu Xue (xuexu6666)
temporarily deployed
to
terraform-privileged
July 31, 2026 01:28 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Xu Xue (xuexu6666)
temporarily deployed
to
terraform-privileged
July 31, 2026 01:28 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Contributor
There was a problem hiding this comment.
Pull request overview
Removes the hard-coded AKS Kubernetes minor version pin (1.33) from the Azure inputs for the three GPU-related perf-eval scenarios so AKS can select a currently supported default version, preventing failures as older minors age into LTS-only.
Changes:
- Removed
kubernetes_version = "1.33"from theaks_cli_config_listentries in the three GPU-related Azure tfvars files. - Leaves version selection to AKS defaults (the
aks-climodule treats a missing/nullkubernetes_version as “omit--kubernetes-version”).
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| scenarios/perf-eval/k8s-ray-scheduling/terraform-inputs/azure.tfvars | Unpins AKS Kubernetes version for ray scheduling scenario so AKS selects a supported default. |
| scenarios/perf-eval/k8s-gpu-scheduling/terraform-inputs/azure.tfvars | Unpins AKS Kubernetes version for GPU scheduling scenario so AKS selects a supported default. |
| scenarios/perf-eval/k8s-gpu-cluster-crud/terraform-inputs/azure.tfvars | Unpins AKS Kubernetes version for GPU cluster CRUD scenario so AKS selects a supported default. |
Leonard Du (LeonardCareer)
approved these changes
Jul 31, 2026
Leonard Du (LeonardCareer)
left a comment
Leonard Du (LeonardCareer)
left a comment
Collaborator
There was a problem hiding this comment.
LGTM
Collaborator
|
/run-tf-integration |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
All scheduled runs of the GPU Cluster CRUD pipeline (ADO
akstelescope/telescopedef 61) — and the sibling GPU scheduling / Ray scheduling pipelines — have been failing onaz aks create:Root cause
The GPU scenarios pinned
kubernetes_version = "1.33". AKS has since aged the entire 1.33 minor into LTS-only support. On aStandard-tier cluster without LTS enabled,--kubernetes-version 1.33resolves to1.33.13and is rejected. Community-supported versions are now 1.34 / 1.35 / 1.36.Fix
Remove the version pin from the Azure inputs of all three GPU scenarios. The
aks-climodule already treatskubernetes_version = nullas "omit--kubernetes-version", so AKS selects its current default supported version. This also prevents recurrence as versions age out.scenarios/perf-eval/k8s-gpu-cluster-crud/terraform-inputs/azure.tfvarsscenarios/perf-eval/k8s-gpu-scheduling/terraform-inputs/azure.tfvarsscenarios/perf-eval/k8s-ray-scheduling/terraform-inputs/azure.tfvarsThe EKS
aws.tfvars(k8s-gpu-cluster-crud) is left on 1.33 — EKS still supports it and no GPU pipeline runs on AWS.Note
The same stale
1.33pin exists in ~12 other Azure perf-eval scenarios and will fail identically; scoped this PR to the GPU pipelines per the request. Happy to follow up with a sweep for the rest.