Skip to content

Ensure read write many first#339

Open
nikola-jokic wants to merge 4 commits intomainfrom
nikola-jokic/rwx-volume-by-default
Open

Ensure read write many first#339
nikola-jokic wants to merge 4 commits intomainfrom
nikola-jokic/rwx-volume-by-default

Conversation

@nikola-jokic
Copy link
Copy Markdown
Collaborator

No description provided.

Copilot AI review requested due to automatic review settings April 22, 2026 15:24
@nikola-jokic nikola-jokic requested a review from a team as a code owner April 22, 2026 15:24
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the k8s runner hooks to use a shared PersistentVolumeClaim-backed work volume (instead of ad-hoc copy/exec flows) and switches container-step execution from standalone Pods to batch Jobs, aligning storage behavior across job/step containers.

Changes:

  • Introduce a single shared volume mount model (POD_VOLUME_NAME PVC + containerVolumes()), and update job/step execution to rely on it.
  • Replace container-step “pod per step” with a Kubernetes Job + helper to resolve the job’s pod name.
  • Update test setup to provision StorageClass/PV/PVC for e2e tests and clean them up.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
packages/k8s/tests/test-setup.ts Provisions StorageClass/PV/PVC for tests; updates initialization/cleanup sequencing around the new shared volume model.
packages/k8s/src/k8s/utils.ts Adds containerVolumes() and writeEntryPointScript(); updates argument handling (fixArgs) for exec invocations.
packages/k8s/src/k8s/index.ts Switches job pod volume to PVC, adds Job creation path for container steps, updates permissions, and introduces node pinning/affinity helpers.
packages/k8s/src/hooks/run-script-step.ts Removes copy/merge temp-dir logic and runs via entrypoint script expected to be present on the shared volume.
packages/k8s/src/hooks/run-container-step.ts Runs container steps as Jobs, streams logs, and optionally injects env via Secret.
packages/k8s/src/hooks/prepare-job.ts Uses createPod, mounts via containerVolumes, and copies externals into the shared work volume path.
Comments suppressed due to low confidence (1)

packages/k8s/tests/test-setup.ts:90

  • cleanupK8sResources deletes the PVC/PV before deleting the pods that may be using the claim. This can leave the PVC stuck terminating or cause the PV delete to fail (still bound). Delete the job/workflow pods first, then the PVC, then the PV, and finally the StorageClass.
  async cleanupK8sResources(): Promise<void> {
    await k8sApi
      .deleteNamespacedPersistentVolumeClaim({
        name: `${this.podName}-work`,
        namespace: 'default',
        gracePeriodSeconds: 0
      })
      .catch((e: k8s.ApiException<any>) => {
        if (e.code !== 404) {
          console.error(JSON.stringify(e))
        }
      })
    await k8sApi
      .deletePersistentVolume({ name: `${this.podName}-pv` })
      .catch((e: k8s.ApiException<any>) => {
        if (e.code !== 404) {
          console.error(JSON.stringify(e))
        }
      })
    await k8sApi
      .deleteNamespacedPod({
        name: this.podName,
        namespace: 'default',
        gracePeriodSeconds: 0
      })

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +30 to +44
if (stepContainer.environmentVariables) {
try {
const envs = JSON.parse(
JSON.stringify(stepContainer.environmentVariables)
)
envs['GITHUB_ACTIONS'] = 'true'
if (!('CI' in envs)) {
envs.CI = 'true'
}
secretName = await createSecretForEnvs(envs)
} catch (err) {
core.debug(`createSecretForEnvs failed: ${JSON.stringify(err)}`)
const message = (err as any)?.response?.body?.message || err
throw new Error(`failed to create script environment: ${message}`)
}
Copy link

Copilot AI Apr 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Environment variables like GITHUB_ACTIONS/CI are only added (via a Secret) when stepContainer.environmentVariables is defined. If it’s undefined, the container step runs without these vars, which is a behavior change from the previous implementation. Consider always constructing an env map (starting with GITHUB_ACTIONS and CI) and creating a secret (or setting podContainer.env) regardless of whether the caller provided additional env vars.

Suggested change
if (stepContainer.environmentVariables) {
try {
const envs = JSON.parse(
JSON.stringify(stepContainer.environmentVariables)
)
envs['GITHUB_ACTIONS'] = 'true'
if (!('CI' in envs)) {
envs.CI = 'true'
}
secretName = await createSecretForEnvs(envs)
} catch (err) {
core.debug(`createSecretForEnvs failed: ${JSON.stringify(err)}`)
const message = (err as any)?.response?.body?.message || err
throw new Error(`failed to create script environment: ${message}`)
}
try {
const envs = JSON.parse(
JSON.stringify(stepContainer.environmentVariables ?? {})
)
envs['GITHUB_ACTIONS'] = 'true'
if (!('CI' in envs)) {
envs.CI = 'true'
}
secretName = await createSecretForEnvs(envs)
} catch (err) {
core.debug(`createSecretForEnvs failed: ${JSON.stringify(err)}`)
const message = (err as any)?.response?.body?.message || err
throw new Error(`failed to create script environment: ${message}`)

Copilot uses AI. Check for mistakes.
@@ -288,18 +396,5 @@ function mergeLists<T>(base?: T[], from?: T[]): T[] {
}

export function fixArgs(args: string[]): string[] {
Copy link

Copilot AI Apr 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixArgs currently always re-tokenizes by shlex.split(args.join(' ')). This breaks commands of the form ['sh','-c','<script>'] because the script string gets split into multiple args (e.g. tests call execPodStep(['sh','-c','[ "$(cat ...)" = ... ] || exit 5'], ...)). Reintroduce the sh -c preservation logic (or equivalent) so the third argument remains a single script string.

Suggested change
export function fixArgs(args: string[]): string[] {
export function fixArgs(args: string[]): string[] {
if (args.length >= 3 && args[0] === 'sh' && args[1] === '-c') {
return [args[0], args[1], args.slice(2).join(' ')]
}

Copilot uses AI. Check for mistakes.
'sh',
'-c',
`[ $(cat /etc/*release* | grep -i -e "^ID=*alpine*" -c) != 0 ] || exit 1`
`'[ $(cat /etc/*release* | grep -i -e "^ID=*alpine*" -c) != 0 ] || exit 1'`
Copy link

Copilot AI Apr 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The sh -c command string here is wrapped in single quotes, which prevents shell expansions like $() from running and will likely make the Alpine detection always fail. Pass the test expression without the outer quotes (and rely on fixArgs preserving sh -c scripts) so the subshell and grep execute as intended.

Suggested change
`'[ $(cat /etc/*release* | grep -i -e "^ID=*alpine*" -c) != 0 ] || exit 1'`
'[ $(cat /etc/*release* | grep -i -e "^ID=*alpine*" -c) != 0 ] || exit 1'

Copilot uses AI. Check for mistakes.
Comment on lines +147 to +160
job.metadata.name = getStepPodName()
job.metadata.labels = { [runnerInstanceLabel.key]: runnerInstanceLabel.value }
job.metadata.annotations = {}

job.spec = new k8s.V1JobSpec()
job.spec.ttlSecondsAfterFinished = 300
job.spec.backoffLimit = 0
job.spec.template = new k8s.V1PodTemplateSpec()

job.spec.template.spec = new k8s.V1PodSpec()
job.spec.template.metadata = new k8s.V1ObjectMeta()
job.spec.template.metadata.labels = {}
job.spec.template.metadata.annotations = {}
job.spec.template.spec.containers = [container]
Copy link

Copilot AI Apr 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

createJob applies RunnerInstanceLabel only to the Job object, but the Pods created by the Job will not automatically inherit Job metadata labels. Since prunePods() selects pods by RunnerInstanceLabel, these step pods won't be pruned. Add the same label to job.spec.template.metadata.labels (and keep it when merging any extension metadata).

Copilot uses AI. Check for mistakes.
)
} catch (error) {
core.warning('Failed to copy _temp from pod')
fs.rmSync(runnerPath)
Copy link

Copilot AI Apr 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fs.rmSync(runnerPath) in the finally block will throw if the file is already removed or was never created, which can mask the original error from execPodStep. Use force: true (or wrap in a try/catch) to make cleanup non-fatal.

Suggested change
fs.rmSync(runnerPath)
fs.rmSync(runnerPath, { force: true })

Copilot uses AI. Check for mistakes.
@nikola-jokic nikola-jokic force-pushed the nikola-jokic/rwx-volume-by-default branch from c47c78e to 5d337ff Compare April 22, 2026 20:59
nikola-jokic and others added 2 commits April 23, 2026 00:09
…nity

- RWX volumes allow job pods to be scheduled on any cluster node
- RWO volumes require affinity to pin job pods to runner's node
- Remove ACTIONS_RUNNER_USE_KUBE_SCHEDULER from RWX migration steps
- Emphasize resource utilization benefits of RWX free scheduling

Co-authored-by: Sisyphus <[email protected]>
Change ACTIONS_RUNNER_USE_KUBE_SCHEDULER to ACTIONS_RUNNER_DISABLE_KUBE_SCHEDULER
to make affinity-based scheduling the first-class (default) implementation.

Breaking Change:
- OLD: Set ACTIONS_RUNNER_USE_KUBE_SCHEDULER=true to enable affinity (opt-in)
- NEW: Affinity is enabled by default, set ACTIONS_RUNNER_DISABLE_KUBE_SCHEDULER=true to disable (opt-out)

Code changes:
- utils.ts: Rename constant and invert useKubeScheduler() logic
- rwo-affinity-test.ts: Update tests to verify default affinity behavior
- ADR 0135: Update to reflect opt-out model
- README: Update guidance to reflect default behavior

Co-authored-by: Sisyphus <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants