fix: unbreak CI and platform RBAC after chart hardening (#226)#433
Merged
Conversation
Commit 1efd648 hardened the Helm chart but broke both CI and the RBAC model on any real cluster. These are the four small, high-confidence fixes. 1. runAsNonRoot with a non-numeric image user. podSecurityContext set runAsNonRoot: true with no runAsUser, and the Dockerfile declares USER by name (mortise). Kubelet cannot verify a non-numeric image user against runAsNonRoot, so the operator pod fails with CreateContainerConfigError, never passes readiness, and helm --wait times out. Both the Integration and UI E2E jobs die here. Add numeric runAsUser/runAsGroup 65532 (mirrors observer.yaml) and make the Dockerfile USER numeric. 2. RBAC escalation-prevention deadlock. The project controller creates a RoleBinding per pj-* namespace referencing ClusterRole mortise-controller-ns, but the operator SA was never granted the bind verb on that role, so Kubernetes forbids the binding and every Project reconcile fails on any RBAC-enforcing cluster. Add a bind rule pinned by resourceNames, plus the kubebuilder marker so config/rbac/role.yaml stays in sync. 3. CI observability. On a helm/cluster failure the logs ended with a single "context deadline exceeded" line and the cluster was then destroyed, leaving every timeout undebuggable. Add if: failure() diagnostics (pods, describe, operator logs, events) before teardown in both cluster jobs, and upload the Playwright report artifact. 4. Release/chart hygiene. Drop the stale mortise-core image override (0.1.0) from the umbrella values so the subchart default and the CI-stamped tag flow through; add projectmembers/status to RBAC so the member-add flow can update status; add alias: metricsServer to the metrics-server dependency so metricsServer.* values (including --kubelet-insecure-tls) actually reach the subchart. Co-Authored-By: Claude Fable 5 <[email protected]>
This was referenced Jul 5, 2026
The camelCase alias `metricsServer` caused Helm to generate resource names like `mortise-metricsServer`, which Kubernetes rejects (RFC 1123 requires lowercase). Remove the alias (the chart name `metrics-server` is already valid) and update all values references to `metrics-server`. Co-Authored-By: Claude Opus 4.6 <[email protected]>
The Makefile deleted the k3d cluster on test failure before the CI diagnostics step could inspect it. Now the Makefile leaves the cluster for diagnostics, and a new always-run step in CI handles teardown. Co-Authored-By: Claude Opus 4.6 <[email protected]>
The #226 RBAC split made every write into pj-* namespaces depend on a RoleBinding that only the async Project reconciler stamped. Two paths raced it: - API handlers create env namespaces synchronously (env create, clone, rename) and then write secrets into them in the same request. Until the reconciler stamped the RoleBinding those writes came back forbidden and surfaced as 500s. The handlers now stamp the binding themselves via the shared internal/nsrbac helper and wait for the apiserver's RBAC cache to reflect it before proceeding. - The App finalizer GC deletes workload resources across env namespaces. A Terminating namespace GCs its RoleBinding before its contents, so those deletes returned forbidden (authz runs before the 404 would), wedging the finalizer with escalating backoff and stalling every delete flow. deleteMatching now skips forbidden deletes when the target namespace is gone or Terminating, since namespace GC owns that cleanup. Co-Authored-By: Claude Opus 4.6 <[email protected]>
…ead-only FS BuildKit's docker auth provider mkdirs $DOCKER_CONFIG (default $HOME/.docker) during every solve, even against anonymous registries. With readOnlyRootFilesystem that mkdir fails and every git-source build dies with "failed to solve: mkdir /home/mortise/.docker: read-only file system". Reproduced on a local k3d cluster; a rebuild under DOCKER_CONFIG=/tmp/.docker succeeded end to end (clone, build, push). Co-Authored-By: Claude Opus 4.6 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Commit
1efd648("harden Helm chart security", #226) hardened the chart but broke CI onmainand the RBAC model on any real cluster.make testandmake test-chartsstill pass at that commit, so none of these failures were visible locally: they only manifest against a real kubelet + RBAC enforcement, which envtest and chart lint cannot see. What started as four small fixes grew as CI kept surfacing deeper fallout from the same hardening commit; each fix below was verified against a real cluster (CI k3d or a local k3d repro).Fixes
1. CI red:
runAsNonRootwith a non-numeric image usercharts/mortise-core/values.yamlsetrunAsNonRoot: truewith norunAsUser, and the Dockerfile declaresUSER mortise(a name, UID 65532 but non-numeric). Kubelet cannot verify a non-numeric image user againstrunAsNonRoot, so the operator container fails withCreateContainerConfigError, never passes readiness, andhelm --waittimes out. Fix: numericrunAsUser/runAsGroup: 65532in the pod security context and a numeric DockerfileUSER.2. RBAC escalation-prevention deadlock
project_controller.ensureRoleBindingcreates a RoleBinding perpj-*namespace referencing ClusterRolemortise-controller-ns, but the operator SA was never granted thebindverb on that role, so Kubernetes escalation prevention forbids the binding and every Project reconcile fails on an RBAC-enforcing cluster. Fix: abindrule pinned byresourceNames: [mortise-controller-ns], plus the kubebuilder marker soconfig/rbac/role.yamlstays in sync.3. Sync API writes raced the async RoleBinding stamp
API handlers (env create, clone, rename) create env namespaces synchronously, but write access inside them arrived only via the async Project reconciler. Secret writes into a fresh namespace returned
forbiddenand surfaced as 500s. Fix: handlers stamp the RoleBinding themselves via the sharedinternal/nsrbachelper and wait for the apiserver's RBAC cache to reflect it (SelfSubjectAccessReview poll) before proceeding.4. App finalizer GC wedged on Terminating namespaces
A Terminating namespace GCs its RoleBinding before its contents, so the finalizer's cross-namespace deletes returned
403 Forbidden(authz runs before the 404 would), wedging Apps in Terminating with escalating backoff and stalling every delete flow. Fix:deleteMatchingskips forbidden deletes when the target namespace is gone or Terminating, since namespace GC owns that cleanup.5.
readOnlyRootFilesystemkilled every git-source buildBuildKit's docker auth provider mkdirs
$DOCKER_CONFIG(default$HOME/.docker) during every solve, even against anonymous registries. Under the hardened read-only root FS every build died withfailed to solve: mkdir /home/mortise/.docker: read-only file system. Reproduced on a local k3d cluster; after settingDOCKER_CONFIG=/tmp/.docker(the writable emptyDir, same pattern asTMPDIR) a rebuild succeeded end to end (clone, build, push).6. CI observability: failures were undebuggable
Failures ended with a single
context deadline exceededline and the cluster was torn down, destroying the evidence. Addedif: failure()diagnostics (pods, describe, operator logs, events) to both cluster jobs, a Playwright report upload, and moved integration-cluster teardown out of the Makefile failure path into anif: always()CI step so diagnostics run against the live cluster.7. Release/chart hygiene
mortise-core.imageoverride (0.1.0) from the umbrella values;release.ymlnever restamped it.projectmembers/statusto the RBAC role (member-add callsStatus().Update()and gotForbiddenunder real RBAC).metrics-servervalues key (a camelCase alias generatedmortise-metricsServerresource names, which Kubernetes rejects as invalid RFC 1123 names).Verification
All six CI jobs green on this branch, including Integration and UI E2E, which had been red on
mainsince1efd648. The git-build fix was additionally validated on a live local k3d cluster where all 12 BuildRuns failed with the mkdir error and a rebuild under the fixed env succeeded.Found via a broader quality audit; remaining audit findings are tracked in #434-#444.