Setup helm chart for garage#1
Conversation
alexlanz
left a comment
alexlanz
left a comment
There was a problem hiding this comment.
@stplasim You mentioned that we don't a job to create the buckets.
Initially we assumed we'd need an init container (or a post-install Job) to create the initial bucket and access key. Turns out Garage v2.3.0 added native flags for exactly this: garage server --single-node --default-bucket reads GARAGE_DEFAULT_ACCESS_KEY / GARAGE_DEFAULT_SECRET_KEY / GARAGE_DEFAULT_BUCKET from the environment and provisions them on first boot. So the bootstrap is fully declarative; no init container, no extra Job, no CLI scripting.
But what if we need multiple buckets or multiple users (ex. one with just read access). Can we handle that then with the current implementation?
| # Set your own (any sufficiently random string) so it is known up front for | ||
| # managing buckets/keys via the admin API. Prefer --set / Terraform over | ||
| # committing it to git. | ||
| token: "" |
There was a problem hiding this comment.
Like this the token is passed as plain text. Do you see no concerns with this approach?
There was a problem hiding this comment.
Not really. In Terraform we pull the token from 1Password and pass it through a sensitive = true variable, same as we already do for e.g. the Grafana admin password or SMTP credentials.
On the cluster side the chart puts it only into a Kubernetes Secret (config is Secret-mounted, no ConfigMap). So the only copies are TF state and Helm's release secret, which hold all our other credentials anyway. An existingSecret indirection wouldn't remove it from TF state either.
That said, we could improve it further by going the 1Password operator route like we do for Loki. But that would mean installing the 1Password Connect operator on the client clusters too, and I'm not sure that's planned.
There was a problem hiding this comment.
Sorry, you are right. Nevermind.
| # Must start with "GK" by Garage convention (GK + 16 hex chars). | ||
| accessKey: "" | ||
| # -- Secret access key for the default bucket. Required when bootstrap.enabled=true. | ||
| secretKey: "" |
You can create as many buckets and tokens as you need. This is only intended for the initial setup. The Helm chart should not reconcile these resources after bootstrapping. Once the server is up, you can use the CLI to manage it however you want. You can also disable bootstrapping entirely and configure everything manually if you prefer. |
|
Oh, I see. This is where we then need the custom operator that manages all the setup of buckets and users right? |
ThoSap
left a comment
ThoSap
left a comment
There was a problem hiding this comment.
Awesome 🔥
Some very minor things.
| runs-on: ubuntu-24.04 | ||
| timeout-minutes: 15 | ||
| steps: | ||
| - uses: actions/checkout@v5 |
There was a problem hiding this comment.
| - uses: actions/checkout@v5 | |
| - uses: actions/checkout@v7 |
| token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }} | ||
| - uses: aboutbits/github-actions-base/git-setup@v2 | ||
| - name: Set up Helm | ||
| uses: azure/setup-helm@v4 |
There was a problem hiding this comment.
| uses: azure/setup-helm@v4 | |
| uses: azure/setup-helm@v5 |
| - name: Set up Helm | ||
| uses: azure/setup-helm@v4 | ||
| with: | ||
| version: 3.21.3 |
There was a problem hiding this comment.
As Helm v4 is out since November 2025, we should think of upgrading to v4 in the future.
v3 is fine for now.
There was a problem hiding this comment.
We could add an additional helm lint workflow for the PR CI.
https://helm.sh/docs/helm/helm_lint/
| - host: s3.example.com | ||
| paths: | ||
| - path: / | ||
| pathType: Prefix |
There was a problem hiding this comment.
If I remember correctly Prefix will log a warning in newer NGINX Ingress versions, we should use the default ImplementationSpecific instead
| pathType: Prefix | |
| pathType: ImplementationSpecific |
| probes: | ||
| liveness: | ||
| httpGet: | ||
| path: /health | ||
| port: admin | ||
| initialDelaySeconds: 10 | ||
| periodSeconds: 10 | ||
| failureThreshold: 6 | ||
| readiness: | ||
| httpGet: | ||
| path: /health | ||
| port: admin | ||
| initialDelaySeconds: 5 | ||
| periodSeconds: 5 |
There was a problem hiding this comment.
These won't work if someone decides to set admin.enabled: false as the named port will not be created, so the Garage StatefulSet will never start.
But I think this is a good start anyway, and a tradeoff we can accept for now.
| - host: s3.example.com | ||
| paths: | ||
| - path: / | ||
| pathType: Prefix |
There was a problem hiding this comment.
| pathType: Prefix | |
| pathType: ImplementationSpecific |
| version: 0.1.0 | ||
| appVersion: "v2.3.0" |
There was a problem hiding this comment.
If we ever accidentally release a version like 1.0 instead of using SemVer 1.0.0, YAML will interpret it as a floating-point number if we don't quote it.
| version: 0.1.0 | |
| appVersion: "v2.3.0" | |
| version: "0.1.0" | |
| appVersion: "v2.3.0" |
| sources: | ||
| - https://git.deuxfleurs.fr/Deuxfleurs/garage |
There was a problem hiding this comment.
Let's also add the GitHub URL.
| sources: | |
| - https://git.deuxfleurs.fr/Deuxfleurs/garage | |
| sources: | |
| - https://git.deuxfleurs.fr/Deuxfleurs/garage | |
| - https://github.com/datahub-local/garage-helm |
3 participants
Setup of the AboutBits Garage Helm chart
Adds a Helm chart for deploying Garage (S3-compatible object storage).
Derived from the official chart, but trimmed to what we need: a single node. No node discovery, clustering, CRDs, or ClusterRoles. Just
replication_factor=1, one StatefulSet replica.First-boot bootstrap without an init container
Initially we assumed we'd need an init container (or a post-install Job) to create the initial bucket and access key. Turns out Garage v2.3.0 added native flags for exactly this:
garage server --single-node --default-bucketreadsGARAGE_DEFAULT_ACCESS_KEY/GARAGE_DEFAULT_SECRET_KEY/GARAGE_DEFAULT_BUCKETfrom the environment and provisions them on first boot. So the bootstrap is fully declarative; no init container, no extra Job, no CLI scripting.Other stuff