Storage controller for Garage (S3-compatible) clusters.

Overview

garage-storage-controller handles bucket and access key management for Garage storage clusters.

Project status

This project is in beta and the core functionality is present. Due to the limited scope and complexity, no major changes are expected.

Important considerations:

• CRDs are still alpha and the API might need to change.
• The controller focuses on the Garage admin API for its functionality.
  • Open an issue if an important bucket config option should be exposed.
• Bucket configuration is primarily managed through the S3 API
  • Use s3 clients and client libs for e.g. policies.
  • Check Garage: S3 compatibility status for details.

Limitations:

• Accessing buckets outside the 'owning' namespace is not allowed.
• ConfigMap drift is hard to detect due to the current RBAC & security model.

Prerequisites

• A running Garage cluster
• A Kubernetes cluster with the controller installed (see Install)
• The controller's ServiceAccount granted access to your namespace (see Enrolling namespaces)

Quick start

apiVersion: garage.getclustered.net/v1alpha1
kind: Bucket
metadata:
  name: bucket-sample
spec:
  name: foo-global-name
  maxSize: 2Gi

---
apiVersion: garage.getclustered.net/v1alpha1
kind: AccessKey
metadata:
  name: accesskey-sample
spec:
  secretName: foo-bucket-access-rw

---
apiVersion: garage.getclustered.net/v1alpha1
kind: AccessPolicy
metadata:
  name: accesspolicy-sample
spec:
  accessKey: accesskey-sample
  bucket: bucket-sample
  permissions:
    read: true
    write: true
    owner: false

This will create a bucket and an access key on Garage. Permissions will be set via the admin API.

Namespace access required: The controller must be granted permission to read and write ConfigMaps and Secrets in your namespace before reconciliation will work. If resources stay in a pending state, this is the most likely cause. See Enrolling namespaces.

Bucket configuration and access key will be stored in Kubernetes objects:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: bucket-sample
data:
  bucket-name: "foo-global-name-abcd1234"
  s3-endpoint: "https://s3.garage.foo.net"

---
apiVersion: v1
kind: Secret
metadata:
  name: foo-bucket-access-rw
data:
  access-key-id: "ABC..."
  secret-access-key: "..."

CRDs

• Bucket: Creates S3 buckets on the Garage cluster
• AccessKey: Generate credentials and store them as namespace secrets
• AccessPolicy: Control which keys can access a bucket

Objectives and scope

Creating and managing storage buckets through Kubernetes API resources should be simple:

• Focus on the common use cases
• Create buckets and access keys for namespace workloads
• Keep this controller safe and easy to operate

Note: Deleting Bucket API resources will never remove the actual buckets on the storage backend. Data loss is not boring. And we aim for boring here.

Accessing buckets in workloads

The S3 API address and the access key are available in the aforementioned ConfigMap and Secret in the namespace.

Mounting bucket details as environment variables

The configmap as well as the secret can be consumed as environment variables:

kind: Pod
spec:
  containers:
  - name: workload
    env:
    # s3 client config:
    - name: S3_ENDPOINT
      valueFrom:
        configMapKeyRef:
          name: bucket-sample
          key: s3-endpoint
    # bucket config values:
    - name: FOO_BUCKET_NAME
      valueFrom:
        configMapKeyRef:
          name: bucket-sample
          key: bucket-name
    - name: FOO_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: foo-bucket-access-rw
          key: access-key-id
    - name: FOO_ACCESS_KEY_SECRET
      valueFrom:
        secretKeyRef:
          name: foo-bucket-access-rw
          key: secret-access-key

Overriding the ConfigMap name

By default the controller will create a ConfigMap with the same name as the Bucket.

If you need a different name e.g. in case of a conflict, set configMapName:

apiVersion: garage.getclustered.net/v1alpha1
kind: Bucket
metadata:
  name: bucket-sample
spec:
  name: foo-global-name
  configMapName: bucket-sample-config
  maxSize: 10Gi

Importing existing buckets

1. Obtain a key with owner permissions on the Garage bucket and create a secret in the target namespace.

2. Create a Bucket API resource and set spec.existingBucket accordingly.

Example:

apiVersion: garage.getclustered.net/v1alpha1
kind: Bucket
metadata:
  name: foo-objects
spec:
  existingBucket:
    name: foo-existing-bucket-d7f8a   # exact name in Garage
    ownerKeySecret: foo-owner-key     # Secret reference
---

apiVersion: v1
kind: Secret
metadata:
  name: foo-owner-key
data:
  access-key-id: "XYZ..."
  secret-access-key: "..."

Install

Configuration

The controller expects the following configuration to be available as environment variables.

Env variable	Description
GARAGE_API_ENDPOINT	Endpoint address of the Garage admin API.
GARAGE_API_TOKEN	Token used to authenticate requests to the Garage admin API.
GARAGE_S3_API_ENDPOINT	Endpoint address of the S3 client API provided to workloads.

See the Garage Administration API docs on how to obtain an admin token.

The controller needs the following Garage Admin API permissions:

Currently used:

Endpoint	Purpose
POST /v2/CreateKey	Create S3 access keys for tenants
GET /v2/GetKeyInfo	Look up a key by ID or name
POST /v2/DeleteKey	Delete S3 access keys
POST /v2/CreateBucket	Create new buckets
GET /v2/GetBucketInfo	Look up a bucket by alias or ID
POST /v2/UpdateBucket	Set bucket quotas
POST /v2/AllowBucketKey	Grant key permissions on a bucket
POST /v2/DenyBucketKey	Revoke key permissions from a bucket

Planned:

Endpoint	Purpose
GET /v2/ListBuckets	Discover orphaned buckets and audit
POST /v2/DeleteBucket	Force bucket deletion (future policy/override feature)

Install with Helm (recommended)

The Helm chart oci bundle is at oci://ghcr.io/bmarinov/charts/garage-storage-controller.
See chart/README.md for details on the available app and chart versions.

Chart values

Provide required configuration values:

# values.yaml
config:
  garageApiEndpoint: "http://garage:3903"
  garageS3ApiEndpoint: "http://garage:3900"

One of existingSecret and apiToken must be set.

Existing secret:

Create a secret containing the admin token:

apiVersion: v1
kind: Secret
metadata:
  name: garage-api-token
stringData:
  api-token: "***"

Set the value accordingly

existingSecret: "garage-api-token"

Alternatively, pass the API token directly as a chart value. A secret with a default name will be created:

helm install ... --set apiToken="$garage_api_token"

Install

helm install garage-storage-controller \
  oci://ghcr.io/bmarinov/charts/garage-storage-controller \
  -n garage-controller-system --create-namespace \
  -f myvalues.yaml

Upgrade

Apply the updated CRDs first, then upgrade the chart. Pass --reuse-values to keep your existing configuration:

kubectl apply -f https://github.com/bmarinov/garage-storage-controller/releases/download/<version>/crds-<version>.yaml

helm upgrade garage-storage-controller \
  oci://ghcr.io/bmarinov/charts/garage-storage-controller \
  --version <chart-version> \
  -n garage-controller-system \
  --reuse-values

Enrolling namespaces

The controller watches custom resources cluster-wide. It requires explicit permission to read and write ConfigMaps and Secrets in tenant namespaces. Without this, reconciliation will fail:

{"error": "configmaps \"...\" is forbidden: User \"system:serviceaccount:garage-controller-system:...\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"default\""}

After installation, the ServiceAccount name is printed in the Helm notes. Use it in the RoleBinding subject below.
The following example enrolls the default namespace:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: garage-controller-namespace-access
  namespace: default # target namespace
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["create", "delete", "get", "list", "patch", "update"]
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["create", "delete", "get", "list", "patch", "update"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: garage-controller-namespace-access-rolebinding
  namespace: default # target namespace
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: garage-controller-namespace-access
subjects:
- kind: ServiceAccount
  name: garage-storage-controller # controller ServiceAccount
  namespace: garage-controller-system # controller namespace 

Manual installation

This method is not supported as it is more error-prone and not fully customizable.

The recommendation is to stick to helm template and helm show crds as a fallback.

Configuration

Create a configmap and a secret in the controller namespace and reference them in the deployment manifest.

Examples can be found in config/env:

• kustomization.yaml contains env var patches for the deployment.
• configmap.yaml - configmap with keys for the endpoint addresses.
• secret.yaml - details on creating a valid secret.

Custom resources

Output CRD manifests to a file:

kubectl kustomize ./config/install/crds -o crds.yaml

Or install directly in the cluster:

kubectl apply -k ./config/install/crds

RBAC

Controller Role and ServiceAccount

kubectl kustomize ./config/install/rbac -o rbac.yaml

This customization creates:

• the controller namespace and a service account
• cluster roles for CRDs
• leader election roles

Namespaced / tenant roles

Managing resources in a given namespace requires permissions over Secrets and ConfigMap resources.

Example with the included default namespace overlay:

kubectl kustomize ./config/rbac/namespaces/default -o default-ns-access.yaml

This will create a role and a role binding in one specific namespace.
It is not recommended, but you can also provide cluster-wide access to configmaps and secrets.

Deployment

The controller kustomization includes an example deployment manifest:

kubectl kustomize ./config/install/controller -o deployment.yaml

The deployment will remain in status CreateContainerConfigError until the expected ConfigMap and Secret are created. See Configuration for more details and an example.

Full manifest

Manifests can also be rendered to a single file:

kubectl kustomize ./config/install/full > dist/install.yaml

Permissions model and security

The controller needs cluster-level permissions for its CRDs. ConfigMap and Secret access can be restricted to specific namespaces.

CRDs

The controller manages the following custom resources in the garage.getclustered.net group:

• accesskeys
• accesspolicies
• buckets

See config/rbac/role.yaml for the ClusterRole definition.

ConfigMaps and Secrets

The controller stores bucket connection details in ConfigMaps and S3 credentials in Secrets. These resources are created in the same namespace as the Bucket and AccessKey resources.

Grant access and enroll a namespace (example):

kubectl apply -k "config/rbac/namespaces/${namespace}"

See config/rbac/base/role_namespace_access.yaml for the Role definition.

Note: The controller does not need cluster-wide access to ConfigMaps or Secrets. Use namespace roles.

Observability

Metrics

The controller exposes Prometheus metrics for its Garage admin API dependency:

Metric	Type	Description
garage_controller_admin_api_up	Gauge	1 when the Garage admin API is reachable, 0 otherwise. On a 30s refresh.
garage_controller_admin_api_requests_total	Counter	Admin API requests; labels: status_class (2xx/3xx/4xx/5xx or error).
garage_controller_admin_api_request_duration_seconds	Histogram	Admin API request latency in seconds.

Logs

Logs follow controller-runtime conventions, using info and error levels.

Relevant for operations:

• Preflight check logged on startup (config validation and auth).
• Garage connectivity loss logs at Error and recovery at Info.

Development

Project setup

Scaffolding done with kubebuilder. See docs for more info.

Running tests

E2E

Run suite with make test-e2e. Requires Kind to be installed and available.:

• Download a specific release:

version="v0.30.0"
[ "$(uname -m)" = x86_64 ] && curl -Lo ./kind https://kind.sigs.k8s.io/dl/$version/kind-linux-amd64
chmod +x ./kind
sudo mv ./kind /usr/local/bin/kind

• Or latest from Homebrew: brew install kind

Debugging

E2E Tests

Setup environment with make setup-test-e2e and launch tests in e2e package.

VS Code launch profile:

{
  "configurations": [
    {
        "name": "Debug E2E Tests",
        "type": "go",
        "request": "launch",
        "mode": "test",
        "program": "${workspaceFolder}/test/e2e",
        "env": {
            "KIND_CLUSTER": "garage-storage-controller-test-e2e",
            "RUN_E2E": "true"
        }
    }
  ]
}