feat(clickhouse): GCS backups + clickhouse-backup engine with keyless auth#1
Merged
Merged
Conversation
…less auth Adds two backup capabilities to the chart: - GCS as a native backup destination (backup.destination.type=gcs), using ClickHouse's BACKUP ... TO S3() against the GCS S3-interop endpoint with HMAC credentials. - An opt-in Altinity clickhouse-backup engine (backup.engine=clickhouse-backup) that runs as a sidecar in the server pods and is driven over its REST API by a CronJob. Unlike the native engine it supports keyless cloud auth — EKS IRSA for S3 and GKE Workload Identity for GCS (the only way to reach GCS keyless, since the native S3-interop path is HMAC-only) — plus static keys. Full multi-shard coverage: data backup from each shard's lead replica, optional schema-only on the others. Image pinned to altinity/clickhouse-backup:2.7.2-fips. The native engine remains the default; existing values are unchanged. Bumps chart version 0.1.0 -> 0.3.0. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Extends the chart's backup support in two ways:
backup.destination.type=gcs) — uses ClickHouse'sBACKUP … TO S3()against the GCS S3-interop endpoint with HMAC credentials.clickhouse-backupengine (backup.engine=clickhouse-backup) — runs as a sidecar in the server pods (it needs file access to/var/lib/clickhouse) and is triggered over its REST API by a CronJob.The native engine remains the default; existing values are unchanged. Chart version
0.1.0→0.3.0.Why the new engine
The native engine's object-store auth is HMAC-only. For GCS that means the S3-interop endpoint, which cannot use GKE Workload Identity. The
clickhouse-backupengine speaks native cloud SDKs, enabling keyless auth:s3.auth=iam+serviceAccount.annotationsrole-arngcs.auth=workloadIdentity+saEmail+ SA annotationStatic keys are still supported (
s3.auth=keys/gcs.auth=key, inline or viaexistingSecret).Behavior
clickhouse-backup server(REST API on:7171, basic-auth reusing the admin credential).backup.schedulecallscreate_remoteagainst each shard's lead replica (full multi-shard), plus optional schema-only on the other replicas (schemaOnlyOtherReplicas). It polls/backup/statusand fails the Job on error.altinity/clickhouse-backup:2.7.2-fips(overridable).Files
values.yaml/values.schema.json—backup.engine+backup.clickhouseBackupconfig block.templates/_helpers.tpl—chbEnabled,chbSecretName, sharedchbEnv.templates/statefulset-server.yaml— sidecar container.templates/service.yaml— API port on the headless service.templates/cronjob-backup-chb.yaml(new) — trigger CronJob.templates/cronjob-backup.yaml/secret-backup.yaml— guarded/branched by engine.README.md,Chart.yaml.Testing
helm lintpasses.🤖 Generated with Claude Code