Skip to content

fix(clickhouse): load profile settings via users.d in customConfig#290

Merged
NitinKumar004 merged 2 commits into
mainfrom
fix/clickhouse-custom-config-usersd
Jun 18, 2026
Merged

fix(clickhouse): load profile settings via users.d in customConfig#290
NitinKumar004 merged 2 commits into
mainfrom
fix/clickhouse-custom-config-usersd

Conversation

@NitinKumar004

Copy link
Copy Markdown
Contributor

Problem

customConfig in the ClickHouse chart was mounted only into /etc/clickhouse-server/config.d/. ClickHouse reads server-level settings from config.d, but profile/user settings are read only from the users config tree (users.d). So a <profiles> block placed in customConfig was silently ignored — no error, no warning, just no effect.

Observed live on the zopnight release: max_server_memory_usage_to_ram_ratio (server-level) applied correctly, but max_memory_usage, max_bytes_before_external_group_by, and max_bytes_before_external_sort (profile-level) never loaded (changed=0, default profile empty). This contributed to ClickHouse OOMs (code 241) on heavy aggregation queries, since the intended per-query cap and spill-to-disk thresholds were never in effect.

This is standard ClickHouse behavior on the official image (not a Bitnami/chart quirk) — config.d and users.d are separate trees read by different parts of the server.

Fix

Mount the same customConfig document into both trees:

  • /etc/clickhouse-server/config.d/custom.xml — server-level settings
  • /etc/clickhouse-server/users.d/custom.xml — profile/user settings

ClickHouse reads the relevant sections from each tree and ignores the rest, so a single document can carry both. No change to deploy values is required — existing customConfig blobs start working after a helm upgrade.

Verification (minikube, official clickhouse/clickhouse-server:25.3)

  • Render: empty customConfig → no ConfigMap/mounts; server-only / profiles-only / mixed → both mounts render.
  • Live deploy of the real mixed config:
    • Pod Running 1/1, 0 restarts; no config-related errors/warnings in logs.
    • Server setting max_server_memory_usage_to_ram_ratio=0.9changed=1.
    • Profile settings max_memory_usage=1.5G, max_bytes_before_external_group_by=750M, max_bytes_before_external_sort=750M → all changed=1; default profile populated; enforced via getSetting().
    • Non-default users: created aggregator_tpbh exactly as the chart does (SQL CREATE USER, no settings profile) and confirmed it inherits all three limits via the server default_profile.

Changes

  • templates/statefulset.yaml — add users.d/custom.xml mount of the existing custom-config ConfigMap.
  • values.yaml — document both mount paths in the customConfig comment.
  • README.md — document dual-mount behavior with example.

🤖 Generated with Claude Code

customConfig was mounted only into /etc/clickhouse-server/config.d/, which
ClickHouse uses for server-level settings. Profile/user settings (e.g.
max_memory_usage, max_bytes_before_external_group_by,
max_bytes_before_external_sort) are read only from the users config tree
(users.d), so a <profiles> block placed in customConfig was silently ignored.

Mount the same customConfig file into /etc/clickhouse-server/users.d/ as well,
so ClickHouse picks up server settings from config.d and profile settings from
users.d. Each tree ignores the sections it does not own, so a single document
can carry both. No change to deploy values is required.

Verified on minikube with the official clickhouse/clickhouse-server:25.3 image:
server and profile settings both load (changed=1), the pod starts with 0
restarts, no config errors, and the limits apply to non-default service users
(e.g. aggregator_*) created without an explicit settings profile, since they
inherit the server default_profile.

@arunesh-j arunesh-j left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update to a newer version and package it ..

@NitinKumar004

Copy link
Copy Markdown
Contributor Author

Done @arunesh-j — bumped the chart to v0.0.3 and packaged it the same way as the last release (#289):

  • charts/clickhouse/Chart.yaml: 0.0.20.0.3
  • packaged docs/clickhouse-v0.0.3.tgz (helm package --version v0.0.3)
  • regenerated docs/index.yaml (merged, older v0.0.2/v0.0.1 entries retained)

helm lint passes. Pushed in c2f46ee.

@NitinKumar004 NitinKumar004 force-pushed the fix/clickhouse-custom-config-usersd branch from c2f46ee to e30a792 Compare June 17, 2026 07:33
@aryanmehrotra aryanmehrotra requested a review from arunesh-j June 18, 2026 05:16
@NitinKumar004 NitinKumar004 merged commit b483213 into main Jun 18, 2026
1 check passed
@NitinKumar004 NitinKumar004 deleted the fix/clickhouse-custom-config-usersd branch June 18, 2026 05:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants