ci(api): Add ClickHouse to the local and CI test stack by khvn26 · Pull Request #7848 · Flagsmith/flagsmith

khvn26 · 2026-06-22T13:08:30Z

Thanks for submitting a PR! Please check the boxes below:

I have read the Contributing Guide.
I have added information to docs/ if required so people know about the feature.
I have filled in the "Changes" section below.
I have filled in the "How did you test this code" section below.

Changes

Contributes to #7408

Stands up a real ClickHouse in the test stack so segment membership code can be tested against actual ClickHouse execution instead of a mocked cursor.

Add a clickhouse service to docker-compose.local.yml, so it comes up for make test both locally and in CI.
Wire CLICKHOUSE_* into .env-ci/.env-local, and wait for it in make wait-for-db. The suite targets the always-present default database; the test runner creates its own test_* database on top, so we don't depend on first-run DB init.
Add a clickhouse_db fixture and a clickhouse marker. ClickHouse has no transactional rollback, so the fixture truncates IDENTITIES on teardown for per-test isolation.
Move the segment-membership tests that stubbed out ClickHouse — compute, refresh, backfill, and log_comment attribution — to live-ClickHouse integration tests under tests/integration/segment_membership/. Skip/defensive branches that never reach ClickHouse stay mocked.
Add a thin ClickHouse test backend that implements _clone_test_db. Without it, pytest --ci results in NotImplementedError.

How did you test this code?

make docker-up brings up the new clickhouse service alongside Postgres.
make test opts='tests/integration/segment_membership/test_segment_membership_clickhouse.py tests/unit/segment_membership/' — all pass.

Verified specifically:

The four migrated integration tests pass against real ClickHouse.
Run under xdist (parallel test-DB cloning works) and with -n0 (two tests sharing one test DB both see the correct counts, confirming truncation-based isolation).
pytest-django creates the CH test database and runs the existing 0001_create_identities migration (JSON-column DDL accepted under allow_experimental_json_type); make wait-for-db waits on ClickHouse.
The remaining mocked segment-membership tests still pass with the alias present.

Stand up a real ClickHouse so segment-membership code can be tested against actual ClickHouse execution instead of a mocked cursor. - Add a clickhouse service (clickhouse-server:25.12, matching ClickHouse Cloud) to docker-compose.local.yml, so it comes up for `make test` both locally and in CI. - Wire CLICKHOUSE_* into .env-ci/.env-local and wait for it in `make wait-for-db`. - Add a `clickhouse_db` fixture and a `clickhouse` marker. ClickHouse has no transactional rollback, so the fixture truncates IDENTITIES on teardown for per-test isolation. - Move the segment-membership tests that stubbed out ClickHouse -- compute, refresh, backfill, and log_comment attribution -- to live-ClickHouse integration tests; leave skip/defensive paths mocked.

vercel · 2026-06-22T13:08:36Z

The latest updates on your projects. Learn more about Vercel for GitHub.

3 Skipped Deployments

Project	Deployment	Actions	Updated (UTC)
docs	Ignored	Preview	Jun 22, 2026 6:31pm
flagsmith-frontend-preview	Ignored	Preview	Jun 22, 2026 6:31pm
flagsmith-frontend-staging	Ignored	Preview	Jun 22, 2026 6:31pm

github-actions · 2026-06-22T13:09:37Z

Docker builds report

Image	Build Status	Security report
`ghcr.io/flagsmith/flagsmith-e2e:pr-7848`	Finished ✅	Skipped
`ghcr.io/flagsmith/flagsmith-api-test:pr-7848`	Finished ✅	Skipped
`ghcr.io/flagsmith/flagsmith-frontend:pr-7848`	Finished ✅	Results ✅
`ghcr.io/flagsmith/flagsmith-api:pr-7848`	Finished ✅	Results ✅
`ghcr.io/flagsmith/flagsmith:pr-7848`	Finished ✅	Results ✅
`ghcr.io/flagsmith/flagsmith-private-cloud:pr-7848`	Finished ✅	Results ✅

github-actions · 2026-06-22T13:15:49Z

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

1 passed

Details

1 test across 1 suite
44.2 seconds
8365717
🔄 Run: #17707 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-16)

1 passed

Details

1 test across 1 suite
36.8 seconds
8365717
🔄 Run: #17707 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-16)

3 passed

Details

3 tests across 3 suites
54.9 seconds
8365717
🔄 Run: #17707 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)

2 passed

Details

2 tests across 2 suites
59 seconds
8365717
🔄 Run: #17707 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)

21 passed

Details

21 tests across 16 suites
1 minute, 18 seconds
7f1342e
🔄 Run: #17708 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-16)

1 passed

Details

1 test across 1 suite
35.3 seconds
7f1342e
🔄 Run: #17708 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

1 passed

Details

1 test across 1 suite
44.8 seconds
7f1342e
🔄 Run: #17708 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-16)

2 passed

Details

2 tests across 2 suites
47.7 seconds
7f1342e
🔄 Run: #17708 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

1 passed

Details

1 test across 1 suite
45.8 seconds
2de66db
🔄 Run: #17710 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-16)

1 passed

Details

1 test across 1 suite
35.4 seconds
2de66db
🔄 Run: #17710 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)

2 passed

Details

2 tests across 2 suites
59 seconds
2de66db
🔄 Run: #17710 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-16)

1 passed

Details

1 test across 1 suite
51.9 seconds
2de66db
🔄 Run: #17710 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-16)

1 passed

Details

1 test across 1 suite
35.6 seconds
a1a7863
🔄 Run: #17715 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

1 passed

Details

1 test across 1 suite
43.8 seconds
a1a7863
🔄 Run: #17715 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-16)

3 passed

Details

3 tests across 3 suites
33 seconds
a1a7863
🔄 Run: #17715 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)

2 passed

Details

2 tests across 2 suites
1 minute, 14 seconds
a1a7863
🔄 Run: #17715 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-16)

1 passed

Details

1 test across 1 suite
39.7 seconds
40cb5a2
🔄 Run: #17716 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

2 passed

Details

2 tests across 2 suites
45.2 seconds
40cb5a2
🔄 Run: #17716 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-16)

3 passed

Details

3 tests across 3 suites
34.1 seconds
40cb5a2
🔄 Run: #17716 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)

1 passed

Details

1 test across 1 suite
1 minute, 8 seconds
40cb5a2
🔄 Run: #17716 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-16)

2 passed

Details

2 tests across 2 suites
41.6 seconds
6dcf528
🔄 Run: #17718 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

1 passed

Details

1 test across 1 suite
44 seconds
6dcf528
🔄 Run: #17718 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-16)

3 passed

Details

3 tests across 3 suites
34.3 seconds
6dcf528
🔄 Run: #17718 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)

4 passed

Details

4 tests across 4 suites
1 minute
6dcf528
🔄 Run: #17718 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-16)

1 passed

Details

1 test across 1 suite
40.5 seconds
6eecd36
🔄 Run: #17719 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

1 passed

Details

1 test across 1 suite
44.1 seconds
6eecd36
🔄 Run: #17719 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-16)

3 passed

Details

3 tests across 3 suites
32.1 seconds
6eecd36
🔄 Run: #17719 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)

3 passed

Details

3 tests across 3 suites
1 minute, 9 seconds
6eecd36
🔄 Run: #17719 (attempt 1)

github-actions · 2026-06-22T13:16:27Z

Visual Regression

19 screenshots compared. See report for details.
View full report

The upstream clickhouse_backend leaves _clone_test_db unimplemented, so setup_databases(parallel=...) raises NotImplementedError and aborts the whole session under `pytest -n auto`. Add a thin backend subclass that clones the primary test database per xdist worker, and point the engine at it. beep boop

codecov · 2026-06-22T14:23:45Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.59%. Comparing base (fe8d77f) to head (6eecd36).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #7848   +/-   ##
=======================================
  Coverage   98.59%   98.59%           
=======================================
  Files        1467     1472    +5     
  Lines       57332    57370   +38     
=======================================
+ Hits        56525    56564   +39     
+ Misses        807      806    -1

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Zaimwa9

Non blocking comment on my side, anticipating new tests around experimentation coming soon. I leave it up to your appreciation, the rest makes sense to me (once confirmed by CI) and thanks for the issue in clickhouse

emyller

Looks fine; few non-blocking questions.

Co-authored-by: Evandro Myller <[email protected]>

This read-endpoint test leaked into the ClickHouse test-stack branch and references segment_membership.services.get_segment_members, which doesn't exist here. Remove it so the PR only contains ClickHouse changes. beep boop

khvn26 requested review from a team as code owners June 22, 2026 13:08

khvn26 requested review from emyller and removed request for a team June 22, 2026 13:08

flagsmith-engineering Bot assigned emyller Jun 22, 2026

github-actions Bot added api Issue related to the REST API docs Documentation updates ci-cd Build, test and deployment related and removed docs Documentation updates labels Jun 22, 2026