Skip to content

feat(sql): support batch-scoped ClickHouse settings - #945

Draft
catkins wants to merge 1 commit into
warpstreamlabs:mainfrom
catkins:catkins/sql-insert-clickhouse-settings
Draft

feat(sql): support batch-scoped ClickHouse settings#945
catkins wants to merge 1 commit into
warpstreamlabs:mainfrom
catkins:catkins/sql-insert-clickhouse-settings

Conversation

@catkins

@catkins catkins commented Jul 23, 2026

Copy link
Copy Markdown

🤖 Adds batch-scoped ClickHouse query settings to sql_insert, primarily to support native insert deduplication without falling back to http_client URL parameters.

Motivation

ClickHouse deduplication requires an insert_deduplication_token that is unique between logical batches but stable when the same insert is retried. The existing sql_insert output had no way to attach clickhouse-go settings to the native prepared batch.

API and semantics

output:
  sql_insert:
    driver: clickhouse
    dsn: clickhouse://default:@localhost:9000/default
    table: events
    columns: [event_id, payload]
    args_mapping: 'root = [this.event_id, this]'
    clickhouse_settings:
      insert_deduplication_token: '${! uuid_v4() }'
      deduplicate_blocks_in_dependent_materialized_views: '1'
    batching:
      count: 1000
      period: 1s

Each interpolated setting is resolved exactly once against the first message of a dispatched output batch. The resulting ClickHouse context is shared by every row appended to the native prepared batch and reused unchanged across Bento's ErrNotConnected reconnect retry for that output transaction.

An upstream nack/reprocessing, process restart, or later rebatching is a new output transaction and reevaluates the settings. This feature therefore preserves retry identity inside the sql_insert dispatch lifecycle; it does not provide a durable token across arbitrary upstream replays.

Design trade-offs

  • Adds a generic, optional BatchOutputContextPreparer extension so lifecycle-owned values can be derived at the actual output-batch boundary and reused by the reconnect loop.
  • Keeps clickhouse_settings explicitly ClickHouse-only because database/sql has no portable query-settings contract.
  • Wraps only configured ClickHouse outputs with the optional hook. Existing drivers and configurations keep the previous writer and Prepare paths unchanged.
  • Uses Tx.PrepareContext, which clickhouse-go maps to its native PrepareBatch; per-record Exec calls append rows to that one batch before commit.

Relevant driver behavior: clickhouse-go PrepareContext and native batch preparation. ClickHouse documents insert_deduplication_token and deduplicate_blocks_in_dependent_materialized_views.

Verification

  • mise exec -- go test ./internal/component/output ./internal/impl/sql ./public/service
  • mise exec -- go test ./internal/impl/sql -run '^TestIntegrationClickhouse$' -timeout 5m -count=1 -v
  • mise x [email protected] -- make lint (0 issues)
  • mise exec -- go mod tidy (no changes)
  • Generated component docs with make docs

The new focused tests cover different tokens between batches, one token across all native row appends, stable identity on reconnect retry, preparation failure behavior, non-ClickHouse rejection, and unchanged optional-hook behavior for existing outputs. The live ClickHouse integration inserts the same two-row batch twice with a fixed token and verifies only two rows remain.

The repository-wide mise exec -- make test completed all changed packages successfully but failed in five pre-existing macOS filesystem-event tests under internal/impl/io due CHMOD/WRITE event ordering and subdirectory watch timing. The full docs lint phase is locally blocked by its CGO/libzmq requirement; the generated sql_insert documentation is included, and upstream CI installs libzmq3-dev for that matrix.

Rollback

Revert this commit. The new configuration field is opt-in and requires no data migration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant