feat(sql): support batch-scoped ClickHouse settings - #945
Draft
catkins wants to merge 1 commit into
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 Adds batch-scoped ClickHouse query settings to
sql_insert, primarily to support native insert deduplication without falling back tohttp_clientURL parameters.Motivation
ClickHouse deduplication requires an
insert_deduplication_tokenthat is unique between logical batches but stable when the same insert is retried. The existingsql_insertoutput had no way to attach clickhouse-go settings to the native prepared batch.API and semantics
Each interpolated setting is resolved exactly once against the first message of a dispatched output batch. The resulting ClickHouse context is shared by every row appended to the native prepared batch and reused unchanged across Bento's
ErrNotConnectedreconnect retry for that output transaction.An upstream nack/reprocessing, process restart, or later rebatching is a new output transaction and reevaluates the settings. This feature therefore preserves retry identity inside the
sql_insertdispatch lifecycle; it does not provide a durable token across arbitrary upstream replays.Design trade-offs
BatchOutputContextPreparerextension so lifecycle-owned values can be derived at the actual output-batch boundary and reused by the reconnect loop.clickhouse_settingsexplicitly ClickHouse-only becausedatabase/sqlhas no portable query-settings contract.Preparepaths unchanged.Tx.PrepareContext, which clickhouse-go maps to its nativePrepareBatch; per-recordExeccalls append rows to that one batch before commit.Relevant driver behavior: clickhouse-go
PrepareContextand native batch preparation. ClickHouse documentsinsert_deduplication_tokenanddeduplicate_blocks_in_dependent_materialized_views.Verification
mise exec -- go test ./internal/component/output ./internal/impl/sql ./public/servicemise exec -- go test ./internal/impl/sql -run '^TestIntegrationClickhouse$' -timeout 5m -count=1 -vmise x [email protected] -- make lint(0 issues)mise exec -- go mod tidy(no changes)make docsThe new focused tests cover different tokens between batches, one token across all native row appends, stable identity on reconnect retry, preparation failure behavior, non-ClickHouse rejection, and unchanged optional-hook behavior for existing outputs. The live ClickHouse integration inserts the same two-row batch twice with a fixed token and verifies only two rows remain.
The repository-wide
mise exec -- make testcompleted all changed packages successfully but failed in five pre-existing macOS filesystem-event tests underinternal/impl/iodueCHMOD/WRITEevent ordering and subdirectory watch timing. The full docs lint phase is locally blocked by its CGO/libzmq requirement; the generatedsql_insertdocumentation is included, and upstream CI installslibzmq3-devfor that matrix.Rollback
Revert this commit. The new configuration field is opt-in and requires no data migration.