feat: columnar DataFrame ingest (Arrow / Polars / Pandas) + Arrow egress#153
Open
kafka1991 wants to merge 301 commits into
Open
feat: columnar DataFrame ingest (Arrow / Polars / Pandas) + Arrow egress#153kafka1991 wants to merge 301 commits into
kafka1991 wants to merge 301 commits into
Conversation
|
Important Review skippedToo many files! This PR contains 181 files, which is 31 over the limit of 150. To get a review, narrow the scope: Upgrade to a paid plan to raise the limit. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (2)
📒 Files selected for processing (181)
You can disable this status message by setting the Use the checkbox below for a quick retry:
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
kafka1991
changed the title
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 9
🧹 Nitpick comments (8)
CMakeLists.txt (1)
396-398: 💤 Low valueMisleading comment: no
fatal_errorgate for Arrow.The comment references a "fatal_error gate" that forces
QUESTDB_ENABLE_ARROW=ON, but no such gate exists. Lines 51-55 have aFATAL_ERRORforQUESTDB_ENABLE_READER, not Arrow. Arrow is silently auto-enabled at lines 89-91 viamessage(STATUS)+set().Consider updating the comment to accurately describe the auto-enable mechanism.
📝 Suggested comment fix
- # Apache Arrow C Data Interface tests. The fatal_error gate above - # forces QUESTDB_ENABLE_ARROW=ON when tests are enabled, so these - # always build alongside the rest of the suite. + # Apache Arrow C Data Interface tests. QUESTDB_ENABLE_ARROW is + # auto-enabled (lines 89-91) when tests are enabled, so these + # always build alongside the rest of the suite.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@CMakeLists.txt` around lines 396 - 398, The comment incorrectly claims a "fatal_error gate" forces QUESTDB_ENABLE_ARROW=ON; instead update the comment near the Apache Arrow C Data Interface tests to state that QUESTDB_ENABLE_ARROW is auto-enabled via the CMake logic that emits message(STATUS) and calls set(QUESTDB_ENABLE_ARROW ON) (the auto-enable mechanism), not via any FATAL_ERROR gate like the one used for QUESTDB_ENABLE_READER; reference QUESTDB_ENABLE_ARROW and the message(STATUS)/set() auto-enable behavior when rewriting the comment.questdb-rs-ffi/src/column_sender.rs (1)
1-1700: Remember to run cargo fmt and clippy before commit.As per coding guidelines, before every commit run:
cargo fmt --manifest-path questdb-rs-ffi/Cargo.tomlcargo clippy --manifest-path questdb-rs-ffi/Cargo.toml --tests(without-D warnings)The implementation looks solid: input validation is comprehensive, Arrow ownership handling is correct, and the FFI surface properly handles all error cases.
As per coding guidelines for
**/*.rsand**/Cargo.tomlfiles.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@questdb-rs-ffi/src/column_sender.rs` around lines 1 - 1700, You need to format and lint the Rust crate before committing: run `cargo fmt --manifest-path questdb-rs-ffi/Cargo.toml` and then `cargo clippy --manifest-path questdb-rs-ffi/Cargo.toml --tests` and fix any clippy warnings (do not deny warnings). Focus fixes around the FFI surface in column_sender.rs (e.g. functions like column_sender_chunk_new, column_sender_chunk_append_numpy_column, resolve_numpy_dtype and column_sender_flush) so formatting and lints are clean before pushing.cpp_test/test_arrow_c.c (1)
52-70: ⚡ Quick winConsider surfacing initialization failures in test helpers.
Both
make_tableandmake_colswallowline_sender_table_name_init/line_sender_column_name_initerrors and return the structure regardless. If initialization fails, the returned struct may be in an undefined state. While this is test code and may be intentional for negative test cases, consider one of:
- Add a comment documenting this is intentional for tests
- Add a boolean out-param to signal failure
- Return a sentinel value or use a different pattern for error cases
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@cpp_test/test_arrow_c.c` around lines 52 - 70, The helpers make_table and make_col currently swallow initialization errors from line_sender_table_name_init and line_sender_column_name_init and return potentially-uninitialized structs; change both functions to detect if err is non-NULL, log or print the error, free the error, and fail-fast (e.g., fprintf to stderr and exit or assert) so tests do not proceed with invalid values — update make_table and make_col to free err and abort on error instead of silently returning; include a brief comment above each helper explaining the fail-fast behavior.questdb-rs/src/egress/arrow/convert.rs (1)
289-326: 🏗️ Heavy lift
Large*mismatch isn’t produced by the egress schema, but the array builders still ignore width/container type
questdb-rs/src/egress/arrow/schema.rs(andpolars.rs) constructsDataType::Utf8/DataType::BinaryandDataType::List(notLargeUtf8/LargeBinary/LargeList), and the Arrow tests assertList(...)—so the specificRecordBatch::try_newrejection described forLarge*types shouldn’t occur in the normal egress path.However,
varlen_string_array()/varlen_binary_array()andnest_lists()ignore the provided field/container width and always buildUtf8/BinaryandListArraywith i32 offsets (e.g.,DataType::Utf8/DataType::Binary,offsets_i32,DataType::Listinnest_lists). If any alternate caller ever passesLarge*types into these helpers, the produced array types would not match the schema. Consider threading/handling thefield/requestedDataType(chooseLarge*+ i64 offsets) or constraining/guarding these helpers accordingly.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@questdb-rs/src/egress/arrow/convert.rs` around lines 289 - 326, The helpers varlen_string_array, varlen_binary_array (and nest_lists) ignore the Field's DataType and always build Utf8/Binary and i32-offset List arrays, which will mismatch if a LargeUtf8/LargeBinary/LargeList is passed; update these helpers to inspect the provided field.type() and branch: for DataType::Utf8/DataType::Binary/DataType::List use i32 offsets and current builders, and for DataType::LargeUtf8/DataType::LargeBinary/DataType::LargeList use the Large variants with i64 offsets (use offsets_i64 or equivalent and construct LargeUtf8/LargeBinary/LargeList ArrayData), or alternatively add an explicit guard that returns an error if a Large* type is supplied. Ensure you reference and switch on the Field's DataType in varlen_string_array, varlen_binary_array and nest_lists so produced ArrayData matches the schema.questdb-rs/src/egress/decoder.rs (1)
798-812: ⚡ Quick winAdd boundary tests for the new per-width decimal scale guard.
This change introduces a new protocol constraint, but there isn’t a targeted test pinning the DECIMAL64 boundary (e.g.
scale=18accepted,scale=19rejected). A focused test here would protect this behavior from silent regressions.Suggested test additions
+ #[test] + fn decode_decimal64_scale_boundary_enforced() { + // scale=18 should pass, scale=19 should fail for DECIMAL64. + let ok = vec![0x00u8, 18u8]; + let bad = vec![0x00u8, 19u8]; + // ...build 1-row DECIMAL64 payloads and assert: + // ok => decode_result_batch(...).is_ok() + // bad => ProtocolError containing "DECIMAL64 scale" + }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@questdb-rs/src/egress/decoder.rs` around lines 798 - 812, Add unit tests that explicitly exercise the new per-width decimal scale guard by constructing DECIMAL values with widths that map to the per_width_max logic (e.g., width 8 => per_width_max 18 for DECIMAL64) and asserting acceptance at the boundary (scale = 18) and rejection just above it (scale = 19). Target the code paths that compute per_width_max and return the ProtocolError with the message "DECIMAL{} scale {} exceeds per-width maximum {}", using the same input construction used by the decoder (invoke the decoder entry that handles DECIMAL widths or send a serialized DECIMAL packet into the decoding routine) so tests fail if per_width_max enforcement changes. Ensure tests cover at least width=8 (DECIMAL64) and one other width branch (e.g., width=16 or 32) to lock in behavior across branches.doc/COLUMN_SENDER_PLAN.md (1)
172-175: ⚡ Quick winKeep the “new module” list aligned with the implemented tree.
This section lists
sender.rs,validity.rs, anderror.rs, but this PR’s actual module set underquestdb-rs/src/ingress/column_sender/differs. Updating this list will prevent implementation drift in follow-up tasks.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@doc/COLUMN_SENDER_PLAN.md` around lines 172 - 175, Update the "New module" list in doc/COLUMN_SENDER_PLAN.md so it exactly matches the implemented module tree under questdb-rs/src/ingress/column_sender/ (use the actual filenames present, e.g., include or remove sender.rs, validity.rs, error.rs as appropriate) and also update the re-export line (questdb::ingress::column_sender::{QuestDb, ColumnSender, Chunk, Validity}) to reflect the real public items implemented in db.rs, sender.rs, chunk.rs, validity.rs, encoder.rs, error.rs; ensure the doc’s module list and re-exports are synchronized with the codebase to prevent future drift.questdb-rs/src/egress/config.rs (1)
589-595: ⚡ Quick winAdd explicit tests for
qwpws/qwpwssaliases.This parser branch is new behavior and should be locked with direct tests asserting
tlsand URL scheme mapping for both aliases.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@questdb-rs/src/egress/config.rs` around lines 589 - 595, Add explicit unit tests that lock the new branch handling the "qwpws" and "qwpwss" aliases: create two tests (e.g., test_qwpws_alias and test_qwpwss_alias) that call the same parser/constructor used by the egress config (the function that interprets the scheme and returns the tls flag and normalized URL scheme) and assert that "qwpws" yields tls = false and normalizes to "ws", and that "qwpwss" yields tls = true and normalizes to "wss"; place them in the existing config/egress tests module so future changes to the branch will break the tests if behavior changes.questdb-rs/src/ingress/column_sender/conn.rs (1)
422-433: 💤 Low value
set_timeoutsis a no-op and may leave stale socket timeouts.The method accepts timeout parameters but discards them. The comment mentions exposing a setter on
WsStreamfor long flushes, but currently any timeout refresh is silently skipped. If the socket's initial timeouts become inappropriate for long frame writes, this could lead to unexpectedWouldBlockor timeout errors.Consider either implementing the timeout refresh or removing the dead parameters to avoid confusion.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@questdb-rs/src/ingress/column_sender/conn.rs` around lines 422 - 433, The set_timeouts function currently ignores its read/write parameters, which can leave stale socket timeouts; update set_timeouts (in conn.rs) to apply the provided read and write Durations to the underlying TCP socket obtained from the WsStream accessor (use the tcp_stream accessor used elsewhere) by calling set_read_timeout and set_write_timeout (converting None to Ok(None) semantics) and return an error if either syscall fails, or alternatively remove the unused parameters and document that timeouts are fixed — pick one: implement timeout refresh by setting TcpStream::set_read_timeout/set_write_timeout and propagate io::Error as Err, or remove the read: Option<Duration>, write: Option<Duration> params from set_timeouts and callers, and update comments referencing WsStream/establish_connection in qwp_ws.rs accordingly.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@cpp_test/smoke_column_sender.c`:
- Around line 76-80: The test uses the wrong type and API names from the header:
replace all occurrences of column_sender and
questdb_db_borrow_sender/questdb_db_return_sender with the actual names defined
in column_sender.h (use qwpws_conn* as the variable type and call
questdb_db_borrow_conn()/questdb_db_return_conn() instead); update the variable
name (e.g., sender -> conn) and all corresponding borrow/return call sites
mentioned (around the blocks at 76, 86, 98-99, 110, 133, 147, 155, 163, 169) so
the types and function names match the header signatures (qwpws_conn*,
questdb_db_borrow_conn, questdb_db_return_conn) and ensure all cleanup paths
call questdb_db_return_conn where sender cleanup was previously used.
In `@cpp_test/test_arrow_ingress.cpp`:
- Around line 35-38: The test uses Arrow-only symbols (qdb::column_sender_conn,
column_sender_flush_arrow_batch / column_sender_flush_arrow_batch_at_column) but
the TU is compiled without QUESTDB_CLIENT_ENABLE_ARROW; guard the Arrow-specific
test sections (the blocks using qdb::column_sender_conn and calls to
flush_arrow_batch/flush_arrow_batch_at_column) with `#ifdef`
QUESTDB_CLIENT_ENABLE_ARROW ... `#endif` (or define QUESTDB_CLIENT_ENABLE_ARROW
for this TU), and apply the same guards to the other indicated ranges (lines
around 48-51, 61-64, 213-236, 266-282, 639-642) so compilation only includes
these symbols when Arrow support is enabled.
In `@doc/COLUMN_SENDER_PLAN.md`:
- Line 131: The markdown fenced code block in COLUMN_SENDER_PLAN.md is untyped
and triggers MD040; update its opening fence from ``` to include a language tag
(e.g., ```text) so the block is typed. Locate the unnamed fenced block and
change the opening fence to include the language specifier (suggested: "text")
while leaving the closing fence as-is.
In `@examples/line_sender_cpp_example_arrow.cpp`:
- Around line 29-34: The AppendValues and Finish calls on the Arrow builders
(ts_b.AppendValues, price_b.AppendValues, ts_b.Finish, price_b.Finish) currently
call .ok() and ignore the returned Status; modify the code to check the returned
Status for each AppendValues and Finish call, handle errors (log and
return/exit) when Status.IsOk() is false, and avoid using ts_arr or price_arr if
Finish failed; ensure each failure path cleans up or aborts before proceeding to
use the arrays.
In `@include/questdb/ingress/line_sender.hpp`:
- Around line 1811-1821: The code currently throws for sender_protocol ==
protocol::qwpws/qwpwss in new_buffer(), breaking existing C++ callers and the
empty-buffer fallback used by flush_and_get_fsn / flush_and_keep_and_get_fsn /
flush_and_keep[_with_flags]; instead of throwing, restore a row-buffer path for
WebSocket senders: remove the throw and map qwpws/qwpwss to a compatible backend
(e.g., reuse line_sender_buffer::_backend_kind::ilp or add a new backend like
qwp_ws_row) so line_sender_buffer construction still succeeds, and ensure the
existing empty-buffer fallback logic in flush_and_get_fsn and
flush_and_keep_and_get_fsn continues to work for WebSocket protocols.
In `@questdb-rs/src/egress/arrow/schema.rs`:
- Around line 228-246: The loop over shape_offsets.windows(2) currently only
checks dims > shapes.len(), which misses cases where w[1] (end) is out of bounds
but end-start is small; update the check in the block handling shape_offsets to
validate absolute bounds first: verify both w[0] and w[1] are <= shapes.len()
(and keep the existing monotonicity check), and if either index is out of range
return the ProtocolError used elsewhere (same fmt! message but referencing the
absolute bound failure); then compute dims = (w[1] - w[0]) as usize and proceed
as before, removing the dims > shapes.len() condition.
In `@questdb-rs/src/ingress.rs`:
- Around line 2474-2487: build_qwp_ws_raw_stream currently omits the same config
validation that build() performs, allowing an incompatible combination (manual
progress + initial_connect_retry=async); to fix, invoke the same validation
sequence used in build(): call qwp_ws.apply_reconnect_implies_initial_retry()
and then reject_unsupported_qwp_ws_sf_config(&qwp_ws)? at the start of
build_qwp_ws_raw_stream (before establishing the connection) so the function
rejects the unsupported config the same way as build().
In `@questdb-rs/src/ingress/column_sender/conf.rs`:
- Around line 159-161: The wildcard match arm currently lets unknown keys pass
through silently; change it so keys starting with "pool_" are rejected instead
of passed to SenderBuilder. In the match fallback (the `_ => { ... }` arm)
detect if the key starts_with("pool_") and return an Err (e.g., a
ConfigError::UnknownKey or UnknownPoolKey with the offending key) so typoed
pool_* settings surface as errors; leave non-pool passthrough behavior for other
keys to continue being handled by SenderBuilder.
In `@questdb-rs/src/ingress/column_sender/conn.rs`:
- Around line 268-281: The function try_drain_acks currently increments drained
for every response from try_recv_qwp_response, but the docstring promises
"number of OK acks consumed"; update the counting logic so you only increment
drained for OK ack responses: when you get Some(response) from
try_recv_qwp_response, inspect the response variant (e.g., differentiate OK vs
DurableAck) and call process_response(response) as before but only increment
drained for the OK variant; if inspecting the variant requires changing
ownership, adjust to match by reference or change process_response to
accept/by-reference or return the response type to avoid double-consuming.
Alternatively, if you prefer minimal change, update the docstring of
try_drain_acks to state it returns the number of responses consumed (not just OK
acks).
---
Nitpick comments:
In `@CMakeLists.txt`:
- Around line 396-398: The comment incorrectly claims a "fatal_error gate"
forces QUESTDB_ENABLE_ARROW=ON; instead update the comment near the Apache Arrow
C Data Interface tests to state that QUESTDB_ENABLE_ARROW is auto-enabled via
the CMake logic that emits message(STATUS) and calls set(QUESTDB_ENABLE_ARROW
ON) (the auto-enable mechanism), not via any FATAL_ERROR gate like the one used
for QUESTDB_ENABLE_READER; reference QUESTDB_ENABLE_ARROW and the
message(STATUS)/set() auto-enable behavior when rewriting the comment.
In `@cpp_test/test_arrow_c.c`:
- Around line 52-70: The helpers make_table and make_col currently swallow
initialization errors from line_sender_table_name_init and
line_sender_column_name_init and return potentially-uninitialized structs;
change both functions to detect if err is non-NULL, log or print the error, free
the error, and fail-fast (e.g., fprintf to stderr and exit or assert) so tests
do not proceed with invalid values — update make_table and make_col to free err
and abort on error instead of silently returning; include a brief comment above
each helper explaining the fail-fast behavior.
In `@doc/COLUMN_SENDER_PLAN.md`:
- Around line 172-175: Update the "New module" list in doc/COLUMN_SENDER_PLAN.md
so it exactly matches the implemented module tree under
questdb-rs/src/ingress/column_sender/ (use the actual filenames present, e.g.,
include or remove sender.rs, validity.rs, error.rs as appropriate) and also
update the re-export line (questdb::ingress::column_sender::{QuestDb,
ColumnSender, Chunk, Validity}) to reflect the real public items implemented in
db.rs, sender.rs, chunk.rs, validity.rs, encoder.rs, error.rs; ensure the doc’s
module list and re-exports are synchronized with the codebase to prevent future
drift.
In `@questdb-rs-ffi/src/column_sender.rs`:
- Around line 1-1700: You need to format and lint the Rust crate before
committing: run `cargo fmt --manifest-path questdb-rs-ffi/Cargo.toml` and then
`cargo clippy --manifest-path questdb-rs-ffi/Cargo.toml --tests` and fix any
clippy warnings (do not deny warnings). Focus fixes around the FFI surface in
column_sender.rs (e.g. functions like column_sender_chunk_new,
column_sender_chunk_append_numpy_column, resolve_numpy_dtype and
column_sender_flush) so formatting and lints are clean before pushing.
In `@questdb-rs/src/egress/arrow/convert.rs`:
- Around line 289-326: The helpers varlen_string_array, varlen_binary_array (and
nest_lists) ignore the Field's DataType and always build Utf8/Binary and
i32-offset List arrays, which will mismatch if a LargeUtf8/LargeBinary/LargeList
is passed; update these helpers to inspect the provided field.type() and branch:
for DataType::Utf8/DataType::Binary/DataType::List use i32 offsets and current
builders, and for DataType::LargeUtf8/DataType::LargeBinary/DataType::LargeList
use the Large variants with i64 offsets (use offsets_i64 or equivalent and
construct LargeUtf8/LargeBinary/LargeList ArrayData), or alternatively add an
explicit guard that returns an error if a Large* type is supplied. Ensure you
reference and switch on the Field's DataType in varlen_string_array,
varlen_binary_array and nest_lists so produced ArrayData matches the schema.
In `@questdb-rs/src/egress/config.rs`:
- Around line 589-595: Add explicit unit tests that lock the new branch handling
the "qwpws" and "qwpwss" aliases: create two tests (e.g., test_qwpws_alias and
test_qwpwss_alias) that call the same parser/constructor used by the egress
config (the function that interprets the scheme and returns the tls flag and
normalized URL scheme) and assert that "qwpws" yields tls = false and normalizes
to "ws", and that "qwpwss" yields tls = true and normalizes to "wss"; place them
in the existing config/egress tests module so future changes to the branch will
break the tests if behavior changes.
In `@questdb-rs/src/egress/decoder.rs`:
- Around line 798-812: Add unit tests that explicitly exercise the new per-width
decimal scale guard by constructing DECIMAL values with widths that map to the
per_width_max logic (e.g., width 8 => per_width_max 18 for DECIMAL64) and
asserting acceptance at the boundary (scale = 18) and rejection just above it
(scale = 19). Target the code paths that compute per_width_max and return the
ProtocolError with the message "DECIMAL{} scale {} exceeds per-width maximum
{}", using the same input construction used by the decoder (invoke the decoder
entry that handles DECIMAL widths or send a serialized DECIMAL packet into the
decoding routine) so tests fail if per_width_max enforcement changes. Ensure
tests cover at least width=8 (DECIMAL64) and one other width branch (e.g.,
width=16 or 32) to lock in behavior across branches.
In `@questdb-rs/src/ingress/column_sender/conn.rs`:
- Around line 422-433: The set_timeouts function currently ignores its
read/write parameters, which can leave stale socket timeouts; update
set_timeouts (in conn.rs) to apply the provided read and write Durations to the
underlying TCP socket obtained from the WsStream accessor (use the tcp_stream
accessor used elsewhere) by calling set_read_timeout and set_write_timeout
(converting None to Ok(None) semantics) and return an error if either syscall
fails, or alternatively remove the unused parameters and document that timeouts
are fixed — pick one: implement timeout refresh by setting
TcpStream::set_read_timeout/set_write_timeout and propagate io::Error as Err, or
remove the read: Option<Duration>, write: Option<Duration> params from
set_timeouts and callers, and update comments referencing
WsStream/establish_connection in qwp_ws.rs accordingly.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 27742921-fcfb-4275-92b7-e403b14badb9
⛔ Files ignored due to path filters (1)
questdb-rs-ffi/Cargo.lockis excluded by!**/*.lock
📒 Files selected for processing (77)
CMakeLists.txtci/compile.yamlci/run_all_tests.pyci/run_fuzz_pipeline.yamlci/run_tests_pipeline.yamlcpp_test/qwp_mock_c.cppcpp_test/qwp_mock_c.hcpp_test/qwp_mock_server.cppcpp_test/smoke_column_sender.ccpp_test/test_arrow_c.ccpp_test/test_arrow_egress.cppcpp_test/test_arrow_ingress.cppdoc/COLUMN_SENDER_FFI_ABI.mddoc/COLUMN_SENDER_PERF.mddoc/COLUMN_SENDER_PLAN.mdexamples/line_reader_c_example_arrow.cexamples/line_reader_cpp_example_arrow.cppexamples/line_sender_cpp_example_arrow.cppinclude/questdb/egress/line_reader.hinclude/questdb/egress/line_reader.hppinclude/questdb/ingress/column_sender.hinclude/questdb/ingress/column_sender.hppinclude/questdb/ingress/line_sender.hinclude/questdb/ingress/line_sender.hppinclude/questdb/ingress/line_sender_core.hppquestdb-rs-ffi/Cargo.tomlquestdb-rs-ffi/src/column_sender.rsquestdb-rs-ffi/src/egress.rsquestdb-rs-ffi/src/lib.rsquestdb-rs/Cargo.tomlquestdb-rs/benches/column_sender.rsquestdb-rs/examples/polars.rsquestdb-rs/examples/qwp_ws_l1_quotes.rsquestdb-rs/src/egress/arrow/convert.rsquestdb-rs/src/egress/arrow/mod.rsquestdb-rs/src/egress/arrow/polars.rsquestdb-rs/src/egress/arrow/reader.rsquestdb-rs/src/egress/arrow/schema.rsquestdb-rs/src/egress/arrow/tests.rsquestdb-rs/src/egress/config.rsquestdb-rs/src/egress/decoder.rsquestdb-rs/src/egress/error.rsquestdb-rs/src/egress/mod.rsquestdb-rs/src/egress/reader.rsquestdb-rs/src/error.rsquestdb-rs/src/ingress.rsquestdb-rs/src/ingress/buffer.rsquestdb-rs/src/ingress/buffer/qwp.rsquestdb-rs/src/ingress/column_sender/arrow_batch.rsquestdb-rs/src/ingress/column_sender/chunk.rsquestdb-rs/src/ingress/column_sender/conf.rsquestdb-rs/src/ingress/column_sender/conn.rsquestdb-rs/src/ingress/column_sender/db.rsquestdb-rs/src/ingress/column_sender/encoder.rsquestdb-rs/src/ingress/column_sender/mod.rsquestdb-rs/src/ingress/column_sender/numpy_wire.rsquestdb-rs/src/ingress/column_sender/sender.rsquestdb-rs/src/ingress/column_sender/validity.rsquestdb-rs/src/ingress/column_sender/wire.rsquestdb-rs/src/ingress/polars.rsquestdb-rs/src/ingress/sender.rsquestdb-rs/src/ingress/sender/qwp_ws.rsquestdb-rs/src/tests.rsquestdb-rs/src/tests/column_sender_pool.rsquestdb-rs/src/tests/qwp_ws.rsquestdb-rs/tests/qwp_egress_bounds_fuzz.rssystem_test/arrow_alignment_fuzz.pysystem_test/arrow_egress_fuzz.pysystem_test/arrow_ffi.pysystem_test/arrow_fuzz_common.pysystem_test/arrow_ingress_fuzz.pysystem_test/arrow_polars_fuzz.pysystem_test/arrow_polars_per_dtype.pysystem_test/arrow_round_trip_fuzz.pysystem_test/questdb_line_sender.pysystem_test/test.pysystem_test/test_arrow_fuzz_common_unit.py
👮 Files not reviewed due to content moderation or server errors (6)
- cpp_test/test_arrow_egress.cpp
- doc/COLUMN_SENDER_FFI_ABI.md
- questdb-rs/src/egress/error.rs
- questdb-rs/src/egress/mod.rs
- questdb-rs/src/egress/reader.rs
- questdb-rs/src/error.rs
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
doc/COLUMN_SENDER_FFI_ABI.md (1)
92-100:⚠️ Potential issue | 🔴 Critical | ⚡ Quick winCritical: Buffer lifetime contract is contradictory.
Section §2.3 (lines 92-100) explicitly states that buffers "must remain alive and unchanged until the next
column_sender_flushcall on the chunk returns" and that "The FFI stores raw pointers into the caller's buffers; it does not copy at append time."However, Section §16 (lines 1206-1209) states: "Buffers are copied into the chunk during the call, so they can be dropped after the call returns."
These are directly contradictory. If Python wrapper developers follow §16 and drop buffers immediately after the append call, but the Rust FFI implementation follows §2.3 and stores raw pointers without copying, this will cause use-after-free bugs and memory corruption.
One of these statements must be wrong. Based on the PR objectives mentioning "zero-overhead-where-possible" and §2.3's explicit rationale ("required to hit memcpy-bandwidth throughput"), the zero-copy design (§2.3) appears intentional. If so, §16 lines 1207-1209 must be corrected to state that buffers must remain alive until flush returns, matching §2.3.
📝 Proposed fix for §16
- **Object lifetimes** — keep the source `np.ndarray` / `pa.Array` - alive for the duration of the FFI call. Buffers are copied into the - chunk during the call, so they can be dropped after the call - returns. + alive until the next `flush()` call returns (or until the chunk is + freed/cleared). The FFI stores raw pointers without copying at append + time for zero-copy throughput. Do not drop or mutate the source arrays + between `append` and `flush`.Also applies to: 1206-1209
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@doc/COLUMN_SENDER_FFI_ABI.md` around lines 92 - 100, The doc contains a contradictory buffer-lifetime contract: §2.3 says buffers passed to any column_sender_chunk_* function are not copied and "must remain alive and unchanged until the next column_sender_flush call on the chunk returns" (or until column_sender_chunk_free / column_sender_chunk_clear is called), but §16 claims buffers are copied during the call and can be dropped immediately; reconcile by updating §16 to match the intended zero-copy design: change the wording to explicitly state that buffers passed to column_sender_chunk_* are not copied and must remain valid and unchanged until column_sender_flush returns (or until column_sender_chunk_free / column_sender_chunk_clear is called), and add a short note pointing to the performance rationale and to §2.3 for details.
🧹 Nitpick comments (2)
doc/COLUMN_SENDER_FFI_ABI.md (2)
537-538: 💤 Low valueClarify VARCHAR as the replacement for removed STRING type.
The note mentions that "The older STRING wire type (
0x08) has been removed from the spec," but doesn't explicitly state that VARCHAR is the replacement. Users migrating from older code might benefit from a brief clarification.Suggested clarification
-QWP has exactly one variable-width text type: VARCHAR (wire code -`0x0F`). The wire format is `uint32` offsets + concatenated bytes. The -older STRING wire type (`0x08`) has been removed from the spec and is -not exposed here. +QWP has exactly one variable-width text type: VARCHAR (wire code +`0x0F`), which replaces the older STRING wire type (`0x08`). +The wire format is `uint32` offsets + concatenated bytes.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@doc/COLUMN_SENDER_FFI_ABI.md` around lines 537 - 538, Update the note that currently states "The older STRING wire type (`0x08`) has been removed from the spec" to explicitly state that VARCHAR is the intended replacement for the removed STRING type; mention the removed symbol "STRING (`0x08`)" and the replacement symbol "VARCHAR" so readers migrating older code know to map STRING uses to VARCHAR and any relevant wire-type differences.
810-811: 💤 Low valueConsider emphasizing the zero-value data-loss trap for BYTE and SHORT.
Lines 810-811 document that source values of 0 in
i8andi16columns are automatically converted to NULL, regardless of the validity bitmap. This means users cannot store the literal value 0 in BYTE or SHORT columns.While this is a wire-protocol constraint (sentinel values) rather than an API design choice, it's a significant footgun that could surprise users. Consider adding a callout box or warning earlier in the spec (perhaps in §6 or §11) to make this limitation more prominent before users encounter it in the coverage matrix.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@doc/COLUMN_SENDER_FFI_ABI.md` around lines 810 - 811, Add a prominent warning about the zero-value sentinel behavior for i8/BYTE and i16/SHORT: state that source values of 0 are treated as NULL regardless of the validity bitmap so literal 0 cannot be stored, and surface this as a callout in an earlier high-visibility section (suggest §6 or §11) and reference the coverage matrix entries `i8`/`BYTE` and `i16`/`SHORT` (sentinel = 0) so readers see the trap before reaching the column type table; keep the callout short, explicit about data loss, and link back to the table rows that document the sentinel behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@doc/COLUMN_SENDER_FFI_ABI.md`:
- Around line 92-100: The doc contains a contradictory buffer-lifetime contract:
§2.3 says buffers passed to any column_sender_chunk_* function are not copied
and "must remain alive and unchanged until the next column_sender_flush call on
the chunk returns" (or until column_sender_chunk_free /
column_sender_chunk_clear is called), but §16 claims buffers are copied during
the call and can be dropped immediately; reconcile by updating §16 to match the
intended zero-copy design: change the wording to explicitly state that buffers
passed to column_sender_chunk_* are not copied and must remain valid and
unchanged until column_sender_flush returns (or until column_sender_chunk_free /
column_sender_chunk_clear is called), and add a short note pointing to the
performance rationale and to §2.3 for details.
---
Nitpick comments:
In `@doc/COLUMN_SENDER_FFI_ABI.md`:
- Around line 537-538: Update the note that currently states "The older STRING
wire type (`0x08`) has been removed from the spec" to explicitly state that
VARCHAR is the intended replacement for the removed STRING type; mention the
removed symbol "STRING (`0x08`)" and the replacement symbol "VARCHAR" so readers
migrating older code know to map STRING uses to VARCHAR and any relevant
wire-type differences.
- Around line 810-811: Add a prominent warning about the zero-value sentinel
behavior for i8/BYTE and i16/SHORT: state that source values of 0 are treated as
NULL regardless of the validity bitmap so literal 0 cannot be stored, and
surface this as a callout in an earlier high-visibility section (suggest §6 or
§11) and reference the coverage matrix entries `i8`/`BYTE` and `i16`/`SHORT`
(sentinel = 0) so readers see the trap before reaching the column type table;
keep the callout short, explicit about data loss, and link back to the table
rows that document the sentinel behavior.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: e3d24b42-a01b-473b-aa99-0866e399212b
📒 Files selected for processing (4)
doc/COLUMN_SENDER_FFI_ABI.mdinclude/questdb/ingress/column_sender.hquestdb-rs-ffi/src/column_sender.rsquestdb-rs/src/ingress/column_sender/numpy_wire.rs
🚧 Files skipped from review as they are similar to previous changes (2)
- include/questdb/ingress/column_sender.h
- questdb-rs-ffi/src/column_sender.rs
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
questdb-rs-ffi/src/column_sender.rs (2)
393-404:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winThe constructor contract and the tested behavior disagree.
Line 393 says
column_sender_chunk_newvalidates the table name to<= 127 bytes, but the test below explicitly asserts that a 128-byte name succeeds and is only rejected later. Either enforce the limit here or relax the documented contract; otherwise FFI callers get a successful handle for an input the API says should fail.Also applies to: 1471-1482
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@questdb-rs-ffi/src/column_sender.rs` around lines 393 - 404, The comment/documentation and runtime validation disagree: update the validation to match the test expectations by allowing table names up to 128 bytes (instead of enforcing <=127) so column_sender_chunk_new returns a valid handle for a 128-byte name; modify the underlying name_str length check (or the constant it uses) to permit 128 bytes and update the doc comment on column_sender_chunk_new accordingly, and apply the same consistent change to the other constructor/validation at the 1471-1482 region so both places enforce the same max-length policy.
122-136:⚠️ Potential issue | 🔴 Critical | 🏗️ Heavy liftFix FFI boundary: stop taking
#[repr(C)]enums by value inextern "C"APIs
column_sender_synctakesack_level: column_sender_ack_levelby value, andcolumn_sender_chunk_append_numpy_columntakesdtype: column_sender_numpy_dtypeby value. If a C caller supplies an out-of-range discriminant, Rust has an invalid enum value at the call boundary (undefined behavior) before yourinto()/conversion logic can reject it.Change both parameters to a fixed-width integer type in the C/Rust boundary (e.g.,
u32), thenTryFrom/validate and returnInvalidApiCallfor unknown values.Also,
column_sender_numpy_dtypedocuments it as “mirrored to the C ABI as a 32-bit enum”, but#[repr(C)]does not guarantee a fixed 32-bit width across C ABIs—use an explicit#[repr(u32)](or, better, acceptu32at the boundary) so the ABI is actually 32-bit.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@questdb-rs-ffi/src/column_sender.rs` around lines 122 - 136, The extern-C boundary must not accept #[repr(C)] enums by value; change the parameters ack_level in column_sender_sync and dtype in column_sender_chunk_append_numpy_column from their enum types (column_sender_ack_level, column_sender_numpy_dtype) to a fixed-width integer (u32) at the FFI boundary, then validate by TryFrom/try_into into the Rust enum (or convert via a match) and return InvalidApiCall for unknown discriminants; if you keep enum definitions for internal use, give them an explicit #[repr(u32)] to document size, but do validation in the functions named column_sender_sync and column_sender_chunk_append_numpy_column before any enum conversion.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@questdb-rs-ffi/src/column_sender.rs`:
- Around line 393-404: The comment/documentation and runtime validation
disagree: update the validation to match the test expectations by allowing table
names up to 128 bytes (instead of enforcing <=127) so column_sender_chunk_new
returns a valid handle for a 128-byte name; modify the underlying name_str
length check (or the constant it uses) to permit 128 bytes and update the doc
comment on column_sender_chunk_new accordingly, and apply the same consistent
change to the other constructor/validation at the 1471-1482 region so both
places enforce the same max-length policy.
- Around line 122-136: The extern-C boundary must not accept #[repr(C)] enums by
value; change the parameters ack_level in column_sender_sync and dtype in
column_sender_chunk_append_numpy_column from their enum types
(column_sender_ack_level, column_sender_numpy_dtype) to a fixed-width integer
(u32) at the FFI boundary, then validate by TryFrom/try_into into the Rust enum
(or convert via a match) and return InvalidApiCall for unknown discriminants; if
you keep enum definitions for internal use, give them an explicit #[repr(u32)]
to document size, but do validation in the functions named column_sender_sync
and column_sender_chunk_append_numpy_column before any enum conversion.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 16ca2d8e-8939-4f3e-bbe2-67f272211260
📒 Files selected for processing (1)
questdb-rs-ffi/src/column_sender.rs
Accept plain FixedSizeBinary(16) as UUID for Arrow ingestion instead of requiring extension metadata. This matches the Python Client.dataframe contract and the server e2e UUID round-trip tests, at the cost of no longer treating FSB16 as a generic opaque fixed-size binary shape in this path. Reject null timestamp field columns before publishing, matching the existing designated-timestamp and Python planner validation policy. Nullable timestamp fields can be revisited later only with an explicit server/protocol contract.
The 'Vs QuestDB master' integration job runs the full system_test/test.py suite on an agent that only installs numpy (not pyarrow/polars). PR #166 added unconditional top-level imports of the Arrow fuzz modules, each of which imports pyarrow at load time, so test.py crashed on import with 'ModuleNotFoundError: No module named pyarrow' before any test ran and before QuestDB started (which also produced the downstream data/log not-found and artifact-publish errors). Wrap the Arrow imports in try/except ImportError: when pyarrow/polars are absent the Arrow TestCases simply aren't registered, so a full-suite run skips them instead of failing at import. Jobs that actually run the TestArrow* cases install pyarrow/polars, so nothing is lost there.
# Conflicts: # ci/run_tests_pipeline.yaml # system_test/test.py
kafka1991
force-pushed
the
jh_conn_pool_refactor
branch
from
5bbe9a0 to
b791145
Compare
main reverted PR #160 ("run Vs QuestDB master against the nightly docker image") in #161, removing QuestDbDockerFixture, the fixture.py `_docker` helpers, the test.py `--docker`/`_new_fixture` plumbing and the docker failover server. This branch was based on #160 and still carried all of it, so the PR's refs/pull/166/merge silently dropped the class + import (deleted on main, unchanged here) while keeping every usage, breaking: - Vs QuestDB master / integration test: NameError: name 'QuestDbDockerFixture' is not defined - QWP/WS fuzz suite / TestArrowWsFuzz: ImportError: cannot import name 'QuestDbDockerFixture' from '__main__' Bring the revert into the branch and reconcile the new suites to the from-source fixture, without losing any tests: - fixture.py / test_egress_failover.py / exhaustion_client.rs: take the revert (auto-merged) — no more docker fixture or `_docker` helper. - test.py: keep the branch's BUILD_MODE_SEED work; drop the `is_docker` plumbing; the QWP/WS smoke + protocol suites now require a QuestDbFixture (the managed, from-source server). - arrow_fuzz_common.py / arrow_ingress_fuzz.py: drop QuestDbDockerFixture from the lazy `from test import ...` and the isinstance() guards; the live/managed checks use QuestDbFixture (+ QuestDbExternalFixture). - ci/run_tests_pipeline.yaml: keep the from-source "Vs QuestDB master" job (Maven compile + `test.py run --repo ./questdb`), drop the docker-nightly variant. All Arrow/e2e/failover tests are preserved; they run against the source-built QuestDbFixture.
The QWP/WS store-and-forward reconnect policy (NACK v2) treats transport outages as retry-forever: `reconnect_max_duration_millis` bounds a single reconnect round, not the sender's lifetime, and only auth/protocol errors are terminal. The old `test_reconnect_gives_up_after_cap` still asserted the pre-refactor give-up-after-cap behavior and failed because a stopped server never surfaces a terminal error. Rename to `test_reconnect_retries_forever_past_cap` and assert the new contract: flushes keep succeeding past several reconnect budgets while the server is down, and every accepted row drains once the endpoint reappears. Update the SFA row-test inventory reference accordingly.
…_java_e2e_scenarios # Conflicts: # system_test/test.py
QWP/WebSocket client. Add the poison escalation dwell window, defaulting to 5s through config while preserving zero-window behavior for raw send-core callers and tests. Poison strikes now retain the first strike timestamp, survive reconnects, reset on ACK progress, and terminal errors report the actual strike count. Add paced below-threshold recycles for retriable NACKs, pre-send retriable rejects, and non-orderly server closes after a send. The first reconnect attempt is delayed with Java-style jittered exponential pacing without consuming reconnect-attempt budget. Also fix adjacent reconnect and close handling: - honor pending ReconnectDelay in the background runner - avoid holding the store mutex across reconnect I/O - exempt orderly 1000/1001 closes and zero-send closes from poison strikes - accept max_frame_rejections and poison_min_escalation_window_millis as ingress-only no-op keys in egress config
The mid-stream role-switch exactly-once invariant is now implemented, so test_qwp_store_and_forward_uncoordinated_role_switch passes on all three bindings (cpp/rust/c). Under xfail(strict=True) an unexpected pass (XPASS) is reported as a failure, breaking the e2e pytest task. Remove the marker so the now-passing test runs as a normal test and real regressions fail.
The mock recycling server sets the listener non-blocking so its accept
loop can honor the run_for deadline. On macOS/BSD and Windows the socket
returned by accept() inherits the listener's non-blocking flag (Linux
does not), so set_read_timeout became a no-op and read_frame returned an
immediate WouldBlock. That kind is not in the whitelisted clean-disconnect
set, so the server thread hit panic!("recycling server failed to read
frame"), which propagated through handle.join().unwrap() and failed
qwp_ws_nack_recycles_are_paced_against_healthy_server and
qwp_ws_non_orderly_close_recycles_are_paced on mac/windows CI.
Force the accepted stream back to blocking mode right after accept(),
matching the existing pattern in spawn_role_reject_upgrade_server.
…_java_e2e_scenarios
Windows was hitting the 90-minute timeout and mac ran ~66m. The two biggest costs were the Rust dependency graph being compiled twice (a second CMake tree existed only to compile the headers under C++20) and everything running serially on one agent. - CMakeLists: build the C++ test/example targets under both the caller's standard and C++20 within a single tree (new opt-in QUESTDB_TEST_CXX20_VARIANTS, default OFF) instead of a separate build_CXX20 tree. Rust is compiled once and shared; only the C++ translation units are recompiled for the `_cxx20` twins. A downstream QUESTDB_TESTS_AND_EXAMPLES=ON build is unchanged (single standard). - compile.yaml: drop the second make; pass the new flag via a template parameter so CI opts in and other jobs stay single-standard. - run_all_tests.py: split into cargo / cpp / unit / integration modes; run each C++ binary plus its `_cxx20` twin when present. - pipeline: keep linux combined in RunOn (it carries the toolchain matrix); split the slow mac/windows legs into parallel RustTests (cargo) and NativeTests (build + C++ + integration) jobs. Unit tests run before the QuestDB build so they fail fast; Rust examples build on linux only.
…_java_e2e_scenarios
test(e2e): port java-client QWP e2e scenarios to c-questdb-client
On an undersized CI box a bounce's post-restart SFA replay storm can saturate the freshly-started server and starve its HTTP accept loop for well over a minute (observed ~96s), during which a producer can neither reconnect nor drain. With the drain budget at 2 min it can be fully consumed by one such window, so close_drain times out even though the server recovers — flaking test_all_mixed_with_bounce and friends. Raise both close_flush_timeout_millis and reconnect_max_duration_millis to 4 min for bounce variants (kept equal so the reconnect sub-budget never trips before the close_drain deadline). Non-bounce variants never reconnect and keep the tighter 2 min ceiling.
- clamp the direct connection's max_buf_size to the handshake's X-QWP-Max-Batch-Size, as the row sender already does (the error docs promised the negotiated cap but the connect path ignored it) - consume deferred-window exhaustion inside frame splitting: commit the published prefix, drain, and retry the range instead of latching the connection must-close (a split's extra frames are an internal detail callers cannot budget for); the top-level flush keeps the explicit "call sync()" contract via FrameOutcome::NoSlot - re-export the caller's Arrow array on every provably-not-delivered failure (\!in_doubt), not just FailoverRetry, so a deferred-capacity failure leaves the batch retryable Found by the py-questdb-client direct-path fault-injection tests (server-advertised cap + defer-aware acks against a mock server).
dir layoyt changed after allowing to borrow more than one sender from pool
Sender::wait(AckLevel::Durable, ..) silently degraded to plain acceptance on a connection that never negotiated durable acks, so a caller asking for durability could get an OK-level wait and never know.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR is the combined landing of #148 and #150 — two independently-developed columnar I/O tracks that both target QWP/WebSocket and share the same connection pool /
SymbolGlobalDict, shipped together so downstream consumers (py-questdb-client, in-tree C/C++ tests) see one self-consistent surface.#148 — Column-major sender (
column_sender)A DataFrame → Table ingest API.
QuestDbconnection pool +BorrowedSender+Chunk(per-columnVec<u8>that stacks wire bytes directly) + synchronousflush(AckLevel). Covers bool / signed integers / floats / UUID / Long256 / IPv4 / timestamps / VARCHAR /symbol_dict_{i8,i16,i32}bulk-intern. The connection-scopedSchemaRegistry(FULL / REFERENCE emit modes) andSymbolGlobalDictare shared with the row API, preserving the 1M-per-connection symbol cap on huge PandasCategoricaldicts. Full C ABI (include/questdb/ingress/column_sender.h) and a Criterion bench suite (column path ≈ memcpy ceiling; bulk-intern ~16× faster than per-row HashMap).#150 — Apache Arrow + Polars integration
Both directions over QWP/WebSocket:
Buffer::append_arrow/append_arrow_at_columnconsumes a wholeRecordBatchin one call, column-major dense bulk path (one memcpy per column; QWP null bitmap built by byte-stride OR-with-NOT of the Arrow validity buffer when boundaries align, per-row fallback only when bit-offsets are unaligned).Cursor::as_record_batch_reader()streamingRecordBatchiterator; Polars sub-feature provides the DataFrame bridge.line_sender_buffer_append_arrow*andline_reader_cursor_next_arrow_batch. Every producer-suppliedArrowArray/ArrowSchemais pre-validated beforefrom_ffi(schema depth ≤ 64,row_count≤ 16M, bounded per-node buffer/child counts, and rejection of NULL or under-sized buffer arrays), so a malformed struct returns an error instead of aborting the FFI crate'spanic = "abort"profile. The manual per-column FFI path caps each variable-length payload ati32::MAX.Why merged
The two tracks were developed on the same
jh_conn_pool_refactorbranch and converged on shared infrastructure (connection pool,SymbolGlobalDict, QWP/WS transport). Splitting them at review time would force one to ship behind a compatibility shim for the other; merging them together avoids that churn and gives C/C++ callers — and the upcoming Pandas / Polars wrapper inpy-questdb-client— a column-major, zero-redundant-copy path into QuestDB in one cut.Public surface
See the original PRs' "Public surface" / "What's in the box" sections:
Feature gating
questdb-rs:arrow+polarsare opt-in features, excluded fromalmost-all-features; column_sender lives behindsync-sender-qwp-ws.questdb-rs-ffi:arrowfeature mirrors.CMakeLists.txt:QUESTDB_ENABLE_ARROW=OFFby default; auto-flipped toONwhenQUESTDB_TESTS_AND_EXAMPLES=ONso tests / examples exercise the Arrow path without explicit opt-in.Test plan
test_arrow_c.c/test_arrow_egress.cpp/test_arrow_ingress.cppwired into CMake and exercised in CIarrow_egress_fuzz/arrow_ingress_fuzz/arrow_round_trip_fuzz/arrow_alignment_fuzzcargo bench --features sync-sender-qwp-ws --bench column_senderpy-questdb-client, WS-7)Closes #148, #150.
Summary by CodeRabbit
New Features
Examples
Documentation
Tests
Chores (CI)
Benchmarks