Zero-copy Arrow interop (to_arrow / __arrow_c_stream__ / from_arrow)

[wudidapaopao]

Adds Arrow interop to DataStore, built on the Arrow PyCapsule interface (the pandas 3.0 standard). Lets DataStore results flow to Polars / DuckDB / pandas 3.0 without a pandas round-trip.

What

• to_arrow(types='native'|'pandas') — a fully pushed-down pipeline returns a pyarrow.Table straight from chDB's ArrowTable output (no pandas round-trip), keeping ClickHouse-native types (UInt64→uint64, nullable ints stay nullable). Falls back to converting the pandas result for pipelines that need pandas-side ops.
• __arrow_c_stream__() — pa.table(ds), pl.from_arrow(ds), duckdb.sql('... FROM ds') and pd.DataFrame.from_arrow(ds) all work with no glue code.
• from_arrow(data) — ingest any PyCapsule producer (pyarrow Table/RecordBatch/reader, Polars, pandas 3.x, DuckDB, another DataStore).
• Adds Connection.query_arrow / Executor.query_arrow.

Notes

• Additive only; to_df() and existing behavior unchanged.
• Arrow has no row index, so group keys from groupby(as_index=True) appear as regular columns.
• Ingest goes through pyarrow (like pandas 3.0's own from_arrow), so it is not zero-copy on import; the zero-copy/native-typed path is on export.

Tests

datastore/tests/test_arrow_interop.py — 23 tests: native types, filter/select/groupby pushdown, interop (pyarrow/Polars/DuckDB/pandas), ingest, round-trips. Passing locally.

Closes #595

Note

Add zero-copy Arrow interop (to_arrow, from_arrow, __arrow_c_stream__) to DataStore

• Adds DataStore.to_arrow(types='native'|'pandas') which returns a pyarrow.Table; in 'native' mode it pushes down a single SQL query to chDB's Arrow output format, falling back to pandas conversion when pushdown is unavailable.
• Adds DataStore.from_arrow(data) classmethod accepting Arrow PyCapsule objects, pyarrow.Table/RecordBatch/RecordBatchReader, and pandas/Polars DataFrames.
• Implements DataStore.__arrow_c_stream__ so DataStore objects are directly consumable by PyArrow, Polars, DuckDB, and pandas 3.x without intermediate copies.
• Adds Connection.query_arrow and Executor.query_arrow to execute SQL and return a pyarrow.Table via chDB's ArrowTable output format.
• A new test module covers round-trips, native type preservation (decimal, list, map, struct, binary), empty/null columns, requested_schema casting, and error handling.

^{Macroscope summarized 54b97e9.}

[wudidapaopao]

@chibugai, review this PR.

[chibugai]

Unlike execute and query_df, this method does not wrap the _conn.query() call in a try/except. On failure, raw chDB exceptions propagate untranslated instead of being converted into a friendly ExecutionError. This breaks the established error-handling pattern and exposes low-level errors to callers. Consider wrapping the call consistently:

try:
    return self._conn.query(sql, "ArrowTable")
except Exception as e:
    self._logger.error("[chDB] Arrow query failed: %s", e)
    friendly_msg = translate_remote_error(e)
    raise ExecutionError(f"Query execution failed: {friendly_msg}\nSQL: {sql}")

[wudidapaopao]

Good catch — fixed in 54b97e9. query_arrow now wraps the _conn.query() call in the same try/except as execute(), logging and translating failures via translate_remote_error into a friendly ExecutionError instead of leaking the raw chDB exception. Added a regression test (TestQueryArrowErrorHandling) that asserts an invalid query raises ExecutionError.

[chibugai]

The fallback uses preserve_index=False, which drops the index instead of converting it to columns. This contradicts the docstring Note ("group keys from groupby(as_index=True) appears as regular columns here") and silently loses data: a groupby(as_index=True).agg(...) pipeline routed through this fallback (e.g. a DataFrame source, or any pandas-side op, or types="pandas") returns a Table missing the group-key columns. This also diverges from the native path (_try_native_arrow), where group keys stay as columns. Consider resetting the index into columns before converting, e.g.:

df = self._execute()
if df.index.name is not None or isinstance(df.index, pd.MultiIndex) or df.index.names != [None]:
    df = df.reset_index()
return pa.Table.from_pandas(df, preserve_index=False)

(or use preserve_index=True only for named/group indexes) so the fallback matches both the documented behavior and the native path.

[wudidapaopao]

Agreed, this was a real data-loss bug — fixed in 54b97e9. The fallback now resets a meaningful index into columns before conversion:

df = self._execute()
if df.index.names != [None]:
    df = df.reset_index()
return pa.Table.from_pandas(df, preserve_index=False)

So group keys from groupby(as_index=True) (and any named index / MultiIndex) are preserved as columns, matching both the native path and the docstring Note. A plain unnamed RangeIndex is still left out (no spurious index column). Confirmed the repro (from_df(...).groupby("city").agg(...) previously returned only ["score"], now ["city", "score"]). Added regressions for the from_df-groupby, types="pandas"-groupby, and named-index cases.