Skip to content

perf(runtime): add typed columnar aggregate reducers#57

Merged
ashione merged 4 commits into
mainfrom
auto/columnar-first-kernel-performance
Jun 14, 2026
Merged

perf(runtime): add typed columnar aggregate reducers#57
ashione merged 4 commits into
mainfrom
auto/columnar-first-kernel-performance

Conversation

@ashione

@ashione ashione commented Jun 13, 2026

Copy link
Copy Markdown
Owner

Summary

  • add state-columnar typed reducer paths for single-INT64 and two-INT64 SUM/COUNT/AVG aggregate shapes
  • extend source pushdown shape classification to predicate single-key COUNT/SUM/AVG and move CSV predicate aggregate execution onto typed reducer paths
  • add Phase 1 internal columnar execution substrate skeleton: ColumnarExecBatch, ColumnarExecColumn, and ColumnarExecView
  • preserve Arrow-backed lazy input inside the execution batch and make row materialization an explicit boundary
  • add a local columnar kernel benchmark gate covering aggregate typed-shape selection, file-source pushdown ratios, and string builtin plan reuse
  • document measured proof points and rejected mixed-key attempts in the plan/design docs
  • address review-found edge cases: safe int64 dense-domain estimation, predicate AVG empty-numeric semantics, and execution-batch backing row-count validation

Performance Evidence

  • single-int64 SUM: 52.0 ms -> 38.5 ms (~1.35x)
  • single-int64 COUNT: 41.0 ms -> 28.0 ms (~1.46x)
  • single-int64 AVG: 53.5 ms -> 38.0 ms (~1.41x)
  • two-int64 SUM: 163.0 ms -> 121.5 ms (~1.34x)
  • two-int64 COUNT: 159.0 ms -> 108.0 ms (~1.47x)
  • two-int64 AVG: 166.0 ms -> 122.5 ms (~1.36x)
  • CSV mixed predicate source count: 267,744 us -> 235,904 us (~1.13x)
  • rejected mixed string/int64 dictionary-id reducer attempt after mixed-string-int64 regressed to 634 ms

Validation

  • bazel test //:planner_v03_test //:file_source_test //:columnar_batch_test --test_output=errors
  • bazel test //:core_regression --test_output=errors
  • bazel test //:experimental_regression --test_output=errors
  • bazel build //:sql_demo //:df_demo //:stream_demo
  • bazel run //:actor_rpc_smoke
  • ./scripts/run_columnar_kernel_benchmark_gate.sh
  • bazel run //:batch_aggregate_benchmark -- 1048576 5
  • bazel run //:file_source_benchmark -- 200000 3
  • bazel run //:string_builtin_benchmark -- 100000 5
  • git diff --check
  • targeted secret-pattern scan over changed files

@ashione ashione force-pushed the auto/columnar-first-kernel-performance branch from d99a12d to daa040a Compare June 13, 2026 16:37

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d99a12df4a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

out.selected[i + j] = 1;
out.indices.push_back(i + j);
++out.selected_count;
if (bounded && out.selected_count >= max_selected) goto scalar_tail;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Advance SIMD tail after bounded vector hit

On AVX2 builds, if a bounded selection (for example WHERE ... LIMIT n via combinedFilterSelection(..., max_selected)) reaches max_selected inside this vector loop, goto scalar_tail leaves i at the start of the same vector block. The scalar tail then reprocesses lanes that were already emitted and appends duplicate indices (e.g. LIMIT 1 can produce [0,0]), so filterTable can duplicate rows and exceed the requested limit; return immediately or advance the tail past the processed lane/block.

Useful? React with 👍 / 👎.

auto& col = cache->columns[ci];
for (auto& v : col.values) {
if (v.isNull()) continue;
v = Value(v.asFloat());

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep Float32 values serializable

When a SQL FLOAT/REAL table receives rows, this rewrites cells to the new DataType::Float32, but the existing table codecs were not updated for that type: BinaryRowBatchCodec falls through and deserializes it as a string, ProtoLikeSerializer decodes type 6 as null, and nanoarrow materialization infers string then calls asString(). Any inserted float column crossing RPC/plan serialization/export boundaries is therefore corrupted or throws, so either handle Float32 in those codecs or avoid emitting it here.

Useful? React with 👍 / 👎.

@ashione ashione force-pushed the auto/columnar-first-kernel-performance branch from daa040a to c40379d Compare June 13, 2026 16:49
Use state-columnar typed reducer paths for single- and two-INT64 SUM/COUNT/AVG aggregate shapes, and classify single-key predicate source COUNT/SUM/AVG pushdown onto typed shapes with CSV predicate evaluation support.

Add focused optimizer/file-source coverage, benchmark scenarios, columnar-first design notes, and Mobius delivery evidence.

Validation: bazel test //:core_regression --test_output=errors; bazel test //:experimental_regression --test_output=errors; bazel run //:batch_aggregate_benchmark -- 1048576 5; bazel run //:file_source_benchmark -- 200000 3; bazel run //:string_builtin_benchmark -- 100000 5; bazel build //:sql_demo //:df_demo //:stream_demo; bazel run //:actor_rpc_smoke; git diff --check; targeted secret scan.
@ashione ashione force-pushed the auto/columnar-first-kernel-performance branch from c40379d to d65e3c3 Compare June 13, 2026 16:53
@ashione ashione merged commit ae244d8 into main Jun 14, 2026
4 checks passed
@ashione ashione deleted the auto/columnar-first-kernel-performance branch June 14, 2026 15:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant