Skip to content

perf(querier): classify time-series columns once per query - #12611

Open
tushar-signoz wants to merge 1 commit into
tvats-json-column-scanfrom
tvats-consume-column-plan
Open

perf(querier): classify time-series columns once per query#12611
tushar-signoz wants to merge 1 commit into
tvats-json-column-scanfrom
tvats-consume-column-plan

Conversation

@tushar-signoz

Copy link
Copy Markdown
Contributor

Description

Stacked on #12555 — the first commit here is that PR; review only the perf(querier) commit.

The graph reader decided what each result column was by switching on its Go pointer type, for every cell of every row — and the switch only listed 32- and 64-bit numeric widths. Anything narrower fell through and was silently dropped, so a raw ClickHouse panel aggregating or grouping by a small-integer or boolean column (max(severity_number), GROUP BY has_error, kind) either rendered an empty chart or merged every group into one unlabelled line.

  • Each column is now classified once per query — timestamp, numeric, bool, string, or document — and every row just acts on that. Numeric coverage comes from the same helper that already counted numeric columns, so the two can no longer disagree and any width works.
  • Rows reuse scratch instead of allocating per row: the aggregation slots, the label slices, and the label objects for rows that land in an already-seen series.
  • Values skip the reflection boxing on the way out of the driver for the common types.
  • The __result_<n> index is bounded before it sizes anything. The alias is user-written in raw SQL, and an overflowing or huge index (__result_99999999999999999999) crashed the reader with a panic — on main too, where the result buckets are sized from the same number after the rows are read. Indices past 1000 are now read as plain numeric columns.

Additional Information

Behaviour before/after, raw SQL panels on a 1M-row stack:

query before after
max(severity_number) over time (UInt8) 200 but an empty chart values returned
grouped by severity_number 1 unlabelled series, 25 groups merged 25 labelled series
count() AS __result_99999999999999999999 panic value returned

Cost, measured per request on a GROUP BY trace_id query returning 200k series, from pprof allocation deltas on a live server: 25% fewer bytes allocated and 23% less CPU (35% fewer allocations inside the reader itself). Response time is unchanged — these queries are dominated by ClickHouse and by encoding the response.

The time-series reader decided what each cell was by switching on its pointer
type, per cell per row — and the switch listed 32- and 64-bit widths only, so
a raw ClickHouse query aggregating or grouping by an Int8, UInt8, Int16,
UInt16 or Bool column (max(severity_number), GROUP BY has_error) either
emptied the chart or merged every group into one unlabelled series.

Classify each column once, up front, into what it contributes to every row:
timestamp, numeric, bool, string, or document. Numeric coverage now comes
from the same helper that counts numeric columns, so the two cannot disagree,
and any width works. Rows reuse the scratch that used to be allocated per row
— the aggregation map, the label slices, and the label objects for rows that
land in existing series — and values dodge the reflection boxing on the way
out of the driver. On a 200k-series group-by this is a quarter fewer bytes
allocated and a fifth less CPU per request.

The __result_<n> index is now also bounded before it sizes the aggregation
slots: the alias is user-written in raw SQL, and an overflowing or huge index
panicked the reader — on main too, where the bucket slice is sized from the
same number after the rows are read.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant