docs: fetch read coalescing (DOC-2386) - #1821
Conversation
New consume-data page for the 26.2 opt-in per-shard fetch read coalescer: concept, enable procedure, hit-ratio evaluation guidance, and limitations. Adds a nav entry and an env-gated cross-link TIP on the follower-fetching page (single-sourced; cloud build unaffected). Sourced from the engineering TOI (cupboard engineering/core/toi/v26.2/2026-07-21-fetch-read-coalescing.md). Co-Authored-By: Claude Fable 5 <[email protected]>
✅ Deploy Preview for redpanda-docs-preview ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Reword the enable guidance (disabled by default, little-to-no benefit) and add an intro sentence to Limitations. Co-Authored-By: Claude Fable 5 <[email protected]>
Per SME (Brandon Allard), the feature can be enabled on BYOC and Dedicated clusters. Wrap the page in a single-source tag with env gating: cloud gets an availability line, the enable procedure with a pointer to cloud cluster configuration, and the limitations; the internal-metrics evaluation section is excluded from the cloud build (only public_metrics is customer-visible in Cloud). Companion stub: cloud-docs#645. Co-Authored-By: Claude Fable 5 <[email protected]>
|
In addition to my other comments it's likely worth making a note somewhere that fetch coalescing should only be enabled for high fanout workloads. Enabling it for workloads with 1:1 or 1:2 fanout will likely result in higher reactor util due to the overhead the de-duplication logic introduces. |
The term was invented by this page and never defined; spell out the condition instead (same partition, same offset, same fetch settings). Co-Authored-By: Claude Fable 5 <[email protected]>
Per ballard26's review: kafka_fetch_read_coalescing_enabled is not whitelisted for user control in Redpanda Cloud. Gate the rpk procedure to Self-Managed and route cloud readers to Redpanda Support. Revert to self-serve wording if the property is whitelisted before release. Co-Authored-By: Claude Fable 5 <[email protected]>
Per ballard26: the de-duplication logic runs on every fetch read, so 1:1 or 1:2 fan-out likely increases reactor utilization. Strengthen the enable guidance in both env variants and add a limitations bullet. Co-Authored-By: Claude Fable 5 <[email protected]>
|
Added in 66eed08 — and worth being loud about, since the page previously undersold this as "little-to-no benefit": the enable guidance in both the Self-Managed and Cloud variants now says to enable/request coalescing only for high fan-out workloads because at low fan-out (one or two consumers per partition) the de-duplication overhead can increase reactor utilization, and there's a new Limitations bullet making the same point where readers check before enabling. |
Per leadership decision: add the enterprise-license NOTE and a license prerequisite to the page (Self-Managed only; Cloud portions unchanged), and list the feature in the Enterprise features table. Co-Authored-By: Claude Fable 5 <[email protected]>
Per PM feedback: the page's actor is the platform operator (cluster config enablement, Enterprise license, broker metrics evaluation), not the consuming-application developer. Update nav, the follower-fetching TIP, the licensing table xref, and page-categories to match. Co-Authored-By: Claude Fable 5 <[email protected]>
The lead-in says "Before enabling..." but the section sat after the enable and evaluate procedures. Follow the cloud-topics precedent: decision-relevant limitations go before the action, after the mechanics section its bullets depend on. Co-Authored-By: Claude Fable 5 <[email protected]>
Cloud readers request the feature rather than enable it: "Before enabling..." for Self-Managed, "Before requesting..." for Cloud. Co-Authored-By: Claude Fable 5 <[email protected]>
There was a problem hiding this comment.
Approving, this is in good shape. The single-sourcing wiring all checks out: I verified develop:consume-data/follower-fetching.adoc and manage:monitor-cloud.adoc both exist in cloud-docs, and the cloud-docs#645 stub mirrors the final page path.
I've left some optional inline suggestions. The main one picks up @ballard26's point about implementation detail in the How it works section: keep the section, trim it to behavior. None of these need to block merge, take or leave them.
Two things I couldn't leave inline because the files aren't in this diff:
-
Release notes:
get-started/release-notes/redpanda.adoconbetacovers the other 26.2 features but has no entry for this one yet. Something like this, maybe after the User-based throughput quotas section:== Fetch read coalescing Redpanda can now coalesce concurrent fetch reads that request the same data with the same fetch settings, sharing one read result across all requesting consumers. For workloads where many consumers tail the same partitions, this reduces read CPU and fetch-response memory from roughly N times to one. Fetch read coalescing is an Enterprise feature and is disabled by default. See xref:manage:cluster-maintenance/fetch-read-coalescing.adoc[] to get started. -
Manage Throughput: it's the natural tuning-story neighbour, so a
See alsoline onmanage-throughput.adocpointing here would help discovery. Note that page is single-sourced to cloud-docs, so gate the link withifndef::env-cloud[]until cloud-docs#645 merges (same reason as your follower-fetching TIP).
Follow-ups for after the 26.2 regens land: link kafka_fetch_read_coalescing_enabled back to this page via property-overrides.json and swap the code-font property mention here for an xref, and same idea for the four coalescer metrics once the internal metrics regen lands.
….adoc Co-authored-by: Jake Cahill <[email protected]>
Co-authored-by: Jake Cahill <[email protected]>
Resolves https://redpandadata.atlassian.net/browse/DOC-2386
Adds documentation for fetch read coalescing, the opt-in per-shard fetch read coalescer new in 26.2 that removes redundant read CPU and fetch-response memory under high consumer fan-out.
Changes
modules/manage/pages/cluster-maintenance/fetch-read-coalescing.adoc: contextualized intro (theN×→1× redundancy story and an explicit use-when), how it works (coalescing key, the ready-hit/in-flight-hit/miss outcomes, weak retention, per-shard scope), enable procedure (kafka_fetch_read_coalescing_enabled, live toggle, ~0.5 MB/shard cache), an Evaluate coalescing effectiveness section with the fourvectorized_kafka_fetch_read_coalescer_*counters and the hit-ratio formula for judging workload fit, and limitations.follower-fetching.adoc: anifndef::env-cloud[]-gated TIP cross-linking the new page (that page is single-sourced to cloud-docs; the new page is docs-only, so the cloud build never parses the xref).All content is sourced from the engineering TOI: engineering/core/toi/v26.2/2026-07-21-fetch-read-coalescing.md (Brandon Allard, SME), and verified against the implementation PR redpanda#31010 (merged to dev 2026-07-13, 26.2-only, no backports): property description, coalescing-key definition, obligatory-vs-strict duplicate-read caveat, and weak-retention behavior all match the source.
Single-sourcing for Redpanda Cloud (added after SME input)
Per Brandon Allard: fetch read coalescing can be enabled on BYOC and Dedicated clusters, so the page is now single-sourced:
// tag::single-source[]wraps the page; companion stub PR: cloud-docs#645 (draft — must merge only after this content reaches docsmainat 26.2 promotion, since cloud-docs consumesstreamingfrommain).ifndef::env-cloud[]-gated: Cloud exposes onlypublic_metricsto customers, and the coalescer counters are internalvectorized_*metrics. (Worth a Brandon confirmation that there's no Cloud-side visibility planned.)follower-fetching.adocstaysifndef::env-cloud[]-gated for now; un-gating it is a follow-up after the cloud stub merges (otherwise the cloud build gets an xref to a page that doesn't exist there yet).Review notes
kafka_fetch_read_coalescing_enabledis not yet in the generated cluster-properties partial (arrives with the 26.2 property regen), so the page mentions it in code font without an xref; the link can be added after regen.🤖 Generated with Claude Code
Preview pages
Updates during review
get-started:licensing/overview.adoc.modules/manage/pages/cluster-maintenance/fetch-read-coalescing.adoc, next to Manage Throughput; nav, TIP xref, and licensing-table xref updated; cloud-docs#645 stub mirrored.