Skip to content

docs: fetch read coalescing (DOC-2386) - #1821

Merged
Feediver1 merged 14 commits into
betafrom
DOC-2386-fetch-read-coalescing
Jul 24, 2026
Merged

docs: fetch read coalescing (DOC-2386)#1821
Feediver1 merged 14 commits into
betafrom
DOC-2386-fetch-read-coalescing

Conversation

@Feediver1

@Feediver1 Feediver1 commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Resolves https://redpandadata.atlassian.net/browse/DOC-2386

Adds documentation for fetch read coalescing, the opt-in per-shard fetch read coalescer new in 26.2 that removes redundant read CPU and fetch-response memory under high consumer fan-out.

Changes

  • New page modules/manage/pages/cluster-maintenance/fetch-read-coalescing.adoc: contextualized intro (the N×→1× redundancy story and an explicit use-when), how it works (coalescing key, the ready-hit/in-flight-hit/miss outcomes, weak retention, per-shard scope), enable procedure (kafka_fetch_read_coalescing_enabled, live toggle, ~0.5 MB/shard cache), an Evaluate coalescing effectiveness section with the four vectorized_kafka_fetch_read_coalescer_* counters and the hit-ratio formula for judging workload fit, and limitations.
  • Nav: entry after Follower Fetching under Consume Data.
  • follower-fetching.adoc: an ifndef::env-cloud[]-gated TIP cross-linking the new page (that page is single-sourced to cloud-docs; the new page is docs-only, so the cloud build never parses the xref).

All content is sourced from the engineering TOI: engineering/core/toi/v26.2/2026-07-21-fetch-read-coalescing.md (Brandon Allard, SME), and verified against the implementation PR redpanda#31010 (merged to dev 2026-07-13, 26.2-only, no backports): property description, coalescing-key definition, obligatory-vs-strict duplicate-read caveat, and weak-retention behavior all match the source.

Single-sourcing for Redpanda Cloud (added after SME input)

Per Brandon Allard: fetch read coalescing can be enabled on BYOC and Dedicated clusters, so the page is now single-sourced:

  • // tag::single-source[] wraps the page; companion stub PR: cloud-docs#645 (draft — must merge only after this content reaches docs main at 26.2 promotion, since cloud-docs consumes streaming from main).
  • Cloud build gets: an availability line (BYOC/Dedicated), the enable procedure plus a gated pointer to cloud's cluster-configuration page, and the limitations.
  • The Evaluate coalescing effectiveness section is ifndef::env-cloud[]-gated: Cloud exposes only public_metrics to customers, and the coalescer counters are internal vectorized_* metrics. (Worth a Brandon confirmation that there's no Cloud-side visibility planned.)
  • The cross-link TIP on follower-fetching.adoc stays ifndef::env-cloud[]-gated for now; un-gating it is a follow-up after the cloud stub merges (otherwise the cloud build gets an xref to a page that doesn't exist there yet).

Review notes

  • kafka_fetch_read_coalescing_enabled is not yet in the generated cluster-properties partial (arrives with the 26.2 property regen), so the page mentions it in code font without an xref; the link can be added after regen.
  • The four coalescer metrics arrive via the 26.2 internal-metrics regen.
  • Deliberately no version callout in prose (the page ships in the 26.2 docs version).
  • Per the ticket chain: findings from this write-up inform the final disposition of the Cloud Topics tuning-material ticket (DOC‑2129, written with a non-linking hyphen on purpose — this PR implements DOC-2386 only).

🤖 Generated with Claude Code

Preview pages

Updates during review

  • Enterprise feature (leadership decision 2026-07-23): enterprise-license NOTE + license prerequisite on the page (Self-Managed only) and a new row in the Enterprise features table in get-started:licensing/overview.adoc.
  • Moved to Manage > Cluster Maintenance (PM feedback): page now lives at modules/manage/pages/cluster-maintenance/fetch-read-coalescing.adoc, next to Manage Throughput; nav, TIP xref, and licensing-table xref updated; cloud-docs#645 stub mirrored.

New consume-data page for the 26.2 opt-in per-shard fetch read
coalescer: concept, enable procedure, hit-ratio evaluation guidance,
and limitations. Adds a nav entry and an env-gated cross-link TIP on
the follower-fetching page (single-sourced; cloud build unaffected).

Sourced from the engineering TOI (cupboard
engineering/core/toi/v26.2/2026-07-21-fetch-read-coalescing.md).

Co-Authored-By: Claude Fable 5 <[email protected]>
@Feediver1
Feediver1 requested a review from a team as a code owner July 22, 2026 17:17
@netlify

netlify Bot commented Jul 22, 2026

Copy link
Copy Markdown

Deploy Preview for redpanda-docs-preview ready!

Name Link
🔨 Latest commit 9e62122
🔍 Latest deploy log https://app.netlify.com/projects/redpanda-docs-preview/deploys/6a63d06a65f09c00082086fe
😎 Deploy Preview https://deploy-preview-1821--redpanda-docs-preview.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@coderabbitai

coderabbitai Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: fb626dca-81d7-4b59-97c9-5a8badc796a5

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch DOC-2386-fetch-read-coalescing

Comment @coderabbitai help to get the list of available commands.

Comment thread modules/develop/pages/consume-data/fetch-read-coalescing.adoc Outdated
Comment thread modules/manage/pages/cluster-maintenance/fetch-read-coalescing.adoc Outdated
Reword the enable guidance (disabled by default, little-to-no benefit)
and add an intro sentence to Limitations.

Co-Authored-By: Claude Fable 5 <[email protected]>
Per SME (Brandon Allard), the feature can be enabled on BYOC and
Dedicated clusters. Wrap the page in a single-source tag with env
gating: cloud gets an availability line, the enable procedure with a
pointer to cloud cluster configuration, and the limitations; the
internal-metrics evaluation section is excluded from the cloud build
(only public_metrics is customer-visible in Cloud). Companion stub:
cloud-docs#645.

Co-Authored-By: Claude Fable 5 <[email protected]>
Comment thread modules/develop/pages/consume-data/fetch-read-coalescing.adoc Outdated
Comment thread modules/develop/pages/consume-data/fetch-read-coalescing.adoc Outdated
Comment thread modules/manage/pages/cluster-maintenance/fetch-read-coalescing.adoc
Comment thread modules/develop/pages/consume-data/fetch-read-coalescing.adoc Outdated
@ballard26

Copy link
Copy Markdown

In addition to my other comments it's likely worth making a note somewhere that fetch coalescing should only be enabled for high fanout workloads. Enabling it for workloads with 1:1 or 1:2 fanout will likely result in higher reactor util due to the overhead the de-duplication logic introduces.

The term was invented by this page and never defined; spell out the
condition instead (same partition, same offset, same fetch settings).

Co-Authored-By: Claude Fable 5 <[email protected]>
Comment thread modules/develop/pages/consume-data/fetch-read-coalescing.adoc Outdated
Feediver1 and others added 3 commits July 23, 2026 13:16
Per ballard26's review: kafka_fetch_read_coalescing_enabled is not
whitelisted for user control in Redpanda Cloud. Gate the rpk procedure
to Self-Managed and route cloud readers to Redpanda Support. Revert to
self-serve wording if the property is whitelisted before release.

Co-Authored-By: Claude Fable 5 <[email protected]>
Per ballard26: the de-duplication logic runs on every fetch read, so
1:1 or 1:2 fan-out likely increases reactor utilization. Strengthen
the enable guidance in both env variants and add a limitations bullet.

Co-Authored-By: Claude Fable 5 <[email protected]>
@Feediver1

Copy link
Copy Markdown
Contributor Author

Added in 66eed08 — and worth being loud about, since the page previously undersold this as "little-to-no benefit": the enable guidance in both the Self-Managed and Cloud variants now says to enable/request coalescing only for high fan-out workloads because at low fan-out (one or two consumers per partition) the de-duplication overhead can increase reactor utilization, and there's a new Limitations bullet making the same point where readers check before enabling.

@ballard26 ballard26 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Feediver1 and others added 4 commits July 23, 2026 16:22
Per leadership decision: add the enterprise-license NOTE and a license
prerequisite to the page (Self-Managed only; Cloud portions unchanged),
and list the feature in the Enterprise features table.

Co-Authored-By: Claude Fable 5 <[email protected]>
Per PM feedback: the page's actor is the platform operator (cluster
config enablement, Enterprise license, broker metrics evaluation), not
the consuming-application developer. Update nav, the follower-fetching
TIP, the licensing table xref, and page-categories to match.

Co-Authored-By: Claude Fable 5 <[email protected]>
The lead-in says "Before enabling..." but the section sat after the
enable and evaluate procedures. Follow the cloud-topics precedent:
decision-relevant limitations go before the action, after the
mechanics section its bullets depend on.

Co-Authored-By: Claude Fable 5 <[email protected]>
Cloud readers request the feature rather than enable it: "Before
enabling..." for Self-Managed, "Before requesting..." for Cloud.

Co-Authored-By: Claude Fable 5 <[email protected]>

@JakeSCahill JakeSCahill left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving, this is in good shape. The single-sourcing wiring all checks out: I verified develop:consume-data/follower-fetching.adoc and manage:monitor-cloud.adoc both exist in cloud-docs, and the cloud-docs#645 stub mirrors the final page path.

I've left some optional inline suggestions. The main one picks up @ballard26's point about implementation detail in the How it works section: keep the section, trim it to behavior. None of these need to block merge, take or leave them.

Two things I couldn't leave inline because the files aren't in this diff:

  1. Release notes: get-started/release-notes/redpanda.adoc on beta covers the other 26.2 features but has no entry for this one yet. Something like this, maybe after the User-based throughput quotas section:

    == Fetch read coalescing
    
    Redpanda can now coalesce concurrent fetch reads that request the same data with the same fetch settings, sharing one read result across all requesting consumers. For workloads where many consumers tail the same partitions, this reduces read CPU and fetch-response memory from roughly N times to one. Fetch read coalescing is an Enterprise feature and is disabled by default. See xref:manage:cluster-maintenance/fetch-read-coalescing.adoc[] to get started.
  2. Manage Throughput: it's the natural tuning-story neighbour, so a See also line on manage-throughput.adoc pointing here would help discovery. Note that page is single-sourced to cloud-docs, so gate the link with ifndef::env-cloud[] until cloud-docs#645 merges (same reason as your follower-fetching TIP).

Follow-ups for after the 26.2 regens land: link kafka_fetch_read_coalescing_enabled back to this page via property-overrides.json and swap the code-font property mention here for an xref, and same idea for the four coalescer metrics once the internal metrics regen lands.

Comment thread modules/manage/pages/cluster-maintenance/fetch-read-coalescing.adoc Outdated
Comment thread modules/manage/pages/cluster-maintenance/fetch-read-coalescing.adoc Outdated
Comment thread modules/manage/pages/cluster-maintenance/fetch-read-coalescing.adoc Outdated
Comment thread modules/manage/pages/cluster-maintenance/fetch-read-coalescing.adoc Outdated
Comment thread modules/manage/pages/cluster-maintenance/fetch-read-coalescing.adoc Outdated
Comment thread modules/manage/pages/cluster-maintenance/fetch-read-coalescing.adoc Outdated
Comment thread modules/get-started/pages/licensing/overview.adoc Outdated
Comment thread modules/get-started/pages/licensing/overview.adoc Outdated
@Feediver1
Feediver1 merged commit ca8a2bb into beta Jul 24, 2026
7 checks passed
@Feediver1
Feediver1 deleted the DOC-2386-fetch-read-coalescing branch July 24, 2026 20:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants