Buffer Abstraction

[zkfriendly]

Summary

This PR introduces the buffer abstraction layer. All buffer abstraction traits are defined in the buffer crate (src/buffer/mod.rs), with a CPU implementation that essentially wraps the existing in-memory code paths.

At this moment, the buffer crate still depends on existing algebra and protocol crates. Future work can consolidate the relevant logic directly into the buffer crate as the abstraction settles.

Since the original review, the abstraction has been simplified substantially: the slice/view layer (BufferRead/BufferWrite, CpuSlice/CpuSliceMut) is removed in favor of two traits (BufferOps, Buffer) over owned buffers only. Host readbacks (as_slice) now occur only at transcript boundaries where the data must reach the host anyway.

Notes

There is also an experimental Metal implementation, but it is not included in this PR.

zk_whir is not fully ported to the buffer abstraction yet. It uses .as_slice() in places to keep the code working, so behavior is essentially unchanged compared to directly using CPU buffers. Since WHIR v3 is coming and is expected to replace zk_whir, that path is intentionally not fully ported for now.

---

Supersedes #260 (same branch, previously from my fork — GitHub cannot retarget a PR head, so reopened from the in-repo branch). Review history and comment threads are on #260.

[codspeed-hq]

Merging this PR will not alter performance

✅ 10 untouched benchmarks
⏩ 22 skipped benchmarks¹

---

_{Comparing zkfr/buffer-abstraction (70fd192) with main (655783c)}

Footnotes

1. 22 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[shreyas-londhe]

Left a few minor comments, but overall looks clean. I'm not completely happy with including Merkle Tree ops in BufferOps, but it is not a blocker and we can move it out in a later PR when the need arises. Thanks for the PR @zkfriendly

[zkfriendly]

I'm not completely happy with including Merkle Tree ops in BufferOps

What about it doesn't fit into BufferOps?

[shreyas-londhe]

What about it doesn't fit into BufferOps?

It is about concern separation, we should follow a hierarchal trait system, where we compose traits instead of adding all the functionality in a single trait. Currently I have to implement the merkle tree ops even if I don't use it in code for example. But it is a non-blocker for now.

[xrvdg]

Discussed offline:

• Message parameter for RS
• Leverage type system for mutability and slices rather than specialised types.
• Naming guidelines for ad-hoc conversion

Non-blocking / follow-up PR:

• Merklize feels out of place inside BufferOps.
• We could get better ergonomics by replacing from_vec/from_slice with the standard From trait for a specific backend.
• Traits are usually best when they are composable rather than hierarchical. Since BufferOps seems to handle host <-> backend communication, and Buffer should ideally only care about backend-specific operations, we might be able to remove BufferOps as a trait dependency on Buffer.

I think after this, the follow up PRs and a bit of usage a more intuitive name for the traits will come to mind.