feat(runtime): Implement pull based arroyo (hackweek) - #560

Draft

tryangul wants to merge 32 commits into

mainfrom

rbroughan/pull-based-arroyo-part-2

Commits on Aug 26, 2026

Full stage based implementation of pull streaming
tryangul
committed
fc8f07c
Handle N:1 and N:0 use cases. Refactor StageResult, handlers and organize for clarity.
tryangul
committed
44271a6
Support concurrent stage application.
tryangul
committed
f3bda85
Actually commit offsets.
tryangul
committed
3ef9891
Add graceful termination aka cancellation support.
tryangul
committed
d899bca
Comment tweaks.
tryangul
committed
1c35ec6
handle fatal kafka errors, preserve dlq headers, init coarsetime updater, move offset tracker.
tryangul
committed
907eef7
Basic rebalance handling.
tryangul
committed
e75addb
Factor out PipelineRunner. Update examples.
tryangul
committed
4324ecd
Formatting and comment cleanup.
tryangul
committed
8da0a08
Fix stage name lifetimes.
tryangul
committed
9cb6059
Use singleton updater to prevent leak.
tryangul
committed
4204697
Synchronize drain on rebalance.
tryangul
committed
3180d1d
Drain regardless of outcome
tryangul
committed
cafbf7e
Revert drain on rebalance due to potential threading deadlock.
tryangul
committed
3fef820
Add BatchStage
tryangul
committed
5c28651
Break out stream collector.
tryangul
committed
402326a
Add flush timer
tryangul
committed
15e9d56
Buffer now has an Out type
tryangul
committed
4cfa420
Formatting
tryangul
committed
7f8b802
Add Pipeline trait
tryangul
committed
9932f36
Fmt.
tryangul
committed
a1c4b29
Doc test import
tryangul
committed
53bf902
Cleanup types with aliases. Convert kafka stats logging to metrics.
tryangul
committed
abe5392
Cargo fmt.
tryangul
committed
ad99a41

Commits on Aug 28, 2026

ref(pull): remove unnecessary Arc around raw payload
tryangul
committed
fa15d64
ref(pull): use coarsetime for stage durations.
tryangul
committed
8d60f47
ref(pull): use parking_lot mutex for batch stage to avoid mutex syscall.
tryangul
committed
2f92802
ref(pull): boxed dyn producer for dlq handler.
tryangul
committed
3abd21c
ref(pull): owned streams.
tryangul
committed
0a0c1f8
ref(pull): Map_emit.
tryangul
committed
b0e94e2

Commits on Aug 29, 2026

ref(pull): Fix-up pipeline runner.
tryangul
committed
5145c34