Streaming reads for cloud LeRobot-v3 & Zarr (inspect + ingest)#35
Merged
Conversation
… inspect)
`forge inspect s3://…` / `gs://…` now reads metadata over the network with range
requests instead of downloading the whole dataset. Inspecting a 464 MB cloud
LeRobot-v3 dataset fetches ~3 KB (its info.json) in ~0.3s — a ~150,000x
reduction in bytes moved — and returns the identical DatasetInfo.
- forge/io/source.py: `DataSource` — a dataset root (local or fsspec-remote)
with the small metadata reads streaming inspection needs (json, parquet
footer/schema, exists, glob), fetched as range requests for remote sources.
- lerobot-v3 reader: `_inspect_remote` reads only meta/info.json (which carries
episode/frame counts, fps, robot, per-camera dims, and the action/observation
schema, so the mp4 probe is skipped). Local inspect path is untouched;
info.json→DatasetInfo mapping factored into `_populate_from_info` and shared.
- zarr reader: opens the store natively over fsspec (`zarr.open("s3://…")`
range-reads chunks); can_read checks markers remotely.
- registry: `detect_format` keeps remote URIs as strings (non-streamable readers
safely return False, so only streamable formats match without a download);
`STREAMABLE_FORMATS = {lerobot-v3, zarr}`.
- CLI: `forge inspect` streams remote streamable formats; falls back to download
for `--deep`, `--generate-config`, or other formats/commands.
Tests (tests/test_streaming.py): correctness over memory:// (right DatasetInfo,
no temp download) for both formats, plus a moto-S3 byte-budget assertion proving
inspect fetches < 1 MB even next to an 8 MB payload. Existing cloud-inspect test
updated (inspect now streams, `--deep` still downloads). No other tests change.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Extend streaming from inspect to the ingest/scoring path: `forge ingest s3://…` registers and quality-scores episodes by reading only the proprio parquet columns (state/action/timestamp) via range reads — never the videos. Ingesting a 464 MB cloud LeRobot-v3 dataset streams ~3 MB (vs 464 MB), in ~0.5s, producing identical episodes and quality scores. - lerobot-v3 reader: `_read_episodes_remote` yields proprio-only Episodes from parquet column reads (observation.state/action/timestamp), plus streamed tasks + episode metadata for language. state_dim/action_dim are left None to match the local reader so an episode's content_hash is identical whether streamed or downloaded — cross-path re-ingest stays idempotent (verified: 100/100 hashes match; streaming into a download-built catalog skips all 100). Zarr streams via its native fsspec store (already wired). - catalog/ingest.py: `_resolve_for_ingest` streams remote streamable formats instead of localizing; other sources unchanged. Embeddings/video quality still download frames. Tests: streaming ingest over memory:// (registers + scores, no temp download) and a moto byte-budget assertion (ingest reads < 2 MB with an 8 MB video payload alongside). No regressions. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
arpitg1304
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Streaming reads for cloud datasets —
forge inspectandforge ingestons3:///gs://LeRobot-v3 and Zarr now read over the network with range requests instead of downloading the whole dataset.Both produce identical results to the local read — same DatasetInfo, same 100 episodes, same quality scores (avg 8.26). This is the credibility gate for the pretraining-scale story: point Forge at a huge cloud corpus and register + score it without pulling the videos.
How
forge/io/source.py—DataSource: a dataset root (local or fsspec-remote) exposing the metadata + column reads streaming needs (json, parquet footer/schema, parquet columns,exists,glob) as range requests for remote sources._inspect_remotereads onlymeta/info.json(episode/frame counts, fps, robot, per-camera dims, schema — so the mp4 probe is skipped); zarr opens the store natively over fsspec._read_episodes_remoteyields proprio-only Episodes from parquet column reads (observation.state/action/timestamp) — never touching the videos — so content-hash + the proprio quality metrics run streamed. Zarr streams via its native fsspec store.catalog/ingest.pystreams remote streamable formats instead of localizing.detect_formatkeeps remote URIs as strings so non-streamable readers safely returnFalse;STREAMABLE_FORMATS = {lerobot-v3, zarr}.Idempotency across paths
Streamed episodes deliberately leave
state_dim/action_dimNone(like the local reader) so an episode's content hash is identical whether streamed or downloaded. Verified: 100/100 hashes match between the two paths, and streaming into a download-built catalog skips all 100. Cross-path re-ingest stays a no-op.Scope
Streamed: inspect (metadata) and ingest (register + proprio quality). Still download-to-temp:
forge inspect --deep, embeddings and video quality (they need frames), and non-streamable formats (HDF5, rosbag, video seeking). See the roadmap.Testing (rigorous)
tests/test_streaming.py:memory://for inspect and ingest (right output, and an assertion that no temp download happened).🤖 Generated with Claude Code