Skip to content

Feat: add dataset module in tracel core#166

Open
Zatiji wants to merge 20 commits into
mainfrom
feat/add-dataset-modules
Open

Feat: add dataset module in tracel core#166
Zatiji wants to merge 20 commits into
mainfrom
feat/add-dataset-modules

Conversation

@Zatiji

@Zatiji Zatiji commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

This branch adds dataset streaming from Station, so you can pull a Station-hosted dataset into training as a burn::data::dataset::Dataset<T>.

Concretely:

  • A DatasetProvider trait + DatasetModule (tracel-core/src/dataset/mod.rs): the abstraction for fetching pages of raw dataset items over the network.

  • AnnotationDataset<T> (burn.rs): wraps that in burn's Dataset<T> trait: builds a lightweight index of valid items on first use, decodes each item as JSON into T, skips malformed items cleanly.

  • Station wiring (station.rs): implements DatasetProvider by calling tracel-client's dataset HTTP endpoints, plus a not-found error path that reports whether it's the dataset or the version that's missing.

  • Plugged into Context: context.datasets() returns a DatasetModule when connected via Station, None otherwise, same pattern as the existing model_registry().

  • Exposed publicly as tracel::dataset::{AnnotationDataset, DatasetModule}.

Usage: context.datasets().unwrap().as_burn_dataset::<MyItem>(name, version) gives you a burn::Dataset<MyItem> ready to hand to a Dataloader.

Note: caching is out of scope for this PR. It'll be added in a follow-up once the core logic here is approved

Copilot AI review requested due to automatic review settings July 17, 2026 16:01

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants