v0.4.0 Updated to GoMLX v0.28.0, added SAM2 and Gemma4; add CLI tool `cmd/hubinfo`.

v0.4.0: Updated to GoMLX v0.28.0, added SAM2 and Gemma4; add CLI tool cmd/hubinfo.

• #46, #47: Expanded the HuggingFace repository metadata retrieval to support detailed file/LFS size information and adds a new command-line tool cmd/hubinfo to display this metadata in the terminal.
  • Added hub.Repo.IterFileInfos() to iterate over files with complete FileInfo records without triggering file downloads.
  • New CLI Tool (cmd/hubinfo):
• Improve support for large datasets, on-demand downloads, and CLI styling`
• Refreshed README.md, with fixes and updates to new features. Including fixing of the accompanying go-huggingface.ipynb notebook.
• Added SAM2 and Gemma4 models support.

v0.3.5 Added `datasets`, `tokenizers/bucket`, significant improvements on `transformer` and `safetensors`.

• Added .github with continuous integration (CI) workflow: only for Linux/amd64 because it should work the
  same on other platforms.
• Package tokenizers: API change!
  • Updated api.Tokenizer interface: added EncodeWithAnnotations, VocabSize(), Normalize() and Config methods,
    and cleaned up the API.
• Package tokenizers/hftokenizer:
  • Added support to AddBosToken and AddEosToken.
• Package tokenizers/sentencepiece:
  • Added post-processing.
  • Spans only generated for EncodeWithAnnotations.
  • Added support to AddBosToken and AddEosToken.
• Package tokenizers/bucket
  • Added bucket package for streaming tokenization of sentences into buckets (or batches) of discrete sizes,
    to minimize padding.
  • Added "Two-Bits Bucketing" strategy.
• Package datasets:
  • Added datasets package for downloading and iterating over parquet files of datasets from the HuggingFace Hub.
  • Added cmd/generate_dataset_structs for generating Go structs for dataset records.
  • Added ParquetFixListSchema for fixing list schema parsed from Go struct (a bug? in parquet-go where it hard-codes
    the group/element node names in lists).
  • Added CreateParquetReader for creating a parquet reader for the given dataset, config and split.
    Less convenient than an iterator, but allows for random access.
• Package transformer
  • Renamed main method to AllLayers: it returns both the final hidden state and all layer outputs;
    it added RoPE positional embeddings support; added support for scaling factor.
  • Updates to modified GoMLX transformer API.
  • Added support for configured task prompts (via task_prompts.json):
    • QueryPrompt builds the full query prompt, based on a task code.
    • RegisteredPromptTasks returns a list of all task codes for which prompts are registered.
    • GetTaskPrompt returns the prompt string for the given task code.
  • LoadContext now accepts an optional (nil-lable) backend and loads the variables directly into the backend.
  • Fixed mask support for batches with padding.
• Package safetensors:
  • IterTensors and IterTensorsFromRepo now take an optional (nil-lable) backend for reading tensors directly into
    a backend.
  • IterTensors and IterTensorsFromRepo implemented with github.com/edsrzf/mmap-go (to avoid one kernel buffer copy to normal memory).
  • IterTensorsFromRepo uses now a parallelized
    pipeline (including parallelizing transfers to the GPU, to ensure XLA will optimize them in a queue) for loading model throughput.
  • Loading of KaLM-Gemma3 12B model (with 22 GB of weights) now takes ~1.5s where previously it took ~8.5s (and slightly faster than in Python using ~1.7s).
• Examples (/examples/...):
  • Added Tecent's KaLM-Gemma3 12B model sentence embedder example.
  • Added MSMARCO Dataset example.
  • Added BAAI (Beijing Academy of Artificial Intelligence) BGE Small Sentence Embedder (English) v1.5: a small and very performatic sentence embedder (BERT based).
  • Added MSMARCO benchmark_embed
    command-line benchmark of sentence embeddings, that also serves as an example of how to use the library.

v0.3.4 go-huggingface transformers

(Skipped v0.3.3)

• Added models/transformer package for loading HuggingFace transformer models (Experimental).
  (Tested with tencent/KaLM-Embedding-Gemma3-12B-2511)
• Added PostProcessor support and EncodeWithOptions to the tokenizer API.
• Split on added tokens before pre-tokenization in hftokenizer.

v0.3.2 Model loading

• Added safetensors support.
• Added support for GGUF file format.
• Tokenizer API improvements.

v0.3.1

v0.3.1

• Fixed go.mod/go.sum.

v0.3.0

• Bumped the version of GoMLX in tests and documentation.
• Bumped version of dependencies: including github.com/daulet/tokenizers, which requires a fresh download of the
  corresponding c++ library libtokenizers.a.

v0.3.0 Updated dependencies

v0.3.0

• Bumped the version of GoMLX in tests and documentation.
• Bumped version of dependencies: including github.com/daulet/tokenizers, which requires a fresh download of the
  corresponding c++ library libtokenizers.a.

v0.2.2 Fixed file truncation issues during download.

• Fixed file truncation issues during download.

v0.2.1 Forced refresh of revision hash before using it.

• Forcefully refresh (download) the revision's hash at least once before using.

v0.2.0 Add Windows support by moving to cross platform flock

• Add Windows support by moving to cross platform flock: see PR #6, thanks to @mrmichaeladavis

v0.1.2 Custom end-point support (also `$HF_ENDPOINT` support)

• If verbosity is 0, it won't print progress.
• Added support for custom end-points. Default being "https://huggingface.co" or the environment variable
  $HF_ENDPOINT if defined.