by ZSpaceLabs

wordchipper is a high-performance Rust byte-pair encoder tokenizer for the OpenAI GPT-2 tokenizer family. It achieves throughput speedups relative to tiktoken-rs in rust on a 64 core machine of ~4.3-5.7x (4 to 64 cores) for general regex BPE vocabularies, and ~6.9x-9.2x when using custom DFA lexers for specific OpenAI vocabularies. Under python wrappers, we see a range of ~2x-4x (4 to 64 cores) speedups over tiktoken.

Status

The productionization towards an LTR stable release can be tracked in the Alpha Release Tracking Issue.

Encode Side-by-Side Benchmarks

x 64 Core	r50k rust	gpt2 python	o200k rust	o200k python
wordchipper:logos	2.7 GiB/s	114.1 MiB/s	2.4 GiB/s	123.7 MiB/s
wordchipper	1.7 GiB/s	110.5 MiB/s	1.5 GiB/s	106.5 MiB/s
tiktoken*	386.0 MiB/s	25.5 MiB/s	265.2 MiB/s	32.7 MiB/s
bpe-openai			60.9 MiB/s	11.1 MiB/s
tokenizers	49.7 MiB/s	20.8 MiB/s	50.2 MiB/s	23.2 MiB/s

Read the full performance paper: wordchipper: Fast BPE Tokenization with Substitutable Internals

no_std Support

The core tokenization pipeline (spanning, encoding, decoding, vocabulary lookup) works in no_std environments. This is CI-verified against wasm32-unknown-unknown and thumbv7m-none-eabi targets.

[dependencies]
wordchipper = { version = "0.7", default-features = false }

Features that require std (training, file I/O, download, rayon parallelism) are behind feature flags that imply std.

Language Bindings

Python

pip install wordchipper

from wordchipper import Tokenizer

tok = Tokenizer.from_pretrained("cl100k_base")
tokens = tok.encode("hello world")  # [15339, 1917]
text = tok.decode(tokens)  # "hello world"

See bindings/python for full API and benchmarks.

JavaScript / TypeScript (WASM)

import {Tokenizer} from "./js/dist/index.js";

const tok = await Tokenizer.fromPretrained("o200k_base");
const tokens = tok.encode("hello world"); // Uint32Array
const text = tok.decode(tokens);          // "hello world"

See bindings/wasm for full API, build instructions, and examples.

Components

Published Crates

• wordchipper - main crate.
• wordchipper-training - training library.
• wordchipper-cli - chipper multifunction command-line tool.

Internally Sourced Dep Crates

You should never need to import these directly.

• wordchipper-disk-cache - disk cache.

Bindings

• wordchipper-python - Python bindings (PyO3/maturin)
• wordchipper-wasm - WASM bindings (wasm-bindgen) with TypeScript wrapper

Development Crates

A Number of internal crates are used for development.

• dev-crates

Acknowledgements

• Thank you to @karpathy and nanochat for the work on rustbpe.
• Thank you to tiktoken for their work in the rust tokenizer space.

License

wordchipper is distributed under the terms of both the MIT license and the Apache License (Version 2.0). See LICENSE-APACHE and LICENSE-MIT for details. Opening a pull request is assumed to signal agreement with these licensing terms