Skip to content

feat: DocumentSplitter — add split_by="token" mode using tiktoken #12528

Description

@rautaditya2606

Is your feature request related to a problem? Please describe.
RecursiveDocumentSplitter supports split_unit="token" (tiktoken, o200k_base) for token-aware
chunking. DocumentSplitter does not — split_by="token" is not a valid value. Users who need
simple (non-recursive) token-aware chunking are forced to use RecursiveDocumentSplitter even when
they don't need recursion, or implement their own splitting logic outside Haystack.

Describe the solution you'd like
Add "token" as a valid split_by mode in DocumentSplitter, using tiktoken for encoding.
The behaviour should mirror RecursiveDocumentSplitter: encode the full text to tokens, chunk by
split_length tokens with split_overlap overlap, decode each chunk back to a string.

A tokenizer_encoding param (default "o200k_base", matching the existing component) should be
exposed so users can choose other encodings (e.g. "cl100k_base" for GPT-3.5/4).
tiktoken should be a lazy optional dependency — same pattern as RecursiveDocumentSplitter.

splitter = DocumentSplitter(split_by="token", split_length=256, split_overlap=32)
result = splitter.run(documents=[Document(content="...")])

Describe alternatives you've considered
Using RecursiveDocumentSplitter with a single separator is the closest workaround, but it adds
unnecessary complexity for pipelines that just need flat token-based chunking. Using remove_regex
or a custom splitting_function requires users to reimplement tokenization logic themselves.

Additional context

  • Component: haystack/components/preprocessors/document_splitter.py
  • RecursiveDocumentSplitter already has the tiktoken pattern — this aligns the two components.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions