Is your feature request related to a problem? Please describe.
RecursiveDocumentSplitter supports split_unit="token" (tiktoken, o200k_base) for token-aware
chunking. DocumentSplitter does not — split_by="token" is not a valid value. Users who need
simple (non-recursive) token-aware chunking are forced to use RecursiveDocumentSplitter even when
they don't need recursion, or implement their own splitting logic outside Haystack.
Describe the solution you'd like
Add "token" as a valid split_by mode in DocumentSplitter, using tiktoken for encoding.
The behaviour should mirror RecursiveDocumentSplitter: encode the full text to tokens, chunk by
split_length tokens with split_overlap overlap, decode each chunk back to a string.
A tokenizer_encoding param (default "o200k_base", matching the existing component) should be
exposed so users can choose other encodings (e.g. "cl100k_base" for GPT-3.5/4).
tiktoken should be a lazy optional dependency — same pattern as RecursiveDocumentSplitter.
splitter = DocumentSplitter(split_by="token", split_length=256, split_overlap=32)
result = splitter.run(documents=[Document(content="...")])
Describe alternatives you've considered
Using RecursiveDocumentSplitter with a single separator is the closest workaround, but it adds
unnecessary complexity for pipelines that just need flat token-based chunking. Using remove_regex
or a custom splitting_function requires users to reimplement tokenization logic themselves.
Additional context
- Component:
haystack/components/preprocessors/document_splitter.py
RecursiveDocumentSplitter already has the tiktoken pattern — this aligns the two components.
Is your feature request related to a problem? Please describe.
RecursiveDocumentSplittersupportssplit_unit="token"(tiktoken,o200k_base) for token-awarechunking.
DocumentSplitterdoes not —split_by="token"is not a valid value. Users who needsimple (non-recursive) token-aware chunking are forced to use
RecursiveDocumentSplittereven whenthey don't need recursion, or implement their own splitting logic outside Haystack.
Describe the solution you'd like
Add
"token"as a validsplit_bymode inDocumentSplitter, using tiktoken for encoding.The behaviour should mirror
RecursiveDocumentSplitter: encode the full text to tokens, chunk bysplit_lengthtokens withsplit_overlapoverlap, decode each chunk back to a string.A
tokenizer_encodingparam (default"o200k_base", matching the existing component) should beexposed so users can choose other encodings (e.g.
"cl100k_base"for GPT-3.5/4).tiktoken should be a lazy optional dependency — same pattern as
RecursiveDocumentSplitter.Describe alternatives you've considered
Using
RecursiveDocumentSplitterwith a single separator is the closest workaround, but it addsunnecessary complexity for pipelines that just need flat token-based chunking. Using
remove_regexor a custom
splitting_functionrequires users to reimplement tokenization logic themselves.Additional context
haystack/components/preprocessors/document_splitter.pyRecursiveDocumentSplitteralready has the tiktoken pattern — this aligns the two components.