Skip to content

PythonCodeSplitter with strip_docstrings=True emits invalid Python for docstring-only function/method/class bodies #12330

Description

@AmirF194

Describe the bug

PythonCodeSplitter with strip_docstrings=True emits a syntactically invalid chunk for any function, method, or class whose body consists solely of a docstring, for example a one-line custom exception class or a stub method. The docstring is the unit's only statement, so removing it leaves the def/class header with no body at all.

Error message

SyntaxError: expected an indented block after class definition on line 2

Expected behavior

A unit whose body is only a docstring should either keep the docstring in place or otherwise remain valid Python after strip_docstrings=True. It should never come out unparsable.

To Reproduce

from haystack import Document
from haystack.components.preprocessors import PythonCodeSplitter

source = '''
class MyError(Exception):
    """Custom error raised when the widget explodes."""
'''

splitter = PythonCodeSplitter(min_effective_lines=1, max_effective_lines=10, strip_docstrings=True)
result = splitter.run(documents=[Document(content=source)])

import ast
for chunk in result["documents"]:
    ast.parse(chunk.content)  # raises SyntaxError

Same result for a docstring-only top-level function and a docstring-only method.

Additional context

strip_docstrings=True is documented as a RAG-oriented option to shrink stored chunk content while still letting the docstring influence retrieval via meta_fields_to_embed=["docstrings"]. With it enabled, exactly the compact/simple declarations (one-line custom exceptions, stub/abstract methods) get corrupted into unparsable text before they are embedded or indexed.

System:

  • Haystack version: HEAD (0c4b23dbd)

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions