Index table structure and layout-ordered text from xberg extraction#551
Merged
Conversation
Tables in PDFs were only ever indexed as flattened text, so row/column meaning was lost to both BM25 and the embedder. With table_extraction on, xberg's table structure recognition runs during extraction and each table is indexed as its own chunk carrying the markdown serialization (new chunk_type 'table', covered by the 'raw' search scope), with long tables split header-repeating. With layout_detection on, page text is ordered by detected reading order and the running header/footer bands are stripped. Both flags are off by default and marked reindex-required; the xberg pin moves to 1.0.0rc29. bb-9t16 bb-orox
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Tables in PDFs are indexed only as flattened text, so row and column meaning is lost to both BM25 and the embedder, and multi-column pages with running headers pollute chunks with interleaved text and repeated boilerplate.
Solution
With
table_extractionon, xberg's table structure recognition runs during extraction and each table is indexed as its own chunk carrying the markdown serialization (new chunk typetable, covered by the raw search scope), with long tables split header-repeating. Withlayout_detectionon, page text is ordered by detected reading order and the header/footer margin bands are stripped. Both flags are off by default and marked reindex-required; the xberg pin moves to 1.0.0rc29.