fix(parse): keep a leading whitespace-only field at the start of a row#1243
Open
spokodev wants to merge 1 commit into
Open
fix(parse): keep a leading whitespace-only field at the start of a row#1243spokodev wants to merge 1 commit into
spokodev wants to merge 1 commit into
Conversation
A whitespace-only field at the start of a row was dropped or truncated,
while the same field in any other column was preserved:
parseString(' \n', { headers: false }) // [[]] expected [[' ']]
parseString(' ,b\n', { headers: false }) // [['', 'b']] expected [[' ', 'b']]
`getStartToken` located the row/field start with `nextNonSpaceToken`, which
skips leading whitespace, so that whitespace never reached the column
parser. RFC 4180 section 2.4 treats spaces as part of a field, and `trim`
defaults to `false`, so the leading whitespace is field content.
Use `nextCharacterToken` for the row start so leading whitespace flows into
the column parser. It differs from `nextNonSpaceToken` only when the next
character is whitespace, and `ColumnParser.parse` still uses
`nextNonSpaceToken` for quote detection, so whitespace before a quoted
field is unaffected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A whitespace-only field at the start of a row is dropped or truncated, even though the identical field in any other column is preserved (default options,
trim: false):058B...\t \t \t...(the existing tab-delimiter test) shows whitespace-only fields are expected to be preserved — it just only holds for non-first columns.RFC 4180 section 2.4: "Spaces are considered part of a field and should not be ignored." With
trimdefaulting tofalse, dropping this whitespace is data loss / a wrong row count.Cause
RowParser.getStartTokenlocates the row/field start withscanner.nextNonSpaceToken, which skips leading whitespace, so for" \n"it jumps to the row delimiter (empty row) and for" ,b"it jumps to the,(pushing an empty first column). The leading whitespace — the actual unquoted field content — never reaches the column parser.Fix
Use
scanner.nextCharacterTokenfor the row start, so leading whitespace flows intoNonQuotedColumnParseras field content. The two getters differ only when the next character is whitespace, so any non-whitespace row start is byte-identical. Whitespace before a quoted field is unaffected becauseColumnParser.parsestill usesnextNonSpaceTokenfor quote detection ("x",ystill parses as["x", "y"]).Verification
RowParser.spec.tscases for a leading whitespace-only field (' ,b\n'→[' ', 'b'],' \n'→[' ']). They fail onmainand pass with the fix.@fast-csv/parsesuite green (374 tests).Note (out of scope, flagging for awareness):
RowParser.isEmptyRowusesreplace(/\s+/g, ''), soignoreEmpty: truewould also discard a legitimate whitespace-only row. Left untouched to keep this PR to one fix.