Use custom handling when reading text files

[cbersch]

I would like to use my own handling of line endings when reading text files. I don't feel comfortable fiddling around with the eol settings in the .gitattributes on existing verified files, especially because it possibly involves every developer to reset the work directory, also on agents, and might get problems again, when switching between branches etc. (Yes, I found related discussions #1435 and #1786)

Is there any existing possibility, that I can configure, how text files are read in? Looking at https://github.com/VerifyTests/Verify/blob/main/src/Verify/Compare/Comparer.cs#L14 I don't see an option.

[SimonCropp]

no, there's no hook for that currently, and i'm hesitant to add a general one.

the issue is that the received file isn't just a diff artifact. accepting a snapshot is File.Move(received, verified) — the received bytes literally become the new verified file. so any transform applied only on the read side breaks the round trip.

concretely, if verify normalized \r\n on read: your tests would pass, but the first time you accept a changed snapshot, that file flips to lf. repeat over a few months and you end up with a repo that's half crlf and half lf, with whole-file diffs on the ones that flipped. that's worse than the current hard failure, and it's the opposite of what you're asking for. the same asymmetry shows up immediately in diff tools — verified stays crlf, received is written lf, so every line reads as changed depending on how the tool is configured.

on the .gitattributes migration, i think it's less invasive than you're expecting. it's one commit: add the text eol=lf entries, then git add --renormalize .. devs pick it up on the next pull and checkout rewrites their working tree — no reset required. the one real gotcha is build agents, and it's the same one hit in #1786: cached repo checkouts need clearing once. after that it stays fixed.

[cbersch]

no, there's no hook for that currently, and i'm hesitant to add a general one.

the issue is that the received file isn't just a diff artifact. accepting a snapshot is File.Move(received, verified) — the received bytes literally become the new verified file. so any transform applied only on the read side breaks the round trip.

concretely, if verify normalized \r\n on read: your tests would pass, but the first time you accept a changed snapshot, that file flips to lf. repeat over a few months and you end up with a repo that's half crlf and half lf, with whole-file diffs on the ones that flipped. that's worse than the current hard failure, and it's the opposite of what you're asking for. the same asymmetry shows up immediately in diff tools — verified stays crlf, received is written lf, so every line reads as changed depending on how the tool is configured.

I thought, that it's not that easy. Thanks for clarifying.
In my case it's the core.autocrlf which breaks the round trip as well. Yes, thankfully, that's also documented.

on the .gitattributes migration, i think it's less invasive than you're expecting. it's one commit: add the text eol=lf entries, then git add --renormalize .. devs pick it up on the next pull and checkout rewrites their working tree — no reset required. the one real gotcha is build agents, and it's the same one hit in #1786: cached repo checkouts need clearing once. after that it stays fixed.

Stupid question: What is the different between a dev pulling the updates vs. a build agent doing so?

And I'm afraid that the build agent would run into problems when switching between newer branches with have the new .gitattributes and older ones.

I'll probably rename the folders with the current verified changes and rename them at a later point. Even if that's clumsy.

[SimonCropp]

What is the difference between a dev pulling the updates vs. a build agent doing so?

not much at the git level. the real difference is fresh clone vs reused workspace. git only re-applies eol filters when a file's blob changes — if the blob is identical it leaves the working tree file alone, stale endings and all. a fresh clone has no stale working tree so it's always correct; an agent reusing a cached workspace can hold crlf files indefinitely.

which means i need to correct what i said above, because autocrlf changes the picture. with core.autocrlf=true the blobs in your repo are already lf — autocrlf normalized them on the way in. so git add --renormalize . finds nothing to change, and since no blob changes, nothing forces your working tree to update either. you do need the one-time refresh: git rm --cached -r . then git reset --hard. that discards uncommitted work, so land the .gitattributes commit first. my "no reset required" was the generic case, not yours — your original instinct was right.

And I'm afraid that the build agent would run into problems when switching between newer branches with have the new .gitattributes and older ones.

mostly fine, for the same reason: the blobs don't change, so switching branches doesn't rewrite those files. a verified file that genuinely differs on an old branch will come back crlf and fail there — but that's the state old branches are already in. the .gitattributes commit doesn't make them worse.

on renaming the folders: it does work, in that delete+add forces renormalization on commit. but it costs blame and a large diff to reach the same place those two commands get you.

[cbersch]

on renaming the folders: it does work, in that delete+add forces renormalization on commit. but it costs blame and a large diff to reach the same place those two commands get you.

Thanks so much for the detailed explanation.

I'll probably still go with this clumsy solution, too many other devs/devops/non-devs/machines involved :)

[SimonCropp]

FYI https://github.com/VerifyTests/Verify/blob/main/docs/newline-tolerance.md

[cbersch]

@SimonCropp Thank you so much for that!