Prevent remaster from updating fallback versioned expected outputs

[mchris86]

Adds tests for fallback detection

Type of Changes

	Type
✓	🐛 Bug fix

Description

Refs #10844

This PR builds on the discussion in #10950 and adapts the fix to the pytest-remaster branch.

When running functional tests on a newer Python version (e.g. 3.14) , pylint may fall back to an older versioned expected output file (e.g. .313.txt) if no exact match exists. These fallback files are valid comparison targets but should not be updated/deleted.

In the current implementation:

golden_master.check(
    lint_test.serialize_output(actual_output), test_file.expected_output,
)

if expected_output resolves to a fallback file, it may be updated/deleted during remaster.

This change detects fallback outputs and ensures they are only used for comparison.
For fallback cases, a compare-only check is done instead of calling golden_master.check().

Testing

Added tests for fallback detection logic

[mchris86]

One thought while working on this: it might be useful for GoldenMaster to expose a compare-only mode, to avoid duplicating comparison logic when remastering is not desired

[Pierre-Sassoulas]

Thank you for working on this.

One thought while working on this: it might be useful for GoldenMaster to expose a compare-only mode, to avoid duplicating comparison logic when remastering is not desired

That would be the --no-remaster option ?

[mchris86]

I think --no-remaster would disable updates for all files.

What i meant here, is when expected_output resolves to a fallback, we still need to compare against it, but not use it as a remaster target. Like skipping remaster for specific files.

This way, we could call something like golden_master.compare_only(...) if fallback is detected and golden_master.check(...) otherwise

[Pierre-Sassoulas]

when expected_output resolves to a fallback,

I don't understand what it means, do you mind giving an example ?

[mchris86]

For example, suppose we have:

• test.py
• test.313.txt

and we run the tests on Python 3.14.

Since there is no test.314.txt, expected_output will become test.313.txt.

In this case, test.313.txt is a fallback. It is used to compare actual with expected, but it was generated for Python 3.13, so we don't want to overwrite it when running on 3.14.

[Pierre-Sassoulas]

Ha thank you for explaining. Yes it definitely need an evolution in pytest-remaster. Maybe an output name can take a callable (and in pylint we'd give a function that return a string based on the interpreter) ? What do you think ?

[mchris86]

The problem is that even if we pass a callable that returns a string, pytest-remaster still won't know if that path should be updated or not.

That is why I am thinking it may be simpler to keep on the pylint side: if the output is a fallback (eg. .131.txt while running 3.14), call something like compare_only(), otherwise use check() to update.

I pushed a small implementation of this in this PR if that helps.

[Pierre-Sassoulas]

Thanks for the discussion, I did some design in pytest-remaster to address this concern. I'd love your opinion on the "dimensions" parameter:

  golden_master.check(                                                                                                                                                                                                     
      lint_test.serialize_output(actual_output),
      test_file.expected_output_base,     # test.txt (generic)                                                                                                                                                             
      dimensions={                                          
          "version": f"{sys.version_info[0]}{sys.version_info[1]}",
          "platform": sys.platform,                                                                                                                                                                                        
      },
  )                                                                                                                                                                                                                        

For version="314" on Linux, pytest-remaster generates and checks this chain:

test.314.linux.txt → test.314.txt → test.linux.txt → test.txt

If the new file is identical to a less specific one, it's automatically removed as redundant. It could also work with cpython/pypy or other specificity I possibly don't remember (python interpreter, py-version parameter, operating system and python implementation are already pretty covering I think). See feature branch: Pierre-Sassoulas/pytest-remaster#1

There's a pylint specific example with a cursed checker that has an output that differ for each python interpreters / platform : https://github.com/Pierre-Sassoulas/pytest-remaster/tree/pylint-functional-tests/tests/demo_pylint

This is still a draft. Does this design work for the use case you had in mind, or do you see issues we should address? We can adjust the API if needed.

[mchris86]

Thanks for sharing this! I took some time go through the PR and the demo and I think the new design is really strong.

One thing I wanted to ask:

With:

golden_master.check(
    lint_test.serialize_output(actual_output),
    test_file.expected_output_base,  # test.txt (generic)
    dimensions={
        "version": f"{sys.version_info[0]}{sys.version_info[1]}",
        "platform": sys.platform,
    },
)

it looks like remaster will create the most specific file for the current run, for example: test.313.linux.txt

How would the less specific files be generated, like: test.txt, or test.313.txt?

Will new tests only contain this kind of expected files (test.313.linux.txt), and no generic base or less specific files?

[Pierre-Sassoulas]

Right, for existing tests (the pylint migration case), the base file (test.txt) already exists. Dimensions add version/platform-specific overrides on top only when the output diverges. If an override matches a less-specific file, _dedup_chain removes it automatically. For brand new tests you're right that remaster currently creates the most-specific file (e.g. test.314.linux.txt) when it shoud create test.txt imo. I missed this use case because it doesn't happen very often, but we should implement it. What do you think?

[mchris86]

I think a simple approach could be:

• If no expected files exist (for a new test), create test.txt.
• After that, golden_master.check() only creates more specific files if the output is different from what is already covered by an existing file.

For example:
test.linux.txt is only created if it differs from test.txt
test.314.txt is only created if it differs from test.linux.txt
test.314.linux.txt is only created if it differs from test.314.txt

This way, we can keep the number of files minimal and add more only when needed.

If we don't want to change current design, just creating the base test.txt for new tests would work too

[Pierre-Sassoulas]

I released 0.0.5, we can refine it when we realize what's blocking.

[mchris86]

Good, I will integrate it with pylint and see how it works