Compatibility issues with Python 3.15

[frenzymadness]

I've spent some time trying to make the python-astroid RPM package build in Fedora Linux with Python 3.15, and I've found some incompatibilities there. We don't run all the tests in the build process, so the list is very likely incomplete. The codebase is complex, and I don't understand all the details, but some of the bugs mentioned below seem not to be related to Python 3.15 but likely surfaced because other changes or other (un)related failures happened.

First failure

______________ TypingBrain.test_typing_object_notsubscriptable_3 _______________

args = (<Attribute.ByteString l.3 at 0x7f1824b51d10>,)
kwargs = {'context': <astroid.context.InferenceContext object at 0x7f1823ef4a00>}
generator = <generator object Attribute._infer at 0x7f1823d2f890>

    def inner(*args: _P.args, **kwargs: _P.kwargs) -> Generator[InferenceResult]:
        generator = func(*args, **kwargs)
        try:
>           yield next(generator)
                  ^^^^^^^^^^^^^^^
E           StopIteration

astroid/decorators.py:81: StopIteration

The above exception was the direct cause of the following exception:

self = <tests.brain.test_brain.TypingBrain testMethod=test_typing_object_notsubscriptable_3>

    def test_typing_object_notsubscriptable_3(self):
        """The ByteString class of the typing module is not
        subscriptable (whereas it is in the collections' module)"""
        right_node = builder.extract_node("""
        import typing
        typing.ByteString
        """)
>       inferred = next(right_node.infer())
                   ^^^^^^^^^^^^^^^^^^^^^^^^

tests/brain/test_brain.py:789: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
astroid/nodes/node_ng.py:166: in infer
    for i, result in enumerate(self._infer(context=context, **kwargs)):
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

args = (<Attribute.ByteString l.3 at 0x7f1824b51d10>,)
kwargs = {'context': <astroid.context.InferenceContext object at 0x7f1823ef4a00>}
generator = <generator object Attribute._infer at 0x7f1823d2f890>

    def inner(*args: _P.args, **kwargs: _P.kwargs) -> Generator[InferenceResult]:
        generator = func(*args, **kwargs)
        try:
            yield next(generator)
        except StopIteration as error:
            # generator is empty
            if error.args:
                raise InferenceError(**error.args[0]) from error
>           raise InferenceError(
                "StopIteration raised without any error information."
            ) from error
E           astroid.exceptions.InferenceError: StopIteration raised without any error information.

astroid/decorators.py:86: InferenceError

I don't know much about this, but it seems to be related to PEP 479, which is kinda old. It seems that the problem is not strictly in PEP 479, raising RuntimeError, but other issues there that caused this to surface.

Workaround for this is:

diff --git a/astroid/decorators.py b/astroid/decorators.py
index 05d2dd3..7c63b2f 100644
--- a/astroid/decorators.py
+++ b/astroid/decorators.py
@@ -61,12 +61,13 @@ def yes_if_nothing_inferred(
         generator = func(*args, **kwargs)
 
         try:
-            yield next(generator)
+            first_value = next(generator)
         except StopIteration:
             # generator is empty
             yield util.Uninferable
             return
 
+        yield first_value
         yield from generator
 
     return inner
@@ -78,7 +79,7 @@ def raise_if_nothing_inferred(
     def inner(*args: _P.args, **kwargs: _P.kwargs) -> Generator[InferenceResult]:
         generator = func(*args, **kwargs)
         try:
-            yield next(generator)
+            first_value = next(generator)
         except StopIteration as error:
             # generator is empty
             if error.args:
@@ -91,6 +92,7 @@ def raise_if_nothing_inferred(
                 f"RecursionError raised with limit {sys.getrecursionlimit()}."
             ) from error
 
+        yield first_value
         yield from generator
 
     return inner

Second failure

_______________ test_regression_parse_deeply_nested_parentheses ________________

    def test_regression_parse_deeply_nested_parentheses() -> None:
        """Regression test for issue #2643."""
        with pytest.raises(AstroidSyntaxError, match="Parsing Python code failed:") as ctx:
            extract_node(
                "A=((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((c,j=t"
            )
        expected = (
            SyntaxError if platform.python_implementation() == "PyPy" else MemoryError
        )
>       assert isinstance(ctx.value.error, expected)
E       assert False
E        +  where False = isinstance(SyntaxError("'(' was never closed", ('<unknown>', 1, 202, 'A=((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((c,j=t\n', 1, 0)), <class 'MemoryError'>)
E        +    where SyntaxError("'(' was never closed", ('<unknown>', 1, 202, 'A=((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((c,j=t\n', 1, 0)) = AstroidSyntaxError('Parsing Python code failed:\n{error}').error
E        +      where AstroidSyntaxError('Parsing Python code failed:\n{error}') = <ExceptionInfo AstroidSyntaxError('Parsing Python code failed:\n{error}') tblen=5>.value

tests/test_regrtest.py:541: AssertionError

Python 3.15 seems to be raising SyntaxError for deeply nested parentheses. I've tested this with 1000 nested levels, and all Pythons versions newer than 3.8 raised SyntaxError and only 3.8 and older raised MemoryError. So this test might be exercising some other limit.

The workaround here is easy:

diff --git a/tests/test_regrtest.py b/tests/test_regrtest.py
index a207057..e2b9916 100644
--- a/tests/test_regrtest.py
+++ b/tests/test_regrtest.py
@@ -535,7 +535,10 @@ def test_regression_parse_deeply_nested_parentheses() -> None:
         extract_node(
             "A=((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((c,j=t"
         )
+    # Python 3.15+ returns SyntaxError for deeply nested parentheses
+    # PyPy returns SyntaxError
+    # Python <3.15 CPython returns MemoryError
     expected = (
-        SyntaxError if platform.python_implementation() == "PyPy" else MemoryError
+        SyntaxError if (platform.python_implementation() == "PyPy" or sys.version_info >= (3, 15)) else MemoryError
     )
     assert isinstance(ctx.value.error, expected)

Exception groups are tuples

I'm not sure why we see this now – it seems that exceptions in exceptions groups have always been tuples and the following change works fine for 3.14 as well:

diff --git a/astroid/protocols.py b/astroid/protocols.py
index 6565688..c19c761 100644
--- a/astroid/protocols.py
+++ b/astroid/protocols.py
@@ -545,7 +545,7 @@ ExceptionGroup
 """)))
         assigned = objects.ExceptionInstance(eg)
         assigned.instance_attrs["exceptions"] = [
-            nodes.List.from_elements(_generate_assigned())
+            nodes.Tuple.from_elements(_generate_assigned())
         ]
         yield assigned
     else:

and

diff --git a/tests/test_group_exceptions.py b/tests/test_group_exceptions.py
index 9680664..fbdb020 100644
--- a/tests/test_group_exceptions.py
+++ b/tests/test_group_exceptions.py
@@ -126,12 +126,12 @@ def test_star_exceptions_infer_exceptions() -> None:
     assert isinstance(node, nodes.TryStar)
     inferred_ve = next(node.handlers[0].statement().name.infer())
     assert inferred_ve.name == "ExceptionGroup"
-    assert isinstance(inferred_ve.getattr("exceptions")[0], nodes.List)
+    assert isinstance(inferred_ve.getattr("exceptions")[0], nodes.Tuple)
     assert (
         inferred_ve.getattr("exceptions")[0].elts[0].pytype() == "builtins.ValueError"
     )
 
     inferred_te = next(node.handlers[1].statement().name.infer())
     assert inferred_te.name == "ExceptionGroup"
-    assert isinstance(inferred_te.getattr("exceptions")[0], nodes.List)
+    assert isinstance(inferred_te.getattr("exceptions")[0], nodes.Tuple)
     assert inferred_te.getattr("exceptions")[0].elts[0].pytype() == "builtins.TypeError"

ByteString is deprecated and no longer available at module-level

See: python/cpython@293b05c

Workaround:

diff --git a/astroid/brain/brain_typing.py b/astroid/brain/brain_typing.py
index e0eb15f..2e580e3 100644
--- a/astroid/brain/brain_typing.py
+++ b/astroid/brain/brain_typing.py
@@ -15,7 +15,7 @@ from typing import Final
 from astroid import context, nodes
 from astroid.brain.helpers import register_module_extender
 from astroid.builder import AstroidBuilder, _extract_single_node, extract_node
-from astroid.const import PY312_PLUS, PY313_PLUS, PY314_PLUS
+from astroid.const import PY312_PLUS, PY313_PLUS, PY314_PLUS, PY315_PLUS
 from astroid.exceptions import (
     AstroidSyntaxError,
     AttributeInferenceError,
@@ -464,6 +464,10 @@ def _typing_transform():
         @classmethod
         def __class_getitem__(cls, item): return cls
     """)
+    if PY315_PLUS:
+        code += textwrap.dedent("""
+    class ByteString: pass
+    """)
     return AstroidBuilder(AstroidManager()).string_build(code)
 
 
diff --git a/astroid/const.py b/astroid/const.py
index dcce074..e4e7978 100644
--- a/astroid/const.py
+++ b/astroid/const.py
@@ -10,6 +10,7 @@ PY312_PLUS = sys.version_info >= (3, 12)
 PY313 = sys.version_info[:2] == (3, 13)
 PY313_PLUS = sys.version_info >= (3, 13)
 PY314_PLUS = sys.version_info >= (3, 14)
+PY315_PLUS = sys.version_info >= (3, 15)
 
 WIN32 = sys.platform == "win32"
 

I'm going to build pylint on top of this, so I'll find issues there as well and possibly some transitive ones in astroid I haven't found yet.

[Pierre-Sassoulas]

Hey, thank you for looking into this. We generally don't bother with compatibility with alphas, because we don't have a lot of time and it's possible that something is going to be reverted later and that the work done will be for nothing. So it's expected that python 3.15 is not working atm. But if you propose to work on this we can definitely open a PR that we're going to merge when "3.15-dev" means the first beta version of python in github. (See here https://github.com/actions/python-versions/releases)

[frenzymadness]

Understood. We (in Fedora Linux) need to make thousands of packages ready for the next Python, so we have to start sooner than you; that's totally fine.

I've found one more issue while rebuilding pylint (which will get it's own issue soon):

Python 3.15+ raises ModuleNotFoundError instead of KeyError when NamespacePath encounters unimported parent packages, which requires this fix:

diff --git a/astroid/interpreter/_import/util.py b/astroid/interpreter/_import/util.py
index df1aca2..59cd955 100644
--- a/astroid/interpreter/_import/util.py
+++ b/astroid/interpreter/_import/util.py
@@ -72,7 +72,7 @@ def is_namespace(modname: str) -> bool:
                 # Workaround for "py" module
                 # https://github.com/pytest-dev/apipkg/issues/13
                 return False
-        except KeyError:
+        except (KeyError, ModuleNotFoundError):
             # Intermediate steps might raise KeyErrors
             # https://github.com/python/cpython/issues/93334
             # TODO: update if fixed in importlib
@@ -80,6 +80,8 @@ def is_namespace(modname: str) -> bool:
             # >>> from importlib.machinery import PathFinder
             # >>> PathFinder.find_spec('a.b', ['a'])
             # KeyError: 'a'
+            # Python 3.15+ raises ModuleNotFoundError instead of KeyError
+            # when NamespacePath encounters unimported parent packages.
 
             # Repair last_submodule_search_locations
             if last_submodule_search_locations:

With this last patch (and a couple more for pylint package), the build of both works fine, and the subset of their tests passes.

[DanielNoord]

Thanks for the investigation you have done here @frenzymadness!

I think we can help you out by merging some of the changes you are suggesting. I have created a PR that includes them, which also adds testing on Python 3.15 to our CI.
Considering the first beta is around the corner it seems as good as time as any to add it.
With the help of AI tools it seems a lot easier to do the incremental changes needed in case a beta does change behaviour.

To that end I have created #3033

[DanielNoord]

We just released astroid 4.2.0b2. We'll work on pylint next, but perhaps that can already help you @frenzymadness. You will need to patch pylint to work around its strict version requirements for astroid (for now).

[DanielNoord]

pylint-dev/pylint#10983 shows we indeed need to handle that KeyError differently.

Funnily enough it was actually raised by "one of our own" in python/cpython#93334 and fixed by them in python/cpython#98100 😄

@jacobtylerwalls Does the suggested patch look correct to you? Or should we change something else? And if so, can we add a test for this?

[jacobtylerwalls]

@jacobtylerwalls Does the suggested patch look correct to you? Or should we change something else? And if so, can we add a test for this?

Yep, it's good, when we finally drop Python 3.14 we can stop catching KeyError. We can adjust the comment. When I wrote "when eventually fixed in importlib" I was assuming they wouldn't raise an error at all, but that's not what happened.