Python 3.12 support

[bluetech]

This is currently blocked on sissaschool/xmlschema#342.

Blocked on python 3.12-beta.2 release in https://github.com/actions/python-versions/blob/main/versions-manifest.json

[nicoddemus]

Probably good to leave a comment here explaining why we are skipping macos-py38.

[bluetech]

I noticed macos-py311 was missing, so I added py312 and removed py38 to keep the job count the same. With the faster Python release cycle, it is starting to become a lot.

[bluetech]

The xmlschema problem is fixed. Now there is a test failure on windows. I don't have windows but seems like a simple issue, will try to fix it.

______________ TestWINLocalPath.test_owner_group_not_implemented ______________

self = <test_local.TestWINLocalPath object at 0x0000026A748AE450>
path1 = local('C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-unknown\\pytest-0\\path1')

    def test_owner_group_not_implemented(self, path1):
        with pytest.raises(NotImplementedError):
>           path1.stat().owner

testing\_py\test_local.py:1222: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <_pytest._py.path.Stat object at 0x0000026A74A25C40>, name = 'owner'

    def __getattr__(self, name: str) -> Any:
>       return getattr(self._osstatresult, "st_" + name)
E       AttributeError: 'os.stat_result' object has no attribute 'st_owner'

.tox\py312\Lib\site-packages\_pytest\_py\path.py:216: AttributeError
=========================== short test summary info ===========================
FAILED testing/_py/test_local.py::TestWINLocalPath::test_owner_group_not_implemented - AttributeError: 'os.stat_result' object has no attribute 'st_owner'

Job: https://github.com/pytest-dev/pytest/actions/runs/4704035427/jobs/8343245002

[bluetech]

The failure doesn't make sense to me.

Relevant code in Stat:

class Stat:
    ...

    def __getattr__(self, name: str) -> Any:
        return getattr(self._osstatresult, "st_" + name)

    def __init__(self, path, osstatresult):
        self.path = path
        self._osstatresult = osstatresult

    @property
    def owner(self):
        if iswin32:
            raise NotImplementedError("XXX win32")
        import pwd

        entry = error.checked_call(pwd.getpwuid, self.uid)  # type:ignore[attr-defined]
        return entry[0]
    ...

The failing test (py.path.local tests):

    def test_owner_group_not_implemented(self, path1):
        with pytest.raises(NotImplementedError):
            path1.stat().owner

fails as seen in previous comment.

How come the __getattr__ is reached for owner attribute, when there is an owner property? It only happens on Windows so I'm assuming it has to do with the raise NotImplementedError("XXX win32") code in the property (but the property isn't present in the stack trace...).

I reran the job and it happened again so it's reproducible.

[nicoddemus]

You are right, it does not make much sense. I debugged it, and here is what happens in 3.12:

1. It reaches owner, and raises NotImplementedError as expected;
2. At this points, somehow __getattr__ gets called.

In Python 3.10 (and I supposed 3.11 but I did not test), 2) never happens -- NotImplementedError is raised from the property, and __getattr__ does not get called.

Seems like a bug in CPython, I will post an issue upstream.

EDIT: posted upstream: python/cpython#103551

[nicoddemus]

This has been fixed already and will be available on the next release. 👍

[bluetech]

• Renamed this PR to just "Python 3.12 support"
• Added Python 3.12 classifier
• Added setuptools testing dependency, needed for test which uses pkg_resources
• Brought Avoid ast deprecation warnings on Python 3.12 #10981 into this PR

[bluetech]

Hmm some autocommit hook deletes the classifier so I'll drop it for now.

[RonnyPfannschmidt]

@bluetech you can probably up the max version in the setup.cfg firmat hook config

[hugovk]

Yes, it's this: https://github.com/asottile/setup-cfg-fmt#adds-python-version-classifiers

diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index af6cd262..fbd5d39c 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -50,7 +50,7 @@ repos:
     rev: v2.2.0
     hooks:
     -   id: setup-cfg-fmt
-        args: ["--max-py-version=3.11", "--include-version-classifiers"]
+        args: ["--max-py-version=3.12", "--include-version-classifiers"]
 -   repo: https://github.com/pre-commit/pygrep-hooks
     rev: v1.10.0
     hooks:

[bluetech]

Thanks, missed that.

[bluetech]

github updated to Python 3.12 beta, I reran the job and now there is a new deprecation (showing one, rest are the same):

 _________________ ERROR collecting testing/python/metafunc.py __________________
testing/python/metafunc.py:16: in <module>
    import hypothesis
<frozen importlib._bootstrap>:1266: in _find_and_load
    ???
<frozen importlib._bootstrap>:1237: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:841: in _load_unlocked
    ???
.tox/py312-coverage/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:169: in exec_module
    source_stat, co = _rewrite_test(fn, self.config)
.tox/py312-coverage/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:352: in _rewrite_test
    rewrite_asserts(tree, source, strfn, config)
.tox/py312-coverage/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:413: in rewrite_asserts
    AssertionRewriter(module_path, config, source).run(mod)
.tox/py312-coverage/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:691: in run
    doc = item.value.s
/opt/hostedtoolcache/Python/3.12.0-beta.1/x64/lib/python3.12/ast.py:532: in _s_getter
    warnings._deprecated(
/opt/hostedtoolcache/Python/3.12.0-beta.1/x64/lib/python3.12/warnings.py:529: in _deprecated
    warn(msg, DeprecationWarning, stacklevel=3)
E   DeprecationWarning: Attribute s is deprecated and will be removed in Python 3.14; use value instead
=========================== short test summary info ============================
ERROR testing/test_error_diffs.py - DeprecationWarning: Attribute s is deprecated and will be removed in Python...
ERROR testing/test_meta.py - DeprecationWarning: Attribute s is deprecated and will be removed in Python...
ERROR testing/test_runner_xunit.py - DeprecationWarning: Attribute s is deprecated and will be removed in Python...
ERROR testing/test_terminal.py - DeprecationWarning: Attribute s is deprecated and will be removed in Python...
ERROR testing/python/metafunc.py - DeprecationWarning: Attribute s is deprecated and will be removed in Python...

Fixing.

[bluetech]

Added this to the ast deprecations commit:

diff --git a/src/_pytest/assertion/rewrite.py b/src/_pytest/assertion/rewrite.py
index 2f9038ee1..00bb4feb9 100644
--- a/src/_pytest/assertion/rewrite.py
+++ b/src/_pytest/assertion/rewrite.py
@@ -688,7 +688,10 @@ class AssertionRewriter(ast.NodeVisitor):
                 and isinstance(item, ast.Expr)
                 and isinstance(item.value, astStr)
             ):
-                doc = item.value.s
+                if sys.version_info >= (3, 8):
+                    doc = item.value.value
+                else:
+                    doc = item.value.s
                 if self.is_rewrite_disabled(doc):
                     return
                 expect_docstring = False

Some more errors now, fixing these too:

Details

=================================== FAILURES ===================================
_______________ test_get_assertion_exprs[backslash continuation] _______________
[gw0] darwin -- Python 3.12.0 /Users/runner/work/pytest/pytest/.tox/py312-xdist/bin/python

src = b'def x():\n    assert 1 + \\\n        2\n'
expected = {2: '1 + \\\n        2'}

    @pytest.mark.parametrize(
        ("src", "expected"),
        (
            # fmt: off
            pytest.param(b"", {}, id="trivial"),
            pytest.param(
                b"def x(): assert 1\n",
                {1: "1"},
                id="assert statement not on own line",
            ),
            pytest.param(
                b"def x():\n"
                b"    assert 1\n"
                b"    assert 1+2\n",
                {2: "1", 3: "1+2"},
                id="multiple assertions",
            ),
            pytest.param(
                # changes in encoding cause the byte offsets to be different
                "# -*- coding: latin1\n"
                "def ÀÀÀÀÀ(): assert 1\n".encode("latin1"),
                {2: "1"},
                id="latin1 encoded on first line\n",
            ),
            pytest.param(
                # using the default utf-8 encoding
                "def ÀÀÀÀÀ(): assert 1\n".encode(),
                {1: "1"},
                id="utf-8 encoded on first line",
            ),
            pytest.param(
                b"def x():\n"
                b"    assert (\n"
                b"        1 + 2  # comment\n"
                b"    )\n",
                {2: "(\n        1 + 2  # comment\n    )"},
                id="multi-line assertion",
            ),
            pytest.param(
                b"def x():\n"
                b"    assert y == [\n"
                b"        1, 2, 3\n"
                b"    ]\n",
                {2: "y == [\n        1, 2, 3\n    ]"},
                id="multi line assert with list continuation",
            ),
            pytest.param(
                b"def x():\n"
                b"    assert 1 + \\\n"
                b"        2\n",
                {2: "1 + \\\n        2"},
                id="backslash continuation",
            ),
            pytest.param(
                b"def x():\n"
                b"    assert x, y\n",
                {2: "x"},
                id="assertion with message",
            ),
            pytest.param(
                b"def x():\n"
                b"    assert (\n"
                b"        f(1, 2, 3)\n"
                b"    ),  'f did not work!'\n",
                {2: "(\n        f(1, 2, 3)\n    )"},
                id="assertion with message, test spanning multiple lines",
            ),
            pytest.param(
                b"def x():\n"
                b"    assert \\\n"
                b"        x\\\n"
                b"        , 'failure message'\n",
                {2: "x"},
                id="escaped newlines plus message",
            ),
            pytest.param(
                b"def x(): assert 5",
                {1: "5"},
                id="no newline at end of file",
            ),
            # fmt: on
        ),
    )
    def test_get_assertion_exprs(src, expected) -> None:
>       assert _get_assertion_exprs(src) == expected
E       AssertionError: assert {2: '1 + \\\n...\\n        2'} == {2: '1 + \\\n        2'}
E         Differing items:
E         {2: '1 + \\\n    assert 1 + \\\n        2'} != {2: '1 + \\\n        2'}
E         Use -v to get more diff

testing/test_assertrewrite.py:1810: AssertionError
___________ test_get_assertion_exprs[escaped newlines plus message] ____________
[gw0] darwin -- Python 3.12.0 /Users/runner/work/pytest/pytest/.tox/py312-xdist/bin/python

src = b"def x():\n    assert \\\n        x\\\n        , 'failure message'\n"
expected = {2: 'x'}

    @pytest.mark.parametrize(
        ("src", "expected"),
        (
            # fmt: off
            pytest.param(b"", {}, id="trivial"),
            pytest.param(
                b"def x(): assert 1\n",
                {1: "1"},
                id="assert statement not on own line",
            ),
            pytest.param(
                b"def x():\n"
                b"    assert 1\n"
                b"    assert 1+2\n",
                {2: "1", 3: "1+2"},
                id="multiple assertions",
            ),
            pytest.param(
                # changes in encoding cause the byte offsets to be different
                "# -*- coding: latin1\n"
                "def ÀÀÀÀÀ(): assert 1\n".encode("latin1"),
                {2: "1"},
                id="latin1 encoded on first line\n",
            ),
            pytest.param(
                # using the default utf-8 encoding
                "def ÀÀÀÀÀ(): assert 1\n".encode(),
                {1: "1"},
                id="utf-8 encoded on first line",
            ),
            pytest.param(
                b"def x():\n"
                b"    assert (\n"
                b"        1 + 2  # comment\n"
                b"    )\n",
                {2: "(\n        1 + 2  # comment\n    )"},
                id="multi-line assertion",
            ),
            pytest.param(
                b"def x():\n"
                b"    assert y == [\n"
                b"        1, 2, 3\n"
                b"    ]\n",
                {2: "y == [\n        1, 2, 3\n    ]"},
                id="multi line assert with list continuation",
            ),
            pytest.param(
                b"def x():\n"
                b"    assert 1 + \\\n"
                b"        2\n",
                {2: "1 + \\\n        2"},
                id="backslash continuation",
            ),
            pytest.param(
                b"def x():\n"
                b"    assert x, y\n",
                {2: "x"},
                id="assertion with message",
            ),
            pytest.param(
                b"def x():\n"
                b"    assert (\n"
                b"        f(1, 2, 3)\n"
                b"    ),  'f did not work!'\n",
                {2: "(\n        f(1, 2, 3)\n    )"},
                id="assertion with message, test spanning multiple lines",
            ),
            pytest.param(
                b"def x():\n"
                b"    assert \\\n"
                b"        x\\\n"
                b"        , 'failure message'\n",
                {2: "x"},
                id="escaped newlines plus message",
            ),
            pytest.param(
                b"def x(): assert 5",
                {1: "5"},
                id="no newline at end of file",
            ),
            # fmt: on
        ),
    )
    def test_get_assertion_exprs(src, expected) -> None:
>       assert _get_assertion_exprs(src) == expected
E       AssertionError: assert {2: 'rt \\\n ...\\\n    asse'} == {2: 'x'}
E         Differing items:
E         {2: 'rt \\\n        x\\\n    asse'} != {2: 'x'}
E         Use -v to get more diff

testing/test_assertrewrite.py:1810: AssertionError
_______ TestConftestCustomization.test_issue2369_collect_module_fileext ________
[gw0] darwin -- Python 3.12.0 /Users/runner/work/pytest/pytest/.tox/py312-xdist/bin/python

self = <collect.TestConftestCustomization object at 0x10c437d70>
pytester = <Pytester PosixPath('/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pytest-of-runner/pytest-0/popen-gw0/test_issue2369_collect_module_fileext0')>

    def test_issue2369_collect_module_fileext(self, pytester: Pytester) -> None:
        """Ensure we can collect files with weird file extensions as Python
        modules (#2369)"""
        # We'll implement a little finder and loader to import files containing
        # Python source code whose file extension is ".narf".
        pytester.makeconftest(
            """
            import sys, os, imp
            from _pytest.python import Module
    
            class Loader(object):
                def load_module(self, name):
                    return imp.load_source(name, name + ".narf")
            class Finder(object):
                def find_module(self, name, path=None):
                    if os.path.exists(name + ".narf"):
                        return Loader()
            sys.meta_path.append(Finder())
    
            def pytest_collect_file(file_path, parent):
                if file_path.suffix == ".narf":
                    return Module.from_parent(path=file_path, parent=parent)"""
        )
        pytester.makefile(
            ".narf",
            """\
            def test_something():
                assert 1 + 1 == 2""",
        )
        # Use runpytest_subprocess, since we're futzing with sys.meta_path.
        result = pytester.runpytest_subprocess()
>       result.stdout.fnmatch_lines(["*1 passed*"])
E       Failed: remains unmatched: '*1 passed*'

/Users/runner/work/pytest/pytest/testing/python/collect.py:928: Failed
----------------------------- Captured stdout call -----------------------------
running: /Users/runner/work/pytest/pytest/.tox/py312-xdist/bin/python -mpytest --basetemp=/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pytest-of-runner/pytest-0/popen-gw0/test_issue2369_collect_module_fileext0/runpytest-0
     in: /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pytest-of-runner/pytest-0/popen-gw0/test_issue2369_collect_module_fileext0
----------------------------- Captured stderr call -----------------------------
ImportError while loading conftest '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pytest-of-runner/pytest-0/popen-gw0/test_issue2369_collect_module_fileext0/conftest.py'.
conftest.py:1: in <module>
    import sys, os, imp
E   ModuleNotFoundError: No module named 'imp'
=========================== short test summary info ============================
FAILED testing/test_assertrewrite.py::test_get_assertion_exprs[backslash continuation] - AssertionError: assert {2: '1 + \\\n...\\n        2'} == {2: '1 + \\\n     ...
FAILED testing/test_assertrewrite.py::test_get_assertion_exprs[escaped newlines plus message] - AssertionError: assert {2: 'rt \\\n ...\\\n    asse'} == {2: 'x'}
FAILED testing/python/collect.py::TestConftestCustomization::test_issue2369_collect_module_fileext - Failed: remains unmatched: '*1 passed*'
===== 3 failed, 3298 passed, 128 skipped, 12 xfailed in 114.77s (0:01:54) ======

[bluetech]

Pushed a fix for imp module removal. Remaining two failures are in assertion rewriting:

FAILED testing/test_assertrewrite.py::test_get_assertion_exprs[backslash continuation] - AssertionError: assert {2: '1 + \\\n    assert 1 + \\\n        2'} == {2: '1 + \\\n        2'}
FAILED testing/test_assertrewrite.py::test_get_assertion_exprs[escaped newlines plus message] - AssertionError: assert {2: 'rt \\\n        x\\\n    asse'} == {2: 'x'}

They stem from what seems like tokenization changes around line continuations. Given the following code

def x():
    assert 1 + \
        2

tokenizes as follows in Python 3.11:

 TokenInfo((ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line=''),
 TokenInfo((NAME), string='def', start=(1, 0), end=(1, 3), line='def x():\n'),
 TokenInfo((NAME), string='x', start=(1, 4), end=(1, 5), line='def x():\n'),
 TokenInfo((OP), string='(', start=(1, 5), end=(1, 6), line='def x():\n'),
 TokenInfo((OP), string=')', start=(1, 6), end=(1, 7), line='def x():\n'),
 TokenInfo((OP), string=':', start=(1, 7), end=(1, 8), line='def x():\n'),
 TokenInfo((NEWLINE), string='\n', start=(1, 8), end=(1, 9), line='def x():\n'),
 TokenInfo((INDENT), string='    ', start=(2, 0), end=(2, 4), line='    assert 1 + \\\n'),
 TokenInfo((NAME), string='assert', start=(2, 4), end=(2, 10), line='    assert 1 + \\\n'),
 TokenInfo((NUMBER), string='1', start=(2, 11), end=(2, 12), line='    assert 1 + \\\n'),
 TokenInfo((OP), string='+', start=(2, 13), end=(2, 14), line='    assert 1 + \\\n'),
 TokenInfo((NUMBER), string='2', start=(3, 8), end=(3, 9), line='        2\n'),
 TokenInfo((NEWLINE), string='\n', start=(3, 9), end=(3, 10), line='        2\n'),
 TokenInfo((DEDENT), string='', start=(4, 0), end=(4, 0), line=''),
 TokenInfo((ENDMARKER), string='', start=(4, 0), end=(4, 0), line='')

and as follows in Python 3.12:

 TokenInfo((ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line=''),
 TokenInfo((NAME), string='def', start=(1, 0), end=(1, 3), line='def x():\n'),
 TokenInfo((NAME), string='x', start=(1, 4), end=(1, 5), line='def x():\n'),
 TokenInfo((OP), string='(', start=(1, 5), end=(1, 6), line='def x():\n'),
 TokenInfo((OP), string=')', start=(1, 6), end=(1, 7), line='def x():\n'),
 TokenInfo((OP), string=':', start=(1, 7), end=(1, 8), line='def x():\n'),
 TokenInfo((NEWLINE), string='\n', start=(1, 8), end=(1, 9), line='def x():\n'),
 TokenInfo((INDENT), string='    ', start=(2, 0), end=(2, 4), line='    assert 1 + \\\n'),
 TokenInfo((NAME), string='assert', start=(2, 4), end=(2, 10), line='    assert 1 + \\\n'),
 TokenInfo((NUMBER), string='1', start=(2, 11), end=(2, 12), line='    assert 1 + \\\n'),
 TokenInfo((OP), string='+', start=(2, 13), end=(2, 14), line='    assert 1 + \\\n'),
 TokenInfo((NUMBER), string='2', start=(3, 8), end=(3, 9), line='    assert 1 + \\\n        2\n'),
 TokenInfo((NEWLINE), string='\n', start=(3, 9), end=(3, 10), line='    assert 1 + \\\n        2\n'),
 TokenInfo((DEDENT), string='', start=(3, 10), end=(3, 10), line='    assert 1 + \\\n        2\n'),
 TokenInfo((ENDMARKER), string='', start=(4, 0), end=(4, 0), line='')

Will look at it some other time (unless someone beats me to it, which would be great).

[bluetech]

cc @asottile on previous comment, you know tokenization better than anyone, maybe you have an idea :)

[asottile]

yeah that's all to be expected. the DEDENT might change back to be less disruptive in python/cpython#104976

[bluetech]

Thanks, I see the PR is merged now, so let's wait for the next beta.

[hugovk]

Thanks, I see the PR is merged now, so let's wait for the next beta.

3.12.0b2 is now out:

• https://www.python.org/downloads/release/python-3120b2/

[bluetech]

Even though 3.12-beta.2 appears in https://github.com/actions/python-versions/releases, I can't get it to be picked up by the setup-python action. I tried setting check-latest: true but this doesn't work either. I'm assuming it needs more time. If anyone knows how to speed it up, let me know.

I see now that it needs to appear in https://github.com/actions/python-versions/blob/main/versions-manifest.json not in the releases.

[The-Compiler]

Yup, I believe actions/python-versions#232 is the thing to watch.

[hugovk]

actions/python-versions#232 has been merged and beta 2 is now available (for example).

[bluetech]

OK this is good to go now, marked as ready for review.

[bluetech]

(I changed the branch protection checks already so this can be green)

[Pierre-Sassoulas]

Great work ! Just curious, I wonder if the python 3.7 specific code path will be removed in 2 weeks for python 3.7 eol ? Is there also a 'cleanup' of the code when removing support for an interpreter ?

[bluetech]

Great work ! Just curious, I wonder if the python 3.7 specific code path will be removed in 2 weeks for python 3.7 eol ? Is there also a 'cleanup' of the code when removing support for an interpreter ?

Yes, I believe our policy is that new pytest feature releases only support non-EOL python versions. Then we also clean up all of the compat code for that version.