8.0.0 reverses order of parametrize when calling individual tests

[okken]

Summary
Marking a test with @pytest.mark.parametrize('i', [1, 2, 3, 4]), I expect the test to run 1, 2, 3, 4.
As a file, pytest does this as expected.
Running individual tests, though, results in 4, 3, 2, 1.
This is surprising behavior and new to 8.0.0.

(foo) $ pytest test_param.py::test_param -v
...
platform win32 -- Python 3.11.7, pytest-8.0.0, ...
...
test_param.py::test_param[4] PASSED                                      [ 25%]
test_param.py::test_param[3] PASSED                                      [ 50%]
test_param.py::test_param[2] PASSED                                      [ 75%]
test_param.py::test_param[1] PASSED                                      [100%]
...

Version 7.4.4 runs the test cases in the expected 1, 2, 3, 4 order.

(foo) $ pytest test_param.py::test_param -v
...
platform win32 -- Python 3.11.7, pytest-7.4.4, ...
...
test_param.py::test_param[1] PASSED                                      [ 25%]
test_param.py::test_param[2] PASSED                                      [ 50%]
test_param.py::test_param[3] PASSED                                      [ 75%]
test_param.py::test_param[4] PASSED                                      [100%]
...

Detail

• a detailed description of the bug or problem you are having
  When running individual parametrized tests, the order is reversed.
  This is new in 8.0.0, and not the case in 7.4.4
  Since 8.0.0 doesn't guarantee backward compatibility, I'm not sure if this is a bug or a feature. :)
  But I didn't see it in the notes, so I'm guessing this is an unintended change, and it is an unexpected, surprising behavior.

• output of pip list from the virtual environment you are using

$ pip list
Package    Version
---------- -------
colorama   0.4.6
iniconfig  2.0.0
packaging  23.2
pip        23.3.1
pluggy     1.3.0
pytest     8.0.0
setuptools 65.5.0

• pytest and operating system versions

platform win32 -- Python 3.11.7, pytest-8.0.0, pluggy-1.3.0

• minimal example if possible

$ cat test_param.py
import pytest

@pytest.mark.parametrize('i', [1, 2, 3, 4])
def test_param(i):
    ...

Output from 8.0.0, shows expected 1,2,3,4 order with file invocation, but reverse order with individual test, 4, 3, 2, 1

(foo) $ pytest test_param.py -v
============================= test session starts =============================
platform win32 -- Python 3.11.7, pytest-8.0.0, pluggy-1.3.0 -- C:\Users\okken\projects\foo\venv\Scripts\python.exe
cachedir: .pytest_cache
rootdir: C:\Users\okken\projects\foo
configfile: pytest.ini
collecting ... collected 4 items

test_param.py::test_param[1] PASSED                                      [ 25%]
test_param.py::test_param[2] PASSED                                      [ 50%]
test_param.py::test_param[3] PASSED                                      [ 75%]
test_param.py::test_param[4] PASSED                                      [100%]

============================== 4 passed in 0.01s ==============================
(foo) $ pytest test_param.py::test_param -v
============================= test session starts =============================
platform win32 -- Python 3.11.7, pytest-8.0.0, pluggy-1.3.0 -- C:\Users\okken\projects\foo\venv\Scripts\python.exe
cachedir: .pytest_cache
rootdir: C:\Users\okken\projects\foo
configfile: pytest.ini
collecting ... collected 4 items

test_param.py::test_param[4] PASSED                                      [ 25%]
test_param.py::test_param[3] PASSED                                      [ 50%]
test_param.py::test_param[2] PASSED                                      [ 75%]
test_param.py::test_param[1] PASSED                                      [100%]

============================== 4 passed in 0.02s ==============================

Output from 7.4.4, shows expected 1,2,3,4 order with both file and individual test invocation.

(foo) $ pytest test_param.py -v
============================= test session starts =============================
platform win32 -- Python 3.11.7, pytest-7.4.4, pluggy-1.3.0 -- C:\Users\okken\projects\foo\venv\Scripts\python.exe
cachedir: .pytest_cache
rootdir: C:\Users\okken\projects\foo
configfile: pytest.ini
collecting ... collected 4 items

test_param.py::test_param[1] PASSED                                      [ 25%]
test_param.py::test_param[2] PASSED                                      [ 50%]
test_param.py::test_param[3] PASSED                                      [ 75%]
test_param.py::test_param[4] PASSED                                      [100%]

============================== 4 passed in 0.02s ==============================
(foo) $ pytest test_param.py::test_param -v
============================= test session starts =============================
platform win32 -- Python 3.11.7, pytest-7.4.4, pluggy-1.3.0 -- C:\Users\okken\projects\foo\venv\Scripts\python.exe
cachedir: .pytest_cache
rootdir: C:\Users\okken\projects\foo
configfile: pytest.ini
collecting ... collected 4 items

test_param.py::test_param[1] PASSED                                      [ 25%]
test_param.py::test_param[2] PASSED                                      [ 50%]
test_param.py::test_param[3] PASSED                                      [ 75%]
test_param.py::test_param[4] PASSED                                      [100%]

============================== 4 passed in 0.00s ==============================

[The-Compiler]

Bisected to 385796b:

• PR: Rework Session and Package collection  #11646
• Fix for How should package collection work? #7777

cc @bluetech

[bluetech]

Thanks @okken for the clear report and @The-Compiler for bisecting. Submitted a fix in #11957.

[okken]

@The-Compiler I really need to lean on bisect more often. You're an inspiration.
@bluetech That's awesome that you found a fix so quickly. You rock.

[The-Compiler]

Hah, the moment I found out about git bisect, it felt like magic!

For reference, for something like this, what I do is something like:

• Make sure I did a pip install -e .
• git bisect start
• git checkout 8.0.0 and make sure I can repro
• git bisect bad
• git checkout 7.4.4 and make sure it works fine
• git bisect good
• Follow instructions and git bisect good or git bisect bad as necessary.

After the initial range, I needed to test < 10 times to find the culprit, so it's something that can usually be done in minutes once there is a nice fast reproducer.