pytest-asyncio-0.23.2 loads packages more eagerly #729

seifertm · 2023-12-24T11:01:13Z

In our case it is actually causing issues. In short, we have Python files in our tests dir that should only be loaded in a specific context. I'll elaborate in more detail below. We could probably work around it if we need to, but for now we have pinned pytest-asyncio to the previous version until this discussion reaches a conclusion.

Our specifix use case: we are developing a Python project with a plugin system. Put simply, these plugins are Python packages that register themselves at load time. They are expected to be loaded by a custom importlib finder/loader and they do not work as standalone package.

Since this plugin system is an integral part of the application we're developing, we test their behavior thoroughly. Therefore we have a bunch of these plugin "packages" as resources underneath our tests directory. They are meant as nothing more than data: our application will find and load them when appropriate. They will raise an error when loaded as a standalone package, which pytest v0.23 now does.

Originally posted by @sanderr in #713 (comment)

The text was updated successfully, but these errors were encountered:

xmatthias · 2023-12-31T16:36:11Z

We're seeing something similar (i suspect) - though due to an even odder case.

Our tests work fine up to pytest-asyncio-0.23.x ... at which point, ONLY the mac runs are failing (they do fail with Fatal Python error: Segmentation fault).
Based on my tests, this is related to the torch module (uninstalling torch completely will fix the issue).

Now torch (or pytorch) is a problematic package to depend on on it's own...
it's doing some funky stuff when imported (it messes with logging, for one - which can be triggered by simply running import torch).

We do have an auto-use fixture to completely mock the torch module on macos to work around this problem - but (i suspect) due to the eager loading of pytest-asyncio, it's loading torch BEFORE actually running the auto-use fixture, therefore importing torch - which causes the random segfaults.

I could also confirm this by messing with the installed torch file (simply placing a raise ValueError() in the __init__.py file confirmed that pytest-asyncio==0.21.1 doesn't import this file, while pytest-asyncio==0.23.2 does during the collection phase (see logs below).

It's unclear to me why pytest-asyncio does need to mess with the importing logic and would import packages that

don't have a direct import in any test file
are not imported in a nested way
are not imported when using an older version of pytest-asyncio

Now torch is imported in some files within the project - which are only referenced through a custom resolver responsible to load the package (similar to the original author of this issue) - but these files are not touched or imported directly anywyere.

In my eyes/understanding (correct me if i'm wrong), pytest-asyncio is a plugin to provide async support for tests, not to modify the test sequence, or help find additional tests - so what's the exact reason to have to modify the importing logic?

Exception details of "importing torch" error

added raise ValueError("IMported torch!") at the bottom of /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/torch/__init__.py on my CI runner

bash-3.2$ pytest --random-order-seed=469386 
=========================================================== test session starts ============================================================
platform darwin -- Python 3.11.7, pytest-7.4.3, pluggy-1.3.0
Using --random-order-bucket=module
Using --random-order-seed=469386

rootdir: /Users/runner/work/freqtrade/freqtrade
configfile: pyproject.toml
plugins: random-order-1.1.0, time-machine-2.13.0, cov-4.1.0, asyncio-0.23.2, mock-3.12.0, anyio-3.7.1, xdist-3.5.0
asyncio: mode=Mode.AUTO
collected 0 items                                                                                                                          
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/_pytest/main.py", line 271, in wrap_s
ession
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>                          ^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/_pytest/main.py", line 324, in _main
INTERNALERROR>     config.hook.pytest_collection(session=session)
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __cal
l__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _ho
okexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pluggy/_callers.py", line 152, in _mu
lticall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pluggy/_result.py", line 114, in get_
result
INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _mul
ticall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/_pytest/main.py", line 335, in pytest
_collection
INTERNALERROR>     session.perform_collect()
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/_pytest/main.py", line 675, in perfor
m_collect
INTERNALERROR>     self.items.extend(self.genitems(node))
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/_pytest/main.py", line 842, in genite
ms
INTERNALERROR>     rep = collect_one_node(node)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/_pytest/runner.py", line 546, in coll
ect_one_node
INTERNALERROR>     ihook.pytest_collectstart(collector=collector)
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __cal
l__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _ho
okexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pluggy/_callers.py", line 113, in _mu
lticall
INTERNALERROR>     raise exception.with_traceback(exception.__traceback__)
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _mul
ticall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pytest_asyncio/plugin.py", line 612, 
in pytest_collectstart
INTERNALERROR>     pyobject = collector.obj
INTERNALERROR>                ^^^^^^^^^^^^^
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/_pytest/python.py", line 310, in obj
INTERNALERROR>     self._obj = obj = self._getobj()
INTERNALERROR>                       ^^^^^^^^^^^^^^
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/_pytest/python.py", line 528, in _get
obj
INTERNALERROR>     return self._importtestmodule()
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/_pytest/python.py", line 617, in _imp
orttestmodule
INTERNALERROR>     mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/_pytest/pathlib.py", line 567, in imp
ort_path
INTERNALERROR>     importlib.import_module(module_name)
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/importlib/__init__.py", line 126, in import_module
INTERNALERROR>     return _bootstrap._gcd_import(name[level:], package, level)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
INTERNALERROR>   File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
INTERNALERROR>   File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
INTERNALERROR>   File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
INTERNALERROR>   File "<frozen importlib._bootstrap_external>", line 940, in exec_module
INTERNALERROR>   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
INTERNALERROR>   File "/Users/runner/work/freqtrade/freqtrade/freqtrade/freqai/tensorboard/__init__.py", line 3, in <module>
INTERNALERROR>     from freqtrade.freqai.tensorboard.tensorboard import TensorBoardCallback, TensorboardLogger
INTERNALERROR>   File "/Users/runner/work/freqtrade/freqtrade/freqtrade/freqai/tensorboard/tensorboard.py", line 5, in <module>
INTERNALERROR>     from torch.utils.tensorboard import SummaryWriter
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/torch/__init__.py", line 1836, in <mo
dule>
INTERNALERROR>     raise ValueError("IMported torch!")
INTERNALERROR> ValueError: IMported torch!

bnavigator · 2024-01-03T22:05:31Z

We also see this on a minimal Pandas installation. Pandas has lots of optional dependencies but imports fine with the minimal requirements. With older pytest-asyncio the test suite would collect fine and skip tests not applicable. But with the new version, the optional Pandas submodules cause an early pytest internalerror.

[    4s] + python3.11 -c 'import pandas; print(pandas.__path__); print(pandas.show_versions())'
[    8s] /usr/lib/python3.11/site-packages/_distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils.
[    8s]   warnings.warn("Setuptools is replacing distutils.")
[    8s] ['/home/abuild/rpmbuild/BUILD/pandas-2.1.4/pandas']
[    8s] 
[    8s] INSTALLED VERSIONS
[    8s] ------------------
[    8s] commit              : a671b5a8bf5dd13fb19f0e88edc679bc9e15c673
[    8s] python              : 3.11.6.final.0
[    8s] python-bits         : 64
[    8s] OS                  : Linux
[    8s] OS-release          : 6.6.6-1-default
[    8s] Version             : #1 SMP PREEMPT_DYNAMIC Mon Dec 11 09:46:39 UTC 2023 (a946a9f)
[    8s] machine             : x86_64
[    8s] processor           : x86_64
[    8s] byteorder           : little
[    8s] LC_ALL              : en_US.UTF-8
[    8s] LANG                : en_US.UTF-8
[    8s] LOCALE              : en_US.UTF-8
[    8s] 
[    8s] pandas              : 2.1.4
[    8s] numpy               : 1.26.2
[    8s] pytz                : 2023.3.post1
[    8s] dateutil            : 2.8.2
[    8s] setuptools          : 69.0.3
[    8s] pip                 : 23.3.2
[    8s] Cython              : 0.29.36
[    8s] pytest              : 7.4.4
[    8s] hypothesis          : 6.92.1
[    8s] sphinx              : None
[    8s] blosc               : None
[    8s] feather             : None
[    8s] xlsxwriter          : None
[    8s] lxml.etree          : None
[    8s] html5lib            : None
[    8s] pymysql             : None
[    8s] psycopg2            : None
[    8s] jinja2              : None
[    8s] IPython             : None
[    8s] pandas_datareader   : None
[    8s] bs4                 : None
[    8s] bottleneck          : None
[    8s] dataframe-api-compat: None
[    8s] fastparquet         : None
[    8s] fsspec              : None
[    8s] gcsfs               : None
[    8s] matplotlib          : None
[    8s] numba               : None
[    8s] numexpr             : None
[    8s] odfpy               : None
[    8s] openpyxl            : None
[    8s] pandas_gbq          : None
[    8s] pyarrow             : None
[    8s] pyreadstat          : None
[    8s] pyxlsb              : None
[    8s] s3fs                : None
[    8s] scipy               : None
[    8s] sqlalchemy          : None
[    8s] tables              : None
[    8s] tabulate            : None
[    8s] xarray              : None
[    8s] xlrd                : None
[    8s] zstandard           : None
[    8s] tzdata              : None
[    8s] qtpy                : None
[    8s] pyqt5               : None
[    8s] None
[    8s] + xvfb-run pytest-3.11 -vvv -rsfE -n0 -m 'not (network or clipboard or single_cpu)' -k 'not (test_pivot_number_of_levels_larger_than_int32 or psycopg2_engine or psycopg2_conn or pymysql_engine or pymysql_conn or test_git_version)' pandas
[   12s] ============================= test session starts ==============================
[   12s] platform linux -- Python 3.11.6, pytest-7.4.4, pluggy-1.3.0 -- /usr/bin/python3.11
[   12s] cachedir: .pytest_cache
[   12s] hypothesis profile 'ci' -> deadline=None, suppress_health_check=[HealthCheck.too_slow, HealthCheck.differing_executors], database=DirectoryBasedExampleDatabase(PosixPath('/home/abuild/rpmbuild/BUILD/pandas-2.1.4/.hypothesis/examples'))
[   12s] rootdir: /home/abuild/rpmbuild/BUILD/pandas-2.1.4/pandas
[   12s] configfile: pyproject.toml
[   12s] plugins: asyncio-0.23.3, xdist-3.5.0, hypothesis-6.92.1
[   12s] asyncio: mode=Mode.STRICT
[   12s] collecting ... collected 0 items
[   12s] INTERNALERROR> Traceback (most recent call last):
[   12s] INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/python.py", line 617, in _importtestmodule
[   12s] INTERNALERROR>     mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
[   12s] INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[   12s] INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/pathlib.py", line 567, in import_path
[   12s] INTERNALERROR>     importlib.import_module(module_name)
[   12s] INTERNALERROR>   File "/usr/lib64/python3.11/importlib/__init__.py", line 126, in import_module
[   12s] INTERNALERROR>     return _bootstrap._gcd_import(name[level:], package, level)
[   12s] INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[   12s] INTERNALERROR>   File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
[   12s] INTERNALERROR>   File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
[   12s] INTERNALERROR>   File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
[   12s] INTERNALERROR>   File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
[   12s] INTERNALERROR>   File "<frozen importlib._bootstrap_external>", line 940, in exec_module
[   12s] INTERNALERROR>   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
[   12s] INTERNALERROR>   File "/home/abuild/rpmbuild/BUILD/pandas-2.1.4/pandas/core/_numba/kernels/__init__.py", line 1, in <module>
[   12s] INTERNALERROR>     from pandas.core._numba.kernels.mean_ import (
[   12s] INTERNALERROR>   File "/home/abuild/rpmbuild/BUILD/pandas-2.1.4/pandas/core/_numba/kernels/mean_.py", line 13, in <module>
[   12s] INTERNALERROR>     import numba
[   12s] INTERNALERROR> ModuleNotFoundError: No module named 'numba'
[   12s] INTERNALERROR> 
[   12s] INTERNALERROR> The above exception was the direct cause of the following exception:
[   12s] INTERNALERROR> 
[   12s] INTERNALERROR> Traceback (most recent call last):
[   12s] INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/main.py", line 271, in wrap_session
[   12s] INTERNALERROR>     session.exitstatus = doit(config, session) or 0
[   12s] INTERNALERROR>                          ^^^^^^^^^^^^^^^^^^^^^
[   12s] INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/main.py", line 324, in _main
[   12s] INTERNALERROR>     config.hook.pytest_collection(session=session)
[   12s] INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
[   12s] INTERNALERROR>     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
[   12s] INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[   12s] INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
[   12s] INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
[   12s] INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[   12s] INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_callers.py", line 152, in _multicall
[   12s] INTERNALERROR>     return outcome.get_result()
[   12s] INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^
[   12s] INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_result.py", line 114, in get_result
[   12s] INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
[   12s] INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _multicall
[   12s] INTERNALERROR>     res = hook_impl.function(*args)
[   12s] INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
[   12s] INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/main.py", line 335, in pytest_collection
[   12s] INTERNALERROR>     session.perform_collect()
[   12s] INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/main.py", line 675, in perform_collect
[   12s] INTERNALERROR>     self.items.extend(self.genitems(node))
[   12s] INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/main.py", line 842, in genitems
[   12s] INTERNALERROR>     rep = collect_one_node(node)
[   12s] INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^
[   12s] INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/runner.py", line 546, in collect_one_node
[   12s] INTERNALERROR>     ihook.pytest_collectstart(collector=collector)
[   12s] INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
[   12s] INTERNALERROR>     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
[   12s] INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[   12s] INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
[   12s] INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
[   12s] INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[   12s] INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_callers.py", line 113, in _multicall
[   12s] INTERNALERROR>     raise exception.with_traceback(exception.__traceback__)
[   12s] INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _multicall
[   12s] INTERNALERROR>     res = hook_impl.function(*args)
[   12s] INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
[   12s] INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pytest_asyncio/plugin.py", line 626, in pytest_collectstart
[   12s] INTERNALERROR>     pyobject = collector.obj
[   12s] INTERNALERROR>                ^^^^^^^^^^^^^
[   12s] INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/python.py", line 310, in obj
[   12s] INTERNALERROR>     self._obj = obj = self._getobj()
[   12s] INTERNALERROR>                       ^^^^^^^^^^^^^^
[   12s] INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/python.py", line 528, in _getobj
[   12s] INTERNALERROR>     return self._importtestmodule()
[   12s] INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^
[   12s] INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/python.py", line 642, in _importtestmodule
[   12s] INTERNALERROR>     raise self.CollectError(
[   12s] INTERNALERROR> _pytest.nodes.Collector.CollectError: ImportError while importing test module '/home/abuild/rpmbuild/BUILD/pandas-2.1.4/pandas/core/_numba/kernels/__init__.py'.
[   12s] INTERNALERROR> Hint: make sure your test modules/packages have valid Python names.
[   12s] INTERNALERROR> Traceback:
[   12s] INTERNALERROR> /usr/lib/python3.11/site-packages/_pytest/python.py:617: in _importtestmodule
[   12s] INTERNALERROR>     mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
[   12s] INTERNALERROR> /usr/lib/python3.11/site-packages/_pytest/pathlib.py:567: in import_path
[   12s] INTERNALERROR>     importlib.import_module(module_name)
[   12s] INTERNALERROR> /usr/lib64/python3.11/importlib/__init__.py:126: in import_module
[   12s] INTERNALERROR>     return _bootstrap._gcd_import(name[level:], package, level)
[   12s] INTERNALERROR> <frozen importlib._bootstrap>:1204: in _gcd_import
[   12s] INTERNALERROR>     ???
[   12s] INTERNALERROR> <frozen importlib._bootstrap>:1176: in _find_and_load
[   12s] INTERNALERROR>     ???
[   12s] INTERNALERROR> <frozen importlib._bootstrap>:1147: in _find_and_load_unlocked
[   12s] INTERNALERROR>     ???
[   12s] INTERNALERROR> <frozen importlib._bootstrap>:690: in _load_unlocked
[   12s] INTERNALERROR>     ???
[   12s] INTERNALERROR> <frozen importlib._bootstrap_external>:940: in exec_module
[   12s] INTERNALERROR>     ???
[   12s] INTERNALERROR> <frozen importlib._bootstrap>:241: in _call_with_frames_removed
[   12s] INTERNALERROR>     ???
[   12s] INTERNALERROR> pandas/core/_numba/kernels/__init__.py:1: in <module>
[   12s] INTERNALERROR>     from pandas.core._numba.kernels.mean_ import (
[   12s] INTERNALERROR> pandas/core/_numba/kernels/mean_.py:13: in <module>
[   12s] INTERNALERROR>     import numba
[   12s] INTERNALERROR> E   ModuleNotFoundError: No module named 'numba'
[   12s] 
[   12s] ============================ no tests ran in 0.58s =============================

sanderr · 2024-01-04T12:18:27Z

@seifertm Thanks for your efforts in triaging this. I see you added this ticket to the v0.23 milestone. Does that mean you aim to address it sometime soon? I'm just trying to determine whether we should give it some more time, or start looking for a longer term workaround.

xmatthias · 2024-01-04T12:21:37Z

@sanderr i'm not entirely sure what a longer-term workaround would look like (?)

I've had a look around, but it doesn't seem that there's alternative plugins that provide the same functionality.
For the moment, i think we're fine to just pin the dependency to a non-broken version - there's no (known) security enhancements that would force us upgrade - at least not for the moment ..

sanderr · 2024-01-04T12:39:30Z

I have to confess that I haven't given it a lot of thought yet, but at least for my scenario I guess it would be possible (albeit through a bit of a hack) to not use __init__.py files as testing artifacts, or to modify their load-time behavior when loaded in an unexpected context.

I agree there's no urgent need to upgrade for now, but I also don't want to lag behind indefinitely. Therefore, if this turns out to be considered a low-priority issue, I might consider trying to work around it from our side, rather than to keep the pinned version. This would allow us to keep moving forward now, rather than if/when a need to upgrade does pop up.

Don't get me wrong, I'd much prefer it if it the old behavior would be restored, but that doesn't necessarily mean that the maintainer(s?) feel the same, or that it fits their agenda, hence the question.

seifertm · 2024-01-04T15:20:40Z

It's unclear to me why pytest-asyncio does need to mess with the importing logic and would import packages that
* don't have a direct import in any test file
* are not imported in a nested way
* are not imported when using an older version of pytest-asyncio

@xmatthias The concept of scoped event loops essentially requires generating pytest fixtures dynamically, depending on the structure of the test suite. This is a feature that's currently not supported in pytest. The current workaround, which is also used by pytest internally for calling things such as setUp in unittest suites, is to programmatically add fixture definitions to a module during collection time. See the following code example:
https://github.com/pytest-dev/pytest/blob/23906106968eb95afbd61adfbc7bbb795fc9aaa9/src/_pytest/python.py#L564-L577

Accessing a the obj attribute of a pytest.Module collector triggers a module import. Accessing pytest.Package.obj imports the package's __init__.py. That means when pytest-asyncio tries to attach a scoped event loop fixture to a package, that package is imported.

Starting from pytest v8, pytest.Package no longer has an obj attribute. We could try attaching the package-scoped event loop fixture to a "virtual" module that doesn't have a corresponding file on the file system.

@sanderr In my opinion, a pytest plugin should not change the way tests are collected and the old behaviour should be restored. I'm planning to do this in a patch release for v0.23. I don't think you should have to look into changes of your test suite, because of this issue.

I set aside some time for pytest-asyncio this weekend, but I cannot promise I will resolve this issue then. I'm sorry the 0.23 releases are causing so many problems, but currently my support of pytest-asyncio happens on a best-effort basis in my free time.

As for alternative testing packages, there's anyio, which comes with a pytest-plugin. I have no experience how much effort it is to migrate to anyio, though.

sanderr · 2024-01-04T15:53:15Z

Thanks for the update. I understand that you can not provide any guarantees with respect to the timeline but it's good to have rough idea of how and when this will move forward. That's all I was after at this point. Thank you for looking into this, and for the clear communication.

seifertm · 2024-01-09T17:40:00Z

@sanderr @xmatthias @bnavigator
Pytest-asyncio v0.23.4a0 was just released. I encourage you to try the pre-release and I'd love to hear if it fixes your problems.

For example, I tested the bcrypt test suite based on the instructions that @mgorny provided in #738 and the tests pass with v0.23.4a0, whereas the crashed with a different 0.23 version.

bnavigator · 2024-01-09T19:03:34Z

I can confirm that the pandas test suite now sucessfully collects the tests in the minimal environment:

Pandas environment and pytest header

[   53s] + python3.11 -c 'import pandas; print(pandas.__path__); print(pandas.show_versions())'
[   55s] /usr/lib/python3.11/site-packages/_distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils.
[   55s]   warnings.warn("Setuptools is replacing distutils.")
[   55s] ['/home/abuild/rpmbuild/BUILD/pandas-2.1.4/pandas']
[   55s] 
[   55s] INSTALLED VERSIONS
[   55s] ------------------
[   55s] commit              : a671b5a8bf5dd13fb19f0e88edc679bc9e15c673
[   55s] python              : 3.11.7.final.0
[   55s] python-bits         : 64
[   55s] OS                  : Linux
[   55s] OS-release          : 6.6.9-1-default
[   55s] Version             : #1 SMP PREEMPT_DYNAMIC Tue Jan  2 07:19:30 UTC 2024 (61d1d44)
[   55s] machine             : x86_64
[   55s] processor           : x86_64
[   55s] byteorder           : little
[   55s] LC_ALL              : en_US.UTF-8
[   55s] LANG                : en_US.UTF-8
[   55s] LOCALE              : en_US.UTF-8
[   55s] 
[   55s] pandas              : 2.1.4
[   55s] numpy               : 1.26.2
[   55s] pytz                : 2023.3.post1
[   55s] dateutil            : 2.8.2
[   55s] setuptools          : 69.0.3
[   55s] pip                 : 23.3.2
[   55s] Cython              : 0.29.36
[   55s] pytest              : 7.4.4
[   55s] hypothesis          : 6.92.2
[   55s] sphinx              : None
[   55s] blosc               : None
[   55s] feather             : None
[   55s] xlsxwriter          : None
[   55s] lxml.etree          : None
[   55s] html5lib            : None
[   55s] pymysql             : None
[   55s] psycopg2            : None
[   55s] jinja2              : None
[   55s] IPython             : None
[   55s] pandas_datareader   : None
[   55s] bs4                 : None
[   55s] bottleneck          : None
[   55s] dataframe-api-compat: None
[   55s] fastparquet         : None
[   55s] fsspec              : None
[   55s] gcsfs               : None
[   55s] matplotlib          : None
[   55s] numba               : None
[   55s] numexpr             : None
[   55s] odfpy               : None
[   55s] openpyxl            : None
[   55s] pandas_gbq          : None
[   55s] pyarrow             : None
[   55s] pyreadstat          : None
[   55s] pyxlsb              : None
[   55s] s3fs                : None
[   55s] scipy               : None
[   55s] sqlalchemy          : None
[   55s] tables              : None
[   55s] tabulate            : None
[   55s] xarray              : None
[   55s] xlrd                : None
[   55s] zstandard           : None
[   55s] tzdata              : None
[   55s] qtpy                : None
[   55s] pyqt5               : None
[   55s] None
[   56s] + xvfb-run pytest-3.11 -vvv -rsfE -n0 -m 'not (network or clipboard or single_cpu)' -k 'not (test_pivot_number_of_levels_larger_than_int32 or psycopg2_engine or psycopg2_conn or pymysql_engine or pymysql_conn or test_git_version)' pandas
[   59s] ============================= test session starts ==============================
[   59s] platform linux -- Python 3.11.7, pytest-7.4.4, pluggy-1.3.0 -- /usr/bin/python3.11
[   59s] cachedir: .pytest_cache
[   59s] hypothesis profile 'ci' -> deadline=None, suppress_health_check=[HealthCheck.too_slow, HealthCheck.differing_executors], database=DirectoryBasedExampleDatabase(PosixPath('/home/abuild/rpmbuild/BUILD/pandas-2.1.4/.hypothesis/examples'))
[   59s] rootdir: /home/abuild/rpmbuild/BUILD/pandas-2.1.4/pandas
[   59s] configfile: pyproject.toml
[   59s] plugins: asyncio-0.23.4a0, xdist-3.5.0, hypothesis-6.92.2
[   59s] asyncio: mode=Mode.STRICT
[  128s] collecting ... collected 203562 items / 2692 deselected / 59 skipped / 200870 selected
[  128s]

blink1073 · 2024-01-10T02:54:08Z

Here's what I'm seeing with 0.23.4a0:

https://github.com/blink1073/ipykernel/actions/runs/7469345207/job/20326312710?pr=1

xmatthias · 2024-01-10T06:22:03Z

Hi @seifertm,

Thanks for this - the test that was failing (in my case, the MacOS version of our CI) is working fine now (previous errors: https://github.com/freqtrade/freqtrade/actions/runs/7442875401).

With the new alpha version - i'm now getting a (new) failure on Windows which i don't quite understand - but is most likely windows specific, and based on your idea posted above, most likely caused by the new virtual module that's being added.

Link to the Actions run: https://github.com/xmatthias/freqtrade/actions/runs/7471096789/job/20330820415

Error on Windows

============================= test session starts =============================
platform win32 -- Python 3.[11](https://github.com/xmatthias/freqtrade/actions/runs/7471096789/job/20330820415#step:6:12).7, pytest-7.4.4, pluggy-1.3.0
Using --random-order-bucket=module
Using --random-order-seed=78313

rootdir: D:\a\freqtrade\freqtrade
configfile: pyproject.toml
plugins: anyio-4.2.0, asyncio-0.23.4a0, cov-4.1.0, mock-3.[12](https://github.com/xmatthias/freqtrade/actions/runs/7471096789/job/20330820415#step:6:13).0, random-order-1.1.0, xdist-3.5.0, time-machine-2.[13](https://github.com/xmatthias/freqtrade/actions/runs/7471096789/job/20330820415#step:6:14).0
asyncio: mode=Mode.AUTO
collected 3319 items / 1 error / 250 deselected / 3069 selected

=================================== ERRORS ====================================
_____________________ ERROR collecting tests/__init__.py ______________________
C:\hostedtoolcache\windows\Python\3.11.7\x64\Lib\site-packages\pytest_asyncio\plugin.py:716: in _patched_collect
    fixturemanager.parsefactories(virtual_module.obj, nodeid=pkg_nodeid)
C:\hostedtoolcache\windows\Python\3.11.7\x64\Lib\site-packages\_pytest\python.py:310: in obj
    self._obj = obj = self._getobj()
C:\hostedtoolcache\windows\Python\3.11.7\x64\Lib\site-packages\_pytest\python.py:528: in _getobj
    return self._importtestmodule()
C:\hostedtoolcache\windows\Python\3.11.7\x64\Lib\site-packages\_pytest\python.py:617: in _importtestmodule
    mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
C:\hostedtoolcache\windows\Python\3.11.7\x64\Lib\site-packages\_pytest\pathlib.py:567: in import_path
    importlib.import_module(module_name)
C:\hostedtoolcache\windows\Python\3.11.7\x64\Lib\importlib\__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1204: in _gcd_import
    ???
<frozen importlib._bootstrap>:1176: in _find_and_load
    ???
<frozen importlib._bootstrap>:1[14](https://github.com/xmatthias/freqtrade/actions/runs/7471096789/job/20330820415#step:6:15)7: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:690: in _load_unlocked
    ???
<frozen importlib._bootstrap_external>:936: in exec_module
    ???
<frozen importlib._bootstrap_external>:1073: in get_code
    ???
<frozen importlib._bootstrap_external>:1130: in get_data
    ???
E   PermissionError: [Errno 13] Permission denied: 'D:\\a\\freqtrade\\freqtrade\\tests\\pytest_asyncio_virtual_module_oq5zt_ka.py'
=========================== short test summary info ===========================
ERROR tests/__init__.py - PermissionError: [Errno 13] Permission denied: 'D:\\a\\freqtrade\\freqtrade\\tests\\pytest_asyncio_virtual_module_oq5zt_ka.py'
!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
====================== [25](https://github.com/xmatthias/freqtrade/actions/runs/7471096789/job/20330820415#step:6:26)0 deselected, 1 error in 6.91s =======================

edit seems to be a similar error / identical than the one posted by @blink1073 above ...

sanderr · 2024-01-10T12:51:59Z

Thanks. I can confirm that the issue I observed on our test suite is fixed with version 0.23.4a0.

seifertm · 2024-01-10T19:02:31Z

Thank you for your feedback!

There are still problems when running on Windows due to platform specific behavior of tempfile.NamedTemporaryFile. Therefore, I enabled testing on Windows in the pytest-asyncio CI, where I could reproduce the issue. I just merged #746 which seems to fix it.

@xmatthias @blink1073 I tagged the pre-release v0.23.4.a1 which you can use for testing. If you can verify that it also works for you, I'll tag v0.23.4

xmatthias · 2024-01-10T19:34:41Z

@seifertm thanks a lot for your efforts!

the v0.23.4.a1 release seems to work fine now! 👍

https://github.com/xmatthias/freqtrade/actions/runs/7479730385/job/20357544175

blink1073 · 2024-01-10T20:21:55Z

v23.4.a1 is failing for a different reason: https://github.com/blink1073/ipykernel/actions/runs/7480301750/job/20359431173?pr=1

mgorny · 2024-01-11T19:15:16Z

After upgrading from 0.23.3 to 0.23.4a1, I'm seeing random packages failing with pytest-xdist with errors like:

Different tests were collected between gw2 and gw7.

Example log (from setuptools): dev-python:setuptools-69.0.3:20240111-191012.log

…c package-scoped fixtures. The temporary files used for this mechanism appearing as disappear after they have been collected. This seems to create issues in some projects, such as setuptools. see pytest-dev#729 (comment) Signed-off-by: Michael Seifert <m.seifert@digitalernachschub.de>

…c package-scoped fixtures. The temporary files used for this mechanism appearing as disappear after they have been collected. This seems to create issues in some projects, such as setuptools. see #729 (comment) Signed-off-by: Michael Seifert <m.seifert@digitalernachschub.de>

seifertm · 2024-01-16T17:09:54Z

I created another pre-release (v0.23.4a2) which no longer creates temporary files during test runs. The dynamic package-scoped fixtures are now attached to the first test module in each package.

This should eliminate any issues with test suites that rely on a specific set of collected items (I assume this is the issue for setuptools).

I tested the pre-release against ipythonkernel and bcrypt successfully. One other user also confirmed that the code fixes their INTERNALERROR.

I'll tag a proper patch release by the end of the week, unless I receive any further reports regarding this issue.

mgorny · 2024-01-17T07:43:33Z

Thanks a lot. I've just tested it with known suspects, and I'll be running it doing version bumps for the next days. I'll let you know if I hit something else.

xmatthias · 2024-01-17T08:38:31Z

The 2nd pre-release appears to work fine on all OS's, from python 3.9 to python 3.12. 👍

mgorny · 2024-01-19T14:28:17Z

I'm very sorry but I've found another regression.

git clone https://github.com/pydata/pydata-sphinx-theme/
cd pydata-sphinx-theme
pip install .[test]
python -m pytest  # → collected 66 items / 1 skipped
pip install pytest-asyncio==0.23.4a2
python -m pytest  # → collected 0 items / 1 skipped

seifertm · 2024-01-28T19:14:44Z

I'm very sorry but I've found another regression.

git clone https://github.com/pydata/pydata-sphinx-theme/
cd pydata-sphinx-theme
pip install .[test]
python -m pytest  # → collected 66 items / 1 skipped
pip install pytest-asyncio==0.23.4a2
python -m pytest  # → collected 0 items / 1 skipped

Good catch! I made yet another change to the test collection that seems to fix the issue. Rather than another pre-release, I did a proper release of v0.23.4, because pytest 8 has been released and older pytest-asyncio version don't have proper upper bounds on the pytest version.

Mark90 · 2024-03-14T14:36:49Z

Hello, I think there's still a problem as of pytest-asyncio==0.23.5. When I have pytest configured to run doctests and ignore any modules that raise import errors, like so:

[tool:pytest]
addopts=--doctest-modules --doctest-ignore-import-errors

then this raises an ImportError during test collection. When I downgrade to pytest-asyncio==0.21.0 this is not the case.

(Sorry if this is not entirely relevant, I didn't read the entire thread)

seifertm added the bug label Dec 24, 2023

xmatthias mentioned this issue Dec 31, 2023

Bump pytest-asyncio from 0.21.1 to 0.23.2 freqtrade/freqtrade#9507

Closed

seifertm added this to the v0.23 milestone Jan 1, 2024

FlorentClarret mentioned this issue Jan 2, 2024

Pin the pytest-asyncio version DataDog/integrations-core#16507

Merged

3 tasks

sanderr mentioned this issue Jan 2, 2024

follow up pytest-asyncio breaking change inmanta/inmanta-core#6848

Closed

seifertm mentioned this issue Jan 2, 2024

v0.23.2 will cause INTERNALERROR if ImportWarning is raised in an imported module #713

Closed

bnavigator mentioned this issue Jan 3, 2024

pytest-asyncio makes pytest collect modules not matching python_files, causing ModuleNotFoundError #738

Closed

seifertm mentioned this issue Jan 7, 2024

Prevent import of unrealted packages during test collection #741

Merged

seifertm linked a pull request Jan 7, 2024 that will close this issue

Prevent import of unrealted packages during test collection #741

Merged

seifertm mentioned this issue Jan 7, 2024

0.23.3 breaking ImportError exception #740

Closed

xmatthias mentioned this issue Jan 8, 2024

Bump pytest-asyncio from 0.21.1 to 0.23.3 freqtrade/freqtrade#9649

Closed

seifertm closed this as completed in #741 Jan 9, 2024

seifertm reopened this Jan 9, 2024

seifertm mentioned this issue Jan 9, 2024

Startup failure on Windows with Version 0.23.3 #739

Closed

seifertm mentioned this issue Jan 10, 2024

Fixes a bug that breaks virtual modules on Windows #746

Merged

ianthomas23 mentioned this issue Jan 11, 2024

CI failing with INTERNALERROR when skipping tests ipython/ipykernel#1188

Open

seifertm mentioned this issue Jan 14, 2024

No virtual modules & No scoped loops for unknown collectors. #747

Merged

seifertm mentioned this issue Jan 28, 2024

Prevent collection issues with skip statements inside packages #764

Merged

seifertm closed this as completed Jan 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pytest-asyncio-0.23.2 loads packages more eagerly #729

pytest-asyncio-0.23.2 loads packages more eagerly #729

seifertm commented Dec 24, 2023 •

edited

xmatthias commented Dec 31, 2023 •

edited

bnavigator commented Jan 3, 2024

sanderr commented Jan 4, 2024

xmatthias commented Jan 4, 2024 •

edited

sanderr commented Jan 4, 2024

seifertm commented Jan 4, 2024

sanderr commented Jan 4, 2024

seifertm commented Jan 9, 2024

bnavigator commented Jan 9, 2024

blink1073 commented Jan 10, 2024

xmatthias commented Jan 10, 2024 •

edited

sanderr commented Jan 10, 2024

seifertm commented Jan 10, 2024

xmatthias commented Jan 10, 2024

blink1073 commented Jan 10, 2024

mgorny commented Jan 11, 2024

seifertm commented Jan 16, 2024

mgorny commented Jan 17, 2024

xmatthias commented Jan 17, 2024

mgorny commented Jan 19, 2024

seifertm commented Jan 28, 2024

Mark90 commented Mar 14, 2024

pytest-asyncio-0.23.2 loads packages more eagerly #729

pytest-asyncio-0.23.2 loads packages more eagerly #729

Comments

seifertm commented Dec 24, 2023 • edited

xmatthias commented Dec 31, 2023 • edited

bnavigator commented Jan 3, 2024

sanderr commented Jan 4, 2024

xmatthias commented Jan 4, 2024 • edited

sanderr commented Jan 4, 2024

seifertm commented Jan 4, 2024

sanderr commented Jan 4, 2024

seifertm commented Jan 9, 2024

bnavigator commented Jan 9, 2024

blink1073 commented Jan 10, 2024

xmatthias commented Jan 10, 2024 • edited

sanderr commented Jan 10, 2024

seifertm commented Jan 10, 2024

xmatthias commented Jan 10, 2024

blink1073 commented Jan 10, 2024

mgorny commented Jan 11, 2024

seifertm commented Jan 16, 2024

mgorny commented Jan 17, 2024

xmatthias commented Jan 17, 2024

mgorny commented Jan 19, 2024

seifertm commented Jan 28, 2024

Mark90 commented Mar 14, 2024

seifertm commented Dec 24, 2023 •

edited

xmatthias commented Dec 31, 2023 •

edited

xmatthias commented Jan 4, 2024 •

edited

xmatthias commented Jan 10, 2024 •

edited