Module code is executed twice when collecting doctests #11306

flying-sheep · 2023-08-11T10:47:35Z

a detailed description of the bug or problem you are having
output of pip list from the virtual environment you are using
pytest and operating system versions
minimal example if possible

Bug description and minimal example

https://github.com/flying-sheep/pytest-doctest-import-twice

Versions

OS: Arch Linux

pip list --format=json --exclude-editable | from json | to md --pretty

name	version
iniconfig	2.0.0
packaging	23.1
pip	23.2.1
pluggy	1.2.0
pytest	7.4.0
setuptools	68.0.0
wheel	0.41.0

The text was updated successfully, but these errors were encountered:

The-Compiler · 2023-08-11T11:12:13Z

Thanks for the detailed example! However, it seems to fail initially with:

        File "/tmp/pip-build-env-d1izs26m/overlay/lib/python3.11/site-packages/hatchling/metadata/core.py", line 508, in readme
          raise OSError(message)
      OSError: Readme file does not exist: README.md

after fixing that:

diff --git i/pyproject.toml w/pyproject.toml
index ecb2f55..c797e64 100644
--- i/pyproject.toml
+++ w/pyproject.toml
@@ -7,7 +7,7 @@ name = 'pytest-doctest-import-twice'
 version = '1.0'
 description = 'pytest reproducer for doctest import behavior'
 authors = [{ name = 'Philipp A.', email = 'flying-sheep@web.de' }]
-readme = 'README.md'
+readme = 'README.rst'
 license = 'GPL-3.0-or-later'
 requires-python = '>=3.11'
 dependencies = [

it looks like it also needs an export PYTHONPATH=. to work.

I ended up creating a virtualenv manually and using .venv/bin/pytest --import-mode=importlib --doctest-modules -o pythonpath=. to reproduce instead. Surprisingly, the example works fine with pytest 7.1.0, and I was able to bisect this to:

Pass importmode to import_path in DoctestModule #10088

However, that looks like it just happens to expose this problem, and isn't actually the culprit itself.

flying-sheep · 2023-08-11T11:54:49Z

it looks like it also needs an export PYTHONPATH=. to work.

Or equivalent. hatch run … (in the README) creates a venv for you and installs your package in dev mode before running the specified script. As far as the import system is concerned, a dev install is just adding the parent directory to sys.path, only using a .pth file instead of an env variable.

If you do a regular install and run Python in safepath (-P) mode, the error is still reproducible:

$ hatch run python -Pm site
sys.path = [
    '/usr/lib/python311.zip',
    '/usr/lib/python3.11',
    '/usr/lib/python3.11/lib-dynload',
    '<venv>/lib/python3.11/site-packages',
]
[…]
$ hatch run python -Pm pytest
[…]
collected 1 item / 1 error
[…]

the same applies to doing a regular install and running the pytest binary (which is probably equivalent to safepath mode anyway)

$ hatch run pytest
[…]
collected 1 item / 1 error
[…]

flying-sheep · 2023-08-11T12:07:06Z

The takeaway of all this is that this is not an issue of setting the PYTHONPATH wrong:

It’s an issue with the way pytest’s imports things by path. ~~This code path should be hit the second time:~~

pytest/src/_pytest/pathlib.py

Lines 524 to 527 in 556e075

    
           if mode is ImportMode.importlib: 
        
               module_name = module_name_from_path(path, root) 
        
               with contextlib.suppress(KeyError): 
        
                   return sys.modules[module_name]

/edit: wait a second, that code path doesn’t exist in my local copy

The-Compiler · 2023-08-11T12:15:21Z

yes, I'm not saying this is caused by PYTHONPATH. But your example needs the PYTHONPATH set to run, even when using hatch run, at least on my system:

$ hatch run pytest
============================= test session starts ==============================
platform linux -- Python 3.11.4, pytest-7.4.0, pluggy-1.2.0
rootdir: /home/florian/tmp/11306/pytest-doctest-import-twice
configfile: pyproject.toml
collected 0 items / 3 errors                                                   

==================================== ERRORS ====================================
___________ ERROR collecting pytest_doctest_import_twice/__init__.py ___________
../../../.local/share/hatch/env/virtual/pytest-doctest-import-twice/fxat3dFr/pytest-doctest-import-twice/lib/python3.11/site-packages/_pytest/runner.py:341: in from_call
    result: Optional[TResult] = func()
../../../.local/share/hatch/env/virtual/pytest-doctest-import-twice/fxat3dFr/pytest-doctest-import-twice/lib/python3.11/site-packages/_pytest/runner.py:372: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
../../../.local/share/hatch/env/virtual/pytest-doctest-import-twice/fxat3dFr/pytest-doctest-import-twice/lib/python3.11/site-packages/_pytest/doctest.py:547: in collect
    module = import_path(
../../../.local/share/hatch/env/virtual/pytest-doctest-import-twice/fxat3dFr/pytest-doctest-import-twice/lib/python3.11/site-packages/_pytest/pathlib.py:538: in import_path
    spec.loader.exec_module(mod)  # type: ignore[union-attr]
<frozen importlib._bootstrap_external>:940: in exec_module
    ???
<frozen importlib._bootstrap>:241: in _call_with_frames_removed
    ???
pytest_doctest_import_twice/__init__.py:1: in <module>
    from .singleton import MyVindictiveSingleton
E   ModuleNotFoundError: No module named 'pytest_doctest_import_twice'
_____________________ ERROR collecting tests/test_basic.py _____________________
../../../.local/share/hatch/env/virtual/pytest-doctest-import-twice/fxat3dFr/pytest-doctest-import-twice/lib/python3.11/site-packages/_pytest/runner.py:341: in from_call
    result: Optional[TResult] = func()
../../../.local/share/hatch/env/virtual/pytest-doctest-import-twice/fxat3dFr/pytest-doctest-import-twice/lib/python3.11/site-packages/_pytest/runner.py:372: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
../../../.local/share/hatch/env/virtual/pytest-doctest-import-twice/fxat3dFr/pytest-doctest-import-twice/lib/python3.11/site-packages/_pytest/doctest.py:547: in collect
    module = import_path(
../../../.local/share/hatch/env/virtual/pytest-doctest-import-twice/fxat3dFr/pytest-doctest-import-twice/lib/python3.11/site-packages/_pytest/pathlib.py:538: in import_path
    spec.loader.exec_module(mod)  # type: ignore[union-attr]
../../../.local/share/hatch/env/virtual/pytest-doctest-import-twice/fxat3dFr/pytest-doctest-import-twice/lib/python3.11/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
tests/test_basic.py:1: in <module>
    from pytest_doctest_import_twice import instance
E   ImportError: cannot import name 'instance' from 'pytest_doctest_import_twice' (unknown location)
_____________________ ERROR collecting tests/test_basic.py _____________________
ImportError while importing test module '/home/florian/tmp/11306/pytest-doctest-import-twice/tests/test_basic.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/test_basic.py:1: in <module>
    from pytest_doctest_import_twice import instance
E   ImportError: cannot import name 'instance' from 'pytest_doctest_import_twice' (unknown location)
=========================== short test summary info ============================
ERROR pytest_doctest_import_twice/__init__.py - ModuleNotFoundError: No module named 'pytest_doctest_import_twice'
ERROR tests/test_basic.py - ImportError: cannot import name 'instance' from 'pytest_doctest_import_twic...
ERROR tests/test_basic.py
!!!!!!!!!!!!!!!!!!! Interrupted: 3 errors during collection !!!!!!!!!!!!!!!!!!!!
============================== 3 errors in 0.10s ===============================

flying-sheep · 2023-08-11T12:31:06Z

That’s shouldn’t be the case. As said, hatch should just make a venv, dev-install the project into it, and then run the script.

Are you using an old hatch version (≠1.7.0)? Maybe try manually creating a venv?

#!/bin/bash
test -d .venv || python -m virtualenv .venv
source .venv/bin/activate
pip install -e .  # or pip install .
pytest

It shouldn’t matter (as said, both PYTHONPATH and regular install reproduce the bug) but it’s worrying that something on our systems behaves so differently.

You can also try to recreate the env using hatch env prune and then hatch run pytest.

flying-sheep · 2023-08-11T12:49:45Z

wait a second, the code path I saw above has not been released. It’s part of #11148

I updated the test code to use pytest master and add some context about what pytest is doing.

Seems like that code didn’t quite fix the problem, I see:

E   module_name: 'pytest_doctest_import_twice.__init__'
E  […]
E   -----------------------------------
E   module_name: 'pytest_doctest_import_twice.__init__'
E  […]

shouldn’t that cause the return sys.modules[module_name] line to succeed one of the two times?

flying-sheep · 2023-08-11T13:05:42Z

@nicoddemus any idea how it’s possible that the line throws a KeyError twice with the same argument?

nicoddemus · 2023-08-11T14:09:58Z

Hi @flying-sheep sorry not a glance, I think this might need debugging to understand what is happening.

flying-sheep · 2023-08-11T14:20:25Z

I hope I provided a good enough reproducer for that! Glad it’s on your radar.

nicoddemus · 2023-08-11T14:27:51Z

Oh sorry I was not clear, I did not mean I will be working on this soon, I'm on vacation at the moment.

flying-sheep · 2023-08-11T14:55:35Z

Haha no worries, enjoy!

flying-sheep · 2023-09-05T10:12:05Z

Hia, is there a chance you could take a look at this? Would be great to be able to run our doctests

For packages, `import_path` receives the path to the package's `__init__.py` file, however module names (as they live in `sys.modules`) should not include the `__init__` part. For example, `app/core/__init__.py` should be imported as `app.core`, not as `app.core.__init__`. Fix pytest-dev#11306

nicoddemus · 2023-09-05T12:34:45Z

@flying-sheep thanks for the ping and the reproducible example.

Opened #11390 with the fix. 👍

For packages, `import_path` receives the path to the package's `__init__.py` file, however module names (as they live in `sys.modules`) should not include the `__init__` part. For example, `app/core/__init__.py` should be imported as `app.core`, not as `app.core.__init__`. Fix pytest-dev#11306

For packages, `import_path` receives the path to the package's `__init__.py` file, however module names (as they live in `sys.modules`) should not include the `__init__` part. For example, `app/core/__init__.py` should be imported as `app.core`, not as `app.core.__init__`. Fix #11306

This PR contains the following updates: | Package | Update | Change | |---|---|---| | [pytest](https://docs.pytest.org/en/latest/) ([source](https://github.com/pytest-dev/pytest), [changelog](https://docs.pytest.org/en/stable/changelog.html)) | patch | `==7.4.0` -> `==7.4.2` | --- ### Release Notes <details> <summary>pytest-dev/pytest (pytest)</summary> ### [`v7.4.2`](https://github.com/pytest-dev/pytest/releases/tag/7.4.2): pytest 7.4.2 (2023-09-07) [Compare Source](pytest-dev/pytest@7.4.1...7.4.2) ### Bug Fixes - [#11237](pytest-dev/pytest#11237): Fix doctest collection of `functools.cached_property` objects. - [#11306](pytest-dev/pytest#11306): Fixed bug using `--importmode=importlib` which would cause package `__init__.py` files to be imported more than once in some cases. - [#11367](pytest-dev/pytest#11367): Fixed bug where `user_properties` where not being saved in the JUnit XML file if a fixture failed during teardown. - [#11394](pytest-dev/pytest#11394): Fixed crash when parsing long command line arguments that might be interpreted as files. ### Improved Documentation - [#11391](pytest-dev/pytest#11391): Improved disclaimer on pytest plugin reference page to better indicate this is an automated, non-curated listing. ### [`v7.4.1`](https://github.com/pytest-dev/pytest/releases/tag/7.4.1): pytest 7.4.1 (2023-09-02) [Compare Source](pytest-dev/pytest@7.4.0...7.4.1) ## Bug Fixes - [#10337](pytest-dev/pytest#10337): Fixed bug where fake intermediate modules generated by `--import-mode=importlib` would not include the child modules as attributes of the parent modules. - [#10702](pytest-dev/pytest#10702): Fixed error assertion handling in `pytest.approx` when `None` is an expected or received value when comparing dictionaries. - [#10811](pytest-dev/pytest#10811): Fixed issue when using `--import-mode=importlib` together with `--doctest-modules` that caused modules to be imported more than once, causing problems with modules that have import side effects. </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).  Reviewed-on: https://git.apud.pl/jacek/adventofcode/pulls/32 Co-authored-by: Renovate <renovate@apud.pl> Co-committed-by: Renovate <renovate@apud.pl>

flying-sheep changed the title ~~Modules executed twice~~ Module code is executed twice when collecting doctests Aug 11, 2023

flying-sheep mentioned this issue Aug 11, 2023

run doctests scverse/scanpy#2605

Merged

Zac-HD added topic: collection related to the collection phase plugin: doctests related to the doctests builtin plugin labels Aug 20, 2023

nicoddemus mentioned this issue Sep 5, 2023

Fix import_path for packages #11390

Merged

nicoddemus closed this as completed in #11390 Sep 5, 2023

flying-sheep mentioned this issue Jan 15, 2024

ModuleNotFoundError when using --doctest-modules, a src/ layout, and --import-mode=importlib (and no editable mode install) #11475

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Module code is executed twice when collecting doctests #11306

Module code is executed twice when collecting doctests #11306

flying-sheep commented Aug 11, 2023

The-Compiler commented Aug 11, 2023

flying-sheep commented Aug 11, 2023 •

edited

flying-sheep commented Aug 11, 2023 •

edited

The-Compiler commented Aug 11, 2023

flying-sheep commented Aug 11, 2023 •

edited

flying-sheep commented Aug 11, 2023 •

edited

flying-sheep commented Aug 11, 2023 •

edited

nicoddemus commented Aug 11, 2023

flying-sheep commented Aug 11, 2023

nicoddemus commented Aug 11, 2023

flying-sheep commented Aug 11, 2023

flying-sheep commented Sep 5, 2023

nicoddemus commented Sep 5, 2023 •

edited

Module code is executed twice when collecting doctests #11306

Module code is executed twice when collecting doctests #11306

Comments

flying-sheep commented Aug 11, 2023

Bug description and minimal example

Versions

The-Compiler commented Aug 11, 2023

flying-sheep commented Aug 11, 2023 • edited

flying-sheep commented Aug 11, 2023 • edited

The-Compiler commented Aug 11, 2023

flying-sheep commented Aug 11, 2023 • edited

flying-sheep commented Aug 11, 2023 • edited

flying-sheep commented Aug 11, 2023 • edited

nicoddemus commented Aug 11, 2023

flying-sheep commented Aug 11, 2023

nicoddemus commented Aug 11, 2023

flying-sheep commented Aug 11, 2023

flying-sheep commented Sep 5, 2023

nicoddemus commented Sep 5, 2023 • edited

flying-sheep commented Aug 11, 2023 •

edited

flying-sheep commented Aug 11, 2023 •

edited

flying-sheep commented Aug 11, 2023 •

edited

flying-sheep commented Aug 11, 2023 •

edited

flying-sheep commented Aug 11, 2023 •

edited

nicoddemus commented Sep 5, 2023 •

edited