ModuleNotFound for editable install of namespace package with two dots #4039
Comments
mauritsvanrees
added
bug
Needs Triage
|
Thank you very much @mauritsvanrees for opening the issue. This is my take on the minimal reproducer: > docker run --rm -it python:3.11-bullseye /bin/bash
mkdir -p /tmp/pkg1/plone/app/uuid
cd /tmp/pkg1
echo '__import__("pkg_resources").declare_namespace(__name__)' > plone/__init__.py
echo '__import__("pkg_resources").declare_namespace(__name__)' > plone/app/__init__.py
echo 'print(f"hello world {__name__}")' > plone/app/uuid/__init__.py
cat <<EOF > setup.py
from setuptools import setup, find_packages
setup(
name="plone.app.uuid",
version="42",
packages=find_packages(),
namespace_packages=[
"plone",
"plone.app",
]
)
EOF
cat <<EOF > pyproject.toml
[build-system]
requires = ["setuptools==68.1.2", "wheel"]
build-backend = "setuptools.build_meta"
EOF
mkdir -p /tmp/pkg2/plone/app/other
cd /tmp/pkg2
echo '__import__("pkg_resources").declare_namespace(__name__)' > plone/__init__.py
echo '__import__("pkg_resources").declare_namespace(__name__)' > plone/app/__init__.py
echo 'print(f"hello world {__name__}")' > plone/app/other/__init__.py
cat <<EOF > setup.py
from setuptools import setup, find_packages
setup(
name="plone.app.other",
version="42",
packages=find_packages(),
namespace_packages=[
"plone",
"plone.app",
]
)
EOF
cat <<EOF > pyproject.toml
[build-system]
requires = ["setuptools==68.1.2", "wheel"]
build-backend = "setuptools.build_meta"
EOF
cd /tmp
python3.11 -m venv .venv
.venv/bin/python -m pip install -U 'pip==23.2'
.venv/bin/python -m pip install -e pkg1
.venv/bin/python -m pip install -e pkg2
.venv/bin/python -c 'from plone.app import uuid, other; print(uuid, other)'I think this is related to the fact that it is very hard to support namespace packages with PEP 660. I have the impression that the I think a fix similar to #4020 would not be possible2. Instead maybe the best is to fix the Footnotes
|
|
I may have found a way around this. You mentioned PEP 660. Quoting from the part about what to put in the wheel (emphasis mine):
This got me thinking: would moving to a src layout help? And it seems it does! Ah, but maybe this only works for one package at a time. I adapted your minimal example (thanks!), and there Output of the last two commands: |
|
I created a PR for |
|
Hi @mauritsvanrees, does #4041 work for your original problem? You can test by first installing it (together with pip install wheel https://github.com/abravalheri/setuptools/archive/refs/heads/issue-4039.zipand then installing your project with |
Yeah, the There is still a problem with the naming of the |
|
I still had to modify a bit the logic of I was a bit disappointed because I was expecting the If I understood correctly the way the For example, if I get the part of #4041 that fix the name uniqueness of > docker run --rm -it python:3.11-bullseye /bin/bash
mkdir -p /tmp/pkg1/plone/app/uuid
cd /tmp/pkg1
echo '__import__("pkg_resources").declare_namespace(__name__)' > plone/__init__.py
echo '__import__("pkg_resources").declare_namespace(__name__)' > plone/app/__init__.py
echo 'print(f"hello world {__name__}")' > plone/app/uuid/__init__.py
cat <<EOF > setup.py
from setuptools import setup, find_packages
setup(
name="plone.app.uuid",
version="42",
packages=find_packages(),
namespace_packages=[
"plone",
"plone.app",
]
)
EOF
cat <<EOF > pyproject.toml
[build-system]
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta"
EOF
mkdir -p /tmp/pkg2/plone/app/other
cd /tmp/pkg2
echo '__import__("pkg_resources").declare_namespace(__name__)' > plone/__init__.py
echo '__import__("pkg_resources").declare_namespace(__name__)' > plone/app/__init__.py
echo 'print(f"hello world {__name__}")' > plone/app/other/__init__.py
cat <<EOF > setup.py
from setuptools import setup, find_packages
setup(
name="plone.app.other",
version="42",
packages=find_packages(),
namespace_packages=[
"plone",
"plone.app",
]
)
EOF
cat <<EOF > pyproject.toml
[build-system]
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta"
EOF
cd /tmp
python3.11 -m venv .venv
.venv/bin/python -m pip install -U 'pip==23.2'
.venv/bin/python -m pip install https://github.com/abravalheri/setuptools/archive/39245fc7503a82d7ba7f1706b2e599c5aecb1cdf.zip
.venv/bin/python -m pip install 'wheel==0.41.2'
.venv/bin/python -m pip install -e ./pkg1 --no-build-isolation
.venv/bin/python -m pip install ./pkg2 --no-build-isolation
.venv/bin/python -c 'from plone.app import uuid, other; print(uuid, other)'
# hello world plone.app.other
# Traceback (most recent call last):
# File "<string>", line 1, in <module>
# ImportError: cannot import name 'uuid' from 'plone.app' (unknown location)Investigating further we can see that the .venv/bin/python
Python 3.11.4 (main, Jul 28 2023, 05:20:14) [GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.modules["plone.app"].__path__
_NamespacePath(['/tmp/.venv/lib/python3.11/site-packages/plone/app', '/tmp/pkg1/plone/app'])
>>> import plone.app.other
hello world plone.app.other
>>> sys.modules["plone.app"].__path__
_NamespacePath(['/tmp/.venv/lib/python3.11/site-packages/plone/app'])So the only thing I can do is to modify the meta path finder again to cover for this case as well... (With #3994 in mind, any attempt to "guess" file location for modules during import time becomes very un-maintainable/difficult-to-do/un-performant because Python does not have good tools to deal with case-insensitive-yet-case-preserving file systems in the way PEP 235 requires - at least not exposed as part of the public API). At the end of the day, I settled for using the information explicitly given via |
Hi @abravalheri. Checking with src-layout... yes, this works as well. Thank you! I am not familiar with the code, and can't really judge whether the fix is good, except that it works for me. I suppose if the choice is to support either pkg_resources or pkgutil namespaces, it makes more sense to support pkg_resources, as that is part of setuptools. Note that in Plone we do want to move to native namespaces, but that will have to wait till a next major release, which I expect at the earliest in 2025. |
|
Thank you @mauritsvanrees for testing. I plan to release a beta version soon that will allow users to experiment with the accumulated changes. If everything goes well we can release a final version next week.
I don't think we have this choice right now (unless we decide to reimplement the convoluted way how stdlib's import machinery deals with case-insensitive-but-preserving file systems). |
Well, I tried to create a beta version... but the CI process automatically converted the beta into a full blown version, so I had no choice there and the new version is out 😅. |
|
I tried my original problem repo again, without any special setuptools settings. This failed. So the new release works. Thanks! |
|
Out of scope for this issue, but I wonder why setuptools 68.1.2 is still used if I don't explicitly ask for a newer version in I don't have a question here, just sharing some notes in case someone else comes across this issue and wonders why a certain setuptools version is used. And this "someone else" can be me in a few months. ;-) Because if I undo the With my I am not sure why it picks 68.1.2 when I do not explicitly require a minimum version. The tox env where the editable package is used, has an older version: So where does the 68.1.2 version come from? Ah, this comes from the environment where the Well, not entirely. When I downgrade or upgrade setuptools in that env, still 68.1.2 is used. Ah, the python in there is really a Python installed by Mac homebrew . I thought I was using pyenv everywhere, but maybe Interestingly, the When I call Does it help if I let pipx use a fresh tox with explicitly this python? So: no, this does not help. It may be specific for how tox installs the development package. Manually it is fine: And the tests pass. I can install the previous setuptools version in the env and it still works: If I then try without build isolation, then I expect it to fail: Update to latest setuptools and it should work again: Summary: it can be tricky to predict which
For tox, you can influence the packaging environment and force a specific setuptools version for building packages: I tried the following for my use case, but the Other ways to get tox to behave how you want, would be let it not install the package itself, but use the exact command that you want, something like this: Anyway, best seems to be to specify the build-system in |
|
Hi @mauritsvanrees , you are right, that is very difficult to understand, and if someone needs a specific version, it is better to use an explicit lower/upper bound in If I had to guess I would say it boils down to Does it make sense to |
|
No, I have removed the |
setuptools version
68.1.2, same with main
Python version
3.11.5
OS
MacOS
Additional environment information
Also happening on GitHub-actions on a Linux env.
See also my comment here: plone/meta#172 (comment)
Locally, on Python 3.10 all goes well, on 3.11 not.
Description
I maintain various setuptools/pkg_resources style namespace packages with two dots, for example:
plone.app.uuid. Until the beginning of August, an editable install worked fine. Now it fails, because setuptools 68.1.0+ is used:ModuleNotFoundError: No module named 'plone.app.uuid'Packages with just one dot seem fine, for example
plone.session.Expected behavior
After
pip install -e .of theplone.app.uuidpackage, I should be able to doimport plone.app.uuid.How to Reproduce
I tried to create a stripped-down copy of
plone.app.uuid, but there an import works fine. I suspect it only goes wrong if anotherplone.app.*package is installed next to it. So use the original repo, but use a branch I created for this issue with minor changes (mainly a much smallertox.ini):This can take a few minutes, because it needs lots of packages.
Output
The editable install generated this file from a template in setuptools:
When I edit
MAPPINGto the following, and directly call the test command, instead of running tox again, it works:I don't know if that makes any sense, or if the
NAMESPACESshould be filled.Workaround: force using an older
setuptools, by adding this inpyproject.toml:I suspect this is a regression caused by PR #3995, so cc @aganders3.
It may need a fix similar to PR #4020.
The text was updated successfully, but these errors were encountered: