Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Recommendations for caching poetry dependencies causes the wrong python version to be used for poetry install #846

Open
4 of 5 tasks
sdb9696 opened this issue Apr 15, 2024 · 14 comments
Assignees
Labels
bug Something isn't working

Comments

@sdb9696
Copy link

sdb9696 commented Apr 15, 2024

Description:

The recommendations for caching poetry dependencies causes the wrong python version to be used for poetry install.

Because pipx install poetry happens before setup-python and By default, Poetry will try to use the Python version used during Poetry’s installation to create the virtual environment for the current project.

Maybe the solution here is to update the docs to make sure that users know to run poetry env use ${{ steps.setup-python.outputs.python-path }}python prior to running poetry install. I have not tested that works however.

Action version:

v5

Platform:

  • Ubuntu
  • macOS
  • Windows

Runner type:

  • Hosted
  • Self-hosted

Tools version:

All versions of python affected except the default version installed on the github runner.

Repro steps:

Follow the steps outlined in advance-usage.md for caching poetry dependencies.

Run poetry install

Expected behavior:

poetry install should use the version of python set by setup-python

Actual behavior:

poetry install uses the default version of python from the runner os.

@sdb9696 sdb9696 added bug Something isn't working needs triage labels Apr 15, 2024
@HarithaVattikuti
Copy link
Contributor

Hello @sdb9696
Thank you for creating this issue. We will investigate it and get back to you as soon as we have some feedback.

@andwaredev
Copy link

Hello @sdb9696 – my team ran into this same issue recently. We solved it by pinning the python version in our pyproject.toml file, under [tool.poetry.dependencies]. See example below.

[tool.poetry.dependencies]
python = "3.11.4"

@sdb9696
Copy link
Author

sdb9696 commented Apr 18, 2024

Thanks @andwaredev but we run our CI against all supported python versions to check there are no issues so not sure this workaround will help.

@silverwind
Copy link

silverwind commented Apr 19, 2024

I find it highly concerning that the recommend and only working solution for caching poetry venvs is to install poetry before setup-python. It can not be assumed that python is present before setup-python has run because some runner images will not have python pre-installed.

This is what I would like to see working:

- uses: actions/setup-python@v5
  with:
    python-version: '3.12'
    cache: 'poetry'
- run: pipx install poetry

@evilhamsterman
Copy link

It's always struck me as a bad chicken/egg problem that poetry is the only cache type that requires the environment manager to be installed before it can be used. Even pipenv is not required to be installed

// because pipenv is not preinstalled on hosted images and virtualenv is not created:
it just uses the default pipenv location.

The Poetry default cache location is well known and I bet in CI the vase majority of projects are using that case. The big difference appears to be that the Poetry option is trying to support the fact that Poetry can install virtual environments in the project. Pipenv can also do this but it's just ignored for that option. I'd recommend just caching the default location like Pipenv, or you could have an option for enabling caching in project virtual environments it people use it.

@evilhamsterman
Copy link

evilhamsterman commented Apr 24, 2024

@silverwind You can tell poetry to use a specific version of python prior to running install poetry env use <python executable> then run poetry install

https://python-poetry.org/docs/managing-environments#switching-between-environments

@silverwind
Copy link

silverwind commented Apr 24, 2024

@evilhamsterman doesn't really help me if there is simply no python installed before setup-python ran. Of course one could do workarounds by running setup-python twice, but I rather skip on enabling cache than doing such hacks.

@adminy
Copy link

adminy commented May 2, 2024

I find it highly concerning that the recommend and only working solution for caching poetry venvs is to install poetry before setup-python. It can not be assumed that python is present before setup-python has run because some runner images will not have python pre-installed.

This is what I would like to see working:

- uses: actions/setup-python@v5
  with:
    python-version: '3.12'
    cache: 'poetry'
- run: pipx install poetry

This also makes the assumtion that you have pipx which I do not :)

I'd highly recommend to install poetry as part of this python version fetching if cache is set to poetry.

Chances are everyone is going to need poetry if they are using poetry cache, so maybe get the specific python version with poetry pre-installed. This seems to be the only viable solution.

@silverwind
Copy link

This also makes the assumtion that you have pipx which I do not :)

I think setup-python does install pip and its variants currently, right?

@silverwind
Copy link

This also makes the assumtion that you have pipx which I do not :)

I think setup-python does install pip and its variants currently, right?

Apparently, it doesn't so to install poetry without pip and having a poetry cache, this would be the ideal:

- uses: actions/setup-python@v5
  with:
    python-version: '3.12'
    cache: 'poetry'
- uses: snok/install-poetry@v1

@LaurensBosscher
Copy link

LaurensBosscher commented May 17, 2024

This is indeed inconvenient. We install Python and Poetry in the following way:

      - name: Install Poetry
        run: |
          python -m pip install poetry
          
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: 3.12
          cache: poetry
          cache-dependency-path: projects/investment_decision_support_system/poetry.lock
          
      - name: Set Poetry environment
        run: |
          python -m pip install poetry
          cd projects/investment_decision_support_system/
          poetry env use 3.12

This triggered the following error:

  - Installing xmltodict (0.13.0 9c3ec3c)
  ChefBuildError
  Backend subprocess exited when trying to invoke get_requires_for_build_wheel
  
  Traceback (most recent call last):
    File "/opt/pipx/venvs/poetry/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 373, in <module>
      main()
    File "/opt/pipx/venvs/poetry/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 357, in main
      json_out["return_val"] = hook(**hook_input["kwargs"])
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/pipx/venvs/poetry/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 128, in get_requires_for_build_wheel
      backend = _build_backend()
                ^^^^^^^^^^^^^^^^
    File "/opt/pipx/venvs/poetry/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 70, in _build_backend
      obj = import_module(mod_path)
            ^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/importlib/__init__.py", line 90, in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
    File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
    File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
    File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
    File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
    File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
    File "<frozen importlib._bootstrap_external>", line 995, in exec_module
    File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
    File "/tmp/tmpigmi429k/.venv/lib/python3.12/site-packages/setuptools/__init__.py", line 16, in <module>
      import setuptools.version
    File "/tmp/tmpigmi429k/.venv/lib/python3.12/site-packages/setuptools/version.py", line 1, in <module>
      import pkg_resources
    File "/tmp/tmpigmi429k/.venv/lib/python3.12/site-packages/pkg_resources/__init__.py", line 2191, in <module>
      register_finder(pkgutil.ImpImporter, find_on_path)
                      ^^^^^^^^^^^^^^^^^^^
  AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
  
  at /opt/pipx/venvs/poetry/lib/python3.10/site-packages/poetry/installation/chef.py:164 in _prepare
      160│ 
      161│                 error = ChefBuildError("\n\n".join(message_parts))
      162│ 
      163│             if error is not None:
    → 164│                 raise error from None
      165│ 
      166│             return path
      167│ 
      168│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Do note how the installation is failing because Poetry is using Python 3.10 while the specified Python version is 3.12.

This took a while to chase down. I'm not sure what exactly depends on Poetry but perhaps it would be possible to integrate those functions into setup-python instead of having a hard dependency on the whole package?

Alternatively, if there's a workaround it would be great if the documentation could be updated.

@evilhamsterman
Copy link

Ultimately we're considering just using actions/cache directly to cache poetry rather than setup-python's built in support

@till
Copy link

till commented May 21, 2024

I just ran into this issue as well. I used an example in the docs that triggers this bug beautifully. Maybe time to ramp up the tests for this action?

@till
Copy link

till commented May 21, 2024

I am running the following workflow which works:

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - run: pipx install poetry==1.8.3
    - run: echo "/root/.local/bin" >> $GITHUB_PATH
    - uses: actions/setup-python@v5
      with:
        python-version: '3.11'
        cache: poetry
    - run: pipx install poetry==1.8.3
    - run: poetry install
      working-directory: app
    - run: poetry run flake8
      working-directory: app

pipx is smart enough not to install poetry again, and somehow it ends up in the correct environment without any hacks.

What I had to do was add /root/.local/bin to the $GITHUB_PATH, because otherwise it wouldn't be able to find poetry anywhere and fail at actions/setup-python:

[pr/lint]   💬  ::debug::Found 1 files to hash.
[pr/lint]   💬  ::debug::followSymbolicLinks 'true'
[pr/lint]   💬  ::debug::implicitDescendants 'true'
[pr/lint]   💬  ::debug::matchDirectories 'true'
[pr/lint]   💬  ::debug::omitBrokenSymbolicLinks 'true'
[pr/lint]   💬  ::debug::Search path '/Users/till/Documents/workspaces/project/pdf-test'
[pr/lint]   💬  ::debug::Processing Poetry project at /Users/till/Documents/workspaces/project/pdf-test/app
[pr/lint]   ❗  ::error::Unable to locate executable file: poetry. Please verify either the file path exists or the file can be found within a directory specified by the PATH environment variable. Also check the file mode to verify the file is executable.
[pr/lint]   ❌  Failure - Main actions/setup-python@v5
[pr/lint]   ⚙  ::set-env:: Python_ROOT_DIR=/opt/hostedtoolcache/Python/3.11.8/x64
[pr/lint]   ⚙  ::set-env:: Python2_ROOT_DIR=/opt/hostedtoolcache/Python/3.11.8/x64
[pr/lint]   ⚙  ::set-env:: Python3_ROOT_DIR=/opt/hostedtoolcache/Python/3.11.8/x64
[pr/lint]   ⚙  ::set-env:: LD_LIBRARY_PATH=/opt/hostedtoolcache/Python/3.11.8/x64/lib
[pr/lint]   ⚙  ::set-env:: pythonLocation=/opt/hostedtoolcache/Python/3.11.8/x64
[pr/lint]   ⚙  ::set-env:: PKG_CONFIG_PATH=/opt/hostedtoolcache/Python/3.11.8/x64/lib/pkgconfig
[pr/lint]   ⚙  ::set-output:: python-version=3.11.8
[pr/lint]   ⚙  ::set-output:: python-path=/opt/hostedtoolcache/Python/3.11.8/x64/bin/python
[pr/lint]   ⚙  ::add-path:: /opt/hostedtoolcache/Python/3.11.8/x64
[pr/lint]   ⚙  ::add-path:: /opt/hostedtoolcache/Python/3.11.8/x64/bin
[pr/lint] exitcode '1': failure

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

9 participants