-
Notifications
You must be signed in to change notification settings - Fork 21.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
PyTorch 1.13 cannot be installed using poetry
on Mac M1
#88049
Comments
This issue maybe related to pypi repo ? I just downloaded the M1 version of wheel from pypi:
I see following Metadata requirements:
No mentions of the
These are Linux specific requirements that should not be used in Mac M1 |
Dependencies to linux package should have |
FWIW, this also affects Macs with Intel chips because (at a minimum), nvidia-cuda-nvrtc-cu11 does not supply any Mac wheels on PyPI. Hence, Poetry goes out and finds nothing and throws a RuntimeError about installing torch 1.13.0. Specifying the |
Due to this issue: pytorch/pytorch#88049
Due to this issue: pytorch/pytorch#88049
The issue here is that Poetry does not support split metadata across wheels; instead, markers should be used with unified metadata. This is not just a Poetry decision; PyPI's metadata API currently gathers metadata from the first distfile to be uploaded. Until such time that PEP 658 is implemented in PyPI, having inconsistent metadata between distfiles is likely to confuse non-Pip tools. |
|
I was really looking to get the improved M1 support in 1.13. For now, my work-around is to hard-code a path to the wheel in
It seems that the PyPi API doesn't support different requirements on different platforms. |
I submitted #88826 to address this for future releases. 🤞 It doesn't solve the larger issue @neersighted mentioned of split metadata across wheels, but should at least have the platform marker. I guess the opposite problem of Poetry inspecting metadata for one of the non-Linux wheels and thus omitting these Linux deps on Linux systems might be possible (I'm not sure if the order the wheels are picked is deterministic?). |
It's based on the order we get them back from PyPI (and PyPI uses upload order iirc; this is modeled on the fact that the PyPI JSON API uses the first artifact to be uploaded). |
this worked for me |
…8826) Fixes pytorch#88049 pytorch#85097 added new extra dependencies on `nvidia-*`. They are linux (GPU) only packages, but were not marked as such, causing issues installing pytorch 1.13 via Poetry (and possibly other tools that follow PyPI's metadata API) on non-Linux systems. This "fixes" the issue by adding the `; platform_system = 'Linux'` marker on these dependencies, but the main problem of different metadata for different wheels is a [somewhat larger issue](pytorch#88049 (comment)). pytorch#85097 used `;` as a delimiter for splitting the different deps, but that is the delimiter used in markers, so I changed to split on `|`. Pull Request resolved: pytorch#88826 Approved by: https://github.com/neersighted, https://github.com/lalmei, https://github.com/malfet
…8826) Fixes pytorch#88049 pytorch#85097 added new extra dependencies on `nvidia-*`. They are linux (GPU) only packages, but were not marked as such, causing issues installing pytorch 1.13 via Poetry (and possibly other tools that follow PyPI's metadata API) on non-Linux systems. This "fixes" the issue by adding the `; platform_system = 'Linux'` marker on these dependencies, but the main problem of different metadata for different wheels is a [somewhat larger issue](pytorch#88049 (comment)). pytorch#85097 used `;` as a delimiter for splitting the different deps, but that is the delimiter used in markers, so I changed to split on `|`. Pull Request resolved: pytorch#88826 Approved by: https://github.com/neersighted, https://github.com/lalmei, https://github.com/malfet
…89924) Fixes #88049 #85097 added new extra dependencies on `nvidia-*`. They are linux (GPU) only packages, but were not marked as such, causing issues installing pytorch 1.13 via Poetry (and possibly other tools that follow PyPI's metadata API) on non-Linux systems. This "fixes" the issue by adding the `; platform_system = 'Linux'` marker on these dependencies, but the main problem of different metadata for different wheels is a [somewhat larger issue](#88049 (comment)). #85097 used `;` as a delimiter for splitting the different deps, but that is the delimiter used in markers, so I changed to split on `|`. Pull Request resolved: #88826 Approved by: https://github.com/neersighted, https://github.com/lalmei, https://github.com/malfet Co-authored-by: Jacob Hayes <jacob.r.hayes@gmail.com>
Met same issue, was trying based on some solutions in this post: [tool.poetry.dependencies]
torch = {version = "1.13.0", source = "pytorch"}
[[tool.poetry.source]]
name = "pytorch"
url = "https://download.pytorch.org/whl"
default = false
secondary = true but no luck. I end up with a solution similar to @mawicks : [tool.poetry.dependencies]
python = "3.10.x"
torch = {url = "https://download.pytorch.org/whl/cpu/torch-1.13.0-cp310-none-macosx_11_0_arm64.whl"} You can find all versions at either one of (No support for Python 3.11 of macOS with M1 chip (arm64) at the time of writing this post) |
Seems like the fix has been added to torch 1.13.1, which right now is in release candidate stage merge of the fix into torch: |
…8826) Fixes pytorch#88049 pytorch#85097 added new extra dependencies on `nvidia-*`. They are linux (GPU) only packages, but were not marked as such, causing issues installing pytorch 1.13 via Poetry (and possibly other tools that follow PyPI's metadata API) on non-Linux systems. This "fixes" the issue by adding the `; platform_system = 'Linux'` marker on these dependencies, but the main problem of different metadata for different wheels is a [somewhat larger issue](pytorch#88049 (comment)). pytorch#85097 used `;` as a delimiter for splitting the different deps, but that is the delimiter used in markers, so I changed to split on `|`. Pull Request resolved: pytorch#88826 Approved by: https://github.com/neersighted, https://github.com/lalmei, https://github.com/malfet
I am using mac for local development and a docker for publishing. The following allowed me to use a single pyproject file for both, hope it will help someone:
|
Torch 1.13 has some weird extra NVIDIA dependencies, described [here] (pytorch/pytorch#88049), which don't seem to recognize that cuda is loaded as a module. So as it currently stands, you need to install top-bottom and then uninstall those nvidia packages, and then maybe muck around a bit more so that everything points to the right place. I don't see any issue with pinning torch 1.12 for now.
2023 current macOS platform name is >>> import sys
>>> sys.platform
'linux'
>>> import sys
>>> sys.platform
'darwin' # macos m1
so it will be better work in local macOS m1 ( [tool.poetry.dependencies]
python = "3.10.9"
torch = [
{ markers = "sys_platform == 'darwin'", url = "https://download.pytorch.org/whl/cpu/torch-1.13.1-cp310-none-macosx_11_0_arm64.whl#sha256=0122806b111b949d21fa1a5f9764d1fd2fcc4a47cb7f8ff914204fd4fc752ed5" },
{ markers = "sys_platform == 'linux'", url = "https://download.pytorch.org/whl/cpu/torch-1.13.1%2Bcpu-cp310-cp310-linux_x86_64.whl#sha256=11692523b87c45b79ddfb5148b12a713d85235d399915490d94e079521f7e014" }
] |
The issue was closed as completed, but it seems, both from comments on this issue and my experience, that this issue remains. What's the recommended way to avoid installing CUDA related dependencies on MacOS but installing them on Linux?
Is it possible to specify the URL for MacOS but use the default on every other platform, as this problem seems to only be apparent on MacOS? |
Follow up on: #102604 Address this comment: #102604 (comment) Whl metadata for all wheels published to pypi must match, otherwise poetry install will fail see this comment: #88049 (comment) Pull Request resolved: #121014 Approved by: https://github.com/malfet
Follow up on: pytorch#102604 Address this comment: pytorch#102604 (comment) Whl metadata for all wheels published to pypi must match, otherwise poetry install will fail see this comment: pytorch#88049 (comment) Pull Request resolved: pytorch#121014 Approved by: https://github.com/malfet
🐛 Describe the bug
This is an installation issue.
Poetry parses the endpoint at
https://pypi.org/pypi/torch/1.13/json
to get dependency metadata. The endpoint returns:However, the packages
nvidia-cuda-runtime-cu11
,nvidia-cudnn-cu11
,nvidia-cuda-nvrtc-cu11
, andnvidia-cuda-nvrtc-cu11
do not exist for M1, andpoetry update
fails withThis is specific to v1.13. This problem did not exist in v1.12.1
Versions
This probably isn't relevant until after it's installed, but I get:
cc @ezyang @seemethere @malfet
The text was updated successfully, but these errors were encountered: