Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG(?): Symbol not found: _cblas_caxpy$NEWLAPACK #25026

Closed
mdhaber opened this issue Oct 29, 2023 · 15 comments · Fixed by #25040
Closed

BUG(?): Symbol not found: _cblas_caxpy$NEWLAPACK #25026

mdhaber opened this issue Oct 29, 2023 · 15 comments · Fixed by #25040
Labels

Comments

@mdhaber
Copy link
Contributor

mdhaber commented Oct 29, 2023

Describe the issue:

On my M1 mac, I can build commit d10d17e and import NumPy without issue. With the next commit, 34afbb9, the build seems to be successful, but I get an error when I try to import numpy at the Python prompt.

I get the full error message below when I use an editible install and try to import NumPy. When I use spin test -v, the build seems OK, but it fails with the same ImportError.

In case it matters, I use Miniforge mamba to set up the numpy-dev environment. This is on a fresh system; I did a factory reset in my attempt to debug this. I've read through https://numpy.org/devdocs/user/troubleshooting-importerror.html as recommended and couldn't find a solution there.

Update: I'm upgrading from Ventura to Sonoma to see if that helps.

Reproduce the code example:

import numpy

Error message:

Traceback (most recent call last):
  File "/Users/matthaberland/numpy/numpy/core/__init__.py", line 24, in <module>
    from . import multiarray
  File "/Users/matthaberland/numpy/numpy/core/multiarray.py", line 10, in <module>
    from . import overrides
  File "/Users/matthaberland/numpy/numpy/core/overrides.py", line 8, in <module>
    from numpy.core._multiarray_umath import (
ImportError: dlopen(/Users/matthaberland/numpy/build/cp39/numpy/core/_multiarray_umath.cpython-39-darwin.so, 0x0002): Symbol not found: _cblas_caxpy$NEWLAPACK
  Referenced from: <47F947F4-FE38-32EE-BDCE-DBA4188AAD7D> /Users/matthaberland/numpy/build/cp39/numpy/core/_multiarray_umath.cpython-39-darwin.so
  Expected in:     <E6C2D470-8878-372D-B9DB-0EDCA1303272> /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/matthaberland/numpy/numpy/__init__.py", line 114, in <module>
    from numpy.__config__ import show as show_config
  File "/Users/matthaberland/numpy/build/cp39/numpy/__config__.py", line 4, in <module>
    from numpy.core._multiarray_umath import (
  File "/Users/matthaberland/numpy/numpy/core/__init__.py", line 50, in <module>
    raise ImportError(msg)
ImportError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.9 from "/Users/matthaberland/miniforge3/envs/numpy-dev/bin/python"
  * The NumPy version is: "2.0.0.dev0+git20231007.f204c02"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: dlopen(/Users/matthaberland/numpy/build/cp39/numpy/core/_multiarray_umath.cpython-39-darwin.so, 0x0002): Symbol not found: _cblas_caxpy$NEWLAPACK
  Referenced from: <47F947F4-FE38-32EE-BDCE-DBA4188AAD7D> /Users/matthaberland/numpy/build/cp39/numpy/core/_multiarray_umath.cpython-39-darwin.so
  Expected in:     <E6C2D470-8878-372D-B9DB-0EDCA1303272> /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/matthaberland/numpy/numpy/__init__.py", line 119, in <module>
    raise ImportError(msg) from e
ImportError: Error importing numpy: you should not try to import numpy from
        its source directory; please exit the numpy source tree, and relaunch
        your python interpreter from there.

Runtime information:

>>> import sys, numpy; print(numpy.__version__); print(sys.version); print(numpy.show_runtime())
2.0.0.dev0+git20231010.d10d17e
3.9.18 | packaged by conda-forge | (main, Aug 30 2023, 03:53:08) 
[Clang 15.0.7 ]
WARNING: `threadpoolctl` not found in system! Install it by `pip install threadpoolctl`. Once installed, try `np.show_runtime` again for more detailed build information
[{'numpy_version': '2.0.0.dev0+git20231010.d10d17e',
  'python': '3.9.18 | packaged by conda-forge | (main, Aug 30 2023, '
            '03:53:08) \n'
            '[Clang 15.0.7 ]',
  'uname': uname_result(system='Darwin', node='Matts-MacBook-Air.local', release='22.1.0', version='Darwin Kernel Version 22.1.0: Sun Oct  9 20:14:30 PDT 2022; root:xnu-8792.41.9~2/RELEASE_ARM64_T8103', machine='arm64')},
 {'simd_extensions': {'baseline': ['NEON', 'NEON_FP16', 'NEON_VFPV4', 'ASIMD'],
                      'found': ['ASIMDHP'],
                      'not_found': ['ASIMDFHM']}}]
None
@mattip
Copy link
Member

mattip commented Oct 29, 2023

Does conda list show any openblas alternatives? Can you attach the meson build log for the failed build? The commit you point to is a massive refactor of the way BLAS is detected, perhaps something in this particular build configuration is off.

@mdhaber
Copy link
Contributor Author

mdhaber commented Oct 29, 2023

Output of conda list:

# Name                    Version                   Build  Channel
accessible-pygments       0.0.4              pyhd8ed1ab_0    conda-forge
alabaster                 0.7.13             pyhd8ed1ab_0    conda-forge
appnope                   0.1.3              pyhd8ed1ab_0    conda-forge
asttokens                 2.4.1              pyhd8ed1ab_0    conda-forge
attrs                     23.1.0             pyh71513ae_1    conda-forge
babel                     2.13.1             pyhd8ed1ab_0    conda-forge
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                pyhd8ed1ab_3    conda-forge
backports.functools_lru_cache 1.6.5              pyhd8ed1ab_0    conda-forge
backports.zoneinfo        0.2.1            py39h2804cbe_8    conda-forge
beautifulsoup4            4.12.2             pyha770c72_0    conda-forge
breathe                   4.35.0             pyhd8ed1ab_1    conda-forge
brotli-python             1.1.0            py39hb198ff7_1    conda-forge
bzip2                     1.0.8                h3422bc3_4    conda-forge
c-compiler                1.6.0                hd291e01_0    conda-forge
ca-certificates           2023.7.22            hf0a4a13_0    conda-forge
ccache                    4.8.1                hbf08be0_0    conda-forge
cctools                   973.0.1             hd1ac623_15    conda-forge
cctools_osx-arm64         973.0.1             h2a25c60_15    conda-forge
certifi                   2023.7.22          pyhd8ed1ab_0    conda-forge
cffi                      1.16.0           py39he153c15_0    conda-forge
charset-normalizer        3.3.1              pyhd8ed1ab_0    conda-forge
clang                     15.0.7               hce30654_3    conda-forge
clang-15                  15.0.7          default_h5dc8d65_3    conda-forge
clang_impl_osx-arm64      15.0.7               h77e971b_6    conda-forge
clang_osx-arm64           15.0.7               h54d7cd3_6    conda-forge
clangxx                   15.0.7          default_h610c423_3    conda-forge
clangxx_impl_osx-arm64    15.0.7               h768a7fd_6    conda-forge
clangxx_osx-arm64         15.0.7               h77e971b_6    conda-forge
click                     8.1.7           unix_pyh707e725_0    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
compiler-rt               15.0.7               hf8d1dfb_1    conda-forge
compiler-rt_osx-arm64     15.0.7               hf8d1dfb_1    conda-forge
compilers                 1.6.0                hce30654_0    conda-forge
coverage                  7.3.2            py39h0f82c59_0    conda-forge
cxx-compiler              1.6.0                h1995070_0    conda-forge
cython                    3.0.4            py39h4ce5507_0    conda-forge
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
docutils                  0.19             py39h2804cbe_1    conda-forge
doxygen                   1.9.8                h0e2417a_0    conda-forge
exceptiongroup            1.1.3              pyhd8ed1ab_0    conda-forge
execnet                   2.0.2              pyhd8ed1ab_0    conda-forge
executing                 1.2.0              pyhd8ed1ab_0    conda-forge
fortran-compiler          1.6.0                h5a50232_0    conda-forge
gettext                   0.21.1               h0186832_0    conda-forge
gfortran                  12.3.0               h1ca8e4b_1    conda-forge
gfortran_impl_osx-arm64   12.3.0               hbbb9e1e_1    conda-forge
gfortran_osx-arm64        12.3.0               h57527a5_1    conda-forge
gitdb                     4.0.11             pyhd8ed1ab_0    conda-forge
gitpython                 3.1.40             pyhd8ed1ab_0    conda-forge
gmp                       6.2.1                h9f76cd9_0    conda-forge
hypothesis                6.88.1             pyha770c72_0    conda-forge
icu                       73.2                 hc8870d7_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
imagesize                 1.4.1              pyhd8ed1ab_0    conda-forge
importlib-metadata        6.8.0              pyha770c72_0    conda-forge
iniconfig                 2.0.0              pyhd8ed1ab_0    conda-forge
ipython                   8.16.1             pyh31c8845_0    conda-forge
isl                       0.25                 h9a09cb3_0    conda-forge
jedi                      0.19.1             pyhd8ed1ab_0    conda-forge
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
ld64                      609                 h89fa09d_15    conda-forge
ld64_osx-arm64            609                 hc4dc95b_15    conda-forge
libclang-cpp15            15.0.7          default_h5dc8d65_3    conda-forge
libcxx                    16.0.6               h4653b0c_0    conda-forge
libffi                    3.4.2                h3422bc3_5    conda-forge
libgfortran               5.0.0           13_2_0_hd922786_1    conda-forge
libgfortran-devel_osx-arm64 12.3.0               hc62be1c_1    conda-forge
libgfortran5              13.2.0               hf226fd6_1    conda-forge
libglib                   2.78.0               h24e9cb9_0    conda-forge
libhiredis                1.0.2                hbec66e7_0    conda-forge
libiconv                  1.17                 he4db4b2_0    conda-forge
libllvm15                 15.0.7               h504e6bf_3    conda-forge
libopenblas               0.3.24          openmp_hd76b1f2_0    conda-forge
libsqlite                 3.43.2               h091b4b1_0    conda-forge
libxml2                   2.11.5               h25269f3_1    conda-forge
libzlib                   1.2.13               h53f4e23_5    conda-forge
llvm-openmp               17.0.3               hcd81f8e_0    conda-forge
llvm-tools                15.0.7               h504e6bf_3    conda-forge
markupsafe                2.1.3            py39h0f82c59_1    conda-forge
matplotlib-inline         0.1.6              pyhd8ed1ab_0    conda-forge
meson                     1.2.3              pyhd8ed1ab_0    conda-forge
meson-python              0.15.0             pyh0c530f3_0    conda-forge
mpc                       1.3.1                h91ba8db_0    conda-forge
mpfr                      4.2.1                h9546428_0    conda-forge
mypy                      1.5.1            py39h0f82c59_1    conda-forge
mypy_extensions           1.0.0              pyha770c72_0    conda-forge
ncurses                   6.4                  h7ea286d_0    conda-forge
ninja                     1.11.1               hffc8910_0    conda-forge
nomkl                     1.0                  h5ca1d4c_0    conda-forge
numpy                     2.0.0.dev0               pypi_0    pypi
numpydoc                  1.4.0              pyhd8ed1ab_1    conda-forge
openblas                  0.3.24          openmp_hce3e5ba_0    conda-forge
openssl                   3.1.4                h0d3ecfb_0    conda-forge
packaging                 23.2               pyhd8ed1ab_0    conda-forge
parso                     0.8.3              pyhd8ed1ab_0    conda-forge
pcre2                     10.40                hb34f9b4_0    conda-forge
pexpect                   4.8.0              pyh1a96a4e_2    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pip                       23.3.1             pyhd8ed1ab_0    conda-forge
pkg-config                0.29.2            hab62308_1008    conda-forge
pluggy                    1.3.0              pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.39             pyha770c72_0    conda-forge
prompt_toolkit            3.0.39               hd8ed1ab_0    conda-forge
psutil                    5.9.5            py39h0f82c59_1    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
pycodestyle               2.7.0              pyhd8ed1ab_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pydata-sphinx-theme       0.13.3             pyhd8ed1ab_0    conda-forge
pygments                  2.16.1             pyhd8ed1ab_0    conda-forge
pyproject-metadata        0.7.1              pyhd8ed1ab_0    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
pytest                    7.4.3              pyhd8ed1ab_0    conda-forge
pytest-cov                4.1.0              pyhd8ed1ab_0    conda-forge
pytest-xdist              3.3.1              pyhd8ed1ab_0    conda-forge
python                    3.9.18          hfa1ae8a_0_cpython    conda-forge
python_abi                3.9                      4_cp39    conda-forge
pytz                      2023.3.post1       pyhd8ed1ab_0    conda-forge
readline                  8.2                  h92ec313_1    conda-forge
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
setuptools                59.2.0           py39h2804cbe_0    conda-forge
sigtool                   0.1.3                h44b9a77_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
smmap                     5.0.0              pyhd8ed1ab_0    conda-forge
snowballstemmer           2.2.0              pyhd8ed1ab_0    conda-forge
sortedcontainers          2.4.0              pyhd8ed1ab_0    conda-forge
soupsieve                 2.5                pyhd8ed1ab_1    conda-forge
sphinx                    6.2.1              pyhd8ed1ab_0    conda-forge
sphinx-design             0.5.0              pyhd8ed1ab_0    conda-forge
sphinxcontrib-applehelp   1.0.7              pyhd8ed1ab_0    conda-forge
sphinxcontrib-devhelp     1.0.5              pyhd8ed1ab_0    conda-forge
sphinxcontrib-htmlhelp    2.0.4              pyhd8ed1ab_0    conda-forge
sphinxcontrib-jsmath      1.0.1              pyhd8ed1ab_0    conda-forge
sphinxcontrib-qthelp      1.0.6              pyhd8ed1ab_0    conda-forge
sphinxcontrib-serializinghtml 1.1.9              pyhd8ed1ab_0    conda-forge
spin                      0.7             unix_pyh707e725_0    conda-forge
stack_data                0.6.2              pyhd8ed1ab_0    conda-forge
tapi                      1100.0.11            he4954df_0    conda-forge
tk                        8.6.13               hb31c410_0    conda-forge
toml                      0.10.2             pyhd8ed1ab_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
traitlets                 5.12.0             pyhd8ed1ab_0    conda-forge
typing_extensions         4.8.0              pyha770c72_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
urllib3                   2.0.7              pyhd8ed1ab_0    conda-forge
wcwidth                   0.2.8              pyhd8ed1ab_0    conda-forge
wheel                     0.41.2             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h57fd34a_0    conda-forge
zipp                      3.17.0             pyhd8ed1ab_0    conda-forge
zlib                      1.2.13               h53f4e23_5    conda-forge
zstd                      1.5.5                h4f39d0f_0    conda-forge

Log is at https://gist.github.com/mdhaber/cad25624c63a15fa9c392716799b4657

I was planning to update macOS to Sonoma. LMK if I should hold off.

@mattip
Copy link
Member

mattip commented Oct 29, 2023

The relevant part is this, your build is using a version of Accelerate. @rgommers might know what is going on.

Looking for framework Accelerate in /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks
Running compile:
Working directory:  /Users/matthaberland/Desktop/numpy/build/cp39/meson-private/tmpuvzn118o
Code:
 int main(void) { return 
0; }
-----------
Command line: `arm64-apple-darwin20.0.0-clang++ -L/Users/matthaberland/miniforge3/envs/numpy-dev/lib /Users/matthaberland/Desktop/numpy/build/cp39/meson-private/tmpuvzn118o/testfile.cpp -o /Users/matthaberland/Desktop/numpy/build/cp39/meson-private/tmpuvzn118o/output.exe -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe -stdlib=libc++ -fvisibility-inlines-hidden -fmessage-length=0 -isystem /Users/matthaberland/miniforge3/envs/numpy-dev/include -D_FORTIFY_SOURCE=2 -isystem /Users/matthaberland/miniforge3/envs/numpy-dev/include -O0 -fpermissive -Werror=implicit-function-declaration -F/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks -framework Accelerate -Wl,-pie -Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs -Wl,-rpath,/Users/matthaberland/miniforge3/envs/numpy-dev/lib` -> 0
Run-time dependency accelerate found: YES
../../numpy/meson.build:119: WARNING: Project targets '>=1.2.99' but uses feature introduced in '1.3.0': dep 'accelerate' custom lookup.
Message: BLAS symbol suffix: $NEWLAPACK
Program _build_utils/process_src_template.py found: YES (/Users/matthaberland/miniforge3/envs/numpy-dev/bin/python /Users/matthaberland/Desktop/numpy/numpy/_build_utils/process_src_template.py)
Program _build_utils/tempita.py found: YES (/Users/matthaberland/Desktop/numpy/numpy/_build_utils/tempita.py)

@mdhaber
Copy link
Contributor Author

mdhaber commented Oct 29, 2023

Yes, that's why I was thinking of updating my OS. I saw that 34afbb9 is just tested with macOS 13.3+. I currently have 13.0 after the factory reset. Not sure what I had before, though.

@andyfaff
Copy link
Contributor

The following works for me:

git fetch numpy
git checkout main
git rebase numpy/main
git submodule update
conda create -n numpydev python=3.9
conda activate numpydev
pip install -r build_requirements.txt     # install requirements with pip
pip install -r test_requirements.txt 
spin build    # automatically uses Accelerate
spin test

On macOS 14.0 with Xcode 15 and gfortran12 from https://github.com/fxcoudert/gfortran-for-macOS/releases. I wonder if it's a consequence of using the conda compilers?

@rgommers
Copy link
Member

I was planning to update macOS to Sonoma. LMK if I should hold off.

I saw that 34afbb9 is just tested with macOS 13.3+. I currently have 13.0 after the factory reset. Not sure what I had before, though.

@mdhaber yes it'd be great if you could hold off on upgrading to 14.0 to see if we can sort this out. This is the first time a bug against building with Accelerate was reported, and I'd like to get to the bottom of it.

The way this is supposed to work is that Accelerate is only detected for macOS >=13.3 and if XCode is new enough so the relevant SDK is installed. This is tested during the build with:

xcrun -sdk macosx --show-sdk-version

so the output of that on your local system is relevant.

Conda env/compilers yes/no should not be relevant here. The build log output for this Accelerate detection looks as expected. It's kinda unfortunate that you don't know what OS version you were on before. Can you still reproduce the issue? If so, can you check:

  • OS version
  • XCode version
  • output of xcrun command above?

@mdhaber
Copy link
Contributor Author

mdhaber commented Oct 30, 2023

Yes, I can still reproduce the issue.
OS: Ventura 13.0
XCode: How do you want me to check independently? Here is the output of a few commands I found online.

> pkgutil --pkg-info=com.apple.pkg.CLTools_Executables | grep version
version: 14.3.1.0.1.1683849156
> xcode-select --version
xcode-select version 2396.
> softwareupdate --history
Command Line Tools for Xcode                       14.3       10/28/2023, 16:38:53  

xcrun output: 13.3

@rgommers
Copy link
Member

Thanks, that helps. This is kinda the opposite problem of #24053 (comment), which was "OS is 13.3, but SDK is older". What is happening here is that your OS version is 13.0 but you do have an SDK for 13.3, which is picked as the default. Hence, a binary is built that will run on 13.3 - but not on 13.0. I didn't realize that this was a thing, I thought the default would always be the current OS version (and using MACOSX_DEPLOYMENT_TARGET to target an older version, but never a newer version). It's kinda weird that the default picked by the Apple tooling here is something that results in binary that won't run on the system on which it is built.

We're going to need an extra safeguard for this. Just checking: does sw_vers --productVersion return 13.0 for you?

@rgommers
Copy link
Member

Ah, why this "SDK too new" normally isn't really a problem is because the default deployment target indeed is "current OS version" (see https://opensource.apple.com/source/cctools/cctools-667.10.0/libstuff/macosx_deployment_target.c.auto.html) and then using a header will hide the too-new stuff. But in this case we're not actually using the Accelerate.h header at all; rather we're using npy_cblas.h which is a portable-across-blas-libs header. So we won't notice that the NEWLAPACK-suffixed symbols are not present.

@rgommers
Copy link
Member

rgommers commented Oct 30, 2023

@mdhaber I think this change will fix it: rgommers/meson@1435e7e. Could you try that? It should have the effect of not selecting Accelerate when you're on <13.3, and once you upgrade to 13.3 or later it will pick it up (and then it'll work).

@rgommers
Copy link
Member

Updated commit linked above to also include MACOSX_DEPLOYMENT_TARGET.

@mdhaber
Copy link
Contributor Author

mdhaber commented Oct 31, 2023

sw_vers -productVersion (single dash) returns 13.0, yes. And with that change (single dash), it's working!

@mdhaber
Copy link
Contributor Author

mdhaber commented Oct 31, 2023

While we're at it, Testing Builds says to python -m pip install -r test_requirements.txt, but if the user just created a conda environment from environment.yml, those packages are already installed. Should the instructions branch between conda and virtualenv like the "Building SciPy from Source" do?

@rgommers
Copy link
Member

Should the instructions branch between conda and virtualenv like the "Building SciPy from Source" do?

Yes, I planned to rewrite basically all the docs, and mostly copying the structure SciPy has because that is much clearer and more comprehensive.

@rgommers
Copy link
Member

Figured out there's no need to call sw_vers, we can use platform.mac_ver() instead. gh-25040 should close this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants