MAINT, BUG: bump OpenBLAS

[tylerjereddy]

• Fixes BUG: Hang on Windows in scikit-learn with 1.13rc1 and 1.14.dev (maybe due to OpenBLAS 0.3.26 update?) #20294.

• Pull in new OpenBLAS binaries as discussed in the above issue, to solve a bug that affects downstream consumers like scikit-learn. The OpenBLAS binary builds happened via MAINT: build new OpenBLAS for SciPy MacPython/openblas-libs#149 with CI results at links below; the single s390x arch failure should be safe to ignore, we don't distribute binaries for that arch:

  • https://app.travis-ci.com/github/MacPython/openblas-libs/builds/269741262
  • https://github.com/MacPython/openblas-libs/actions/runs/8493017997
  • https://github.com/MacPython/openblas-libs/actions/runs/8493018404

• I had an issue when testing locally with CIBW_BUILD=cp311-* cibuildwheel --platform linux --archs x86_64. Let's see if the regular CI catches anything before I try a full blown wheel build here, hopefully that's just some local issue I have.

  [210/1479] Linking target scipy/special/_ellip_harm_2.cpython-311-x86_64-linux-gnu.so
  FAILED: scipy/special/_ellip_harm_2.cpython-311-x86_64-linux-gnu.so
  cc  -o scipy/special/_ellip_harm_2.cpython-311-x86_64-linux-gnu.so scipy/special/_ellip_harm_2.cpython-311-x86_64-linux-gnu.so.p/meson-generated__ellip_harm_2.c.o scipy/special/_ellip_harm_2.cpython-311-x86_64-linux-gnu.so.p/sf_error.c.o -Wl,--as-needed -Wl,--allow-shlib-undefined -Wl,-O1 -shared -fPIC -Wl,--start-group -lm -Wl,--version-script=/project/scipy/_build_utils/link-version-pyinit.map -L/usr/local/lib '-l$(libprefix}openblas' -Wl,--end-group
  /opt/rh/devtoolset-10/root/usr/libexec/gcc/x86_64-redhat-linux/10/ld: cannot find -l$(libprefix}openblas
  collect2: error: ld returned 1 exit status

[skip circle]

[tylerjereddy]

Ok, the CI replicates my failure, maybe the hash size/format on the new strings then.

[tylerjereddy]

Ah, the shared lib naming scheme probably just changed a bit.

[tylerjereddy]

Current layout from the tools/openblas_support.py download in /tmp/openblas/lib locally on this branch:

cmake  libopenblas.a  libopenblasp-r0.3.26.dev.a  libopenblasp-r0.3.26.dev.so  libopenblas.so  libopenblas.so.0  pkgconfig

Last known working layout from a non-stable release:

cmake  libopenblas.a  libopenblasp-r0.3.21.dev.a  libopenblasp-r0.3.21.dev.so  libopenblas.so  libopenblas.so.0  pkgconfig

I confirmed that reverting to the original versions/hashes before gh-20215 allows cibuildwheel to get past the linking problem. So, will need to dig around a bit to debug that I think, probably upstream somewhere.

[tylerjereddy]

On Windows the error looks like:

c:/rtools40/ucrt64/bin/../lib/gcc/x86_64-w64-mingw32/10.3.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot find -l$(libprefix}openblas_v0.3.26-382-gb1e8ba50-

On local Linux box:

/opt/rh/devtoolset-10/root/usr/libexec/gcc/x86_64-redhat-linux/10/ld: cannot find -l$(libprefix}openblas

Of course git grep -E -i "openblas_support" shows various calls for each wheel build platform.

There are quite a few harsh/version/searching related commits at: https://github.com/MacPython/openblas-libs/commits/scipy/

[tylerjereddy]

On the assumption that the linker stuff gets propagated by pkg-config/meson somehow I compared the Libs: lines in .pc files for a few different versions of the OpenBLAS binary we use (I renamed the files myself for clarity):

openblas_1_12_0.pc:Libs: -L${libdir} -lopenblas${libsuffix}
openblas_1_13_rc1.pc:Libs: -L${libdir} -lopenblas${libsuffix}
openblas_new.pc:Libs: -L${libdir} -l$(libprefix}openblas${libnamesuffix}

Probably not a coincidence that there's a change in substitution pattern at the first observed failure (last line above/version used in this branch).

There are some suffix changes recently upstream here:
MacPython/openblas-libs@9c64109

[larsoner]

Maybe a stupid question but in your earlier comment is it really correct to have cannot find -l$(libprefix}openblas with a $( money-open-paren to start and a } close-brace to end? That seems like a typo but maybe that's just some gcc naivete from my end in which case feel free to ignore!

[tylerjereddy]

I think you may be right. The smoke seems to be coming from the pkg-config *.pc file and there are at least two things that have changed:

1. The general substitution pattern.
2. A likely-invalid syntax usage.

[tylerjereddy]

I've initiated an initial attempt to rebuild the OpenBLAS binaries upstream with this patch: MacPython/openblas-libs@082affc

Wheel builds are at:

• Windows OpenBLAS: https://github.com/MacPython/openblas-libs/actions/runs/8499665681
• Posix OpenBLAS: https://github.com/MacPython/openblas-libs/actions/runs/8499664684
• Linux OpenBLAS alternative archs: https://app.travis-ci.com/github/MacPython/openblas-libs/builds/269743876

This assumes that the uploads will be able to overwrite the previous binaries.

[tylerjereddy]

Although my attempt to fix this upstream failed miserably (the .pc files were completely unchanged), adding a shim to our own openblas_support.py to restore the old formatting of openblas.pc was successful for me locally on x86_64 Linux with cibuildwheel testing.

So, I'll flush the CI to see if the Windows version is also happy here, and if so then I'll try full blown wheel builds here after that. Perhaps the increasingly-ugly openblas_support.py shims can be forgiven given our impending migration away from that (gh-20074) to the wheels infrastructure anyway.

[tylerjereddy]

The only failing CI check here now is unrelated (gh-20365). So, I'll push in an empty commit to probe the wheel builds proper with the new OpenBLAS + shims in this branch.

[tylerjereddy]

Summarizing current status here, after solving some of the initial problems with shims in this PR branch:

1. 2 CI failures are coming from MacOS arm64 wheel builds, which fail for Python 3.10 and 3.11 only. I note that MacOS M1 GHA runners are still in beta, and the failures do not appear to be related to my shims here at first glance. In short, pip install pipx fails with No such file or directory: '/usr/local/opt/pipx/venvs'. Something looks weird here, in the cp311 matrix entry, I see a 3.12 version of Python: /Users/runner/hostedtoolcache/Python/3.12.2/arm64/bin/python" -m pip install pipx. Same for the 3.10 failure. Python 3.9 and 3.12 (which both pass), use Python 3.11.8 to do the initial pipx install, so maybe the mismatch isn't a problem upfront before the the real cibuildwheel work starts, but the inconsistency is still a bit weird to read.
2. All 4 musllinux x86_64 wheel build jobs fail at python tools/openblas_support.py because their download target URLs don't seem to exist. The 3.9 example is: Could not download "https://anaconda.org/multibuild-wheels-staging/openblas-libs/v0.3.26-382-gb1e8ba50/download/openblas-v0.3.26-382-gb1e8ba50-musllinux_1_1_x86_64.tar.gz". The last time this job succeeded, for 1.13.0rc1, the URL was: https://anaconda.org/multibuild-wheels-staging/openblas-libs/v0.3.26/download/openblas-v0.3.26-musllinux_1_1_x86_64.tar.gz. I'm not sure what's going on with ILP64 stuff vs. other prefix/postfix naming work, but the closest I see at the moment is: https://anaconda.org/multibuild-wheels-staging/openblas-libs/v0.3.26-382-gb1e8ba50/download/openblas64_-v0.3.26-382-gb1e8ba50-musllinux_1_1_x86_64.tar.gz. I suppose we could add the trailing 64_ here if we're comfortable with that and it doesn't mean something else that we don't want?

Certainly it is clear why I don't like to frequently try bumping OpenBLAS, I guess some upstream changes have already made things trickier between the fairly recent last bump for the non-wheel infra.

[rgommers]

2. I suppose we could add the trailing 64_ here if we're comfortable with that and it doesn't mean something else that we don't want?

No, those are the ILP64 builds, they're not usable for SciPy. We need the ones without 64 in the name.

[rgommers]

The musllinux problem is because the builds weren't uploaded (see CI log) due to:

 Error:  ('Service Unavailable: The server cannot process the request due to a high load ([GET] https://api.anaconda.org/package/multibuild-wheels-staging/openblas-libs -> 503)', 503)

Just rerunning the failed builds on that job should fix the problem. (side note: I don't have permissions to do that, could I be added @mattip?)

[rgommers]

The macOS arm64 failures look like a cibuildwheel problem that depends on the runner image version:

• Jobs with macOS 14.2.1 and runner image version 20240116.1 (which has Python 3.11.8) pass
• Jobs with macOS 14.4 and runner image version 20240325.1 (which has Python 3.12.2) fail

The runners get picked at random, a re-run made one of the two jobs pass. The second one I retriggered three times, but no luck getting the older runner image.

[rgommers]

@tylerjereddy I bet an upgrade to cibuildwheel 2.17.0 will fix the macOS issue. We're still using 2.16.5, and that only has preliminary support, 2.17.0 has official support and contains a bunch of fixes: https://github.com/pypa/cibuildwheel#v2170

[tylerjereddy]

The musllinux problem is because the builds weren't uploaded (see CI log) due to:

Ah, I ignored that because the job had already passed in the previous iteration, but now I see not only did it fail to upload, it also deleted the correct asset before it did:

Warning: Distribution "openblas-v0.3.26-382-gb1e8ba50-musllinux_1_1_x86_64.tar.gz" already exists. Removing.

[tylerjereddy]

Just rerunning the failed builds on that job should fix the problem. (side note: I don't have permissions to do that, could I be added @mattip?)

@rgommers Thanks for the suggestions, I added you as an admin on that repo, so that should do the trick. Once the job that failed to upload completes I'll push a cibuildwheel version bump and flush wheel builds again here.

[tylerjereddy]

Upstream CI upload seems fixed now, and I'm retrying the wheel builds with the latest cibuildwheel stable release to see if things are green yet.

[tylerjereddy]

I think the one Cirrus Linux ARM job just timed out at the 1 hour mark exactly, and I believe we've seen that before on occasion and is not related to any changes here. The rest of the CI/wheel builds are green so I may proceed with squash-merging.

[tylerjereddy]

The shims in openblas_support are a bit hacky, but seem to get the job done for now.