[MNT]: Numpy 2.0 support strategy

[timhoffm]

Summary

Numpy will release version 2.0 around end of December 2023 with substantial API clearnup: numpy/numpy#24300

There are already some releated PRs:
#26762, #26756, #26706, #26687, #26664, #26566

This ticket should serve two purposes:

1. Define the migration strategy
   NumPy 2.0 development status & announcements numpy/numpy#24300 (comment) has "Key guidance for users and downstream package authors"
   Are we heavy users that we have to pin <2 for now and fix everything once they have their RC out? Or could we fix everything already now (ideally still for 3.8.0)? Note that the pinning is also only a bandaid and package managers may choose to resolve to numpy=2 and matplotlib=3.7.x (which is still unpinned).
2. Track the progress and open steps

[ksunden]

Yeah, unfortunately even we were a bit late on the "pin to <2" train, though not sure there was anything that could have been done to prevent having some version that was not pinned and thus purported to be compatible when it isn't. We did end up pinning for 3.7.3 and 3.8.0, though <=3.7.2 is unpinned.

For us, I think the biggest things left to change are that they are willing/expecting ABI to break, and thus wheels made with np<2 will not work with np==2. (though my understanding is wheels made with np==2 will work with np<2)

These changes have not happened yet, though some precursors have, which on original implementation broke mpl, but were reworked such that they didn't with relative ease. (See numpy/numpy#24634)

I think the changes to the python API are largely taken care of now, though there may be a few that come in later. There was an ambitious initial proposal that would have had a lot more requiring change, but that got pared down significantly. And I suspect our usage of the C API will not be too affected either, other than requiring re-compiling (though I have some reserves about pybind11 requiring changes, which may slow it down from just "recompile").

While continuing to monitor and watch for things that break us, I think the biggest thing is to be ready to do a patch release compiled with np 2.0 (possibly their RC, to allow for timing/etc) in coordination with the numpy release, such that users have a wheel to install that works immediately.

The advise I got was to not pin on main, but pin on release branches.

Luckily(?) we are used in numpy CI, so there is a reasonable chance that at least at the point where the first actual breakage happens, we are likely to know at the PR stage. (Though they may end up having to install compiled against their own numpy for CI at that point, though we will still find out if recompiling is not enough, I guess.)

Also see #26422, which I opened when the initial idea of much more significantly paring down the main np namespace was proposed, and found where we used all of the possibly removed functions. I closed this when all of the items seemed to be taken care of/dropped from the list of removals, but it only originally covered the main namespace and then they moved on to the ndarray class namespace cleanups (which I think are less broad than even the pared down main namespace cleanup, and I have not seen any list of further planned but not implemented changes there, though that doesn't mean they can't happen)

[timhoffm]

Thanks for the detailed answer. To sum up:

• We need to sync a release with numpy=2 because we need to compile for it.
• We have done what we can for now.
• We are an in the loop for potential additional problems that need to be addressed on our side.

[scottshambaugh]

FWIW (it looks like someone had already done this but I didn't see it in the linked tickets), I ran the ruff tool as recommended by https://numpy.org/devdocs/numpy_2_0_migration_guide.html against the main branch and it came back clean.
ruff check path/to/code/ --select NPY201

The 2.0.0b1 release is out for testing as of March 11 and it looks like 2.0.0rc1 is due any day now.

[ksunden]

We have been running tests against nightlies, and only recently started failing, see #27892

Currently waiting on pybind11 to catch up (seberg has a PR in) which is causing at least a few of the failures... (mostly in tri- related things and contouring) I suspect this is delaying the RC timeline to a degree...

There are more failures on CI than I see locally, but suspect that is due to inconsistent np2 support, so hopefully will be worked out with little to no direct action over time.

[ksunden]

#27993 addresses this as far as the 3.8 series is concerned

[ksunden]

Closing as this has been addressed with the merge up and we are now building np2 compatible by default