-
Notifications
You must be signed in to change notification settings - Fork 213
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Request: release a patch version that pins cython<3.0 #1342
Comments
For most projects, a release is fairly simple as you mention. However, due to a requirement on maintainers free time, building wheels, and testing with downstream Linux release managers, it is not quite as simple for pyproj. The next release of pyproj depends on python 3.12 compatibility updates from a release from numpy to ensure stability. We welcome assistance preparing pyproj for the next release with Python 3.12 wheels. |
The blocker for the next release is #1330. Once that is ready, the release will follow. That likely depends on the next release of numpy, but there may be a way to get it to work now. The instructions you linked to are correct. However, slightly out of date. The wheels are automatically uploaded except from Cirrus CI as those need to be manually uploaded to pypi currently. |
@snowman2 I had some luck with the vispy project (builds wheels with a single Cython extension) with forcing the pre-release of numpy to be installed. I see you're already doing that, but it looks like for some reason it is trying to build numpy from source. Here's vispys cibuildwheel env vars: If needed I can try to take a look tomorrow, but no guarantees on time. I also don't build 32-bit wheels for vispy (or any of my packages) and it looks like you do...how do you handle that with numpy not providing 32-bit wheels? |
I think that it is the pypy wheels and win32 wheels that are building from source. Depending on the win32 failures, that one would be okay to disable. |
There are suggestions for workarounds linked in #1330 to numpy issues. |
My kid had to stay home sick today so I'm not really getting any work done today and probably not tomorrow. |
No worries @djhoese. I hope that your kid feels better soon. |
Coming soon ... #1344 |
Thanks @snowman2! I would still be happy to help with patch or "post" releases for older versions for the sake of any users who can't upgrade for whatever reason. In my case, updating to the newest version is fine. |
Contributions to help with releases are welcome 👍 |
@snowman2 could you point me to the release instructions? In your opinion what is the hardest part? Would you consider pyproj's release process much harder than other python packages you maintain (ex. rioxarray) given how tightly it is tied to the PROJ library? Or are there other reasons? |
https://github.com/pyproj4/pyproj/blob/main/HOW_TO_RELEASE.md
The wheels. The matrix of python versions (3.9-3.12), operations systems (Windows, MacOS, Linux), architectures (x86, i686, x86_64, amd64), and python implementations (cpython, pypy) make for long build times and a high potential for failures. Due to the long build times, it takes a while to debug as the CICD process takes a long time to complete. It is a never ending game of wackamole to keep it stable. Most of the wheels are build using GitHub Actions. However due to need to support MacOS arm64 and Linux aarch64, the wheels are build on Cirrus CI and Travis CI. With Travis CI, we have a limited amount of credits, so reducing the frequency of the releases helps to stretch the credits farther.
Essentially, yes. PROJ makes it required to provide wheels. With rioxarray, I can make a new release in 1 minute and it is all automatically uploaded to pypi. I make a lot of rioxarray releases as soon as features are added (release early, release often 😄 ). With pyproj, it sucks hours of my time both preparing for and making sure the builds complete properly (which rarely happens the first time around). So, I try to limit the number of pyproj releases to reduce the amount of time I have to dedicate to making releases. Additionally, there are downstream linux distribution package managers that are kind enough to run tests on pyproj before each release. I usually try to limit the number of pyproj releases to be respectful of their time. In general, a release every 4-6 months is the current cadence of pyproj releases. |
A couple thoughts:
|
I should have maybe started with: this is just me brainstorming and not suggesting that you alone should tackle these things. So as a project, what should pyproj do to speed this up and what has been tried/avoided in the past? Oh:
|
It is used so the wheels generated can be tested. You can generate the wheels on Actions, but cannot test them. In my experience, it is a bad Idea to release something you haven't tested.
Automating it is on the TODO list.
IIRC someone requested them a while back...
I am open to this idea.
That is definitely something to look into.
Those are optional test dependencies. If it works, great. If not, not a big deal.
numpy is the only bottleneck at the moment. Not sure I would want to release without testing using numpy.
Sounds worth looking into. |
I can understand that. Usually the temptation is too great and I just end up not testing for the hard-to-test platforms. In my usual cases though these are just simple Cython extensions with simple for loops over some numpy arrays so the compatibility usually leans on Cython and numpy's testing so I'm a lot less scared about my own libraries doing something incompatible. My only worry with two separate build systems is if one uploads to PyPI but the other fails and you get this weird partial release. I suppose that is one advantage to the non-automatic cirrus wheels.
Hm, what I saw was a doctest failure. I guess if that's integrated with your pytest and is an xfail or whatever then 👍
I guess I was thinking more breaking up the categories of tests. Like basic python-heavy functionality probably isn't going to break between platforms but if they're fast then whatever include them. The heavy PROJ compatibility is probably always needed. I guess it depends on what tests take the longest. Is it an even distribution for test execution time or are there some that are like 20 seconds each that could be skipped. Otherwise... it was brought to my attention today that pykdtree and pyresample wheel building are not in the modern times and I need to overhaul them. I'm working on that now and if they have any decent amount of build time maybe I'll try that splitting per-platform thing and see how github actions feels about it. I could then port that to pyproj. |
Oh about not testing on GA, you're talking about not being able to test on the emulated platforms? |
Re: PyPy wheels, I actually first ran into this problem because I was trying to install Pyproj on PyPy, and there wasn't a wheel available for my particular platform, so it tried to build from source and failed. Given that problems like the current one are very rare, I don't think it's such a bad idea to drop wheel builds that are an undue maintenance burden. |
That sounds about right. |
The tests are pretty quick, so I wouldn't spend too much time optimizing those. The main bottleneck is dependencies. |
Of the 28m45s of the ubuntu wheel building for the last release, the testing makes up 11+ minutes (~37%). If I can get PROJ building cached testing becomes ~58% of the build time. I also suggest we drop PyPy 3.9 given that numpy will likely drop it: And if that's dropped then that does take ~5 minutes off of the total build time. PyPy tests take most of that time because they try to build shapely from source. So yeah the dependencies being installed don't help, but I'm not sure there is much that can be done there without caching them...which they might be already internal to cibuildwheel...I'll look at that too. |
And I've decided against splitting the environments based on python version. If I get caching working for PROJ then I'll reconsider it. The main downside though is that updating cibuildwheel doesn't automatically get you wheels for new versions of Python because you're likely explicitly setting what versions of Python to build. Edit: ...and you're already splitting on platform/arch because of the GA versus cirrus versus appveyor split. |
Currently, it's impossible to build Pyproj 3.6.0 with PEP 517 build isolation because Pip will install Cython 3.x, which Pyproj 3.6.0 is incompatible with.
This can be easily fixed by releasing a Pyproj 3.6.1 or 3.6.0.post1 version that does simply adds
<3.0
to the Cython build dep: https://github.com/pyproj4/pyproj/compare/3.6.0...gwerbin:pin-cython-below-3.0?expand=1I understand that the current master branch updates Pyproj to use only Cython 3.x, but I'm not aware of a release date for it. This patch would be very useful for end users who need to build their own wheels (e.g. for PyPy), while the next release is still in development.
The text was updated successfully, but these errors were encountered: