our releases break too many user sites

[drammock]

Every time we push out a release, a flood of new issues comes in. As a maintainer I feel responsible to fix them, but as a maintainer with other, higher-priority responsibilities I don't really have time to drop everything and make hotfixes.

For our most recent release I suggested we do an RC, which happened, but then a full release was done almost immediately (AFAICT after only one user site had tested it). In my mind that defeats the purpose of an RC, which is to give several test users time to try it and find bugs, so that we have time to fix those bugs at our own pace before those bugs break all user's sites.

I'd like to propose a change to our release process, which is to always require and RC and to set a mandatory minimum delay between the RC and the full release.

[12rambau]

My issue with RC is that appart from matplotlib and pandas that are checking it in their CI, nobody sees the RC as they are not pulled by pip automatically. Latex FAQ for example which usually provides a lot of feedback is not aware of our changes before we actually push a new release.

I see 2 output that could help us:

1. move forward with a community of advanced users and ensure they capture the updates and give feedbacks before we push RC to real releases. Create a "pydata community representatives" team to give feedback? #645
2. Our tests are weak and even if it's slow we should add a playwrite one everytime we create something new, testing html is obviously not sufficient. Test our JavaScript with Playwright #585

[trallard]

The release guidelines suggest a week or so between an RC and a full release, so my natural questions are:

1. was this followed? If so then perhaps this period needs extending
2. if it was not followed, why?

If it was not followed because only two projects are checking the RCs (or there might be more we are not aware of) it does not feel like enough of an argument to do so.

move forward with a community of advanced users and ensure they capture the updates and give feedbacks before we push RC to real releases. #645

Even if there is a small community of advanced users, they will need sufficient time to provide feedback. This brings us back to having to either extend or enforce the period between an RC and a full release.

Our tests are weak and even if it's slow we should add a playwrite one everytime we create something new, testing html is obviously not sufficient.

We should look at adding more playwright tests. We already have at least some initial tests that were added for accessibility checks, so we can build from that. That is something perhaps Gabriel and I can help with alongside some of the accessibility improvements in the theme. Still, it will be a considerable amount of work, I anticipate, so not necessarily a quick fix. But it is undoubtedly needed considering how much churn and issues we see on every release to avoid needing so many hotfixes every time.

[drammock]

My issue with RC is that appart from matplotlib and pandas that are checking it in their CI, nobody sees the RC as they are not pulled by pip automatically.

I don't think we really know how many projects pull in RCs. MNE-Python used to build off of the theme's main for example, and we plan to move back to doing it again. We started pinning last year because changes in the theme broke our site a few times in a fairly short time frame, and I didn't have time to keep figuring out why / update our template overrides and/or CSS to adapt. Anyway, as @trallard said: even if it's only a few projects, we need to give them enough time to discover breakages (which are often not full-blown sphinx failures and require someone to look at the site --- sometimes even just one or a few pages are affected! --- and notice that something is different/wrong; see for example scipy/scipy#16660 (comment)).

We should look at adding more playwright tests.

+1. Our tests are fast compared to other libraries I work on, and lack of thorough/comprehensive tests is what is stalling #1583 for example. We should still try to be efficient when adding them (playwright tests are slow, as are sphinx test builds) but I think we have some headroom before the tests become slow enough that they're really a bottleneck for theme development.

[tupui]

Speaking for SciPy, now that we are (soon) back on track, I would be happy to update our CI to help you test RC.

[trallard]

We should still try to be efficient when adding them (playwright tests are slow, as are sphinx test builds), but we have some headroom before the tests become slow enough that they're really a bottleneck for theme development.

Ditto - I think we can come up with some good balance on running some "unit tests" on all PRs/pushes and leave, for example, regression or more time-consuming ones merged to the main or as part of pre-release checks.

[dopplershift]

Just piping in to add that MetPy is running the RC's as they appear in our nightly CI run. If I see something here that calls attention to the RC being out, then I usually check the rendered artifacts.

[lwasser]

Hey y'all! 👋 i wanted to chime in here as i've been seeing these releases and they have been breaking our online builds. i actually came here to report a bug with 15.1. i've been thinking i may start pinning this theme. it would be helpful to somehow know when there are breaking changes or how some of us (i'm happy to help if i can in some way) in the ecosystem could perhaps help you when RC's come out? i don't follow the release cycle closely but do notice when i go to work on the site, (our packaging and peer review guides) i often find things don't work as they used to.

i know this is super tricky, I can see from both a user and maintainer side how it impacts folks. thank you all for this work on this amazing theme!! ✨

[12rambau]

A starting point could be:
Notify a community of meaningful (and willing) projects to review the latest RC. They will provide their feedback directly in the issue related to the release, which may be transformed to stand-alone issues if needed. Once we reach a good amount of answers (let's say 75%) we can transform the rc into a release.

As an initial list of project I would see the all list of featured project:

1. Arviz
2. Bokeh
3. Jupyter
4. Matplotlib
5. executablebook (I don't know which repository is the most meaningful)
6. matplotlib
7. scipy
8. pandas
9. sepal
10. numpy
11. networkx
12. MNE

additional project that have been very nice to review things also include:

1. ipyleaflet
2. jupyterlite
3. voila
4. metPy
5. pyOpenSci
6. brightway

So a grand total of 18 projects which seems more than enough to me.

As I'm re-writting the release.md in #1621 I think an official process could be a nice addition.

I see 2 options to reach everyone:

1. these projects include a CI check on RC that will be automatically triggered and we wait for their feedback
2. We open issues in their repository to notify everyone that documentation might break and we need some feedback

what do you think ?