DOC Update website to pydata-sphinx-theme

[Charlie-XIAO]

This issue is a continuation of #26809 and aims to migrate the scikit-learn website towards pydata-sphinx-theme. As this is an ambitious goal, this issue is to track the steps of the migration. cc @lucyleeow who guided me to open this issue.

Quick links

• The new_web_theme branch
• Tracker for upstream issues
• Live preview: https://scikit-learn.org/_pst_preview

TODO before merging into main

• doc-min-dependencies is bypassed in CI for now and we need to reactivate it before merging into main. Also, we need to make sure all dependencies are documented, in particular sphinx-design which was not added in the very first setup PR. See also #28379.
• Remove the new_web_theme part in .circleci/config.yml.
• In conf.py, change the version switcher link to https://scikit-learn.org/dev/_static/versions.json.
• Remove themes/ and update exclude_patterns in conf.py: these are only useful for the old theme.
• Pin higher versions of sphinx, pydata-sphinx-theme, and sphinx-gallery. See also tracker for upstream issues.

Tracking work towards the new_web_theme branch

• DOC [PST] conf, setup, general styling #28132
• DOC [PST] landing page #28331
• DOC [PST] install page #28336
• DOC [PST] version switcher and warning banner #28347
• MAINT [PST] set up live preview #28353
• DOC [PST] fix lock files and add sphinx-design #28379
• DOC [PST] fix dropdowns #28401
• DOC [PST] tune toctree-related styles #28408
• DOC [PST] fix changelog badges #28409
• DOC [PST] better integrate gallery with pydata-sphinx-theme #28415
• DOC [PST] refactor API structure and improve display #28428
• DOC [PST] tune FAQ page styling #28448
• DOC add Algolia search bar for improved website search #28478
• DOC [PST] use jinja2 template for rst generation #28511
• DOC [PST] disable gallery link tweaks when html-noplot to avoid warnings #28512
• DOC [PST] fix styling/simplify structure of testimonial and about pages #28623
• DOC [PST] FIX/RFC machine learning map #28630
• Fix admonitions (Todo: make e.g. .. note:: something into .. admonition:: something with :class: note)

Tracking work related by towards the main branch

• MAINT remove unneeded robots.txt #28387
• MAINT create robots.txt for setting up pydata-sphinx-theme preview #28376
• DOC fix a sphinx warning and a rendering issue #28315
• DOC nitpicks on the FAQ page #28272
• DOC fix some hooks that fail to capture the titles in pydata-sphinx-theme setup #28271
• DOC restructure changelog (in particular for switching to pydata-sphinx-theme) #28255
• DOC solve some sphinx errors when updating to pydata-sphinx-theme #28134
• DOC fix wrong indentations in the documentation that lead to undesired blockquotes #28107

[adrinjalali]

@Charlie-XIAO I rebased the new_web_theme to the latest main.

This is awesome work, thanks.

[glemaitre]

This is awesome work, thanks.

Indeed, this is.

Please feel free to ping me whenever you need reviews.

[Charlie-XIAO]

@adrinjalali @glemaitre Thanks for your confirmation, and thanks for updating new_web_theme! I've opened #28132 for the very first step.

[Charlie-XIAO]

Tracking the list of upstream issues in pydata-sphinx-theme, bootstrap, sphinx, etc.

Latest stable release of pydata-sphinx-theme: 0.15.2
Latest stable release of sphinx-design: 0.15.0
Latest stable release of sphinx: 7.2.6

• Fix sticky header pydata/pydata-sphinx-theme#1630
• fix: align the search button with other icons pydata/pydata-sphinx-theme#1620
• FIX make theme switcher have consistent widths pydata/pydata-sphinx-theme#1651 (pydata-sphinx-theme>0.15.2)
• FIX make search button closer to other icons in topbar pydata/pydata-sphinx-theme#1659 (pydata-sphinx-theme>0.15.2)
• ENH make search result heading a bit away from the search input box pydata/pydata-sphinx-theme#1690 (pydata-sphinx-theme>0.15.2)
• ENH animation for the top banner pydata/pydata-sphinx-theme#1693 (pydata-sphinx-theme>0.15.2)
• FIX secondary sidebar persists even if it is empty pydata/pydata-sphinx-theme#1688 (pydata-sphinx-theme>0.15.2)
• Closes #11884 fix searchtools.js: docContent.textContent sphinx-doc/sphinx#11885 (sphinx>7.2.6)
• [ENH] Minigallery can take arbitrary files/glob patterns as input sphinx-gallery/sphinx-gallery#1226 (sphinx-gallery>0.15.0)

---

• Section navigation shows up even if it contains nothing pydata/pydata-sphinx-theme#1662
  FIX empty primary sidebar showing up pydata/pydata-sphinx-theme#1682
  Shows empty navigation. This is an pretty large problem such that if unsolved upstream we would need to pin 0.14.x.
• FEA Integrate download links and binder/juputerlite buttons with pydata-sphinx-theme secondary sidebar sphinx-gallery/sphinx-gallery#1258
  Integrate the badges and download links with the secondary sidebar.
• minigallery with multiple files: deduplicate and should not start new rows sphinx-gallery/sphinx-gallery#1259
  minigallery with multiple files should not start a new row for each file, and should remove duplicated examples.
• broken reference in API builds pydata/pydata-sphinx-theme#1435
  Use bootstrap ScrollSpy with anchor using dots? twbs/bootstrap#39248
  Scrollspy (i.e., secondary sidebar highlight following the scrolling of the page) does not work with IDs that contain dots, and autodoc generates a lot of those IDs. This needs to be fixed in bootstrap (the reason is that currently it selects #xxx.xxx.xxx where .xxx would be interpreted as a class instead).
• page is not displayed in the breadcrumb list pydata/pydata-sphinx-theme#1624
  The breadcrumbs component is broken when having numbered sections. If not fixed upstream we need to vendor the component and make modifications on our side.

[adrinjalali]

Now that #28132 (comment) is merged, I think it might not be a bad idea to have that branch pushed somewhere so that we can have a live preview.

[adrinjalali]

I was thinking more something alone the lines of https://scikit-learn.org/new_web_theme though. to really test things, and have it updated automatically with every merged PR.

[Charlie-XIAO]

Ah I see, so essentially I can modify the deploy step of CircleCI to do something when CIRCLE_BRANCH is new_web_theme right? I can open a PR to try. Update: #28347.

[betatim]

Should we add a label for PRs related to this effort? I'm generally not the biggest fan of labels, but it seems like it could be useful here to filter PRs to just those related to this issue. WDYT?

[Charlie-XIAO]

Yea a label may be useful. Maybe it can be automatically added with some workflow since I currently give related PRs titles with a shared prefix?

Or alternatively I'm keeping track of all PRs towards the branch, related PRs towards main, and upstream issues (which you can find or redirect to from the very first comment). So essentially you can easily find everything related from the first comment in this issue, and maintainers can edit if more shortcuts are wanted.

[Charlie-XIAO]

A question: is it possible to directly add the dependencies (pydata-sphinx-theme, sphinx-remove-toctrees, etc.) in main? I think git is not able to merge the lock files from main correctly into new_web_theme.

[lesteve]

A question: is it possible to directly add the dependencies
I think git is not able to merge the lock files from main correctly into new_web_theme.

I think you can keep working in the preview branch. Lock-files are quite likely to have conflicts but you can sort them out like any other conflicts in git, see #28336 (comment)

[Charlie-XIAO]

Thanks for the reply @lesteve. If we do not worry about the conflicts then merging main into new_web_theme would indeed solve my problem.

[Charlie-XIAO]

The only thing is that every time after merging main into new_web_theme I think the lock files will no longer have the dependencies for pydata-sphinx-theme so we need to manually update the lock files again with a PR, otherwise things will stop working.

[lesteve]

Yes there is an additional manual process involved to solve the conflicts, I think this is unavoidable. This would be a bit weird to have additional dependencies in main that we don't use ...

Unless I am missing something, if you want to merge main into new_web_theme this is already a manual process right and you need to open a PR? The additional step is to fix conflicts before opening the PR. There is probably a way to make it less painful by running something like this (not tested so probably need to be adapted):

# I never know --theirs vs --ours and it depends whether you do rebase or merge I think
git checkout --theirs -- build_tools
python build_tools/update_environments_and_lock_files.py --select-build 'doc$'
git a build_tools
# or use rebase if you do rebase
git merge --continue

[Charlie-XIAO]

I do not have the right to merge, so I need to leave it to maintainers. So, either every time maintainers do this when merging new_web_theme into main, or I make PRs for this every time maintainers do a merge.

Initially I asked about adding to main because that would be "once and for all" (if I am not mistaken). But yes having things we do not use in main is strange... especially since the doc will get modified as well (though very minor places, by which I mean the table on this page that is written based on _min_dependencies.py).

I am okay will both options, and I think this might not be something I can decide. @adrinjalali who have helped me do the merging job recently.

[lesteve]

I think it is fine if new_web_theme branch lags behind main and is updated only from time to time, for example say every week or even less frequently.

I guess the first thing is to get feed-back on how it looks and for this this does not matter if you don't have the doc updates from the last two weeks.

The mechanism to update new_web_theme preview is to do a PR (once it's merged you have to wait a while, ~1 hour, before the doc is built on the branch and the content pushed to .github.io). If you merge main into your PR branch and you have lock-files conflicts, you need to fix the conflicts before you open a PR.

[Charlie-XIAO]

Yes you are right and in fact I need the theme branch to be updated only when I directly push some related changes to main.

The mechanism to update new_web_theme preview is to do a PR (once it's merged you have to wait a while, ~1 hour, before the doc is built on the branch and the content pushed to .github.io). If you merge main into your PR branch and you have lock-files conflicts, you need to fix the conflicts before you open a PR.

Yes true. And I believe I shouldn't need to merge main into my PR branch if I'm targeting the theme branch.

So the decision is not to disturb main. I do need to make a PR to fix what's already broken though. Thanks for your careful explanations @lesteve

[glemaitre]

I merged the latest PR and we should not have any pending PR. Now, I think that we need:

• Merge main into the new website branch and solve conflict
• Have a look if we can alleviate some of the remaining issue through the configuration file.

Then, I think that we should be merging the PR in main.

[glemaitre]

FYI, I create a specific board that could be useful to track any PR issues in other repositories: https://github.com/orgs/scikit-learn/projects/6

[Charlie-XIAO]

Nice! I will take a look at the merging tomorrow; I think there is going to be some conflicts in particular regarding the drop-down which git often fails resolve elegantly. AFAIC with the current changes the new website is definitely "usable". We may navigate around once again when I open the PR targeting main. There are indeed styling issues left e.g. admonitions, misused block quotes in APIs, etc. but those can be tackled afterwards.

[glemaitre]

I'm closing this issue since we did most of the items.
I aggregated the remaining issues (that needs upstream work) in the following board: https://github.com/orgs/scikit-learn/projects/6