linkcheck failing after Sphinx 7.1.0 release

[larrybradley]

Describe the bug

Starting with Sphinx 7.1.0, my package(s) started reporting linkcheck failures due to "Anchor not found", e.g., https://github.com/astropy/photutils/actions/runs/5688763395/job/15419142358.

Reverting to Sphinx 7.0.1 fixes the issue.

git bisect reveals the issue started with e45fb5e, this PR: #11432

How to Reproduce

$ git clone git@github.com:astropy/photutils.git
$ cd photutils
$ tox -e linkcheck

Environment Information

Platform:              darwin; (macOS-13.5-x86_64-i386-64bit)
Python version:        3.11.3 (main, May 26 2023, 21:36:22) [Clang 14.0.3 (clang-1403.0.22.14.1)])
Python implementation: CPython
Sphinx version:        7.1.1
Docutils version:      0.20.1
Jinja2 version:        3.1.2
Pygments version:      2.15.1

Sphinx extensions

No response

Additional context

No response

[larrybradley]

cc: @pllim

[pllim]

I wouldn't have suspected Sphinx itself. Thanks for investigating!

[jayaddison]

@larrybradley @pllim @adamtheturtle thanks for reporting this (both here and in #11532). The regression was indeed introduced in #11432.

I'm working on fix for this and hope to have that in place relatively soon - a fixup should optimistically be available within the next few days.

Despite the problem, to try to make the most of a learning opportunity: could I ask out of curiosity why you prefer to run linkchecking with anchor checking disabled?

[larrybradley]

@jayaddison Thanks. It's been so long that I don't even recall setting linkcheck_anchors = False. Perhaps it was because of this statement from the docs:
"Since this requires downloading the whole document, it’s considerably slower when enabled."

I may just switch to linkcheck_anchors = True.

[jayaddison]

Thanks @larrybradley. Enabling anchor checking is likely to increase network bandwidth usage, because the linkchecker will use HTTP GET to retrieve entire page content instead of HTTP HEAD to check whether the page exists. Scanning for the anchors in the content also requires a little extra compute time.

As long as those aren't significant costs in your use case, enabling anchor checking sounds good to me - I think the counterbalancing benefit of checking the validity of references within pages is worthwhile in most circumstances 👍

FWIW: we have some further network usage optimisations planned in #11346 that may further reduce the network costs of anchor-checking in future.

[pllim]

Hello! Maybe related... I find that intersphinx does not work well with this new feature.

Here is the original page I am trying to refer to: https://github.com/spacetelescope/synphot_refactor/blob/d5feb4ec8e89912a117f111e355f78fa4534e9a9/docs/synphot/spectrum.rst?plain=1#L485

.. _synphot-vega-spec:

Vega
----

Here is the doc downstream that refers back to it using intersphinx: https://github.com/spacetelescope/stsynphot_refactor/blob/7912bdac481e44f8622fbe8244125f9e656c8bdf/docs/stsynphot/spectrum.rst?plain=1#L19

* ``stsynphot.spectrum.Vega``, which is a preloaded
  :ref:`synphot:synphot-vega-spec` spectrum. It can be reloaded using
  :func:`~stsynphot.spectrum.load_vega` if necessary.

Example log with failure: https://github.com/spacetelescope/stsynphot_refactor/actions/runs/5723206136/job/15507435536

(stsynphot/spectrum: line   18) broken    https://synphot.readthedocs.io/en/latest/synphot/spectrum.html#synphot-vega-spec - Anchor 'synphot-vega-spec' not found

I expect the synphot-vega-spec intersphinx label to automatically resolves to #Vega anchor in the final HTML but that is not the case.

[jayaddison]

Thanks @pllim - yep, that is the same issue by the looks of it.

A fix is in progress in #11544, or if you would prefer to try enabling anchor linkchecking -- while bearing in mind the tradeoffs mentioned previously -- you can adjust the synphot_refactor linkcheck_anchors setting to True (or remove that line -- the default value for it is True).

[AA-Turner]

Sphinx 7.1.2 has been released.

A

[pllim]

Thanks! The failing jobs are passing again for me.

[jayaddison]

@pllim brilliant, glad to hear that - thanks for confirming.