Refactor: linkcheck builder: reduce the lifetime of the 'response' variable

[jayaddison]

Feature or Bugfix

• Refactoring

Purpose

• Reduce the runtime duration during which the response variable that holds the results of HTTP request results should be considered live and is non-garbage-collectable.

Detail

• I think that delayed resource-collection of the response variable may be a contributing factor to timeouts that occur during our unit tests.

Relates

• Flaky unit tests: linkcheck builder (tests/test_build_linkcheck.py) #11299.
• linkcheck tests: test webservers: enable HTTP/1.1 protocol #11392 (comment)

  • The response object has a kind of code smell about it in this case - it seems to exist for longer than is necessary (yes, we read some information from it later -- but those items of information could be retrieved up-front, and then the code that requires them could be refactored to accept parameters). While it's held, I don't think that the connection resources associated with it can be released -- because the server-side of the connection is open, this is HTTP1.1.

[AA-Turner]

Thanks @jayaddison!

A

[AA-Turner]

https://github.com/sphinx-doc/sphinx/actions/runs/5615478788/job/15215837543 -- LaTeX timed out

FAILED tests/test_build_linkcheck.py::test_defaults - assert "Anchor 'top' not found" in "links.rst:6: [broken] http://localhost:7777/#top: HTTPConnectionPool(host='localhost', port=7777): Read timed out. (read timeout=0.05)\nlinks.rst:4: [broken] http://localhost:7777/#!bar: HTTPConnectionPool(host='localhost', port=7777): Read timed out. (read timeout=0.05)\nlinks.rst:9: [broken] path/to/notfound: \nlinks.rst:7: [broken] http://localhost:7777#does-not-exist: Anchor 'does-not-exist' not found\nlinks.rst:11: [broken] http://localhost:7777/image.png: 404 Client Error: Not Found for url: http://localhost:7777/image.png\nlinks.rst:13: [broken] http://localhost:7777/image2.png: 404 Client Error: Not Found for url: http://localhost:7777/image2.png\n"

[jayaddison]

https://github.com/sphinx-doc/sphinx/actions/runs/5615478788/job/15215837543 -- LaTeX timed out

See also a similar test failure at: https://github.com/sphinx-doc/sphinx/actions/runs/5615865316/job/15217324918

[jayaddison]

It looks like I introduced a regression with the rewrite of these lines - and the regression is reported here: #11532 (comment)

Basically: when anchor checking is not enabled -- that is, self.linkcheck_anchors is False -- the first retrieval method returned is an HTTP HEAD request. It can have response.ok, and the anchor value may be non-empty -- because we continue to extract (partition) the anchor from the uri regardless of whether anchor linkchecking will be performed. The HTTP HEAD response doesn't contain a content body, so contains_anchor will return false, and the exception is raised.

I'll prepare a fixup/patch for this and some test coverage to cover this situation within the next few days.

cc @adamtheturtle