New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
linkcheck builder: streamed HTTP requests do not close their local socket #11317
Comments
Note: TCP connection re-use -- something that would likely provide performance benefits and resource reduction for the linkchecker -- requires use of That's a slight tangent, because somehow I think that there could still be an additional cost introduced here by ensuring that each request socket is closed. But I think closing them is probably a sensible thing to do, and that use of sessions could be introduced separately. |
|
That makes sense, and #11318 should perform the relevant request-and-socket closing. I'm not sure we should merge that until we determine the reason that #11299 is happening, though: it may indicate a problem here or elsewhere.
Yep, I think we should begin using sessions as a followup improvement for the linkchecker. It'd be nice to be able to demonstrate how the connection-pooling behaviour changed before-and-after that. |
Describe the bug
Running
pytest tests/test_build_linkcheck.py
emitsResourceWarning
s related to unclosed sockets, such as:This occurs when HTTP requests are made using the
requests
library withstream=True
enabled (approximate behaviour: do not begin consuming content from the connection immediately, and provide it iteratively as it is received).This occurs in a couple of places currently:
sphinx/sphinx/builders/linkcheck.py
Lines 319 to 320 in 776d01e
sphinx/sphinx/builders/linkcheck.py
Lines 341 to 343 in 776d01e
I'm not certain whether this implies a bug/problem. The upstream
requests
project mentions that connection pooling and the cost of recreating sockets (as opposed to re-using existing ones) can be significant.I've found two possible resolutions locally (either: make requests using a context-manager, so that the
__exit__
implicitly calls.close
, or: call.close
on the response object explicitly), and will offer a pull request with one of those.How to Reproduce
Simple repro case:
$ pytest tests/test_build_linkcheck.py`
To trace where the resource allocations originally occurred:
Environment Information
Sphinx extensions
None
Additional context
Discovered while investigating #11299, although does not appear directly related.
The text was updated successfully, but these errors were encountered: