Testing: issue 11299 replicability

[jayaddison]

Using this branch to perform repeat continuous integration testing for potential flaky-test fixes.

Tally	Issue encountered	Issue not encountered
Without sphinx-doc#11323	2	48
With sphinx-doc#11323	0	50

Test failures:

• test_raw_node: https://github.com/jayaddison/sphinx/actions/runs/4697985339/jobs/8329708670?pr=1#step:10:2447
• test_raw_node: https://github.com/jayaddison/sphinx/actions/runs/4698589467/jobs/8331061144?pr=1#step:10:2452

[jayaddison]

The failure at https://github.com/jayaddison/sphinx/actions/runs/4697985339/jobs/8329708670?pr=1#step:10:2447 was a timeout in the test_raw_node test case, seeming to indicate that the problem is in fact not specific to the test_linkcheck_allowed_redirects test.

[jayaddison]

Why would sphinx-doc#11323 make any difference to the result of other unrelated unit tests?

And in particular, why would it reduce the number of timeouts encountered by HTTP API clients during those other tests?

It doesn't seem to make sense. I think that the lack of test failures so far here when running with the fix applied must be a coincidence. I'll continue a few more rounds of testing.

[jayaddison]

Hmm, this is proving fairly tricky to replicate. I wonder if any/some of the failures could've been due to something environmental like resource contention (limited CPU time available on the GitHub Actions runners available to the tests when they were running, for example) and not a code-related issue at all. Not too far from a roundish number of 100 evaluations currently, so at that point I'll probably call this complete, regardless of whether it provides much additional information by then.

[jayaddison]

After 100 experimental evaluations (each involving running the tests across a matrix of three Python versions, two DocUtils versions, and doubled for good measure (one GitHub Actions push event and one pull_request each)), the only failures that occurred were without the fix applied.

Despite that, I don't see any reason why the fix should improve the situation here; my preference is to increase the timeout threshold in the linkcheck tests instead.