Proposed fix that ensures parallel pip cache downloads can work

[itamarst]

Fixes #12361

TODOs:

• The presumed invariant here is that downloading the same URL always give the same results, so that mixing metadata and body from concurrent downloads is OK. That is, this PR only fixes the race condition as it affects reads. I assume that invariant holds for pip downloads, but it would be good to have that validated.
• I don't have a reproducer, or better yet, end-to-end test. My attempt at reproducer didn't work. This isn't strictly necessary but I can't prove this is a real fix.
• Need to a file a bug against CacheControl

[pfmoore]

This seems like a reasonable approach. I can't confirm that the assumption you state is valid, but it seems correct to me. I think I'd be happy to say that we don't need to support URLs that serve different content at different times (although I don't have experience with some of the big development infrastructures that people work with, so it's possible I'm wrong).

[itamarst]

I am going to add references to the upstream issue, and document the invariant at least.

[itamarst]

Even if the writes assumption is wrong in some edge cases, this doesn't make things worse compared to 23.3 in those edge cases, and it makes things better everywhere else.

[RonnyPfannschmidt]

i vaguely recall this can happen on devpi for non-finalized wheels,
however its also highly unlikely

being race-safe on pypi goes a long way however

[itamarst]

This PR also fixes the case where pip crashes half-way through writing the cache entry, i.e. after writing metadata but before writing the body.

Assuming you're happy with the code, it seems like an immediate improvement, and a long-term fix can be done once CacheControl has fixes on its side for the two issues I filed.

[pradyunsg]

Thanks @itamarst! ^>^

[sbidoul]

Thank you so much for the prompt followup on your initial contribution, @itamarst!