also use PyErr::SetObject on Python versions before 3.12

[davidhewitt]

Fixes #3432

This PR changes versions before 3.12 to also use PyErr::SetObject when raising (or normalizing) a lazy PyErr. This helps to unify the code paths between 3.12 and older versions as well as fixes the bugs discussed in those recent issues.

[mejrs]

Can you explain how this fixes those issues? I'm reading the code and the C api docs but I don't quite see it.

[davidhewitt]

I can't speak in detail for the PyPy fix, which is a little bit arcane. My intuition there was that before this PR we were not "normalizing" exceptions from our Lazy state before calling PyErr_Restore. Somehow in the StopIteration case with PyPy having the exception be unnormalized was not behaving the same as CPython. Potentially we could argue it's a PyPy bug which we should report upstream, but if switching to PyErr_SetObject fixed it, hey, that's a nice improvement given I was exploring that anyway.

The motivation for calling PyErr_SetObject is that this has the logic for chaining exceptions baked into it, and on 3.12 it also normalizes the exception eagerly (since raised exceptions are always stored in the normalized state).

Given we're forced to use it on 3.12 to make the machinery work, because PyErr_NormalizeException is deprecated, it seems like a simplification to the code to also use it on pre-3.12 given it adds the exception chaining functionality we want.

[davidhewitt]

See also this comment; I think in principle we could implement an equivalent of PyErr_SetObject in Rust, which would have the upside of allowing us to then refactor and adjust behaviour later if we so desire. I wanted to start by just calling PyErr_SetObject as it avoids moving a lot of complexity to Rust.

[davidhewitt]

Please review #3471 first.

[davidhewitt]

Rebased; this PR is now a very minimal diff which hopefully highlights the way that this adjustment makes the pre-3.12 code paths match 3.12, while also fixing a bug (and adding a test).

[davidhewitt]

As this makes a behavioural change which I believe we want to bring as part of the 0.20 release, I'm going to merge this soon (probably tomorrow) so I can proceed with preparing the release. Please forgive me if this turns out to be unwanted! 🙏