Make pytest.warns() re-emit all non-matching warnings

[reaganjlee]

PR Description:

This issue fixes part of issue #9288 by allowing pytest.warns() to emit all unmatched warnings.

One question I have with the design is how to handle the exceptions once we re-emit them. Having pytest.warns() calls return all exceptions will require refactoring of other tests. There doesn't seem to be a way to check whether the code is within a WarningsChecker frame, so a way to solve this is an optional-argument for re-emitting?

[RonnyPfannschmidt]

this one is tricky, as the key question to solve here is whether to remove one or all matched warnings

unlike the case of only checking, if a match exists, now we need a decision on what to re-emmit

i believe the possibly desired use-cases are all, unmatched, "all but first match" and its going to be hard to decide a good api for this

[reaganjlee]

Since we want to reduce number of refactors (and potentially, number of optional arguments), we can assume that if a user specifies the particular class they'd like to catch warnings from, then they would use this same warning type for similar checks, leading us to filter that exception class and remitting only the different, unexpected exceptions from the one specified? If there is a specific match given, then ignore this and do the usual re-emit besides the one/s found.

This may or may not still lead to some of the existing, unrelated tests failing, though I can implement this soon and see how that goes

[RonnyPfannschmidt]

I'm currently not able to allocate some time to write the details down to reiterate the options

My current recommendation would be start with a implementation that filters all matching warnings and then Iterating on documentation for a future api where the other use cases can be served

[reaganjlee]

Let me know if there are any documentation issues to fix for this. And if this works, I can fix up the code to match

[Zac-HD]

Hey Reagan - let's try to get this merged soon!

• Rebase on the latest main branch
• I think that we should re-emit all and only warnings which do not match (by type and possibly regex pattern).
• We'll need to fix the mypy warnings from pre-commit ci; add a changelog entry, and make sure you're in the authors list

and I think that's it; your docs look good to me and I don't think we need to handle any additional complexity here.

[Zac-HD]

As you note, we'll need a changelog, docs, and adding you to the authors list (🎊); after those, fixing the mypy CI errors, and comments below I think we'll be ready to merge!

[nicoddemus]

Thanks @reaganjlee and everyone for the comments.

Left a few smaller comments too, other than that LGTM. 👍

[The-Compiler]

I don't think I'll be able to review this - I barely ever use warnings (given my main project is an application rather than a library), so I'd rather leave this to someone else.

[Zac-HD]

Thanks @nicoddemus, all done!

Since Python 3.7 is now end-of-life, we should probably test on a newer version of PyPy (3.8 through 3.10 are now supported) and remove the old CI jobs. I'd rather not add that into this long-running PR though 😅

[nicoddemus]

Not sure what this TODO means here... can we clarify it? Also not clear if this is something meant to be done still in this PR, or later.

[Zac-HD]

Ah, that's a note-to-self about rebasing #11129, which I'll do immediately after this is merged.

[nicoddemus]

we should probably test on a newer version of PyPy (3.8 through 3.10 are now supported) and remove the old CI jobs. I'd rather not add that into this long-running PR though

Agreed, we can do any adjustments later after main is adapted. 👍

[pllim]

I think this broke downstream tests reported at astropy/astropy#15015 . Our intention of the affected tests are to make sure a particular warning (that we are checking with pytest.warns) is emitted, not necessarily that it has to be the only warning. If we want to make sure that it is the only warning, we check the len later. I suspect this change would break multiple downstream packages when released. Is there a way to get the old behavior back? Or did I misdiagnose the downstream failures?

[Zac-HD]

I think you're correct in tracing the change to this PR.

We're aware that this will lead to more warnings being emitted, which can make tests fail with -Werror (that's why it's documented as a breaking change!), but we think ceasing to hide unrelated warnings is worth making that change in Pytest 8.0 - see #11126 for the release plan.

To ignore any other warnings, without asserting that there must be other warnings, you can use the stdlib warnings.catch_warnings() + warnings.simplefilter("ignore") functions as we document here. I contributed a bit of API sugar so from Python 3.11 you can use with catch_warnings(action="ignore"):, but it might be a while before you can rely on that everywhere.

---

I did notice skimming through the failures that this test (source here) could be affected by #11129 - up until now, a warns() within a raises() did literally nothing at all 😓. This gzip warning also looks real, though I don't know enough about FITS to tell for most of the rest.

[pllim]

Thanks for the clarification! I guess I'll have to go in and clean stuff up. 😸