[tests] add public properties SphinxTestApp.status and SphinxTestApp.warning

[picnixz]

This is a first step to the test suite modernization. I can extract those changes from my current implementation without changing anything (which is good from a review PoV).

cc @jayaddison

btw, the name of the branch is totally unrelated (at first, I just wanted to type things but then I saw that to type them I need this small change).

[picnixz]

I wanted to have two implementation of two plugins side-by-side and use the new plugin only for some tests but it's not possible in pytest... So I'll need to merge both the new and legacy implementation together and maintain them so that we can incrementally change the tests.

[chrisjsewell]

This is a first step to the test suite modernization.

Heya, can you explain what this means, as obviously its quite vague:

• What are the current issues with the test suite?
• and how does this PR (and further ones) seek to address these issues?

[picnixz]

What are the current issues with the test suite?

There are many many side-effects which makes it impossible to use with xdist (see #11285) (I forgot to mention that PR actually). By the way, the test suite takes roughly 8 minutes to finish. With xdist, it should boil down to 2-3 minutes I think (and locally it's around 1 minute in general).

and how does this PR (and further ones) seek to address these issues?

Each test should be isolated in a way that it does not affect other tests. Some tests use the same test-root but they sometimes change the sources in-place. Other tests import modules but sometimes they do not unload them and thus annotations are kept (and mess other tests). We should never have side-effects like that in general. Also many tests are actually C/C of others and some of them use app.build(force_all=True) but not all tests should do this ! IMO, force_all=True should only be done inside a test if more than one build is done. If the test is parametrized, then the parametrization is semantically equivalent to a big for loop so you just need a single app.build() with the same test directory for those parametrized tests.

This PR is actually just one thing I could extract from the huge PR with all the modifications. Also, there are various typing issues that were never resolved or assumptions that were wrong (like, assuming that app._status is a StringIO is actually incorrect because nothing is checked.

[chrisjsewell]

Each test should be isolated in a way that it does not affect other tests.

Completely agree, this should be a goal of the test suite 👍
I don't think this really has anything to do with "modernization", its just fixing something that is currently broken 😅

[picnixz]

I don't think this really has anything to do with "modernization", its just fixing something that is currently broken 😅

Ah yes, actually it's only some parts of the test suite that need to be modernized (like, using monkeypatch instead of manual patching).

[picnixz]

Here's my thought:

• Currently, the signature of SphinxTestApp is

def __init__(self, buildername='html', srcdir=None, builddir=..., ...)

and inside the constructor we have assert srcdir is not None. In other words, you cannot write:

app = SphinxTestApp()
app = SphinxTestApp('html')
app = SphinxTestApp('html', builddir=...)
app = SphinxTestApp(builddir=...)

srcdir = None
app = SphinxTestApp('html', srcdir)

In addition, we have

kwargs['srcdir'] = srcdir = sphinx_test_tempdir / kwargs.get('srcdir', testroot)

In other words, there will ALWAYS be a srcdir keyword argument when using the marker. Thus, you cannot have both the positional and the keyword argument at the same time (meaning that you will not be able to use the positional one argument in the marker!). In particular, when you use make_app instead of SphinxTestApp, you will call the constructor with a keyword argument.

Finally, the make_app docstring says

Provides make_app function to initialize SphinxTestApp instance.
if you want to initialize 'app' in your test function. please use this
instead of using SphinxTestApp class [directly].

In other words, the SphinxTestApp class is NOT meant to be part of the public API. As such, you should only be able to construct a SphinxTestApp instance using srcdir as a keyword argument. Therefore, the signature I suggest here where srcdir (and whatever follows) is a keyword argument is actually the one that we expected for years.

Finally, the sphinx marker never included the srcdir as a marker keyword so... it was probably not meant to be supported from a public PoV (and only meant for us).

Note that in Sphinx 4.2, we added the builddir argument but just after the srcdir one. So, we already changed the signature at that time ! and any codebase using positional arguments would likely have been updated since then. I think there shouldn't be a lot of codebase which does not use keyword arguments for constructing a SphinxTestApp unless they really want to rely on a fragile signature.

As such, I will still do this kind of breaking changes because 1) the SphinxTestApp class was never meant to be public 2) SphinxTestApp is not meant to be public and only constructed via make_app whose implementation we gave until now couldn't work if we did not use a keyword argument for srcdir 3) the keyword argument is mandatory, hence everything after should be specified as keyword argument as well (from a syntax PoV).

[picnixz]

Since the plugin will be a bit long to implement, I'll use separate branch. That way, I can rebase and you can see the diffs of the 3 PRs instead of having every chunk at each PR.

EDIT: I cannot transfer the branches easily :( let's forget about this then...

[picnixz]

@chrisjsewell Now it should be fine. I'll change the way the markers (in the next PR) are processed however because the way it was done until now assumed keyword arguments and not just positional ones (except for the 'buildername').

[chrisjsewell]

Actually, this one was not enforced during the years (and there are a lot of internals that are named without _).

Well they are not internals then, they are part of the public API 😉

but yeh thanks @picnixz, I don't wanna block you from moving things forward, and obviously its annoying when things have maybe not have been handled correctly in the past (one of my loves of Rust is that everything is private and immutable by default, so its more diffucult to make these mistakes).

But we should be very careful if breaking the "public API"; that it is really needed and, if so, then it is clearly documented

[chrisjsewell]

cheers!

I would commit it with a title like "Add SphinxTestApp.status and SphinxTestApp.warning public properties" though,
since the current PR title/description is not very explanatory 😅

[jayaddison]

I'm still not sure about this. I'm OK with the changes to make the __init__ signature of SphinxTestApp more similar to Sphinx itself -- the test class should tend towards an accurate representation of the way a normal Sphinx build would behave.

But I don't understand the reasoning for making methods public on SphinxTestApp that aren't public on Sphinx. Do we plan to make those public on Sphinx in future?

[chrisjsewell]

But I don't understand the reasoning for making methods public on SphinxTestApp that aren't public on Sphinx. Do we plan to make those public on Sphinx in future?

Checking the output, in particular of warnings, is highly common in sphinx extension test suites.

You just have to look at https://github.com/search?q=app._warning&type=code, to see the many test suites currently having to use the private attribute for this

[jayaddison]

Ok, thank you. People do tend to find and copy code in test cases into non-test code, so I worry that making the attributes public on SphinxTestApp only could cause confusion at some point. I understand that people are already using it, and that extra type validation is useful, but I'm not sure about the visibility modification yet.

[picnixz]

It's something that we must somewhat expose because there is no other way if you don't use the app fixture + status/warning fixture. If you use make_app, you need to access the status/warnings like that.

[jayaddison]

I think my concerns are unfounded, and so please go ahead. Thank you both for explaining.

So that it's communicated, I'll attempt to explain what I was worrying about:

• We push this change.
• Code that uses SphinxTestApp begins using the public accessor method (looks nicer, more acceptable, type-checked - a good idea).
• Other code copies stuff from that test code and attempts it in production and it breaks.

Why I think my concerns are unfounded:

• It should fairly clearly be test code, so it should be rare for people to copy it.
• If they do move it into production it'll break quickly because the relevant attribute names don't exist on the Sphinx class.

...and also I can't think of a suggestion to type-check an attribute value on an object that wouldn't require introducing other dependencies or doing unnecessarily complicated things, and I don't think we should do those.

So, longwinded explanation, but: I withdraw my concerns.

[picnixz]

No worries! I'll merge tomorrow (from my computer rather than my phone).

[picnixz]

@chrisjsewell Fun fact, I forgot about the commit message ! I just changed the PR title since I don't want to amend commits on master

[jayaddison]

@chrisjsewell @picnixz some trivia from archaeological code curiosity: the way that test code used to be provided with the status and warning output from a Sphinx test app build was that the fixture (as available using a @with_app decorator) would:

• Create fresh StringIO objects to hold the results.
• Call SphinxTestApp with those as warning/status attribute initializers.
• Call the underlying test case with the StringIO objects as arguments, so that the test can inspect them.

That approach was deprecated during the migration to pytest as far as I can tell. It seems like it was quite a neat approach.

[jayaddison]

@chrisjsewell @picnixz some trivia from archaeological code curiosity: the way that test code used to be provided with the status and warning output from a Sphinx test app build was that the fixture (as available using a @with_app decorator) would:

• Create fresh StringIO objects to hold the results.
• Call SphinxTestApp with those as warning/status attribute initializers.
• Call the underlying test case with the StringIO objects as arguments, so that the test can inspect them.

That approach was deprecated during the migration to pytest as far as I can tell. It seems like it was quite a neat approach.

The legacy of this remains in the current code, in a with_text_app decorator:

sphinx/tests/test_builders/test_build_text.py

Lines 9 to 15 in bf0bec3

	def with_text_app(*args, **kw):
	default_kw = {
	'buildername': 'text',
	'testroot': 'build-text',
	}
	default_kw.update(kw)
	return pytest.mark.sphinx(*args, **default_kw)

that remains in use.

[picnixz]

It's still being used when using the shared result by the way. You can see it in the app_params() fixture.

[jayaddison]

The way that the parameters were added as (optional, I think?) test arguments is what I think was particularly nice (although they were still StringIO objects, where I guess most test authors would want to compare against str values directly).

sphinx/tests/test_build.py

Line 82 in 5dd51a7

def test_circular_toctree(app, status, warning):

[picnixz]

Actually, that's what I would have expected as well because most of the time you do status.getvalue() when you request the fixture... so... I think we can (in the future) have a new fixture which simply returns status.getvalue() directly (something like app_stdout with normal messages and app_stderr with warnings).

[chrisjsewell]

when you request the fixture

personally I feel that having separate fixtures is over-complicating things 😬

but why we are on the subject, actually one of the most annoying things is having to deal with colored output of warnings;
in general, when you want to check warnings, you don't want the to have console color codes in.
Unless I am missing something, there is no clear way to temporarily turn off color codes

[picnixz]

Mmh, you should technically be able to turn off colors by adding NO_COLOR to the environment.

EDIT: Did not work, the logger appears to ignore this.
EDIT 2: ok it's easy to make it work. I'll do it tomorrow.

[chrisjsewell]

EDIT 2: ok it's easy to make it work. I'll do it tomorrow.

I would encorage you to have a look at #12130 first, lets have a think about this from a top-level perspective 😄

[picnixz]

personally I feel that having separate fixtures is over-complicating things 😬

Well, I agree that I would prefer using app.status instead of requesting status, but I think that having a fixture which gives you a view on the lines without colors (or possibly with colors) is better (even better, we should return an object that can be compared to list of strings + handle regex, a bit similar to _pytest.pytester.LineMatcher).

I would encorage you to have a look at #12130 first, lets have a think about this from a top-level perspective 😄

I've looked at it but to me it's something unrelated. Actually, the main issue is that the ColorizeFormatter of the logger does not care about whether colorization should be done or not.

[chrisjsewell]

Actually, the main issue is that the ColorizeFormatter of the logger does not care about whether colorization should be done or not.

As I've just commented in #12134 (comment), it does actually already obey the colorization setting