experiment with the effect of different terminal widths on the test result and testing in general

[RonnyPfannschmidt]

when we took note that certain terminal widths affect the testing, we just configured a default for the full test suite

however this demonstrated that we have a blind spot there
we really need to get a idea what happens to the tests/ux in general when the width is either too small or too big

initial experiments could just vary COLUMNS to identify the areas where things may go south - once pinpointed exact tests an be added

Originally posted by @RonnyPfannschmidt in #11425 (comment)

[Tbruno25]

I'm happy to submit a PR for this 👍

[RonnyPfannschmidt]

Please give it a go

[Tbruno25]

For initial characterization I ran the test suite and logged its output for all column values between 40 and 400 which produced the following result

Tests Failed	Column Value
0	40-84, 86-97, 350-400
1	85
2	n/a
3	98-199, 201-349
4	200

It seems only a small handful of tests are affected which appear to be the same that were identified in #11423

I recommend we refactor these offending tests to be agnostic to the column size -- which I'm happy to do 🙂

It also might be nice to add a test which runs the suite with a subset of different column values to preserve this behavior, however maybe this should exist within CI only as it will take longer to complete.

Thoughts?

[Tbruno25]

After evaluating the failed tests, I added the following test case to test_pytester.py which seems like it should pass successfully but doesn't.

def test_linematcher_wildcards_on_same_line() -> None:
    lm = LineMatcher(["Foo Bar Baz"])
    lm.fnmatch_lines(["*Foo*", "*Bar*"])

Seems like a bug within the LineMatcher._match_lines method.

[RonnyPfannschmidt]

That particular line match test should indeed fail,

The match asks for 2 lines,one is given

[Tbruno25]

That particular line match test should indeed fail,

The match asks for 2 lines,one is given

I suppose? With consecutive=False, I guess I would expect linematcher to check that the wildcard matches any line within the input. However, based on your comment it seems like the requirement that each line must be unique is the correct behavior?