Pytest become 4x slower when updated to 3.3.0

[skontar]

Hi all.

We have updated the pytest from 3.2.5 to 3.3.0. The tests become 4 times slower (from 2 minutes to 8 minutes). I can reproduce it on my system running Fedora 26, Python 2.7.14, and our CI running RHEL 7, Python 2.7.13.

It can be easily fixed by downgrading pytest. It seems to be tied to a lot of text being printed to stdout by running tests. I cannot include the environment right now, neither I have minimal example and the code / tests are unfortunately not open source.

I will try to reproduce the issue in a minimal example and let you know. I wanted to report it now, maybe someone will know what change between those versions may have caused it.

[nicoddemus]

Hi @skontar, thanks for writing.

Please try:

1. Disable logging (-p no:logging).
2. Use old-style console output: -o console_output_style=classic.

[skontar]

1. Disable logging (-p no:logging).
   This definitely helps. I get the same performance in 3.3.0 with -p no:logging as in 3.2.5 without it.
2. Use old-style console output: -o console_output_style=classic.
   Did not help.

[pytestbot]

GitMate.io thinks the contributor most likely able to help you is @nicoddemus.

[nicoddemus]

Thanks @skontar for testing this.

Could you do some other tests please:

1. log level with --log-leve=WARNING
2. Disable live logging with -o log_cli_level=9999

FWIW we're discussing changes to the builtin logging in #3013; if you have thoughts on that we would love to hear them

[skontar]

Here is minimal example. On my system, it is 2s vs 30s between 3.2.5 to 3.3.0.

pytest_test.zip

[skontar]

@nicoddemus neither of those helps with minimal example.

[nicoddemus]

@skontar thanks a lot for the feedback.

The reproducible example is useful, it clearly demonstrates that the logging plugin generates a huge overhead with its default settings for suites which generate a ton of log messages.

For now it seems the only workaround is to disable the logging plugin with -p no:logging (this can be added to your pytest.ini as well btw).

I'm closing this for now and moving the discussion over to #3013.