Deprecate includeUncoveredFiles configuration option

[nostadt]

Hi Sebastian,

before I start, I want to give my appreciation for the time and effort you put into this project.

In the code coverage section (v11.3) of the documentation it is mentioned that it is highly recommended to keep the default (true) for includeUncoveredFiles. Otherwise the report is not complete and honest. That made me wonder, why can I disable it at all? Perhaps it can be removed, with the goal to reduce complexity.

Looking forward to hearing from you,

Alexander

[sebastianbergmann]

That made me wonder, why can I disable it at all? Perhaps it can be removed, with the goal to reduce complexity.

Good question and excellent point. Thank you for raising it!

Before php-code-coverage used static analysis (using PHP-Parser) to (also) analyse uncovered files, uncovered files were actually loaded using require. This resulted in any code outside of code units such as classes or functions to be executed. This in turn could cause problems and was the reason why it was made optional whether uncovered files should be processed or not.

So, yes, the includeUncoveredFiles option could have been removed a while ago.

[nostadt]

You're welcome. Thank you for the response and happy to hear that it can be removed nowadays.

[mgleska]

There is one important reason to leave includeUncoveredFiles as it was before - daily work performance.

In my daily work I often run coverage test for only one test file. To check if my recent update of test file covers all lines and branches in tested class.
With includeUncoveredFiles=false, on coverage result index page I get only tested class and a few related files.
With includeUncoveredFiles=true, on coverage result index page I get coverage result for whole system - mostly with negative coverage score.

For very simple system it looks like this:

vs

For real (big) system second list will be much longer.
It is easy to see, that on longer list we get some (a lot of) unwanted information, which is useless in this context and disturb quick perception of coverage result of tested class.

@sebastianbergmann Please consider restoring includeUncoveredFiles in config file or adding a CLI option for to enable this functionality.

[sebastianbergmann]

@theseer, @localheinz, @staabm, @Schrank, @sebastianheuer, @Tesla91, and I discussed this issue during the PHPUnit Code Sprint in Munich today. We would like to understand why you look at the overview page when you seem to only be interested in the details of a single source file (or a subdirectory of the project root directory).

[mgleska]

@sebastianbergmann Because I am starting browsing of coverage results from index.html file.

Here is the content of coverage directory from my example (generated without includeUncoveredFiles=false):

/app/coverage # ls -p
Admin/                  CommonInfrastructure/   Kernel.php_branch.html  Printer/                _js/
Api/                    Customer/               Kernel.php_path.html    _css/                   dashboard.html
Auth/                   Kernel.php.html         Order/                  _icons/                 index.html

index.html is a natural starting point.

Now I see, that I can go to CommonInfrastructure directory and open index.html from this location.
Before this discussion I never thought to do so.

OK. I can start browsing in specific directory.
But there is another issue - performance. From my small demo project I get following execution time:

includeUncoveredFiles="false"	includeUncoveredFiles="true"
[00:00.070]	[00:02.256]
The src directory contains less then 100 files.

Is there any other option to not waste my time, CPU power and energy?
How to not waste resources? I want to save the planet 😃

[janedbal]

Removing this flag significantly slows down our CI jobs:

• XML generation time:
  • Before: [00:00.002]
  • After: [01:03.994] ⚠️

This would make us unable to upgrade to PHPUnit 12. Let me explain our usecases:

• We have so many tests that we run them in 16 parallel CI jobs (in order to keep each CI job time around 7 mins)
  • Thus, we need partial coverage reports that are later merged together
• We also generate code coverage for changed files only
  • This is then used only to show in GitLab UI so that reviewer knows which parts of the code is not tested in the MR diff
  • Thus, we need partial coverage reports

@sebastianbergmann Would you please consider some change to enable those usecases again? I believe many huge projects will face this.

[sebastianbergmann]

The deprecation of the includeUncoveredFiles configuration option has been reverted in 11.5 and the removal of the includeUncoveredFiles configuration option has been reverted in main.

[janedbal]

Thank you very much!

[janedbal]

and the removal of the includeUncoveredFiles configuration option has been reverted in main.

@sebastianbergmann I believe it was never properly reverted within phpunit, I just tried PHPUnit 12 and the generation time is huge again. Can you double check please?

[sebastianbergmann]

Thank you for bringing this to my attention.

It appers that somehow the commit reverting the removal of the includeUncoveredFiles configuration option got lost. It has been restored in e743ec6. Can you please check with the current state of 12.0? Thank you!

[fedorovsergey]

Hello! Could you please cherry-pick it in 11.5 version too. Your first revert was there. Now in release 11.5 xsd allows to use attribute "includeUncoveredFiles" but it takes no effect because code does not exists.

[sebastianbergmann]

The code in question was never removed in 11.5. It was deprecated in ee2a7c6. This deprecation was reverted in bfab44d.

[janedbal]

dev-main seems to be back to ~zero, thank you!