Running coverlet global tool in docker with already compiled binary does not find any coverage

[mabead]

I have a case where the coverlet global tool works fine when executed directly on my machine (windows 10) but that fails to report coverage when executed in docker against an already compiled dll.

Here's a procedure to reproduce the issue.

First clone the following repository that contains a very simple test project.

> git clone https://github.com/mabead/CoverletInDocker
> cd CoverletInDocker

Compile the solution directly on your host machine (mine was a windows machine).

> dotnet build --configuration Release

Make sure that invoking coverlet outside of docker works fine. Note that I explicitely pass the dll to the 'dotnet test' command. This will be required later under the docker.

> coverlet .\SomeProject.Tests\bin\Release\netcoreapp3.1\SomeProject.Tests.dll --target dotnet --targetargs "test .\SomeProject.Tests\bin\Release\netcoreapp3.1\SomeProject.Tests.dll"
Microsoft (R) Test Execution Command Line Tool Version 17.0.0
Copyright (c) Microsoft Corporation.  All rights reserved.
Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Good morning
Good night
Passed!  - Failed:     0, Passed:     2, Skipped:     0, Total:     2, Duration: 4 ms - SomeProject.Tests.dll (netcoreapp3.1)

Calculating coverage result...
  Generating report 'C:\Users\mbeaudry\CoverletInDocker\coverage.json'
+-------------+--------+--------+--------+
| Module      | Line   | Branch | Method |
+-------------+--------+--------+--------+
| SomeProject | 66.66% | 50%    | 100%   |
+-------------+--------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 66.66% | 50%    | 100%   |
+---------+--------+--------+--------+
| Average | 66.66% | 50%    | 100%   |
+---------+--------+--------+--------+

Ok, so we know that coverlet works fine and that the line coverage is 66.66%. Let's now open a docker prompt where we will try to do about the same thing:

> docker run --rm -it -v YOUR_CURRENT_DIRECTORY:/src -w /src mcr.microsoft.com/dotnet/core/sdk:3.1 bash

You should now be in a bash shell within docker. Let's first install coverlet. Note that I install version 1.7.2 because I am using .net core 3.1

> dotnet tool install -g coverlet.console --version 1.7.2
> export PATH="$PATH:/root/.dotnet/tools"

And finally, let's run coverlet within docker:

> coverlet ./SomeProject.Tests/bin/Release/netcoreapp3.1/SomeProject.Tests.dll --target dotnet --targetargs "test ./SomeProject.Tests/bin/Release/netcoreapp3.1/SomeProject.Tests.dll"
Microsoft (R) Test Execution Command Line Tool Version 16.7.1
Copyright (c) Microsoft Corporation.  All rights reserved.
Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Good morning
Good night
Test Run Successful.
Total tests: 2
     Passed: 2
 Total time: 1.4477 Seconds

Calculating coverage result...
  Generating report '/src/coverage.json'
+--------+------+--------+--------+
| Module | Line | Branch | Method |
+--------+------+--------+--------+

+---------+------+--------+--------+
|         | Line | Branch | Method |
+---------+------+--------+--------+
| Total   | 100% | 100%   | 100%   |
+---------+------+--------+--------+
| Average | NaN% | NaN%   | NaN%   |
+---------+------+--------+--------+

As you can see, my tests are executed correctly but coverlet does not report any coverage.

Let's now try to recompile my project in docker and run coverlet again

> dotnet build --configuration Release
> coverlet ./SomeProject.Tests/bin/Release/netcoreapp3.1/SomeProject.Tests.dll --target dotnet --targetargs "test ./SomeProject.Tests/bin/Release/netcoreapp3.1/SomeProject.Tests.dll"
Microsoft (R) Test Execution Command Line Tool Version 16.7.1
Copyright (c) Microsoft Corporation.  All rights reserved.
Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Good morning
Good night
Test Run Successful.
Total tests: 2
     Passed: 2
 Total time: 1.4434 Seconds

Calculating coverage result...
  Generating report '/src/coverage.json'
+-------------+--------+--------+--------+
| Module      | Line   | Branch | Method |
+-------------+--------+--------+--------+
| SomeProject | 66.66% | 50%    | 100%   |
+-------------+--------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 66.66% | 50%    | 100%   |
+---------+--------+--------+--------+
| Average | 66.66% | 50%    | 100%   |
+---------+--------+--------+--------+

We now see that coverlet now works fine. But unfortunately, I had to recompile my code in docker. For this small project, recompiling is not a big problem but in large projects, this can add significant delays to the CI.

So, do you have any idea on how I can make this coverage work under docker without recompiling my code?

[MarcoRossignoli]

Thanks for the repro, at first glance it should work as you expect. We'll take a look asap.

[petli]

I suspect this happens because coverlet can't find the source code at the same path inside the docker container as outside it. When instrumenting DLLs, it checks that the sources can be found as a heuristic to avoid instrumenting third-party libraries without requiring you to filter out them manually. The docker command

docker run --rm -it -v YOUR_CURRENT_DIRECTORY:/src -w /src mcr.microsoft.com/dotnet/core/sdk:3.1 bash

remaps the source directory to /src which most likely isn't the path outside docker. That's why it works when you build inside docker and then test: coverlet can find the source code files referenced by the built DLLs, since they now reference the code in the /src directory inside the container.

#1164 suggest a new parameter to let you control this heuristic which could help with this use case, but it would be at the price of having to filter out any third-party libraries so they don't clutter up your results.

[mabead]

Thanks for the hint @petli . Wit this in mind, I was able to successfully run coverlet in docker when the assembly was previously compiled on linux. For this to work, I had to replace -v YOUR_CURRENT_DIRECTORY:/src by -v YOUR_CURRENT_DIRECTORY:YOUR_CURRENT_DIRECTORY. That way, the docker path and local paths are identical, thus coverlet properly detects the assemblies to instrument.

This solution is obviously not perfect since it does not work when the host machine is running Windows. But for my use case, it does help me to fix what I wanted. Thanks!

[MarcoRossignoli]

I'll close as a dup and as @petli said we have already some solution in mind #1164