lcov file is empty when using C# source generator #1099

aalmada · 2021-02-23T23:45:26Z

I've been using Coverlet with Coveralls on my project for a long time. I don't know when it started happening but now the report only includes methods from a dependent project.

I'm using the latest versions of xunit, Microsoft.NET.Test.Sdk, and coverlet.collector. I'm using net5. 0.

In the command line I'm typing:
dotnet test ./NetFabric.Hyperlinq.UnitTests/NetFabric.Hyperlinq.UnitTests.csproj -f:net5.0 -c:Release --collect:"XPlat Code Coverage" --settings coverlet.runsettings --diag:log.txt

The coverlet.runsettings file only has the following:

<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="XPlat code coverage">
        <Configuration>
          <Format>lcov</Format>
          <!-- <Include>[NetFabric.Hyperlinq]*</Include> -->
          <!-- <Exclude>[NetFabric.Assertive]*</Exclude> -->
          <UseSourceLink>true</UseSourceLink>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
</RunSettings>

In the generated logs I can see references to the classes of my main project but they keep not showing up in the coverage.info file.

The text was updated successfully, but these errors were encountered:

aalmada · 2021-02-23T23:56:57Z

Here's a portion of the host log:

TpTrace Warning: 0 : 14504, 15, 2021/02/23, 23:36:29.866, 1878667252485, testhost.dll, No DataCollection Data set for the test case 3139059c-ce64-cf4f-db62-ee57dda3ab00
TpTrace Warning: 0 : 14504, 15, 2021/02/23, 23:36:29.866, 1878667252791, testhost.dll, TestRunCache: InProgressTests is null
TpTrace Warning: 0 : 14504, 15, 2021/02/23, 23:36:29.866, 1878667253034, testhost.dll, TestRunCache: No test found corresponding to testResult 'NetFabric.Hyperlinq.UnitTests.Element.ElementAt.ReadOnlyListTests.ElementAt_Skip_Take_PredicateAt_With_ValidData_Must_Return_Some(source: [1], skipCount: -1, takeCount: 9, predicate: Func`3 { Method = Boolean <get_SkipTakePredicateAtSingle>b__113_4(Int32, Int32), Target = <>c { } })' in inProgress list.
TpTrace Verbose: 0 : 14504, 15, 2021/02/23, 23:36:29.867, 1878667253481, testhost.dll, TestExecutionRecorder.RecordResult: Received result for test: NetFabric.Hyperlinq.UnitTests.Element.ElementAt.ReadOnlyListTests.ElementAt_Skip_Take_PredicateAt_With_ValidData_Must_Return_Some.
TpTrace Warning: 0 : 14504, 15, 2021/02/23, 23:36:29.867, 1878667253777, testhost.dll, No DataCollection Data set for the test case 3139059c-ce64-cf4f-db62-ee57dda3ab00
TpTrace Warning: 0 : 14504, 15, 2021/02/23, 23:36:29.867, 1878667254018, testhost.dll, TestRunCache: InProgressTests is null
TpTrace Warning: 0 : 14504, 15, 2021/02/23, 23:36:29.867, 1878667254240, testhost.dll, TestRunCache: No test found corresponding to testResult 'NetFabric.Hyperlinq.UnitTests.Element.ElementAt.ReadOnlyListTests.ElementAt_Skip_Take_PredicateAt_With_ValidData_Must_Return_Some(source: [1], skipCount: 0, takeCount: 9, predicate: Func`3 { Method = Boolean <get_SkipTakePredicateAtSingle>b__113_5(Int32, Int32), Target = <>c { } })' in inProgress list.
TpTrace Verbose: 0 : 14504, 15, 2021/02/23, 23:36:29.867, 1878667254654, testhost.dll, TestExecutionRecorder.RecordResult: Received result for test: NetFabric.Hyperlinq.UnitTests.Element.ElementAt.ReadOnlyListTests.ElementAt_Skip_Take_PredicateAt_With_ValidData_Must_Return_Some.
TpTrace Warning: 0 : 14504, 15, 2021/02/23, 23:36:29.867, 1878667254942, testhost.dll, No DataCollection Data set for the test case 3139059c-ce64-cf4f-db62-ee57dda3ab00
TpTrace Warning: 0 : 14504, 15, 2021/02/23, 23:36:29.867, 1878667255181, testhost.dll, TestRunCache: InProgressTests is null
TpTrace Warning: 0 : 14504, 15, 2021/02/23, 23:36:29.867, 1878667255402, testhost.dll, TestRunCache: No test found corresponding to testResult 'NetFabric.Hyperlinq.UnitTests.Element.ElementAt.ReadOnlyListTests.ElementAt_Skip_Take_PredicateAt_With_ValidData_Must_Return_Some(source: [1], skipCount: 0, takeCount: 9, predicate: Func`3 { Method = Boolean <get_SkipTakePredicateAtSingle>b__113_6(Int32, Int32), Target = <>c { } })' in inProgress list.
TpTrace Verbose: 0 : 14504, 15, 2021/02/23, 23:36:29.867, 1878667255864, testhost.dll, TestExecutionRecorder.RecordResult: Received result for test: NetFabric.Hyperlinq.UnitTests.Element.ElementAt.ReadOnlyListTests.ElementAt_Skip_Take_PredicateAt_With_ValidData_Must_Return_Some.

Do all this "no"s mean there's something wrong?

petli · 2021-02-24T11:29:14Z

Am I correct in that unit test only references this project file:
https://github.com/NetFabric/NetFabric.Hyperlinq/blob/Coverage/NetFabric.Hyperlinq/NetFabric.Hyperlinq.csproj

It has these two project references in turn:

  <ItemGroup>
    <ProjectReference Include="..\NetFabric.Hyperlinq.Abstractions\NetFabric.Hyperlinq.Abstractions.csproj" />
    <ProjectReference Include="..\NetFabric.Hyperlinq.SourceGenerator\NetFabric.Hyperlinq.SourceGenerator.csproj" OutputItemType="Analyzer" ReferenceOutputAssembly="false" />  
  </ItemGroup>

For which of the projects are you getting coverage output?

aalmada · 2021-02-25T00:01:52Z

@petli Hi!

I'm using GitHub actions to build the solution. Let me point to results that I'm getting.

On the main branch, I'm using MSBUILD integration with the following script:

name: Build and test

on: [push]

jobs:
  windows:
    runs-on: windows-latest

    steps:
    - name: Check out repository
      uses: actions/checkout@v2
    - name: Setup .NET Core
      uses: actions/setup-dotnet@v1.7.2
      with:
        dotnet-version: '5.0.x'
    - name: Test source generator
      run: dotnet test ./NetFabric.Hyperlinq.SourceGenerator.UnitTests/NetFabric.Hyperlinq.SourceGenerator.UnitTests.csproj 
    - name: Build solution
      run: dotnet build
    - name: Tests and coverage
      run: dotnet test ./NetFabric.Hyperlinq.UnitTests/NetFabric.Hyperlinq.UnitTests.csproj -f:net5.0 -p:CollectCoverage=true -p:CoverletOutputFormat=lcov -p:CoverletOutput=TestResults/ /p:ExcludeByAttribute="GeneratedCodeAttribute" /p:ExcludeByFile="../NetFabric.Hyperlinq.Interfaces/*.cs"
    - name: Publish coverage report to coveralls.io
      uses: coverallsapp/github-action@master
      with:
        github-token: ${{ secrets.GITHUB_TOKEN }}
        path-to-lcov: ./NetFabric.Hyperlinq.UnitTests/TestResults/coverage.net5.0.info

You can see in Coveralls that it only includes coverage from classes in NetFabric.Hyperlinq.Abstractions. That's a dependency project.

Please ignore the ExcludeByAttribute in the script. It's not doing anything.

After many attempts, I decided to try VS Test integration instead. It can be found in the Coverage branch. I changed the script to the following:

name: Build and test

on: [push]

jobs:
  windows:
    runs-on: windows-latest

    steps:
    - name: Check out repository
      uses: actions/checkout@v2
    - name: Setup .NET Core
      uses: actions/setup-dotnet@v1.7.2
      with:
        dotnet-version: '5.0.x'
    - name: Test source generator
      run: dotnet test ./NetFabric.Hyperlinq.SourceGenerator.UnitTests/NetFabric.Hyperlinq.SourceGenerator.UnitTests.csproj 
    - name: Build solution
      run: dotnet build
    - name: Tests and coverage
      run: dotnet test ./NetFabric.Hyperlinq.UnitTests/NetFabric.Hyperlinq.UnitTests.csproj -f:net5.0 -c:Release --collect:"XPlat Code Coverage" --settings coverlet.runsettings
    - name: Get path to lcov file
      id: get_lcov_path
      shell: bash
      run: printf '::set-output name=lcov_path::%s\n' ./NetFabric.Hyperlinq.UnitTests/TestResults/*/coverage.info
    - name: Publish coverage report to coveralls.io
      uses: coverallsapp/github-action@master
      with:
        github-token: ${{ secrets.GITHUB_TOKEN }}
        path-to-lcov: ${{ steps.get_lcov_path.outputs.lcov_path }}

with the coverlet.runsettings as

<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="XPlat code coverage">
        <Configuration>
          <Format>lcov</Format>
          <Include>[NetFabric.Hyperlinq]*</Include>
          <Exclude>[NetFabric.Assertive]*</Exclude>
          <UseSourceLink>true</UseSourceLink>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
</RunSettings>

Now I get the following:

Using lcov file: ./NetFabric.Hyperlinq.UnitTests/TestResults/8315cd24-363c-46e7-b38a-f835c342f447/coverage.info
Error:  "2021-02-23T23:21:14.739Z"  'error from lcovParse: ' 'Failed to parse string'
Error:  "2021-02-23T23:21:14.741Z"  'input: ' ''
Error:  "2021-02-23T23:21:14.741Z"  'error from convertLcovToCoveralls'
Error: Failed to parse string

This is because the file coverage.info is empty.

Sorry for wasting your time but I've tried everything I could...

petli · 2021-02-25T10:10:14Z

EDIT: I think I misunderstood, so I've removed this comment. I need to think a bit more to understand what's happening.

petli · 2021-02-25T14:09:56Z

I think I found it, have a look at this comment: #1084 (comment)

When I modify https://github.com/NetFabric/NetFabric.Hyperlinq/blob/Coverage/NetFabric.Hyperlinq.SourceGenerator/OverloadsGenerator.cs#L40 to change the file ending to .g.cs I get code coverage generated locally. Can you give it a shot and see if it works in your CI flow too?

aalmada · 2021-02-25T23:32:42Z

Oh! So, it was the source generator...
Sorry but I haven't looked in a while into the coverage reports so I could not really say what changed to make this happen. Also, sorry for failing to mention that I'm using a source generator.
Running from the command line, I can see that the coverage.info file contains information. I now need to figure out why Coveralls reports no coverage... 🤔

aalmada · 2021-03-01T23:17:42Z

Hi! Opening this issue again because it's still not fully working.
So, now the coverage.info file is not empty but I'm not getting a coverage report as I had with MSBUILD integration.
The log is very long but in the end, there's this:

  "AttachmentSets": [
    {
      "Uri": "datacollector://Microsoft/CoverletCodeCoverage/1.0",
      "DisplayName": "XPlat code coverage",
      "Attachments": [
        {
          "Description": "",
          "Uri": "file://C:/Projects/NetFabric.Hyperlinq/NetFabric.Hyperlinq.UnitTests/TestResults/42616beb-30b0-4046-ba50-fd0e53cafb7e/coverage.info"
        }
      ]
    }
  ],
  "Metrics": {}
}

Should Metrics have something?

MarcoRossignoli · 2021-03-02T07:21:25Z

That logs looks strange we generate only coverage.info, from where it comes from?that seems pretty easy(list of string https://github.com/coverlet-coverage/coverlet/blob/master/src/coverlet.core/Reporters/LcovReporter.cs#L17).
Can you "diff" msbuild an new one coverage.info to understand what is missing(classes/methods)?

aalmada · 2021-03-08T23:56:29Z

Hi @MarcoRossignoli

That JSON is from the log.txt generated when following the steps in troubleshooting.

Comparing the two outputs, as you suggested:

When I use VSTest integration the result is

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.

Attachments:
  C:\Projects\NetFabric.Hyperlinq\NetFabric.Hyperlinq.UnitTests\TestResults\86455514-8fe7-47a2-a86b-06908e6e89b8\coverage.info
Passed!  - Failed:     0, Passed:  7077, Skipped:     0, Total:  7077, Duration: 4 s - NetFabric.Hyperlinq.UnitTests.dll (net5.0)

and generates a 2,293 KB coverage.info file.

When I use MSBuild integration the result is

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.

Passed!  - Failed:     0, Passed:  7077, Skipped:     0, Total:  7077, Duration: 4 s - NetFabric.Hyperlinq.UnitTests.dll (net5.0)

Calculating coverage result...
  Generating report 'TestResults\coverage.net5.0.info'

+----------------------------------+--------+--------+--------+
| Module                           | Line   | Branch | Method |
+----------------------------------+--------+--------+--------+
| NetFabric.Hyperlinq.Abstractions | 19.04% | 9.52%  | 19.04% |
+----------------------------------+--------+--------+--------+
| NetFabric.Hyperlinq              | 51.3%  | 56.09% | 33.15% |
+----------------------------------+--------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 51.18% | 55.42% | 33.04% |
+---------+--------+--------+--------+
| Average | 35.17% | 32.8%  | 26.09% |
+---------+--------+--------+--------+

and generates a 2,296 KB coverage.net5.0.info file.

The diff shows that the coverage.info file is missing many of the DA: lines existing in the coverage.net5.0.info file.

So, this is no longer related to the C# code generator. Maybe I should file a different issue...

MarcoRossignoli · 2021-03-09T07:01:31Z

If you see difference between VSTest integration(collectors) and msbuild integration(less coverage) it's likely you're hitting a known issue related to test engine behavior https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/KnownIssues.md#vstest-stops-process-execution-early

The most reliable way for coverlet is VSTest collectors integration.
The same coverage engine is shared between all three drivers(collectors, msbuild tasks, .NET Tool).

aalmada · 2021-03-09T10:07:03Z

Hi @MarcoRossignoli

It's good to know that the most reliable is VSTest collectors integration and that's what I'm trying to use but unsuccessfully.

For me, it's not reporting 0% coverage. It's not reporting. The outputs I show above come from the command line, not CI. Note that for VSTest integration there are no tables shown.

I have it working with MSBUILD integration: https://coveralls.io/github/NetFabric/NetFabric.Hyperlinq

MarcoRossignoli · 2021-03-09T10:34:13Z

Note that for VSTest integration there are no tables shown.

These are the differences between integrations https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/DriversFeatures.md

MarcoRossignoli · 2021-03-09T10:36:50Z

The diff shows that the coverage.info file is missing many of the DA: lines existing in the coverage.net5.0.info file.

Do you have the filename/class/lines details?

aalmada · 2021-03-22T23:24:20Z

I'm going to close this issue. I'm not sure if the issue is with coverlet or CoverAlls. It works with MSBUILD integration but not with VSTest integration.
Thanks for all the help!

MarcoRossignoli · 2021-03-30T10:44:32Z

Sorry for delay @aalmada super busy period, anyway should work also with vstest, I'll keep this open for future investigation

potatopeelings · 2021-05-09T22:01:44Z

I ran into a similar issue and was able to replicate it using a simple hello world project and the command line. Repo repository here - https://github.com/potatopeelings/coverlet1099 with coverlet.msbuild 3.0.3

I have 3 projects

console with Program.cs printing hello world
tests just running the Main static method
source generator project generating an additional class (which is not used)

If I do

dotnet test . -r C:\Temp\ -p:CollectCoverage=true -p:CoverletOutputFormat=\"opencover\" /p:CoverletOutput=C:\Temp\opencover.xml

I get no Module nodes in opencover.xml and the coverage result looks like this

+--------+------+--------+--------+
| Module | Line | Branch | Method |
+--------+------+--------+--------+

+---------+------+--------+--------+
|         | Line | Branch | Method |
+---------+------+--------+--------+
| Total   | 100% | 100%   | 100%   |
+---------+------+--------+--------+
| Average | NaN% | NaN%   | NaN%   |
+---------+------+--------+--------+

If I simply remove the reference to the generator project, I get the expected Module nodes and this

+---------------------+------+--------+--------+
| Module              | Line | Branch | Method |
+---------------------+------+--------+--------+
| coverlet1099Console | 50%  | 100%   | 50%    |
+---------------------+------+--------+--------+

+---------+------+--------+--------+
|         | Line | Branch | Method |
+---------+------+--------+--------+
| Total   | 50%  | 100%   | 50%    |
+---------+------+--------+--------+
| Average | 50%  | 100%   | 50%    |
+---------+------+--------+--------+

Note: I was running into some weird caching like issues when trying to replicate (I had to sometimes remove and re-add the generator project reference for it to stick). Usually toggling the project reference and rerunning a couple of times sorted that out.

petli · 2021-05-10T10:36:05Z

@potatopeelings Your issue is likely that the generated file does not end in .g.cs, which is what coverlet uses to identify them (as a somewhat de-facto standard). Try changing it to that and see if you get a result. (See also #1084)

potatopeelings · 2021-05-10T11:04:40Z

@petli I don't emit any files in that - the source generator generated classes get directly added to the compilation.

edits:

I use version 3.0.3.
I think its possible related to Code coverage not working with MVC/Razor applications targeting ASP.NET 6 preview 3+ #1140 (which I missed on my initial search because I was looking for SourceGenerators).

petli · 2021-05-10T12:50:03Z

As I understand it a source generator always emits files, and you say what file name should be when it is generated with the first parameter to AddSource:
https://github.com/potatopeelings/coverlet1099/blob/master/coverlet1099Generator/Program.cs#L32
You can try adding .g.cs to the end of the name there, and see if that helps.

potatopeelings · 2021-05-10T13:00:46Z

Thanks! That worked. I seem to have misunderstood the <EmitCompilerGeneratedFiles> setting.

Apologies for the false alarm.

petli · 2021-05-10T13:14:15Z

Glad it worked, @potatopeelings! Those files end up in the obj directory so they can be inspected to aid in tracking down issues with the generated code.

MarcoRossignoli · 2022-01-16T09:26:58Z

Closing for stale conversation, feel free to reopen if needed.
We have this one in the queue also for this kind of issues #1164

daveMueller added the untriaged To be investigated label Feb 24, 2021

daveMueller removed the untriaged To be investigated label Feb 25, 2021

MarcoRossignoli added the untriaged To be investigated label Feb 25, 2021

petli closed this as completed Feb 25, 2021

petli added waiting for customer Waiting for customer action and removed untriaged To be investigated labels Feb 25, 2021

petli reopened this Feb 25, 2021

aalmada changed the title ~~Coverage report only includes methods from a dependent project~~ lcov file is empty when using C# source generator Feb 27, 2021

aalmada closed this as completed Feb 27, 2021

aalmada reopened this Mar 1, 2021

aalmada closed this as completed Mar 22, 2021

MarcoRossignoli reopened this Mar 30, 2021

MarcoRossignoli added untriaged To be investigated with repro Issue with repro and removed waiting for customer Waiting for customer action labels Mar 30, 2021

MarcoRossignoli closed this as completed Jan 16, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

lcov file is empty when using C# source generator #1099

lcov file is empty when using C# source generator #1099

aalmada commented Feb 23, 2021 •

edited

aalmada commented Feb 23, 2021

petli commented Feb 24, 2021

aalmada commented Feb 25, 2021

petli commented Feb 25, 2021 •

edited

petli commented Feb 25, 2021

aalmada commented Feb 25, 2021

aalmada commented Mar 1, 2021

MarcoRossignoli commented Mar 2, 2021 •

edited

aalmada commented Mar 8, 2021 •

edited

MarcoRossignoli commented Mar 9, 2021 •

edited

aalmada commented Mar 9, 2021

MarcoRossignoli commented Mar 9, 2021 •

edited

MarcoRossignoli commented Mar 9, 2021

aalmada commented Mar 22, 2021

MarcoRossignoli commented Mar 30, 2021

potatopeelings commented May 9, 2021 •

edited

petli commented May 10, 2021

potatopeelings commented May 10, 2021 •

edited

petli commented May 10, 2021

potatopeelings commented May 10, 2021

petli commented May 10, 2021

MarcoRossignoli commented Jan 16, 2022 •

edited

lcov file is empty when using C# source generator #1099

lcov file is empty when using C# source generator #1099

Comments

aalmada commented Feb 23, 2021 • edited

aalmada commented Feb 23, 2021

petli commented Feb 24, 2021

aalmada commented Feb 25, 2021

petli commented Feb 25, 2021 • edited

petli commented Feb 25, 2021

aalmada commented Feb 25, 2021

aalmada commented Mar 1, 2021

MarcoRossignoli commented Mar 2, 2021 • edited

aalmada commented Mar 8, 2021 • edited

MarcoRossignoli commented Mar 9, 2021 • edited

aalmada commented Mar 9, 2021

MarcoRossignoli commented Mar 9, 2021 • edited

MarcoRossignoli commented Mar 9, 2021

aalmada commented Mar 22, 2021

MarcoRossignoli commented Mar 30, 2021

potatopeelings commented May 9, 2021 • edited

petli commented May 10, 2021

potatopeelings commented May 10, 2021 • edited

petli commented May 10, 2021

potatopeelings commented May 10, 2021

petli commented May 10, 2021

MarcoRossignoli commented Jan 16, 2022 • edited

aalmada commented Feb 23, 2021 •

edited

petli commented Feb 25, 2021 •

edited

MarcoRossignoli commented Mar 2, 2021 •

edited

aalmada commented Mar 8, 2021 •

edited

MarcoRossignoli commented Mar 9, 2021 •

edited

MarcoRossignoli commented Mar 9, 2021 •

edited

potatopeelings commented May 9, 2021 •

edited

potatopeelings commented May 10, 2021 •

edited

MarcoRossignoli commented Jan 16, 2022 •

edited