Slow performance on discovery / running due to discovering TestReporters

[kiwidev]

Once a test project has a number of files inside the output folder, then discovery of test reporters can add significant time onto both discovery and run phases inside Visual Studio. On a project with 100 dlls + regional versions (not a large number with modern libs), iterating through all of the dlls is adding on 1-3 seconds to each phase. This makes for a slow red-green-refactor cycle inside VS. (The current code iterates through every type inside every dll in the output folder).

I've done some local tests where I explicitly specified the list of dlls containing reporters (e.g. xunit.runner.reporters.netcoreapp10.dll) inside a runsettings file and this eliminated the overhead / reduced it to < 50ms. As an example, with a simple project this takes the "run test" process after changing a file (discovery and then run, excluding build) from 4 seconds to 1.5.

Would the team be open to some sort of explicit flag like this inside runsettings or xunit config? Or is there a better way to do it?

Example (additional logging inserted by me, warm runs with fast NVMe disk, VS2022):

========== Starting test discovery ==========
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.6-preview.3+7479759cc5 (64-bit .NET 6.0.5)
[xUnit.net 00:00:00.08] Get runner reporter
[xUnit.net 00:00:01.31] After Get runner reporter
[xUnit.net 00:00:01.36]   Discovering: Foo.RedactedTests
[xUnit.net 00:00:01.73]   Discovered:  Foo.RedactedTests
========== Test discovery finished: 400 Tests found in 2.3 sec ==========
Executing test method: Foo.RedactedTests.EmptyTest
Using automatically detected runsettings file(s). To learn more visit https://aka.ms/vs-runsettings.
========== Starting test run ==========
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.6-preview.3+7479759cc5 (64-bit .NET 6.0.5)
[xUnit.net 00:00:00.02] Get runner reporter
[xUnit.net 00:00:01.24] After get runner reporter
[xUnit.net 00:00:01.32]   Starting:    Foo.RedactedTests
[xUnit.net 00:00:01.47]   Finished:    Foo.RedactedTests
========== Test run finished: 1 Tests (1 Passed, 0 Failed, 0 Skipped) run in 1.5 sec ==========

With config on:

========== Starting test discovery ==========
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.6-preview.3+7479759cc5 (64-bit .NET 6.0.5)
[xUnit.net 00:00:00.09] Get runner reporter
[xUnit.net 00:00:00.11] After Get runner reporter
[xUnit.net 00:00:00.21]   Discovering: Foo.RedactedTests
[xUnit.net 00:00:00.54]   Discovered:  Foo.RedactedTests
========== Test discovery finished: 400 Tests found in 1.2 sec ==========
Executing test method: Foo.RedactedTests.EmptyTest
Using automatically detected runsettings file(s). To learn more visit https://aka.ms/vs-runsettings.
========== Starting test run ==========
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.6-preview.3+7479759cc5 (64-bit .NET 6.0.5)
[xUnit.net 00:00:00.02] Get runner reporter
[xUnit.net 00:00:00.04] After get runner reporter
[xUnit.net 00:00:00.16]   Starting:    Foo.RedactedTests
[xUnit.net 00:00:00.29]   Finished:    Foo.RedactedTests
========== Test run finished: 1 Tests (1 Passed, 0 Failed, 0 Skipped) run in 310 ms ==========

[nohwnd]

I've done some local tests where I explicitly specified the list of dlls containing reporters .. inside a runsettings file.

What exactly did you do?

On a project with 100 dlls + regional versions (not a large number with modern libs), iterating through all of the dlls is adding on 1-3 seconds to each phase.

Is this specific to xunit? Or is that vstest problem in general, do you have some logs/code where I can see the iterating happening?

Would the team be open to some sort of explicit flag like this inside runsettings or xunit config? Or is there a better way to do it?

We have extension loading policy options in vstest, which by default look for all dlls based on naming convention plus some other based on folder location. There is also a strict explicit policy where you need to list in runsettings the exact dlls. We are trying to steer away from inspecting all possibly relevant dlls, because as you say it adds a lot of unnecessary overhead.

[kiwidev]

Hi,

The code that's causing the issue is:
https://github.com/xunit/visualstudio.xunit/blob/main/src/xunit.runner.visualstudio/VsTestRunner.cs#L691

where the VsTestRunner iterates through every type in every dll in the path attempting to find reporters.

The local change I made for testing passed in a set of explicit dlls into that method and used that to filter the dlls that would be enumerated.

This only affects the VS / dotnet plugin (both when calling dotnet test and through Visual Studio).

[PureKrome]

Also @kiwidev - could be kewl to have some ways to use wildcards, instead of having to list each one. So if you know your namespace (which should match the filenames, in this scenario) .. you can do MyTestProject.* and that should hopefully reduce the list of files to read in, let alone find the reporters in them?

[nohwnd]

Mentioned it also here, but this thread is more appropriate.

FWIW in vstest we currently use a naming convention e.g. *TestAdapter.dll, and then an assembly level attribute also by convention (we know the shape of it, but the extension author defines it in their extension):

[assembly: TestExtensionTypes(new global::System.Type[]
{
	typeof(MSTestDiscoverer),
	typeof(MSTestExecutor)
})]

When found we inspect only those types for the desired type of extension, rather than inspecting all types.

[justhoma-zz]

The introduction of parallel test discovery has definitely helped alleviate the issue, however it would be even better if this discovery process could be optimized.

[bradwilson]

@nohwnd That attribute is marked as internal (and not in Microsoft.TestPlatform.ObjectModel) so it doesn't appear to be something that's targeted at general purpose usage.