stopOnFail is ignored when synchronous message reporting is requested

[tteguayco]

I have a project with some xunit tests which includes xunit.runner.visualstudio as a dependency (targeting version 2.5.1).

Inside that project, I added an xunit.runner.json file with the following content:

{
  "$schema": "https://xunit.net/schema/current/xunit.runner.schema.json",
  "parallelizeTestCollections": false,
  "stopOnFail": true
}

When running my tests from VisualStudio 2022, the option parallelizeTestCollections seems to be working fine but stopOnFail is completely ignored; even if one test fails, all the remaining will run (even if the latter started after the failing one finished).

Am I missing something here? Happy to provide more context as necessary.

[bradwilson]

What versions of the NuGet packages are you using? Can you provide a repro project?

[bradwilson]

Here you can see me illustrate the feature working: #189 (comment)

[bradwilson]

With this unit test:

using Xunit;

namespace Empty;

public class UnitTest
{
        [Fact] public void Passing1() { }
        [Fact] public void Passing2() { }
        [Fact] public void Passing3() { }
        [Fact] public void Passing4() { }
        [Fact] public void Failing1() => Assert.Fail("I'm a failing test");
        [Fact] public void Passing5() { }
        [Fact] public void Passing6() { }
        [Fact] public void Passing7() { }
        [Fact] public void Passing8() { }
        [Fact] public void Failing2() => Assert.Fail("I'm a failing test");
        [Fact] public void Passing9() { }
        [Fact] public void Passing10() { }
        [Fact] public void Passing11() { }
        [Fact] public void Passing12() { }
        [Fact] public void Failing3() => Assert.Fail("I'm a failing test");
        [Fact] public void Passing13() { }
        [Fact] public void Passing14() { }
        [Fact] public void Passing15() { }
        [Fact] public void Passing16() { }
        [Fact] public void Failing4() => Assert.Fail("I'm a failing test");
        [Fact] public void Passing17() { }
        [Fact] public void Passing18() { }
        [Fact] public void Passing19() { }
        [Fact] public void Passing20() { }
        [Fact] public void Failing5() => Assert.Fail("I'm a failing test");
}

and this project:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <LangVersion>11.0</LangVersion>
    <Nullable>enable</Nullable>
    <TargetFramework>net6.0</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.7.2" />
    <PackageReference Include="xunit" Version="2.6.0" />
    <PackageReference Include="xunit.runner.visualstudio" Version="2.5.3" />
  </ItemGroup>

</Project>

dotnet test:

  All projects are up-to-date for restore.
  empty -> C:\Dev\repro\empty\bin\Debug\net6.0\empty.dll
Test run for C:\Dev\repro\empty\bin\Debug\net6.0\empty.dll (.NETCoreApp,Version=v6.0)
Microsoft (R) Test Execution Command Line Tool Version 17.7.2 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
[xUnit.net 00:00:00.13]     Empty.UnitTest.Failing2 [FAIL]
[xUnit.net 00:00:00.13]     Empty.UnitTest.Failing1 [FAIL]
  Failed Empty.UnitTest.Failing2 [2 ms]
  Error Message:
   I'm a failing test
  Stack Trace:
     at Empty.UnitTest.Failing2() in C:\Dev\repro\empty\UnitTest.cs:line 16
  Failed Empty.UnitTest.Failing1 [< 1 ms]
  Error Message:
   I'm a failing test
  Stack Trace:
     at Empty.UnitTest.Failing1() in C:\Dev\repro\empty\UnitTest.cs:line 11
[xUnit.net 00:00:00.15]     Empty.UnitTest.Failing5 [FAIL]
  Failed Empty.UnitTest.Failing5 [< 1 ms]
  Error Message:
   I'm a failing test
  Stack Trace:
     at Empty.UnitTest.Failing5() in C:\Dev\repro\empty\UnitTest.cs:line 31
[xUnit.net 00:00:00.15]     Empty.UnitTest.Failing4 [FAIL]
[xUnit.net 00:00:00.15]     Empty.UnitTest.Failing3 [FAIL]
  Failed Empty.UnitTest.Failing4 [< 1 ms]
  Error Message:
   I'm a failing test
  Stack Trace:
     at Empty.UnitTest.Failing4() in C:\Dev\repro\empty\UnitTest.cs:line 26
  Failed Empty.UnitTest.Failing3 [< 1 ms]
  Error Message:
   I'm a failing test
  Stack Trace:
     at Empty.UnitTest.Failing3() in C:\Dev\repro\empty\UnitTest.cs:line 21

Failed!  - Failed:     5, Passed:    20, Skipped:     0, Total:    25, Duration: 22 ms - empty.dll (net6.0)

dotnet test -- xUnit.StopOnFail=true:

  Determining projects to restore...
  Restored C:\Dev\repro\empty\empty.csproj (in 261 ms).
  empty -> C:\Dev\repro\empty\bin\Debug\net6.0\empty.dll
Test run for C:\Dev\repro\empty\bin\Debug\net6.0\empty.dll (.NETCoreApp,Version=v6.0)
Microsoft (R) Test Execution Command Line Tool Version 17.7.2 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
[xUnit.net 00:00:00.13]     Empty.UnitTest.Failing2 [FAIL]
  Failed Empty.UnitTest.Failing2 [2 ms]
  Error Message:
   I'm a failing test
  Stack Trace:
     at Empty.UnitTest.Failing2() in C:\Dev\repro\empty\UnitTest.cs:line 16

Failed!  - Failed:     1, Passed:     0, Skipped:     0, Total:     1, Duration: < 1 ms - empty.dll (net6.0)

[tteguayco]

Thank you for the detailed answers. I'll have a look as soon as I get a chance.

[tteguayco]

Ok, I have created this dummy test repo. It has the same NuGet packages you @bradwilson depicted above.

Before adding the file xunit.runner.json to the root of the project, if I run dotnet test -- xUnit.StopOnFail=true, everything runs as expected: the execution of the tests stop when a failing test is encountered.

However, if then I add the xunit.runner.json file according to these instructions here, it doesn't matter if I run dotnet run or dotnet run -- xUnit.StopOnFail=true, all tests are run ignoring the property "stopOnFail": true in the JSON file.

Any ideas?

[tteguayco]

lol. It looks like setting parallelizeTestCollections: false makes the config stopOnFail to be ignored.

Is that the expected behavior?

[tteguayco]

Also, running the tests from VisualStudio 2022 (right click on project > Run Tests) seems to ignore the config set in xunit.runner.json.

[bradwilson]

It looks like setting parallelizeTestCollections: false makes the config stopOnFail to be ignored.

Is that the expected behavior?

Definitely not. That seems like the bug. 👍🏼

[bradwilson]

Have narrowed down that this appears to be a problem unique to xunit.runner.visualstudio, as the console runner w/ .NET Framework works as expected:

As well as with .NET:

[bradwilson]

This is a core framework bug, despite only being visible in the VSTest runner, because the bug is in the synchronous message bus (which gets enabled by VsTestRunner when parallel test collections are turned off):

visualstudio.xunit/src/xunit.runner.visualstudio/VsTestRunner.cs

Lines 535 to 539 in 8b76089

	if (!runInfo.Assembly.Configuration.ParallelizeTestCollectionsOrDefault)
	{
	executionOptions.SetSynchronousMessageReporting(true);
	executionOptions.SetDisableParallelization(true);
	}

And we can see that stop-on-fail was never added to the SynchronousMessageBus, as evidenced by the creation here:

https://github.com/xunit/xunit/blob/4e99b52d8e7a480e9ede44db8e25687ff26fc6b7/src/xunit.execution/Sdk/Frameworks/Runners/TestAssemblyRunner.cs#L135-L141

[bradwilson]

Because this is a core framework bug, getting the fix means updating your xunit reference, and not your xunit.runner.visualstudio reference.

Available in v2: 2.6.2-pre.10
Available in v3: 0.1.1-pre.321

[tteguayco]

Thank you for kind assistance. It looks like we need for now parallelizeTestCollections set to true, so hopefully this bug will not be affecting us for now.

Again, thank you @bradwilson