Unit tests ignore xunit.runner.json - Explains why the new stopOnFail setting wasn't working

[tig]

See xunit/visualstudio.xunit#189

Fix is to set xunit.runner.json to Copy On Newer:

(Adding this to UnitTests.csproj:

  <ItemGroup>
    <Content Include="xunit.runner.json" CopyToOutputDirectory="PreserveNewest" />
  </ItemGroup>

We can remove this line from ./UnitTests/AssemblyInfo.cs (it WAS what was preventing tests from running in parallel, but is no longer needed if xunit.runner.json is working):

// Since Application is a singleton we can't run tests in parallel
[assembly: CollectionBehavior (DisableTestParallelization = true)]

Once this is done, stopOnFail can be enabled by adding "stopOnFail": true to xunit.runner.json.

I'd like to know if anyone has an objection to setting stopOnFail to true. I'd DEF want this for CI/CD, but am not sure if I like it for dev work.

[BDisp]

There is situations which is preferable the unit tests stooped instead of some never end.

[tig]

There is situations which is preferable the unit tests stooped instead of some never end.

Sorry @BDisp but I can't parse what you're saying? Can you try again?

[BDisp]

Sorry @BDisp but I can't parse what you're saying? Can you try again?

I was referring that some unit tests fail silent and only opening the Tests on the drop down of the Output window we can verify there is an exception. But never mind.

[tznind]

I'd like to know if anyone has an objection to setting stopOnFail to true. I'd DEF want this for CI/CD, but am not sure if I like it for dev work.

I think for local dev it would be nice to see all the failing tests rather than break on first (if that's what stopOnFail does).

If you want it it in CI/CD though you could declare it as false in the json and add a CI build step that replaces false with true:

sed -i 's/"stopOnFail": false/"stopOnFail": true/g' ./UnitTests/xunit.runner.json

[BDisp]

Well I also agree with @tznind. If the intention is stop the tests when one of them fails, then we stop all tests. I know running the tests in a command line the failed tests is only show when they finished, but we can do Ctrl+C to stop all. For CI/CD the @tznind suggestion is a good choice.

[tig]

Cool. That's how we'll address this then.

[tig]

I've also fixed this in #2612 so it will be picked up for v2 when I merge that.