OuterLoop, and SkipOnPlatform attributes conflict, and don't work correctly when used together

[radical]

Consider
https://github.com/dotnet/runtime/blob/978df67ced885aeca5f7e75379451bc1a57cc219/src/libraries/System.Net.WebSockets/tests/WebSocketCreateTest.cs#L40-L45

        [OuterLoop("Uses external servers")]
        [Theory]
        [MemberData(nameof(EchoServers))]
        [SkipOnPlatform(TestPlatforms.Browser, "System.Net.Sockets is not supported on this platform.")]
        [ActiveIssue("https://github.com/dotnet/runtime/issues/34690", TestPlatforms.Windows, TargetFrameworkMonikers.Netcoreapp, TestRuntimes.Mono)]
        public async Task WebSocketProtocol_CreateFromConnectedStream_CanSendReceiveData(Uri echoUri)

This has both OuterLoop, and SkipOnPlatform attributes. This should run when using category=outerloop, but get skipped if it is running on the browser. Instead, the tests run anyway, and fail because the operations are not supported on browser.

I can reproduce this with ./dotnet.sh build /bl src/libraries/System.Net.WebSockets/tests/System.Net.WebSockets.Tests.csproj /p:Configuration=Release /p:TargetOS=Browser /p:TargetArchitecture=wasm /t:Test -p:Scenario=WasmTestOnBrowser -p:TestScope=outerloop -p:XUnitClassName=System.Net.WebSockets.Tests.WebSocketTests

this is the xharness command line:
wasm test-browser --app=. --output-directory=/workspaces/runtime/artifacts/bin/System.Net.WebSockets.Tests/Release/net7.0/browser-wasm/AppBundle/xharness-output --browser-arg=--no-sandbox -s dotnet.js.symbols --symbol-patterns wasm-symbol-patterns.txt --browser-arg=--no-sandbox -- --run WasmTestRunner.dll System.Net.WebSockets.Tests.dll -class System.Net.WebSockets.Tests.WebSocketTests -trait category=OuterLoop -notrait category=failing.

I don't think this is wasm, or SkipOnPlatform specific.

cc @akoeplinger @premun

[premun]

I think this is something that is actually implemented in runtime?
https://github.com/dotnet/runtime/blob/2f38fb55e965d685fff9ee07e0f9c50d3e6bafaf/docs/workflow/testing/libraries/filtering-tests.md

My understanding was that you provide the filters and also this file with ignores/excludes and XHarness only triggers the regular Xunit runner over that? But @akoeplinger might know more

[akoeplinger]

That doc is mostly relevant for the regular xunit console runner, but we implement most of that logic ourselves, e.g.

xharness/src/Microsoft.DotNet.XHarness.TestRunners.Xunit/WasmApplicationEntryPoint.cs

Lines 30 to 31 in 5a55841

	runner.SkipCategories(ExcludedTraits);
	runner.SkipCategories(IncludedTraits, isExcluded: false);

It's most likely some issue there.

[premun]

Closing this as this is not implemented in xharness but in runtime

[akoeplinger]

This is still an issue, reopening.