System.Numerics.Tensors.Net8.Tests does not appear to actually run against .NET 8 #101846

tannergooding · 2024-05-03T14:20:25Z

As per the title, the System.Numerics.Tensors.Net8.Tests do not appear to currently be running against .NET 8 as intended. Instead, they are compiled for .NET 8 and appear to be running on the .NET 9 runtime via the roll forward mechanism.

This can be seen in one of the CI legs for #101800 where the Net8.Tests project has the following failure:

System.Numerics.Tensors.Tests.SingleGenericTensorPrimitives.SpanDestinationFunctions_SpecialValues(tensorPrimitivesMethod: SpanDestinationDelegate { Method = Void ReciprocalSqrtEstimate[Single](System.ReadOnlySpan`1[System.Single], System.Span`1[System.Single]), Target = null }, expectedMethod: Func`2 { Method = Single ReciprocalSqrtEstimate(Single), Target = null }, tolerance: 0.01171875) [FAIL]
      Assert.All() Failure: 253 out of 256 items in the collection did not pass.
      [3]:   Item:  4
             Error: Assert.All() Failure: 9 out of 24 items in the collection did not pass.
                    [12]: Item:  -1.40129846E-45
                          Error: Assert.Equal() Failure: Values differ
                                 Expected: NaN
                                 Actual:   -∞

The actual test is comparing the result of the TensorPrimitives.ReciprocalSqrtEstimate function against the in-box float.ReciprocalSqrtEstimate API. On .NET 8, the float version will call into Sse.ReciprocalSqrtScalar which treats subnormal values as flush-to-zero and therefore does 1 / Sqrt(-0.0f) which returns -∞, however the expected value is NaN indicated that the float version isntead called into Avx512F.Reciprocal14SqrtScalar which handles all non-zero negatives as properly negative and thus is doing 1 / Sqrt(nonZeroNegative) which returns NaN. This highlights that it is actually running against .NET 9 and not testing what we think it should be testing.

The text was updated successfully, but these errors were encountered:

dotnet-policy-service · 2024-05-03T14:20:48Z

Tagging subscribers to this area: @dotnet/area-infrastructure-libraries
See info in area-owners.md if you want to be subscribed.

ViktorHofer · 2024-05-03T14:36:48Z

As per the title, the System.Numerics.Tensors.Net8.Tests do not appear to currently be running against .NET 8 as intended. Instead, they are compiled for .NET 8 and appear to be running on the .NET 9 runtime via the roll forward mechanism.

Correct. None of the libraries tests in dotnet/runtime run against a previous version of .NETCoreApp: #54639

We only test the NetCoreAppCurrent and NetFrameworkCurrent/NetFrameworkMinimum TFMs via tests. That's a known gap that we never had resources to close.

stephentoub · 2024-05-03T14:55:05Z

As per the title, the System.Numerics.Tensors.Net8.Tests do not appear to currently be running against .NET 8 as intended. Instead, they are compiled for .NET 8 and appear to be running on the .NET 9 runtime via the roll forward mechanism.

Correct. None of the libraries tests in dotnet/runtime run against a previous version of .NETCoreApp: #54639

We only test the NetCoreAppCurrent and NetFrameworkCurrent/NetFrameworkMinimum TFMs via tests. That's a known gap that we never had resources to close.

Ok, so this project is using the .NET 8 build of SNT (and thus exercising .NET 8 code paths in the assembly) but running on .NET 9?

tannergooding · 2024-05-03T15:01:13Z

That's what it looks like to me.

Hence the issue is that the System.Numerics.Tensors v8.0 tests expects ReciprocalSqrtEstimate to behave one way (doesn't use AVX512) but System.Private.Corelib v9 does it a different way (does use AVX512). Leading the tests to fail since they can't compare the TensorPrimitives results against the scalar (float) results.

The Estimate APIs are intentionally allowed to differ cross processes, and so changing behavior over time is expected. It's just this oddity in how our tests are written that causes this to be a problem for us since we can't guarantee the behavior is synchronized for our own out of band library.

v-wenyuxu · 2024-05-15T01:59:11Z

Failed in: runtime-coreclr libraries-jitstress 20240514.1

Failed tests:

net9.0-linux-Release-x64-jitstress2_tiered-Ubuntu.2204.Amd64.Open
    - System.Numerics.Tensors.Tests.NonGenericSingleTensorPrimitivesTests.CosineSimilarity_AllLengths
    - System.Numerics.Tensors.Tests.SingleGenericTensorPrimitives.CosineSimilarity_AllLengths
net9.0-linux-Release-x64-jitstress1_tiered-Ubuntu.2204.Amd64.Open
    - System.Numerics.Tensors.Tests.NonGenericSingleTensorPrimitivesTests.CosineSimilarity_AllLengths
    - System.Numerics.Tensors.Tests.NonGenericSingleTensorPrimitivesTests.CosineSimilarity_AllLengths
net9.0-windows-Release-x64-jitstress1_tiered-Windows.10.Amd64.Open
    - System.Numerics.Tensors.Tests.NonGenericSingleTensorPrimitivesTests.CosineSimilarity_AllLengths
    - System.Numerics.Tensors.Tests.NonGenericSingleTensorPrimitivesTests.CosineSimilarity_AllLengths
    - System.Numerics.Tensors.Tests.SingleGenericTensorPrimitives.CosineSimilarity_AllLengths

Error message:

 Assert.All() Failure: 4 out of 256 items in the collection did not pass.
[3]: Item:  4
     Error: Assert.Equal() Failure: Values differ
            Expected: 0.151808769
            Actual:   0.25312683
[4]: Item:  5
     Error: Assert.Equal() Failure: Values differ
            Expected: 0.253379166
            Actual:   0.138663024
[5]: Item:  6
     Error: Assert.Equal() Failure: Values differ
            Expected: -0.344092488
            Actual:   -0.0114496415
[6]: Item:  7
     Error: Assert.Equal() Failure: Values differ
            Expected: -0.410494238
            Actual:   -0.248638168

Stack trace:

   at System.Numerics.Tensors.Tests.TensorPrimitivesTests`1.CosineSimilarity_AllLengths() in /_/src/libraries/System.Numerics.Tensors/tests/TensorPrimitivesTests.cs:line 740
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr) in /_/src/libraries/System.Private.CoreLib/src/System/Reflection/MethodBaseInvoker.cs:line 57

stephentoub · 2024-05-16T01:26:39Z

@v-wenyuxu, that last failure doesn't appear to be related to this discussion. And while the one prior to that is related, it's not the same thing. Can you please open new issues for each if there's nothing else relevant open now? Thanks.

v-wenyuxu · 2024-05-17T02:22:51Z

Failed in: runtime-coreclr libraries-jitstress 20240516.4

Failed tests:

net9.0-linux-Release-x64-jitstress2_tiered-Ubuntu.2204.Amd64.Open
    - System.Numerics.Tensors.Tests.SingleGenericTensorPrimitives.CosineSimilarity_AllLengths
net9.0-linux-Release-x64-jitstress1_tiered-Ubuntu.2204.Amd64.Open
    - System.Numerics.Tensors.Tests.NonGenericSingleTensorPrimitivesTests.CosineSimilarity_AllLengths
net9.0-windows-Release-x64-jitstress1_tiered-Windows.10.Amd64.Open
    - System.Numerics.Tensors.Tests.SingleGenericTensorPrimitives.CosineSimilarity_AllLengths
    - System.Numerics.Tensors.Tests.NonGenericSingleTensorPrimitivesTests.CosineSimilarity_AllLengths
    - System.Numerics.Tensors.Tests.SingleGenericTensorPrimitives.CosineSimilarity_AllLengths
    - System.Numerics.Tensors.Tests.NonGenericSingleTensorPrimitivesTests.CosineSimilarity_AllLengths

Error message:

 Assert.All() Failure: 4 out of 256 items in the collection did not pass.
[3]: Item:  4
     Error: Assert.Equal() Failure: Values differ
            Expected: 0.151808769
            Actual:   0.25312683
[4]: Item:  5
     Error: Assert.Equal() Failure: Values differ
            Expected: 0.253379166
            Actual:   0.138663024
[5]: Item:  6
     Error: Assert.Equal() Failure: Values differ
            Expected: -0.344092488
            Actual:   -0.0114496415
[6]: Item:  7
     Error: Assert.Equal() Failure: Values differ
            Expected: -0.410494238
            Actual:   -0.248638168

Stack trace:

   at System.Numerics.Tensors.Tests.TensorPrimitivesTests`1.CosineSimilarity_AllLengths() in /_/src/libraries/System.Numerics.Tensors/tests/TensorPrimitivesTests.cs:line 740
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr) in /_/src/libraries/System.Private.CoreLib/src/System/Reflection/MethodBaseInvoker.cs:line 57

v-wenyuxu · 2024-05-20T01:38:20Z

Failed in: runtime-coreclr libraries-jitstress 20240519.1

Failed tests:

net9.0-linux-Release-x64-jitstress2_tiered-Ubuntu.2204.Amd64.Open
    - System.Numerics.Tensors.Tests.NonGenericSingleTensorPrimitivesTests.CosineSimilarity_AllLengths
    - System.Numerics.Tensors.Tests.NonGenericSingleTensorPrimitivesTests.CosineSimilarity_AllLengths
net9.0-linux-Release-x64-jitstress1_tiered-Ubuntu.2204.Amd64.Open
    - System.Numerics.Tensors.Tests.NonGenericSingleTensorPrimitivesTests.CosineSimilarity_AllLengths
    - System.Numerics.Tensors.Tests.NonGenericSingleTensorPrimitivesTests.CosineSimilarity_AllLengths
net9.0-windows-Release-x64-jitstress1_tiered-Windows.10.Amd64.Open
    - System.Numerics.Tensors.Tests.NonGenericSingleTensorPrimitivesTests.CosineSimilarity_AllLengths
    - System.Numerics.Tensors.Tests.NonGenericSingleTensorPrimitivesTests.CosineSimilarity_AllLengths
    - System.Numerics.Tensors.Tests.SingleGenericTensorPrimitives.CosineSimilarity_AllLengths

Error message:

 Assert.All() Failure: 4 out of 256 items in the collection did not pass.
[3]: Item:  4
     Error: Assert.Equal() Failure: Values differ
            Expected: 0.151808769
            Actual:   0.25312683
[4]: Item:  5
     Error: Assert.Equal() Failure: Values differ
            Expected: 0.253379166
            Actual:   0.138663024
[5]: Item:  6
     Error: Assert.Equal() Failure: Values differ
            Expected: -0.344092488
            Actual:   -0.0114496415
[6]: Item:  7
     Error: Assert.Equal() Failure: Values differ
            Expected: -0.410494238
            Actual:   -0.248638168

Stack trace:

   at System.Numerics.Tensors.Tests.TensorPrimitivesTests`1.CosineSimilarity_AllLengths() in /_/src/libraries/System.Numerics.Tensors/tests/TensorPrimitivesTests.cs:line 740
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr) in /_/src/libraries/System.Private.CoreLib/src/System/Reflection/MethodBaseInvoker.cs:line 57

v-wenyuxu · 2024-05-21T01:16:49Z

Failed in: runtime-coreclr libraries-jitstress 20240520.1

Failed tests:

net9.0-linux-Release-x64-jitstress2_tiered-Ubuntu.2204.Amd64.Open
    - System.Numerics.Tensors.Tests.SingleGenericTensorPrimitives.CosineSimilarity_AllLengths
    - System.Numerics.Tensors.Tests.SingleGenericTensorPrimitives.CosineSimilarity_AllLengths
net9.0-linux-Release-x64-jitstress1_tiered-Ubuntu.2204.Amd64.Open
    - System.Numerics.Tensors.Tests.NonGenericSingleTensorPrimitivesTests.CosineSimilarity_AllLengths
    - System.Numerics.Tensors.Tests.SingleGenericTensorPrimitives.CosineSimilarity_AllLengths
net9.0-windows-Release-x64-jitstress1_tiered-Windows.10.Amd64.Open
    - System.Numerics.Tensors.Tests.NonGenericSingleTensorPrimitivesTests.CosineSimilarity_AllLengths
    - System.Numerics.Tensors.Tests.NonGenericSingleTensorPrimitivesTests.CosineSimilarity_AllLengths
    - System.Numerics.Tensors.Tests.SingleGenericTensorPrimitives.CosineSimilarity_AllLengths

Error message:

 Assert.All() Failure: 4 out of 256 items in the collection did not pass.
[3]: Item:  4
     Error: Assert.Equal() Failure: Values differ
            Expected: 0.151808769
            Actual:   0.25312683
[4]: Item:  5
     Error: Assert.Equal() Failure: Values differ
            Expected: 0.253379166
            Actual:   0.138663024
[5]: Item:  6
     Error: Assert.Equal() Failure: Values differ
            Expected: -0.344092488
            Actual:   -0.0114496415
[6]: Item:  7
     Error: Assert.Equal() Failure: Values differ
            Expected: -0.410494238
            Actual:   -0.248638168

Stack trace:

   at System.Numerics.Tensors.Tests.TensorPrimitivesTests`1.CosineSimilarity_AllLengths() in /_/src/libraries/System.Numerics.Tensors/tests/TensorPrimitivesTests.cs:line 740
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr) in /_/src/libraries/System.Private.CoreLib/src/System/Reflection/MethodBaseInvoker.cs:line 57

dotnet-policy-service · 2024-05-21T23:16:42Z

Tagging subscribers to this area: @dotnet/area-system-numerics-tensors
See info in area-owners.md if you want to be subscribed.

v-wenyuxu · 2024-05-22T01:20:47Z

Failed in: runtime-coreclr libraries-jitstress 20240521.1

Failed tests:

net9.0-linux-Release-x64-jitstress2_tiered-Ubuntu.2204.Amd64.Open
    - System.Numerics.Tensors.Tests.SingleGenericTensorPrimitives.CosineSimilarity_AllLengths
    - System.Numerics.Tensors.Tests.NonGenericSingleTensorPrimitivesTests.CosineSimilarity_AllLengths
net9.0-linux-Release-x64-jitstress1_tiered-Ubuntu.2204.Amd64.Open
    - System.Numerics.Tensors.Tests.NonGenericSingleTensorPrimitivesTests.CosineSimilarity_AllLengths
    - System.Numerics.Tensors.Tests.NonGenericSingleTensorPrimitivesTests.CosineSimilarity_AllLengths
net9.0-windows-Release-x64-jitstress1_tiered-Windows.10.Amd64.Open
    - System.Numerics.Tensors.Tests.NonGenericSingleTensorPrimitivesTests.CosineSimilarity_AllLengths
    - System.Numerics.Tensors.Tests.NonGenericSingleTensorPrimitivesTests.CosineSimilarity_AllLengths
    - System.Numerics.Tensors.Tests.SingleGenericTensorPrimitives.CosineSimilarity_AllLengths

Error message:

 Assert.All() Failure: 4 out of 256 items in the collection did not pass.
[3]: Item:  4
     Error: Assert.Equal() Failure: Values differ
            Expected: 0.151808769
            Actual:   0.25312683
[4]: Item:  5
     Error: Assert.Equal() Failure: Values differ
            Expected: 0.253379166
            Actual:   0.138663024
[5]: Item:  6
     Error: Assert.Equal() Failure: Values differ
            Expected: -0.344092488
            Actual:   -0.0114496415
[6]: Item:  7
     Error: Assert.Equal() Failure: Values differ
            Expected: -0.410494238
            Actual:   -0.248638168

Stack trace:

   at System.Numerics.Tensors.Tests.TensorPrimitivesTests`1.CosineSimilarity_AllLengths() in /_/src/libraries/System.Numerics.Tensors/tests/TensorPrimitivesTests.cs:line 740
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr) in /_/src/libraries/System.Private.CoreLib/src/System/Reflection/MethodBaseInvoker.cs:line 57

tannergooding added the area-Infrastructure-libraries label May 3, 2024

dotnet-policy-service bot added the untriaged New issue has not been triaged by the area owner label May 3, 2024

ericstj added area-System.Numerics.Tensors and removed area-Infrastructure-libraries labels May 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

System.Numerics.Tensors.Net8.Tests does not appear to actually run against .NET 8 #101846

System.Numerics.Tensors.Net8.Tests does not appear to actually run against .NET 8 #101846

tannergooding commented May 3, 2024

dotnet-policy-service bot commented May 3, 2024

ViktorHofer commented May 3, 2024

stephentoub commented May 3, 2024

tannergooding commented May 3, 2024 •

edited

v-wenyuxu commented May 15, 2024

stephentoub commented May 16, 2024

v-wenyuxu commented May 17, 2024

v-wenyuxu commented May 20, 2024

v-wenyuxu commented May 21, 2024

dotnet-policy-service bot commented May 21, 2024

v-wenyuxu commented May 22, 2024

System.Numerics.Tensors.Net8.Tests does not appear to actually run against .NET 8 #101846

System.Numerics.Tensors.Net8.Tests does not appear to actually run against .NET 8 #101846

Comments

tannergooding commented May 3, 2024

dotnet-policy-service bot commented May 3, 2024

ViktorHofer commented May 3, 2024

stephentoub commented May 3, 2024

tannergooding commented May 3, 2024 • edited

v-wenyuxu commented May 15, 2024

stephentoub commented May 16, 2024

v-wenyuxu commented May 17, 2024

v-wenyuxu commented May 20, 2024

v-wenyuxu commented May 21, 2024

dotnet-policy-service bot commented May 21, 2024

v-wenyuxu commented May 22, 2024

tannergooding commented May 3, 2024 •

edited