Change `.Verify` to fail when no issues are reported #9250

Tim-Pohlmann · 2024-05-03T14:10:42Z

Fixes #9237
Based on #9259

This branch has two commits:

Change Verifier to raise on errors: This is the functional change and directly related UTs. Review carefully!
Update UTs: This updates a lot of rule UTs by doing one of the following actions:
1. Change Verify to VerifyNoIssues. Sometimes with a comment, sometimes the reason is clear from context.
2. Change Verify to VerifyNoIssuesIgnoreErrors. Pay special attention to these, because they can, if used wrongly, hide problems.
3. Add a Noncompliant line to the test case.
4. In rare circumstances I had to fix the test case.
5. Everything that does not fall in one of these categories is CaYC (in addition to one of the points above).

Side note: Ignore the branch names. They are confusing, unfortunately.

CristianAmbrosini

Wow that was a lot, left some comments (some are nitpicks!)

CristianAmbrosini · 2024-05-09T07:35:22Z

analyzers/tests/SonarAnalyzer.TestFramework/Verification/DiagnosticVerifier.cs

@@ -27,21 +27,20 @@ public static class DiagnosticVerifier
    private const string AD0001 = nameof(AD0001);
    private const string LineContinuationVB12 = "BC36716";  // Visual Basic 12.0 does not support line continuation comments.

-    public static void Verify(
+    public static int Verify(


Please align the parameters consistently across the file, either placing Compilation compilation inlined with the method declaration or changing the other methods

CristianAmbrosini · 2024-05-09T07:37:02Z

analyzers/tests/SonarAnalyzer.TestFramework/Verification/DiagnosticVerifier.cs

-    public static void Verify(
-            Compilation compilation,
-            DiagnosticAnalyzer[] analyzers,
-            CompilationErrorBehavior checkMode, // ToDo: Remove this parameter in https://github.com/SonarSource/sonar-dotnet/issues/8588
-            string additionalFilePath = null,
-            string[] onlyDiagnostics = null,
-            string[] additionalSourceFiles = null)


I think I prefer the old parameter alignment tho (opinionated)

I agree, I was simply following our coding style.

analyzers/tests/SonarAnalyzer.TestFramework/Verification/Verifier.cs

CristianAmbrosini · 2024-05-09T08:48:49Z

...alyzer.Test/TestCases/SymbolicExecution/Roslyn/HashesShouldHaveUnpredictableSalt.CSharp10.cs

+        int a;
+        (var shortSalt, a) = (new byte[15], 42);


To make it actually require C# 10 (mixing assignment and declaration in deconstruction). However, since I removed the comment in the test file, this change does not need to be in this PR. I'll revert it.

CristianAmbrosini · 2024-05-09T08:49:01Z

...lyzer.Test/TestCases/SymbolicExecution/Roslyn/InitializationVectorShouldBeRandom.CSharp10.cs

+        int a;
+        (var rgb, a) = (new byte[16], 42);


Same as the other. Reverted.

I still see it 🤔

analyzers/tests/SonarAnalyzer.TestFramework.Test/Verification/VerifierTest.cs

CristianAmbrosini · 2024-05-09T08:57:16Z

analyzers/tests/SonarAnalyzer.Test/Rules/__Sandbox__.cs

What is this file about? if it's just adding more UTs for SE rules please cherry-pick the changes and move them into a different PR

Added by accident.

CristianAmbrosini · 2024-05-09T08:57:42Z

analyzers/SonarAnalyzer.lutconfig

Educational: what is this file about?

Live Unit Testing. I committed it by accident.

analyzers/tests/SonarAnalyzer.TestFramework/Verification/DiagnosticVerifier.cs

CristianAmbrosini

Nice! Only nitpicks left

CristianAmbrosini · 2024-05-13T13:21:35Z

analyzers/tests/SonarAnalyzer.Test/Rules/AvoidExcessiveClassCouplingTest.cs

+                {
+                    public void Foo<T>(Action action1, Action<T> action2, Action<T, T> action3, Action<T, T, T> action4, Action<T, T, T, T> action5) { }
+                }
+


CristianAmbrosini · 2024-05-13T13:21:44Z

analyzers/tests/SonarAnalyzer.Test/Rules/AvoidExcessiveClassCouplingTest.cs

+                {
+                    public void Foo<T>(Func<T> func1, Func<T, T> func2, Func<T, T, T> func3, Func<T, T, T, T> func4) { }
+                }
+


CristianAmbrosini · 2024-05-13T13:22:00Z

analyzers/tests/SonarAnalyzer.Test/Rules/AvoidExcessiveClassCouplingTest.cs

+                {
+                    public void Foo(int* pointer) { } // Error [CS0214]
+                }
+


CristianAmbrosini · 2024-05-13T13:22:10Z

analyzers/tests/SonarAnalyzer.Test/Rules/AvoidExcessiveClassCouplingTest.cs

+                {
+                    public ConsoleColor Foo(ConsoleColor c) { return ConsoleColor.Black; }
+                }
+


CristianAmbrosini · 2024-05-13T13:33:08Z

analyzers/tests/SonarAnalyzer.Test/Rules/CallerInformationParametersShouldBeLastTest.cs

@@ -44,7 +44,8 @@ public class CallerInformationParametersShouldBeLastTest

        [TestMethod]
        public void CallerInformationParametersShouldBeLast_CSharp11() =>
-            builder.AddPaths("CallerInformationParametersShouldBeLast.CSharp11.cs").WithOptions(ParseOptionsHelper.FromCSharp11).Verify();
+            builder.AddPaths("CallerInformationParametersShouldBeLast.CSharp11.cs").WithOptions(ParseOptionsHelper.FromCSharp11)
+                .VerifyNoIssues();   // overriding an abstract default implementation is compliant


This was not solved although it's marked with 👍

CristianAmbrosini · 2024-05-14T14:58:24Z

analyzers/tests/SonarAnalyzer.Test/TestCases/MemberInitializedToDefault.CSharp8.cs

@@ -10,7 +10,8 @@ public class C
        // If the field is not initialized and nullable is set to enabled then
        // there's a warning "CS8618: Non-nullable property 'MyProperty' must contain a non-null value when exiting constructor. Consider declaring the property as nullable."
        // To silence the warning one needs to use a field initializer and the suppression operator. We should not raise and issue in the case ! is present.
-        public object MyProperty { get; set; } = default!; // Compliant
+        public object Noncompliant { get; set; } = default; // FN


Is it documented?

Are you asking if there is a ticket for this? I did not create one. The rule is non-sonarway - I don't think it's worth tracking this.

CristianAmbrosini · 2024-05-14T14:58:52Z

analyzers/tests/SonarAnalyzer.Test/TestCases/MethodOverloadsShouldBeGrouped.CSharp12.cs

@@ -1,8 +1,10 @@
 class SomeClass(int arg)
 {
-    public void RandomMethod1() { }
+    public void RandomMethod1() { }         // Noncompliant


Why was this compliant before? 🤔

Because there was no other overload for RandomMethod1 present.

CristianAmbrosini · 2024-05-14T14:59:17Z

analyzers/tests/SonarAnalyzer.Test/TestCases/NativeMethodsShouldBeWrapped.CSharp9.cs

+interface Foo
+{
+    [DllImport("mynativelib")]
+    extern public static void Bar(string s, int x);     // FN


Documented somewhere? aka GH issue

I don't know. I did not write the test. I just merged it from the .CSharp10 file.

CristianAmbrosini · 2024-05-14T15:00:18Z

...lyzer.Test/TestCases/SymbolicExecution/Roslyn/InitializationVectorShouldBeRandom.CSharp10.cs

+        int a;
+        (var rgb, a) = (new byte[16], 42);


I still see it 🤔

CristianAmbrosini · 2024-05-14T15:00:33Z

...alyzer.Test/TestCases/SymbolicExecution/Sonar/InitializationVectorShouldBeRandom.CSharp10.cs

+        int a;
+        (var rgb, a) = (new byte[16], 42);


sonarcloud · 2024-05-14T15:41:33Z

Quality Gate passed for 'Sonar .NET Java Plugin'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud

sonarcloud · 2024-05-14T15:43:39Z

Quality Gate passed for 'SonarAnalyzer for .NET'

Issues
0 New issues
1 Accepted issue

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

Tim-Pohlmann force-pushed the Tim/VerifiyNoIssuesReported branch from f784e57 to a61b70a Compare May 8, 2024 06:59

Tim-Pohlmann changed the base branch from master to Tim/VerifyNoIssues May 8, 2024 07:00

Tim-Pohlmann marked this pull request as draft May 8, 2024 07:01

Tim-Pohlmann requested a review from CristianAmbrosini May 8, 2024 07:15

Tim-Pohlmann force-pushed the Tim/VerifyNoIssues branch from a3ecee4 to 6cf0904 Compare May 8, 2024 12:21

Base automatically changed from Tim/VerifyNoIssues to master May 8, 2024 12:37

Tim-Pohlmann force-pushed the Tim/VerifiyNoIssuesReported branch from a61b70a to fd4d546 Compare May 8, 2024 12:47

Tim-Pohlmann marked this pull request as ready for review May 8, 2024 12:51

CristianAmbrosini requested changes May 9, 2024

View reviewed changes

pavel-mikula-sonarsource reviewed May 10, 2024

View reviewed changes

analyzers/tests/SonarAnalyzer.TestFramework/Verification/DiagnosticVerifier.cs Show resolved Hide resolved

Tim-Pohlmann force-pushed the Tim/VerifiyNoIssuesReported branch from 0739ae1 to 994abcc Compare May 10, 2024 11:51

Tim-Pohlmann requested a review from CristianAmbrosini May 13, 2024 11:22

Tim-Pohlmann added 4 commits May 13, 2024 16:48

Change Verifier to raise on errors

a9e9585

Update UTs

17e03d6

Review 1

86551ee

Add tests for VerifyNoIssuesIgnoreErrors

d67893b

Tim-Pohlmann force-pushed the Tim/VerifiyNoIssuesReported branch 3 times, most recently from 16a8b88 to 8f4f5f6 Compare May 13, 2024 15:01

Review 1 Addendum

9f79a69

Tim-Pohlmann force-pushed the Tim/VerifiyNoIssuesReported branch from 8f4f5f6 to 9f79a69 Compare May 13, 2024 15:05

CristianAmbrosini approved these changes May 14, 2024

View reviewed changes

Review 2

fc6812d

Tim-Pohlmann enabled auto-merge (squash) May 14, 2024 15:30

Tim-Pohlmann merged commit 2081bf4 into master May 14, 2024
26 checks passed

Tim-Pohlmann deleted the Tim/VerifiyNoIssuesReported branch May 14, 2024 15:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Change `.Verify` to fail when no issues are reported #9250

Change `.Verify` to fail when no issues are reported #9250

Tim-Pohlmann commented May 3, 2024 •

edited

CristianAmbrosini left a comment

CristianAmbrosini May 9, 2024

CristianAmbrosini May 9, 2024

Tim-Pohlmann May 10, 2024

CristianAmbrosini May 9, 2024

Tim-Pohlmann May 10, 2024

CristianAmbrosini May 9, 2024

Tim-Pohlmann May 10, 2024

CristianAmbrosini May 14, 2024

CristianAmbrosini May 9, 2024

Tim-Pohlmann May 10, 2024

CristianAmbrosini May 9, 2024

Tim-Pohlmann May 10, 2024

CristianAmbrosini left a comment

CristianAmbrosini May 13, 2024

CristianAmbrosini May 13, 2024

CristianAmbrosini May 13, 2024

CristianAmbrosini May 13, 2024

CristianAmbrosini May 13, 2024

CristianAmbrosini May 14, 2024

Tim-Pohlmann May 14, 2024

CristianAmbrosini May 14, 2024

Tim-Pohlmann May 14, 2024

CristianAmbrosini May 14, 2024

Tim-Pohlmann May 14, 2024

CristianAmbrosini May 14, 2024

CristianAmbrosini May 14, 2024

sonarcloud bot commented May 14, 2024

sonarcloud bot commented May 14, 2024

Change .Verify to fail when no issues are reported #9250

Change .Verify to fail when no issues are reported #9250

Conversation

Tim-Pohlmann commented May 3, 2024 • edited

CristianAmbrosini left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CristianAmbrosini left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarcloud bot commented May 14, 2024

Quality Gate passed for 'Sonar .NET Java Plugin'

sonarcloud bot commented May 14, 2024

Quality Gate passed for 'SonarAnalyzer for .NET'

Change `.Verify` to fail when no issues are reported #9250

Change `.Verify` to fail when no issues are reported #9250

Tim-Pohlmann commented May 3, 2024 •

edited