Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[No Jira] Improve Autoscan testing #4520

Merged
merged 6 commits into from
Nov 9, 2023

Conversation

johann-beleites-sonarsource
Copy link
Contributor

Please ensure your pull request adheres to the following guidelines:

  • Use the following formatting style: SonarSource/sonar-developer-toolset
  • Unit tests are passing and you provided a unit test for your fix
  • ITs should pass : To run ITs locally, checkout the README of the project.
  • If there is a Jira ticket available, please make your commits and pull request start with the ticket number (SONARJAVA-XXXX)

Sorry, something went wrong.

knownDiffs.put(diff.ruleKey, diff);
}

// store new unexpected diffs in JSON files - serializable
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For easier debugging, we only output diffs that are different from the expected. There is no value in outputting diffs that are expected that way.

String differences = Files.readString(pathFor(TARGET_ACTUAL + PROJECT_KEY + "-no-binaries_differences"));
softly.assertThat(differences).isEqualTo("Issues differences: 3473");
softly.assertThat(differences).isEqualTo("Issues differences: " + expectedDiffs);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm fine with removing this assertion entirely (I don't see much value here that goes beyond the comparison of the actual and new diffs collections), however the modification is okay as well, as it doesn't need to be edited when diffs change.

softly.assertThat(newTotal).isEqualTo(knownTotal);
softly.assertThat(rulesCausingFPs).hasSize(6);
softly.assertThat(rulesNotReporting).hasSize(7);
softly.assertThat(rulesSilenced).hasSize(74);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The vast majority of rules relying on semantics will be silenced by the absence of binaries. Nowadays, it seems to me that most of our rules that we write rely on semantics. I don't think having this check is very valuable in addition to the diff data, that already tells us if a rule is silenced.

Comment on lines 200 to 199
softly.assertThat(rulesCausingFPs).hasSize(6);
softly.assertThat(rulesNotReporting).hasSize(7);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's okay to leave these two assertions because this should actually be unexpected in most cases and require some additional verification. It's changed very rarely.

Copy link
Contributor

@leveretka leveretka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Thanks for the effort.
Let's merge this PR after the release

Copy link

sonarqube-next bot commented Nov 9, 2023

Quality Gate passed Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No Coverage information No data about Coverage
0.0% 0.0% Duplication on New Code

See analysis details on SonarQube

@johann-beleites-sonarsource johann-beleites-sonarsource merged commit 9b5bf3f into master Nov 9, 2023
@johann-beleites-sonarsource johann-beleites-sonarsource deleted the jb/autoscan-test branch November 9, 2023 10:15
rguihard pushed a commit to rguihard/sonar-java that referenced this pull request Jan 15, 2024
* Split autoscan test results into files per rule, to avoid merge conflicts

* Remove / optimize redundant assertions
rguihard pushed a commit to rguihard/sonar-java that referenced this pull request Jan 15, 2024
* Split autoscan test results into files per rule, to avoid merge conflicts

* Remove / optimize redundant assertions
rguihard pushed a commit to rguihard/sonar-java that referenced this pull request Jan 15, 2024
* Split autoscan test results into files per rule, to avoid merge conflicts

* Remove / optimize redundant assertions
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants