[trainer] fix issues where number of failures would always be zero

[tahirmt]

Checklist

• I've run bundle exec rspec from the root directory to see all new and existing tests pass
• I've followed the fastlane code style and run bundle exec rubocop -a to ensure the code style is valid
• I see several green ci/circleci builds in the "All checks have passed" section of my PR (connect CircleCI to GitHub if not)
• I've read the Contribution Guidelines
• I've updated the documentation if necessary.

Motivation and Context

Closes #21565

Description

If the test case name has spaces or any special characters, the reported number_of_failures will always be 0. This will also not include any failed tests inside the junit report.

The reason this was happening was because during the comparison to check the test failure there was a sanitization step that's not applied to the identifier itself so it never matched.

This is from the log output of such a test.

** TEST FAILED **
[15:11:03]: Exit status: 65
+--------------------+---+
|      Test Results      |
+--------------------+---+
| Number of tests    | 3 |
| Number of failures | 0 |
+--------------------+---+

[15:11:04]: Test execution failed. Exit status: 65

Testing Steps

The easiest way to reproduce it was to use https://github.com/Quick/quick because Quick 7 produces test case names with spaces if the spec names had spaces in it. And when you run scan the output will always say the number of failures is 0.

I have attached a sample project.

1. Download project
2. Run bundle install
3. Run bundle exec fastlane test

When it runs the output will show number of failures as 0.

Then you can uncomment the branch line in the Gemfile and run bundle install and switch to this branch. Then you will see that the number of failure is 1.

SpaceTests.zip

[tahirmt]

The failure is unrelated to my changes. Likely existing issue.

[tahirmt]

@joshdholtz any chance you can take a look at this?

[rogerluan]

Thank you so much for providing a unit test, a fixture .xcresult and a sample project @tahirmt !

I reviewed your solution and it was great, but I thought of merging in the approach that @TheMetalCode applied in his PR that addresses this very same issue #21565 (but his PR was lacking unit tests 😅). In summary, I replaced the "String extension" that you created with a simple proc, and changed the implementation of the sanitizer from using several individual custom string replacements, with a single regex-replace that looks for all \W (i.e. non-word characters).

I think this is more fail-proof, and passes all the tests 😊

This PR LGTM already. I've made the changes above and solved some merge conflicts that were present. I'll give a few more days before merging it to see if there's anyone else interested in reviewing it too 🙇 maybe @lacostej ?

Thanks once again for this @tahirmt , this is a long-lasting issue! 🙏

[rogerluan]

PS: @tahirmt let me know if you have any thoughts on the changes I made 🙇

[tahirmt]

@rogerluan the solution looks good to me. Thanks for updating it and moving it forward.

[rogerluan]

No problem! Looking forward to merging this soon!

[lacostej]

LGTM

[tahirmt]

@rogerluan any idea when we can expect it to be merged?

[rogerluan]

Hi @tahirmt, I'm OOO till Sunday, but I'll take a look once I'm back :)

[rogerluan]

Thanks for another great contribution to fastlane @tahirmt 🤗 this is going in now! Thank you for your patience 🙇

[tahirmt]

@rogerluan sorry to bug you but do you have any idea when the next fastlane release would go out? It's been weeks since it was merged but no release yet.

[rogerluan]

Nope, sorry 😬 I don't have control over the releases yet 😅 I'll try to get a release out soon though 🙏

[jaysoffian]

This change actually broke parsing for at least some result files:

If I can't figure this out myself, I'll attach the sanitized output of xcrun xcresulttool get --format json for the xcresult bundle in question. I'm just noting this PR introduced a regression.

Result bundle was generated by Xcode 15.1.

Edit: the bug is in the find_failure method, specifically in the updated failures.find block.

• self.identifier : OPHUIManageAccountsTestCase/testManageAccountsSignIn
• sanitized_identifier : OPHUIManageAccountsTestCase_testManageAccountsSignIn
• failure.test_case_name : -[OPHUIManageAccountsTestCase testManageAccountsSignIn]
• sanitized_test_case_name: __OPHUIManageAccountsTestCase_testManageAccountsSignIn_

Since the sanitized_identifier does not match the sanitized_test_case_name, the failures.find block returns nil, so find_failure returns nil to the summaries_to_data function that calls it.

Now to figure out what this sanitation is all about, and how best to handle the case where we know a test failed but we aren't able to find it.

Edit 2: here's the data it's failing to parse correctly run through gron and eliding a path slightly:

$ cat ActionTestPlanRunSummaries.json | jq '.summaries._values[0].testableSummaries._values[0].tests._values[0].subtests._values[0].subtests._values[0].subtests._values[1]' | gron
json = {};
json._type = {};
json._type._name = "ActionTestMetadata";
json._type._supertype = {};
json._type._supertype._name = "ActionTestSummaryIdentifiableObject";
json._type._supertype._supertype = {};
json._type._supertype._supertype._name = "ActionAbstractTestSummary";
json.duration = {};
json.duration._type = {};
json.duration._type._name = "Double";
json.duration._value = "38.547163009643555";
json.identifier = {};
json.identifier._type = {};
json.identifier._type._name = "String";
json.identifier._value = "OPHUIManageAccountsTestCase/testManageAccountsSignIn";
json.identifierURL = {};
json.identifierURL._type = {};
json.identifierURL._type._name = "String";
json.identifierURL._value = "test://com.apple.xcode/<elided>/OPHUIManageAccountsTestCase/testManageAccountsSignIn";
json.name = {};
json.name._type = {};
json.name._type._name = "String";
json.name._value = "testManageAccountsSignIn";
json.summaryRef = {};
json.summaryRef._type = {};
json.summaryRef._type._name = "Reference";
json.summaryRef.id = {};
json.summaryRef.id._type = {};
json.summaryRef.id._type._name = "String";
json.summaryRef.id._value = "<elided>";
json.summaryRef.targetType = {};
json.summaryRef.targetType._type = {};
json.summaryRef.targetType._type._name = "TypeDefinition";
json.summaryRef.targetType.name = {};
json.summaryRef.targetType.name._type = {};
json.summaryRef.targetType.name._type._name = "String";
json.summaryRef.targetType.name._value = "ActionTestSummary";
json.summaryRef.targetType.supertype = {};
json.summaryRef.targetType.supertype._type = {};
json.summaryRef.targetType.supertype._type._name = "TypeDefinition";
json.summaryRef.targetType.supertype.name = {};
json.summaryRef.targetType.supertype.name._type = {};
json.summaryRef.targetType.supertype.name._type._name = "String";
json.summaryRef.targetType.supertype.name._value = "ActionTestSummaryIdentifiableObject";
json.summaryRef.targetType.supertype.supertype = {};
json.summaryRef.targetType.supertype.supertype._type = {};
json.summaryRef.targetType.supertype.supertype._type._name = "TypeDefinition";
json.summaryRef.targetType.supertype.supertype.name = {};
json.summaryRef.targetType.supertype.supertype.name._type = {};
json.summaryRef.targetType.supertype.supertype.name._type._name = "String";
json.summaryRef.targetType.supertype.supertype.name._value = "ActionAbstractTestSummary";
json.testStatus = {};
json.testStatus._type = {};
json.testStatus._type._name = "String";
json.testStatus._value = "Failure";

$ cat ActionsInvocationRecord.json | jq '.issues.testFailureSummaries._values[0]' | gron
json = {};
json._type = {};
json._type._name = "TestFailureIssueSummary";
json._type._supertype = {};
json._type._supertype._name = "IssueSummary";
json.documentLocationInCreatingWorkspace = {};
json.documentLocationInCreatingWorkspace._type = {};
json.documentLocationInCreatingWorkspace._type._name = "DocumentLocation";
json.documentLocationInCreatingWorkspace.concreteTypeName = {};
json.documentLocationInCreatingWorkspace.concreteTypeName._type = {};
json.documentLocationInCreatingWorkspace.concreteTypeName._type._name = "String";
json.documentLocationInCreatingWorkspace.concreteTypeName._value = "DVTTextDocumentLocation";
json.documentLocationInCreatingWorkspace.url = {};
json.documentLocationInCreatingWorkspace.url._type = {};
json.documentLocationInCreatingWorkspace.url._type._name = "String";
json.documentLocationInCreatingWorkspace.url._value = "file:///<elided>/Tests/Activities/OPHAuthenticationActivity.m#EndingLineNumber=18&StartingLineNumber=18";
json.issueType = {};
json.issueType._type = {};
json.issueType._type._name = "String";
json.issueType._value = "Uncategorized";
json.message = {};
json.message._type = {};
json.message._type._name = "String";
json.message._value = "(([[[OPHUITestHomePage alloc] init] userPresentWithUser: userIdp:) is true) failed";
json.testCaseName = {};
json.testCaseName._type = {};
json.testCaseName._type._name = "String";
json.testCaseName._value = "-[OPHUIManageAccountsTestCase testManageAccountsSignIn]";

Edit 3: okay, so the fundamental problem is that the two raw strings (from the parsed output of xcresulttool it's trying to match up are:

1. OPHUIManageAccountsTestCase/testManageAccountsSignIn (identifier._value from ActionTestPlanRunSummaries)
2. -[OPHUIManageAccountsTestCase testManageAccountsSignIn] (testCaseName._value from ActionsInvocationRecord)

When it "sanitizes" these values they become:

1. OPHUIManageAccountsTestCase_testManageAccountsSignIn
2. __OPHUIManageAccountsTestCase_testManageAccountsSignIn_

This whole endeavor seems fraught with peril. The old code worked (in this specific instance) because it stripped the extra characters from the test case name:

• Sanitize for Swift by replacing "." for "/"
• Sanitize for Objective-C by removing "-", "[", "]", and replacing " " for ?/

The new code no longer works because instead of stripping these characters entirely, it's replacing them with underscores which is not sufficient to match the normalization process that (apparently) xcode/xcresult uses.

I think this PR may have misunderstood what the original code was trying to do. It's not sanitizing the test case name so much as normalizing it in order to match the identifier value from ActionTestPlanRunSummaries. For it to work correctly it has to normalize the strings identically to how xcresulttool does so, not blindly sanitize each of them.

[jaysoffian]

This change is a regression. See #21432 (comment)

[jaysoffian]

Fix #21990