Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

NativeAOT legs timing out in CI #102239

Open
stephentoub opened this issue May 15, 2024 · 7 comments
Open

NativeAOT legs timing out in CI #102239

stephentoub opened this issue May 15, 2024 · 7 comments
Labels
area-NativeAOT-coreclr blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' Known Build Error Use this to report build issues in the .NET Helix tab os-linux Linux OS (any supported distro) untriaged New issue has not been triaged by the area owner

Comments

@stephentoub
Copy link
Member

stephentoub commented May 15, 2024 •
edited by build-analysis bot

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=675463
Build error leg or test failing: Build / linux-x64 Debug NativeAOT
Pull request: #102176

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "ran longer than the maximum time of 120 minutes",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=675463
Error message validated: [ran longer than the maximum time of 120 minutes]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 5/15/2024 3:00:04 AM UTC

Report

Build Definition Step Name Console log Pull Request
690305 dotnet/runtime linux-arm64 Release NativeAOT Log #102790
689901 dotnet/runtime linux-x64 Debug NativeAOT Log #102628
689859 dotnet/runtime linux-x64 Debug NativeAOT Log #102774
689831 dotnet/runtime linux-x64 Release NativeAOT Log #102315
689470 dotnet/runtime linux-x64 Release NativeAOT Log #102754
689821 dotnet/runtime linux-x64 Release NativeAOT Log #102771
689811 dotnet/runtime linux-x64 Debug NativeAOT Log #102739
2461554 dotnet-runtime Performance osx x64 release iOSNativeAOT JIT ios_scenarios perfiphone12mini NoJS False True True net9.0 Log
689748 dotnet/runtime linux-x64 Release NativeAOT Log #102769
689733 dotnet/runtime linux-x64 Release NativeAOT Log #99183
689721 dotnet/runtime linux-x64 Release NativeAOT Log #101232
689653 dotnet/runtime linux-x64 Debug NativeAOT Log #102720
689637 dotnet/runtime linux-x64 Release NativeAOT Log #102702
689627 dotnet/runtime linux-x64 Release NativeAOT Log #101796
689613 dotnet/runtime linux-x64 Debug NativeAOT Log #102760
689603 dotnet/runtime linux-x64 Debug NativeAOT Log #102758
689594 dotnet/runtime linux-x64 Debug NativeAOT Log #102756
689490 dotnet/runtime CoreCLR Product Build OSX arm64 checked Log #102558
689473 dotnet/runtime linux-x64 Debug NativeAOT Log #102755
689413 dotnet/runtime linux-x64 Release NativeAOT Log #102736
688826 dotnet/runtime CoreCLR Product Build OSX arm64 release Log #102727
688566 dotnet/runtime Build MacCatalyst x64 Debug AllSubsets_Mono #102558
688422 dotnet/runtime CoreCLR Product Build OSX arm64 release Log
687900 dotnet/runtime CoreCLR Product Build OSX arm64 release Log
686115 dotnet/runtime linux-arm64 Release NativeAOT Log
686107 dotnet/runtime linux-arm64 Release NativeAOT Log
686053 dotnet/runtime linux-arm64 Release NativeAOT Log #102628
686024 dotnet/runtime linux-arm64 Release NativeAOT Log #102592
684967 dotnet/runtime Build OSX x64 release Runtime_Release Log #102588
2458479 dotnet-runtime CoreCLR Product Build OSX x64 checked Log #39678
683553 dotnet/runtime linux-x64 Debug NativeAOT Log #102364
683439 dotnet/runtime windows-x64 Release NativeAOT Log #102528
683433 dotnet/runtime windows-x64 Release NativeAOT Log #102499
683428 dotnet/runtime windows-x64 Debug NativeAOT Log #102499
683420 dotnet/runtime windows-x64 Release NativeAOT Log #102409
683374 dotnet/runtime linux-x64 Debug NativeAOT Log #102298
683341 dotnet/runtime linux-x64 Release NativeAOT Log #102415
683331 dotnet/runtime linux-x64 Debug NativeAOT Log #85694
683262 dotnet/runtime linux-x64 Release NativeAOT Log
682899 dotnet/runtime linux-x64 Release NativeAOT Log #102297
682905 dotnet/runtime linux-x64 Release NativeAOT Log #102469
683209 dotnet/runtime linux-x64 Debug NativeAOT Log #102520
683204 dotnet/runtime linux-x64 Release NativeAOT Log #102519
682742 dotnet/runtime linux-x64 Release NativeAOT Log #102493
683154 dotnet/runtime linux-x64 Release NativeAOT Log #102180
683123 dotnet/runtime linux-x64 Release NativeAOT Log #102475
683105 dotnet/runtime linux-x64 Release NativeAOT Log #102508
683085 dotnet/runtime linux-x64 Release NativeAOT Log #102424
683055 dotnet/runtime linux-x64 Release NativeAOT Log #102512
683043 dotnet/runtime linux-x64 Release NativeAOT Log #102498
2457311 dotnet-runtime Performance osx x64 release iOSNativeAOT JIT ios_scenarios perfiphone12mini NoJS False False True net9.0 Log
682950 dotnet/runtime linux-x64 Release NativeAOT Log #101295
682938 dotnet/runtime linux-x64 Release NativeAOT Log #102248
682872 dotnet/runtime linux-x64 Debug NativeAOT Log #102262
682680 dotnet/runtime windows-x64 Release NativeAOT Log #102490
682671 dotnet/runtime windows-x64 Release NativeAOT Log #101801
682649 dotnet/runtime linux-x64 Release NativeAOT Log #102489
682643 dotnet/runtime linux-x64 Debug NativeAOT Log #102488
682617 dotnet/runtime linux-x64 Release NativeAOT Log #102487
682593 dotnet/runtime linux-x64 Debug NativeAOT Log #99871
682019 dotnet/runtime CoreCLR Product Build OSX arm64 checked Log
681993 dotnet/runtime Build OSX x64 Release AllSubsets_Mono_LLVMJIT #102096
682003 dotnet/runtime CoreCLR Product Build OSX arm64 checked Log
2456451 dotnet-runtime Performance osx x64 release iOSNativeAOT JIT ios_scenarios perfiphone12mini NoJS False True True net9.0 Log
681898 dotnet/runtime Build windows x64 Release AllSubsets_Mono Log #102466
681897 dotnet/runtime Build OSX x64 release Runtime_Release Log #102402
681832 dotnet/runtime Mono Product Build OSX arm64 debug Log
681111 dotnet/runtime CoreCLR Product Build OSX x64 checked Log #102444
681093 dotnet/runtime CoreCLR Product Build OSX arm64 checked Log #102443
681073 dotnet/runtime osx-arm64 Release NativeAOT Log #102366
680869 dotnet/runtime osx-arm64 Release NativeAOT Log #102434
680860 dotnet/runtime osx-arm64 Release NativeAOT Log #102433
680841 dotnet/runtime osx-arm64 Release NativeAOT Log #102432
2455420 dotnet-runtime windows-x86 release CoreCLR
680010 dotnet/runtime CoreCLR Product Build OSX arm64 debug Log
679808 dotnet/runtime CoreCLR Product Build OSX arm64 release Log #102401
2454354 dotnet-runtime Performance osx x64 release iOSNativeAOT JIT ios_scenarios perfiphone12mini NoJS False False True net9.0 Log
2454199 dotnet-runtime CoreCLR Product Build OSX x64 release Log
679595 dotnet/runtime CoreCLR Product Build OSX arm64 checked Log
2454004 dotnet-runtime Build tvOS arm64 release AllSubsets_Mono Log
2453876 dotnet-runtime CoreCLR Product Build OSX x64 release Log
678386 dotnet/runtime windows-x64 Release AllSubsets_Mono Log
678374 dotnet/runtime Build windows x64 Release AllSubsets_Mono Log #102346
2452732 dotnet-runtime CoreCLR Product Build OSX arm64 checked Log #39614
2452639 dotnet-runtime windows-x86 release CoreCLR
2452397 dotnet-runtime Performance osx x64 release iOSMono JIT ios_scenarios perfiphone12mini NoJS False False True net9.0 Log
677199 dotnet/runtime windows-arm64 Release NativeAOT Log #102271
677144 dotnet/runtime windows-x64 Release NativeAOT Log #102277
677133 dotnet/runtime windows-x64 Debug NativeAOT Log #102286
677127 dotnet/runtime windows-x64 Debug NativeAOT Log #102291
676829 dotnet/runtime linux-x64 Debug NativeAOT Log
2452156 dotnet-runtime Installer Build and Test coreclr windows_x86 Release Log
676802 dotnet/runtime linux-x64 Release NativeAOT Log #102280
676765 dotnet/runtime linux-x64 Release NativeAOT Log #102218
676743 dotnet/runtime linux-x64 Release NativeAOT Log #101001
676726 dotnet/runtime linux-x64 Release NativeAOT Log #102278
676686 dotnet/runtime linux-x64 Debug NativeAOT Log #102248
676635 dotnet/runtime windows-x64 Release NativeAOT Log #102237
676620 dotnet/runtime windows-x64 Release NativeAOT Log #102143
676613 dotnet/runtime windows-x64 Release NativeAOT Log #102270
Displaying 100 of 144 results

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
20 30 144
@stephentoub stephentoub added blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' Known Build Error Use this to report build issues in the .NET Helix tab labels May 15, 2024
@dotnet-policy-service dotnet-policy-service bot added the untriaged New issue has not been triaged by the area owner label May 15, 2024
@dotnet-policy-service Dotnet Policy Service
Copy link
Contributor

Tagging subscribers to this area: @dotnet/area-infrastructure-libraries
See info in area-owners.md if you want to be subscribed.

@MichalStrehovsky
Copy link
Member

Clicking through, the problem is always the same - we finish product build in 20 minutes and send 5 workitems to helix (each of which takes less than a minute to run). We then wait for 100 minutes for these to finish. Then we timeout. Then 2 more hours later, the Helix workitems get finally scheduled and finish.

Digging into Helix logs, it always looks something like this:

    "Delay": "04:17:08.6820000",
    "Duration": "00:00:17.8290000",

We could increase the timeout to 5 hours but that feels excessive.

@kunalspathak kunalspathak mentioned this issue May 15, 2024
Merged
@MichalStrehovsky MichalStrehovsky mentioned this issue May 16, 2024
Merged
@agocke
Copy link
Member

agocke commented May 16, 2024

@markwilkie Could you comment on what "Delay" means here? Is there something holding up the run?

@markwilkie markwilkie assigned markwilkie and unassigned markwilkie May 16, 2024
@markwilkie
Copy link
Member

markwilkie commented May 16, 2024 •
edited

@chcosta - any thoughts as to what 'Delay' means here?

"Delay": "04:17:08.6820000",
"Duration": "00:00:17.8290000",

@chcosta
Copy link
Member

chcosta commented May 16, 2024

looking

@chcosta
Copy link
Member

chcosta commented May 17, 2024

I haven't had much time to dig into this yet

@dotnet-policy-service Dotnet Policy Service
Copy link
Contributor

Tagging subscribers to this area: @agocke, @MichalStrehovsky, @jkotas
See info in area-owners.md if you want to be subscribed.

@ericstj ericstj added the os-linux Linux OS (any supported distro) label May 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area-NativeAOT-coreclr blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' Known Build Error Use this to report build issues in the .NET Helix tab os-linux Linux OS (any supported distro) untriaged New issue has not been triaged by the area owner
Projects
AppModel
Status: No status
Milestone
No milestone
Development

No branches or pull requests
6 participants
@agocke @stephentoub @markwilkie @ericstj @chcosta @MichalStrehovsky