Build fails with "eng/common/tools.sh: line 474: 537 Segmentation fault"

[jkotas]

2022-10-07T15:14:51.2484448Z Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli
2022-10-07T15:14:51.2485788Z --------------------------------------------------------------------------------------
2022-10-07T15:14:51.6083403Z /__w/1/s/.dotnet/sdk/7.0.100-rc.1.22431.12/MSBuild.dll /nologo -maxcpucount /m -verbosity:m /v:minimal /bl:/__w/1/s/artifacts/log/Release/ToolsetRestore.binlog /clp:Summary /clp:ErrorsOnly;NoSummary /nr:false /p:TreatWarningsAsErrors=true /p:ContinuousIntegrationBuild=true /p:__ToolsetLocationOutputFile=/__w/1/s/artifacts/toolset/8.0.0-beta.22503.1.txt /t:__WriteToolsetLocation /warnaserror /__w/1/s/artifacts/toolset/restore.proj
2022-10-07T15:14:55.6156152Z /__w/1/s/eng/common/tools.sh: line 474:   537 Segmentation fault      (core dumped) "$_InitializeBuildTool" "$@"
2022-10-07T15:14:55.6156978Z Build failed with exit code 139. Check errors above.

Failed in #76737. Full log: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/44249/logs/166

{
  "ErrorPattern": "eng/common/tools.sh: line [0-9]+:   [0-9]+ Segmentation fault",
  "BuildRetry": true
}

Report

Build	Definition	Step Name	Console log	Pull Request
2454004	dotnet-runtime	Restore internal tools	Log

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	1

Known issue validation

Build: 🔎
Result validation: ⚠️ Validation could not be done without an Azure DevOps build URL on the issue. Please add it to the "Build: 🔎" line.

[ghost]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

---

2022-10-07T15:14:51.2484448Z Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli
2022-10-07T15:14:51.2485788Z --------------------------------------------------------------------------------------
2022-10-07T15:14:51.6083403Z /__w/1/s/.dotnet/sdk/7.0.100-rc.1.22431.12/MSBuild.dll /nologo -maxcpucount /m -verbosity:m /v:minimal /bl:/__w/1/s/artifacts/log/Release/ToolsetRestore.binlog /clp:Summary /clp:ErrorsOnly;NoSummary /nr:false /p:TreatWarningsAsErrors=true /p:ContinuousIntegrationBuild=true /p:__ToolsetLocationOutputFile=/__w/1/s/artifacts/toolset/8.0.0-beta.22503.1.txt /t:__WriteToolsetLocation /warnaserror /__w/1/s/artifacts/toolset/restore.proj
2022-10-07T15:14:55.6156152Z /__w/1/s/eng/common/tools.sh: line 474:   537 Segmentation fault      (core dumped) "$_InitializeBuildTool" "$@"
2022-10-07T15:14:55.6156978Z Build failed with exit code 139. Check errors above.

Failed in #76737. Full log: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/44249/logs/166

{
  "ErrorMessage": "eng/common/tools.sh: line 474:   537 Segmentation fault",
  "BuildRetry": true
}

Author:	jkotas
Assignees:	-
Labels:	area-Infrastructure, Known Build Error
Milestone:	-

[EgorBo]

I hit the same issue in #76112 yesterday, disappeared after re-run

[janvorli]

I've hit the same issue in #76972 and noticed it in another PR few days ago.

[janvorli]

In both the case @jkotas referenced and in my PR, the failure was in arm64 cross build of native aot. Not sure if that's just a coincidence though:
/__w/1/s/build.sh -ci -arch arm64 -os Linux -s clr.aot+libs+libs.tests -c Release /p:TestNativeAot=true /p:RunSmokeTestsOnly=true /p:ArchiveTests=true /p:CrossBuild=true /p:MonoLLVMUseCxx11Abi=true

[MichalStrehovsky]

In both the case @jkotas referenced and in my PR, the failure was in arm64 cross build of native aot. Not sure if that's just a coincidence though: /__w/1/s/build.sh -ci -arch arm64 -os Linux -s clr.aot+libs+libs.tests -c Release /p:TestNativeAot=true /p:RunSmokeTestsOnly=true /p:ArchiveTests=true /p:CrossBuild=true /p:MonoLLVMUseCxx11Abi=true

Coincidence.

This one is failing while restoring host+packs: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/118290/logs/995

2022-12-22T01:40:48.7387119Z /__w/1/s/build.sh -subset host+packs -ci  -arch arm /p:CrossBuild=true /p:PortableBuild=true /p:SkipTests=True  /p:RuntimeArtifactsPath=/__w/1/s/artifacts/transport/coreclr /p:RuntimeConfiguration=release  /p:LibrariesConfiguration=Release /p:Configuration=Debug /p:TargetOS=Linux /p:TargetArchitecture=arm /p:RuntimeFlavor=coreclr 
2022-12-22T01:40:48.7388339Z ========================== Starting Command Output ===========================
2022-12-22T01:40:48.7433628Z [command]/bin/bash --noprofile --norc /__w/_temp/b3185808-2639-4823-907b-8265987e8dd7.sh
2022-12-22T01:40:48.8261028Z __DistroRid: linux-musl-arm
2022-12-22T01:40:48.9440350Z Downloading 'https://dotnet.microsoft.com/download/dotnet/scripts/v1/dotnet-install.sh'
2022-12-22T01:40:49.0004221Z Attempting to install dotnet from public_location.
2022-12-22T01:40:49.0104471Z dotnet-install: Note that the intended use of this script is for Continuous Integration (CI) scenarios, where:
2022-12-22T01:40:49.0107504Z dotnet-install: - The SDK needs to be installed without user interaction and without admin rights.
2022-12-22T01:40:49.0109451Z dotnet-install: - The SDK installation doesn't need to persist across multiple CI runs.
2022-12-22T01:40:49.0112241Z dotnet-install: To set up a development environment or to run apps, use installers rather than this script. Visit https://dotnet.microsoft.com/download to get the installer.
2022-12-22T01:40:49.0113208Z 
2022-12-22T01:40:49.4297716Z dotnet-install: Attempting to download using primary link https://dotnetcli.azureedge.net/dotnet/Sdk/7.0.100/dotnet-sdk-7.0.100-linux-x64.tar.gz
2022-12-22T01:40:50.7548065Z dotnet-install: Extracting zip from https://dotnetcli.azureedge.net/dotnet/Sdk/7.0.100/dotnet-sdk-7.0.100-linux-x64.tar.gz
[REDACTED FOR BREVITY]
2022-12-22T01:40:57.3692043Z /__w/1/s/.dotnet/sdk/7.0.100/MSBuild.dll /nologo -maxcpucount /m -verbosity:m /v:minimal /bl:/__w/1/s/artifacts/log/Debug/ToolsetRestore.binlog /clp:Summary /clp:ErrorsOnly;NoSummary /nr:false /p:TreatWarningsAsErrors=true /p:ContinuousIntegrationBuild=true /p:__ToolsetLocationOutputFile=/__w/1/s/artifacts/toolset/8.0.0-beta.22606.1.txt /t:__WriteToolsetLocation /warnaserror /__w/1/s/artifacts/toolset/restore.proj
2022-12-22T01:40:59.5418606Z /__w/1/s/eng/common/tools.sh: line 467:   637 Segmentation fault      (core dumped) "$_InitializeBuildTool" "$@"

[agocke]

Changing to area-CoreCLR. This is likely a product issue, not an infra issue.

[jkotas]

Closing. This stopped occurring after updating .NET SDK LKG to .NET 8 P7.

[jkotas]

Happened again in #91554 on osx-x64 release Runtime_Debug build

Full log https://dev.azure.com/dnceng-public/public/_build/results?buildId=395071&view=logs&jobId=74a332a6-4c62-563e-0293-081993a1bb75&j=74a332a6-4c62-563e-0293-081993a1bb75&t=26d9c75b-b006-57b7-96be-abd7e0e03816

...
  System.Security.Cryptography.Pkcs -> /Users/runner/work/1/s/artifacts/bin/System.Security.Cryptography.Pkcs/ref/Release/net9.0/System.Security.Cryptography.Pkcs.dll
  System.ComponentModel.Composition -> /Users/runner/work/1/s/artifacts/bin/System.ComponentModel.Composition/ref/Release/net9.0/System.ComponentModel.Composition.dll
  System.Composition.Runtime -> /Users/runner/work/1/s/artifacts/bin/System.Composition.Runtime/Release/net9.0/System.Composition.Runtime.dll
/Users/runner/work/1/s/eng/common/tools.sh: line 498:  2233 Segmentation fault: 11  "$_InitializeBuildTool" "$@"
Build failed with exit code 139. Check errors above.

[mangod9]

Looks like there is overall flakiness on macOS x64 recently with many different failure modes.