[mono] Restore failure due to The process cannot access the file x because it is being used by another process
#77364
Comments
|
I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label. |
|
This is confirmed reproducible on |
|
False alarm - updating to latest commit of |
|
Never mind, indeed it happens less, but it still occurs enough to make our build fail consistently: |
|
Tagging subscribers to this area: @dotnet/runtime-infrastructure Issue DetailsDescriptionWhen building runtime with mono-flavored runtime, restore often fails in CI environment with Reproduction StepsIn an Alpine Edge git clone https://gitlab.alpinelinux.org/ayakael/aports -b dotnet7/mono-restore
cd aports/testing/dotnet7-stage0
abuild deps unpack prepare buildIt should eventually fail. The aport builds a minimum set of components (runtime-mono, roslyn, sdk, aspnetcore, installer) to be able to build an SDK tar, then using that produced tarball with mono-flavored runtime it builds the whole stack again. This aport is usually used to crossbuild to other platforms, but in this case I am using it to easily reproduce the bug. You'd likely be able to reproduce this on Expected behaviorRestore should occur without issue Actual behaviorRestore fails with the following error: Regression?This error is confirmed not present with .NET runtime 6.0.10 in Known WorkaroundsNone so far. ConfigurationVersion: SDK Other informationlogs
|
|
Not sure what causes this, I've never see this particular error in my builds ... Looking at the runtime sources, it seems this message is triggered only if some Linux native library call returns with errno set to |
Indeed, can't make heads or tails. One piece of data is that it going from tarball from Diff between those two commits is hard to figure out. @omajid Have you started trying build of dotnet7 on |
|
The diff to git-info shows the following: --- dotnet-d41bfecf5090e9163aa2da251246abed9e756e53/git-info/AllRepoVersions.props
+++ dotnet-e6dd91c290b808f971a1ac69c2fb29395bbf1051/git-info/AllRepoVersions.props
@@ -15,8 +15,8 @@
<templatingOutputPackageVersion>7.0.100-rtm.22510.12</templatingOutputPackageVersion>
<xdtGitCommitHash>9a1c3e1b7f0c8763d4c96e593961a61a72679a7b</xdtGitCommitHash>
<xdtOutputPackageVersion>7.0.0-preview.22423.2</xdtOutputPackageVersion>
- <arcadeGitCommitHash>720af493900b2f2bdc48e9ee12577983a5c9be36</arcadeGitCommitHash>
- <arcadeOutputPackageVersion>7.0.0-beta.22464.4</arcadeOutputPackageVersion>
+ <arcadeGitCommitHash>02e28316bf35d1028683ee313f0794776bff18d1</arcadeGitCommitHash>
+ <arcadeOutputPackageVersion>7.0.0-beta.22513.4</arcadeOutputPackageVersion>
<aspnetcoreGitCommitHash>c6865355c01e1fb170b65427846d939559ab3789</aspnetcoreGitCommitHash>
<aspnetcoreOutputPackageVersion>7.0.0-rtm.22512.1</aspnetcoreOutputPackageVersion>
<deploymenttoolsGitCommitHash>c3ad00ae84489071080a606f6a8e43c9a91a5cc2</deploymenttoolsGitCommitHash>
@@ -49,7 +49,7 @@
<vstestOutputPackageVersion>17.4.0-release-20220926-01</vstestOutputPackageVersion>
<xlifftasksGitCommitHash>740189d758fb3bbdc118c5b6171ef1a7351a8c44</xlifftasksGitCommitHash>
<xlifftasksOutputPackageVersion>1.0.0-beta.22427.1</xlifftasksOutputPackageVersion>
- <installerGitCommitHash>d41bfecf5090e9163aa2da251246abed9e756e53</installerGitCommitHash>
+ <installerGitCommitHash>e6dd91c290b808f971a1ac69c2fb29395bbf1051</installerGitCommitHash>
<installerOutputPackageVersion>7.0.100</installerOutputPackageVersion>
</PropertyGroup>
</Project>The following components have thus changed:
|
|
In turn, diff of arcade between those two commits goes as follows: diff --git a/.config/tsaoptions.json b/.config/tsaoptions.json
new file mode 100644
index 00000000..ae219a82
--- /dev/null
+++ b/.config/tsaoptions.json
@@ -0,0 +1,10 @@
+{
+ "instanceUrl": "https://devdiv.visualstudio.com/",
+ "template": "TFSDEVDIV",
+ "projectName": "DEVDIV",
+ "areaPath": "DevDiv\\NET Fundamentals\\Infrastructure\\Arcade\\SDL",
+ "iterationPath": "DevDiv",
+ "notificationAliases": [ "dnceng@microsoft.com" ],
+ "repositoryName":"Arcade",
+ "codebaseName": "Arcade"
+}
\ No newline at end of file
diff --git a/azure-pipelines-codeql.yml b/azure-pipelines-codeql.yml
index ea648627..d85d52d2 100644
--- a/azure-pipelines-codeql.yml
+++ b/azure-pipelines-codeql.yml
@@ -1,49 +1,63 @@
+parameters:
+ # Optionally do not publish to TSA. Useful for e.g. verifying fixes before PR.
+- name: TSAEnabled
+ displayName: Publish results to TSA
+ type: boolean
+ default: true
+
variables:
- - name: _TeamName
- value: DotNetCore
- - group: SDL_Settings
-
+- template: eng/common-variables.yml
+ # CG is handled in the primary CI pipeline
+- name: skipComponentGovernanceDetection
+ value: true
+ # Force CodeQL enabled so it may be run on any branch
+- name: Codeql.Enabled
+ value: true
+ # Do not let CodeQL 3000 Extension gate scan frequency
+- name: Codeql.Cadence
+ value: 0
+ # CodeQL needs this plumbed along as a variable to enable TSA
+- name: Codeql.TSAEnabled
+ value: ${{ parameters.TSAEnabled }}
+
+ # Build variables
+- name: _BuildConfig
+ value: Release
+
trigger: none
schedules:
- cron: 0 12 * * 1
- displayName: Weekly Monday CodeQL/Semmle run
+ displayName: Weekly Monday CodeQL run
branches:
include:
- main
+ - release/6.0
+ - release/7.0
always: true
-stages:
-- stage: build
- displayName: Build
- # Three phases for each of the three OSes we want to run on
- jobs:
- - template: /eng/common/templates/jobs/codeql-build.yml
- parameters:
- jobs:
- - job: Windows_NT_CSharp
- timeoutInMinutes: 90
- pool:
- name: NetCore1ESPool-Svc-Internal
- demands: ImageOverride -equals windows.vs2019.amd64
+jobs:
+- job: codeql
+ displayName: CodeQL
+ pool:
+ name: NetCore1ESPool-Svc-Internal
+ demands: ImageOverride -equals 1es-windows-2022
+ timeoutInMinutes: 90
- steps:
- - checkout: self
- clean: true
+ steps:
- - template: /eng/common/templates/steps/execute-codeql.yml
- parameters:
- executeAllSdlToolsScript: 'eng/common/sdl/execute-all-sdl-tools.ps1'
- buildCommands: 'eng\common\cibuild.cmd -configuration Release -prepareMachine /p:Test=false /p:Sign=false'
- language: csharp
- publishGuardianDirectoryToPipeline: true
- additionalParameters: '-SourceToolsList @("semmle")
- -TsaInstanceURL $(_TsaInstanceURL)
- -TsaProjectName $(_TsaProjectName)
- -TsaNotificationEmail $(_TsaNotificationEmail)
- -TsaCodebaseAdmin $(_TsaCodebaseAdmin)
- -TsaBugAreaPath $(_TsaBugAreaPath)
- -TsaIterationPath $(_TsaIterationPath)
- -TsaRepositoryName "Arcade"
- -TsaCodebaseName "Arcade"
- -TsaPublish $True'
+ - task: UseDotNet@2
+ inputs:
+ useGlobalJson: true
+
+ - task: CodeQL3000Init@0
+ displayName: CodeQL Initialize
+
+ - script: eng\common\cibuild.cmd
+ -configuration $(_BuildConfig)
+ -prepareMachine
+ /p:Test=false
+ displayName: Windows Build
+
+ - task: CodeQL3000Finalize@0
+ displayName: CodeQL Finalize
diff --git a/eng/Version.Details.xml b/eng/Version.Details.xml
index 5916476a..6002233c 100644
--- a/eng/Version.Details.xml
+++ b/eng/Version.Details.xml
@@ -47,9 +47,9 @@
<Uri>https://github.com/dotnet/xharness</Uri>
<Sha>fbeb09787a4cdcf8a375382cf7a4f5edfaf1b9d7</Sha>
</Dependency>
- <Dependency Name="Microsoft.Net.Compilers.Toolset" Version="4.4.0-2.22426.8">
+ <Dependency Name="Microsoft.Net.Compilers.Toolset" Version="4.4.0-3.22472.2">
<Uri>https://github.com/dotnet/roslyn</Uri>
- <Sha>a9ddbc56659be7d9fdd956c926af3b7cd5b8e44f</Sha>
+ <Sha>7a48ae565f3c36155fc8f606384e20b2a54f98c8</Sha>
</Dependency>
<Dependency Name="Microsoft.NET.ILLink.Tasks" Version="6.0.100-1.22103.2">
<Uri>https://github.com/dotnet/linker</Uri>
diff --git a/eng/Versions.props b/eng/Versions.props
index e83705a2..540ea350 100644
--- a/eng/Versions.props
+++ b/eng/Versions.props
@@ -40,7 +40,7 @@
<MicrosoftExtensionsFileSystemGlobbingVersion>2.0.0</MicrosoftExtensionsFileSystemGlobbingVersion>
<MicrosoftExtensionsLoggingConsoleVersion>2.1.1</MicrosoftExtensionsLoggingConsoleVersion>
<MicrosoftNETCorePlatformsVersion>2.1.0</MicrosoftNETCorePlatformsVersion>
- <MicrosoftNetCompilersToolsetVersion>4.4.0-2.22426.8</MicrosoftNetCompilersToolsetVersion>
+ <MicrosoftNetCompilersToolsetVersion>4.4.0-3.22472.2</MicrosoftNetCompilersToolsetVersion>
<MicrosoftNetTestSdkVersion>17.4.0-preview-20220707-01</MicrosoftNetTestSdkVersion>
<MicrosoftNETILLinkTasksVersion>6.0.100-1.22103.2</MicrosoftNETILLinkTasksVersion>
<MicrosoftSignedWixVersion>1.0.0-v3.14.0.5722</MicrosoftSignedWixVersion>
diff --git a/eng/common/build.ps1 b/eng/common/build.ps1
index 8943da24..e0420a64 100644
--- a/eng/common/build.ps1
+++ b/eng/common/build.ps1
@@ -26,6 +26,7 @@ Param(
[string] $runtimeSourceFeed = '',
[string] $runtimeSourceFeedKey = '',
[switch] $excludePrereleaseVS,
+ [switch] $nativeToolsOnMachine,
[switch] $help,
[Parameter(ValueFromRemainingArguments=$true)][String[]]$properties
)
@@ -66,6 +67,7 @@ function Print-Usage() {
Write-Host " -prepareMachine Prepare machine for CI run, clean up processes after build"
Write-Host " -warnAsError <value> Sets warnaserror msbuild parameter ('true' or 'false')"
Write-Host " -msbuildEngine <value> Msbuild engine to use to run build ('dotnet', 'vs', or unspecified)."
+ Write-Host " -nativeToolsOnMachine Sets the native tools on machine environment variable (indicating that the script should use native tools on machine)"
Write-Host " -excludePrereleaseVS Set to exclude build engines in prerelease versions of Visual Studio"
Write-Host ""
@@ -146,6 +148,9 @@ try {
$nodeReuse = $false
}
+ if ($nativeToolsOnMachine) {
+ $env:NativeToolsOnMachine = $true
+ }
if ($restore) {
InitializeNativeTools
}
diff --git a/eng/common/init-tools-native.ps1 b/eng/common/init-tools-native.ps1
index 8d48ec56..fbc67eff 100644
--- a/eng/common/init-tools-native.ps1
+++ b/eng/common/init-tools-native.ps1
@@ -98,11 +98,12 @@ try {
Write-Error "Arcade tools directory '$ArcadeToolsDirectory' was not found; artifacts were not properly installed."
exit 1
}
- $ToolDirectory = (Get-ChildItem -Path "$ArcadeToolsDirectory" -Filter "$ToolName-$ToolVersion*" | Sort-Object -Descending)[0]
- if ([string]::IsNullOrWhiteSpace($ToolDirectory)) {
+ $ToolDirectories = (Get-ChildItem -Path "$ArcadeToolsDirectory" -Filter "$ToolName-$ToolVersion*" | Sort-Object -Descending)
+ if ($ToolDirectories -eq $null) {
Write-Error "Unable to find directory for $ToolName $ToolVersion; please make sure the tool is installed on this image."
exit 1
}
+ $ToolDirectory = $ToolDirectories[0]
$BinPathFile = "$($ToolDirectory.FullName)\binpath.txt"
if (-not (Test-Path -Path "$BinPathFile")) {
Write-Error "Unable to find binpath.txt in '$($ToolDirectory.FullName)' ($ToolName $ToolVersion); artifact is either installed incorrectly or is not a bootstrappable tool."
@@ -112,6 +113,7 @@ try {
$ToolPath = Convert-Path -Path $BinPath
Write-Host "Adding $ToolName to the path ($ToolPath)..."
Write-Host "##vso[task.prependpath]$ToolPath"
+ $env:PATH = "$ToolPath;$env:PATH"
$InstalledTools += @{ $ToolName = $ToolDirectory.FullName }
}
}
diff --git a/eng/common/sdk-task.ps1 b/eng/common/sdk-task.ps1
index c35087a0..39be08d4 100644
--- a/eng/common/sdk-task.ps1
+++ b/eng/common/sdk-task.ps1
@@ -64,7 +64,7 @@ try {
$GlobalJson.tools | Add-Member -Name "vs" -Value (ConvertFrom-Json "{ `"version`": `"16.5`" }") -MemberType NoteProperty
}
if( -not ($GlobalJson.tools.PSObject.Properties.Name -match "xcopy-msbuild" )) {
- $GlobalJson.tools | Add-Member -Name "xcopy-msbuild" -Value "17.2.1" -MemberType NoteProperty
+ $GlobalJson.tools | Add-Member -Name "xcopy-msbuild" -Value "17.3.1" -MemberType NoteProperty
}
if ($GlobalJson.tools."xcopy-msbuild".Trim() -ine "none") {
$xcopyMSBuildToolsFolder = InitializeXCopyMSBuild $GlobalJson.tools."xcopy-msbuild" -install $true
diff --git a/eng/common/tools.ps1 b/eng/common/tools.ps1
index aba6308a..44912694 100644
--- a/eng/common/tools.ps1
+++ b/eng/common/tools.ps1
@@ -365,8 +365,8 @@ function InitializeVisualStudioMSBuild([bool]$install, [object]$vsRequirements =
# If the version of msbuild is going to be xcopied,
# use this version. Version matches a package here:
- # https://dev.azure.com/dnceng/public/_packaging?_a=package&feed=dotnet-eng&package=RoslynTools.MSBuild&protocolType=NuGet&version=17.2.1&view=overview
- $defaultXCopyMSBuildVersion = '17.2.1'
+ # https://dev.azure.com/dnceng/public/_packaging?_a=package&feed=dotnet-eng&package=RoslynTools.MSBuild&protocolType=NuGet&version=17.3.1view=overview
+ $defaultXCopyMSBuildVersion = '17.3.1'
if (!$vsRequirements) {
if (Get-Member -InputObject $GlobalJson.tools -Name 'vs') {
diff --git a/global.json b/global.json
index 00c6f187..4a5989c8 100644
--- a/global.json
+++ b/global.json
@@ -1,6 +1,6 @@
{
"tools": {
- "dotnet": "7.0.100-rc.1.22431.12"
+ "dotnet": "7.0.100-rc.2.22477.23"
},
"msbuild-sdks": {
"Microsoft.DotNet.Arcade.Sdk": "7.0.0-beta.22426.8",
diff --git a/src/Microsoft.DotNet.Arcade.Sdk/tools/SdkTasks/VisualStudio.BuildIbcTrainingSettings.proj b/src/Microsoft.DotNet.Arcade.Sdk/tools/SdkTasks/VisualStudio.BuildIbcTrainingSettings.proj
index abf2b732..ce73ed93 100644
--- a/src/Microsoft.DotNet.Arcade.Sdk/tools/SdkTasks/VisualStudio.BuildIbcTrainingSettings.proj
+++ b/src/Microsoft.DotNet.Arcade.Sdk/tools/SdkTasks/VisualStudio.BuildIbcTrainingSettings.proj
@@ -17,7 +17,6 @@
-->
<Import Project="Directory.Build.props" />
- <Import Project="Directory.Build.targets" />
<PropertyGroup>
<_VisualStudioBuildTasksAssembly>$(NuGetPackageRoot)microsoft.dotnet.build.tasks.visualstudio\$(MicrosoftDotNetBuildTasksVisualStudioVersion)\tools\net472\Microsoft.DotNet.Build.Tasks.VisualStudio.dll</_VisualStudioBuildTasksAssembly>
diff --git a/src/Microsoft.DotNet.Arcade.Sdk/tools/SourceBuild/SourceBuildArcadeBuild.targets b/src/Microsoft.DotNet.Arcade.Sdk/tools/SourceBuild/SourceBuildArcadeBuild.targets
index 336aafb0..02c7d543 100644
--- a/src/Microsoft.DotNet.Arcade.Sdk/tools/SourceBuild/SourceBuildArcadeBuild.targets
+++ b/src/Microsoft.DotNet.Arcade.Sdk/tools/SourceBuild/SourceBuildArcadeBuild.targets
@@ -82,11 +82,6 @@
<!-- The inner build needs to reference the overall output dir for nupkg transport etc. -->
<InnerBuildArgs>$(InnerBuildArgs) /p:SourceBuildOutputDir=$(SourceBuildOutputDir)</InnerBuildArgs>
<InnerBuildArgs>$(InnerBuildArgs) /p:SourceBuiltBlobFeedDir=$(SourceBuiltBlobFeedDir)</InnerBuildArgs>
-
- <!-- Work around issue where local clone may cause failure using non-origin remote fallback: https://github.com/dotnet/sourcelink/issues/629 -->
- <InnerBuildArgs>$(InnerBuildArgs) /p:EnableSourceControlManagerQueries=false</InnerBuildArgs>
- <InnerBuildArgs>$(InnerBuildArgs) /p:EnableSourceLink=false</InnerBuildArgs>
- <InnerBuildArgs>$(InnerBuildArgs) /p:DeterministicSourcePaths=false</InnerBuildArgs>
</PropertyGroup>
<ItemGroup>
diff --git a/src/Microsoft.DotNet.Helix/Sdk/tools/dotnet-cli/DotNetCli.props b/src/Microsoft.DotNet.Helix/Sdk/tools/dotnet-cli/DotNetCli.props
index b0b5978c..f594af9f 100644
--- a/src/Microsoft.DotNet.Helix/Sdk/tools/dotnet-cli/DotNetCli.props
+++ b/src/Microsoft.DotNet.Helix/Sdk/tools/dotnet-cli/DotNetCli.props
@@ -1,7 +1,7 @@
<Project>
<PropertyGroup>
<IncludeDotNetCli Condition=" '$(IncludeDotNetCli)' != 'true' ">false</IncludeDotNetCli>
- <AspNetCoreRuntimeVersion>7.0.0-rc.1.22423.7</AspNetCoreRuntimeVersion>
+ <AspNetCoreRuntimeVersion>7.0.0-rc.2.22476.2</AspNetCoreRuntimeVersion>
<DotNetCliPackageType Condition=" '$(DotNetCliPackageType)' == '' ">runtime</DotNetCliPackageType>
<DotNetCliVersion Condition=" '$(DotNetCliVersion)' == '' AND '$(DotNetCliPackageType)' == 'runtime' ">$(BundledNETCoreAppPackageVersion)</DotNetCliVersion>
<!-- TODO (https://github.com/dotnet/arcade/issues/7022): We are hardcoding this version to use the one tied to the SDK version from global.json -->
diff --git a/src/Microsoft.DotNet.Helix/Sdk/tools/xharness-runner/XHarnessRunner.targets b/src/Microsoft.DotNet.Helix/Sdk/tools/xharness-runner/XHarnessRunner.targets
index 5af41dc7..c1c8414d 100644
--- a/src/Microsoft.DotNet.Helix/Sdk/tools/xharness-runner/XHarnessRunner.targets
+++ b/src/Microsoft.DotNet.Helix/Sdk/tools/xharness-runner/XHarnessRunner.targets
@@ -2,7 +2,7 @@
<PropertyGroup Condition=" '$(IncludeXHarnessCli)' == 'true' ">
<IncludeDotNetCli>true</IncludeDotNetCli>
<XHarnessTargetFramework Condition=" '$(XHarnessTargetFramework)' == '' ">net7.0</XHarnessTargetFramework>
- <DotNetCliVersion Condition=" '$(XHarnessTargetFramework)' == 'net7.0' ">7.0.100-rc.1.22431.12</DotNetCliVersion>
+ <DotNetCliVersion Condition=" '$(XHarnessTargetFramework)' == 'net7.0' ">7.0.100-rc.2.22477.23</DotNetCliVersion>
<DotNetCliVersion Condition=" '$(XHarnessTargetFramework)' == 'net6.0' ">6.0.202</DotNetCliVersion>
<DotNetCliPackageType>sdk</DotNetCliPackageType>
</PropertyGroup>
diff --git a/src/Microsoft.DotNet.SharedFramework.Sdk/targets/sharedfx.targets b/src/Microsoft.DotNet.SharedFramework.Sdk/targets/sharedfx.targets
index 902c8db7..09ea6fb3 100644
--- a/src/Microsoft.DotNet.SharedFramework.Sdk/targets/sharedfx.targets
+++ b/src/Microsoft.DotNet.SharedFramework.Sdk/targets/sharedfx.targets
@@ -360,7 +360,7 @@
</Target>
<Target Name="_CreatePlatformManifest" DependsOnTargets="_GenerateTemplatedPlatformManifest;_GeneratePlatformManifestFromRuntimePack"
- Condition="'$(PlatformPackageType)' == 'TargetingPack'">
+ Condition="'$(PlatformPackageType)' == 'TargetingPack' and '$(SkipGeneratingPlatformManifest)' == ''">
<ItemGroup>
<_PlatformManifestFile Include="$(IntermediateOutputPath)PlatformManifest.txt" TargetPath="data" GeneratedBuildFile="true" />
</ItemGroup>Anything look suspicious? |
|
I haven not seen this error on s390x, though I still occasionally run into roslyn hangs. |
|
Please share an msbuild binlog when that happens again which might help us to root cause the issue. |
|
Finally got the error to reappear on my format.binlog.log Had to append .log to allow upload on github. |
|
Higher quality binlogs extracted directly from Alpine pipelines for ppc64le. For some reason, ppc64le seems to have this error more than s390x. ppc64le - runtime: The roslyn binlog seems much richer in information when replying in diag mode. Strange observation: when building with |
|
Binlogs mention src/NuGet.Core/NuGet.Common/ConcurrencyUtilities.cs |
|
Exporting |
|
Closing in favor of NuGet/Home#12242 |
|
@tmds @uweigand Moving back the conversation to runtime, as a nuget dev has confirmed that this is likely a mono issue. The referenced issue has a bunch of different binlogs for when the issue occurs as datapoints. A workaround was setting The error: « while trying to create a directory, it looks like some syscall (mkdir?) returns EAGAIN/EWOULDBLOCK. This is not expected. » (@tmds) Because this occurs on both Exploring the above, this patch was attempted, but unfortunately there was no positive result. diff --git a/src/native/libs/System.Native/pal_io.c b/src/native/libs/System.Native/pal_io.c
index d211328f71..fe5ebc25b9 100644
--- a/src/native/libs/System.Native/pal_io.c
+++ b/src/native/libs/System.Native/pal_io.c
@@ -227,7 +227,7 @@ int32_t SystemNative_Stat(const char* path, FileStatus* output)
{
struct stat_ result;
int ret;
- while ((ret = stat_(path, &result)) < 0 && errno == EINTR);
+ while ((ret = stat_(path, &result)) < 0 && (errno == EINTR || errno == EAGAIN));
if (ret == 0)
{
@@ -689,7 +689,7 @@ int32_t SystemNative_FcntlGetIsNonBlocking(intptr_t fd, int32_t* isNonBlocking)
int32_t SystemNative_MkDir(const char* path, int32_t mode)
{
int32_t result;
- while ((result = mkdir(path, (mode_t)mode)) < 0 && errno == EINTR);
+ while ((result = mkdir(path, (mode_t)mode)) < 0 && (errno == EINTR || errno == EAGAIN));
return result;
} |
|
Have you tried building with mono runtime on x64 (should be possible with dotnet/installer#14792)? Does that also reproduce the issue? That @omajid and @uweigand have not seen it on RHEL builds suggests it may be related to the Alpine build environment (musl/filesystem/...). If you can reproduce it with an Alpine mono x64 build, I will try if I can reproduce it on Fedora. Errno is stored thread local (both unmanaged as managed). Maybe we're picking up someone else's errno. |
|
The design of the interop marshalling source generators assumes that it is possible to reliably retrieve We had a few bugs where various lazy helpers did not preserve |
This comment was marked as resolved.
This comment was marked as resolved.
|
It might be possible to modify mono to save/restore errno, but there are a lot of random code paths which could get executed between the 2 pinvokes, so this seems like a brittle approach. |
This comment was marked as resolved.
This comment was marked as resolved.
Yes, it requires special care to ensure every path preserves Anyone source-building .NET using mono runtime is now using the |
|
Looking at In runtime/src/mono/mono/utils/mono-threads-coop.c Lines 365 to 368 in 73b5ac0 Which points to mono/mono#14101 which describes a very similar situation:
So it seems likely that when we exit from GC safe mode while there's already a GC underway, we end up blocking which calls some underlying primitive that clobbers Seems likely we need xplat analogues of /fyi @lateralusX |
|
Interesting thing about musl:
so more evidence that it's the exit from GC safe that is responsible for the clobbered |
lambdageek
added
area-VM-threading-mono
and removed
area-Infrastructure
untriaged
We already save/restore GetLastError on win32. Do it on posix platforms, too. If one thread is in a pinvoke wrapper, while another thread triggers a STW, the pinvoke wrapper will self-suspend the thread and wait for a notification to resume. Depending on the platform we can use win32 primitives, Mach semaphores or POSIX semaphores. win32 and posix can both change the value of last error (errno, respectively) while the thread is suspended. That means that code like this (generated by the LibraryImportAttribute source generator) cannot reliably retrieve the error from the last pinvoke: ```csharp __retVal = __PInvoke(__path_native, mode); // there is a pinvoke wrapper here, that transitions from GC Safe to GC Unsafe mode __lastError = System.Runtime.InteropServices.Marshal.GetLastSystemError(); ``` The solution is to explicitly preserve the value of GetLastError/errno when exiting from GC Safe. Fixes dotnet#77364
ghost
added
the
in-pr
|
Ah well this is funny - just as I sent the last comment, my build of roslyn on s390x failed with the error. Attached is a binlog: edit Redoing tests as to cover ppc64le here: https://gitlab.alpinelinux.org/ayakael/aports/-/merge_requests/12 |
|
This appears to be the best summary of the issue: #77364 (comment) @ViktorHofer looks like there is a binlog to look at. FYI @marek-safar |
The binlog unfortunately doesn't tell where the double write / lock comes from:
|
If the issue occurs less after applying #79856, we could assume that it fixes a case where The fails observed after applying the change could be another case to be handled (or even a different bug). @ayakael did you see a clear improvement after applying #79856? |
Apologies for the late reply, I wanted to package dotnet8-preview1 to test. I've tested on |
* [threads] Save errno when using posix semaphores for thread transitions We already save/restore GetLastError on win32. Do it on posix platforms, too. If one thread is in a pinvoke wrapper, while another thread triggers a STW, the pinvoke wrapper will self-suspend the thread and wait for a notification to resume. Depending on the platform we can use win32 primitives, Mach semaphores or POSIX semaphores. win32 and posix can both change the value of last error (errno, respectively) while the thread is suspended. That means that code like this (generated by the LibraryImportAttribute source generator) cannot reliably retrieve the error from the last pinvoke: ```csharp __retVal = __PInvoke(__path_native, mode); // there is a pinvoke wrapper here, that transitions from GC Safe to GC Unsafe mode __lastError = System.Runtime.InteropServices.Marshal.GetLastSystemError(); ``` The solution is to explicitly preserve the value of GetLastError/errno when exiting from GC Safe. Fixes #77364 * rename W32_DEFINE_LAST_ERROR_RESTORE_POINT to MONO_DEFINE_LAST_ERROR_RESTORE_POINT and W32_RESTORE_LAST_ERROR_FROM_RESTORE_POINT to MONO_RESTORE_LAST_ERROR_FROM_RESTORE_POINT
ghost
removed
the
in-pr
|
It seems like there's a regression. I tested preview.7 on ppc64le, and still occasionally get this issue. Should I open a new issue? |
Description
When building runtime with mono-flavored runtime, restore often fails in CI environment with
The process cannot access the file x because it is being used by another process. This affects runtime, as well as other builds (confirmed roslyn), thus specificity to runtime is that this does not occur on coreclr-flavored runtime.Reproduction Steps
In an Alpine Edge
linux-musl-x64environment, you can use this modified aport to reproduce bug. Following steps to reproduce:git clone https://gitlab.alpinelinux.org/ayakael/aports -b dotnet7/mono-restore cd aports/testing/dotnet7-stage0 abuild deps unpack prepare buildIt should eventually fail.
The aport builds a minimum set of components (runtime-mono, roslyn, sdk, aspnetcore, installer) to be able to build an SDK tar, then using that produced tarball with mono-flavored runtime it builds the whole stack again. This aport is usually used to crossbuild to other platforms, but in this case I am using it to easily reproduce the bug.
You'd likely be able to reproduce this on
linux-x64 by building runtime with/p:PrimaryRuntimeFlavor=Mono` and trying to build runtime with produced artifacts.Expected behavior
Restore should occur without issue
Actual behavior
Restore fails with the following error:
Regression?
This error is confirmed not present with .NET runtime 6.0.10 in
s390x, thus it seems to be a regression.Known Workarounds
None so far.
Configuration
Version: SDK
7.0.100-rtm.22519.39OS: Alpine LInux Edge docker-based CI Pipelines (although occurs to a lesser degree in s390x VM)
Architecure: Confirmed reproducible in s390x, ppc64le and x64 (with mono-flavored runtime)
Specificity: specific to mono on all platforms
Other information
logs
ppc64le with runtime: link
s390x with runtime: link
s390x with roslyn: link
The text was updated successfully, but these errors were encountered: