Eliminate portable runtime build. #14549

tmds · 2022-09-20T06:19:11Z

Contributes to dotnet/source-build#3027.

tmds · 2022-09-20T06:19:49Z

src/SourceBuild/tarball/content/repos/package-source-build.proj

@@ -50,7 +50,7 @@
      <SourceBuiltTarballName>$(OutputPath)$(SourceBuiltArtifactsTarballName).$(installerOutputPackageVersion).tar.gz</SourceBuiltTarballName>
    </PropertyGroup>

-    <Exec Command="tar --numeric-owner --exclude='Microsoft.SourceBuild.Intermediate.*.nupkg' --exclude='runtime.$(TargetRid).*.nupkg' -czf $(SourceBuiltTarballName) *.nupkg *.props SourceBuildReferencePackages/"
+    <Exec Command="tar --numeric-owner --exclude='Microsoft.SourceBuild.Intermediate.*.nupkg' -czf $(SourceBuiltTarballName) *.nupkg *.props SourceBuildReferencePackages/"


This change is to include ilasm for the non-portable rid.

tmds · 2022-09-20T06:20:16Z

src/SourceBuild/tarball/patches/aspnetcore/0004-non-portable-build.patch

@@ -0,0 +1,337 @@
+From 4e92a776a0bb0b8a4ab02ad4e436352594b7eab2 Mon Sep 17 00:00:00 2001


This is dotnet/aspnetcore#43937

tmds · 2022-09-20T06:24:55Z

src/SourceBuild/tarball/patches/installer/0001-non-portable-build.patch

+     <MicrosoftNETCoreAppHostPackageVersion>$(MicrosoftNETCoreAppHostwinx64PackageVersion)</MicrosoftNETCoreAppHostPackageVersion>
+     <MicrosoftNETCoreAppRuntimePackageVersion>$(MicrosoftNETCoreAppRuntimewinx64PackageVersion)</MicrosoftNETCoreAppRuntimePackageVersion>
+-    <MicrosoftAspNetCoreAppRuntimePackageVersion>$(MicrosoftAspNetCoreAppRuntimewinx64PackageVersion)</MicrosoftAspNetCoreAppRuntimePackageVersion>
+    <MicrosoftAspNetCoreAppRuntimePackageVersionBuilt>$(MicrosoftAspNetCoreAppRuntimewinx64PackageVersion)</MicrosoftAspNetCoreAppRuntimePackageVersionBuilt>


I've added this to deal with https://github.com/dotnet/installer/issues/14492. This renames the property so source-build doesn't override it. There are better ways to fix this. I noticed the issue after I wrote this patch.

tmds · 2022-09-20T06:25:30Z

src/SourceBuild/tarball/patches/runtime/0003-non-portable-build.patch

@@ -0,0 +1,67 @@
+From 39d69b410eff3e6a22aa3a25489263a35c7da453 Mon Sep 17 00:00:00 2001


This is dotnet/runtime#75597.

tmds · 2022-09-20T06:32:45Z

I ran this locally using Microsoft's portable sdk.
Then I took the output from the build (non-portable sdk and assets) and made another build.

This built fine with the exception of fsharp which fails on the second build with:

    FSharp.Build -> /home/tmds/tarball61/src/fsharp/artifacts/source-build/self/src/artifacts/bin/FSharp.Build/Proto/netstandard2.0/FSharp.Build.dll
    FSharp.Compiler.Service -> /home/tmds/tarball61/src/fsharp/artifacts/source-build/self/src/artifacts/bin/FSharp.Compiler.Service/Proto/netstandard2.0/FSharp.Compiler.Service.dll
FSC : error : no entrypoint specified in executable binary [/home/tmds/tarball61/src/fsharp/artifacts/source-build/self/src/src/fsc/fscProject/fsc.fsproj::TargetFramework=net7.0] [/home/tmds/tarball61/Tools/source-built/Microsoft.DotNet.Arcade.Sdk/tools/Build.proj]
##vso[task.logissue type=error;sourcepath=FSC;linenumber=0;columnnumber=0;code=;]no entrypoint specified in executable binary [/home/tmds/tarball61/src/fsharp/artifacts/source-build/self/src/src/fsc/fscProject/fsc.fsproj::TargetFramework=net7.0%5D
    fsc -> /home/tmds/tarball61/src/fsharp/artifacts/source-build/self/src/artifacts/bin/fsc/Proto/net7.0/fsc.dll
FSC : error : no entrypoint specified in executable binary [/home/tmds/tarball61/src/fsharp/artifacts/source-build/self/src/src/fsi/fsiProject/fsi.fsproj::TargetFramework=net7.0] [/home/tmds/tarball61/Tools/source-built/Microsoft.DotNet.Arcade.Sdk/tools/Build.proj]
##vso[task.logissue type=error;sourcepath=FSC;linenumber=0;columnnumber=0;code=;]no entrypoint specified in executable binary [/home/tmds/tarball61/src/fsharp/artifacts/source-build/self/src/src/fsi/fsiProject/fsi.fsproj::TargetFramework=net7.0%5D
    fsi -> /home/tmds/tarball61/src/fsharp/artifacts/source-build/self/src/artifacts/bin/fsi/Proto/net7.0/fsi.dll

I have no clue what is causing this or to get better debug info.
It could be that two successive builds without these changes also have the issue. I have not tried.
Can someone help me figure out what the issue is?

tmds · 2022-09-20T13:09:28Z

This built fine with the exception of fsharp which fails on the second build with:

The issue is not caused by these changes. I've created dotnet/source-build#3030.

MichaelSimons · 2022-09-20T13:20:08Z

@tmds - This is a fantastic contribution I've been wanting for some time. Can you clarify your intent on if you are trying to get this into 7.0 or 8.0? I ask because the contributing changes in the repos like runtime and aspnetcore are going into main and this PR is going into 7.0.

MichaelSimons

Once the FSharp issue is resolved (dotnet/source-build#3030), I'd like to trigger the internal CI on this change to validate everything builds cleanly rebuilding the tarball with the SDK and artifacts from these changes.

src/SourceBuild/tarball/content/repos/known-good.proj

tmds · 2022-09-20T14:03:32Z

Can you clarify your intent on if you are trying to get this into 7.0 or 8.0? I ask because the contributing changes in the repos like runtime and aspnetcore are going into main and this PR is going into 7.0.

I'm targeting the main branches to get feedback from the repo owners, and here I'm using 7.0 to validate.

We can decide if we want to do this 7.0 or later depending on the risk. We could also make it opt-in.

It builds, but it isn't finished yet. I think the crossgen2 package that gets built from runtime no longer meets the built-from-source requirement. I've created dotnet/runtime#74721 for that.

Once the FSharp issue is resolved (dotnet/source-build#3030), I'd like to trigger the internal CI on this change to validate everything builds cleanly rebuilding the tarball with the SDK and artifacts from these changes.

Is this a CI job that will build installer twice? One starting with the prebuilts + portable SDK, and then a second build with the output of the first build?

MichaelSimons · 2022-09-20T14:07:51Z

We can decide if we want to do this 7.0 or later depending on the risk. We could also make it opt-in.

Sounds good.

Is this a CI job that will build installer twice? One starting with the prebuilts + portable SDK, and then a second build with the output of the first build?

Correct

tmds · 2022-09-21T04:56:14Z

src/SourceBuild/tarball/patches/sdk/0003-rid-assert.patch

@@ -0,0 +1,37 @@
+From 43f51a115a8f20b0c1374afa180c2a67db2e357e Mon Sep 17 00:00:00 2001


This is dotnet/sdk#28005.

ayakael · 2022-09-26T02:46:19Z

In testing this against 7.0.100-rc1 on Alpine Linux, build of aspnetcore fails with:

Could not locate the package or project for "Microsoft.NETCore.App.Runtime.alpine.3.17-x64"

I reckon this is due to the initial prebuilt bootstrap being portable. Aspnetcore should be able to buld with portable runtime despite targeting non-portable.

edit Ignore this, I made a mistake when porting this pull request to the Alpine building script. It works now.

ayakael · 2022-09-26T02:59:04Z

Incidently this fixes musl support on installer, as the issues had to do entirely with portable builds in source-build. I'm doing more tests, but it looks like it would make #13410 unnecessary.

ayakael · 2022-09-27T04:21:44Z

Finished tests for dotnet7 rc1 on Alpine Linux. Substantial build time savings:

x86_64: 89 min to 66 min
aarch64: 181 min to 97 min
armv7: tbd, but given current build time of 8 hours, going down to 3 would be terrific.

Very happy with this change, and would be happy to see it included in net7.0.

tmds · 2022-09-28T19:51:46Z

The tarball build fails with:

2022-09-28T19:24:52.6807149Z     Build FAILED.
2022-09-28T19:24:52.6807585Z     
2022-09-28T19:24:52.6811751Z     /tarball/src/runtime/artifacts/source-build/self/package-cache/microsoft.dotnet.apicompat.task/7.0.100-rc.1.22429.2/build/Microsoft.NET.ApiCompat.ValidatePackage.targets(32,5): error CP0002: Member 'Microsoft.Extensions.Caching.Memory.PostEvictionDelegate.PostEvictionDelegate(object, System.IntPtr)' exists on lib/netstandard2.0/Microsoft.Extensions.Caching.Abstractions.dll but not on lib/net7.0/Microsoft.Extensions.Caching.Abstractions.dll [/tarball/src/runtime/artifacts/source-build/self/src/src/libraries/Microsoft.Extensions.Caching.Abstractions/src/Microsoft.Extensions.Caching.Abstractions.csproj]
2022-09-28T19:24:52.6816093Z         0 Warning(s)
2022-09-28T19:24:52.6816602Z         1 Error(s)

This seems related to dotnet/sdk#25566.

tmds · 2022-09-29T06:32:23Z

The tarball build fails with:

I had accidentally removed disabling package validation which caused the build failure.

tmds · 2022-09-29T09:00:26Z

src/SourceBuild/tarball/patches/runtime/0003-build-unknown.patch

@@ -0,0 +1,266 @@
+From 65dba1ea70461a99f585daebed67a3ad52b3f48c Mon Sep 17 00:00:00 2001


This is dotnet/runtime#74504 (merged on runtime main).

tmds · 2022-09-29T09:00:58Z

src/SourceBuild/tarball/patches/runtime/0004-update-graph.patch

@@ -0,0 +1,127 @@
+From cd2f0ffb287251c26f03850d1a596abafb99d5c7 Mon Sep 17 00:00:00 2001


This is dotnet/runtime#76068 (merged on runtime main).

tmds · 2022-09-29T09:01:39Z

src/SourceBuild/tarball/patches/runtime/0005-non-portable-build.patch

@@ -0,0 +1,52 @@
+From 2775cf0d524f7a31f8cad3405b3e111056b345e3 Mon Sep 17 00:00:00 2001


This is dotnet/runtime#75597.

tmds · 2022-09-29T15:44:13Z

This now builds without the portable runtime.

There is still an issue with the crossgen2 package that gets built by runtime.

This contains a self-contained app. Self-contained apps get built using artifacts from microsoft.netcore.app.runtime.<rid> package. So the package that now comes out of runtime has prebuilt binaries in it.

It should be possible to use the microsoft.netcore.app.runtime.<rid> that got built as part of the runtime build, but I haven't been able to make that work.
There is an open issue for this: dotnet/runtime#74721.

tmds · 2022-09-30T12:36:58Z

Besides the patches needed to eliminate the portable runtime build, I've included patches that enable source-build to build on distros (and distro versions) that are not yet known in the runtime graph (dotnet/runtime#74504 and dotnet/runtime#76068).

I've included them in this PR to see if they work well with the other RID-related changes.

tmds · 2022-11-08T18:00:02Z

I'll look into making the backport PRs.

We also still need a proper fix for dotnet/source-build#3121.

ayakael · 2022-11-08T18:09:19Z

I'll look into making the backport PRs.

We also still need a proper fix for dotnet/source-build#3121.

Indeed. I've been using your fix, that is to say changing the name of MicrosoftAspNetCoreAppRuntimePackageVersion to MicrosoftAspNetCoreAppRuntimePackageVersionBuilt. In fact, it is what I backported to release/6.0.1xx in #14816. It was my understanding that this is the fix we're putting forward.

tmds · 2022-11-14T07:59:18Z

Status of the backports:

aspnetcore: backported to 6.0, 7.0
runtime: backported to 6.0, 7.0: [release/7.0] source build backports runtime#78395
sdk: 6.0: [release/6.0.1xx] ResolveReadyToRunCompilers: map non-portable rids when targetOS is determined (backport #28380) sdk#28782, 7.0: [release/7.0.1xx] ResolveReadyToRunCompilers: map non-portable rids when targetOS is determined. (#28380) sdk#29052 not yet merged. cc @AntonLapounov @ViktorHofer
installer: source-build: stop overriding aspnetcore version #14938 to be reviewed, and backported cc @MichaelSimons @crummel

tmds · 2022-11-18T07:24:04Z

Update on 7.0 backports:

runtime: to be merged: [release/7.0] source build backports runtime#78395
installer: to be backported: source-build: stop overriding aspnetcore version #14938, Eliminate portable runtime build. #14549

tmds · 2022-12-01T19:23:02Z

installer: to be backported: #14938, #14549

@crummel @MichaelSimons are you looking into backporting these to 7.0? Is there something I can do to help?

MichaelSimons · 2022-12-01T19:31:15Z

This was picked up in 7.0.1 servicing (internally). It will be released in Dec servicing.

MichaelSimons · 2022-12-01T19:32:04Z

The runtime and aspnetcore changes that were backported flowed in and required the installer changes.

tmds · 2022-12-14T04:03:48Z

@MichaelSimons this change is now part of tag/v7.0.101-source-build, but not part of release/7.0.1xx. Is that expected? Will it become part of the 7.0.1xx release branch?

MichaelSimons · 2022-12-14T17:09:13Z

Is that expected?

Yes it is. The internal servicing changes made for 7.0.101 changes aren't getting mirror publicly because of the tight timeline for the January release because of the holiday vacations. The changes will get mirrored during the 7.0.102 release in January. I apologize for any inconveniences this causes.

tmds · 2023-01-11T05:48:24Z

The changes will get mirrored during the 7.0.102 release in January.

The changes are part of the 7.0.102 tag, will they be mirrored automatically to the 7.0.1xx and 7.0.2xx branches?

MichaelSimons · 2023-01-11T14:47:24Z

will they be mirrored automatically to the 7.0.1xx and 7.0.2xx branches?

Yes.

tmds requested a review from a team as a code owner September 20, 2022 06:19

tmds commented Sep 20, 2022

View reviewed changes

tmds mentioned this pull request Sep 20, 2022

source-build: support building aspnetcore using non-portable runtime packages. dotnet/aspnetcore#43937

Merged

This was referenced Sep 20, 2022

source-build: support building runtime using non-portable runtime packages. dotnet/runtime#75597

Merged

[release/7.0.1xx] installer fails to build using a sdk+packages built by installer dotnet/source-build#3030

Closed

MichaelSimons reviewed Sep 20, 2022

View reviewed changes

src/SourceBuild/tarball/content/repos/known-good.proj Show resolved Hide resolved

tmds commented Sep 21, 2022

View reviewed changes

This was referenced Sep 26, 2022

.NET 7: Alpine 3.17 package issue tracker dotnet/source-build#3020

Closed

[release/6.0.1xx] Fix build against musl #13410

Closed

tmds mentioned this pull request Sep 27, 2022

Microsoft.NETCore.App.Crossgen2.sfxproj in source-build dotnet/runtime#74721

Closed

tmds commented Sep 29, 2022

View reviewed changes

tmds mentioned this pull request Oct 3, 2022

[release/7.0.1xx] Wrong version of aspnetcore.runtime.internal is pulled by GenerateLayout dotnet/source-build#3121

Closed

ayakael mentioned this pull request Oct 3, 2022

[main] Adds alternative musl version of MicrosoftAspNetCoreAppRuntimePackageVersion #14644

Closed

ayakael added a commit to ayakael/installer that referenced this pull request Nov 10, 2022

Eliminate runtime portable build (backport dotnet#14549)

055e1e8

crummel pushed a commit to ayakael/installer that referenced this pull request Nov 15, 2022

Eliminate runtime portable build (backport dotnet#14549)

0b55f0a

ayakael added a commit to ayakael/installer that referenced this pull request Nov 15, 2022

Eliminate runtime portable build (backport dotnet#14549)

2085cfe

ayakael added a commit to ayakael/installer that referenced this pull request Nov 16, 2022

Eliminate runtime portable build (backport dotnet#14549)

fd04c9c

This was referenced Nov 21, 2022

Make it unnecessary to add new versions of known distros into the RID graph dotnet/runtime#59803

Closed

Eliminate the 'runtime-portable' build dotnet/source-build#3027

Closed

Add support for banana linux dotnet/source-build#297

Open

This was referenced Nov 30, 2022

[release/7.0.1xx] Build failure when source-build builds on itself due to _GenerateComposites dotnet/source-build#3140

Closed

Add Alpine CI for source-build #14417

Merged

This was referenced Nov 30, 2022

[VMR] Build failure with portable RID (e.g. /p:TargetRid=linux-x64) dotnet/source-build#3131

Closed

[ppc64le][.net7] Unhandled Exception: System.PlatformNotSupportedException: System.Data.ODBC is not supported on this platform dotnet/runtime#78563

Closed

ayakael added a commit to ayakael/installer that referenced this pull request Dec 16, 2022

Eliminate runtime portable build (backport dotnet#14549)

cfb3bcb

ayakael mentioned this pull request Dec 21, 2022

VoidLinux dotnet discussion void-dotnet/void-packages#1

Closed

ayakael added a commit to ayakael/installer that referenced this pull request Jan 10, 2023

Eliminate runtime portable build (backport dotnet#14549)

08486a0

ayakael added a commit to ayakael/installer that referenced this pull request Jan 11, 2023

Eliminate runtime portable build (backport dotnet#14549)

747ef04

ayakael added a commit to ayakael/installer that referenced this pull request Jan 13, 2023

Eliminate runtime portable build (backport dotnet#14549)

4cf6d9f

ayakael added a commit to ayakael/installer that referenced this pull request Jan 13, 2023

Eliminate runtime portable build (backport dotnet#14549)

3008f7e

crummel pushed a commit to ayakael/installer that referenced this pull request Jan 24, 2023

Eliminate runtime portable build (backport dotnet#14549)

d673a17

ayakael mentioned this pull request Sep 9, 2023

[release/6.0.1xx] Fix aspnetcore build on musl-libc platforms #17324

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Eliminate portable runtime build. #14549

Eliminate portable runtime build. #14549

tmds commented Sep 20, 2022

tmds Sep 20, 2022

tmds Sep 20, 2022

tmds Sep 20, 2022

tmds Sep 20, 2022

tmds commented Sep 20, 2022

tmds commented Sep 20, 2022

MichaelSimons commented Sep 20, 2022

MichaelSimons left a comment

tmds commented Sep 20, 2022

MichaelSimons commented Sep 20, 2022

tmds Sep 21, 2022

ayakael commented Sep 26, 2022 •

edited

ayakael commented Sep 26, 2022

ayakael commented Sep 27, 2022

tmds commented Sep 28, 2022

tmds commented Sep 29, 2022

tmds Sep 29, 2022

tmds Sep 29, 2022

tmds Sep 29, 2022

tmds commented Sep 29, 2022

tmds commented Sep 30, 2022

tmds commented Nov 8, 2022

ayakael commented Nov 8, 2022

tmds commented Nov 14, 2022 •

edited

tmds commented Nov 18, 2022

tmds commented Dec 1, 2022

MichaelSimons commented Dec 1, 2022

MichaelSimons commented Dec 1, 2022

tmds commented Dec 14, 2022

MichaelSimons commented Dec 14, 2022

tmds commented Jan 11, 2023

MichaelSimons commented Jan 11, 2023

		@@ -0,0 +1,337 @@
		From 4e92a776a0bb0b8a4ab02ad4e436352594b7eab2 Mon Sep 17 00:00:00 2001

		@@ -0,0 +1,67 @@
		From 39d69b410eff3e6a22aa3a25489263a35c7da453 Mon Sep 17 00:00:00 2001

		@@ -0,0 +1,37 @@
		From 43f51a115a8f20b0c1374afa180c2a67db2e357e Mon Sep 17 00:00:00 2001

		@@ -0,0 +1,266 @@
		From 65dba1ea70461a99f585daebed67a3ad52b3f48c Mon Sep 17 00:00:00 2001

		@@ -0,0 +1,127 @@
		From cd2f0ffb287251c26f03850d1a596abafb99d5c7 Mon Sep 17 00:00:00 2001

		@@ -0,0 +1,52 @@
		From 2775cf0d524f7a31f8cad3405b3e111056b345e3 Mon Sep 17 00:00:00 2001

Eliminate portable runtime build. #14549

Eliminate portable runtime build. #14549

Conversation

tmds commented Sep 20, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tmds commented Sep 20, 2022

tmds commented Sep 20, 2022

MichaelSimons commented Sep 20, 2022

MichaelSimons left a comment

Choose a reason for hiding this comment

tmds commented Sep 20, 2022

MichaelSimons commented Sep 20, 2022

Choose a reason for hiding this comment

ayakael commented Sep 26, 2022 • edited

ayakael commented Sep 26, 2022

ayakael commented Sep 27, 2022

tmds commented Sep 28, 2022

tmds commented Sep 29, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tmds commented Sep 29, 2022

tmds commented Sep 30, 2022

tmds commented Nov 8, 2022

ayakael commented Nov 8, 2022

tmds commented Nov 14, 2022 • edited

tmds commented Nov 18, 2022

tmds commented Dec 1, 2022

MichaelSimons commented Dec 1, 2022

MichaelSimons commented Dec 1, 2022

tmds commented Dec 14, 2022

MichaelSimons commented Dec 14, 2022

tmds commented Jan 11, 2023

MichaelSimons commented Jan 11, 2023

ayakael commented Sep 26, 2022 •

edited

tmds commented Nov 14, 2022 •

edited