Fix non-portable distroRID generation on Alpine Linux

[ayakael]

init-distro-id.sh generates incorrect DistroRid on Alpine. While the expected DistroRid (alpine-x.xx-xxx) should only include macro version, init-distro-id.sh includes the micro version. This patches it to cut off the trailing subversion off of VERSION_ID by treating it the same way RHEL's VERSION_ID is treated.

Made as part of Alpine Linux dotnet31 / dotnet5 packaging project, see dotnet/source-build#2695

[ghost]

Tagging subscribers to this area: @dotnet/area-infrastructure-libraries
See info in area-owners.md if you want to be subscribed.

Issue Details

---

init-distro-id.sh generates incorrect DistroRid on Alpine. While the expected DistroRid (alpine-x.xx-xxx) should only include macro version, init-distro-id.sh includes the micro version. This patches it to cut off the trailing subversion off of VERSION_ID by treating it the same way RHEL's VERSION_ID is treated.

Made as part of Alpine Linux dotnet31 / dotnet5 packaging project, see dotnet/source-build#2695

Author:	ayakael
Assignees:	-
Labels:	area-Infrastructure-libraries, community-contribution
Milestone:	-

[am11]

What is a benefit of a non-portable build? I am successfully using linux-musl-x64 official binary package (as well as custom builds) on Alpine and Void musl for past couple of .NET / .NET Core releases. Same goes for glibc based distros (with generic rid: linux-x64).

re #62184 (comment); non-portable build is a legacy concept from .NET Core 1.x days and official builds stopped using it later (circa 2017). I'm sure we do not want to feed the entire https://distrowatch.com/ database in our code. :) For the version warnings, compat boundaries, the distro package manager can decide relationships between OS and .NET versions and if/when someone will try to force install a latest package on a very older version of OS, they will get exactly the same behavior as from any other software package manager is providing (may break, may work, no guarantees).

[ayakael]

What is a benefit of a non-portable build? I am successfully using linux-musl-x64 official binary package (as well as custom builds) on Alpine and Void musl for past couple of .NET / .NET Core releases. Same goes for glibc based distros (with generic rid: linux-x64).

re #62184 (comment); non-portable build is a legacy concept from .NET Core 1.x days and official builds stopped using it later (circa 2017). I'm sure we do not want to feed the entire https://distrowatch.com/ database in our code. :) For the version warnings, compat boundaries, the distro package manager can decide relationships between OS and .NET versions and if/when someone will try to force install a latest package on a very older version of OS, they will get exactly the same behavior as from any other software package manager is providing (may break, may work, no guarantees).

Thanks for asking! I'm currently working on the package for dotnet3.1, dotnet5 and dotnet6 for Alpine Linux's aport. There were a few factors that informed my going the non-portable build route for the APKBUILD script.

• It striked me as more appropriate to do a non-portable build given that I was building within the context of a packaging script
• It actually didn't dawn me as an option that the build script actually prescribed for default use-case, given that accepted RID for Alpine Linux are updated every release (in src/libraries/Microsoft.NETCore.Platforms/src/runtime.compatibility.json, among others). Thus I assumed that the build is, by default, non-portable, with option to go portable if prescribed by use case.
• I only considered the portable build on dotnet31 given that it fixed a few issues here, and there. That was just temporary though, and in fact as a portable build it causes other build oddities that I couldn't find a fix for.

With that said, I made the pull request as a courtesy in case this was indeed a bug for non-portable builds. I understand that the runtimes / sdk binaries made available by Microsoft should absolutely be portable. Although I am concerned that, without this patch (at least within the build environment for 5.0.209-SDK on Alpine Linux 3.15), non-portable builds do not work.

(edit: oh and I get the distrowatch concern, haha. Makes sense to not want to create if statements for every thing out there. I'm understanding the portable approach now given that. Although, it does beg the question, why update RID for alpine at all?)

[am11]

Although, it does beg the question, why update RID for alpine at all?

This is a very good question, and there are some on-going discussions on this topic to rethink this facility: #59803 and NuGet/Home#5862.

However, as it stands (and for past couple of releases, predating the least supported version: 3.1), the corehost (dotnet(1)) has fallback logic in place, which roughly works like this:

• if it is a portable build with RID linux-x64 (for glibc) or linux-musl-x64 (for musl-libc), traverse the RID graph (runtime.compatibility.json) from that node onwards: linux-x64 -> linux, unix-x64 -> unix.
• if it is a non-portable build with RID like (ubuntu-20.04-x64, alpine3.14-x64 and since so forth), traverse the RID graph from that node onwards..

If it doesn't find any matching nodes, it will bail and error out. Given that, portable variant is more permissive/flexible. The rest of the "compatibility" concern is not a huge issue (empirically speaking), as $other software in aports, debian packages etc. also have similar characteristic that if we force install package from other version lane, there are no guarantees that it will work (you will have to manually download/arrange dependencies for your application; which also is not super inconvenient these days).

More concrete example could be made that a third-party nuget package has a dependency on some version of a lib, which is not available on certain distro version X, then the consumer will get a runtime error, instead of a package restore (dotnet restore) time error with portable runtime. In such case, that nuget package author can add a msbuild target (that runs post-install) to validate library requirement (as opposed to a conservative distro version) on the host system and warn the user that package requirement is not met. Note that package author need that kind of a check even today to properly support portable builds (but it's not mandatory, since it will only give a run-time error which devs should test before pushing their code to production 🙂).

However, in practice, at least I have not seen users running into this kind of compat issues with official builds; where existing tooling leave them clueless (again: official builds, which have significant consumers, are portable for past several releases).

[ayakael]

Although, it does beg the question, why update RID for alpine at all?

This is a very good question, and there are some on-going discussions on this topic to rethink this facility: #59803 and NuGet/Home#5862.

However, as it stands (and for past couple of releases, predating the least supported version: 3.1), the corehost (dotnet(1)) has fallback logic in place, which roughly works like this:

• if it is a portable build with RID linux-x64 (for glibc) or linux-musl-x64 (for musl-libc), traverse the RID graph (runtime.compatibility.json) from that node onwards: linux-x64 -> linux, unix-x64 -> unix.
• if it is a non-portable build with RID like (ubuntu-20.04-x64, alpine3.14-x64 and since so forth), traverse the RID graph from that node onwards..

If it doesn't find any matching nodes, it will bail and error out. Given that, portable variant is more permissive/flexible. The rest of the "compatibility" concern is not a huge issue (empirically speaking), as $other software in aports, debian packages etc. also have similar characteristic that if we force install package from other version lane, there are no guarantees that it will work (you will have to manually download/arrange dependencies for your application; which also is not super inconvenient these days).

More concrete example could be made that a third-party nuget package has a dependency on some version of a lib, which is not available on certain distro version X, then the consumer will get a runtime error, instead of a package restore (dotnet restore) time error with portable runtime. In such case, that nuget package author can add a msbuild target (that runs post-install) to validate library requirement (as opposed to a conservative distro version) on the host system and warn the user that package requirement is not met. Note that package author need that kind of a check even today to properly support portable builds (but it's not mandatory, since it will only give a run-time error which devs should test before pushing their code to production slightly_smiling_face).

However, in practice, at least I have not seen users running into this kind of compat issues with official builds; where existing tooling leave them clueless (again: official builds, which have significant consumers, are portable for past several releases).

Truth be told, I'm not a .NET programmer: I saw an important hole in Alpine's packages, and decided to fill it by writing an APKBUILD, so I appreciate that your sharing your knowledge. If you'll allow me to pick your brain a bit more, I see that portable builds offer more flexibility for nuget packages, thus might be more convenient for the end-user. In trying to figure out best practices, I looked at the packages on Arch Linux and Fedora, both use /p:SkipPortableRuntimeBuild as true, thus going the non-portable route. Why do you think they would go that route, given the by-design inflexibility of non-portable?

[am11]

Why do you think they would go that route, given the by-design inflexibility of non-portable?

I think it was added for .NET Core v1.x compatibility in the olden days and then it became a tech debt in runtime build configurations, which we are dragging along. e.g. all the modern versions of Ubuntu (after 16.04) do not have .NET Core 1.x compatible official package: https://github.com/dotnet/sdk/blob/1814193188014a50aa63d7c2c105336c3a0b6b93/eng/restore-toolset.sh#L23 (and all runtime releases afterwards was made distro-agnostic / portable which is working out just OK; it may have lost some pedantic compatibility warnings here and there but that has not risen to the level of "critical issue"). I haven't find the arguments to the contrary; what breaks if Arch and Fedora (also) switch to portable builds..

I personally consider runtime/build.sh -portable=false as legacy switch which has only contrive real-world benefits and that demands updating untime.compatibility.json for every release of each distro (aka pretty unnatural maintenance headache for any software). To my knowledge, no other language platform (rust, node.js, ruby etc.) is bothering about such distro-level granularity, so we also should try to stay away from that option.

From your perspective, I think the question boils down to: in a long run, would you rather update community/dotnet/APKBUILD file for every .NET release (portable option; something which alpinelinux/aports maintainers regularly do), or would you update it for every .NET release and Alpine release (non-portable; maintainers will find it odd, because it really is .. very odd)?

[ayakael]

Thanks for putting it that way. Indeed, having to update RID at every update strikes me as odd. Okay, I'm sold. I'll go ahead and try a portable build, see how my APKBUILD fairs. This should slim down my patchset, which for maintenance is going to be less headache for me and future maintainers. This would make this merge request unncessary, if future best practice is to move away from non-portable altogether.

[ayakael]

I'm seeing some dissent on this being future best practice in this thread: dotnet/source-build#1529. Hard to figure out what / why people are attached to non-portable RIDs given your above arguments. As a practice, it seems desirable by package maintainer because of a perceived "protection", I suppose. But then why should dotnet do the package manager's job at making sure dependencies are there?

A technicaly question for you: Are the non-portable binaries optimized for that build environement while portable bits aren't for sake of portability? Or do the differences stop at a differing compatibility matrix as enumerated above?

[am11]

Or do the differences stop at a differing compatibility matrix as enumerated above?

This.

From the perspective of dotnet/runtime and its dependencies, AFAIK, there aren't any particular optimizations we can achieve from non-portable build, which otherwise are not possible from portable builds. e.g. one of the coreclr dependency is libunwind. In v1.x, it was required from the platform, but later in the effort to reduce the dependency requirements, it was copied in-tree and eversince it is compiled with coreclr in builds from .NET Foundation. Some distros requested and added a build-time switch to continue using this dependency from the platform, because this library is customized/optimized differently in their ports. That switch can still be used with portable builds and is not tied to non-portable build.

Another scenario where non-portable builds do not have a real benefit over portable ones:
If we install an official binary package (dotnet-sdk-linux-musl-x64.tar.gz which is portable for musl-libc based linux distros) from https://github.com/dotnet/installer#installers-and-binaries or https://dot.net/ on latest Alpine (v3.15) today and publish a .NET app using those bits, then distribute this published app and try to consume it on older version of Alpine or Void Linux (musl flavor), then it may just work, or may fail due to:

• a hard -- dynamically linked -- dependency has a mismatched version which still is fine as long as all the exported symbols required by the binary are available. Otherwise, it will cause a missing symbol error (which consumer can figure out using ldd command).
• a soft dependency (the ones which runtime and its libs dlopen() for libssl, numa, icu etc. and adapt to whichever symbols are available), is missing a symbol that is non-optional or imperative for code to function, that will cause a runtime error. Mostly the resolution for such error from runtime has a helpful error message, searchable over the internets.

(fix in both cases is typically either to update that lib on old version either from OS package manager, or make-install that lib from source)

If we now create a non-portable package, it will not behave any differently in this situation and developer will go through the same experience.

Where would non-portable build make a difference:
Essentially, it will be during the dotnet-restore command. If we are using a nuget package which has a) a native platform dependency, b) specialization for specific version of certain OS/disro (which makes it highly unlikely), that will raise a package restore-time error if we are on incompatible version. However, in practice:

• majority of community nuget packages do not directly depend on native libraries from the system (they are written in managed code: C#/F#/VB)
• and the ones which do (e.g. https://github.com/ericsink/SQLitePCL.raw/blob/c851e2395367ce9b30404630171b9aac534384fc/src/SQLitePCLRaw.lib.e_sqlcipher/SQLitePCLRaw.lib.e_sqlcipher.nuspec#L37), they do not opt for "distro version" level specificity (because it is a huge -- unmaintainable -- undertaking for anyone).

(if I am a nuget package author who really cares that much about a good restore-time compatibility warning or error message, I'd opt for a post-restore "system requirement validation" mechanism using an msbuild target)

So the "benefit" of using non-portable build is little and it depends on too many "what if"s that it is virtually non-existent.

[ayakael]

Super interesting, thanks! Further exchanges with @dagood and @crummel on dotnet/source-build#2695 parallels your explanation of portable vs non-portable vis-à-vis dynamic linking. They consider this a maintenance optimization, as it allows for automatic scanning of dependencies. Since the source-build team has identified non-portable builds as a high priority, I'm going to keep Alpine builds non-portable to follow that standard. Fedora and Red Hat package maintainers know no doubt much more than me, so I'll refer to them. Once I fix the odd kinks in the portabe build, I'll package them along-side in the bootstrap packages for effective N-1 => N builds between distro versions. Apparently, ASP.NET also relies on portable bits so it's necessary work.

[am11]

non-portable vis-à-vis dynamic linking

First, "use lib X from distro package" mechanism is very limited; one (HP libunwind) out of five external dependencies used by .NET Runtime has that option in build infrastructure.

Second, it is not tied to portable vs. non-portable builds.

[ghost]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

---

init-distro-id.sh generates incorrect DistroRid on Alpine. While the expected DistroRid (alpine-x.xx-xxx) should only include macro version, init-distro-id.sh includes the micro version. This patches it to cut off the trailing subversion off of VERSION_ID by treating it the same way RHEL's VERSION_ID is treated.

Made as part of Alpine Linux dotnet31 / dotnet5 packaging project, see dotnet/source-build#2695

Author:	ayakael
Assignees:	-
Labels:	area-Infrastructure, community-contribution
Milestone:	-

[ericstj]

@ayakael @am11 -- can you summarize the next action needed for this PR? Did you conclude it is necessary or no? cc @agocke @crummel

[ayakael]

@ericstj There are two perspectives on this: that of Microsoft, which am11 has brought to the table, and that of package maintainers. This fix isn't needed for Microsoft's use case where portable builds are produced. From a package maintainer's perspective, this is necessary as, without it, non-portable builds fail on Alpine Linux for dotnet 3.1, 5.0 and 6.0. The source-build team has shared with me that it is a priority for them to maintain portable builds. This leads me to want to refer to them when proposing that we merge, regardless of the merits (or lack thereof) of non-portable builds. Equally important to note: the backported version for 3.1 (dotnet/coreclr#28227) has already been approved for merge.

One might fear that this sets a precedent where we might be tempted to include a bunch of systems within this if statement. I think that's a non-issue given that Alpine Linux portable builds are officially produced by Microsoft. In my view, this should strengthen the case to support non-portable builds by upstream on this system.

[am11]

@ericstj, my understanding is that non-portable build of runtime entails hardcoding OS version information without any real-world advantage over portable builds. If that is true, we should discourage non-portable builds of runtime at least for new distro packages (like aports in this case).

There are two perspectives on this: that of Microsoft, which am11 has brought to the table, and that of package maintainers.

Portable build of runtime should not imply build from Microsoft and non-portable, the source-build. If this assumption breaks something in portable source-build, we should fix that. All the build knobs (turn feature X on and Y off) in runtime repo work with portable and non-portable builds alike.

One might fear that this sets a precedent where we might be tempted to include a bunch of systems within this if statement.

This condition + host changes for every distro which has slightly different version string + runtime.compatibility.json updates for every version of each distro

multiply it by every distro in the database

... then we see the scaling problem. This issue was realized in the light of the fact that no other programming platform cares about distro+version level of detail in build and there is no real-world use-case, which is why portable build mechanism (i.e. relaxed host and nuget package resolver) was implemented and made production-ready in .NET Core 2 timeframe.

[ericstj]

@hoyosjs -- can you have a look here since you approved dotnet/coreclr#28227

[hoyosjs]

My take on this was:

• The sourcebuild team seemed supportive of the stream of changes around here. The changes were small and largely innocuous. They didn't really affect how we build the product, but enabled a customer's scenario of how they wanted to build the product. To me, in open source that's something we should support if it's not detrimental to the product.
• Yes, portable builds are great for most things, but for that to work in the .NET ecosystem we always have to build in the lowest common denominator. We don't configure native builds in a way that all places where we toggle logic based on OS features can be done through runtime detection. They are often ifdef's that get enabled based on where we run CMake checks. You can build in a system that has the features you want to enable those paths, but that build is no longer portable. This felt like a reasonable mechanism to signify that. The main downside is the RID graph becomes crazy then for nuget and all the restore logic, but for a distro build of the SDK this didn't feel absurd.

[dagood]

@dotnet/source-build-internal, FYI in case the broader team is not aware of this thread. I think what's missing here is a complete accounting of why Red Hat and Fedora need the non-portable build to continue working. (So new distros can decide if it applies to them, too.)

[agocke]

My take on this is that I think it's a good thing to reduce the build complexity and the benefits from producing distro-specific builds are minimal, if present at all. I think we should try to phase out distro-specific builds entirely.

I would very much appreciate any existing user of the distro-specific builds to point out what problems this would cause.

cc @dleeapho

[trylek]

@janvorli is a long-time expert in the field of Unix distros, it might be interesting to hear his thoughts.

[janvorli]

I think that when dotnet is distributed as part of specific distro itself, the non-portable build makes sense, since that's the way that all the other packages that are part of the distro are built. The packages are supposed to depend only on the versions of packages published in the repo version. That enables the package manager of the specific distro to install all the necessary dependencies automatically and ensure that those versions work correctly with .NET. That would not work with portable build that has some of its dependencies detected dynamically at runtime. I believe that's why Redhat is building non portable builds.

For packages that Microsoft builds though, I don't see any benefit of producing non-portable builds.

[am11]

That would not work with portable build that has some of its dependencies detected dynamically at runtime.

@janvorli, do you know offhand which dependency loads differently just because of portable/non-portable difference? I don't see that in the code.

[janvorli]

@am11 I meant openSSL (FEATURE_DISTRO_AGNOSTIC_SSL). I have thought that we still do have that mechanism for ICU, but I have found that it was removed in the past so that we always dynamically load it.

[tmds]

Looking closer how build is structured, I don't see how would we miss out any freedom by having just one build flavor; without portable and non-portable distinction.

Afaik we aren't discussing portable vs non-portable build configuration. We're discussing the value of the non-portable rids.

Feedom as in FOSS means you can build stuff from source. Yes, Microsoft binaries run everywhere. That shouldn't stop us from building something ourselves.

But building something ourselves shouldn't stop users from using portable artifacts provided by Microsoft.

"build from source on Fedora 35 or Ubuntu 20.04" etc. rather than "build from source for Fedora 35 or Ubuntu 20.04"

I don't understand what you mean by this: we build .NET for Fedora 35 on Fedora 35.

That we use the non-portable rid fedora.35-x64 describes that the resulting artifacts are not portable.

That fedora.35-x64 inherits linux-x64 rid means linux-x64 artifacts are compatible with this platform.

Build on Fedora's official build system can differ by feature flags from build on that of Ubuntu's, like any other software in distro packages. Non-portable build, OTOH, is about only one combination of those feature flags, which is generalization for every distro, i.e. it is completely odd way to handle it. We should let each distro decide which feature flag they want to build.

This is also about build configuration. I agree, each build can chose its own flags.

[ayakael]

@tmds Unfortunately, I still need this patch to build under 6.0, although I don't add the missing Rid using your method. If my reading of that PR is correct, adding /p:AdditionalRuntimeIdentifiers=alpine.3.15-x64 (edit: + /p:TargetRid=alpine.3.15-x64) to ./build.sh should skip force __distroRid as alpine.3.15-x64 rather than alpine.3.15.0-x64. Am I correct?

[am11]

Afaik we aren't discussing portable vs non-portable build configuration. We're discussing the value of the non-portable rids.

This PR is related to non-portable build. RID is implicitly related and it is calculated at run-time instead of build-time.

Feedom as in FOSS means you can build stuff from source. Yes, Microsoft binaries run everywhere. That shouldn't stop us from building something ourselves.

It sounds cool, but I don't see how does that affect our ability of building .NET "from source", "by ourselves"?

But building something ourselves shouldn't stop users from using portable artifacts provided by Microsoft.

It sounds like a real problem, but data shows us it's not #62942 (comment). NuGet package published by Microsoft, Fedora, third-party org, individual etc. all kinds of possibilities cannot be all captured by the RID. It is insufficient for that purpose.

I don't understand what you mean by this: we build .NET for Fedora 35 on Fedora 35.
...
That fedora.35-x64 inherits linux-x64 rid means linux-x64 artifacts are compatible with this platform.

We shouldn't need to hard code distro info into the build artifacts and rely on platform relationships as if they will help solve compatibility issues. This is not normal. That is what I mean by on vs. for.

[ayakael]

@am11 Unless I'm mistaken of the use of eng/native/init-distro-rid.sh, this is executed at build time to parse /etc/os-release info. Due to this script not parsing /etc/os-release correctly on Alpine, it sets _DistroRid as alpine.3.15.0-x64 rather than alpine.3.15-x64. This trailing version number causes overall building of source-build to fail due to every other product expecting the latter rather than the former. I don't believe there's reason for any product to expect alpine.3.15.0-x64 as Rid.

I do appreciate the logic of avoiding hardcoding distro information in code, but so far I havn't found another solution to building dotnet on Alpine Linux without using this patch. I've intuited from @tmds that a solution might be to set /p:AdditionalRuntimeIdentifiers=alpine.3.15-x64 (edit: + /p:OutputRid=alpine.3.15-x64), but I've yet to test it. I've also intuited from your suggestion that the solution is to do portable builds. Using my above preliminary tests, building a portable dotnet using source-build isn't straightforward. I'm sure its possible, but as far as I can tell the build process isn't designed for it.

[am11]

I'm sure its possible, but as far as I can tell the build process isn't designed for it.

At least it was designed to support both flavors https://grep.app/search?q=SourceBuildPortable&filter[repo][0]=dotnet/runtime. If there is a bug somewhere, we should try to fix it (mainly to avoid maintaining dead code).

Thank you for your patience. I hope it was not blocking you from progressing, given both source-build and aports have source patching mechanism.

[ayakael]

@am11 Thanks for the flag, I didn't know about it., I'll do some tests on my side with it.

No worries, indeed APKBUILD has a patching mechanism, thus this hasn't blocked me at all. I sent this PR as a courtesy if it was indeed bug. In addition, I've found the conversation really interesting, it certainly has taught me a lot of the backend in case I encounter any future issues.

[tmds]

It sounds cool, but I don't see how does that affect our ability of building .NET "from source", "by ourselves"?

It enables distributing any package that Microsoft builds for the portable rid without making the portable rid non portable.

all kinds of possibilities cannot be all captured by the RID. It is insufficient for that purpose.

We're looking specifically at being able to build from source and distribute packages that Microsoft publishes under the portable rid.

These packages provide key features for .NET such as SCD and crossgen.

The design shouldn't prevent anyone other than Microsoft to distribute these features.

We shouldn't need to hard code distro info into the build artifacts and rely on platform relationships as if they will help solve compatibility issues.

We shouldn't use linux-x64 for non-portable artifacts.

I've intuited from @tmds that a solution might be to set /p:AdditionalRuntimeIdentifiers=alpine.3.15-x64 (edit: + /p:OutputRid=alpine.3.15-x64),

This should automagically happen by:

    <!-- When building from source, ensure the RID we're building for is part of the RID graph -->
    <AdditionalRuntimeIdentifiers Condition="'$(DotNetBuildFromSource)' == 'true'">$(AdditionalRuntimeIdentifiers);$(OutputRID)</AdditionalRuntimeIdentifiers>

[bartonjs]

Does the build-from-source RID even end up anywhere? Nothing in dotnet/runtime, to my knowledge, ever cares about (e.g.) Fedora 34 vs Fedora 35... just AnyOS/Windows/Unix/Linux/macOS/iOS/tvOS/Android/Browser, at which point it's all just "linux" for a Fedora.

If a distro is building to put things in their package manager, then the ELF files from the build won't go across distros or distro-versions (generally?), so the "what version of glibc was I linked against?" doesn't matter... that only matters for the tgz of "download this runtime onto any Linux that uses glibc" that we host.

As for the portable build vs individual feature flags... I don't think we're married to the idea of one big "I'm a distro" vs "I'm the semi-universal redist" switch... it was just an artifact of when we went for building on/for each distro individually to making the universal redist... it was the flag we needed to change to universal. I'm, personally, supportive of most anything that gets .NET into official distro package feeds. If a distro wants to officially put up a concrete request like "we want to keep ICU in flexible binding, but change the OpenSSL binding to classic dynamic linking" then that's probably something we can entertain... though not sure what level of resourcing is available, so it'd be one of those "PRs welcome" situations (provided that the manner of configuration is maintainable/understandable) -- but, as far as I can tell (with about 3 minutes' investigation), literally the only thing that varies on the portable build switch during compilation is -DFEATURE_DISTRO_AGNOSTIC_SSL=$__PortableBuild

[ayakael]

@tmds Reconfirming build bug as of 6.0.102:

  [18:49:57.11] Building 'runtime in tarball'
  Running command:
    /var/build/dotnet5/testing/dotnet6-bootstrap/src/source-build-tarball-6.0.102/src/runtime.839cdfb0ecca5e0be3dbccd926e7651ef50fdf10/build.sh  --ci --configuration Release --restore --build --pack --publish -bl /p:ArcadeBuildFromSource=true /p:CopyWipIntoInnerSourceBuildRepo=true /p:DotNetBuildOffline=true /p:CopySrcInsteadOfClone=true /p:DotNetPackageVersionPropsPath="/var/build/dotnet5/testing/dotnet6-bootstrap/src/source-build-tarball-6.0.102/artifacts/obj/x64/Release/PackageVersions.props" /p:AdditionalSourceBuiltNupkgCacheDir="/var/build/dotnet5/testing/dotnet6-bootstrap/src/source-build-tarball-6.0.102/artifacts/obj/x64/Release/blob-feed/packages/" /p:ReferencePackageNupkgCacheDir="/var/build/dotnet5/testing/dotnet6-bootstrap/src/source-build-tarball-6.0.102/packages/reference/packages/" /p:PreviouslySourceBuiltNupkgCacheDir="/usr/share/dotnet/artifacts/6.0.102/" /p:TargetRid=alpine.3.15-x64 /p:SourceBuildNonPortable=true  /p:DotNetPackageVersionPropsPath=/var/build/dotnet5/testing/dotnet6-bootstrap/src/source-build-tarball-6.0.102/artifacts/obj/x64/Release/PackageVersions.props /p:DotNetRestoreSourcePropsPath=/var/build/dotnet5/testing/dotnet6-bootstrap/src/source-build-tarball-6.0.102/artifacts/obj/x64/Release/RestoreSources.props /p:DotNetBuildOffline=true
    Log: /var/build/dotnet5/testing/dotnet6-bootstrap/src/source-build-tarball-6.0.102/artifacts/logs/runtime.log
    With Environment Variables:
      DotNetBuildFromSource=true                                                                                                            
      DotNetPackageVersionPropsPath=/var/build/dotnet5/testing/dotnet6-bootstrap/src/source-build-tarball-6.0.102/artifacts/obj/x64/Release/PackageVersions.props
      DotNetRestorePackagesPath=/var/build/dotnet5/testing/dotnet6-bootstrap/src/source-build-tarball-6.0.102/packages/restored/
      DotNetBuildOffline=true
      AddDotnetfeedProjectSource=false
      DOTNET_INSTALL_DIR=/var/build/dotnet5/testing/dotnet6-bootstrap/src/bootstrap/
      DOTNET_PATH=/var/build/dotnet5/testing/dotnet6-bootstrap/src/bootstrap/
      DOTNET_HOST_PATH=/var/build/dotnet5/testing/dotnet6-bootstrap/src/bootstrap/dotnet
      _InitializeDotNetCli=/var/build/dotnet5/testing/dotnet6-bootstrap/src/bootstrap/
      _DotNetInstallDir=/var/build/dotnet5/testing/dotnet6-bootstrap/src/bootstrap/
      _InitializeToolset=/var/build/dotnet5/testing/dotnet6-bootstrap/src/source-build-tarball-6.0.102/Tools/source-built/Microsoft.DotNet.Arcade.Sdk/tools/Build.proj
      _OverrideArcadeInitializeBuildToolFramework=net6.0
      DotNetUseShippingVersions=true                                                                       
      PreReleaseVersionLabel=                                                                            
      PackageVersionStamp=
      PB_VersionStamp=
      ContinuousIntegrationBuild=true
      MSBUILDDISABLENODEREUSE=1
      OfficialBuildId=20220214.1
      BUILD_BUILDNUMBER=20220214.1
      GitCommitCount=
      GitCommitHash=839cdfb0ecca5e0be3dbccd926e7651ef50fdf10
      GitInfoCommitHash=839cdfb0ecca5e0be3dbccd926e7651ef50fdf10
      SourceRevisionId=839cdfb0ecca5e0be3dbccd926e7651ef50fdf10
      RepositoryCommit=839cdfb0ecca5e0be3dbccd926e7651ef50fdf10
      COMMIT_SHA=839cdfb0ecca5e0be3dbccd926e7651ef50fdf10
      GIT_COMMIT=839cdfb0ecca5e0be3dbccd926e7651ef50fdf10
      RepositoryType=Git
      DeterministicSourcePaths=true
      SourceRoot=/var/build/dotnet5/testing/dotnet6-bootstrap/src/source-build-tarball-6.0.102/src/runtime.839cdfb0ecca5e0be3dbccd926e7651ef50fdf10/
      BuildInParallel=false
      LatestCommit=839cdfb0ecca5e0be3dbccd926e7651ef50fdf10
      SOURCE_BUILT_SDK_ID_ARCADE=Microsoft.DotNet.Arcade.Sdk
      SOURCE_BUILT_SDK_ID_ARCADE_PACKAGING=Microsoft.DotNet.Build.Tasks.Packaging
      SOURCE_BUILT_SDK_ID_ARCADE_TGT_FX_SDK=Microsoft.DotNet.Build.Tasks.TargetFramework.Sdk
      SOURCE_BUILT_SDK_ID_ARCADE_SHARED_FX_SDK=Microsoft.DotNet.SharedFramework.Sdk
      SOURCE_BUILT_SDK_VERSION_ARCADE=6.0.0-beta.21609.4
      SOURCE_BUILT_SDK_VERSION_ARCADE_PACKAGING=6.0.0-beta.21609.4
      SOURCE_BUILT_SDK_VERSION_ARCADE_TGT_FX_SDK=6.0.0-beta.21609.4
      SOURCE_BUILT_SDK_VERSION_ARCADE_SHARED_FX_SDK=6.0.0-beta.21609.4
      SOURCE_BUILT_SDK_DIR_ARCADE=/var/build/dotnet5/testing/dotnet6-bootstrap/src/source-build-tarball-6.0.102/Tools/source-built/Microsoft.DotNet.Arcade.Sdk/
      SOURCE_BUILT_SDK_DIR_ARCADE_PACKAGING=/var/build/dotnet5/testing/dotnet6-bootstrap/src/source-build-tarball-6.0.102/Tools/source-built/Microsoft.DotNet.Build.Tasks.Packaging/
      SOURCE_BUILT_SDK_DIR_ARCADE_TGT_FX_SDK=/var/build/dotnet5/testing/dotnet6-bootstrap/src/source-build-tarball-6.0.102/Tools/source-built/Microsoft.DotNet.Build.Tasks.TargetFramework.Sdk/
      SOURCE_BUILT_SDK_DIR_ARCADE_SHARED_FX_SDK=/var/build/dotnet5/testing/dotnet6-bootstrap/src/source-build-tarball-6.0.102/Tools/source-built/Microsoft.DotNet.SharedFramework.Sdk/
/var/build/dotnet5/testing/dotnet6-bootstrap/src/source-build-tarball-6.0.102/repos/Directory.Build.targets(418,5): error MSB3073: The command "/var/build/dotnet5/testing/dotnet6-bootstrap/src/source-build-tarball-6.0.102/src/runtime.839cdfb0ecca5e0be3dbccd926e7651ef50fdf10/build.sh  --ci --configuration Release --restore --build --pack --publish -bl /p:ArcadeBuildFromSource=true /p:CopyWipIntoInnerSourceBuildRepo=true /p:DotNetBuildOffline=true /p:CopySrcInsteadOfClone=true /p:DotNetPackageVersionPropsPath="/var/build/dotnet5/testing/dotnet6-bootstrap/src/source-build-tarball-6.0.102/artifacts/obj/x64/Release/PackageVersions.props" /p:AdditionalSourceBuiltNupkgCacheDir="/var/build/dotnet5/testing/dotnet6-bootstrap/src/source-build-tarball-6.0.102/artifacts/obj/x64/Release/blob-feed/packages/" /p:ReferencePackageNupkgCacheDir="/var/build/dotnet5/testing/dotnet6-bootstrap/src/source-build-tarball-6.0.102/packages/reference/packages/" /p:PreviouslySourceBuiltNupkgCacheDir="/usr/share/dotnet/artifacts/6.0.102/" /p:TargetRid=alpine.3.15-x64 /p:SourceBuildNonPortable=true  /p:DotNetPackageVersionPropsPath=/var/build/dotnet5/testing/dotnet6-bootstrap/src/source-build-tarball-6.0.102/artifacts/obj/x64/Release/PackageVersions.props /p:DotNetRestoreSourcePropsPath=/var/build/dotnet5/testing/dotnet6-bootstrap/src/source-build-tarball-6.0.102/artifacts/obj/x64/Release/RestoreSources.props /p:DotNetBuildOffline=true >> /var/build/dotnet5/testing/dotnet6-bootstrap/src/source-build-tarball-6.0.102/artifacts/logs/runtime.log 2>&1" exited with code 1. [/var/build/dotnet5/testing/dotnet6-bootstrap/src/source-build-tarball-6.0.102/repos/runtime.proj]
[...]
      -- Installing: /var/build/dotnet5/testing/dotnet6-bootstrap/src/source-build-tarball-6.0.102/src/runtime.839cdfb0ecca5e0be3dbccd926e7651ef50fdf10/artifacts/source-build/self/src/artifacts/bin/alpine.3.15.0-x64.Release/corehost_test/libmockhostpolicy.so
      -- Installing: /var/build/dotnet5/testing/dotnet6-bootstrap/src/source-build-tarball-6.0.102/src/runtime.839cdfb0ecca5e0be3dbccd926e7651ef50fdf10/artifacts/source-build/self/src/artifacts/bin/alpine.3.15.0-x64.Release/corehost_test/nativehost
      ~/dotnet5/testing/dotnet6-bootstrap/src/source-build-tarball-6.0.102/src/runtime.839cdfb0ecca5e0be3dbccd926e7651ef50fdf10/artifacts/source-build/self/src/src/native/corehost
      Copying /var/build/dotnet5/testing/dotnet6-bootstrap/src/source-build-tarball-6.0.102/src/runtime.839cdfb0ecca5e0be3dbccd926e7651ef50fdf10/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release//corehost/. to /var/build/dotnet5/testing/dotnet6-bootstrap/src/source-build-tarball-6.0.102/src/runtime.839cdfb0ecca5e0be3dbccd926e7651ef50fdf10/artifacts/source-build/self/src/artifacts/bin/alpine.3.15.0-x64.Release/corehost
      Microsoft.NETCore.DotNetAppHost -> /var/build/dotnet5/testing/dotnet6-bootstrap/src/source-build-tarball-6.0.102/src/runtime.839cdfb0ecca5e0be3dbccd926e7651ef50fdf10/artifacts/source-build/self/src/artifacts/packages/Release/Shipping/Microsoft.NETCore.DotNetAppHost.6.0.2.nupkg
      Microsoft.NETCore.DotNetHost -> /var/build/dotnet5/testing/dotnet6-bootstrap/src/source-build-tarball-6.0.102/src/runtime.839cdfb0ecca5e0be3dbccd926e7651ef50fdf10/artifacts/source-build/self/src/artifacts/packages/Release/Shipping/Microsoft.NETCore.DotNetHost.6.0.2.nupkg
      Microsoft.NETCore.DotNetHostPolicy -> /var/build/dotnet5/testing/dotnet6-bootstrap/src/source-build-tarball-6.0.102/src/runtime.839cdfb0ecca5e0be3dbccd926e7651ef50fdf10/artifacts/source-build/self/src/artifacts/packages/Release/Shipping/Microsoft.NETCore.DotNetHostPolicy.6.0.2.nupkg
      Microsoft.NETCore.DotNetHostResolver -> /var/build/dotnet5/testing/dotnet6-bootstrap/src/source-build-tarball-6.0.102/src/runtime.839cdfb0ecca5e0be3dbccd926e7651ef50fdf10/artifacts/source-build/self/src/artifacts/packages/Release/Shipping/Microsoft.NETCore.DotNetHostResolver.6.0.2.nupkg
    /var/build/dotnet5/testing/dotnet6-bootstrap/src/source-build-tarball-6.0.102/src/runtime.839cdfb0ecca5e0be3dbccd926e7651ef50fdf10/artifacts/source-build/self/package-cache/microsoft.dotnet.build.tasks.packaging/6.0.0-beta.21609.4/build/Packaging.targets(1257,5): error : Error when creating nuget lib package from /var/build/dotnet5/testing/dotnet6-bootstrap/src/source-build-tarball-6.0.102/src/runtime.839cdfb0ecca5e0be3dbccd926e7651ef50fdf10/artifacts/source-build/self/src/artifacts/packages/Release/specs/runtime.alpine.3.15-x64.Microsoft.NETCore.DotNetAppHost.nuspec. System.IO.DirectoryNotFoundException: Could not find a part of the path '/var/build/dotnet5/testing/dotnet6-bootstrap/src/source-build-tarball-6.0.102/src/runtime.839cdfb0ecca5e0be3dbccd926e7651ef50fdf10/artifacts/source-build/self/src/artifacts/bin/alpine.3.15-x64.Release/corehost'. [/var/build/dotnet5/testing/dotnet6-bootstrap/src/source-build-tarball-6.0.102/src/runtime.839cdfb0ecca5e0be3dbccd926e7651ef50fdf10/artifacts/source-build/self/src/src/installer/pkg/projects/Microsoft.NETCore.DotNetAppHost/Microsoft.NETCore.DotNetAppHost.pkgproj]

Missing file is actually in /var/build/dotnet5/testing/dotnet6-bootstrap/src/source-build-tarball-6.0.102/src/runtime.839cdfb0ecca5e0be3dbccd926e7651ef50fdf10/artifacts/source-build/self/src/artifacts/bin/alpine.3.15.0-x64.Release/corehost when runtime build itself expects it to be at /var/build/dotnet5/testing/dotnet6-bootstrap/src/source-build-tarball-6.0.102/src/runtime.839cdfb0ecca5e0be3dbccd926e7651ef50fdf10/artifacts/source-build/self/src/artifacts/bin/alpine.3.15-x64.Release/corehost.

edit: Do note that I add RIDs for Alpine to runtime.compatibility.json, runtime.json and runtimeGroups.props .

[tmds]

Does the build-from-source RID even end up anywhere?

In the host that lives at packs/Microsoft.NETCore.App.Host.<rid>.

And imo we should also consider the packages Microsoft publishes for the portable rid (like Microsoft.NETCore.App.Runtime.linux-x64) and allow these to be built and distributed as FOSS without inappropriately using the portable rid.

Missing file is actually in /var/build/dotnet5/testing/dotnet6-bootstrap/src/source-build-tarball-6.0.102/src/runtime.839cdfb0ecca5e0be3dbccd926e7651ef50fdf10/artifacts/source-build/self/src/artifacts/bin/alpine.3.15.0-x64.Release/corehost when runtime build itself expects it to be at /var/build/dotnet5/testing/dotnet6-bootstrap/src/source-build-tarball-6.0.102/src/runtime.839cdfb0ecca5e0be3dbccd926e7651ef50fdf10/artifacts/source-build/self/src/artifacts/bin/alpine.3.15-x64.Release/corehost.

The difference is alpine.3.15.0 vs alpine.3.15. The alpine rid gets normalized to the alpine.x.y, and that normalization is indeed missing in init-distro-rid.sh.

[tmds]

The alpine rid gets normalized to the alpine.x.y, and that normalization is indeed missing in init-distro-rid.sh.

This change is appropriate to match the normalization that happens in other places like:

runtime/src/native/corehost/hostmisc/pal.unix.cpp

Lines 718 to 736 in a4a6d30

	pal::string_t normalize_linux_rid(pal::string_t rid)
	{
	pal::string_t rhelPrefix(_X("rhel."));
	pal::string_t alpinePrefix(_X("alpine."));
	pal::string_t rockyPrefix(_X("rocky."));
	size_t lastVersionSeparatorIndex = std::string::npos;
	if (rid.compare(0, rhelPrefix.length(), rhelPrefix) == 0)
	{
	lastVersionSeparatorIndex = rid.find(_X("."), rhelPrefix.length());
	}
	else if (rid.compare(0, alpinePrefix.length(), alpinePrefix) == 0)
	{
	size_t secondVersionSeparatorIndex = rid.find(_X("."), alpinePrefix.length());
	if (secondVersionSeparatorIndex != std::string::npos)
	{
	lastVersionSeparatorIndex = rid.find(_X("."), secondVersionSeparatorIndex + 1);
	}
	}

@ericstj can you merge the PR?

The discussion about the sense/non-sense of non-portable rids can be continued on NuGet/Home#5862, #59803 or a new issue.

[ericstj]

We need fresh PR validation results, I'd also prefer if this is merged by someone who's actively working on infra - just in case there's something that breaks. @hoyosjs @bartonjs

[ericstj]

/azp run runtime

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[crummel]

I hit the retry button on the timed-out AzDo jobs. It sounds like we should move the discussion about portable RIDs to NuGet/Home#5862? I can post over there and summarize the discussion from this issue.

[tmds]

This can be merged. The rid discussion has moved to the Simplify RID Model design.

[omajid]

Ping. Can we merge the fix in this PR? I don't think it's controversial at all. I need it to get CI for alpine working.

The RID discussion is worth continuing, but I don't think there's any need to hold back this PR; it's not adding or modifying RIDs to the graph.

[agocke]

Yes, this is fine to merge.