document how bundler and rubygems behave wrt linux, linux-gnu, and linux-musl releases

[flavorjones]

We know that rubygems 3.33.22 / bundler 2.3.21 is required for basic support of linux-gnu and linux-musl platform names; but buggy bundler behavior prior to 2.5.6 may cause challenges for some users.

This issue is to record some research into how the default versions behave with different native platform gems, and to document recommendations for maintainers as to how and when to ship linux-gnu and linux-musl versions.

testing

To help test this, I published "linux + linux-musl" and "linux-gnu + linux-musl" combinations of the gem rcee_precompiled which you can find at https://rubygems.org/gems/rcee_precompiled/versions.

• v0.5.0 and v0.5.0.1 are "linux + linux-musl"
• v0.5.1 is `"linux-gnu + linux--musl"

Note that the versions in question being tested below are:

ruby	gem	bundler
3.0	3.2.33	2.2.33
3.0 (updated rubygems)	3.5.7	2.5.7
3.1	3.3.26	2.3.26
3.2	3.4.19	2.4.19
3.3	3.5.3	2.5.3

and the alpine containers have build-base installed to compile the source gem if necessary.

We test Ruby 3.0 with updated rubygems because its default version is below the known minimum needed for this to work correctly (3.3.22).

testing gem install

target	linux + linux-musl (v0.5.0.1)	linux-gnu + linux-musl (v0.5.1)
3.0 gnu	✔ installs -linux	🤷 installs source gem
3.0 musl	✖ installs -linux	🤷 installs source gem
3.0 gnu (updated rubygems)	✔ installs -linux	✔ installs -linux-gnu
3.0 musl (updated rubygems)	✔ installs -linux-musl	✔ installs -linux-musl
3.1 gnu	✔ installs -linux	✔ installs -linux-gnu
3.1 musl	✔ installs -linux-musl	✔ installs -linux-musl
3.2 gnu	✔ installs -linux	✔ installs -linux-gnu
3.2 musl	✔ installs -linux-musl	✔ installs -linux-musl
3.3 gnu	✔ installs -linux	✔ installs -linux-gnu
3.3 musl	✔ installs -linux-musl	✔ installs -linux-musl

testing bundle install

target	linux + linux-musl (v0.5.0.1)	linux-gnu + linux-musl (v0.5.1)
3.0 gnu	✔ installs -linux	🤷 installs source gem
3.0 musl	🤷 installs source gem	🤷 installs source gem
3.0 gnu (updated rubygems)	✖ fails to install [1]	✔ installs -linux-gnu
3.0 musl (updated rubygems)	✔ installs -linux-musl	✔ installs -linux-musl
3.1 gnu	✔ installs -linux	✔ installs -linux-gnu
3.1 musl	✔ installs -linux-musl	✔ installs -linux-musl
3.2 gnu	✔ installs -linux	✔ installs -linux-gnu
3.2 musl	✔ installs -linux-musl	✔ installs -linux-musl
3.3 gnu	✔ installs -linux	✔ installs -linux-gnu
3.3 musl	✖ installs -linux	✔ installs -linux-musl

Error [1]:

Could not find gems matching 'rcee_precompiled (= 0.5.0.1)' valid for all resolution platforms (aarch64-linux-musl,
aarch64-linux, arm-linux-musl, arm-linux, arm64-darwin, x86-linux-musl, x86-linux, x86_64-darwin, x86_64-linux-musl) in
rubygems repository https://rubygems.org/ or installed locally.

The source contains the following gems matching 'rcee_precompiled (= 0.5.0.1)':
  * rcee_precompiled-0.5.0.1-aarch64-linux
  * rcee_precompiled-0.5.0.1-aarch64-linux-musl
  * rcee_precompiled-0.5.0.1-arm-linux
  * rcee_precompiled-0.5.0.1-arm-linux-musl
  * rcee_precompiled-0.5.0.1-arm64-darwin
  * rcee_precompiled-0.5.0.1
  * rcee_precompiled-0.5.0.1-x64-mingw-ucrt
  * rcee_precompiled-0.5.0.1-x64-mingw32
  * rcee_precompiled-0.5.0.1-x86-linux
  * rcee_precompiled-0.5.0.1-x86-linux-musl
  * rcee_precompiled-0.5.0.1-x86_64-darwin
  * rcee_precompiled-0.5.0.1-x86_64-linux
  * rcee_precompiled-0.5.0.1-x86_64-linux-musl

Recommendations

Given the inconsisent behavior of gem and bundle when the gem in question ships linux and linux-musl platform gems, it seems pretty clear that it's not a good idea to do this.

So the recommendation has to be:

• either:
  • ship a -linux platform gem that works for both gnu and musl systems,
  • or ship both -linux-gnu and -linux-musl platform gems.
• do NOT ship -linux and -linux-musl platform gems for the same version.

The installation docs for users should contain some warnings:

• end users on Ruby 3.0 must be using gem 3.3.22 or later / bundler 2.3.21 or later to get basic support for precompiled gems (will fall back to installing the vanilla "ruby" platform gem) set required_rubygems_version for native gems that specify the linux libc rake-compiler#236
• end users on musl systems should be using bundler 2.5.6 or later to avoid on musl systems, bundler does not include the relevant musl native gem platform in the lock file rubygems/rubygems#7432

[flavorjones]

OK, so understanding that bundler's behavior has changed in at least two unfortunate ways for the "linux + linux-musl" scenario:

• at some version since 2.5.3, on gnu target systems, bundler stopped being able to resolve a gem
• at some version between 2.4.19 and 2.5.3, on musl target systems, bundler stopped being able to resolve a gem

Let's dig in.

gnu target platform

On a gnu system (docker image ruby:3.2), running ruby 3.2.3, with the following Gemfile:

source "https://rubygems.org"

gem "rcee_precompiled", "0.5.0.1"

bundler	status
2.4.19	✔
2.4.22	✔
2.5.0	✔
2.5.3	✔
2.5.4	✔
2.5.5	✔
2.5.6	✖ [1]

Error [1]: git-bisected to rubygems/rubygems@66ee354

Could not find gems matching 'rcee_precompiled (= 0.5.0.1)' valid for all resolution platforms (aarch64-linux-musl,
aarch64-linux, arm-linux-musl, arm-linux, arm64-darwin, x86-linux-musl, x86-linux, x86_64-darwin, x86_64-linux-musl) in
rubygems repository https://rubygems.org/ or installed locally.

The source contains the following gems matching 'rcee_precompiled (= 0.5.0.1)':
  * rcee_precompiled-0.5.0.1-aarch64-linux
  * rcee_precompiled-0.5.0.1-aarch64-linux-musl
  * rcee_precompiled-0.5.0.1-arm-linux
  * rcee_precompiled-0.5.0.1-arm-linux-musl
  * rcee_precompiled-0.5.0.1-arm64-darwin
  * rcee_precompiled-0.5.0.1
  * rcee_precompiled-0.5.0.1-x64-mingw-ucrt
  * rcee_precompiled-0.5.0.1-x64-mingw32
  * rcee_precompiled-0.5.0.1-x86-linux
  * rcee_precompiled-0.5.0.1-x86-linux-musl
  * rcee_precompiled-0.5.0.1-x86_64-darwin
  * rcee_precompiled-0.5.0.1-x86_64-linux
  * rcee_precompiled-0.5.0.1-x86_64-linux-musl

musl target platform

On a musl system (docker image ruby:3.2-alpine), running ruby 3.2.3, with the following Gemfile:

source "https://rubygems.org"

gem "rcee_precompiled", "0.5.0.1"

bundler	status
2.4.19	✔
2.4.20	✔
2.4.21	✔
2.4.22	✔
2.5.0	✖ [1]
2.5.5	✖ [1]
2.5.6	✔ [2]
2.5.7	✔

Error [1]: incorrectly installs x86_64-linux instead of x86_64-linux-musl

# Gemfile.lock
GEM
  remote: https://rubygems.org/
  specs:
    rcee_precompiled (0.5.0.1)
      mini_portile2
    rcee_precompiled (0.5.0.1-aarch64-linux)
    rcee_precompiled (0.5.0.1-aarch64-linux-musl)
    rcee_precompiled (0.5.0.1-arm-linux)
    rcee_precompiled (0.5.0.1-arm-linux-musl)
    rcee_precompiled (0.5.0.1-arm64-darwin)
    rcee_precompiled (0.5.0.1-x86-linux)
    rcee_precompiled (0.5.0.1-x86-linux-musl)
    rcee_precompiled (0.5.0.1-x86_64-darwin)
    rcee_precompiled (0.5.0.1-x86_64-linux)

PLATFORMS
  aarch64-linux
  aarch64-linux-musl
  arm-linux
  arm-linux-musl
  arm64-darwin
  ruby
  x86-linux
  x86-linux-musl
  x86_64-darwin
  x86_64-linux

DEPENDENCIES
  rcee_precompiled (= 0.5.0.1)

BUNDLED WITH
   2.5.0

Note [2]: rubygems/rubygems#7432 was fixed in 2.5.6 by rubygems/rubygems#7441

[flavorjones]

I've updated the recommendations in the description. I'll be opening a bug report on rubygems shortly to report these findings.

[flavorjones]

I've created an issue upstream at test: demonstrate issue with installing linux-musl gems by flavorjones · Pull Request #7583 · rubygems/rubygems for the issue described above in bundler 2.5.6.

[flavorjones]

Recommendations added to the README. Closing.

[ntkme]

Debian unstable really screwed it up: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1069089

Debian unstable's ruby3.1 package currently bundles rubygems 3.4.20. However, Debian's maintainers patched it in a way that behavior like 3.3.15:

diff --git a/lib/rubygems/platform.rb b/lib/rubygems/platform.rb
index b721629b7..a16c6fca9 100644
--- a/lib/rubygems/platform.rb
+++ b/lib/rubygems/platform.rb
@@ -117,6 +117,9 @@ def initialize(arch)
       when /^(\w+_platform)(\d+)?/ then [ $1,          $2  ]
       else                              [ "unknown",   nil ]
       end
+
+      # emulate behavior from 3.3.15 on ruby 3.1
+      @version = nil if (RUBY_VERSION < "3.2" && @os == "linux")
     when Gem::Platform then
       @cpu = arch.cpu
       @os = arch.os

It's completely broken despite 3.4.20 > 3.3.22...

irb(main):001:0> Gem::Platform.new('aarch64-linux-gnu')
=> #<Gem::Platform:0x0000ffffa56fc8e0 @cpu="aarch64", @os="linux", @version=nil>
irb(main):002:0> Gem::Platform.new('aarch64-linux-musl')
=> #<Gem::Platform:0x0000ffffa56c8680 @cpu="aarch64", @os="linux", @version=nil>
irb(main):003:0> Gem::Platform.new('aarch64-linux-musl') == Gem::Platform.new('aarch64-linux-gnu')
=> true

I have already got a few bug reports from users on Debian unstable. What I don't want to do is to block rubygems <= 3.4.20, because ruby3.2 on Debian unstable also have same patched rubygems 3.4.20, but works properly (see the version check in the patch above).

I have tried a few ways to reduce the noises from users by checking if they have a broken rubygems and raise a runtime error, etc. and unfortunately it brought even more noise and confusion. The final solution I end up with is to conditionally block ruby <3.2 on linux native gems, as ruby >=3.2 seems to always have a good default rubygems version (assuming users won't downgrade default rubygems).

  spec.required_ruby_version = if spec.platform == Gem::Platform::RUBY || spec.platform.os != 'linux'
                                 '>= 3.1.0'
                               else
                                 '>= 3.2.0'
                               end

[flavorjones]

@ntkme Thanks for adding this context, that's really too bad about the downstream patches that get applied in the Sid distro. I hope it gets fixed.