Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ruby]: add pre-compiled binaries for ruby 3.2; drop them for ruby 2.6 #32089

Merged
merged 41 commits into from
Jan 19, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
6f8256c
upgrade rake compiler docker
apolcyn Jan 12, 2023
c114501
upgrade docker images
apolcyn Jan 12, 2023
ad1d285
Upgrade Rakefile to add 3.2 and drop 2.6; remove code for building pa…
apolcyn Jan 12, 2023
0435754
update docker image versions
apolcyn Jan 12, 2023
69a4f23
remove ruby 2.6 distrib test
apolcyn Jan 12, 2023
b57158b
update sha256
apolcyn Jan 12, 2023
1accf47
update rake compiler
apolcyn Jan 12, 2023
983a1d7
debug statements
apolcyn Jan 13, 2023
040b9c7
temporary pin to 1.1.7
apolcyn Jan 13, 2023
74878a2
try set no_native = false
apolcyn Jan 13, 2023
72642bb
debug prints
apolcyn Jan 13, 2023
af3796f
dep changes
apolcyn Jan 17, 2023
56f1c4a
remove explicit installation of bundler
apolcyn Jan 17, 2023
eca241e
remove experiment
apolcyn Jan 17, 2023
7936a08
remove bundle install hack
apolcyn Jan 17, 2023
98fe9b0
Merge remote-tracking branch 'origin/master' into ruby_32_support
apolcyn Jan 17, 2023
0b18fb1
add debug logs
apolcyn Jan 17, 2023
614fe51
try stripping on dbg build too
apolcyn Jan 17, 2023
69aacbf
stop stripping output entirely
apolcyn Jan 17, 2023
80a432f
debug prints
apolcyn Jan 17, 2023
8f0e7d4
add more logs
apolcyn Jan 18, 2023
9f302db
try bumping MACOSX_DEPLOYMENT_TARGET to 11.7
apolcyn Jan 18, 2023
712a171
Revert "try bumping MACOSX_DEPLOYMENT_TARGET to 11.7"
apolcyn Jan 18, 2023
aae1f9b
try using ruby 3.1 instead of 3.2 on macos unit tests
apolcyn Jan 18, 2023
2979590
stop using export files as experiment
apolcyn Jan 18, 2023
8e90fc9
log generated makefile
apolcyn Jan 18, 2023
e5ed905
Try running on ruby 2.7.8
apolcyn Jan 18, 2023
934b0f8
improve logs; see if ruby 2.7 works
apolcyn Jan 19, 2023
6ee214c
try disabling chained fixups
apolcyn Jan 19, 2023
1b353ab
Add -wl
apolcyn Jan 19, 2023
09041c6
also check other file
apolcyn Jan 19, 2023
a02fa10
more logs
apolcyn Jan 19, 2023
ddf187f
add more logging
apolcyn Jan 19, 2023
158a627
Fix log
apolcyn Jan 19, 2023
362882a
Try using copy in rakefile
apolcyn Jan 19, 2023
7eb96c8
add hack
apolcyn Jan 19, 2023
b95f0a7
set workaround for macos
apolcyn Jan 19, 2023
183fd0a
remove debug logs
apolcyn Jan 19, 2023
b7b7e14
revert unnecessary change
apolcyn Jan 19, 2023
7e5ab91
updates
apolcyn Jan 19, 2023
63424f6
remove log
apolcyn Jan 19, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
138 changes: 61 additions & 77 deletions Rakefile
Original file line number Diff line number Diff line change
Expand Up @@ -144,91 +144,75 @@ task 'gem:native', [:plat] do |t, args|
verbose = ENV['V'] || '0'

grpc_config = ENV['GRPC_CONFIG'] || 'opt'
ruby_cc_versions = ['3.1.0', '3.0.0', '2.7.0', '2.6.0'].join(':')
ruby_cc_versions = ['3.2.0', '3.1.0', '3.0.0', '2.7.0'].join(':')
selected_plat = "#{args[:plat]}"

if RUBY_PLATFORM =~ /darwin/
if !selected_plat.empty? && selected_plat != 'darwin'
fail "Cannot pass platform as an argument when on Darwin."
end
# use env variable to set artifact build paralellism
nproc_override = ENV['GRPC_RUBY_BUILD_PROCS'] || `nproc`.strip

FileUtils.touch 'grpc_c.32-msvcrt.ruby'
FileUtils.touch 'grpc_c.64-msvcrt.ruby'
FileUtils.touch 'grpc_c.64-ucrt.ruby'
unless '2.5' == /(\d+\.\d+)/.match(RUBY_VERSION).to_s
fail "rake gem:native (the rake task to build the binary packages) is being " \
"invoked on macos with ruby #{RUBY_VERSION}. The ruby macos artifact " \
"build should be running on ruby 2.5."
end
system "bundle exec rake cross native gem RUBY_CC_VERSION=#{ruby_cc_versions} V=#{verbose} GRPC_CONFIG=#{grpc_config}"
# propagate env variables with ccache configuration to the rake-compiler-dock docker container
# and setup ccache symlinks as needed.
prepare_ccache_cmd = "export GRPC_BUILD_ENABLE_CCACHE=\"#{ENV.fetch('GRPC_BUILD_ENABLE_CCACHE', '')}\" && "
prepare_ccache_cmd += "export CCACHE_SECONDARY_STORAGE=\"#{ENV.fetch('CCACHE_SECONDARY_STORAGE', '')}\" && "
prepare_ccache_cmd += "export PATH=\"$PATH:/usr/local/bin\" && "
prepare_ccache_cmd += "source tools/internal_ci/helper_scripts/prepare_ccache_symlinks_rc "

supported_windows_platforms = ['x86-mingw32', 'x64-mingw32', 'x64-mingw-ucrt']
supported_unix_platforms = ['x86_64-linux', 'x86-linux', 'x86_64-darwin', 'arm64-darwin']
supported_platforms = supported_windows_platforms + supported_unix_platforms

if selected_plat.empty?
# build everything
windows_platforms = supported_windows_platforms
unix_platforms = supported_unix_platforms
else
# use env variable to set artifact build paralellism
nproc_override = ENV['GRPC_RUBY_BUILD_PROCS'] || `nproc`.strip

# propagate env variables with ccache configuration to the rake-compiler-dock docker container
# and setup ccache symlinks as needed.
prepare_ccache_cmd = "export GRPC_BUILD_ENABLE_CCACHE=\"#{ENV.fetch('GRPC_BUILD_ENABLE_CCACHE', '')}\" && "
prepare_ccache_cmd += "export CCACHE_SECONDARY_STORAGE=\"#{ENV.fetch('CCACHE_SECONDARY_STORAGE', '')}\" && "
prepare_ccache_cmd += "export PATH=\"$PATH:/usr/local/bin\" && "
prepare_ccache_cmd += "source tools/internal_ci/helper_scripts/prepare_ccache_symlinks_rc "

supported_windows_platforms = ['x86-mingw32', 'x64-mingw32', 'x64-mingw-ucrt']
supported_unix_platforms = ['x86_64-linux', 'x86-linux', 'x86_64-darwin', 'arm64-darwin']
supported_platforms = supported_windows_platforms + supported_unix_platforms

if selected_plat.empty?
# build everything
windows_platforms = supported_windows_platforms
unix_platforms = supported_unix_platforms
# build only selected platform
if supported_windows_platforms.include?(selected_plat)
windows_platforms = [selected_plat]
unix_platforms = []
elsif supported_unix_platforms.include?(selected_plat)
windows_platforms = []
unix_platforms = [selected_plat]
else
# build only selected platform
if supported_windows_platforms.include?(selected_plat)
windows_platforms = [selected_plat]
unix_platforms = []
elsif supported_unix_platforms.include?(selected_plat)
windows_platforms = []
unix_platforms = [selected_plat]
else
fail "Unsupported platform '#{selected_plat}' passed as an argument."
end
fail "Unsupported platform '#{selected_plat}' passed as an argument."
end
end

# Create the windows dlls or create the empty placeholders
Rake::Task['dlls'].execute(plat: windows_platforms)

windows_platforms.each do |plat|
run_rake_compiler(plat, <<~EOT)
#{prepare_ccache_cmd} && \
gem update --system --no-document && \
bundle && \
bundle exec rake clean && \
bundle exec rake native:#{plat} pkg/#{spec.full_name}-#{plat}.gem pkg/#{spec.full_name}.gem \
RUBY_CC_VERSION=#{ruby_cc_versions} \
V=#{verbose} \
GRPC_CONFIG=#{grpc_config} \
GRPC_RUBY_BUILD_PROCS=#{nproc_override}
EOT
end
# Create the windows dlls or create the empty placeholders
Rake::Task['dlls'].execute(plat: windows_platforms)

# Truncate grpc_c.*.ruby files because they're for Windows only and we don't want
# them to take up space in the gems that don't target windows.
File.truncate('grpc_c.32-msvcrt.ruby', 0)
File.truncate('grpc_c.64-msvcrt.ruby', 0)
File.truncate('grpc_c.64-ucrt.ruby', 0)

unix_platforms.each do |plat|
run_rake_compiler(plat, <<~EOT)
#{prepare_ccache_cmd} && \
gem update --system --no-document && \
bundle && \
bundle exec rake clean && \
bundle exec rake native:#{plat} pkg/#{spec.full_name}-#{plat}.gem pkg/#{spec.full_name}.gem \
RUBY_CC_VERSION=#{ruby_cc_versions} \
V=#{verbose} \
GRPC_CONFIG=#{grpc_config} \
GRPC_RUBY_BUILD_PROCS=#{nproc_override}
EOT
end
windows_platforms.each do |plat|
run_rake_compiler(plat, <<~EOT)
#{prepare_ccache_cmd} && \
gem update --system --no-document && \
bundle && \
bundle exec rake clean && \
bundle exec rake native:#{plat} pkg/#{spec.full_name}-#{plat}.gem pkg/#{spec.full_name}.gem \
RUBY_CC_VERSION=#{ruby_cc_versions} \
V=#{verbose} \
GRPC_CONFIG=#{grpc_config} \
GRPC_RUBY_BUILD_PROCS=#{nproc_override}
EOT
end

# Truncate grpc_c.*.ruby files because they're for Windows only and we don't want
# them to take up space in the gems that don't target windows.
File.truncate('grpc_c.32-msvcrt.ruby', 0)
File.truncate('grpc_c.64-msvcrt.ruby', 0)
File.truncate('grpc_c.64-ucrt.ruby', 0)

unix_platforms.each do |plat|
run_rake_compiler(plat, <<~EOT)
#{prepare_ccache_cmd} && \
gem update --system --no-document && \
bundle && \
bundle exec rake clean && \
bundle exec rake native:#{plat} pkg/#{spec.full_name}-#{plat}.gem pkg/#{spec.full_name}.gem \
RUBY_CC_VERSION=#{ruby_cc_versions} \
V=#{verbose} \
GRPC_CONFIG=#{grpc_config} \
GRPC_RUBY_BUILD_PROCS=#{nproc_override}
EOT
end
end

Expand Down
4 changes: 2 additions & 2 deletions grpc.gemspec

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

19 changes: 19 additions & 0 deletions src/ruby/ext/grpc/extconf.rb
Original file line number Diff line number Diff line change
Expand Up @@ -187,3 +187,22 @@ def ext_export_filename()
end
File.rename('Makefile.new', 'Makefile')
end
if ENV['GRPC_RUBY_TEST_ONLY_WORKAROUND_MAKE_INSTALL_BUG']
# Note: this env var setting is intended to work around a problem observed
# with the ginstall command on grpc's macos automated test infrastructure,
# and is not guaranteed to work in the wild.
# Also see https://github.com/rake-compiler/rake-compiler/issues/210.
puts 'Overriding the generated Makefile install target to use cp'
File.open('Makefile.new', 'w') do |o|
File.foreach('Makefile') do |i|
if i.start_with?('INSTALL_PROG = ')
override = 'INSTALL_PROG = cp'
puts "Replacing generated Makefile line: |#{i}|, with: |#{override}|"
o.puts override
else
o.puts i
end
end
end
File.rename('Makefile.new', 'Makefile')
end
4 changes: 2 additions & 2 deletions templates/grpc.gemspec.template
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,8 @@
s.add_development_dependency 'logging', '~> 2.0'
s.add_development_dependency 'simplecov', '~> 0.22'
s.add_development_dependency 'rake', '~> 13.0'
s.add_development_dependency 'rake-compiler', '<= 1.1.1'
s.add_development_dependency 'rake-compiler-dock', '~> 1.2'
s.add_development_dependency 'rake-compiler', '~> 1.2.1'
s.add_development_dependency 'rake-compiler-dock', '~> 1.3'
s.add_development_dependency 'rspec', '~> 3.6'
s.add_development_dependency 'rubocop', '~> 1.41.0'
s.add_development_dependency 'signet', '~> 0.7'
Expand Down
Original file line number Diff line number Diff line change
@@ -1 +1 @@
us-docker.pkg.dev/grpc-testing/testing-images-public/rake_arm64-darwin:69742889e98f7c0dc2d26cc89a43017179515167@sha256:d1b0c35c651a1911ba290184f63c5e81b17fd20291acea20c91e9f2c2d7c8d50
us-docker.pkg.dev/grpc-testing/testing-images-public/rake_arm64-darwin:ef464134d73d8d261b7edb53b737c2dc797f7cee@sha256:e0eb1f9f632fb18d4f244b7297d1a5e7cf60ae58e649ac5b2f8ac6266ea07128
Original file line number Diff line number Diff line change
@@ -1 +1 @@
FROM larskanis/rake-compiler-dock-mri-arm64-darwin:1.2.2
FROM ghcr.io/rake-compiler/rake-compiler-dock-image:1.3.0-mri-arm64-darwin
Original file line number Diff line number Diff line change
@@ -1 +1 @@
us-docker.pkg.dev/grpc-testing/testing-images-public/rake_x64-mingw-ucrt:0b8bed3e125d8952ccc8938f905d9792c84077f5@sha256:ab967122a47b0385a1e66b5b91693d971468a276000ac1c066c54bef577ef4b7
us-docker.pkg.dev/grpc-testing/testing-images-public/rake_x64-mingw-ucrt:3c2e3bdaf402bb1dd329309b8682890cd3cca19f@sha256:63490b0000c6011a19983fef637efc69a2ae0f67b7a4e29cd36db53c881e908d
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM larskanis/rake-compiler-dock-mri-x64-mingw-ucrt:1.2.2
FROM ghcr.io/rake-compiler/rake-compiler-dock-image:1.3.0-mri-x64-mingw-ucrt

RUN find / -name win32.h | while read f ; do sed -i 's/gettimeofday/rb_gettimeofday/' $f ; done

Expand Down
Original file line number Diff line number Diff line change
@@ -1 +1 @@
us-docker.pkg.dev/grpc-testing/testing-images-public/rake_x64-mingw32:47e598526ac5068d348e29237e1dc84e3d01afab@sha256:6f4386f2954821c199d104e9dfe250ef99fdb74f4840311f8a7812c462629a83
us-docker.pkg.dev/grpc-testing/testing-images-public/rake_x64-mingw32:f24841cbe57e448e0cc98498a0a7d9467f91627b@sha256:63ece6e9b336b7cbf66eaa0201505b0579ac06cd7802f19b44c3a816d5617c17
2 changes: 1 addition & 1 deletion third_party/rake-compiler-dock/rake_x64-mingw32/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM larskanis/rake-compiler-dock-mri-x64-mingw32:1.2.2
FROM ghcr.io/rake-compiler/rake-compiler-dock-image:1.3.0-mri-x64-mingw32

RUN find / -name win32.h | while read f ; do sed -i 's/gettimeofday/rb_gettimeofday/' $f ; done

Expand Down
Original file line number Diff line number Diff line change
@@ -1 +1 @@
us-docker.pkg.dev/grpc-testing/testing-images-public/rake_x86-linux:e8ffbe97d21d829ae38ab10ef65cf38e542ee8b9@sha256:dab334b2c651e78f82c9ff7141699ac23365676f3972cff07382f7146702b88f
us-docker.pkg.dev/grpc-testing/testing-images-public/rake_x86-linux:921974cb95c4ca19ef79be0745df716a9bbf1211@sha256:71e3afca0843bf7bd5da7fa04bff40ad976e76aa5867936166b30251d0a692d8
2 changes: 1 addition & 1 deletion third_party/rake-compiler-dock/rake_x86-linux/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM larskanis/rake-compiler-dock-mri-x86-linux:1.2.2
FROM ghcr.io/rake-compiler/rake-compiler-dock-image:1.3.0-mri-x86-linux

#=================
# Install ccache
Expand Down
Original file line number Diff line number Diff line change
@@ -1 +1 @@
us-docker.pkg.dev/grpc-testing/testing-images-public/rake_x86-mingw32:91c4f180dce2a3f4fc9bcf66dc0818f0d8c81415@sha256:09b3f47c5eb8f6b686af4d6cba636637ab310ef1dcfa30e05bb00b183b1b619b
us-docker.pkg.dev/grpc-testing/testing-images-public/rake_x86-mingw32:bf745e7ec6ab838e4f982e25f086d6c8667f71cc@sha256:629be8f57e2d50b123584f2cfa00ff5b968cc2cc3b2a6b874acd07100a3eb96d
2 changes: 1 addition & 1 deletion third_party/rake-compiler-dock/rake_x86-mingw32/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM larskanis/rake-compiler-dock-mri-x86-mingw32:1.2.2
FROM ghcr.io/rake-compiler/rake-compiler-dock-image:1.3.0-mri-x86-mingw32

RUN find / -name win32.h | while read f ; do sed -i 's/gettimeofday/rb_gettimeofday/' $f ; done

Expand Down
Original file line number Diff line number Diff line change
@@ -1 +1 @@
us-docker.pkg.dev/grpc-testing/testing-images-public/rake_x86_64-darwin:56f7dd2c720f4c10a6d4be6a869b11ec1e39a0f3@sha256:b086e560ef215180b843d1d7fe48e2cfbed3112ec4ab57e655ad5dfb61090726
us-docker.pkg.dev/grpc-testing/testing-images-public/rake_x86_64-darwin:03fc718cb9cc9a32b16f1ed270b12ba77b862a05@sha256:8dd11cad778d9fc01c3555a57254016f5db7227309d24f50a192a6db80d4a51c
Original file line number Diff line number Diff line change
@@ -1 +1 @@
FROM larskanis/rake-compiler-dock-mri-x86_64-darwin:1.2.2
FROM ghcr.io/rake-compiler/rake-compiler-dock-image:1.3.0-mri-x86_64-darwin
Original file line number Diff line number Diff line change
@@ -1 +1 @@
us-docker.pkg.dev/grpc-testing/testing-images-public/rake_x86_64-linux:47c34d44a509c824ae84aaff82c14de5aa3bdc12@sha256:5c485b0a0e6bf5e0fc87122b3d49003a0d0aa30cbaca7250902f53823e50bc9c
us-docker.pkg.dev/grpc-testing/testing-images-public/rake_x86_64-linux:73e8372e1b5c1c414a5a8c6b37e76ed0d60a26cc@sha256:9aa77587fa4d4c25c91d0ccca8eb806cf0738a4b67eb4b54d40324185658194e
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM larskanis/rake-compiler-dock-mri-x86_64-linux:1.2.2
FROM ghcr.io/rake-compiler/rake-compiler-dock-image:1.3.0-mri-x86_64-linux

#=================
# Install ccache
Expand Down

This file was deleted.

This file was deleted.

6 changes: 0 additions & 6 deletions tools/run_tests/artifacts/distribtest_targets.py
Original file line number Diff line number Diff line change
Expand Up @@ -431,12 +431,6 @@ def targets():
PythonDistribTest('linux', 'x64', 'arch', source=True),
PythonDistribTest('linux', 'x64', 'ubuntu2004', source=True),
# Ruby
RubyDistribTest('linux',
'x64',
'stretch',
ruby_version='ruby_2_6',
source=True,
presubmit=True),
RubyDistribTest('linux',
'x64',
'stretch',
Expand Down
6 changes: 6 additions & 0 deletions tools/run_tests/helper_scripts/build_ruby.sh
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,12 @@ fi
cd "$(dirname "$0")/../../.."

rm -rf ./tmp

SYSTEM=$(uname | cut -f 1 -d_)
if [ "$SYSTEM" == "Darwin" ]; then
# work around https://github.com/rake-compiler/rake-compiler/issues/210
export GRPC_RUBY_TEST_ONLY_WORKAROUND_MAKE_INSTALL_BUG=true
fi
bundle exec rake compile

# build grpc_ruby_plugin
Expand Down