Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add OpenSSL Provider support #635

Merged
merged 2 commits into from Jun 18, 2023
Merged

Conversation

QWYNG
Copy link
Contributor

@QWYNG QWYNG commented Jun 2, 2023

Hi! Thank you for great gem.

This PR fix #567

Currently Ruby does not seem to have an API to load providers
To add a provider, you need to set the following file in OPENSSL_CONF

  ~ ruby -v
ruby 3.2.2 (2023-03-30 revision e51014f9c0) [arm64-darwin22]
  ~ irb
irb(main):001:0> require 'openssl'
=> true
irb(main):002:0> OpenSSL::VERSION
=> "3.1.0"
irb(main):003:0> OpenSSL::OPENSSL_VERSION
=> "OpenSSL 3.1.0 14 Mar 2023"
irb(main):004:0> OpenSSL::OPENSSL_LIBRARY_VERSION
=> "OpenSSL 3.1.0 14 Mar 2023"
irb(main):005:0> OpenSSL::Cipher.new("RC4")
(irb):5:in `initialize': unsupported (OpenSSL::Cipher::CipherError)
	from (irb):5:in `new'
	from (irb):5:in `<main>'
~~~
  ~ cat .add_provider_conf
openssl_conf = openssl_init

[openssl_init]
providers = provider_sect

[provider_sect]
default = default_sect
legacy = legacy_sect

[default_sect]
activate = 1

[legacy_sect]
activate = 1

  ~ irb
irb(main):001:0> ENV['OPENSSL_CONF'] = ".add_provider_conf"
=> ".add_provider_conf"
irb(main):002:0> require 'openssl'
=> true
# default provider does not support RC4 
irb(main):003:0> OpenSSL::Cipher.new("RC4")
=> #<OpenSSL::Cipher:0x0000000110e015d0>

This change provides an API to add a provider using OSSL_PROVIDER_load

OpenSSL::Provider.load("legacy")
=> #<OpenSSL::Provider name="legacy">
cipher = OpenSSL::Cipher.new("RC4")
=> #<OpenSSL::Cipher:0x0000000108eb3648>

I am just a beginner to Ruby C extensions. I would love to get a review!

ref:
https://www.openssl.org/docs/manmaster/man3/OSSL_PROVIDER_load.html

ext/openssl/ossl_provider.c Outdated Show resolved Hide resolved
@QWYNG QWYNG marked this pull request as ready for review June 2, 2023 13:10
@QWYNG QWYNG marked this pull request as draft June 2, 2023 13:14
@QWYNG QWYNG marked this pull request as ready for review June 2, 2023 13:50
@QWYNG QWYNG marked this pull request as draft June 2, 2023 13:51
@QWYNG QWYNG force-pushed the add_provider_support branch 2 times, most recently from 056a641 to efc2ed1 Compare June 2, 2023 14:06
@QWYNG QWYNG marked this pull request as ready for review June 2, 2023 14:11
Copy link
Member

@rhenium rhenium left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for working on this!

This is something we definitely should have.

ext/openssl/ossl.h Outdated Show resolved Hide resolved
ext/openssl/ossl_provider.h Outdated Show resolved Hide resolved
test/openssl/test_provider.rb Show resolved Hide resolved
ext/openssl/ossl_provider.c Outdated Show resolved Hide resolved
ext/openssl/ossl_provider.c Show resolved Hide resolved
ext/openssl/ossl_provider.c Outdated Show resolved Hide resolved
@QWYNG
Copy link
Contributor Author

QWYNG commented Jun 7, 2023

@rhenium
Thank you for your kind review!
I made some modifications based on your reviews.

ext/openssl/ossl_provider.c Show resolved Hide resolved
ext/openssl/ossl_provider.c Show resolved Hide resolved
test/openssl/test_provider.rb Show resolved Hide resolved
@rhenium
Copy link
Member

rhenium commented Jun 8, 2023

I wonder why CI is failing on Windows.

@QWYNG
Copy link
Contributor Author

QWYNG commented Jun 11, 2023

@rhenium

I wonder why CI is failing on Windows.

With Windows and Ruby over 3.2, OpenSSL 3 will be used.
It seems that OpenSSL3 on Windows has the legacy.dll installed in a different location than the default search path.

refs:
openssl/openssl#20847 (comment)
https://github.com/ruby/setup-msys2-gcc/releases
https://github.com/QWYNG/openssl/actions/runs/5238084041

@MSP-Greg
Copy link
Contributor

I've got msys2 and vcpkg tools on my systems, and they have legacy.dll in different locations. These dll's are bundled with Windows Rubies. Where should they be located? Or, what is the 'default search path'?

@MSP-Greg
Copy link
Contributor

@QWYNG & @rhenium

I've got Windows ucrt fixed locally, the mingw & ucrt builds are being built for GitHub Actions, should be about 40 minutes, as the full ruby/ruby test suite is run.

The mswin build isn't done yet, that may take till tomorrow.

Where the legacy.dll file should be located is in bin/lib/ossl-modules. I also placed the engines-3 folder in bin.

This is due to MSYS2 patching OpenSSL to assign paths for misc OpenSSL folders relative to where the main OpenSSL dll's are located, which is in bin/ruby_builtin_dlls...

Pinging @larskanis for Ruby 3.2.

@MSP-Greg
Copy link
Contributor

@QWYNG & @rhenium

The Windows head builds (mingw, ucrt, & mswin) have been fixed. I patched test.yml in my fork with needed changes.

See Actions here, the patch is here. It removes the ENV['OPENSSL_MODULES'] settings for head builds and sets it for the (yet to be updated) MSYS2 3.2 build.

Note that the existing Actions steps (or something equivalent to them) will probably be needed in ruby/ruby,..

@QWYNG
Copy link
Contributor Author

QWYNG commented Jun 12, 2023

@MSP-Greg
Thank you! I cherry-picked the commit.

@@ -130,7 +136,7 @@ jobs:
run: echo "OPENSSL_CONF=$(pwd)/test/openssl/fixtures/ssl/openssl_fips.cnf" >> $GITHUB_ENV
if: matrix.fips-enabled

- name: set fips enviornment variable for testing.
- name: set fips environment variable for testing.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for finding and fixing a typo. However, I don't think this is related to this PR's main topic. Maybe better to send another PR for that.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I opened the PR #641 for that.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm...
Should I change the commit?

Copy link
Member

@junaruga junaruga Jun 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. I think so. I already merged the commit only fixing the typo. So, I think after you will finish the review and fix everything in this PR, I think you (or a maintainer) needs to squash the commits in this PR into 1 commit on the latest master branch. We call it rebasing the PR on master branch. Could you do that?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The typo fix came from my commit, which modified the Actions workflow file, which isn't part of the gem, or copied to ruby/ruby, etc.

Fixing a pair of transposed comment characters in a file that is already being modified seemed trivial...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you squash the commits in this PR to the 1 commit, it's nice to have the following line in the bottom of the commit message. You can check the examples in the past commits by git log on the master branch.

Co-authored-by: MSP-Greg <Greg.mpls@gmail.com>

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's often nice to see changes as the PR 'evolves'. Maintainers can always use the 'Squash and Merge' option.

Squash_and_Merge_sm

@rhenium
Copy link
Member

rhenium commented Jun 12, 2023

@MSP-Greg @QWYNG Thank you for debugging and fixing it!

@MSP-Greg
Copy link
Contributor

Is there a constant that can be added, similar to OpenSSL::X509::DEFAULT_CERT_DIR, that holds the 'default' location of OPENSSL_MODULES?

These locations are coded in the OpenSSL files, and may be patched by the build system (as in MSYS2, vcpkg, etc). MSYS2 does so and they have changed the locations in the past. I think the current locations are unlikely to change.

ext/openssl/ossl_provider.c Outdated Show resolved Hide resolved
ext/openssl/ossl_provider.c Outdated Show resolved Hide resolved
ext/openssl/ossl_provider.c Show resolved Hide resolved
ext/openssl/ossl_provider.c Show resolved Hide resolved
ext/openssl/ossl_provider.c Outdated Show resolved Hide resolved
@QWYNG
Copy link
Contributor Author

QWYNG commented Jun 13, 2023

@MSP-Greg

Is there a constant that can be added, similar to OpenSSL::X509::DEFAULT_CERT_DIR, that holds the 'default' location of OPENSSL_MODULES?

There seems to be an API in OpenSSL master to get the default load path, but it does not seem to exist in 3.1 or 3.0

https://github.com/openssl/openssl/blob/050dddb06162a8016c004317273f8f01b72ac20a/include/openssl/provider.h#LL22C13-L22C51

https://github.com/openssl/openssl/blob/openssl-3.1/include/openssl/provider.h
https://github.com/openssl/openssl/blob/openssl-3.0/include/openssl/provider.h

openssl/openssl#19752 (comment)

ext/openssl/ossl_provider.c Outdated Show resolved Hide resolved
ext/openssl/ossl_provider.c Outdated Show resolved Hide resolved
ext/openssl/ossl_provider.c Outdated Show resolved Hide resolved
ext/openssl/ossl_provider.c Outdated Show resolved Hide resolved
Co-authored-by: MSP-Greg <Greg.mpls@gmail.com>
Copy link
Member

@rhenium rhenium left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is ready! Thanks so much for your patience.

@rhenium
Copy link
Member

rhenium commented Jun 18, 2023

Let me squash commits and rebase on top of the current master, which has since then updated .github/workflows/test.yaml.

@rhenium rhenium merged commit 97fb410 into ruby:master Jun 18, 2023
40 checks passed
@rhenium
Copy link
Member

rhenium commented Jun 18, 2023

Merged now. Thank you so much!

@QWYNG
Copy link
Contributor Author

QWYNG commented Jun 19, 2023

Thank you, too!

@hsbt
Copy link
Member

hsbt commented Jun 19, 2023

@rhenium @QWYNG mswin and vs2019 environment is broken at ruby/ruby repository.

https://github.com/ruby/ruby/actions/runs/5309762723/jobs/9610845792#step:21:951

Can you look this?

@rhenium
Copy link
Member

rhenium commented Jun 19, 2023

Thanks for informing me. I will look into it. It's odd, though, because the same workflow passed on my rhenium/ruby fork, with both vs2019 and vs2022.

https://github.com/rhenium/ruby/actions/runs/5304896653/jobs/9601511128

@QWYNG
Copy link
Contributor Author

QWYNG commented Jun 19, 2023

@rhenium @hsbt
I am not sure, but I suspect it is a cache issue. @MSP-Greg has updated the Windows build regarding this PR.

#635 (comment)

@rhenium
Copy link
Member

rhenium commented Jun 19, 2023

They are using different versions of OpenSSL from vcpkg. I wasn't able to pinpoint the change from https://github.com/microsoft/vcpkg/commits/master/ports/openssl, but I suppose something fixed the installation paths for providers.

As an easy workaround, I will try clearing the vcpkg caches. Perhaps should the workflow attempt to update the packages?

On ruby/ruby:

2023-06-19T08:47:35.7591300Z ##[group]Run actions/cache@88522ab9f39a2ea568f7027eddc7d8d8bc9d59c8
2023-06-19T08:47:35.7591832Z with:
2023-06-19T08:47:35.7592120Z   path: C:\vcpkg\downloads
2023-06-19T08:47:35.7592633Z   key: Windows-vcpkg-download-windows-2019-e9f62fa5a762d367e3dc1818d29450ee8e9e25ab
2023-06-19T08:47:35.7593340Z   restore-keys: Windows-vcpkg-download-windows-2019-
Windows-vcpkg-download-

2023-06-19T08:47:35.7593815Z   enableCrossOsArchive: false
2023-06-19T08:47:35.7594139Z   fail-on-cache-miss: false
2023-06-19T08:47:35.7594441Z   lookup-only: false
2023-06-19T08:47:35.7594708Z env:
2023-06-19T08:47:35.7595029Z   GITPULLOPTIONS: --no-tags origin refs/heads/master
2023-06-19T08:47:35.7595402Z   PATCH: C:\msys64\usr\bin\patch.exe
2023-06-19T08:47:35.7595710Z   OS_VER: windows-2019
2023-06-19T08:47:35.7595959Z ##[endgroup]
2023-06-19T08:47:38.6993771Z Received 130023424 of 304196632 (42.7%), 123.8 MBs/sec
2023-06-19T08:47:40.4055229Z Received 192937984 of 304196632 (63.4%), 91.3 MBs/sec
2023-06-19T08:47:41.4144097Z Received 268435456 of 304196632 (88.2%), 83.0 MBs/sec
2023-06-19T08:47:41.4145022Z Received 304196632 of 304196632 (100.0%), 81.7 MBs/sec
2023-06-19T08:47:41.4145693Z Cache Size: ~290 MB (304196632 B)
2023-06-19T08:47:41.4151813Z [command]"C:\Program Files\Git\usr\bin\tar.exe" -xf D:/a/_temp/ba77890b-100d-4182-8cff-e9e870266984/cache.tzst -P -C D:/a/ruby/ruby --force-local --use-compress-program "zstd -d"
2023-06-19T08:49:22.2375820Z Cache restored successfully
2023-06-19T08:49:22.3153150Z Cache restored from key: Windows-vcpkg-download-windows-2019-e7440dcaeae68de421356e22c9126b4325ad7bff
2023-06-19T08:49:22.3602226Z ##[group]Run actions/cache@88522ab9f39a2ea568f7027eddc7d8d8bc9d59c8
2023-06-19T08:49:22.3602675Z with:
2023-06-19T08:49:22.3602938Z   path: C:\vcpkg\installed
2023-06-19T08:49:22.3603405Z   key: Windows-vcpkg-installed--e9f62fa5a762d367e3dc1818d29450ee8e9e25ab
2023-06-19T08:49:22.3603978Z   restore-keys: Windows-vcpkg-installed--
Windows-vcpkg-installed-

2023-06-19T08:49:22.3604411Z   enableCrossOsArchive: false
2023-06-19T08:49:22.3604727Z   fail-on-cache-miss: false
2023-06-19T08:49:22.3605018Z   lookup-only: false
2023-06-19T08:49:22.3605273Z env:
2023-06-19T08:49:22.3605577Z   GITPULLOPTIONS: --no-tags origin refs/heads/master
2023-06-19T08:49:22.3605942Z   PATCH: C:\msys64\usr\bin\patch.exe
2023-06-19T08:49:22.3606243Z   OS_VER: windows-2019
2023-06-19T08:49:22.3606503Z ##[endgroup]
2023-06-19T08:49:23.1338919Z Received 19537507 of 19537507 (100.0%), 62.1 MBs/sec
2023-06-19T08:49:23.1364889Z Cache Size: ~19 MB (19537507 B)
2023-06-19T08:49:23.1398506Z [command]"C:\Program Files\Git\usr\bin\tar.exe" -xf D:/a/_temp/801c4ef7-da72-470e-b132-47e81234e3a8/cache.tzst -P -C D:/a/ruby/ruby --force-local --use-compress-program "zstd -d"
2023-06-19T08:49:26.1580806Z Cache restored successfully
2023-06-19T08:49:26.1819381Z Cache restored from key: Windows-vcpkg-installed--e7440dcaeae68de421356e22c9126b4325ad7bff
2023-06-19T08:49:26.2144628Z ##[group]Run vcpkg --triplet x64-windows install libffi libyaml openssl readline zlib
2023-06-19T08:49:26.2145346Z �[36;1mvcpkg --triplet x64-windows install libffi libyaml openssl readline zlib�[0m
2023-06-19T08:49:26.2184499Z shell: C:\Windows\system32\cmd.EXE /D /E:ON /V:OFF /S /C "CALL "{0}""
2023-06-19T08:49:26.2184848Z env:
2023-06-19T08:49:26.2185189Z   GITPULLOPTIONS: --no-tags origin refs/heads/master
2023-06-19T08:49:26.2185576Z   PATCH: C:\msys64\usr\bin\patch.exe
2023-06-19T08:49:26.2185893Z   OS_VER: windows-2019
2023-06-19T08:49:26.2186184Z ##[endgroup]
2023-06-19T08:49:28.8242778Z Computing installation plan...
2023-06-19T08:49:33.3700005Z The following packages are already installed:
2023-06-19T08:49:33.3700456Z     libffi:x64-windows -> 3.4.4
2023-06-19T08:49:33.3700906Z     libyaml:x64-windows -> 0.2.5#3
2023-06-19T08:49:33.3701245Z     openssl:x64-windows -> 3.0.8
2023-06-19T08:49:33.3701583Z     readline:x64-windows -> 0#3
2023-06-19T08:49:33.3702020Z     zlib:x64-windows -> 1.2.13
2023-06-19T08:49:33.4108578Z libffi:x64-windows is already installed
2023-06-19T08:49:33.4111697Z libyaml:x64-windows is already installed
2023-06-19T08:49:33.4112574Z openssl:x64-windows is already installed
2023-06-19T08:49:33.4112967Z readline:x64-windows is already installed
2023-06-19T08:49:33.4113350Z zlib:x64-windows is already installed
2023-06-19T08:49:33.4121486Z Total install time: 10.2 ms
[...]

On rhenium/ruby:

2023-06-18T17:36:17.1884671Z The following packages will be built and installed:
2023-06-18T17:36:17.1885001Z     libffi:x64-windows -> 3.4.4#1
2023-06-18T17:36:17.1885253Z     libyaml:x64-windows -> 0.2.5#4
2023-06-18T17:36:17.1885510Z     openssl:x64-windows -> 3.1.1
2023-06-18T17:36:17.1885744Z     readline:x64-windows -> 0#4
2023-06-18T17:36:17.1886102Z   * readline-win32:x64-windows -> 5.0#5
2023-06-18T17:36:17.1886372Z   * vcpkg-cmake:x64-windows -> 2023-05-04
2023-06-18T17:36:17.1886676Z   * vcpkg-cmake-config:x64-windows -> 2022-02-06#1
2023-06-18T17:36:17.1887001Z   * vcpkg-cmake-get-vars:x64-windows -> 2023-03-02
2023-06-18T17:36:17.1887286Z     zlib:x64-windows -> 1.2.13

@MSP-Greg
Copy link
Contributor

MSP-Greg commented Jun 19, 2023

@rhenium

I suspect that ruby/ruby will need ENV['OPENSSL_MODULES'] set for the mswin CI.

MSYS2 has a patch in their OpenSSL package ('relocation.patch') that makes several (all?) of the OpenSSL default file system locations relative to the OpenSSL dll's.

MSFT vcpkg does not have an equivalent patch, and the values are hard references to various directories.

MSFT vcpkg is broken right now. See microsoft/vcpkg#31908.

@rhenium
Copy link
Member

rhenium commented Jun 19, 2023

Yes, that appears to be the root cause. I installed OpenSSL with vcpkg install openssl[tools], and running C:\vcpkg\installed\x64-windows\tools\openssl\openssl info -modulesdir shows C:\vcpkg\packages\openssl_x64-windows\bin. This directory is not preserved by ruby/ruby's workflow (and is not supposed to be, IIUC).

@QWYNG QWYNG deleted the add_provider_support branch July 3, 2023 06:47
@carolregalin
Copy link

OpenSSL::Provider not loading for me.

Ruby version: 3.2.2
Openssl gem version: 3.2.0.
Operational system: Ubuntu.

001 > require "openssl"
=> true
002 > OpenSSL::VERSION
=> "3.2.0"
003 > OpenSSL::Provider.load("legacy")
uninitialized constant OpenSSL::Provider (NameError)

Any help, please?

@junaruga
Copy link
Member

junaruga commented Oct 9, 2023

What is your used OpenSSL version (OpenSSL::OPENSSL_VERSION and OpenSSL::OPENSSL_LIBRARY_VERSION)? I think the provider feature is available in OpenSSL 3.0 and later versions.

#if OSSL_OPENSSL_PREREQ(3, 0, 0)
# define OSSL_USE_PROVIDER
#endif

@tanna88
Copy link

tanna88 commented Dec 14, 2023

OpenSSL::Provider not loading for me.

Ruby version: 3.2.2 Openssl gem version: 3.2.0. Operational system: Ubuntu.

001 > require "openssl" => true 002 > OpenSSL::VERSION => "3.2.0" 003 > OpenSSL::Provider.load("legacy") uninitialized constant OpenSSL::Provider (NameError)

Any help, please?

have you solved the problem @junaruga ? I get the same problem with you

@tanna88
Copy link

tanna88 commented Dec 14, 2023

It works now. we need to make sure that we use correct native OpenSSL version

@junaruga
Copy link
Member

The provider feature only works with OpenSSL 3 or later versions. Perhaps the error message in OpenSSL 1.0/1.1 can be better.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
7 participants