password decryption doesn't fail 0.4% of the time while given the wrong password (depends on salt)

[lacostej]

In the @fastlane/fastlane (ruby) project, we use openssl password based encryption to encrypt the files we fetch from the Apple Developer portal. Under the hood, the ruby code generates an IV and Key using EVP_BytesToKey and generates a random salt.

If the wrong password is used, the code is supposed to fail with the bad decrypt error.

We found out that in about 0.4% of the time, the decryption doesn't detect that the wrong password was entered. This depends on the salt that was generated.

We have a simple ruby code demonstrating the issue.

We've converted it to C and it appears to fail in the same way. I haven't written C in 20 years so I hope I got this right :)

Is there an issue in openssl?

The ruby code was tested on various systems, from Mac to Linux.

The C code was tested on my Mac M1 Pro against openssl 3.2.0