[release/6.0] Fix RSA OAEP decryption in Android with non-power-of-two key lengths #71696
+34
−1
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Customer Impact
A customer reported in #71607 that decrypting RSA OAEP with SHA2 and a 3072-bit key on Linux resulted in a OAEP de-padding error, while other platforms were able to perform these operations successfully.
Investigation in to the issue led to uncover that RSA OAEP decryption that uses the managed implementation does not work with non-power-of-two keys because we do not slice a rented buffer accordingly. This issue was also present for Android, in addition to the reported platform, Linux. The current implementation only works because
CryptoPool.Rent
happens to give back power-of-two arrays which are exactly the same size as the key.The fix is to slice the data to the correct size.
.NET 7 will address this issue differently, by completely removing the managed RSA OEAP depadding in #71670, so this is not a back port.
This is a port of dotnet/corefx#43153 for Android for release/6.0.
Testing
Unit tests were introduced to test RSA OAEP encryption with a 3072-bit RSA key. These tests will be forward-ported to dotnet/runtime@main.
Risk
Minimal. The change is localized and understood that a
Slice
was missing.