Add legacy RSA fallback for Engine-based applications

[olszomal]

Current Behavior:

With OpenSSL 3.x, RSA key reconstruction uses the provider-based EVP_PKEY_fromdata() path.

In Engine-based applications, a suitable provider context may not always be available for this operation. In that case, EVP_PKEY_fromdata_init() can fail, which prevents SoftHSM from reconstructing the internal RSA key object.

Scope of Changes:

This change explicitly uses the OpenSSL default provider for internal RSA key reconstruction.

If the provider-based RSA reconstruction cannot be initialized, the code falls back to the legacy RSA API while it is still available. This fallback is intended for Engine-based applications and preserves compatibility with OpenSSL 3.x.

The fallback is only used when the provider key creation context cannot be initialized. Failures from EVP_PKEY_fromdata() itself are still treated as RSA key reconstruction errors.

Testing:

Verified RSA signing with SoftHSM, OpenSSL 3.6.2, and the libp11 Engine.

Before this change, RSA signing through the Engine failed because SoftHSM could not initialize the provider-based RSA key reconstruction path. After this change, SoftHSM falls back to the legacy RSA API and signing succeeds.

Summary by CodeRabbit

• Bug Fixes
  • Enhanced RSA cryptographic key handling to improve compatibility with OpenSSL 3.x and 4.x versions.
  • Improved error handling and fallback mechanisms for more reliable RSA key reconstruction operations.
  • Refined diagnostic handling and error state management for better stability across OpenSSL versions.

[coderabbitai]

📝 Walkthrough

Walkthrough

Both RSA key files update reconstruction logic for OpenSSL 3.x+ compatibility. They introduce provider-based context methods to rebuild EVP_PKEY from parameters, add explicit error handling and cleanup paths, and include version-gated fallback to legacy RSA APIs for OpenSSL versions where provider-based reconstruction fails.

Changes

OpenSSL 3.x+ RSA Key Reconstruction Compatibility

Layer / File(s)	Summary
Error Header Infrastructure src/lib/crypto/OSSLRSAPrivateKey.cpp, src/lib/crypto/OSSLRSAPublicKey.cpp	Conditional inclusion of <openssl/err.h> headers in both files to support error-queue operations used in new reconstruction paths.
Private Key Provider-Based Reconstruction src/lib/crypto/OSSLRSAPrivateKey.cpp	EVP_PKEY_CTX initialization with default provider, context-based EVP_PKEY_fromdata reconstruction flow with explicit cleanup and freeing on success; OpenSSL < 4.0 fallback clears error queue and proceeds to legacy RSA API; diagnostic pragmas adjusted.
Public Key Provider-Based and Legacy Reconstruction src/lib/crypto/OSSLRSAPublicKey.cpp	OpenSSL 3+ branch reworked to attempt EVP_PKEY_fromdata with provider context and version-specific failure handling: OpenSSL 4.x+ emits error and aborts, older versions clear errors and fall back to legacy path. Legacy section ensures RSA_set0_key is called and pragma state is restored.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Suggested reviewers

• kalvdans
• bjosv
• jschlyter

Poem

🐰 With OpenSSL's dance between old and new,
Two RSA keys learned tricks to construe,
Provider contexts rise, error queues clear,
Legacy fallbacks stand ready right here! 🔐
Compatibility blooms when versions align,
Cryptographic rabbits make crypto code shine!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately captures the main change: adding a legacy RSA fallback mechanism for Engine-based applications when provider-based reconstruction fails in OpenSSL 3.x.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

src/lib/crypto/OSSLRSAPrivateKey.cpp (1)

385-427: ⚠️ Potential issue | 🟠 Major

Restrict legacy RSA fallback to provider-context/init failures only.

• In src/lib/crypto/OSSLRSAPrivateKey.cpp and src/lib/crypto/OSSLRSAPublicKey.cpp, when OPENSSL_VERSION_NUMBER < 0x40000000L, the code clears the error queue and proceeds to the legacy RSA builder for any provider-based reconstruction failure (including OSSL_PARAM_BLD_to_param()/params == NULL and EVP_PKEY_fromdata() failures, even after EVP_PKEY_fromdata_init() succeeded).
• Split the failure handling so only EVP_PKEY_CTX_new_from_name(...) / EVP_PKEY_fromdata_init(...) failures trigger the legacy fallback; keep EVP_PKEY_fromdata(...) and param-building failures as hard reconstruction errors (log and return with rsa == NULL) to avoid masking malformed/inconsistent RSA components.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/crypto/OSSLRSAPrivateKey.cpp` around lines 385 - 427, The current
logic falls back to the legacy RSA API for any provider-path failure; change it
so only failures to obtain/init the provider context trigger the legacy
fallback: check the results of EVP_PKEY_CTX_new_from_name(...) and
EVP_PKEY_fromdata_init(ctx) first and if either fails then clear errors and jump
to the existing legacy fallback path; treat OSSL_PARAM_BLD_to_param()/params ==
NULL and EVP_PKEY_fromdata(ctx, &rsa, ...) failures as hard errors — log ("Could
not create RSA key object" or similar), free bn_* and params/ctx as needed,
leave rsa NULL and return immediately (no legacy fallback). Update the same
pattern in both OSSLRSAPrivateKey.cpp and OSSLRSAPublicKey.cpp, referencing
EVP_PKEY_CTX_new_from_name, EVP_PKEY_fromdata_init, OSSL_PARAM_BLD_to_param,
params, EVP_PKEY_fromdata and rsa to locate the code to change.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@src/lib/crypto/OSSLRSAPrivateKey.cpp`:
- Around line 385-427: The current logic falls back to the legacy RSA API for
any provider-path failure; change it so only failures to obtain/init the
provider context trigger the legacy fallback: check the results of
EVP_PKEY_CTX_new_from_name(...) and EVP_PKEY_fromdata_init(ctx) first and if
either fails then clear errors and jump to the existing legacy fallback path;
treat OSSL_PARAM_BLD_to_param()/params == NULL and EVP_PKEY_fromdata(ctx, &rsa,
...) failures as hard errors — log ("Could not create RSA key object" or
similar), free bn_* and params/ctx as needed, leave rsa NULL and return
immediately (no legacy fallback). Update the same pattern in both
OSSLRSAPrivateKey.cpp and OSSLRSAPublicKey.cpp, referencing
EVP_PKEY_CTX_new_from_name, EVP_PKEY_fromdata_init, OSSL_PARAM_BLD_to_param,
params, EVP_PKEY_fromdata and rsa to locate the code to change.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 9f363725-b39b-4a37-b442-4980e248fcae

📥 Commits

Reviewing files that changed from the base of the PR and between de25233 and 7d29407.

📒 Files selected for processing (2)

• src/lib/crypto/OSSLRSAPrivateKey.cpp
• src/lib/crypto/OSSLRSAPublicKey.cpp