[PATCH] crypto: openssl: fix verification of parameterized RSA-PSS keys

[Oliver Kaestner]

Do not set RSA_PSS_SALTLEN_AUTO during verification.

This fails for parameterized RSASSA-PSS keys, where the public key
encodes any restrictions, e.g. for the digest algorithm, or salt length.

Setting RSA_PSS_SALTLEN_AUTO explicitly is unnecessary for verification
as the length will be auto-detected by default [1]:

> EVP_PKEY_CTX_set_rsa_pss_saltlen() sets the RSA PSS salt length to
> saltlen. As its name implies it is only supported for PSS padding.
> If this function is not called then the salt length is maximized up
> to the digest length when signing and auto detection when verifying.

But setting this value also causes the verification to fail for
parameterized keys as the OpenSSL docs note [2]:

> The EVP_PKEY_CTX_set_rsa_pss_saltlen() macro is used to set the salt
> length. If the key has usage restrictions then an error is returned
> if an attempt is made to set the salt length below the minimum value.
> It is otherwise similar to the RSA operation except detection of the
> salt length (using RSA_PSS_SALTLEN_AUTO) is not supported for
> verification if the key has usage restrictions.

So remove that call and let OpenSSL do the right thing automatically.

[1] https://docs.openssl.org/3.5/man3/EVP_PKEY_CTX_ctrl/#rsa-parameters
[2] https://docs.openssl.org/3.5/man3/EVP_PKEY_CTX_set_rsa_pss_keygen_md

Link: https://groups.google.com/g/swupdate/c/FMRY6rtuKW8
Signed-off-by: Oliver Kästner <okae...@rosen-nxt.com>
---
crypto/swupdate_rsa_verify_openssl.c | 5 -----
1 file changed, 5 deletions(-)

diff --git a/crypto/swupdate_rsa_verify_openssl.c b/crypto/swupdate_rsa_verify_openssl.c
index fac102ce..195a0f44 100644
--- a/crypto/swupdate_rsa_verify_openssl.c
+++ b/crypto/swupdate_rsa_verify_openssl.c
@@ -71,11 +71,6 @@ static int dgst_verify_init(struct openssl_digest *dgst)
ERROR("EVP_PKEY_CTX_set_rsa_padding failed, error 0x%lx", ERR_get_error());
return -EFAULT; /* failed */
}
- rc = EVP_PKEY_CTX_set_rsa_pss_saltlen(dgst->ckey, -2);
- if (rc <= 0) {
- ERROR("EVP_PKEY_CTX_set_rsa_pss_saltlen failed, error 0x%lx", ERR_get_error());
- return -EFAULT; /* failed */
- }
}

return 0;
--
2.43.0

[Stefano Babic]

Applied to -master, thanks !

Best regards,
Stefano Babic

--
_______________________________________________________________________
Nabla Software Engineering GmbH
Hirschstr. 111A | 86156 Augsburg | Tel: +49 821 45592596
Geschäftsführer : Stefano Babic | HRB 40522 Augsburg
E-Mail: sba...@nabladev.com

[Oliver Kästner]

Great, thanks! Would it be possible to apply it on scarthgap, too? Otherwise I'll just patch it in my bbappend.

- Oliver

[Stefano Babic]

On 3/24/26 15:52, 'Oliver Kästner' via swupdate wrote:
> Great, thanks! Would it be possible to apply it on scarthgap, too?

Not as fix version, so 2025.12.x.

But you can use PREFERRED_VERSION_swupdate = "2025.12+git%"

I have update the _git recipe for all supported branches:

857cb95..d4cfdeb master -> master
0792561..81f4faa scarthgap -> scarthgap
0cbabdc..7e28fc4 whinlatter -> whinlatter
857cb95..b1fbd1d wrynose -> wrynose

Best regards,
Stefano Babic

> parameters <https://docs.openssl.org/3.5/man3/EVP_PKEY_CTX_ctrl/
> #rsa-parameters>
> > [2] https://docs.openssl.org/3.5/man3/
> EVP_PKEY_CTX_set_rsa_pss_keygen_md <https://docs.openssl.org/3.5/
> man3/EVP_PKEY_CTX_set_rsa_pss_keygen_md>
> >
> > Link: https://groups.google.com/g/swupdate/c/FMRY6rtuKW8

> Hirschstr. 111A | 86156 Augsburg | Tel: +49 821 45592596 <tel:
> +49%20821%2045592596>

> Geschäftsführer : Stefano Babic | HRB 40522 Augsburg
> E-Mail: sba...@nabladev.com
>

> --
> You received this message because you are subscribed to the Google
> Groups "swupdate" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to swupdate+u...@googlegroups.com
> <mailto:swupdate+u...@googlegroups.com>.
> To view this discussion visit https://groups.google.com/d/msgid/
> swupdate/50155983-4ea9-4709-81de-a2fcbc7b71b7n%40googlegroups.com
> <https://groups.google.com/d/msgid/swupdate/50155983-4ea9-4709-81de-
> a2fcbc7b71b7n%40googlegroups.com?utm_medium=email&utm_source=footer>.