cms signing error with SHA3-512 & PSS

20 views
Skip to first unread message

Harald Koch

unread,
Jan 21, 2026, 8:00:29 AMJan 21
to openss...@openssl.org
Hello list,

with the lastest openSSL 3.6.0 version, I try to sign a file with the following command (the requirements are to support PSS and SHA3-512!):

openssl cms -sign -in /tmp/input -out /tmp/output -signer /tmp/signed_key_priv.cer -keyopt rsa_padding_mode:pss -nocerts -nodetach -binary -outform DER -md sha3-512

I get the following error:

Error finalizing CMS structure
00000000:error:1C8C0103:Provider routines:rsa_generate_signature_aid:internal error:providers/implementations/signature/rsa_sig.c:352:
00000000:error:17000067:CMS routines:CMS_final:cms datafinal error:crypto/cms/cms_smime.c:926:


When changing SHA3-512 to any other SHA3 algorithm (like SHA3-256 or SHA3-384), everything seems to work fine. Also, when removing the PSS option, the command succeeds with SHA3-512. Is there anything I can improve? May using the C library with an own library call succeed? Am I missing a crucial parameter for this combination?


Regards,
Harald Koch



Matt Caswell

unread,
Jan 21, 2026, 11:27:59 AMJan 21
to Harald Koch, openss...@openssl.org
On Wed, 21 Jan 2026 at 13:00, 'Harald Koch' via openssl-users <openss...@openssl.org> wrote:
Hello list,

with the lastest openSSL 3.6.0 version, I try to sign a file with the following command (the requirements are to support PSS and SHA3-512!):

openssl cms -sign -in /tmp/input -out /tmp/output -signer /tmp/signed_key_priv.cer -keyopt rsa_padding_mode:pss -nocerts -nodetach -binary -outform DER -md sha3-512

I get the following error:

Error finalizing CMS structure
00000000:error:1C8C0103:Provider routines:rsa_generate_signature_aid:internal error:providers/implementations/signature/rsa_sig.c:352:
00000000:error:17000067:CMS routines:CMS_final:cms datafinal error:crypto/cms/cms_smime.c:926:


When changing SHA3-512 to any other SHA3 algorithm (like SHA3-256 or SHA3-384), everything seems to work fine.

Are you sure about that? When I try SHA3-256 or SHA3-384 it fails in the same way. It works with the SHA2 variants, i.e. sha2-256, sha2-384, sha2-512

The reason can be seen in the code here:


We hit the default case above and error out. Following the link in the comment brings you to RFC8017:


"hashAlgorithm identifies the hash function.  It SHALL be an algorithm ID with an OID in the set OAEP-PSSDigestAlgorithms."

Below that sentence is the set of algorithms, which does not include any SHA3 algorithms.

Matt

 
Also, when removing the PSS option, the command succeeds with SHA3-512. Is there anything I can improve? May using the C library with an own library call succeed? Am I missing a crucial parameter for this combination?


Regards,
Harald Koch



--
You received this message because you are subscribed to the Google Groups "openssl-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openssl-user...@openssl.org.
To view this discussion visit https://groups.google.com/a/openssl.org/d/msgid/openssl-users/C57D8068-C17D-4B5F-B2C9-0A9E937EDA00%40c-works.net.

Harald Koch

unread,
Jan 21, 2026, 11:40:35 AMJan 21
to Matt Caswell, openss...@openssl.org
Hello Matt,

Am 21.01.2026 um 17:27 schrieb Matt Caswell <ma...@openssl.org>:
On Wed, 21 Jan 2026 at 13:00, 'Harald Koch' via openssl-users <openss...@openssl.org> wrote:
Hello list,

with the lastest openSSL 3.6.0 version, I try to sign a file with the following command (the requirements are to support PSS and SHA3-512!):

openssl cms -sign -in /tmp/input -out /tmp/output -signer /tmp/signed_key_priv.cer -keyopt rsa_padding_mode:pss -nocerts -nodetach -binary -outform DER -md sha3-512

I get the following error:

Error finalizing CMS structure
00000000:error:1C8C0103:Provider routines:rsa_generate_signature_aid:internal error:providers/implementations/signature/rsa_sig.c:352:
00000000:error:17000067:CMS routines:CMS_final:cms datafinal error:crypto/cms/cms_smime.c:926:


When changing SHA3-512 to any other SHA3 algorithm (like SHA3-256 or SHA3-384), everything seems to work fine.

Are you sure about that? When I try SHA3-256 or SHA3-384 it fails in the same way. It works with the SHA2 variants, i.e. sha2-256, sha2-384, sha2-512

You are right, I just looked in the wrong console output: SHA3-256 and SHA3-384 don’t work either. Thank you for pointing me on this.


The reason can be seen in the code here:


We hit the default case above and error out. Following the link in the comment brings you to RFC8017:


"hashAlgorithm identifies the hash function.  It SHALL be an algorithm ID with an OID in the set OAEP-PSSDigestAlgorithms."

Below that sentence is the set of algorithms, which does not include any SHA3 algorithms.
The list in the RFC also has a line "...  -- Allows for future expansion —„, which implies more digest algorithms to be added. SHA33-512 can be identified as 2.16.840.1.101.3.4.2.10 (https://oid-base.com/get/2.16.840.1.101.3.4.2.10).

Greetings,
Harald




Richard Levitte

unread,
Jan 22, 2026, 2:41:32 AMJan 22
to openss...@openssl.org
On 1/21/26 5:40 PM, 'Harald Koch' via openssl-users wrote:
Hello Matt,
...
The reason can be seen in the code here:


We hit the default case above and error out. Following the link in the comment brings you to RFC8017:


"hashAlgorithm identifies the hash function.  It SHALL be an algorithm ID with an OID in the set OAEP-PSSDigestAlgorithms."

Below that sentence is the set of algorithms, which does not include any SHA3 algorithms.
The list in the RFC also has a line "...  -- Allows for future expansion —„, which implies more digest algorithms to be added. SHA33-512 can be identified as 2.16.840.1.101.3.4.2.10 (https://oid-base.com/get/2.16.840.1.101.3.4.2.10).

This is a matter of interpretation, isn't it?  Does "future expansion" mean ad-hoc, or by newer RFCs?

Ad-hoc could of course be done, if it's safe to assume the same PARAMETERS NULL as for all others (I agree that this is a safe bet, but one never really knows, yeah?).  But either way, OpenSSL providers currently lean on the safe side of algos specified by RFCs, rather than ad-hoc expansion.  (I think this is library policy)

Cheers,
Richard

Harald Koch

unread,
Jan 22, 2026, 6:34:29 AMJan 22
to openss...@openssl.org
Hello Richard,
Since at least Bouncycastle supports this algorithm, I assume their library policy allows the usage of additional OIDs (so do I). If it is a simple task to add it to the list of support algorithms, may it be possible for me to do so (if openSSL doesn’t want to go the same way)? This way I would have a functional framework and could wait for an official implementation. I have to keep up compatibility and want to stay with openSSL.

Greetings,
Harald


Reply all
Reply to author
Forward
0 new messages