tl;dr: is it worth using OpenSSL to build a CMS EnvelopedData message when the key transport algorithm is RSA-OAEP? If so, how?
Long version:
After some more digging, I'll try to make my request more precise.
Some context: I am generating a RSA key pair with an external engine (say a HSM with PKCS #11). This key pair will be used to wrap/unwrap symmetric keys with RSAES-OAEP. RFC 4055 specifies some algorithm identifiers for use with CMS, RSAES-OAEP being one of them (§4). I'm making a certificate signing request for the public key, and of course I'd want its Public Key Algorithm to be RSAES-OAEP (probably with hash = SHA1, mgf = SHA1, param = none, since currently only those parameters are implemented with 1.0.1f).
The idea is to use CMS to envelop some content (i.e. encrypt it with a symmetric key generated on-the-fly, and wrap this key with the public RSA key). The content-encryption-key will be unwrapped by the external engine.
So essentially, I want to use OpenSSL to:
- on the client side, build a CMS structure to hold an EnvelopedData type, the content-key being wrapped with OAEP, and export that to DER/PEM/whatever.
- on the server side, parse the CMS structure, get the wrapped key and the encrypted message. The actual unwrapping/decryption will be made by the external engine.
(Add some signing/verifying to the mix, i.e. encapsulate the EnvelopedData into a SignedData ; OpenSSL will then be used server-side to parse 1) the signature 2) the encapsulated-content-info out of the CMS, the actual verification will be made by the external engine)
Is OpenSSL a good choice here? Client-side, I'll have to build a CMS structure ; demos/cms/cms_enc.c makes it look simple, but the key wrapping mechanism is chosen on the basis of the recipient's certificate... And using OpenSSL, building a CSR saying "THIS KEY IS MEANT TO BE USED WITH OAEP" seems tedious.
I'm not even sure 1.0.1 supports OAEP as a method for key transport though: cms_RecipientInfo_ktri_encrypt() calls EVP_PKEY_CTX_ctrl, which I guess resolves to pkey_rsa_ctrl() ; OAEP padding is only set for EVP_PKEY_CTRL_CMS_DECRYPT. (I... don't really understand this part of the code (rsa_pmeth.c:580). When decrypting, the RecipientInfo structure is parsed and OAEP padding is specified ; but when it comes to encrypting the function just returns 1? ...)
Server-side, I expect I'll only use the crypto library as a CMS/X.509 parser to get the bytes I'm interested in (i.e. the actual signature, wrapped key, encrypted message). Which means struggling with the ASN.1 API to find the correct functions for getting my content back from the internal structures.
In the end I'll mostly use OpenSSL as a glorified CMS parser. Client-side, OAEP support for key wrapping seems unavailable (and if it is, I still don't know how to generate a certificate which actually reads "OAEP" as Public Key Algorithm) ; this means, I guess, that I'll use standard EVP functions to build my encrypted content/wrapped key, and then manually stuff them inside a CMS structure, bypassing the fancy CMS_encrypt function (haven't looked yet but I expect CMS_sign() should be able to handle ECDSA/SHA1... ?).
(If I go for the "manual stuffing" option, I guess I can stop worrying about my Public Key Algorithm not reading "OAEP" and just have my client application "know" that it should wrap with OAEP, whatever the certificate says... After all, the server application will do exactly that, although it would be nice if it could decide the unwrapping mechanism based on the cert)
When I ask if OpenSSL is a "good choice", I don't mean to troll, I just figure some people must have faced the same problems (RFC 4055 has been out since 2005). So I assume those people chose to 1/ use a patched OpenSSL to support their particular mechanism choice (like that guy[1]) 2/ go for the "manual stuffing" option 3/ stop caring about standards and use their own conventions for encrypted content/wrapped key/signature transport, or 4/ find some other framework which actually supports all these algorithms and provides a nice CMS-building/parsing interface.
(Or maybe 5/ they just use XMLENC/XMLDSIG)
[1]
http://stackoverflow.com/questions/22373305/rsa-public-key-encryption-openssl