Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
OpenSSL Windows CryptoAPI certificate and encrypt/decrypt interop
634 views
Skip to first unread message
Nou Dadoun
Mar 7, 2012, 3:40:41 PM3/7/12
to
I'm trying to develop a package which can establish an ssl connection between a windows server and a client using openssl and I'm running into some serious road blocks - I'm a relative novice at both the openssl and wincrypt apis, I've done extensive searching for any hints at how this problem might be solved but I haven't found anything similar which might be applicable.
Here's a high-level description:
On the server side, I'm using both openssl and the windows crypto api, I use the windows crypto api to access the windows certificate store and search for and load the certificate (with its private key) that I'm interested in using. I spent some time trying to extract the private key to use in openssl directly but discovered that the windows crypto api refuses to let go of it in a form that openssl can use directly.
So I'm using the openssl RSA_set_method facility to replace the private key decrypt method (rsa_priv_dec) to pass off the heavy lifting to the cryptoapi (this seems like a reasonable thing to do and the kind of thing that would be done with a smart card or TCM or something where there's no direct access to the private key). I tell openssl to use the x509 encoding of the loaded certificate for incoming connections and wait.
When a client SSL HELLO comes in, the server presents the x509 certificate, the client uses the public key in the certificate to encrypt its "secret" and sends the response to the server to decrypt with the private key.
The server receives the response, my replaced method kicks in and tries to decrypt the 128-byte response using CryptDecrypt with the private key and fails with NTE_BAD_DATA (pretty opaque and not very informative). So the connection fails.
I've done some unit testing on the api's and I can encrypt/decrypt sample messages using CryptEncrypt and CryptDecrypt, but anything that I've encrypted with RSA_public_encrypt gives me the same NTE_BAD_DATA error when I hand it to (private key) CryptDecrypt.
I've extracted the PUBLICKEYBLOB to get the windows representation of the public key and I can see that the key bytes are in reverse order of the public key portion of the x509 certificate. I've read that the crypto api uses little-endian as opposed to everybody else's big-endian but (perhaps foolishly) assumed that that difference was only an internal representation issue, that something encrypted by openssl should be decryptable by CryptoAPI (I tried reversing the bytes of the encrypted message too and that didn't work.)
So I'm a little stymied, am I missing some secret sauce somewhere?
If openssl had an api to load certificates from windows cert stores that would be an alternate route but it seems as though this mechanism of using CryptoApi to handle the certificate access/management shouldn't be as difficult as it seems. And surely the interop problem I'm experiencing can't be unique to my setup. Anyone have any suggestions?
I can post code on request but thought I'd start with a high-level description of the problem to avoid clouding the issue too much.
thanks in advance ... Nou
---
Nou Dadoun
nda...@teradici.com
604-628-1215
______________________________________________________________________
OpenSSL Project http://www.openssl.org
User Support Mailing List openss...@openssl.org
Automated List Manager majo...@openssl.org
Here's a high-level description:
On the server side, I'm using both openssl and the windows crypto api, I use the windows crypto api to access the windows certificate store and search for and load the certificate (with its private key) that I'm interested in using. I spent some time trying to extract the private key to use in openssl directly but discovered that the windows crypto api refuses to let go of it in a form that openssl can use directly.
So I'm using the openssl RSA_set_method facility to replace the private key decrypt method (rsa_priv_dec) to pass off the heavy lifting to the cryptoapi (this seems like a reasonable thing to do and the kind of thing that would be done with a smart card or TCM or something where there's no direct access to the private key). I tell openssl to use the x509 encoding of the loaded certificate for incoming connections and wait.
When a client SSL HELLO comes in, the server presents the x509 certificate, the client uses the public key in the certificate to encrypt its "secret" and sends the response to the server to decrypt with the private key.
The server receives the response, my replaced method kicks in and tries to decrypt the 128-byte response using CryptDecrypt with the private key and fails with NTE_BAD_DATA (pretty opaque and not very informative). So the connection fails.
I've done some unit testing on the api's and I can encrypt/decrypt sample messages using CryptEncrypt and CryptDecrypt, but anything that I've encrypted with RSA_public_encrypt gives me the same NTE_BAD_DATA error when I hand it to (private key) CryptDecrypt.
I've extracted the PUBLICKEYBLOB to get the windows representation of the public key and I can see that the key bytes are in reverse order of the public key portion of the x509 certificate. I've read that the crypto api uses little-endian as opposed to everybody else's big-endian but (perhaps foolishly) assumed that that difference was only an internal representation issue, that something encrypted by openssl should be decryptable by CryptoAPI (I tried reversing the bytes of the encrypted message too and that didn't work.)
So I'm a little stymied, am I missing some secret sauce somewhere?
If openssl had an api to load certificates from windows cert stores that would be an alternate route but it seems as though this mechanism of using CryptoApi to handle the certificate access/management shouldn't be as difficult as it seems. And surely the interop problem I'm experiencing can't be unique to my setup. Anyone have any suggestions?
I can post code on request but thought I'd start with a high-level description of the problem to avoid clouding the issue too much.
thanks in advance ... Nou
---
Nou Dadoun
nda...@teradici.com
604-628-1215
______________________________________________________________________
OpenSSL Project http://www.openssl.org
User Support Mailing List openss...@openssl.org
Automated List Manager majo...@openssl.org
Dr. Stephen Henson
Mar 7, 2012, 8:52:13 PM3/7/12
to
use that with OpenSSL. If the key is unexportable you can't.
OpenSSL has a CryptoAPI ENGINE you can use and that will handle the conversion
between formats for you. It is supported on the command line which you can use
as a test.
Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org
Nou Dadoun
Mar 8, 2012, 3:05:52 PM3/8/12
to
Thanks for the response, I'm trying to allow end-users to use commercially purchased certificates so I'd rather not make the assumption that the key is exportable.
Using the capi engine sounds like a viable alternative, but I've had trouble tracking down details on how to use it.
Unfortunately I have a few restrictions; we're fips-certified on openssl 0.98n so that's the version I'm stuck with (without recertifying). I also want to use the crypto api directly to tell it which certificate to load and use (i.e. user configurable through a gpo setting) and then have the engine use that certificate for the ssl handshake to the peer.
I've read the O'Reilly section on Engines but it's pretty rudimentary and doesn't touch the capi engine, do you have a pointer to any user documentation that might have some examples on using the capi engine?
Thanks again ... N
Using the capi engine sounds like a viable alternative, but I've had trouble tracking down details on how to use it.
Unfortunately I have a few restrictions; we're fips-certified on openssl 0.98n so that's the version I'm stuck with (without recertifying). I also want to use the crypto api directly to tell it which certificate to load and use (i.e. user configurable through a gpo setting) and then have the engine use that certificate for the ssl handshake to the peer.
I've read the O'Reilly section on Engines but it's pretty rudimentary and doesn't touch the capi engine, do you have a pointer to any user documentation that might have some examples on using the capi engine?
Thanks again ... N
Dr. Stephen Henson
Mar 8, 2012, 4:54:50 PM3/8/12
to
On Thu, Mar 08, 2012, Nou Dadoun wrote:
> Thanks for the response, I'm trying to allow end-users to use commercially purchased certificates so I'd rather not make the assumption that the key is exportable.
>
> Using the capi engine sounds like a viable alternative, but I've had trouble tracking down details on how to use it.
>
> Unfortunately I have a few restrictions; we're fips-certified on openssl 0.98n so that's the version I'm stuck with (without recertifying). I also want to use the crypto api directly to tell it which certificate to load and use (i.e. user configurable through a gpo setting) and then have the engine use that certificate for the ssl handshake to the peer.
>
> I've read the O'Reilly section on Engines but it's pretty rudimentary and doesn't touch the capi engine, do you have a pointer to any user documentation that might have some examples on using the capi engine?
>
If you need all crypto to be FIPS compliant (for some value of compliant)
> Thanks for the response, I'm trying to allow end-users to use commercially purchased certificates so I'd rather not make the assumption that the key is exportable.
>
> Using the capi engine sounds like a viable alternative, but I've had trouble tracking down details on how to use it.
>
> Unfortunately I have a few restrictions; we're fips-certified on openssl 0.98n so that's the version I'm stuck with (without recertifying). I also want to use the crypto api directly to tell it which certificate to load and use (i.e. user configurable through a gpo setting) and then have the engine use that certificate for the ssl handshake to the peer.
>
> I've read the O'Reilly section on Engines but it's pretty rudimentary and doesn't touch the capi engine, do you have a pointer to any user documentation that might have some examples on using the capi engine?
>
that's a can of worms because the relevant CSPs might not be and you'd be
mixing various cryptographic operations across boundaries.
Nou Dadoun
Mar 9, 2012, 12:37:13 PM3/9/12
to
I think I'll probably have to jump off that bridge when I get to it, but to make progress I'm going to try to get something going in the interim. I think I'll post some code (I suspect that the padding provided by the capi encrypt/decrypt is somehow different than what openssl is doing - the ms docs are woefully inadequate) but if anyone has pointers on information on how to use the capi engine, I'd greatly appreciate it, thanks! ... N
---
Nou Dadoun
nda...@teradici.com
604-628-1215
-----Original Message-----
From: owner-ope...@openssl.org [mailto:owner-ope...@openssl.org] On Behalf Of Dr. Stephen Henson
Sent: March 8, 2012 1:55 PM
To: openss...@openssl.org
Subject: Re: OpenSSL Windows CryptoAPI certificate and encrypt/decrypt interop
---
Nou Dadoun
nda...@teradici.com
604-628-1215
-----Original Message-----
From: owner-ope...@openssl.org [mailto:owner-ope...@openssl.org] On Behalf Of Dr. Stephen Henson
Sent: March 8, 2012 1:55 PM
To: openss...@openssl.org
Subject: Re: OpenSSL Windows CryptoAPI certificate and encrypt/decrypt interop
Dr. Stephen Henson
Mar 9, 2012, 1:11:21 PM3/9/12
to
On Fri, Mar 09, 2012, Nou Dadoun wrote:
> I think I'll probably have to jump off that bridge when I get to it, but to
> make progress I'm going to try to get something going in the interim. I
> think I'll post some code (I suspect that the padding provided by the capi
> encrypt/decrypt is somehow different than what openssl is doing - the ms
> docs are woefully inadequate) but if anyone has pointers on information on
> how to use the capi engine, I'd greatly appreciate it, thanks! ... N
>
You can actually use the capi ENGINE from the command line to perform private
> I think I'll probably have to jump off that bridge when I get to it, but to
> make progress I'm going to try to get something going in the interim. I
> think I'll post some code (I suspect that the padding provided by the capi
> encrypt/decrypt is somehow different than what openssl is doing - the ms
> docs are woefully inadequate) but if anyone has pointers on information on
> how to use the capi engine, I'd greatly appreciate it, thanks! ... N
>
key operations. You specify -engine capi -inform/-keyform engine and the input
key parameter can take several forms. The default is a string from the
corresponding certificate DN so if it has "CN=steve" then "steve" would do.
At an API level you use ENGINE_load_private_key("steve") which returns an
EVP_PKEY structure which can then be used like any other private key.
Nou Dadoun
Mar 9, 2012, 1:24:42 PM3/9/12
to
That does sound incredibly powerful (and right up the alley of what I'm try to do) but rather than peppering you with individual questions, is there any documentation other than tackling the code? (even an api man page wrt engine usage would help)
E.g. I'd like to specify the location of the windows cert store to look in for the certificate that I want to use being selecting the actual certificate, and it's not clear how I would do that, thanks again for your help ... N
---
Nou Dadoun
nda...@teradici.com
604-628-1215
-----Original Message-----
From: owner-ope...@openssl.org [mailto:owner-ope...@openssl.org] On Behalf Of Dr. Stephen Henson
Sent: March 9, 2012 10:11 AM
To: openss...@openssl.org
Subject: Re: OpenSSL Windows CryptoAPI certificate and encrypt/decrypt interop
E.g. I'd like to specify the location of the windows cert store to look in for the certificate that I want to use being selecting the actual certificate, and it's not clear how I would do that, thanks again for your help ... N
---
Nou Dadoun
nda...@teradici.com
604-628-1215
-----Original Message-----
From: owner-ope...@openssl.org [mailto:owner-ope...@openssl.org] On Behalf Of Dr. Stephen Henson
Sent: March 9, 2012 10:11 AM
To: openss...@openssl.org
Subject: Re: OpenSSL Windows CryptoAPI certificate and encrypt/decrypt interop
Jack
Oct 7, 2023, 1:45:28 AM10/7/23
to
Active engagement on social media platforms can help increase brand awareness and drive traffic to your website. Sharing your content on social media can also encourage others to share it, leading to more backlinks and improved visibility https://techzone-agency.com/
0 new messages