Need some help debugging SSL error thrown from STunnel using OpenSSL-FIPS
David Gillingham
I've been tasked to internally investigate a system that utilizes
STunnel and OpenSSL to create a secure wrapper for a propietary
protocol. Additionally, this solution must eventually be FIPS 140-2
compliant.
So, using instructions outlined in the OpenSSL FIPS Security Policy
and on this mailing list, I have been able to succesfully build a
FIPS-compliant distribution using MinGW and Visual Studio 2005.
Then, I took the STunnel source and modified its SSL initialization
function to invoke OpenSSL's FIPS mode (using FIPS_mode_set(1), as
outlined on page 45 of the security policy), along with changing a few
#includes to allow it build on VS2005.
It is important to note that I was able to succesfully use STunnel
prior to adding in the FIPS mode invocation. However, after building
STunnel with the FIPS mode invocation, I'm encountering some program
errors (which seem to be SSL errors) that I'm having some trouble
deciphering. I understand that the task of deciphering these errors
may be better directed at an STunnel mailing list, but I am unable to
access their page from work.
What follows is a STunnel program log that contains what appears to be
a stack trace of the SSL errors being thrown. In line 8, STunnel
claims that one of the OpenSSL calls is being disabled for FIPS, but
it is not clear to me which call that was. I was hoping that someone
more familiar with OpenSSL in FIPS mode may be able to lend a hand on
that one. Also note that server.pem is a file that contains an RSA
private key and a password-protected, signed certificate in PKCS7
format. Please be aware that I am definitely using the right password
for the cert as I have verified this in the copy of the code not using
OpenSSL's FIPS mode.
----BEGIN STUNNEL LOG----
2006.06.06 18:58:26 LOG7[592:1816]: RAND_status claims sufficient
entropy for the PRNG
2006.06.06 18:58:26 LOG6[592:1816]: PRNG seeded successfully
2006.06.06 18:58:26 LOG7[592:1816]: Certificate: server.pem
2006.06.06 18:58:26 LOG7[592:1816]: Key file: server.pem
2006.06.06 18:58:32 LOG3[592:1816]: error stack: 140B3009 :
error:140B3009:SSL routines:SSL_CTX_use_RSAPrivateKey_file:PEM lib
2006.06.06 18:58:32 LOG3[592:1816]: error stack: 906A065 :
error:0906A065:PEM routines:PEM_do_header:bad decrypt
2006.06.06 18:58:32 LOG3[592:1816]: error stack: 6065064 :
error:06065064:digital envelope routines:EVP_DecryptFinal:bad decrypt
2006.06.06 18:58:32 LOG3[592:1816]: SSL_CTX_use_RSAPrivateKey_file:
608008D: error:0608008D:digital envelope
routines:EVP_DigestInit:disabled for fips
2006.06.06 18:58:32 LOG3[592:1816]: Server is down
----END STUNNEL LOG----
______________________________________________________________________
OpenSSL Project http://www.openssl.org
User Support Mailing List openss...@openssl.org
Automated List Manager majo...@openssl.org
Dr. Stephen Henson
> Hello all,
>
> I've been tasked to internally investigate a system that utilizes
> STunnel and OpenSSL to create a secure wrapper for a propietary
> protocol. Additionally, this solution must eventually be FIPS 140-2
> compliant.
>
> 608008D: error:0608008D:digital envelope
> routines:EVP_DigestInit:disabled for fips
>
That's the problem. I'd guess that this is due to a certificate using an
algorithm that isn't allowed in FIPS mode: probably MD5.
Steve.
--
Dr Stephen N. Henson. Email, S/MIME and PGP keys: see homepage
OpenSSL project core developer and freelance consultant.
Funding needed! Details on homepage.
Homepage: http://www.drh-consultancy.demon.co.uk
Dr. Stephen Henson
> Hello all,
>
> I've been tasked to internally investigate a system that utilizes
> STunnel and OpenSSL to create a secure wrapper for a propietary
> protocol. Additionally, this solution must eventually be FIPS 140-2
> compliant.
>
> 608008D: error:0608008D:digital envelope
> routines:EVP_DigestInit:disabled for fips
>
> 2006.06.06 18:58:32 LOG3[592:1816]: Server is down
> ----END STUNNEL LOG----
Oops! Although my previous reply is valid it isn't the cause of this specific
error. The problem here is the private key format is the OpenSSL "traditional"
form which uses MD5 (a prohibited algorithm) to derive the keys. You need to
convert the key to PKCS#8 format using:
openssl pkcs8 -in key.pem -topk8 -v2 des3 -out pkcs8key.pem
BTW the "user document" is also now online at:
http://www.openssl.org/docs/fips/UserGuide-1.0.pdf