FIPS Failure on newer 32-bit Windows platforms.

[Gwendolyn Hunt]

Folks,
I have run into an issue with FIPS Object Module that has me stumped. I have a multi-platform non-gui C++ application that uses boost::asio with OpenSSL under the covers for network communications. It is mature code over two years old. For Windows it is built on Windows XPSP3 for both 32-bit and 64-bit platforms and for Linux on RHEL5.3 for 32-bit and 64-bit platforms. Nightly, we run extended automated tests on all our platforms. For Windows we test the application on both 32-bit and 64-bit variants of Windows XP, 2003, 2008, Vista and Windows 7. The automated testing heavily exercises SSL communications.

A couple of weeks ago I updated OpenSSL to v1.0.1c and added the FIPS Object Module v2.0.1. After building a FIPS Capable OpenSSL as specified it was straightforward to add FIPS mode to my application. The change was seamless with full operation on all Linux platforms, all Windows 64-bit platforms but we had issues with FIPS on 32-bit Windows 2008, Vista and Windows 7 platforms. It works on 32-bit Windows XP and 2003 but not on the newer 32-bit versions of Windows. I tried all combinations of static and dll linkage, release and debug builds and the results were 100% deterministic: On the newer Windows 32-bit platforms FIPS_mode_set() always fails with error: "error:2D06B06F:FIPS routines:FIPS_check_incore_fingerprint:fingerprint does not match".

Next I tried both OpenSSL 1.0.1d and 1.0.1e with FIPS 2.0.2 and had the same results. Lastly, to make sure it wasn't my application that is causing the issue I wrote a simple test application that does one thing, call FIPS_mode_set(). The failure pattern was exactly the same. Works on all 64-bit Windows, works on 32-bit Windows XP and 2003 but fails on Windows 2008, Vista and Windows 7.

Anyone have any ideas?

Thanks,
Gwen
--
Gwendolyn Hunt
Senior Software Engineer
gh...@tripwire.com

______________________________________________________________________
OpenSSL Project http://www.openssl.org
User Support Mailing List openss...@openssl.org
Automated List Manager majo...@openssl.org

[Dr. Stephen Henson]

On Sat, Feb 16, 2013, Gwendolyn Hunt wrote:

> Folks,
> I have run into an issue with FIPS Object Module that has me stumped. I have a multi-platform non-gui C++ application that uses boost::asio with OpenSSL under the covers for network communications. It is mature code over two years old. For Windows it is built on Windows XPSP3 for both 32-bit and 64-bit platforms and for Linux on RHEL5.3 for 32-bit and 64-bit platforms. Nightly, we run extended automated tests on all our platforms. For Windows we test the application on both 32-bit and 64-bit variants of Windows XP, 2003, 2008, Vista and Windows 7. The automated testing heavily exercises SSL communications.
>
> A couple of weeks ago I updated OpenSSL to v1.0.1c and added the FIPS Object Module v2.0.1. After building a FIPS Capable OpenSSL as specified it was straightforward to add FIPS mode to my application. The change was seamless with full operation on all Linux platforms, all Windows 64-bit platforms but we had issues with FIPS on 32-bit Windows 2008, Vista and Windows 7 platforms. It works on 32-bit Windows XP and 2003 but not on the newer 32-bit versions of Windows. I tried all combinations of static and dll linkage, release and debug builds and the results were 100% deterministic: On the newer Windows 32-bit platforms FIPS_mode_set() always fails with error: "error:2D06B06F:FIPS routines:FIPS_check_incore_fingerprint:fingerprint does not match".
>
> Next I tried both OpenSSL 1.0.1d and 1.0.1e with FIPS 2.0.2 and had the same results. Lastly, to make sure it wasn't my application that is causing the issue I wrote a simple test application that does one thing, call FIPS_mode_set(). The failure pattern was exactly the same. Works on all 64-bit Windows, works on 32-bit Windows XP and 2003 but fails on Windows 2008, Vista and Windows 7.
>
> Anyone have any ideas?
>

It could the the ASLR issue. To address that you need to turn off ASLR in your
application or the FIPS capable OpenSSL build. The /fixed option to the linker
will do this: see the archives for more details.

I thought we did this by default now in the FIPS capable OpenSSL for 32 bit
windows builds: apparently not.

Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org

[Gwendolyn Hunt]

Thanks Stephen and Jeff:

We suspected some address changing going on. I rebuilt FIPS Capable OpenSSL creating static libs. But the only way I could see setting the LINK flags was modifying ms\nt.mak adding "/DYNAMICBASE:NO /FIXED" to LFLAGS. I tried /FIXED at first and that didn't do it for me. But I could have done something wrong there. I'm not sure how to set the link flags and stay compliant with the prescribed build process (we want to leverage the validation for the FIPS Object Module).

Also, added the same arguments to my link phase call to "fipslink.pl" and the cmake link flags I use to build my application.

Thanks for pointing me in the right direction!

Cheers,
Gwen

Gwendolyn Hunt | Senior Software Engineer

TRIPWIRE | CONFIDENCE: SECURED
www.tripwire.com

[Dr. Stephen Henson]

On Sat, Feb 16, 2013, Gwendolyn Hunt wrote:

> Thanks Stephen and Jeff:
>
> We suspected some address changing going on. I rebuilt FIPS Capable OpenSSL
> creating static libs. But the only way I could see setting the LINK flags
> was modifying ms\nt.mak adding "/DYNAMICBASE:NO /FIXED" to LFLAGS. I tried
> /FIXED at first and that didn't do it for me. But I could have done
> something wrong there. I'm not sure how to set the link flags and stay
> compliant with the prescribed build process (we want to leverage the
> validation for the FIPS Object Module).
>
>

The prescribed build process only applies to the FIPS module itself and not
the FIPS capable OpenSSL. So you can modify the build procedure of the FIPS
capable OpenSSL (within reason) without affecting the validation.

Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org