Simple C Application to Test FIPS Installation

[EJ Hammond]

I'm trying to develop an extremely simple application in C in order to verify my FIPS Object Module installation, but I'm having trouble.

Here is my exact build information and results:
On Windows XP 32bit Operating System
Download & Extract openssl-1.0.1c and openssl-fips-2.0
$cd C:\openssl-fips-2.0
$ms\do_fips
>FIPS BUILD SUCCESS
$out32dll\fips_test_suite
>All tests completed with 0 errors
$cd C:\openssl-1.0.1c
$perl Configure VC-WIN32 fips
>Configured for VC-WIN32
$ms\do_nasm
>completes with no errors
$nmake -f ms\ntdll.mak
>completes with no errors
$nmake -f ms\ntdll.mak test
>passed all tests
$nmake -f ms\ntdll.mak install
>completes with no errors
$cd C:\usr\local\ssl\bin
$openssl version -a
>OpenSSL 1.0.1c-fips...
$echo Hello World > hello.txt
$openssl md5 hello.txt
>works
$set OPENSSL_FIPS=1
$openssl md5 hello.txt
>Error disabled for fips
$openssl sha1 hello.txt
>works
------------------------
At this point, I am confident that my installation is correct.
------------------------

Next, I wanted to create a sample program using C to demonstrate how to call the FIPS_mode_set() function. After searching around, I found some simple code here: http://old.nabble.com/AES-cbc--How-to-Init-Openssl--td12475822.html and adapted it slightly for my needs.

--Begin C code--
#include <stdio.h>
#include <string.h>
#include <openssl\err.h>
#include <openssl\fips.h>
#include <openssl\aes.h>

int main(int argc, char *argv[])
{
//32byte key
unsigned char key[] = {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31};

//16byte Initialization Vector
unsigned char iv[] = {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15};

//plain txt input with padding buffer. Since the AES Block Size is 16bytes and 'crypto' is only 6 bytes, it needs 10 bytes of padding
unsigned char plaintxt[1024]="crypto\x0a\x0a\x0a\x0a\x0a\x0a\x0a\x0a\x0a\x0a";

//output
unsigned char encrypted[1024];

AES_KEY aeskey;

//direct the output of the program to out.txt
freopen ("out.txt","w",stdout);

//Setting up FIPS MODE:
if(FIPS_mode()) //Check if FIPS MODE is enabled already
{
printf("FIPS MODE is already engaged\n");
}
else
{
printf("Attempting to enable FIPS MODE\n");
if(FIPS_mode_set(1))
{
printf("FIPS mode set successful\n");
}
else
{
printf("FIPS mode set failure\n");
ERR_load_crypto_strings();
printf("Loaded Crypto Strings\n");
ERR_print_errors_fp(stderr);
printf("Printed Errors \n");
exit(3);
}
}

memset(encrypted, 0, sizeof(encrypted));

AES_set_encrypt_key(key, 256, &aeskey);

AES_cbc_encrypt(plaintxt, encrypted, 16, &aeskey, iv, AES_ENCRYPT);

printf("%s", encrypted);
fclose (stdout);

return(0);
}
--End C code--

I am using Visual Studio C++ to debug this program. To the search directories for include I have added C:\usr\local\ssl\include and C:\usr\local\ssl\fips-2.0\include. To the search directories for libraries I have added C:\usr\local\ssl\lib and C:\usr\local\ssl\fips-2.0\lib. To the additional dependencies I have added the paths to ssleay32.lib, libeay32.lib, and fipscanister.lib.

On running the program without the FIPS section, I get a nice encrypted message in out.txt which I am successfully able to decrypt from the command line using: openssl aes-256-cbc -d -in out.txt -K 000102030405060708090A0B0C0D0E0F101112131415161718191A1B1C1D1E1F -iv 000102030405060708090A0B0C0D0E0F

I took the block that calls the fips methods directly from the FIPS User Manual 2.0 and added the header files that I thought were necessary.
When I run the program as it appears here, the build is successful, but it exits with an error code of 1.
out.txt reads:
Attempting to enable FIPS MODE
FIPS mode set failure
Loaded Crypto Strings

So Fips_mode() returns 0, then FIPS_mode_set(1) returns 0, then ERR_load_crypto_strings() executes, and the program dies at ERR_print_errors_fp(stderr). It never makes it to the print statement which is directly after ERR_print_errors_fp(stderr).

I've spent days looking around online and debugging this program but, at this point, I'm stumped.

[EJ Hammond]

After working on this over the weekend I have made some headway. Scrolling through the FAQs led me to include applink.c and to call CRYPTO_malloc_init() in my application. Here is the new code with the additions labeled 'EDITED':

--Begin C code--
#include <stdio.h>
#include <string.h>
#include <openssl\err.h>
#include <openssl\fips.h>
#include <openssl\aes.h>

#include <openssl\applink.c> //***********EDITED

int main(int argc, char *argv[])
{
//32byte key
unsigned char key[] = {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31};

//16byte Initialization Vector
unsigned char iv[] = {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15};

//plain txt input with padding buffer. Since the AES Block Size is 16bytes and 'crypto' is only 6 bytes, it needs 10 bytes of padding
unsigned char plaintxt[1024]="crypto\x0a\x0a\x0a\x0a\x0a\x0a\x0a\x0a\x0a\x0a";

//output
unsigned char encrypted[1024];

AES_KEY aeskey;

//direct the output of the program to out.txt
freopen ("out.txt","w",stdout);

//Setting up FIPS MODE:

CRYPTO_malloc_init(); //**********EDITED

if(FIPS_mode()) //Check if FIPS MODE is enabled already
{
printf("FIPS MODE is already engaged\n");
}
else
{
printf("Attempting to enable FIPS MODE\n");
if(FIPS_mode_set(1))
{
printf("FIPS mode set successful\n");
}
else
{
printf("FIPS mode set failure\n");
ERR_load_crypto_strings();
printf("Loaded Crypto Strings\n");
ERR_print_errors_fp(stderr);
printf("Printed Errors \n");
exit(3);
}
}

memset(encrypted, 0, sizeof(encrypted));

AES_set_encrypt_key(key, 256, &aeskey);

AES_cbc_encrypt(plaintxt, encrypted, 16, &aeskey, iv, AES_ENCRYPT);

printf("%s", encrypted);
fclose (stdout);

return(0);
}
--End C code--

Now, the errors are printing as they should and I get:
5652:error:0F06D065:common libcrypto routines:FIPS_mode_set:fips not supported:.\crypto\o_fips.c:92:

I will continue debugging and add any progress to this thread hopefully for someone's benefit in the future.

[EJ Hammond]

I traced the error code:

5652:error:0F06D065:common libcrypto routines:FIPS_mode_set:fips not supported:.\crypto\o_fips.c:92:

back to the file o_fips.c and found this block of code:

--Begin C code--
int FIPS_mode_set(int r)
{
OPENSSL_init();
#ifdef OPENSSL_FIPS
#ifndef FIPS_AUTH_USER_PASS
#define FIPS_AUTH_USER_PASS "Default FIPS Crypto User Password"
#endif
if (!FIPS_module_mode_set(r, FIPS_AUTH_USER_PASS))
return 0;
if (r)
RAND_set_rand_method(FIPS_rand_get_method());
else
RAND_set_rand_method(NULL);
return 1;
#else
if (r == 0)
return 1;//line 92 follows
CRYPTOerr(CRYPTO_F_FIPS_MODE_SET, CRYPTO_R_FIPS_MODE_NOT_SUPPORTED);
return 0;
#endif
}
--End C code--

So it appears that OPENSSL_FIPS is not defined so the code takes the #else branch and sends me the error code on line 92. Okay..

As a test, I amended my code to test if OPENSSL_FIPS was defined:

--Begin C code--
#ifdef OPENSSL_FIPS
printf("\nOPENSSL_FIPS is defined\n");

if(FIPS_mode()) //Check if FIPS MODE is enabled already
{
printf("FIPS MODE is already engaged\n");
}
else
{
printf("Attempting to enable FIPS MODE\n");
if(FIPS_mode_set(1))
{
printf("FIPS mode set successful\n");
}
else
{
printf("FIPS mode set failure\n");
ERR_load_crypto_strings();

ERR_print_errors_fp(stderr);
exit(2);
}
}
#else
printf("OPENSSL_FIPS is not defined");
#endif
--End C code--

Strangely, each time I run my code, it prints the statement "OPENSSL_FIPS is defined" but then it gives me the exact same error as before which indicates that OPENSSL_FIPS is *not* defined.

How is this possible?

[EJ Hammond]

Continuation of this topic has been moved to: http://old.nabble.com/Unusual-Fips-Mode-Set-Failure-tp34235609p34235609.html