CALG_AES_256 not working on Windows Vista

[Gabi]

Hi,

I'm trying to use CALG_AES_256 algorithm to encrypt files on Windows
Vista. CryptEncrypt call is returning NTE_BAD_ALGID all the times.

The way that I'm initializing CryptoApi is:

if(!CryptAcquireContext(&m_hCryptProv, NULL, MS_ENH_RSA_AES_PROV,
PROV_RSA_AES, CRYPT_VERIFYCONTEXT))
{....

This call is successful all the time.

The way that I'm crating the key is:

if(!CryptCreateHash(m_hCryptProv, HashAlgId, 0, 0, &hKeyHash))
{

err = E_FAIL;
DWORD dwError = GetLastError();
LOG(L"Error during CryptCreateHash!");
}

if (SUCCEEDED(err))
{
// Hash the password string.

if(!CryptHashData(hKeyHash, pbPassword, cbPassword, 0))
{
err = E_FAIL;
LOG(L"GetKey() Error during CryptHashData!");
}
}

if (SUCCEEDED(err))
{

if (!CryptDeriveKey(m_hCryptProv, Algid, hKeyHash,
CRYPT_EXPORTABLE, &hKey))
{
err = E_FAIL;
}
}

Where Algid = CALG_AES_256 and HashAlgId =CALG_SHA_256.

Any calls to CryptEncrypt using hKey created above is failing with
NTE_BAD_ALGID. I have added some code to get the ALGID from hKey and
make sure that the ALGID is CALG_AES_256.

Any ideas?

Thanks,
Gabi

[Gabi]

Could it be a bug in Microsoft Cryptography Provider implementation?
Here is a dump of the Supported algorithms for MS_ENH_RSA_AES_PROV,
PROV_RSA_AES:

Enumerating the supported algorithms

Algid Bits Type Name Algorithm
Length Name
________________________________________________________
00006602h 128 Encrypt 4 RC2
00006801h 128 Encrypt 4 RC4
00006601h 56 Encrypt 4 DES
00006609h 112 Encrypt 13 3DES TWO KEY
00006603h 168 Encrypt 5 3DES
00008004h 160 Hash 6 SHA-1
0000800ch 256 Hash 8 SHA-256
0000800dh 384 Hash 8 SHA-384
0000800eh 512 Hash 8 SHA-512
00008001h 128 Hash 4 MD2
00008002h 128 Hash 4 MD4
00008003h 128 Hash 4 MD5
00008008h 288 Hash 12 SSL3 SHAMD5
00008005h 0 Hash 4 MAC
00002400h 1024 Signature 9 RSA_SIGN
0000a400h 1024 Exchange 9 RSA_KEYX
00008009h 0 Hash 5 HMAC
0000660eh 128 Encrypt 8 AES 128
0000660fh 192 Encrypt 8 AES 192
00006610h 256 Encrypt 8 AES 256

So, all used algorithms looks like being supported but CryptEncrypt
call still return NTE_BAD_ALGID.

I realy need help to find a way to be able to use AES 256....

Thanks,
Gabi

[Novotney@discussions.microsoft.com Peter Novotney]

Hi,

Here is some sample code that encrypts some data using AES 256 and dereiving
the key in a similair fashion to what you show above. It works on my Vista
Sp1 machine using Visual Studio 2008.

int _tmain(int argc, _TCHAR* argv[])
{
HCRYPTPROV hProv = 0;
HCRYPTKEY hKey = 0;
HCRYPTHASH hHash = 0;
DWORD dwCount = 5;
BYTE rgData[512] = {0x01, 0x02, 0x03, 0x04, 0x05};
LPWSTR wszPassword = L"pass";
DWORD cbPassword = (wcslen(wszPassword)+1)*sizeof(WCHAR);

if(!CryptAcquireContext(
&hProv,
NULL,
MS_ENH_RSA_AES_PROV,
PROV_RSA_AES,
CRYPT_VERIFYCONTEXT))
{
printf("Error %x during CryptAcquireContext!\n",
GetLastError());
goto Cleanup;
}

if(!CryptCreateHash(hProv, CALG_SHA_256, 0, 0, &hHash))
{
printf("Error %x during CryptCreateHash!\n", GetLastError());
goto Cleanup;
}

if(!CryptHashData(hHash, (PBYTE)wszPassword, cbPassword, 0))
{
printf("Error %x during CryptHashData!\n", GetLastError());
goto Cleanup;
}

if (!CryptDeriveKey(hProv, CALG_AES_256, hHash,
CRYPT_EXPORTABLE, &hKey))
{
printf("Error %x during CryptDeriveKey!\n", GetLastError());
goto Cleanup;
}

for(DWORD i = 0; i < dwCount; i++)
{
printf("%d ",rgData[i]);
}
printf("\n");

if (!CryptEncrypt(
hKey,
0,
TRUE,
0,
rgData,
&dwCount,
sizeof(rgData)))
{
printf("Error %x during CryptEncrypt!\n", GetLastError());
goto Cleanup;
}

for(DWORD i = 0; i < dwCount; i++)
{
printf("%d ",rgData[i]);
}
printf("\n");

Cleanup:
if(hKey)
{
CryptDestroyKey(hKey);
}
if(hHash)
{
CryptDestroyHash(hHash);
}
if(hProv)
{
CryptReleaseContext(hProv, 0);
}
return 0;
}

Try giving it a run.

[Gabi]

Hi Peter,

Thanks for the code. It works for me and in same time I was able to
identify the issue in my code.The only difference that I had in my
code was to set the key mode to ECB. For some reason, after I set key
mode to ECB, CryptEncrypt call is failing. I'm continue to dig on this
problem.

Thanks,
Gabi

[Peter Novotney]

Hmm are you sure it's the key mode.

I added a CryptSetKeyParam call to set the mode in the sample code I gave
earlier and the CryptEncrypt call still succeded.

DWORD dwMode = CRYPT_MODE_EBC;
if(!CryptSetKeyParam(hKey, KP_MODE, (PBYTE)&dwMode, 0))
{
printf("Error %x during CryptSetKeyParam!\n", GetLastError());
goto Cleanup;
}

What does your CryptSetKeyParam call look like?

[Gabi]

Hi Peter,

In my code I'm setting EBC mode and also KeyIV values, but because of
my mistake, I've call CryptSetKeyParam to set the mode twice and to
set Key IV value twice before a call to CryptEncrypt(). This cause the
problem that I've mention before.

Thanks a lot for your help.
Gabi

On May 10, 12:24 am, Peter Novotney