CryptEncrypt returns different encrypted strings on different Windows systems

Victor

unread,

Aug 12, 2005, 4:01:40 PM8/12/05

to

Hi everyone,

CryptEncrypt returns different encrypted strings on different Windows
systems when the same user password input string is used . This
creates a problem for us because user can't login to our application
if their passwords are encrypted.

Here is our scenario: All our user passwords have initially encrypted
on a Win2K3 Server and stored in a SQL Server database on the same
computer. Our application is then running on either Win2K3-based,
Win2k-based, WinXP-based, or Win9x-based computers that encrypts and
verifies user login password against the database on the Win2k3 server.

I attached the source code of password encryption that we use. We use
CALG_RC4, but not CALG_RC2. So adding a CryptSetKeyParam call with
passing parameter set to KP_EFFECTIVE_KEYLEN may not be required in my
case.

Any helps would be greately appreciated.

Victor

Source Code:

#include "stdafx.h"
#define _WIN32_WINNT 0x0400
#include <wincrypt.h>
#include "CryptString.h"

CCryptString::CCryptString()
{
m_bInitOK = FALSE;
m_bInitOK = SetupCryptoClient();
}

BOOL CCryptString::SetupCryptoClient()
{

HCRYPTPROV hProv;
HCRYPTKEY hKey;

if ( !CryptAcquireContext( &hProv, NULL, "Microsoft Base
Cryptographic Provider v1.0\0", PROV_RSA_FULL, 0 ) )
{
if (!CryptAcquireContext(&hProv, NULL, MS_DEF_PROV, PROV_RSA_FULL,
CRYPT_NEWKEYSET))
{
return FALSE;
}
}

if (!CryptGetUserKey(hProv, AT_SIGNATURE, &hKey))
{
if (GetLastError() == NTE_NO_KEY)
{
if (!CryptGenKey(hProv, AT_SIGNATURE, 0, &hKey))
{
CryptReleaseContext(hProv, 0);
return FALSE;
}
else
{
CryptDestroyKey(hKey);
}
}
else
{
CryptReleaseContext(hProv, 0);
return FALSE;
}
}

if (!CryptGetUserKey(hProv,AT_KEYEXCHANGE,&hKey))
{
if (GetLastError()==NTE_NO_KEY)
{
if (!CryptGenKey(hProv,AT_KEYEXCHANGE,0,&hKey))
{
CryptReleaseContext(hProv, 0);
return FALSE;
}
else
{
CryptDestroyKey(hKey);
}
}
else
{
CryptReleaseContext(hProv, 0);
return FALSE;
}
}

CryptReleaseContext(hProv, 0);
return TRUE;
}

BOOL CCryptString::EncryptString( TCHAR* szPassword, TCHAR*
szEncryptPwd, TCHAR *szKey )
{

if (!m_bInitOK)
return FALSE;

BOOL bResult = TRUE;
HKEY hRegKey = NULL;
HCRYPTPROV hProv = NULL;
HCRYPTKEY hKey = NULL;
HCRYPTKEY hXchgKey = NULL;
HCRYPTHASH hHash = NULL;
DWORD dwLength;

if ( CryptAcquireContext( &hProv, NULL, "Microsoft Base Cryptographic
Provider v1.0\0", PROV_RSA_FULL, 0 ) )
{
if (CryptCreateHash(hProv, CALG_MD5, 0, 0, &hHash))
{
dwLength = sizeof(TCHAR)*_tcslen(szKey);
if (CryptHashData(hHash, (BYTE *)szKey, dwLength, 0))
{
if (CryptDeriveKey(hProv, CALG_RC4 , hHash, CRYPT_EXPORTABLE,
&hKey))
{
dwLength = sizeof(TCHAR)*_tcslen(szPassword);
BYTE *pbBuffer = (BYTE *)malloc(dwLength);
if (pbBuffer != NULL)
{
memcpy(pbBuffer, szPassword, dwLength);
if (CryptEncrypt(hKey, 0, TRUE, 0, pbBuffer, &dwLength,
dwLength))
{
memcpy(szEncryptPwd, pbBuffer, dwLength);

}
else
{
bResult = FALSE;
}
free(pbBuffer);
}
else
{
bResult = FALSE;
}
CryptDestroyKey(hKey);
}
else
{
bResult = FALSE;
}
}
else
{
bResult = FALSE;
}
CryptDestroyHash(hHash);
}
else
{
bResult = FALSE;
}
CryptReleaseContext(hProv, 0);
}
return bResult;
}

Victor

unread,

Aug 12, 2005, 4:02:52 PM8/12/05

to

Hi Everyone,

CryptEncrypt returns different encrypted strings on different Windows

systems when the same user password input string is used. This creates

a problem for us because user can't login to our application if their
passwords are encrypted.

Here is our scenario: All our user passwords have initially encrypted
on a Win2K3 Server and stored in a SQL Server database on the same
computer. Our application is then running on either Win2K3-based,
Win2k-based, WinXP-based, or Win9x-based computers that encrypts and
verifies user login password against the database on the Win2k3 server.

I attached the source code of password encryption that we use. We use

CALG_RC4, but not CALG_RC2, so I think adding a CryptSetKeyParam call

with passing parameter set to KP_EFFECTIVE_KEYLEN may not be required
in my case.

Any helps would be greately appreciated.

Victor

Souce Code:

John Banes

unread,

Aug 14, 2005, 12:34:01 AM8/14/05

to

The default length of RC4 keys has changed in the various releases of
Windows. It used to be 40 bits, and it's now 128 bits by default, if I
remember correctly. That's probably what's causing your problem. To manually
specify the key length, you need to OR in the key size in the upper word of
the dwFlags field. This should be explained in MSDN.

if (CryptDeriveKey(hProv, CALG_RC4 , hHash, (128 << 16) | CRYPT_EXPORTABLE,
&hKey))

"Victor" <weike...@yahoo.com> wrote in message
news:1123874098....@g14g2000cwa.googlegroups.com...

Victor

unread,

Aug 15, 2005, 2:11:11 PM8/15/05

to

Hi John,

You've been a big help to me and I'm very grateful for your time and
effort in helping me.

I know at least what's the problem now after you explained it to me,
but I still have no
slightest idea why the CryptDeriveKey call will fail if I add the ( 128
<< 16 ) bitwise-OR operation with
the CRYPT_EXPORTABLE as the 'dwFlags' value. The GetLastError()
returned an error code of 87 on Win98 and 2148073481, which is
NTE_BAD_FLAGS, on Win2k3.

Do this has to do with the CryptGenKey( hProv, AT_SIGNATURE, 0, &hKey )
call from
the SetupCryptoClient() in my source code? I'm new to Crypto API so
please bear with me.

Many Thanks,

Victor

unread,

Aug 15, 2005, 2:14:34 PM8/15/05

to

Hi John,

You've been a big help to me and I'm very grateful for your time and
effort in helping me.

I know at least what's the problem now after you explained it to me,
but I still have no
slightest idea why the CryptDeriveKey call will fail if I add the ( 128
<< 16 ) bitwise-OR operation with the CRYPT_EXPORTABLE as the 'dwFlags'
value. The GetLastError()
returned an error code of 87 on Win98 and 2148073481, which is
NTE_BAD_FLAGS, on Win2k3.

Do this has to do with the CryptGenKey( hProv, AT_SIGNATURE, 0, &hKey )
call from

the SetupCryptoClient() in my source code? I'm new to Microsoft Crypto