Compatibility between .NET and CryptoAPI RC2 Encryption

[Marco Padovani]

I am trying to encrypt and decrypt text identically in both unmanaged
C++ using the CryptoAPI functions and C# using .NET. They both work by
themselves, but neither can decrypt the other's encryption. I have
read article pointing to little-endian vs. big-endian issues, but I
have not been able to resolve the problem in my code. Can anyone show
me what's wrong with my code?

Here are extracted snippets of the two versions.

Thanks in advance,

Marco

---------------

.NET C#:
--------
public string EncryptString(
string strInput,
string strPassword)
{

// Convert string to byte array
ASCIIEncoding encCur = new ASCIIEncoding();
byte[] encodedInput = encCur.GetBytes(strInput);

// Get crypto provider and create hash
byte[] encodedPassword = encCur.GetBytes(strPassword);
MD5CryptoServiceProvider MD5Crypto = new MD5CryptoServiceProvider();
byte[] hashedPassword = MD5Crypto.ComputeHash(encodedPassword);

// Create a zeroed-out initialization vector (for now).
byte[] IV = new byte[8];

// Alternative way of getting hashed password (?)
PasswordDeriveBytes pderiver = new PasswordDeriveBytes(strPassword,
null);
byte[] ivZeros = new byte[8];
byte[] pbeKey = pderiver.CryptDeriveKey("RC2", "MD5", 128, ivZeros);

// Create the the encryption service provider object.
RC2CryptoServiceProvider RC2 = new RC2CryptoServiceProvider();

// Get an encryptor.
// (Note that I also tried reversing the bytes of the hashedPassword
here, but
// that did not work.)
ICryptoTransform encryptor = RC2.CreateEncryptor(hashedPassword, IV);

// Prepare to encrypt the data.
MemoryStream msEncrypt = new MemoryStream();
CryptoStream csEncrypt = new CryptoStream(msEncrypt, encryptor,
CryptoStreamMode.Write);

// Convert the input string to a byte array.
byte[] toEncrypt = encCur.GetBytes(strInput);

// Write all data to the crypto stream and flush it.
csEncrypt.Write(toEncrypt, 0, toEncrypt.Length);
csEncrypt.FlushFinalBlock();

// Get encrypted array of bytes.
byte[] encrypted = msEncrypt.ToArray();

// At this point the above "encrypted" array does not match the
CryptoAPI version

...
} // public string EncryptString()

---------------------------------------------------

CryptoAPI C++:
--------------

CString EncryptString(
char* pszSource,
char* pszPWD)
{

// Get a handle to the default Crypto provider.
HCRYPTPROV hProv = NULL;
CryptAcquireContext(&hProv, "MyContainer", MS_DEF_PROV,
PROV_RSA_FULL, CRYPT_MACHINE_KEYSET);

// Derive a session key by creating a one-way hash of the password.
HCRYPTKEY hKey = NULL;
BYTE IV[8] = {0,0,0,0,0,0,0,0};
HCRYPTHASH hHash = NULL;
CryptCreateHash(hProv, CALG_MD5, 0, 0, &hHash);
CryptHashData(hHash, (BYTE*) pszPWD, strlen(pszPWD), 0);
CryptHashData(hHash, (BYTE*) pszPWD, strlen(pszPWD), 0);

// Extract out hash so we can compare it with the .NET version in a
breakpoint
BYTE* pbHashData = NULL;
DWORD dwHashSize = 0;
DWORD dwHashDataLen = sizeof(DWORD);
CryptGetHashParam(hHash, HP_HASHSIZE, (BYTE*) &dwHashSize,
&dwHashDataLen, 0);
dwHashDataLen = dwHashSize;
pbHashData = new BYTE[dwHashDataLen];
CryptGetHashParam(hHash, HP_HASHVAL, pbHashData, &dwHashDataLen, 0);

// Derive the key from the hashed password.
CryptDeriveKey(hProv, CALG_RC2, hHash, 0, &hKey);

// Create an initialization vector (for now initializing to zeroes).
BYTE pbRandomData[8] = {0, 0, 0, 0, 0, 0, 0, 0};

// Set the initialization vector (IV) in the key.
CryptSetKeyParam(hKey, KP_IV, IV, 0);

// Get the length of the input string.
DWORD dwSourceLen = strlen(pszSource);

// Determine the length of the buffer to hold the corresponding
cyphertext.
DWORD dwDataLen = dwSourceLen;
BYTE* pTarget = NULL;
DWORD dwCryptDataLen = dwDataLen;
CryptEncrypt(hKey, 0, TRUE, 0, NULL, &dwCryptDataLen, dwDataLen);

// Allocate memory for the target cyphertext buffer.
pSource = new BYTE[(UINT) dwCryptDataLen + 1];

// Encrypt the plaintext string (in place).
strcpy((char*) pTarget, pszSource);
dwDataLen = dwCryptDataLen;
dwCryptDataLen = dwSourceLen;
CryptEncrypt(hKey, 0, TRUE, 0, pTarget, &dwCryptDataLen, dwDataLen);

// At this point the above "pTarget" array does not match the .NET
version

...

// Cleanup
delete []pbHashData;
CryptDestroyKey(hKey);
CryptReleaseContext(hProv, 0);
delete []pSource;

...
} // CString EncryptString()

[Mike B]

There might be a problem with default keys sizes being different.

Try explicitly setting the key size using CryptSetKeyParam to see if that
makes a difference.

regards
Mike

"Marco Padovani" <MPad...@pdssoftware.com> wrote in message
news:9a288e48.04111...@posting.google.com...

[Ryan Menezes [MSFT]]

Crossposting to microsoft.public.dotnet.security

You might also want to have a look at the KP_EFFECTIVE_KEYLEN parameter to
CryptSetKeyParam if it is relevant.

Are you running the C# and C++ programs on the same platform ?
--
Thanks,
Ryan Menezes [MS]
This posting is provided "AS IS" with no warranties, and confers no rights.

"Mike B" <mbri...@wol.co.za> wrote in message
news:#GhVWTVy...@TK2MSFTNGP11.phx.gbl...

[Rob Teixeira]

OK, you have a lot of stuff going on, so best to try narrowing one step at a
time.
First of all, are you sure you're getting the same session Key?
Instead of derriving it, try using the same binary data on both sides (hard
code it for the test).

Secondly, is there any reason you need to use RC2? There are a lot of funny
behaviours of this cipher, depending on the provider - not to mention that
it's not a very safe cipher to begin with. Just as an example:
http://support.microsoft.com/kb/841715
I'd suggest moving to 3DES or AES (Rijndael) if possible, but if that's not
possible, try using the MS Enhanced or MS Strong provider instead of the
default provider to see if that changes anything. I'm fairly certain the
default key size is affected by this, and I'm pretty certain the .NET code
doesn't use the base provider.

Also, by default, the .NET RC2 class (which actually uses the CryptoAPI
under the covers), defaults to CBC mode with PKCS#7 padding for input plain
text. It also uses an *effective* key size of 128 bits by default (something
Mike and Ryan may be alluding to). The Base provider by default, probably
uses only 40.

Anyway just a few things to check out. And I really stress the part about
avoiding the RC2 cipher if possible.

-Rob Teixeira

"Ryan Menezes [MSFT]" <rya...@online.microsoft.com> wrote in message
news:Olk1Ljfy...@TK2MSFTNGP15.phx.gbl...

[Marco Padovani]

Thanks Rob, Ryan, and Mike! Your ideas were right on the money.

(For those people struggling with similar issues) I made the following
code changes to make the two code snippets work identically. Both
changes were in the C++ CryptoAPI code:

1. At the beginning, in CryptAcquireContext(), I changed the
MS_DEF_PROV parameter to MS_ENHANCED_PROV. (Without this the next
change failed because the base MS provider does not allow effective
key sizes greater than 40.)

2. After the CryptDeriveKey() call, I added the following call:

DWORD dwKeyLen = 128;
CryptSetKeyParam(rhKey, KP_EFFECTIVE_KEYLEN, (BYTE*) &dwKeyLen, 0);

This set effective key length to match the .NET call's length.

(Finally, to answer the specific questions the three of you asked in
your responses to my original request:)
* I am running both on the same platform.
* I picked RC2 for no reason other than some original legacy code used
it. However, I will look into your suggestion of using another.
Thanks.

Again, thank you all for your help.

"Rob Teixeira" <RobTeixeira@@msn.com> wrote in message news:<OJmLXah...@TK2MSFTNGP11.phx.gbl>...

> > > > read articles pointing to little-endian vs. big-endian issues, but I