Crypto++ static library crashed in X64 of Visyal Studio 2017

[Kevin C]

Hello Everyone,

Could someone do me a favor to help it out?

With my 64bit Window7 machine, 

I can build Crypto++ static library in X86 in visual studio 2017, and use it without any issue,

so I built  Crypto++ static library in X64 within visual studio 2017, and tried to use it with 64bit application,

It always crashed on the line m_k[i] = a+b with error message :

System.AccessViolationException: 'Attempted to read or write protected memory. This is often an indication that other memory is corrupt.'

Below is the section of codes when crashed,  the Crypto++ is the latest source.2016.

void Twofish::Base::UncheckedSetKey(const byte *userKey, unsigned int keylength, const NameValuePairs &)

{

AssertValidKeyLength(keylength);

unsigned int len = (keylength <= 16 ? 2 : (keylength <= 24 ? 3 : 4));

SecBlock<word32> key(len*2);

GetUserKey(LITTLE_ENDIAN_ORDER, key.begin(), len*2, userKey, keylength);

unsigned int i;

for (i=0; i<40; i+=2)

{

word32 a = h(i, key, len);

word32 b = rotlFixed(h(i+1, key+1, len), 8);

m_k[i] = a+b;

m_k[i+1] = rotlFixed(a+2*b, 9);

}

.............................................................

Did I miss any setting? 

Thanks so much for any help.

Kevin

[Jeffrey Walton]

With my 64bit Window7 machine, 

I can build Crypto++ static library in X86 in visual studio 2017, and use it without any issue,

so I built  Crypto++ static library in X64 within visual studio 2017, and tried to use it with 64bit application,

It always crashed on the line m_k[i] = a+b with error message :

System.AccessViolationException: 'Attempted to read or write protected memory. This is often an indication that other memory is corrupt.'

Below is the section of codes when crashed,  the Crypto++ is the latest source.2016.

void Twofish::Base::UncheckedSetKey(const byte *userKey, unsigned int keylength, const NameValuePairs &)

{

AssertValidKeyLength(keylength);

unsigned int len = (keylength <= 16 ? 2 : (keylength <= 24 ? 3 : 4));

SecBlock<word32> key(len*2);

GetUserKey(LITTLE_ENDIAN_ORDER, key.begin(), len*2, userKey, keylength);

unsigned int i;

for (i=0; i<40; i+=2)

{

word32 a = h(i, key, len);

word32 b = rotlFixed(h(i+1, key+1, len), 8);

m_k[i] = a+b;

m_k[i+1] = rotlFixed(a+2*b, 9);

}

That code is unchanged for ages. See, for example, Crypto++ 5.5 from 2007 at  https://github.com/weidai11/cryptopp/blob/0f71ab33118145a6185221715129190193aed1b5/twofish.cpp#L52 .

Did I miss any setting?

The problem is likely in the code that is using Twofish. We need to see the userland code around setting the key. Some_Mode<Twofish>::Encryption (and decryption) calls UncheckedSetKey to set the key.

If the calling code looks OK, then please check Master for the problem. I don't recall a problem that would affect the 5.6.5 ZIP but not affect Master. However, I could be wrong.

You can verify the library is OK by running the self tests. Build and run the self tests from Visual Studio or from the command line according to https://www.cryptopp.com/wiki/MSBuild_(Command_Line).

Jeff

[Kevin C]

Jeff,

Thank you so much for your reply.

My C# codes passed parameters with  the key buffKey =  "r8Mkzz23Uq v~}3t" , int iKeyLength = 16, char* pBuff = "7A9867EFC505A424BB9787390F501140845A78ACF11970F41DEF08695DB137B4490D64A1C32DE3BE2B67DDFC7C94FF25490D64A1C32DE3BE2B67DDFC7C94FF25490D64A1C32DE3BE2B67DDFC7C94FF25490D64A1C32DE3BE2B67DDFC7C94FF25490D64A1C32DE3BE2B67DDFC7C94FF25490D64A1C32DE3BE2B67DDFC7C94FF25",int iSrcLength = 256; to unmanged C function DecryptTwofishProcess below,

I shouldl  get  byte* pOut out ,

My codes below, 

void UnManagedCrypto::DecryptTwofishProcess(const byte *buffKey,int iKeyLength,char* pBuff,int iSrcLength, byte* pOut)

{

int i, iLength = 17;

int  iBlockLength = 16;

byte *pSrc;

char *pStr;

int iTemp;

CryptoPP::TwofishDecryption Encr(buffKey, iKeyLength); // crashed inside.

pStr = new char[iSrcLength];

memset((void*)pStr, 0, iSrcLength);

From the line above CryptoPP::TwofishDecryption Encr(buffKey, iKeyLength), call 

BlockCipherFinal(const byte *key, size_t length)

{this->SetKey(key, length);}

then call void Twofish::Base::UncheckedSetKey(const byte *userKey, unsigned int keylength, const NameValuePairs &)

{

AssertValidKeyLength(keylength);

unsigned int len = (keylength <= 16 ? 2 : (keylength <= 24 ? 3 : 4));

SecBlock<word32> key(len*2);

GetUserKey(LITTLE_ENDIAN_ORDER, key.begin(), len*2, userKey, keylength);

unsigned int i;

for (i=0; i<40; i+=2)

{

word32 a = h(i, key, len);

word32 b = rotlFixed(h(i+1, key+1, len), 8);

m_k[i] = a+b; //------------------Crashed here.

That crashed on the line above

The same codes work for 32 bit, even I used the same codes to build 64 bit on 32-bit OS with crypto++561,

we have deployed that 64 bit application, it works on 64 bit machine too.

Thanks for any help.

kevin

[Jeffrey Walton]

On Wednesday, August 30, 2017 at 6:25:16 PM UTC-4, Kevin C wrote:

My C# codes passed parameters with  the key buffKey =  "r8Mkzz23Uq v~}3t" , int iKeyLength = 16, char* pBuff = "7A9867EFC505A424BB9787390F501140845A78ACF11970F41DEF08695DB137B4490D64A1C32DE3BE2B67DDFC7C94FF25490D64A1C32DE3BE2B67DDFC7C94FF25490D64A1C32DE3BE2B67DDFC7C94FF25490D64A1C32DE3BE2B67DDFC7C94FF25490D64A1C32DE3BE2B67DDFC7C94FF25490D64A1C32DE3BE2B67DDFC7C94FF25",int iSrcLength = 256; to unmanged C function DecryptTwofishProcess below,

I shouldl  get  byte* pOut out ,

Sorry, I'm not experienced with C# interop. Its been years since I worked with it.

Maybe the folks on Stack Overflow can help.

Jeff