Seg fault in x64dll.asm in x64 Release configuration

[Andrew Bromba]

Hello,

I'm experiencing strange behaviour using AES 256Bit encryption in my
VS2005SP1 Professional projects. I'm running a Core 2 Quad Q6600 with
Windows Vista Business 64Bit. My crypto++ calling code looks like
this:

CFB_Mode<AES>::Encryption aes(keyBuffer, keyBufferLength, IV);

(using CBC_Mode leads to the same result). Everything works fine in
x64 Debug version, but in x64 Release version a segmentation fault
occurs in x64dll.asm on line 93. The problem seems to be, that

mov rdi, QWORD PTR [?g_cacheLineSize@CryptoPP@@3IA]

in line 69 fills "rdi" with 0x4079dbf000000040 instead of
0x0000000000000040. Also note, that the seg fault occurs upon
instantiation of the AES encryption object (see stack trace).

I saw that g_cacheLineSize is a word32 - could this be the problem? Am
I missing a preprocessor definition?

Inserting the line

printf("c: %x %x %x\n", *(((unsigned int *) &g_cacheLineSize) - 1),
(unsigned int) g_cacheLineSize, *(((unsigned int *) &g_cacheLineSize)
+ 1));

in rijndael.cpp:974 revealed, that 0x4079dbf0 is right next to
g_cacheLineSize == 0x00000040 in memory, the output on the command
line was
"c: 0 40 4079dbe0".

What am I doing wrong?

Kind regards,
Andrew Bromba


Stack trace:
> iddedit.exe!CryptoPP::Rijndael::Enc::AdvancedProcessBlocks(const unsigned char * inBlocks=0x0000000000a2f260, const unsigned char * xorBlocks=0x0000000000000000, unsigned char * outBlocks=0x0000000000a2f2b0, unsigned __int64 length=0x0000000000000010, unsigned int flags=0x00000000) Line 977 C++
iddedit.exe!CryptoPP::Rijndael::Enc::ProcessAndXorBlock(const
unsigned char * inBlock=0x0000000000a2f260, const unsigned char *
xorBlock=0x0000000000000000, unsigned char *
outBlock=0x0000000000a2f2b0) Line 289 C++
iddedit.exe!CryptoPP::CFB_ModePolicy::TransformRegister() Line 52 C+
+
iddedit.exe!CryptoPP::CFB_ModePolicy::CipherResynchronize(const
unsigned char * iv=0x000000000012f660, unsigned __int64
length=0x0000000000000010) Line 61 C++
iddedit.exe!
CryptoPP::CFB_CipherTemplate<CryptoPP::AbstractPolicyHolder<CryptoPP::CFB_CipherAbstractPolicy,CryptoPP::CFB_ModePolicy>
>::UncheckedSetKey(const unsigned char * key=0x0000000000a2b870,
unsigned int length=0x00000020, const CryptoPP::NameValuePairs &
params={...}) Line 165 C++
iddedit.exe!CryptoPP::SimpleKeyingInterface::SetKey(const unsigned
char * key=0x0000000000a2b870, unsigned __int64
length=0x0000000000000020, const CryptoPP::NameValuePairs & params=
{...}) Line 57 C++
iddedit.exe!
CryptoPP::CipherModeFinalTemplate_CipherHolder<CryptoPP::BlockCipherFinal<0,CryptoPP::Rijndael::Enc>,CryptoPP::ConcretePolicyHolder<CryptoPP::Empty,CryptoPP::CFB_EncryptionTemplate<CryptoPP::AbstractPolicyHolder<CryptoPP::CFB_CipherAbstractPolicy,CryptoPP::CFB_ModePolicy>
>,CryptoPP::CFB_CipherAbstractPolicy>
>::CipherModeFinalTemplate_CipherHolder<CryptoPP::BlockCipherFinal<0,CryptoPP::Rijndael::Enc>,CryptoPP::ConcretePolicyHolder<CryptoPP::Empty,CryptoPP::CFB_EncryptionTemplate<CryptoPP::AbstractPolicyHolder<CryptoPP::CFB_CipherAbstractPolicy,CryptoPP::CFB_ModePolicy>
>,CryptoPP::CFB_CipherAbstractPolicy> >(const unsigned char *
key=0x0000000000a2b870, unsigned __int64 length=0x0000000000000020,
const unsigned char * iv=0x000000000012f660) Line 264 + 0xae bytes C+
+
...

[Andrew Bromba]

Hello again,

changing the line 69 in x64dll.asm from

mov rdi, QWORD PTR [?g_cacheLineSize@CryptoPP@@3IA]

to

mov edi, DWORD PTR [?g_cacheLineSize@CryptoPP@@3IA]

solved the problem for me. Please confirm that what I am doing makes
sense, because I am not used to assembly language:

- g_cacheLineSize is a word32 -> DWORD PTR [...] tells the compiler
about it
- edi is 32Bit access to rdi register -> the high order bits in rdi
are zero-padded which is what I want

Best regards,

Andrew Bromba

[Wei Dai]

Yes, that should be the right fix. Thanks for reporting this bug and
providing the fix.

--------------------------------------------------
From: "Andrew Bromba" <Andrew...@gmx.net>
Sent: Friday, July 03, 2009 2:59 AM
To: "Crypto++ Users" <cryptop...@googlegroups.com>
Subject: Re: Seg fault in x64dll.asm in x64 Release configuration

[Robert Roessler]

Wei Dai wrote:
> Yes, that should be the right fix. Thanks for reporting this bug and
> providing the fix.

Is this change going into the SVN repo?

> --------------------------------------------------
> From: "Andrew Bromba"<Andrew...@gmx.net>
> Sent: Friday, July 03, 2009 2:59 AM
> To: "Crypto++ Users"<cryptop...@googlegroups.com>
> Subject: Re: Seg fault in x64dll.asm in x64 Release configuration
>
>>
>> Hello again,
>>
>> changing the line 69 in x64dll.asm from
>>
>> mov rdi, QWORD PTR [?g_cacheLineSize@CryptoPP@@3IA]
>>
>> to
>>
>> mov edi, DWORD PTR [?g_cacheLineSize@CryptoPP@@3IA]
>>
>> solved the problem for me. Please confirm that what I am doing makes
>> sense, because I am not used to assembly language:

Robert

[Wei Dai]

I've just checked in the fix.

--------------------------------------------------
From: "Robert Roessler" <roes...@rftp.com>
Sent: Sunday, July 05, 2009 12:45 PM
To: "Wei Dai" <wei...@weidai.com>
Cc: "Crypto++ Users" <cryptop...@googlegroups.com>