wrong SHA-256 result with optimized assembly on i386 ([pycryptopp] #24: SHA256 failure on NetBSD with multiple segments)

[Zooko O'Whielacronx]

Folks:

A contributor to the open source Tahoe-LAFS project (Midnight Magic)
has written a test script which shows wrong output from SHA-256 if you
use Crypto++ v5.6.0 on i386 (not amd64) with optimized assembly turned
on, and you hash something with SHA-256 in multiple updates, the total
size of which is a multiple of 128 bytes, and the first update of
which is an odd number of bytes less than 64. :-)

You can find his test script attached to the ticket, below. While
awaiting a fix from the upstream Crypto++ folks, we might see if we
can disable optimized assembly, at least on i386, so that we can
proceed with the imminent Tahoe-LAFS v1.5.0 release.

The whole ticket includes some debugging from a different contributor
(Black Dew) who observed that Crypto++ uses the MOVDQA instruction on
his AthlonXP, which valgrind thinks is invalid and which wikipedia
says belongs in SSE2 (which AthlonXP doesn't fully support).

Thanks!

Regards,

Zooko

---------- Forwarded message ----------
From: pycryptopp <tr...@allmydata.org>
Date: Fri, Jul 3, 2009 at 1:51 PM
Subject: Re: [tahoe-dev] [pycryptopp] #24: SHA256 failure on NetBSD
with multiple segments
To:
Cc: taho...@allmydata.org

#24: SHA256 failure on NetBSD with multiple segments
----------------------+-----------------------------------------------------
Reporter:  warner     |           Owner:  bdew
   Type:  defect     |          Status:  new
Priority:  critical   |         Version:  0.5.1
Keywords:  integrity  |   Launchpad_bug:
----------------------+-----------------------------------------------------

Comment(by midnightmagic):

 Replying to [comment:17 midnightmagic]:
 > I have written a test program for crypto++ which demonstrates the same
 problem for a SHA256:
 > the results differ based on whether the input is split up on an odd
 boundary or not. I have
 > compiled my crypto++ library with and (I believe) without the optimized
 assembly routines.

 I lied. Manually disabling the optimized assembly makes my sample program
 output the correct results. Here are the details:

 {{{ g++ -DCRYPTOPP_DISABLE_X86ASM -DNDEBUG -g -O2 -DCRYPTOPP_DISABLE_SSSE3
 -pipe -I. -c mytest2.cpp && g++ -o mytest2 mytest2.o -L. -lcryptopp &&
 ./mytest2 }}}

 will work and results in:

 {{{ STRING: aaa:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
 (etc)
 SHA256: 6836CF13BAC400E9105071CD6AF47084DFACAD4E5E302C94BFED24E013AFB73E
 STRING: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
 (etc)
 SHA256: 6836CF13BAC400E9105071CD6AF47084DFACAD4E5E302C94BFED24E013AFB73E
 }}}

 Just build crypto++ and then plunk mytest2.cpp in the same directory as
 the libcryptopp.a was built in, and run the above.

 {{{ g++ -DNDEBUG -g -O2 -DCRYPTOPP_DISABLE_SSSE3 -pipe -I. -c mytest2.cpp
 && g++ -o mytest2 mytest2.o -L. -lcryptopp && ./mytest2 }}}

 will NOT work, and results in:

 {{{ STRING: aaa:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
 (etc)
 SHA256: D159F4747C64249633829F8355DC84678A67CCCE92ABC8F770D33111AFB43E42
 STRING: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
 (etc)
 SHA256: 6836CF13BAC400E9105071CD6AF47084DFACAD4E5E302C94BFED24E013AFB73E
 }}}

--
Ticket URL: <http://allmydata.org/trac/pycryptopp/ticket/24#comment:18>
pycryptopp <http://allmydata.org/trac/pycryptopp>
Python bindings for the Crypto++ library
_______________________________________________
tahoe-dev mailing list
taho...@allmydata.org
http://allmydata.org/cgi-bin/mailman/listinfo/tahoe-dev

[midnightmagic]

Oh, Hey Zooko. :-)

I just posted my sample program in here too. I guess I should've
included more information including a link to the pycryptopp/tahoe
trac tickets.

[Wei Dai]

Thanks for the sample program. I've checked in the fix to sha.cpp.

The cause of the bug, if anyone is curious, was that I assumed a certain
pointer would be word-aligned, and used its least significant bit to store
whether or not the CPU supports SSE2. It turns out that I forgot that
pointer isn't necessarily word-aligned, so when that happens, the code would
incorrectly use the SSE2 code path. And for some reason, the SSE2
instructions didn't cause an exception on non-SSE2 CPUs.

--------------------------------------------------
From: "midnightmagic" <frav...@gmail.com>
Sent: Friday, July 03, 2009 1:55 PM
To: "Crypto++ Users" <cryptop...@googlegroups.com>
Subject: Re: wrong SHA-256 result with optimized assembly on i386
([pycryptopp] #24: SHA256 failure on NetBSD with multiple segments)

[midnightmagic]

Hey that's wonderful! I really appreciate the explanation. I love
hearing about these sorts of fixes.