Flipped bit in chacha encrypted file, avx vs sse

[austin clifton]

Hi all,

I have an issue which I believe may be a bug. I followed the instructions from the "Bug Report" page on the cryptopp wiki and all tests in cryptest.exe are passing, so figured I should post here first and make sure it isn't a build related issue.

I built cryptopp 8.5.0 as a static library, x64 multi-threaded debug (\MTd) using Visual Studio 2019 v16.10.0, on Windows 10 Pro v10.0.19043, using the .sln file provided with the cryptopp source code.

I used two different machines for this test. One is the machine I built the cryptopp library with. Both are running the same version of Windows 10. One machine has a Ryzen 3700X, the other has an i7-990X. The Ryzen supports AVX, the i7 does not.

I am VERY rarely finding that files encrypted with the chacha cipher by these two machines have differing sha256 hashes. If I do a hex diff against the files, there is a single flipped bit at 0x1EBB4 (0xD8 vs 0xD9).

I am able to reproduce this consistently with a specific key, nonce, and input file, after test encrypting ~50GB of 1MB garbage files on each machine. 

If I disable AVX when building cryptlib by defining CRYPTOPP_DISABLE_AVX and CRYPTOPP_DISABLE_AVX2, the machine with the Ryzen will encrypt the file the same as the i7.

Source code for a minimal reproducible example is here:
https://github.com/austin-clifton/cryptopp-chacha-asm-test/blob/main/src/main.cpp#L208

That repository includes a ready-to-build test case with Visual Studio, minus a built cryptlib.lib which should be added to libs/debug/ before building.

Thanks,

Austin

[Jeffrey Walton]

Ugh, I dislike these reports.

I've seen a similar problem on a Ryzen with SHA. It is GCC67, which is
Ryzen 5 2400G, at https://cfarm.tetaneutral.net/machines/list/. One
self test failed but no others. The one test that failed is one with a
very long input string. I believe it is/was the last test at
https://github.com/weidai11/cryptopp/blob/master/TestVectors/sha2_256_fips_180.txt.

I am thinking the problem is with either GCC's AMD target code or the
Ryzen CPU itself. The problem eventually cleared itself.

With that said, can you give Master a try? We cleared a bug at
https://github.com/weidai11/cryptopp/issues/1010 that may affect GCC's
code generation. It was a bug related to GCC alias violations, and GCC
would remove code on occasion.

Something else to try... Update the firmware on the machine in case a
firmware or microcode update addressed the issue.

Jeff

[Jeffrey Walton]

On Thu, Sep 16, 2021 at 8:42 PM austin clifton <austin....@otoy.com> wrote:
>

Austin,

I fetched main.cpp and run_459_file_76.bin from your GitHub. I
compiled the library with:

CXXFLAGS="-DNDEBUG -g2 -O3 -std=c++17" make

I compiled/linked main.cpp with:

g++ -DNDEBUG -g2 -O3 -fPIC -pthread -std=c++17 -I. main.cxx
./libcryptopp.a -o main.exe

Running the program on a AMD Ryzen 3 3200U and AMD A6-9220e, this is the result.

$ ./main.exe
sha256: 0FC0FADCDF82770086C9DA8513A16FC785620D7B1C187CDD828E433EB0979847
encsha: 6FBEE484EE64A2AB02235DDF29CA0B61EE3B811D227C2729836D3BD6161C9B18

Is this correct?

I have one Windows machine for testing. It is a Core i5. Sorry I don't
have a good test environment.

Jeff

[austin clifton]

Hey Jeff, thanks for the fast response!

Yes, the encrypted sha from your AMD CPUs matches what I get on my Ryzen 7 3700X.

The encrypted sha from the i7 is 8F16077454F8477594CAD4304126B0A6F30C8C4D2536E2441FFFD320656E1DF1. That's also the sha I get if I disable AVX on my Ryzen when compiling cryptopp.

I'm not sure which sha is "correct" but we are seeing the same behavior across the AMD CPUs. Would it make sense to disable assembly altogether to get a reference encrypted sha256?

We are seeing the same behavior from MSVC and GCC compilers. I can try master with GCC here too but sounds like that fix is unrelated?

I'll try to glean more useful data from logs I have here. We distribute encrypted assets to lots of machines (all running Windows) but I have to dig around some.

[austin clifton]

FWIW, I have two more test results from Windows machines with Intel CPUs:

• i7 8750h
  
• i9 10980KK
  

Both are AVX-capable and produce the encrypted sha 6FBEE484EE64A2AB02235DDF29CA0B61EE3B811D227C2729836D3BD6161C9B18.

[Jeffrey Walton]

On Fri, Sep 17, 2021 at 2:31 PM austin clifton <austin....@otoy.com> wrote:
>
> Hey Jeff, thanks for the fast response!
>
> Yes, the encrypted sha from your AMD CPUs matches what I get on my Ryzen 7 3700X.
>
> The encrypted sha from the i7 is 8F16077454F8477594CAD4304126B0A6F30C8C4D2536E2441FFFD320656E1DF1. That's also the sha I get if I disable AVX on my Ryzen when compiling cryptopp.
>
> I'm not sure which sha is "correct" but we are seeing the same behavior across the AMD CPUs. Would it make sense to disable assembly altogether to get a reference encrypted sha256?

Yes, that is what I usually do when I need a reference. I add
-DCRYPTOPP_DISABLE_ASM to CXXFLAGS. That removes all inline assembly
and intrinsics.

In the case of ChaCha there is another option for a reference. You can
use Bernstein's reference implementation. I use the author's
implementation for test vectors, when available. I keep them stashed
away at https://github.com/noloader/cryptopp-test.

Note that Bernstein's ChaCah and the IETF's ChaCha are slightly
different. Bernstein uses a 64-bit counter that starts at 0 and cannot
wrap. IETF uses a 32-bit counter that can wrap. The IETF never
specified what to do when wrap occurs.

> We are seeing the same behavior from MSVC and GCC compilers. I can try master with GCC here too but sounds like that fix is unrelated?

Yeah, I think that is an issue with the CPU, not the compiler.

> I'll try to glean more useful data from logs I have here. We distribute encrypted assets to lots of machines (all running Windows) but I have to dig around some.

The guy to ask is Agner Fog, https://www.agner.org. I'm not sure
whether he runs a mailing list.

Jeff

[Jeffrey Walton]

On Thu, Sep 16, 2021 at 8:42 PM austin clifton <austin....@otoy.com> wrote:
>
> I have an issue which I believe may be a bug. I followed the instructions from the "Bug Report" page on the cryptopp wiki and all tests in cryptest.exe are passing, so figured I should post here first and make sure it isn't a build related issue.

We're tracking this issue at https://github.com/weidai11/cryptopp/issues/1069.

Jeff

[Jeffrey Walton]

We disabled the AVX2 implementation at https://github.com/weidai11/cryptopp/commit/f0a32f0c6a51.

Jeff

[austin clifton]

Great, thanks for the update. I'll follow the conversation over there.

[Jeffrey Walton]

On Friday, September 17, 2021 at 9:19:33 PM UTC-4 austin....@otoy.com wrote:

Great, thanks for the update. I'll follow the conversation over there.

...

We're tracking this issue at https://github.com/weidai11/cryptopp/issues/1069.

We disabled the AVX2 implementation at https://github.com/weidai11/cryptopp/commit/f0a32f0c6a51.

The issue was cleared at https://github.com/weidai11/cryptopp/commit/20962baf4440.

Jeff