cryptest fails, but only when redirecting output to a file
49 views
Skip to first unread message
Bart Vandewoestyne
Mar 18, 2021, 11:46:27 AM3/18/21
to Crypto++ Users
Hello,
I have built Crypto++ 8.20 on Windows using Visual Studio 2019 in both Debug and Release configuration using something like
msbuild cryptest.sln /t:cryptlib;cryptest /p:Configuration="Debug";Platform=Win32;PlatformToolset=v142;WholeProgramOptimization="false""
For both the Debug and Release build, when I run the cryptest.exe as follows:
cryptest.exe v
it seems to run fine because I get:
-------------------------------------------------------------------------------------------------------------------------------------
Using seed: 1616082124
Testing Settings...
passed: Your machine is little endian.
passed: Aligned data access.
passed: sizeof(byte) == 1
passed: sizeof(word16) == 2
passed: sizeof(word32) == 4
passed: sizeof(word64) == 8
passed: sizeof(hword) == 2, sizeof(word) == 4, sizeof(dword) == 8
passed: cacheLineSize == 64
hasSSE2 == 1, hasSSSE3 == 1, hasSSE4.1 == 1, hasSSE4.2 == 1, hasAVX == 1, hasAVX2 == 1, hasAESNI == 1, hasCLMUL == 1, hasRDRAND == 1, hasRDSEED == 1, hasSHA == 0, isP4 == 0
Testing operating system provided nonblocking random number generator...
passed: 10240 generated bytes compressed to 10245 bytes by DEFLATE
passed: IncorporateEntropy with 128 bytes
passed: GenerateWord32 and Crop
passed: DiscardBytes with 1024 bytes
Testing RandomPool generator...
passed: 10240 generated bytes compressed to 10245 bytes by DEFLATE
passed: IncorporateEntropy with 128 bytes
passed: GenerateWord32 and Crop
passed: DiscardBytes with 1024 bytes
Testing AutoSeeded RandomPool generator...
passed: 10240 generated bytes compressed to 10245 bytes by DEFLATE
passed: IncorporateEntropy with 128 bytes
passed: GenerateWord32 and Crop
passed: DiscardBytes with 1024 bytes
Testing OldRandomPool generator...
passed: 10240 generated bytes compressed to 10245 bytes by DEFLATE
passed: IncorporateEntropy with 128 bytes
passed: GenerateWord32 and Crop
passed: DiscardBytes with 1024 bytes
passed: Expected sequence from PGP-style RandomPool (circa 2007)
Testing AutoSeeded X917 generator...
passed: 10240 generated bytes compressed to 10245 bytes by DEFLATE
passed: IncorporateEntropy with 128 bytes
passed: GenerateWord32 and Crop
passed: DiscardBytes with 1024 bytes
Testing Padlock RNG generator...
Padlock RNG generator not available, skipping test.
Testing RDRAND generator...
passed: 10240 generated bytes compressed to 10245 bytes by DEFLATE
passed: IncorporateEntropy with 128 bytes
passed: GenerateWord32 and Crop
passed: DiscardBytes with 1024 bytes
passed: Maurer Randomness Test returned value 1.000000
Testing RDSEED generator...
-------------------------------------------------------------------------------------------------------------------------------------
However, when (in a script or on my windows cmd prompt) I try to redirect the output to a file using
cryptest.exe v > output.txt
or
cryptest.exe v > output.txt 2>&1
then i get my cmd prompt back, but the output.txt file only contains
Using seed: 1616082222
and that's it. What am I doing wrong here?
Regards,
Bart
Jeffrey Walton
Mar 18, 2021, 11:56:00 AM3/18/21
to Crypto++ Users List
On Thu, Mar 18, 2021 at 11:46 AM Bart Vandewoestyne
<bart.vand...@gmail.com> wrote:
>
> I have built Crypto++ 8.20 on Windows using Visual Studio 2019 in both Debug and Release configuration using something like
>
> msbuild cryptest.sln /t:cryptlib;cryptest /p:Configuration="Debug";Platform=Win32;PlatformToolset=v142;WholeProgramOptimization="false""
>
> For both the Debug and Release build, when I run the cryptest.exe as follows:
>
> cryptest.exe v
>
> it seems to run fine because I get:
> ...
<bart.vand...@gmail.com> wrote:
>
> I have built Crypto++ 8.20 on Windows using Visual Studio 2019 in both Debug and Release configuration using something like
>
> msbuild cryptest.sln /t:cryptlib;cryptest /p:Configuration="Debug";Platform=Win32;PlatformToolset=v142;WholeProgramOptimization="false""
>
> For both the Debug and Release build, when I run the cryptest.exe as follows:
>
> cryptest.exe v
>
> it seems to run fine because I get:
>
> However, when (in a script or on my windows cmd prompt) I try to redirect the output to a file using
>
> cryptest.exe v > output.txt
> or
> cryptest.exe v > output.txt 2>&1
>
> then i get my cmd prompt back, but the output.txt file only contains
>
> Using seed: 1616082222
>
> and that's it. What am I doing wrong here?
No ideas. The test program (cryptest.exe v' and 'cryptest.exe tv all')
> However, when (in a script or on my windows cmd prompt) I try to redirect the output to a file using
>
> cryptest.exe v > output.txt
> or
> cryptest.exe v > output.txt 2>&1
>
> then i get my cmd prompt back, but the output.txt file only contains
>
> Using seed: 1616082222
>
> and that's it. What am I doing wrong here?
does not prompt so there should not be a hang.
I'm guessing it has something to do with MinGW or Cygwin. Maybe you
can attach a debugger when its hung.
Jeff
Bart Vandewoestyne
Mar 18, 2021, 11:59:38 AM3/18/21
to Crypto++ Users
On Thursday, March 18, 2021 at 4:56:00 PM UTC+1 Jeffrey Walton wrote:
No ideas. The test program (cryptest.exe v' and 'cryptest.exe tv all')
does not prompt so there should not be a hang.
I'm guessing it has something to do with MinGW or Cygwin. Maybe you
can attach a debugger when its hung.
I'm not using MinGW or Cygwin. It's a plain Windows build (using VS2019) and I'm running it in a normal cmd window...
Bart
Bart Vandewoestyne
Mar 18, 2021, 12:11:16 PM3/18/21
to Crypto++ Users
Hmm... I just discovered that the cryptest.exe v output for my VS2019 build is not complete. It stops at
Testing RDSEED generator...
But a colleague of mine who used VS2015 to build it (also debug build), has a much longer output for 'cryptest v'...
So it seems that the RDSEED generator test fails somehow...
Any ideas on what I can do to find out why this fails? Is there anybody using VS2019 who can confirm this behavior?
Regards,
Testing RDSEED generator...
But a colleague of mine who used VS2015 to build it (also debug build), has a much longer output for 'cryptest v'...
So it seems that the RDSEED generator test fails somehow...
Any ideas on what I can do to find out why this fails? Is there anybody using VS2019 who can confirm this behavior?
Regards,
Bart
Bart Vandewoestyne
Mar 18, 2021, 12:19:49 PM3/18/21
to Crypto++ Users
OK, when debugging with the VS2019 debugger, I found out that an exception is being thrown in the call to MASM_RDSEED_GenerateBlock in
void RDSEED::GenerateBlock(byte *output, size_t size)
{
CRYPTOPP_ASSERT((output && size) || !(output || size));
if (size == 0) return;
#if defined(MASM_RDSEED_ASM_AVAILABLE)
MASM_RDSEED_GenerateBlock(output, size);
.... not relevant ...
}
Will look a bit further into this, but in the meanwhile: any ideas on how to fix this are highly appreciated :-) I want all the tests to pass!
Kind regards,
Bart
Bart Vandewoestyne
Mar 18, 2021, 12:24:31 PM3/18/21
to Crypto++ Users
OK... and it seems like MASM_RDSEED_GenerateBlock is a function written in assembler in rdrand.asm.
Since my assembler knowledge is very limited, I'm lost here. So it looks like the fact that I'm running this using VS2019 causes that assembler code to throw an exception or something?
Since my assembler knowledge is very limited, I'm lost here. So it looks like the fact that I'm running this using VS2019 causes that assembler code to throw an exception or something?
Will try again with VS2015 and see if I have the same problem there...
Kind regards,
Bart
Jeffrey Walton
Mar 18, 2021, 12:30:51 PM3/18/21
to Crypto++ Users List
On Thu, Mar 18, 2021 at 12:24 PM Bart Vandewoestyne
<bart.vand...@gmail.com> wrote:
>
> OK... and it seems like MASM_RDSEED_GenerateBlock is a function written in assembler in rdrand.asm.
>
> Since my assembler knowledge is very limited, I'm lost here. So it looks like the fact that I'm running this using VS2019 causes that assembler code to throw an exception or something?
>
> Will try again with VS2015 and see if I have the same problem there...
Throwing an exception is expected if RDRAND or RDSEED is not
<bart.vand...@gmail.com> wrote:
>
> OK... and it seems like MASM_RDSEED_GenerateBlock is a function written in assembler in rdrand.asm.
>
> Since my assembler knowledge is very limited, I'm lost here. So it looks like the fact that I'm running this using VS2019 causes that assembler code to throw an exception or something?
>
> Will try again with VS2015 and see if I have the same problem there...
available. It is the reason for the try/catch block at
https://github.com/weidai11/cryptopp/blob/master/validat3.cpp#L827.
bool TestRDRAND()
{
std::cout << "\nTesting RDRAND generator...\n\n";
bool pass = true;
member_ptr<RandomNumberGenerator> rng;
try {rng.reset(new RDRAND);}
catch (const RDRAND_Err &) {}
if (rng.get())
{
// Test generator
}
return pass;
}
TestRDSEED() does the same thing.
The RDRAND exception is raised at
https://github.com/weidai11/cryptopp/blob/master/rdrand.cpp#L102. The
RDSEED exception is raised at
https://github.com/weidai11/cryptopp/blob/master/rdrand.cpp#L212.
Jeff
Jeffrey Walton
Mar 18, 2021, 12:33:18 PM3/18/21
to Crypto++ Users List
> OK... and it seems like MASM_RDSEED_GenerateBlock is a function written in assembler in rdrand.asm.
>
> Since my assembler knowledge is very limited, I'm lost here. So it looks like the fact that I'm running this using VS2019 causes that assembler code to throw an exception or something?
There's not much too the assembly instructions. Also see
>
> Since my assembler knowledge is very limited, I'm lost here. So it looks like the fact that I'm running this using VS2019 causes that assembler code to throw an exception or something?
https://www.cryptopp.com/wiki/RDRAND_and_RDSEED.
Jeff
Bart Vandewoestyne
Mar 18, 2021, 3:23:58 PM3/18/21
to Crypto++ Users
OK. I have done some more research and testing. Here's what I found out so far:
On our build server (where the tests do not crash), the test logs tell me:
hasRDRAND == 0, hasRDSEED == 0
and because of that, apparently the tests are skipped, because in the logs I
have:
Testing RDSEED generator...
RDSEED generator not available, skipping test.
On my laptop, it seems like I have RDRAND and RDSEED available, because in the
logs I see:
hasRDRAND == 1, hasRDSEED == 1
and so the test is started:
Testing RDSEED generator...
but that is the last line in the output, the test crashes somewhere.
Using the VS2015 debugger, i found that the call stack at the moment of the crash is:
cryptest.exe!017e9c80() Unknown
cryptest.exe!CryptoPP::RDSEED::GenerateBlock(unsigned char * output, unsigned int size) Line 249 C++
cryptest.exe!CryptoPP::RandomNumberGenerator::GenerateIntoBufferedTransformation(CryptoPP::BufferedTransformation & target, const std::basic_string<char,std::char_traits<char>,std::allocator<char> > & channel, unsigned __int64 length) Line 330 C++
cryptest.exe!CryptoPP::RandomNumberStore::TransferTo2(CryptoPP::BufferedTransformation & target, unsigned __int64 & transferBytes, const std::basic_string<char,std::char_traits<char>,std::allocator<char> > & channel, bool blocking) Line 1227 C++
cryptest.exe!CryptoPP::BufferedTransformation::TransferMessagesTo2(CryptoPP::BufferedTransformation & target, unsigned int & messageCount, const std::basic_string<char,std::char_traits<char>,std::allocator<char> > & channel, bool blocking) Line 650 C++
cryptest.exe!CryptoPP::BufferedTransformation::TransferAllTo2(CryptoPP::BufferedTransformation & target, const std::basic_string<char,std::char_traits<char>,std::allocator<char> > & channel, bool blocking) Line 696 C++
cryptest.exe!CryptoPP::SourceTemplate<CryptoPP::RandomNumberStore>::PumpAll2(bool blocking) Line 1378 C++
cryptest.exe!CryptoPP::Source::PumpAll() Line 1319 C++
cryptest.exe!CryptoPP::Source::SourceInitialize(bool pumpAll, const CryptoPP::NameValuePairs & parameters) Line 1355 C++
cryptest.exe!CryptoPP::RandomNumberSource::RandomNumberSource(CryptoPP::RandomNumberGenerator & rng, int length, bool pumpAll, CryptoPP::BufferedTransformation * attachment) Line 1454 C++
cryptest.exe!CryptoPP::Test::Test_RandomNumberGenerator(CryptoPP::RandomNumberGenerator & prng, bool drain) Line 431 C++
cryptest.exe!CryptoPP::Test::TestRDSEED() Line 866 C++
cryptest.exe!CryptoPP::Test::ValidateAll(bool thorough) Line 56 C++
cryptest.exe!CryptoPP::Test::Validate(int alg, bool thorough, const char * seedInput) Line 915 C++
cryptest.exe!CryptoPP::Test::scoped_main(int argc, char * * argv) Line 396 C++
cryptest.exe!main(int argc, char * * argv) Line 1058 C++
cryptest.exe!invoke_main() Line 64 C++
cryptest.exe!__scrt_common_main_seh() Line 253 C++
cryptest.exe!__scrt_common_main() Line 296 C++
cryptest.exe!mainCRTStartup() Line 17 C++
kernel32.dll!76ea6359() Unknown
[Frames below may be incorrect and/or missing, no symbols loaded for kernel32.dll]
ntdll.dll!77407c14() Unknown
ntdll.dll!77407be4() Unknown
more specifically inside the function RDSEED::GenerateBlock where we have the
statement
MASM_RDSEED_GenerateBlock(output, size);
And in rdrand.cpp we have
#if MASM_RDSEED_ASM_AVAILABLE
extern "C" void CRYPTOPP_FASTCALL MASM_RDSEED_GenerateBlock(byte*, size_t);
#endif
Searching the Visual Studio cryptest solution for 'MASM_RDSEED_GenerateBlock'
tells me that this is a function written in assembler and defined in the file
rdrand.asm.
My main question remains why I have the crash when this MASM_RDSEED_GenerateBlock
assembler function is called. Could it be that I am not using the right
compiler flags when building Crypto++? Because at the top of rdrand.asm I see
the comment:
;; This ASM file provides RDRAND and RDSEED to downlevel Microsoft tool chains.
;; Everything "just works" under Visual Studio. Other platforms will have to
;; run MASM/MASM-64 and then link to the object files.
;; set ASFLAGS=/nologo /D_M_X86 /W3 /Cx /Zi /safeseh
;; set ASFLAGS64=/nologo /D_M_X64 /W3 /Cx /Zi
;; "C:\Program Files (x86)\Microsoft Visual Studio 11.0\VC\bin\ml.exe" %ASFLAGS% /Fo rdrand-x86.obj /c rdrand.asm
;; "C:\Program Files (x86)\Microsoft Visual Studio 11.0\VC\bin\amd64\ml64.exe" %ASFLAGS64% /Fo rdrand-x64.obj /c rdrand.asm
If anybody has an idea on why MASM_RDSEED_GenerateBlock(output, size) crashes
and what I can do to avoid it, it would be highly appreciated :-)
Kind regards,
Bart
Jeffrey Walton
Mar 18, 2021, 3:29:08 PM3/18/21
to Crypto++ Users List
I've seen that hang before, but it was present all the time. It did
not happen in isolation, like during redirection.
I wrote it off to a MS toolchain bug and worked around it. Also see
https://github.com/weidai11/cryptopp/commit/b1c691b53a4c.
You should upgrade to Crypto++ 8.3 or above for the fix.
Jeff
Bart Vandewoestyne
Mar 18, 2021, 3:37:27 PM3/18/21
to Crypto++ Users
Hello Jeffrey,
so it sure looks like I am indeed experiencing this bug with my 8.2.0 version using VS2019 on a Windows 10 machine.
I will try 8.3.0 or later and see if that fixes the problem. I'll report back here.
so it sure looks like I am indeed experiencing this bug with my 8.2.0 version using VS2019 on a Windows 10 machine.
I will try 8.3.0 or later and see if that fixes the problem. I'll report back here.
Kind regards,
Bart
Bart Vandewoestyne
Mar 18, 2021, 4:05:36 PM3/18/21
to Crypto++ Users
On Thursday, March 18, 2021 at 8:37:27 PM UTC+1 Bart Vandewoestyne wrote:
[...]
I will try 8.3.0 or later and see if that fixes the problem. I'll report back here.
I confirm: using 8.3.0 solves the problem. For both
cryptest v
cryptest tv all
all tests now pass without problems.
Kind regards,
Bart
Jeffrey Walton
Mar 18, 2021, 11:24:27 PM3/18/21
to Crypto++ Users List
know the cause.
Jeff
Bart Vandewoestyne
Mar 19, 2021, 5:33:50 AM3/19/21
to Crypto++ Users
Hello Jeffrey,
I'm going to be absolutely honest with you: my assembler knowledge is super-limited, so I do not consider myself knowledgeable enough in that area to understand what's wrong there. On top of that, if upgrading to 8.3.0 or later solves the problem, then I think my (and my company's) policy will be to upgrade and move on with your lives instead of trying to continue to figure out what went wrong... ;-) If a day had 48 hours, I would be happy to investigate... but I'm afraid we're not in such a situation...
Kind regards,
Bart
Reply all
Reply to author
Forward
0 new messages