Library testing (was: Breaking eggs and making omelets)

[Jeffrey Walton]

> Sixth, learning from mistakes and is not license to lower expectations. Each
> time I broke the code, I looked beyond the instance problem and questioned
> why it happened in the first place. I then placed controls to stop future
> breaks. Of all the controls that can be placed, we achieve the highest yield
> from cryptest.sh and additional self tests.
>
> I've been tracking bug reports and performance of cryptest.sh. cryptest.sh
> detected 78% of the bugs before users experienced them. The script caught
> every problem introduced this week. If you are submitting patches, then it
> would behoove you to run cryptest.sh on your integrated changes or risk
> being tarred and feathered by users.

By the way, cryptest.sh can be found at
https://github.com/weidai11/cryptopp/blob/master/cryptest.sh.

Here is the result of running it on a late 2010 MacBook. I'd be
interested to know if anyone else tests their library under 75
different configurations for one platform. Keep in mind I do this
continually for the BSDs, Linux, OS X and Solaris.

(And the 2 to 10 hour runtime explains why a break creeps into Master
for an instance. The script takes a while to run, so we have to wait
for the results to catch unforeseen issues).

************************************************

74 configurations tested
- No Posix NDEBUG or assert
- C++ std::min and std::max
- X86 rotate immediate code generation
- Debug, default CXXFLAGS
- Release, default CXXFLAGS
- Debug, x86_64 minimum arch CXXFLAGS
- Release, x86_64 minimum arch CXXFLAGS
- Debug, mismatched arch capabilities
- Release, mismatched arch capabilities
- Debug, DISABLE_ASM
- Release, DISABLE_ASM
- Debug, c++03
- Release, c++03
- Debug, c++11
- Release, c++11
- Debug, gnu++11
- Release, gnu++11
- Debug, c++14
- Release, c++14
- Debug, gnu++14
- Release, gnu++14
- Debug, MAINTAIN_BACKWARDS_COMPATIBILITY
- Release, MAINTAIN_BACKWARDS_COMPATIBILITY
- Debug, INIT_PRIORITY
- Release, INIT_PRIORITY
- Debug, NO_UNALIGNED_DATA_ACCESS
- Release, NO_UNALIGNED_DATA_ACCESS
- Debug, NO_OS_DEPENDENCE
- Release, NO_OS_DEPENDENCE
- Debug, -O3 optimizations
- Release, -O3 optimizations
- Debug, -O5 optimizations
- Release, -O5 optimizations
- Debug, -Os optimizations
- Release, -Os optimizations
- Debug, -Ofast optimizations
- Release, -Ofast optimizations
- Debug, dead code strip
- Release, dead code strip
- Debug, OpenMP
- Release, OpenMP
- Debug, c++03, Asan
- Release, c++03, Asan
- Debug, c++11, Asan
- Release, c++11, Asan
- Release, c++14, Asan
- Darwin, c++03, libc++ (LLVM)
- Darwin, c++03, libstdc++ (GNU)
- Darwin, c++11, libc++ (LLVM)
- Darwin, c++11, libstdc++ (GNU)
- Darwin, c++14, libc++ (LLVM)
- Darwin, c++14, libstdc++ (GNU)
- Darwin, Intel multiarch, c++03
- Darwin, Intel multiarch, c++11
- Darwin, Intel multiarch, c++14
- Darwin, c++03, Malloc Guards
- Darwin, c++11, Malloc Guards
- Darwin, c++14, Malloc Guards
- AES, RDRAND and RDSEED
- Benchmarks, c++03
- Benchmarks, c++11
- Benchmarks, c++14
- Valgrind, c++03
- Valgrind, c++11
- Valgrind, c++14
- GCC compiler
- MacPorts 5.x GCC compiler
- MacPorts 6.x GCC compiler
- MacPorts 3.7 Clang compiler
- MacPorts 3.8 Clang compiler
- MacPorts 3.9 Clang compiler
- Xcode Clang compiler
- Test install with data directory
- Test remove with data directory

************************************************

Testing started: Wed Sep 21 01:26:40 EDT 2016
Testing finished: Wed Sep 21 11:25:15 EDT 2016