CATCH build performance

[Arto Bendiken]

Howdy,

I've recently switched from Boost.Test to CATCH on all my C++ projects, and I am loving it but for a particular pain point which is becoming increasingly difficult to ignore. Namely, performance. Compiling a test suite takes much longer than previously. Each individual file in the test suite now typically takes on the order of 5+ seconds to compile.

To reduce this down to the simplest possible quantifiable test case, here follow some figures for standalone compilation of the first factorial example from the tutorial:

time g++ factorial.cc                                   # 6.2s

time g++ -std=c++11 factorial.cc                        # 7.8s

time clang++ factorial.cc                               # 4.0s

time clang++ -std=c++11 -stdlib=libc++ factorial.cc     # 4.7s

Now, if we do go still more minimalistic, reducing the input file to simply the following fundamental skeletal minimum:

#define CATCH_CONFIG_MAIN

#include "catch.hpp"

TEST_CASE("skeleton") {}

…the build times do improve, but not markedly so:

time g++ skeleton.cc                                    # 5.6s

time g++ -std=c++11 skeleton.cc                         # 6.6s

time clang++ skeleton.cc                                # 3.3s

time clang++ -std=c++11 -stdlib=libc++ skeleton.cc      # 4.1s

This latter problem is particularly pressing for me, as it means that a test suite which is still mostly in development takes ages to build even though it doesn't yet do much anything, affecting the test-driven development lifecycle. The only way to speed up iterations is to edit the Makefile to remove mentions of stub files, or else to actually comment out the CATCH inclusion from each one. A workaround, at best.

All aforementioned figures were obtained with CATCH 1.0 build 32, timed with GCC 4.8.2 and Clang 3.4 running on Mac OS X 10.7.5 on a MacBook with a 1.8 GHz Intel Core i7 processor and 4 GB of RAM. Each individual timing was averaged from five repeated runs.

Are there perchance any preprocessor macros or the like that one might define in order to reduce the number of lines of magic that the compiler needs to process for each compilation? I don't (at present, anyhow) need any of the more advanced functionality provided by CATCH; just making basic REQUIRE() assertions would pretty much suffice.

Thanks for your consideration,

Arto

-- 

Arto Bendiken | @bendiken | http://ar.to

[Phil Nash]

On Wednesday, 19 March 2014 18:57:45 UTC, Arto Bendiken wrote:

Howdy,

I've recently switched from Boost.Test to CATCH on all my C++ projects, and I am loving it

Great!

 

but for a particular pain point which is becoming increasingly difficult to ignore. Namely, performance.

Not so great.

 

Compiling a test suite takes much longer than previously. Each individual file in the test suite now typically takes on the order of 5+ seconds to compile.

To reduce this down to the simplest possible quantifiable test case, here follow some figures for standalone compilation of the first factorial example from the tutorial:

time g++ factorial.cc                                   # 6.2s

time g++ -std=c++11 factorial.cc                        # 7.8s

time clang++ factorial.cc                               # 4.0s

time clang++ -std=c++11 -stdlib=libc++ factorial.cc     # 4.7s

Unfortunately by reducing to "the simplest possible quantifiable test case" you have changed the nature of what you are quantifying (and we haven't even had to invoke quantum mechanics, although perhaps the mechanics of quantification).

The issue is that in exactly one cpp file you should #define CATCH_CONFIG_MAIN. You're doing that - but now in the only cpp file!

The reason this is significant is that that definition causes Catch to compile in all the code that, if it were not header only, would be in cpp files. So you take the hit of compiling all of Catch in one translation unit but the rest should be free of any overhead due to header-only-ness.

Should be.

There are exceptions and I go through every now and then to purge them but they have a habit of building up.

I've not done any timings for a while so it's possible they've accumulated again.

If not the other thing that can add up is the use of expression templates for decomposing all the assertion expressions.

 

Now, if we do go still more minimalistic, reducing the input file to simply the following fundamental skeletal minimum:

#define CATCH_CONFIG_MAIN

#include "catch.hpp"

TEST_CASE("skeleton") {}

…the build times do improve, but not markedly so:

In this case that suggests that the overhead you are measuring is due to CATCH_CONFIG_MAIN and you should not see it for other TUs. In other 

words you're not measuring what you set out to measure.

Note: I'm not saying there is no problem - you clearly had an issue that prompted you to do the timings in the first place.

This latter problem is particularly pressing for me, as it means that a test suite which is still mostly in development takes ages to build even though it doesn't yet do much anything, affecting the test-driven development lifecycle. The only way to speed up iterations is to edit the Makefile to remove mentions of stub files, or else to actually comment out the CATCH inclusion from each one. A workaround, at best.

It would be interesting if you could come back with timings that take into account what I've now (hopefully) clarified about CATCH_CONFIG_MAIN. I should do the same, although my conclusions may not hold for your situation.

 

All aforementioned figures were obtained with CATCH 1.0 build 32, timed with GCC 4.8.2 and Clang 3.4 running on Mac OS X 10.7.5 on a MacBook with a 1.8 GHz Intel Core i7 processor and 4 GB of RAM. Each individual timing was averaged from five repeated runs.

Are there perchance any preprocessor macros or the like that one might define in order to reduce the number of lines of magic that the compiler needs to process for each compilation?

By now you should know the answer to that ;-)

 

I don't (at present, anyhow) need any of the more advanced functionality provided by CATCH; just making basic REQUIRE() assertions would pretty much suffice

 

However there is not, currently, anyway to selectively remove certain features. It's an interesting idea, though, which I may consider. However I do feel that, once you take the CATCH_CONFIG_MAIN part out of the equation, the next biggest bottleneck is probably the expression templates. It may be possible to have it conditionally compile those in (falling back to old-school assert()'s behaviour - i.e. not decomposing the expression). At least that way you would have the choice of how to make your trade off.

Of course I could be wrong and there could be an unnecessary sub-system of Catch that is significantly adding to build-times. I certainly need to look at it again. A few people have brought this up recently, so there is obviously a problem here. Thanks for taking the time to try and break this down.