Slow compilation
johnny
The module I'm testing has interaction with many other modules through
interfaces (about 20 of them, each with about 15 functions taking and
returning mostly built-in types).
Trying to get the test case to pass, I add mocks for one interface at
a time. Every time I create a new mock class, #include and use it in
the test case, the compilation time increases with 10-20 seconds. As
comparison, all other .cpp files compile in < 1 second. At the moment,
I have mocked 8 interfaces and the compile time is 1m35s. (Of which
linking is probably 30 seconds.) Clearly, turn-around time like this
when unit testing is painful!
Before I provide you with more detailed information on mock usage,
MSVC project settings etc., I wanted to ask if you have seen these
kinds of compile times before. It might just be that I have screwed
something up in the project settings etc. (I have seen big
improvements fiddling with compiler and linker settings.)
Another question that comes to mind is: is there a way to separate
mock function declaration from definintion in gmock? It seems like re-
parsing and recompiling the mock function code every time I change a
test case is a waste of time, right? For fun, I implemented wrapper
macros for mock function declaration and definition, and it seems to
work just fine. Are there any plans in this area?
And last, thank you for a great piece of software! With reasonable
compile times, gmock will be very useful for us!
Best regards / johnny
Vlad Losev
I have a small test project with just a single test case (for now).
The module I'm testing has interaction with many other modules through
interfaces (about 20 of them, each with about 15 functions taking and
returning mostly built-in types).
Trying to get the test case to pass, I add mocks for one interface at
a time. Every time I create a new mock class, #include and use it in
the test case, the compilation time increases with 10-20 seconds. As
comparison, all other .cpp files compile in < 1 second. At the moment,
I have mocked 8 interfaces and the compile time is 1m35s. (Of which
linking is probably 30 seconds.) Clearly, turn-around time like this
when unit testing is painful!
Before I provide you with more detailed information on mock usage,
MSVC project settings etc., I wanted to ask if you have seen these
kinds of compile times before. It might just be that I have screwed
something up in the project settings etc. (I have seen big
improvements fiddling with compiler and linker settings.)
Another question that comes to mind is: is there a way to separate
mock function declaration from definintion in gmock? It seems like re-
parsing and recompiling the mock function code every time I change a
test case is a waste of time, right? For fun, I implemented wrapper
macros for mock function declaration and definition, and it seems to
work just fine. Are there any plans in this area?
And last, thank you for a great piece of software! With reasonable
compile times, gmock will be very useful for us!
Best regards / johnny
Regards,
Vlad
johnny
Thank you for your quick answer!
I looked at the thread you suggested and it was a nice source of
information. Regarding the details discussed in the thread:
- I don't have the /clr flag turned on
- I run out of memory often and have to increase the /Zm flag (e.g. /
Zm700)
- I also got the object file problem and had to add /bigobj
- I would probably be helped by mocking only some of the functions,
but I would like the mock classes to be usable for others. Thus, I
really want the mock classes to have _all_ functions mocked. I realize
this is the key problem.
- My mock class/function definitions are very stable. Compiling them
only once would be great. So, I think that good macros for mock
declarations could help a lot (not like my feeble attempts, but with
most types forward declared) in cases like mine. What do you think?
Do you have any other ideas on how to reduce compile times? If not, I
will just have to think really hard whether or not my module can be
tested efficiently.
Thanks. / j
On 3 Sep, 12:57, Vlad Losev <vladlo...@gmail.com> wrote:
> Hi Johnny,
>
>
>
> On Thu, Sep 3, 2009 at 12:00 PM, johnny <johnny.big...@gmail.com> wrote:
>
> > I have a small test project with just a single test case (for now).
> > The module I'm testing has interaction with many other modules through
> > interfaces (about 20 of them, each with about 15 functions taking and
> > returning mostly built-in types).
>
> > Trying to get the test case to pass, I add mocks for one interface at
> > a time. Every time I create a new mock class, #include and use it in
> > the test case, the compilation time increases with 10-20 seconds. As
> > comparison, all other .cpp files compile in < 1 second. At the moment,
> > I have mocked 8 interfaces and the compile time is 1m35s. (Of which
> > linking is probably 30 seconds.) Clearly, turn-around time like this
> > when unit testing is painful!
>
> > You may be trying to compile gmock code in the managed mode. In that case,
>
> .
Zhanyong Wan (λx.x x)
Which version of VC are you using? Is compiling gmock's own tests
slow too? How many virtual methods are in your class? Could you
provide us a small, but complete, program (and your compiler flags)
that shows the problem?
>> > Before I provide you with more detailed information on mock usage,
>> > MSVC project settings etc., I wanted to ask if you have seen these
>> > kinds of compile times before. It might just be that I have screwed
>> > something up in the project settings etc. (I have seen big
>> > improvements fiddling with compiler and linker settings.)
>>
>> > Another question that comes to mind is: is there a way to separate
>> > mock function declaration from definintion in gmock? It seems like re-
>> > parsing and recompiling the mock function code every time I change a
>> > test case is a waste of time, right? For fun, I implemented wrapper
The speed of compilation hasn't really been a big problem for us who
use gmock inside Google, so we haven't done a lot to optimize it. It
is possible to separate the declaration and definition of mock
methods, but that requires the user to repeat the same information
twice. It hasn't become necessary for us yet. Let's see if we can
fix your performance problem first.
I'm curious: how much savings in compilation time are you seeing when
you separate the declaration and the definition?
Thanks,
>> > macros for mock function declaration and definition, and it seems to
>> > work just fine. Are there any plans in this area?
>>
>> > And last, thank you for a great piece of software! With reasonable
>> > compile times, gmock will be very useful for us!
>>
>> > Best regards / johnny
>>
>> Regards,
>> Vlad
>
--
Zhanyong
johnny
Hi,
I'm using MSVC 2005. I recompiled gmock-1.1.0\test\gmock-generated-
function-mockers_test.cc several times and got 23s compile + link time
on the average. It has 19 virtual functions. Normal files compile in
under a second. 23 seconds for a small test file like this is quite
heavy.
In our code, all interfaces combined (25 of them) has just over 400
virtual functions. The compile + link time on a file with a single
test case including all mock classes (but only using 8 of them, and
about 12 ON_CALL/EXPECTs) is 5m09s.
Just for fun, if we assume compile times increase linearly with the
number of virtual functions, we expect 23 seconds * 400/19 = 8 minutes
compile times. So 5m09s is probably close to expected.
I have estimated (e.g. using grep) that about 100 of the 400 virtual
functions are actually used. Mocking only those would give me compile
times of about 2 minutes then. Having turn-around times of minutes
would be painful. :/
Regarding declarations: If I remember it correctly, using decl/def
separation brought compile times down about 20% (from 1m45s to 1m26s).
With a real effort, declaration could probably be much better (forward
declare all types etc.).
Do you think there is any hope of getting the compile times down?
BR / Johnny
Vlad Losev
Hi Vlad,
Thank you for your quick answer!
I looked at the thread you suggested and it was a nice source of
information. Regarding the details discussed in the thread:
- I don't have the /clr flag turned on
- I run out of memory often and have to increase the /Zm flag (e.g. /
Zm700)
- I also got the object file problem and had to add /bigobj
- I would probably be helped by mocking only some of the functions,
but I would like the mock classes to be usable for others. Thus, I
really want the mock classes to have _all_ functions mocked. I realize
this is the key problem.
Try disabling precompiled headers in your test projects if you are using them.
IIRC, the amount of memory consumed bby the compiler is directly proportional to the number of mocked methods. Thus, splitting your test files so that each one contains minimal amount of mocked classes/interfaces may help with the performance.
- Vlad
johnny
I had to disable precompiled headers, the compiler gave me "fatal
error C1083: Cannot open compiler intermediate file: 'unittest.pch':
Not enough space" :)
The problem with splitting files is that I'm testing a subsystem
containing several classes (i.e. end-to-end tests). Most testcases
will depend upon several of the 20 external interfaces. But I'll do my
best to keep test sets small in each file.
And the good news: when I mocked only the functions that are actually
called, compile times dropped to 20 seconds. Now, the only drawback is
that module interface mocks cannot be reused (only copy-reused). But I
can live with that.
Zhanyong Wan (λx.x x)
On Thu, Sep 3, 2009 at 10:45 PM, johnny<johnny...@gmail.com> wrote:
>
>
> Hi,
>
> I'm using MSVC 2005. I recompiled gmock-1.1.0\test\gmock-generated-
> function-mockers_test.cc several times and got 23s compile + link time
> on the average. It has 19 virtual functions. Normal files compile in
> under a second. 23 seconds for a small test file like this is quite
> heavy.
I get about the same speed on my Windows machine.
FYI: on a Linux machine with similar hardware, gcc is able to compile
and link it in ~10 seconds.
> In our code, all interfaces combined (25 of them) has just over 400
> virtual functions. The compile + link time on a file with a single
> test case including all mock classes (but only using 8 of them, and
> about 12 ON_CALL/EXPECTs) is 5m09s.
>
> Just for fun, if we assume compile times increase linearly with the
> number of virtual functions, we expect 23 seconds * 400/19 = 8 minutes
> compile times. So 5m09s is probably close to expected.
>
> I have estimated (e.g. using grep) that about 100 of the 400 virtual
> functions are actually used. Mocking only those would give me compile
> times of about 2 minutes then. Having turn-around times of minutes
> would be painful. :/
>
> Regarding declarations: If I remember it correctly, using decl/def
> separation brought compile times down about 20% (from 1m45s to 1m26s).
> With a real effort, declaration could probably be much better (forward
> declare all types etc.).
>
> Do you think there is any hope of getting the compile times down?
I'll spend some time thinking about this and report my result. Thanks,
--
Zhanyong
Vlad Losev
I had to disable precompiled headers, the compiler gave me "fatal
error C1083: Cannot open compiler intermediate file: 'unittest.pch':
Not enough space" :)
- Vlad
Zhanyong Wan (λx.x x)
will revisit the problem when I have more time. I created
http://code.google.com/p/googlemock/issues/detail?id=68 to track it.
Thanks,
2009/9/4 Zhanyong Wan (λx.x x) <w...@google.com>:
--
Zhanyong
Jocelyn Legault
I couldn't think of an easy way to boost the compilation speed, but
How about precompiled headers? Isn't it a feature supported by most compilers these days?
joce.
Zhanyong Wan (λx.x x)
Thanks for the suggestion. People can try that, but precompiled
headers have their limitations (many source files have to share the
same headers to pay off, and it incurs more memory usage, etc).
Therefore it would still be nice to improve the compilation speed.
--
Zhanyong
Jocelyn Legault
You can grab the pdf from http://drop.io/nyoevun
I'm rather confident that precompiled headers could be of some help, especially since #include <gtest/gtest.h> pulls a lot of stuff by itself.
It's getting late now, but I'll run some tests tomorrow and I'll let you know if I can make any difference.
Zhanyong Wan (λx.x x)
> Well, just for fun, I generated a project include graph for gmock. It's a
> tad scary! :-)
>
> You can grab the pdf from http://drop.io/nyoevun
Thanks for the pdf! It's interesting.
Most of the complexity comes from windows.h and STL headers.
> I'm rather confident that precompiled headers could be of some help,
> especially since #include <gtest/gtest.h> pulls a lot of stuff by itself.
>
> It's getting late now, but I'll run some tests tomorrow and I'll let you
> know if I can make any difference.
I wouldn't be surprised if precompiled headers help. It's worth
pointing out that the majority of the compilation time (for the slow
cases any way) is spent on instantiating MOCK_METHOD* definitions.
Therefore you'd probably have to move the MOCK_METHOD*s into the
precompiled headers to see a meaningful benefit. Pre-compiling
gmock.h and gtest.h isn't likely to be enough.
Thanks for volunteering to share your experiment result. Looking forward to it!
> 2009/9/15 Zhanyong Wan (λx.x x) <w...@google.com>
>>
>> 2009/9/12 Jocelyn Legault <jocelyn...@gmail.com>:
>> >
>> > 2009/9/10 Zhanyong Wan (λx.x x) <w...@google.com>
>> >>
>> >> I couldn't think of an easy way to boost the compilation speed, but
>> >
>> > How about precompiled headers? Isn't it a feature supported by most
>> > compilers these days?
>>
>> Thanks for the suggestion. People can try that, but precompiled
>> headers have their limitations (many source files have to share the
>> same headers to pay off, and it incurs more memory usage, etc).
>> Therefore it would still be nice to improve the compilation speed.
>>
>> --
>> Zhanyong
>
>
--
Zhanyong