Using Spock to test C++

Michael Putters

unread,

Mar 20, 2014, 2:32:42 PM3/20/14

to spockfr...@googlegroups.com

Hello,

I thought this might interest people who are coming from a background that is not necessarily Java/Groovy-centric:

I am working on a project where the client is 99% C++ and the server 99% Java. The Java side - which really only involves sticking a bunch of libraries together: CXF, Camel, Couchbase, etc. - is tested using Spock, which is as you all know extremely nice. But for C++, things are a bit trickier. There are obviously a lot of frameworks and many of them do a fine job, but they are generally limited by the language itself. Up until now, I was writing the C++ tests using Google Test.

Now a large chunk of the C++ project is a static recompiler: it takes PowerPC instructions, analyzes them, generates an intermediate representation for LLVM and then compiles the whole thing to x86 or ARM and finally executes the instructions. This requires an awful lot of very small unit tests for each instruction. And those tests are heavily data-driven, since you have a starting state for the processor (each registers having specific values) and you want to test the state at the end of the execution. Using Google Test and trying to add some "nice" syntax, I came up with something that was attempting to do what Spock does:

TEST_F(PowerPCFixedArithmetic_ADDZE, canAddToZeroExtendedWithOverflow) {
    given([](ProcessorStateVector & states) {
        states.resize(6);

        states[0].GPR.R14 = 0x1111111111111111;
        states[0].XER.split.CA = 0;

        states[1].GPR.R14 = 0x1111111111111111;
        states[1].XER.split.CA = 1;

        states[2].GPR.R14 = 0xffffffffffffffff;
        states[2].XER.split.CA = 0;

        states[3].GPR.R14 = 0xffffffffffffffff;
        states[3].XER.split.CA = 1;

        states[4].GPR.R14 = 0x7fffffffffffffff;
        states[4].XER.split.CA = 0;

        states[5].GPR.R14 = 0x7fffffffffffffff;
        states[5].XER.split.CA = 1;
    });

    when(R"(
        addzeo %r13, %r14
    )");

    then([](ProcessorStateVector const & states) {
        ASSERT_EQ(0x1111111111111111, states[0].GPR.R13);
        ASSERT_TRUE(states[0].XER.hasOverflow());
        ASSERT_FALSE(states[0].XER.didOverflow());
        ASSERT_FALSE(states[0].XER.hasCarry());

        ASSERT_EQ(0x1111111111111112, states[1].GPR.R13);
        ASSERT_TRUE(states[1].XER.hasOverflow());
        ASSERT_FALSE(states[1].XER.didOverflow());
        ASSERT_FALSE(states[1].XER.hasCarry());

        ASSERT_EQ(0xffffffffffffffff, states[2].GPR.R13);
        ASSERT_FALSE(states[2].XER.hasOverflow());
        ASSERT_FALSE(states[2].XER.didOverflow());
        ASSERT_FALSE(states[2].XER.hasCarry());

        ASSERT_EQ(0x0000000000000000, states[3].GPR.R13);
        ASSERT_FALSE(states[3].XER.hasOverflow());
        ASSERT_FALSE(states[3].XER.didOverflow());
        ASSERT_TRUE(states[3].XER.hasCarry());

        ASSERT_EQ(0x7fffffffffffffff, states[4].GPR.R13);
        ASSERT_TRUE(states[4].XER.hasOverflow());
        ASSERT_FALSE(states[4].XER.didOverflow());
        ASSERT_FALSE(states[4].XER.hasCarry());

        ASSERT_EQ(0x8000000000000000, states[5].GPR.R13);
        ASSERT_TRUE(states[5].XER.hasOverflow());
        ASSERT_TRUE(states[5].XER.didOverflow());
        ASSERT_FALSE(states[5].XER.hasCarry());
    });
}

This is a very simple test, and it is already quite far from eye candy. It's also really easy to make a mistake somewhere in the middle of all those copy-pasted asserts.

So after thinking about how I could get something close to Spock, I told myself "well, why not just use Spock?". I thought it'd be complicated but all I really need is to call one function with a structure that holds the processor state, and get that modified structure back. A few lines of JNI later, I had something like this:

class ADDZESpecification extends PowerPCSpecification {

    def "can add to zero extended with overflow"(long r14, boolean existingCA, long r13, boolean SO, boolean OV, boolean CA) {
        given:
        state.r14 = r14
        state.XER.CA = existingCA

        when:
        execute("addzeo %r13, %r14")

        then:
        state.r13 == r13
        state.XER.SO == SO
        state.XER.OV == OV
        state.XER.CA == CA

        where:
        r14                  | existingCA || r13                     | SO   | OV          | CA
        0x1111111111111111   | false       || 0x1111111111111111      | true | false       | false
        0x1111111111111111   | true        || 0x1111111111111112      | true | false       | false
        0x7fffffffffffffff   | false       || 0x7fffffffffffffff      | true | false       | false
        0x7fffffffffffffff   | true        || 0x8000000000000000      | true | true        | false
        0xffffffffffffffff   | false       || 0xffffffffffffffff      | true | false       | false
        0xffffffffffffffff   | true        || 0x0000000000000000      | true | false       | true
    }

}

Much better looking and far easier to maintain. The only small issue left is that I still have to specify the method's parameters due to all those hex numbers ending up being something other than a long.

So while I wouldn't recommend that for general purpose tests (because it'd require JNI for everything you test), if you have a similar project that's heavily data-driven, I definitely recommend using Spock.

And just to make things groovier, one last thing I didn't mention. This particular client sub-project's structure is:

/src/main/cpp - the project's sources
/src/main/include - the project's internal headers
/src/main/public - the project's external headers
/src/test/cpp - the JNI sources
/src/test/groovy - the Spock specifications

Because yes, I'm obviously using Gradle to build the client (32 sub-projects, all using the new native language plugins, with Windows, Linux, OS X, iOS, Android and Windows Phone targets) ;-)

Michael

Renato Athaydes

unread,

Mar 21, 2014, 4:42:21 PM3/21/14

to spockfr...@googlegroups.com

Wow, good job!

Can we see the code online (especially the JNI you used)?

David Dawson

unread,

Mar 22, 2014, 3:36:52 AM3/22/14

to spockfr...@googlegroups.com

Did you happen to look at JNA instead of JNI by any chance? It's far, far easier to use.

--
David Dawson
CEO, Principal Consultant
Simplicity Itself Limited
Tel +44 7866 011 256
Skype: davidadawson
david....@simplicityitself.com
http://www.simplicityitself.com

On 21 Mar 2014 23:27, "Renato Athaydes" <ren...@athaydes.com> wrote:

Wow, good job!

Can we see the code online (especially the JNI you used)?

--
You received this message because you are subscribed to the Google Groups "Spock Framework - User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spockframewor...@googlegroups.com.
To post to this group, send email to spockfr...@googlegroups.com.
Visit this group at http://groups.google.com/group/spockframework.
For more options, visit https://groups.google.com/d/optout.

Michael Putters

unread,

Mar 22, 2014, 7:51:23 AM3/22/14

to spockfr...@googlegroups.com

Yes I have, but considering I really have only one function to call I figured avoiding having yet another dependency (JNA) was worth the extra JNI work, which is really just 20-ish lines of code

Michael Putters

unread,

Mar 22, 2014, 7:59:14 AM3/22/14

to spockfr...@googlegroups.com

The project is not on github (yet), but: https://gist.github.com/mputters/9706093

David Dawson

unread,

Mar 22, 2014, 8:27:55 AM3/22/14

to spockfr...@googlegroups.com

Fairy nuff.

I'd not even considered Spock before for any C related work. This is fab. Thanks for the idea!

--
David Dawson
CEO, Principal Consultant
Simplicity Itself Limited
Tel +44 7866 011 256
Skype: davidadawson
david....@simplicityitself.com
http://www.simplicityitself.com

Russel Winder

unread,

Mar 22, 2014, 9:18:45 AM3/22/14

to spockfr...@googlegroups.com

On Thu, 2014-03-20 at 11:32 -0700, Michael Putters wrote:
[…]

> now, I was writing the C++ tests using Google Test.

[…]

I would just have switched from Google Test to Catch
https://github.com/philsquared/Catch but I am not sure if Phil has put
the equivalent of Spock's @Unroll into it yet.

I have taken the liberty of forwarding your email to him.

--
Russel.
=============================================================================
Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel...@ekiga.net
41 Buckmaster Road m: +44 7770 465 077 xmpp: rus...@winder.org.uk
London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder

Phil Nash

unread,

Mar 22, 2014, 11:26:40 AM3/22/14

to spockfr...@googlegroups.com

As is typical of Russel, to this group he says, "use Catch". To me he says, "Looks like Groovy and the JVM is the way of testing C++ code" ;-)

I've not really been part of the Java scene for nearly a decade (at which time anything to do with JNI was something to run screaming from - I don't know if that has improved much since) - but kudos to Michael for pulling off the interop necessary to make that work.

I've seen similar in the .Net world where someone asked if there was anything like NUnit for C++ and the response was "yes: NUnit" - using C++/CLI for the interop - which is limited to Windows - but otherwise the principle is the same.

Of course that was before Catch was born and these days when I have to use NUnit I actually miss unit testing in C++ !!!

So I thought it would be interesting to see how you might write Michael's original example using Catch (sorry if this is getting further off topic).

First: Catch has built in support for BDD style testing with GIVEN, WHEN and THEN - which take free-form strings as names.

It also has (unfinished, therefore undocumented) support for parameterised tests (which I presume is what Russel is referring to by @Unroll in Spock - and what the "where" section does in the original example?).

Unfortunately a big part of the "unfinishedness" of Generators is that they don't work with SECTIONS (which is what GIVEN, WHEN and THEN are). So we're not quite there yet.

In this case you don't really need generators anyway, as you can write a simple loop over an input vector to achieve the same results.

Unfortunately that hits the same problem with SECTIONs.

So I've written this example using a loop and GIVEN, WHEN and THEN sections. This doesn't currently work in practice - but if you just comment out the section macros it does work. Not very compelling, I know. I really must fix that issue.

I've had to make some assumptions about Michael's code under test - and included a simplified form here. I've used C++11 liberally, too, which cleans up a lot (although I haven't had to use any lambdas).

The section names are a bit vague as I don't really know what he underlying intent is. These names do not appear in the original examples. The fact that I couldn't reverse-engineer the intent out of the code suggests that putting these names in is a good idea!

#define CATCH_CONFIG_MAIN

#include "catch.hpp"

struct Gpr {

unsigned long R13;

unsigned long R14;

};

struct Xer {

int SO;

int OV;

int CA;

};

struct ProcessorState {

Gpr GPR;

Xer XER;

};

struct ProcessorTestState {

Gpr GPR;

Xer XER;

int existingCA;

};

typedef std::vector<ProcessorTestState> ProcessorStateVector;

void execute(std::string const& expr) { /* ... */ }

SCENARIO( "Can add to zero extended with overflow" ) {

ProcessorStateVector states = {

// R13 R14 SO OV CA existingCA

{ Gpr{ 0x1111111111111111, 0x1111111111111111 }, Xer{ true, false, false }, false },

{ Gpr{ 0x1111111111111111, 0x1111111111111111 }, Xer{ true, false, false }, true }

};

GIVEN("Some existing state")

{

for ( auto testState : states ) {

ProcessorState state;

state.GPR.R14 = testState.GPR.R14;

state.XER.CA = testState.XER.CA;

WHEN( "addzero is executed" )

{

execute( "addzero %r13, %r14" );

THEN( "The overflow flags are set appropriately" )

{

REQUIRE( state.GPR.R13 == testState.GPR.R13 );

REQUIRE( state.XER.SO == testState.XER.SO);

REQUIRE( state.XER.OV == testState.XER.OV );

REQUIRE( state.XER.CA == testState.XER.CA );

}

Luke Daley

unread,

Mar 23, 2014, 7:37:15 PM3/23/14

to spockfr...@googlegroups.com

Thanks for sharing this. Very interesting.

Michael Putters

unread,

Mar 24, 2014, 5:49:50 AM3/24/14

to spockfr...@googlegroups.com

Hi,

Thanks for taking the time to post all that information, it's actually useful (considering all the other subprojects' unit tests are still C++).

Sadly in your example, you're - unknowingly, of course - cheating: I cannot use the C++11 initializers the way you do, because the actual structure is something like:

struct {

UI64 GPRs[32];

UI64 SPRs[14];

UI32 CR;

};

So until C++11 gets the C99 named initialization (don't remember what the proper name is), I'll have to specify all the members even though I only need 1 or 2...

But still, outside the initialization, the code looks cleaner than what I had.

Also, regarding @Unroll: it's related to the where statement but not required. Without the annotation, Spock will test all the rows in a single test (like your loop). With the @Unroll annotation, each row gets its own individual test. And in most cases you don't need to have those columns passed as arguments to the test method, unless you want to force a type like I do (or have to).

Anyway I'll look into Catch.

Reply all

Reply to author

Forward