Group setup/teardown

[Andreas Stenius]

Hi,

I'd like a test group setup/teardown function. 

Is this already possible, or will I need to patch it in to get it to work?

//Andreas

[Bas Vodde]

Hi Andreas,

Nope, it isn't really build in at the moment. Every now and then it pops up and usually there are better ways of doing that than having a test-group setup/teardown. Because of that, I've been hesitant to actually implement it as it seems that when you need it then it is better to think more than to use it.

Could you describe the specific case in which you would want it?

Bas

[Andreas Stenius]

Hi Bas,

I have a module under test. But before carrying out the tests, I need to call a module_init() function. this init call is kind of expensive (time-wise), so I'd rather avoid calling it for each test. And when all tests are done, there are some by products in the file system I'd like to clean up.

Some thinking (thanks for the nudge ;) gave me the idea that I could call the init function from the normal TEST_SETUP, and simply have local flag in the test group set when this call has been done, and skip it for the subsequent tests.

The teardown was trickier, unless there is a way to find out if this is the last test to run for the test group.

I want to keep the temp files during the entire test group, since it is the init call that created them in the first place.

Hope it made some sort of sense... 

I don't mind rethinking my problem to avoid group setup/teardown if that would be possible... it just was the straight forward idea that came to mind... :)

2012/6/24 Bas Vodde <ba...@odd-e.com>

[Bas Vodde]

Hola,

> I have a module under test. But before carrying out the tests, I need to call a module_init() function. this init call is kind of expensive (time-wise), so I'd rather avoid calling it for each test. And when all tests are done, there are some by products in the file system I'd like to clean up.

Oki, splitting the init into parts so that you can only init the things you need for the test isn't an option?

> Some thinking (thanks for the nudge ;) gave me the idea that I could call the init function from the normal TEST_SETUP, and simply have local flag in the test group set when this call has been done, and skip it for the subsequent tests.
> The teardown was trickier, unless there is a way to find out if this is the last test to run for the test group.

Actually, there is, sort-of :) The way thats done internal in the framework is by comparing the test-group of the next test :) This assumes that the developer doesn't mixes multiple testgroups in one file or spread it out over multiple files (an assumption that is yet to be broken :P)
This is internally used for the "TestGroupStart" which is called on TestPlugins (which is another way of doing it, if you insist, it has a groupStart and a groupEnd :P)

When implementing this functionality, I thought about adding a setup/teardown for the group, but after thinking it over for quite long, I decided that adding that functionality would probably promote the wrong behavior...

> I want to keep the temp files during the entire test group, since it is the init call that created them in the first place.

Ok? How about stubbing out the temp files so they don't go to the filesystem?

> Hope it made some sort of sense...

Sort-of :) It confirms that there might be a better way of dealing with it :)

> I don't mind rethinking my problem to avoid group setup/teardown if that would be possible... it just was the straight forward idea that came to mind... :)

Oh, another thing. If you DO go into the setup/teardown direction you described, then you might get into a fight with the memory leak detector as memory allocated in group setup will be considered to be leaking. If you use the plugin mechanism, then you won't have that. This is because the memory leak detector itself is a plugin, so you can control your plugin to be called before the memory leak detector one...

Hope this helps?

Bas

[Andreas Stenius]

مرحبا,

Thanks for your very prompt replies :)

As I started writing the tests, it may have been premature to assume the cost of reinitializing before each test case.

Sure, I can stub out the file system parts (and have done so for other modules in the past). I'll need to work some more with the tests to see where I'm headed with it.

But I will most likely avoid a group setup/teardown.

Thanks for your input. I'll let you know how it turns out ;)

Btw,

I have made some changes to use CppUTest in a gnu autotools project. Some tweaking was needed to get all the pieces together.

For interested parties, my changes are available from https://git.astekk.se/pub/pub/cpputest/

Actually, it wasn't that much, when I look back at it. The two changes that I still rely on are:

https://git.astekk.se/pub/pub/cpputest/commit/?h=pub&id=5f9e33ddef144694badab1118afe9b9d7ca75556

and

https://git.astekk.se/pub/pub/cpputest/commit/?h=pub&id=e7667f545ea1fceea7161a7cf74abb318da79d75

The later had me quite puzzled, as I tried to understand why I had issues with it, and no one else... ?

I resolved to patching the code, as I didn't really come to grips with it.

//Andreas

2012/6/24 Bas Vodde <ba...@odd-e.com>

[Bas Vodde]

Hola,

> As I started writing the tests, it may have been premature to assume the cost of reinitializing before each test case.

Oki, if it ever does get slow, you might want to look at making it faster rather than doing it less often.

However you look at it, using a test-case setup will cause more test dependencies...

> Sure, I can stub out the file system parts (and have done so for other modules in the past). I'll need to work some more with the tests to see where I'm headed with it.
> But I will most likely avoid a group setup/teardown.

Oki! Let me know if I can still help :)

> Thanks for your input. I'll let you know how it turns out ;)

Good.

> Btw,
> I have made some changes to use CppUTest in a gnu autotools project. Some tweaking was needed to get all the pieces together.

Kewl. I'll try to get some autotools support for CppUTest also these days.

> For interested parties, my changes are available from https://git.astekk.se/pub/pub/cpputest/
> Actually, it wasn't that much, when I look back at it. The two changes that I still rely on are:
> https://git.astekk.se/pub/pub/cpputest/commit/?h=pub&id=5f9e33ddef144694badab1118afe9b9d7ca75556
> and
> https://git.astekk.se/pub/pub/cpputest/commit/?h=pub&id=e7667f545ea1fceea7161a7cf74abb318da79d75

Oki, I've integrated these on the trunk (second one slightly different). Let me know if you have others still. Thanks for these!

> The later had me quite puzzled, as I tried to understand why I had issues with it, and no one else... ?

Yah, agree.

> I resolved to patching the code, as I didn't really come to grips with it.

Well, the second one is a switch that is rarely used as it is only when you do kernel development. It is somewhat in development :)

Bas

[Andreas Stenius]

Hi,

2012/6/28 Bas Vodde <ba...@odd-e.com>:

>
> Hola,
>
>> As I started writing the tests, it may have been premature to assume the cost of reinitializing before each test case.
>
> Oki, if it ever does get slow, you might want to look at making it faster rather than doing it less often.

Yeah, as it became, I split the initialization so I can choose what
part to initialize, as well as avoid saving data to disk for later.

> However you look at it, using a test-case setup will cause more test dependencies...
>
>> Sure, I can stub out the file system parts (and have done so for other modules in the past). I'll need to work some more with the tests to see where I'm headed with it.
>> But I will most likely avoid a group setup/teardown.
>
> Oki! Let me know if I can still help :)

As a rather lone developer, simply describing what I do (or try to do) helps :)

>> Thanks for your input. I'll let you know how it turns out ;)
>
> Good.
>
>> Btw,
>> I have made some changes to use CppUTest in a gnu autotools project. Some tweaking was needed to get all the pieces together.
>
> Kewl. I'll try to get some autotools support for CppUTest also these days.

I'd love it if I could build the whole thing with autotools ;)
I've made another change to accomodate my target device with limited memory.
https://git.astekk.se/pub/pub/cpputest/commit/?h=pub&id=9d120aabf6f69180a9482d7008e5aa2b6519b121

That is so I can link my own cpputest test binaries, since I can't fit
the whole lot into memory at once. But still use the standard cpputest
makefiles to build the obj files.

>> For interested parties, my changes are available from https://git.astekk.se/pub/pub/cpputest/
>> Actually, it wasn't that much, when I look back at it. The two changes that I still rely on are:
>> https://git.astekk.se/pub/pub/cpputest/commit/?h=pub&id=5f9e33ddef144694badab1118afe9b9d7ca75556
>> and
>> https://git.astekk.se/pub/pub/cpputest/commit/?h=pub&id=e7667f545ea1fceea7161a7cf74abb318da79d75
>
> Oki, I've integrated these on the trunk (second one slightly different). Let me know if you have others still. Thanks for these!

Nice. I've merged it and pushed a few more changes of my own to
https://git.astekk.se/pub/pub/cpputest/log/?h=pub

It is rather hackish and work in progress, and needs cleaning up.
Still, feedback welcome. ;)
I deserve some whacking along my fingers for not having written any
tests for the MEMCMP_EQUAL stuff... :/
(and the accompanying additions to StringFrom)

>> The later had me quite puzzled, as I tried to understand why I had issues with it, and no one else... ?
>
> Yah, agree.
>
>> I resolved to patching the code, as I didn't really come to grips with it.
>
> Well, the second one is a switch that is rarely used as it is only when you do kernel development. It is somewhat in development :)

Ah, ok. Thanks for cleaning that up a bit ;)

//Andreas

[Bas Vodde]

Hiya,

Old mail. Quick reply.

>> Kewl. I'll try to get some autotools support for CppUTest also these days.
>
> I'd love it if I could build the whole thing with autotools ;)
> I've made another change to accomodate my target device with limited memory.
> https://git.astekk.se/pub/pub/cpputest/commit/?h=pub&id=9d120aabf6f69180a9482d7008e5aa2b6519b121
>
> That is so I can link my own cpputest test binaries, since I can't fit
> the whole lot into memory at once. But still use the standard cpputest
> makefiles to build the obj files.

I've merged this change in MakefileWorker

> It is rather hackish and work in progress, and needs cleaning up.
> Still, feedback welcome. ;)
> I deserve some whacking along my fingers for not having written any
> tests for the MEMCMP_EQUAL stuff... :/
> (and the accompanying additions to StringFrom)

I think we'll need to get a MEMCMP_EQUAL in CppUTest in the future. I opened an issue for this :)

Thanks,

Bas