[Boost-users] [Boost.Test] [1.38] Automated test suie registration with muti-file-test-modules
Greg Christopher
We are starting to expand the number of test modules and are having some growing pains.
Originally we were using the "single header" variant included/unit_test.hpp file. Now we have switched to linking to the dynamic version of the boost test library and are including auto_unit_test.hpp.
We have been counting on the Test Tools to automatically register suites and test cases. This still worked when we just included unit_test.hpp (instead of included/unit_test.hpp).
However with multiple test modules, init_unit_test_suite is being defined in each module.
I am looking for clear documentation on what to do. This seems like the closest thing:
http://www.boost.org/doc/libs/1_31_0/libs/test/doc/components/unit_test_framework/components/test_case/auto_register_facility.html
However following those directions exactly when trying to use multiple test modules (in a single executable) and only one BOOST_AUTO_TEST_MAIN still results in init_unit_test_suite being multiply defined.
I would like multiple test suites, although the documentation for this declares that there is a single global test suite.
I apologize if this is well documented somewhere, but after an hour of googling I have to ask.
Greg
_______________________________________________
Boost-users mailing list
Boost...@lists.boost.org
http://lists.boost.org/mailman/listinfo.cgi/boost-users
Gennadiy Rozental
>
> Hi,
> We are starting to expand the number of test modules and are having
> some growing pains.
>
> Originally we were using the "single header" variant
> included/unit_test.hpp file. Now we have switched
> to linking to the dynamic version of the boost test library
> and are including auto_unit_test.hpp.
Just unit_test.hpp is fine. Older name is deprecated.
> We have been counting on the Test Tools to automatically register
> suites and test cases. This still worked when we just included
> unit_test.hpp (instead of included/unit_test.hpp).
>
> However with multiple test modules, init_unit_test_suite is being
> defined in each module.
This should work as you expect. Please post simple example illustrating
your issues.
> I would like multiple test suites, although the documentation for
> this declares that there is a single global test suite.
You can have multiple test suite under the master test suite.
Gennadiy
Gennadiy Rozental
>
> Hi,
> We are starting to expand the number of test modules and are having
> some growing pains.
>
> Originally we were using the "single header" variant
> included/unit_test.hpp file. Now we have switched
> to linking to the dynamic version of the boost test library
> and are including auto_unit_test.hpp.
Just unit_test.hpp is fine. Older name is deprecated.
> We have been counting on the Test Tools to automatically register
> suites and test cases. This still worked when we just included
> unit_test.hpp (instead of included/unit_test.hpp).
>
> However with multiple test modules, init_unit_test_suite is being
> defined in each module.
This should work as you expect. Please post simple example illustrating
your issues.
> I would like multiple test suites, although the documentation for
> this declares that there is a single global test suite.
You can have multiple test suite under the master test suite.
Gennadiy
_______________________________________________
Greg Christopher
> > Hi,
> > We are starting to expand the number of test modules and are having
> > some growing pains.
> >
> > Originally we were using the "single header" variant
> > included/unit_test.hpp file. Now we have switched
> > to linking to the dynamic version of the boost test library
> > and are including auto_unit_test.hpp.
>
> Just unit_test.hpp is fine. Older name is deprecated.
Ok- hopefully that's true for the version mentioned in the subject line.
>
> > We have been counting on the Test Tools to automatically register
> > suites and test cases. This still worked when we just included
> > unit_test.hpp (instead of included/unit_test.hpp).
> >
> > However with multiple test modules, init_unit_test_suite is being
> > defined in each module.
>
> This should work as you expect. Please post simple example illustrating
> your issues.
Test1.cpp:
-----
#include "stdafx.h"
#define BOOST_TEST_MODULE foo2
#include <boost/test/execution_monitor.hpp>
#include <boost/test/unit_test.hpp>
class DefaultFixture {
public:
DefaultFixture()
{
}
~DefaultFixture() {};
};
BOOST_AUTO_TEST_SUITE(foo2)
BOOST_FIXTURE_TEST_CASE(foo2, DefaultFixture)
{
BOOST_CHECK(4 == 4)
}
-----
Test2.cpp:
-----
#include "stdafx.h"
#define BOOST_TEST_MODULE foo
#include <boost/test/execution_monitor.hpp>
#include <boost/test/unit_test.hpp>
class DefaultFixture {
public:
DefaultFixture()
{
}
~DefaultFixture() {};
};
BOOST_AUTO_TEST_SUITE(foo)
BOOST_FIXTURE_TEST_CASE(foo, DefaultFixture)
{
BOOST_CHECK(4 == 4);
}
BOOST_AUTO_TEST_SUITE_END()
-----
And Link with boost library....
1>TestCase2.obj : error LNK2005: "class boost::unit_test::test_suite * __cdecl init_unit_test_suite(int,char * * const)" (?init_unit_test_suite@@YAPAVtest_suite@unit_test@boost@@HQAPAD@Z) already defined in basic.obj
Googling around, I discovered this cryptic define done in one of the files will fix it:
-----
#define init_unit_test_suite init_auto_unit_test_suite
-----
A strange observation as well....
Take this code:
#include "stdafx.h"
#define BOOST_TEST_MODULE foo
#include <boost/test/execution_monitor.hpp>
#include <boost/test/unit_test.hpp>
class DefaultFixture {
public:
DefaultFixture()
{
}
~DefaultFixture() {};
};
BOOST_AUTO_TEST_SUITE(foo)
BOOST_FIXTURE_TEST_CASE(foo, DefaultFixture)
{
BOOST_CHECK(4 == 4);
}
BOOST_AUTO_TEST_SUITE_END()
And change BOOST_FIXTURE_TEST_CASE toBOOST_TEST_CASE(foo)... it won't compile:
C:\basic\test2.cpp(21) : error C2882: 'foo' : illegal use of namespace identifier in expression
c:\code\main\test\bora-vmsoft\svga\tests\bat\basic\basic\test2.cpp(22) : error C2448: 'boost::unit_test::make_test_case' : function-style initializer appears to be a function definition
-----
Anyway, thanks for your help. The documentation right now seems to be lacking a little bit in giving clear examples of the different styles in which we are able to do test and suite registration.
Greg
Steven Watanabe
Greg Christopher wrote:
> Test1.cpp:
> -----
> #include "stdafx.h"
>
> #define BOOST_TEST_MODULE foo2
>
> <snip>
> -----
> Test2.cpp:
> -----
> #include "stdafx.h"
>
> #define BOOST_TEST_MODULE foo
> <snip>
> -----
> And Link with boost library....
>
Don't #define BOOST_TEST_MODULE in more than
one translation unit.
In Christ,
Steven Watanabe
Gennadiy Rozental
> Don't #define BOOST_TEST_MODULE in more than
> one translation unit.
This is covered in docs I believe. BOOST_TEST_MODULE has similar effect as
BOOST_TEST_MAIN, but also define the name of the master test suite
Gennadiy
Gennadiy Rozental
> And change BOOST_FIXTURE_TEST_CASE toBOOST_TEST_CASE(foo)... it won't compile:
Correct name is BOOST_AUTO_TEST_CASE. BOOST_TEST_CASE is used with
manual test case registration.
Gennadiy
Greg Christopher
> > Hi,
> > We are starting to expand the number of test modules and are having
> > some growing pains.
> >
> > Originally we were using the "single header" variant >
> included/unit_test.hpp file. Now we have switched > to linking to the
> dynamic version of the boost test library > and are including
> auto_unit_test.hpp.
>
> Just unit_test.hpp is fine. Older name is deprecated.
Ok- hopefully that's true for the version mentioned in the subject line.
>
> > We have been counting on the Test Tools to automatically register
> > suites and test cases. This still worked when we just included >
> unit_test.hpp (instead of included/unit_test.hpp).
> >
> > However with multiple test modules, init_unit_test_suite is being
> > defined in each module.
>
> This should work as you expect. Please post simple example
> illustrating your issues.
Test1.cpp:
-----
#include "stdafx.h"
#define BOOST_TEST_MODULE foo2
#include <boost/test/execution_monitor.hpp>
#include <boost/test/unit_test.hpp>
class DefaultFixture {
public:
DefaultFixture()
{
}
~DefaultFixture() {};
};
BOOST_AUTO_TEST_SUITE(foo2)
BOOST_FIXTURE_TEST_CASE(foo2, DefaultFixture) {
BOOST_CHECK(4 == 4)
}
-----
Test2.cpp:
-----
#include "stdafx.h"
#define BOOST_TEST_MODULE foo
#include <boost/test/execution_monitor.hpp>
#include <boost/test/unit_test.hpp>
class DefaultFixture {
public:
DefaultFixture()
{
}
~DefaultFixture() {};
};
BOOST_AUTO_TEST_SUITE(foo)
BOOST_FIXTURE_TEST_CASE(foo, DefaultFixture) {
BOOST_CHECK(4 == 4);
}
BOOST_AUTO_TEST_SUITE_END()
-----
And Link with boost library....
1>TestCase2.obj : error LNK2005: "class boost::unit_test::test_suite *
1>__cdecl init_unit_test_suite(int,char * * const)"
1>(?init_unit_test_suite@@YAPAVtest_suite@unit_test@boost@@HQAPAD@Z)
1>already defined in basic.obj
Googling around, I discovered this cryptic define done in one of the files will fix it:
-----
#define init_unit_test_suite init_auto_unit_test_suite
-----
A strange observation as well....
Take this code:
#include "stdafx.h"
#define BOOST_TEST_MODULE foo
#include <boost/test/execution_monitor.hpp>
#include <boost/test/unit_test.hpp>
class DefaultFixture {
public:
DefaultFixture()
{
}
~DefaultFixture() {};
};
BOOST_AUTO_TEST_SUITE(foo)
BOOST_FIXTURE_TEST_CASE(foo, DefaultFixture) {
BOOST_CHECK(4 == 4);
}
BOOST_AUTO_TEST_SUITE_END()
And change BOOST_FIXTURE_TEST_CASE toBOOST_TEST_CASE(foo)... it won't compile:
C:\basic\test2.cpp(21) : error C2882: 'foo' : illegal use of namespace identifier in expression
c:\code\main\test\bora-vmsoft\svga\tests\bat\basic\basic\test2.cpp(22) : error C2448: 'boost::unit_test::make_test_case' : function-style initializer appears to be a function definition
-----
Anyway, thanks for your help. The documentation right now seems to be lacking a little bit in giving clear examples of the different styles in which we are able to do test and suite registration.
Greg
_______________________________________________
Greg Christopher
> -----Original Message-----
> From: boost-use...@lists.boost.org [mailto:boost-users-
> bou...@lists.boost.org] On Behalf Of Gennadiy Rozental
> Sent: Tuesday, June 23, 2009 6:27 PM
> To: boost...@lists.boost.org
> Subject: Re: [Boost-users] [Boost.Test] [1.38] Automated test suie
> registration with muti-file-test-modules
>
> Steven Watanabe <watanabesj <at> gmail.com> writes:
>
> > Don't #define BOOST_TEST_MODULE in more than
> > one translation unit.
>
> This is covered in docs I believe. BOOST_TEST_MODULE has similar effect
> as
> BOOST_TEST_MAIN, but also define the name of the master test suite
Since I don't really need to define the name of the master test suite, I will not use BOOST_TEST_MODULE.
As it turns out, that still does not yield a working example. I then have no init_unit_test_suite() defined. By messing around, I find that defining BOOST_AUTO_TEST_MAIN fixes the problem. Looks like I need to do that in some module? I thought that the main program would be pulled out of the library if I didn't provide one. While that may be true, init_unit_test_suite is not defined.
So may I assume that for a fully automated situation with multiple test modules, the right approach is simply to insure that BOOST_AUTO_TEST_MAIN is defined in one module?
Thanks again,
Greg
Gennadiy Rozental
> > This is covered in docs I believe. BOOST_TEST_MODULE has similar effect
> > as
> > BOOST_TEST_MAIN, but also define the name of the master test suite
>
> Since I don't really need to define the name of the master test suite,
> I will not use BOOST_TEST_MODULE.
>
> As it turns out, that still does not yield a working example. I then
> have no init_unit_test_suite()
Please post an example and how you build it.
> defined. By messing around, I find that defining BOOST_AUTO_TEST_MAIN
> fixes the problem. Looks like I need to do that in some module?
No. BOOST_AUTO_TEST_MAIN is deprecated and should not be required.
> I thought that the main program would be pulled out of the library if
> I didn't provide one. While that may be true, init_unit_test_suite
> is not defined.
It may not have to be defined it you are using shared library variant of UTF.
Gennadiy
Greg Christopher
> -----Original Message-----
> From: boost-use...@lists.boost.org [mailto:boost-users-
> bou...@lists.boost.org] On Behalf Of Gennadiy Rozental
> Sent: Friday, July 03, 2009 11:48 PM
> To: boost...@lists.boost.org
> Subject: Re: [Boost-users] [Boost.Test] [1.38] Automated test suie
> registration with muti-file-test-modules
>
> Greg Christopher <gchristopher <at> vmware.com> writes:
>
> > > This is covered in docs I believe. BOOST_TEST_MODULE has similar
> effect
> > > as
> > > BOOST_TEST_MAIN, but also define the name of the master test suite
> >
> > Since I don't really need to define the name of the master test
> suite,
> > I will not use BOOST_TEST_MODULE.
> >
> > As it turns out, that still does not yield a working example. I then
> > have no init_unit_test_suite()
>
> Please post an example and how you build it.
The example is basically the same example that was posted earlier:
Test1.cpp:
#include "stdafx.h"
#define BOOST_TEST_MAIN Yes // Remove this line and we stop working
#include <boost/test/execution_monitor.hpp>
#include <boost/test/unit_test.hpp>
BOOST_AUTO_TEST_SUITE(foo1)
BOOST_AUTO_TEST_CASE(foo1case)
{
BOOST_CHECK(4 == 4);
}
BOOST_AUTO_TEST_SUITE_END()
Test2.cpp
#include "stdafx.h"
#include <boost/test/execution_monitor.hpp>
#include <boost/test/unit_test.hpp>
#define BOOST_TEST_MODULE Yeah
BOOST_AUTO_TEST_SUITE(foo2)
BOOST_AUTO_TEST_CASE(foo2case)
{
BOOST_CHECK(4 == 4);
}
BOOST_AUTO_TEST_SUITE_END()
Now when you link to the static version of the lib you get this:
1>libboost_unit_test_framework-vc90-mt-gd-1_38.lib(unit_test_main.obj) : error LNK2019: unresolved external symbol "class boost::unit_test::test_suite * __cdecl init_unit_test_suite(int,char * * const)" (?init_unit_test_suite@@YAPAVtest_suite@unit_test@boost@@HQAPAD@Z) referenced in function _main
Project is a simple Visual Studio 2008 one using static version of boost:
http://www.easy-share.com/1906667710/basic.zip
Thanks,
Greg
Greg Christopher
Ahmed Badran
multi-file test modules, I arrange my files as follows:
file1.cpp
#include <boost/test/unit_test.cpp>
....
BOOST_AUTO_TEST_SUITE(file1_test_suite)
... tests ...
BOOST_AUTO_TEST_SUITE_END()
- end of file 1 -
file2.cpp
#include <boost/test/unit_test.cpp>
....
BOOST_AUTO_TEST_SUITE(file2_test_suite)
... tests ...
BOOST_AUTO_TEST_SUITE_END()
- end of file 2 -
test_main.cpp
#define BOOST_TEST_MODULE master_test_suite
#include <boost/test/included/unit_test.hpp>
- end of test_main.cpp -
This causes a main, a test runner and the automatic generation of the
required functions in test_main.cpp.
I'm not that experienced using boost.test, but I've found this solution
to work nicely and fit my needs.
Regards,
Ahmed
Gennadiy Rozental
> #include <boost/test/execution_monitor.hpp>
> #include <boost/test/unit_test.hpp>
>
> #define BOOST_TEST_MODULE Yeah
You obviously need to define the symbol *before* include, not after.
Gennadiy
Greg Christopher
I am wondering what "non random" means (--random-0).
We are seeing the test framework behave such that with --random-0, it seems to do the tests in an order and stick with that order, but I'm not sure how it's deciding the order. If we re-link the test program, it seems to do them in a different order.
I would like to figure out how to make this deterministic without losing the benefit of not having to create my own test execution driver. I would think that non random would be in alphabetical order of test_case_name (we are doing auto registration) but that's not what we're seeing. Non-random seems to indicate maybe some kind of static initialization order, which is random.
Thanks for any help.
Greg
Steven Watanabe
Greg Christopher wrote:
> I am wondering what "non random" means (--random-0).
>
> We are seeing the test framework behave such that with --random-0, it seems to do the tests in an order and stick with that order, but I'm not sure how it's deciding the order. If we re-link the test program, it seems to do them in a different order.
>
> I would like to figure out how to make this deterministic without losing the benefit of not having to create my own test execution driver. I would think that non random would be in alphabetical order of test_case_name (we are doing auto registration) but that's not what we're seeing. Non-random seems to indicate maybe some kind of static initialization order, which is random.
>
You are correct that the order is effectively the static initialization
order.
This is not completely random. Test cases in the same translation unit
should be executed in order.
In Christ,
Steven Watanabe
P.S. Please start a new thread instead of replying to an existing thread.
Greg Christopher
As mentioned in thread below, test cases on boost.test may be being run based on static initialization. However, we are seeing them run in different order with each new link- perhaps because Microsoft is intentionally changing the way things fall in memory?
In the past I had created a test framework with auto-registration. In that framework, at static initialization time, objects registered themselves with a global parent object that had a container that was alphabetically sorted. Tests always ran in the same order, even if files were moved around in the project or makefile. At this point my tests aren't even running the same way when I recompile the same cpp file (there is only one cpp file in this test).
I guess I'm down to two questions:
1) does this sound like a good feature request, and if so how do I file it?
2) for now what's the shortest path to what I want while retaining auto-registration.
Thanks,
Greg
> -----Original Message-----
> From: boost-use...@lists.boost.org [mailto:boost-users-
> bou...@lists.boost.org] On Behalf Of Steven Watanabe
> Sent: Tuesday, October 13, 2009 8:16 PM
> To: boost...@lists.boost.org
Gennadiy Rozental
> way when I recompile the same cpp file
> (there is only one cpp file in this test).
These is something wrong going on here. The order should be well defined in this
case. Are you sure you do not have 'random' set through either using CLA or
environment? Can you create simple example illustrating the issues and provide
details of your configuration?
Gennadiy