[Googletest] Memory leak detection in Windows
Justin Holzer
April which is now closed:
http://groups.google.com/group/googletestframework/browse_thread/thread/e0123ea04c873283?tvc=2
I have implemented automated memory leak checking in my test fixtures
by using the _Crt* debugging functions provided in Windows. A base
test fixture class checks the state of memory in the SetUp() method,
and then in TearDown(), if HasFailure() returns false, the current
state of memory is compared with the state captured during SetUp() to
check for leaks (using _CrtMemCheckpoint and _CrtMemDifference).
This method works well except in some cases when the test fixture,
which is a derived/child class of the base test fixture class, uses
local member variables to provide common objects for all tests within
the fixture. For instance, the state of a local member variable may
change between SetUp() and TearDown(), and when TearDown() is called,
the test fixture's local data has not yet been cleaned up as the
destructor has not yet been called. This can trick the _Crt* functions
in to thinking that there are memory leaks because of the differences
in memory state between what was captured during SetUp() and TearDown
().
One solution to this issue was to move the code in the base class from
SetUp() into a default constructor, and from TearDown() into a virtual
destructor. This way, the initial memory state will be captured prior
to the child class initializing it's local member variables (since the
base class constructor is always called first), and the child test
fixture will be able to clean up it's local member data prior to the
final memory state being checked (since the base class destructor will
be called after the child class destructor). This solution does have
one caveat though.If a memory leak is detected, I would want to be
able to mark the test as failed (using the FAIL macro). I do not know
of any way to alter the outcome of the test once the destructor has
been reached. In addition, once the base class destructor is reached,
the child class has been cleaned up already.
Obviously, the FAIL macro cannot be used in the destructor since it
returns a value. Is there any possible way to change the pass/fail
status of a test in the destructor of a test fixture? If not, has
anyone come across a better solution for adding memory leak detection
(in Windows) that is safe to use with test fixture local member
variables? To work around this, all objects used by test cases can be
declared within the body of the tests themselves. However, this could
lead to lots of redundant coding. It's certainly not the end of the
world to use this workaround, but it would be optimal to be able to
use test fixtures for creating common objects used by all test cases
within the fixture and still be able to properly check for memory
leaks.
Here are some sample classes in which adding a local member variable
to the derived test fixture may trigger a false positive when checking
for memory leaks:
class BaseTestFixture : public ::testing::Test
{
public:
BaseTestFixture()
{
// to handle local member variables in derived classes, SetUp()
code would go here instead
}
virtual ~BaseTestFixture()
{
// to handle local member variables in derived classes, TearDown()
code would go here instead
}
virtual void SetUp() { // capture initial memory state here }
virtual void TearDown() { // capture ending memory state and compare
with state from SetUp() to check for leaks }
};
class MyTestFixture : public BaseTestFixture
{
protected:
virtual void SetUp() { BaseTestFixture::SetUp(); }
virtual void TearDown() { BaseTestFixture::TearDown(); }
// add any local member variables here
// ex. MySupportClass foo;
};
TEST_F(MyTestFixture, SomeTest) { ... }
- Justin
Jay Sprenkle
This post is really meant as a follow up to a post made back in March/
April which is now closed:
http://groups.google.com/group/googletestframework/browse_thread/thread/e0123ea04c873283?tvc=2
I have implemented automated memory leak checking in my test fixtures
by using the _Crt* debugging functions provided in Windows. A base
test fixture class checks the state of memory in the SetUp() method,
and then in TearDown(), if HasFailure() returns false, the current
state of memory is compared with the state captured during SetUp() to
check for leaks (using _CrtMemCheckpoint and _CrtMemDifference).
The following is the best code I have found so far to see memory usage on Linux.
Unfortunately it returns allocated memory pages which seem to be cached.
I think it's on demand loading code and leaving it in memory for some unknown period/until it needs to reclaim memory.
It's therefore pretty useless for determining if you have an actual leak.
#include <unistd.h>
#include <ios>
#include <iostream>
#include <fstream>
#include <string>
class mem_usage
{
public:
unsigned long vm_usage;
unsigned long resident_set;
mem_usage()
{
vm_usage = 0;
resident_set = 0;
};
const bool operator!=( const mem_usage& b ) const
{
return ( vm_usage != b.vm_usage || resident_set != b.resident_set );
};
};
void process_mem_usage(mem_usage& usage)
{
using std::ios_base;
using std::ifstream;
using std::string;
usage.vm_usage = 0;
usage.resident_set = 0;
ifstream stat_stream( "/proc/self/stat", ios_base::in );
// dummy vars for leading entries in stat that we don't care about
string pid, comm, state, ppid, pgrp, session, tty_nr;
string tpgid, flags, minflt, cminflt, majflt, cmajflt;
string utime, stime, cutime, cstime, priority, nice;
string O, itrealvalue, starttime;
// the two fields we want
unsigned long vsize;
long rss;
stat_stream >> pid >> comm >> state >> ppid >> pgrp >> session >> tty_nr
>> tpgid >> flags >> minflt >> cminflt >> majflt >> cmajflt
>> utime >> stime >> cutime >> cstime >> priority >> nice
>> O >> itrealvalue >> starttime >> vsize >> rss; // don't care about the rest
long page_size = sysconf(_SC_PAGE_SIZE); // / 1024; // in case x86-64 is configured to use 2MB pages
usage.vm_usage = vsize; // / 1024.0;
usage.resident_set = rss * page_size;
}
Justin Holzer
By including the appropriate header file, it will replace the standard
allocation methods with its own implementation (malloc, calloc,
realloc, free, etc.). I don't know if you can necessarily enable or
disable it to check only during specific times, but perhaps it would
be able to properly detect leaks without having to check memory at
specific intervals. I know that in Windows, the static singleton
object used by GoogleTest for running the test cases will cause the
CRT debugging tools to falsely report memory leaks, but that may not
necessarily be the case with other tools.
I also came across this article from 2003, about various applications
and libraries that can help detect memory leaks: http://www.linuxjournal.com/article/6556
As I also mentioned in my email to you yesterday, I have read good
things about Valgrind. Although it is a memory profiling app rather
than a library (to my knowledge), perhaps you could use it in such a
way where you would build the executable for your tests and then use
Valgrind to profile that executable. If that would work then there's
no additional coding require on your part. Just update your makefile
or whatever tool you use for your builds to tell it to automatically
run your test through Valgrind. I'm sure Valgrind can report false
positives, but I think it would at least be worth a try.
On Aug 8, 3:28 pm, Jay Sprenkle <jspren...@gmail.com> wrote:
>
> > This post is really meant as a follow up to a post made back in March/
> > April which is now closed:
>
Zhanyong Wan (λx.x x)
On Fri, Aug 7, 2009 at 12:16 PM, Justin Holzer<jsho...@gmail.com> wrote:
>
> This post is really meant as a follow up to a post made back in March/
> April which is now closed:
> http://groups.google.com/group/googletestframework/browse_thread/thread/e0123ea04c873283?tvc=2
>
> I have implemented automated memory leak checking in my test fixtures
> by using the _Crt* debugging functions provided in Windows. A base
> test fixture class checks the state of memory in the SetUp() method,
> and then in TearDown(), if HasFailure() returns false, the current
> state of memory is compared with the state captured during SetUp() to
> check for leaks (using _CrtMemCheckpoint and _CrtMemDifference).
>
> This method works well except in some cases when the test fixture,
> which is a derived/child class of the base test fixture class, uses
> local member variables to provide common objects for all tests within
> the fixture. For instance, the state of a local member variable may
> change between SetUp() and TearDown(), and when TearDown() is called,
> the test fixture's local data has not yet been cleaned up as the
> destructor has not yet been called. This can trick the _Crt* functions
> in to thinking that there are memory leaks because of the differences
> in memory state between what was captured during SetUp() and TearDown
> ().
>
> One solution to this issue was to move the code in the base class from
> SetUp() into a default constructor, and from TearDown() into a virtual
> destructor.
Sounds good.
Or, you can put the leak checking logic into a separate class (say,
LeakChecker) which does the work in its ctor and dtor. Then you can
add a member variable of type LeakChecker to your test fixture class.
> This way, the initial memory state will be captured prior
> to the child class initializing it's local member variables (since the
> base class constructor is always called first), and the child test
> fixture will be able to clean up it's local member data prior to the
> final memory state being checked (since the base class destructor will
> be called after the child class destructor). This solution does have
> one caveat though.If a memory leak is detected, I would want to be
> able to mark the test as failed (using the FAIL macro). I do not know
> of any way to alter the outcome of the test once the destructor has
> been reached.
Just use FAIL, ADD_FAILURE, or any other assertion macro to cause a
failure.
> In addition, once the base class destructor is reached,
> the child class has been cleaned up already.
This shouldn't be a problem. The test result is not stored in the
test fixture object, so you can generate new failures in the dtor of
the test fixture (or its base class).
> Obviously, the FAIL macro cannot be used in the destructor since it
> returns a value.
This is due to a pecularity of C++. The workaround is simple.
Instead of:
virtual ~Foo() {
...
FAIL(); // Doesn't compile.
...
}
write:
void Destruct() {
...
FAIL();
...
}
virtual ~Foo() { Destruct(); }
For those curious, FAIL() can be used in a void-returning function
only (for some technical reasons), and C++ doesn't consider ctors and
dtors as void-returning functions.
This is documented in
http://code.google.com/p/googletest/wiki/GoogleTestAdvancedGuide#Assertion_Placement.
Once the event listener API is implemented, you should be able to
register a listener that does the leak checking when a test
starts/ends. The benefit of this approach is that all TESTs and
TEST_Fs in your program are automatically checked - you don't have to
modify them to inherit from a leak checking base test fixture class.
However, since both Vlad and I are on vacation now, it will be a while
before this API can be finished.
--
Zhanyong
Justin Holzer
Thanks for the great advice! I think using the helper method in the
base test fixture destructor is going to be a great workaround for the
time being. At least that way, all I will need to do is extend the
base class, and no other code should be necessary. This seems like the
best option until the time when the event listener API is available.
Even though I will have to remember to extend my custom base class, a
quick macro/snippet will help to reduce the chance of errors there.
I will make the changes when I get to work in the morning and post the
code for the base fixture class so that others may hopefully be able
to benefit from it as well.
Thanks for all the help and advice,
- Justin
On Aug 8, 9:37 pm, Zhanyong Wan (λx.x x) <w...@google.com> wrote:
> Hi Justin,
>
>
>
>
>
> On Fri, Aug 7, 2009 at 12:16 PM, Justin Holzer<jshol...@gmail.com> wrote:
>
> > This post is really meant as a follow up to a post made back in March/
> > April which is now closed:
Justin Holzer
memory leak checking in the constructor and destructor. It can be
found here:
http://dev.opaquepixel.com/code/LeakCheckingTestFixture.h
This class is a first draft and has not been heavily tested, so use at
your own risk. The class makes use of Microsoft's CRT (C Runtime)
debugging methods. At this time, I do not have any implementation for
other platforms, as all of my current development is being done on
Win32. If time permits in the future, I may look at attempting to make
the class a bit more robust, but for now, it does the job. It should
be noted that the MSVC++ compiler strips out calls to the CRT
debugging methods if "_DEBUG" is not defined. As a result, the base
class will not currently compile in a non-debug build.
If you are interested enough to have read to this point, then I
strongly recommend reading more in to the CRT memory debugging and
reporting methods:
http://msdn.microsoft.com/en-us/library/974tc9t1.aspx
In order to use the class, some initial configuration/setup is
required:
1. The symbol _CRTDBG_MAP_ALLOC must be #define'd in order to use
the CRT memory debugging methods (this is a requirement from MS, not
me :) )
2. The crt debugging methods are accessed by including <crtdbg.h>
3. The base class will dump memory reports in the case a leak is
detected. By default, reports will be dumped to the debugger's output
stream (i.e. in VC++, the debug output window). However, you can have
the methods dump output to multiple streams (for instance, if you have
a logging implementation or want to use STDERR or STDOUT). The
function _CrtSetReportMode will allow you to set the streams that the
methods will dump reports to.
Example of using _CrtSetReportMode...
int main(int argc, char** argv)
{
// tell CRT debug methods to dump memory reports to the debugger's
output window as well as STDERR
_CrtSetReportMode(_CRT_WARN, _CRTDBG_MODE_DEBUG |
_CRTDBG_MODE_FILE);
_CrtSetReportFile(_CRT_WARN, _CRTDBG_FILE_STDERR);
// initialize Google Mock and Test frameworks
::testing::InitGoogleMock(&argc, argv);
// run tests and immediately return the result
return RUN_ALL_TESTS();
}
I hope some of you guys and gals using GoogleTest in Windows will find
this code useful.
- Justin
Jay Sprenkle
I have posted the code for a test fixture base class that will perform
memory leak checking in the constructor and destructor. It can be
found here:
http://dev.opaquepixel.com/code/LeakCheckingTestFixture.h
Hey Justin,
I did some poking around and found the gnu malloc system includes a call to determine how much memory is currently allocated. The following is able to find failures in both malloc and new, at least with my trivial testing:
#include <malloc.h>
/**
* A base GTest (GoogleTest) text fixture class that supports memory leak checking.
*
* NOTE: when using this class as a base class for a test fixture,
* the derived class should not create any local member variables, as
* they can cause memory leaks to be improperly reported.
*/
class BaseTestFixture : public ::testing::Test
{
public:
{
// capture the memory state at the beginning of the test
struct mallinfo minfo = mallinfo();
beginMemoryState = minfo.uordblks;
}
virtual void TearDown()
{
// only check for memory leaks if the test did not end in a failure
if (!HasFailure())
{
// if there are differences between the memory state at beginning and end, then there are memory leaks
struct mallinfo minfo = mallinfo();
if ( beginMemoryState != minfo.uordblks )
{
FAIL() << "Memory Leak(s) Detected!";
}
}
}
private:
// memory state at the beginning of the test fixture set up
int beginMemoryState;
};
//----------------------------------------------------------------
// check that allocating nothing doesn't throw a false positive
//----------------------------------------------------------------
TEST_F(BaseTestFixture, BaseTestFixtureTest)
{
// should always pass an empty test
}
//----------------------------------------------------------------
// check that malloc()ing something is detected
//----------------------------------------------------------------
TEST_F(BaseTestFixture, BaseTestFixtureMallocTest)
{
// should always fail
void* p = malloc(10);
}
//----------------------------------------------------------------
// check that new()ing something is detected
//----------------------------------------------------------------
TEST_F(BaseTestFixture, BaseTestFixtureNewFailTest)
{
// should always fail
int* array = new int[10];
}
//----------------------------------------------------------------
//
//----------------------------------------------------------------
TEST_F(BaseTestFixture, BaseTestFixtureNewTest)
{
void* p = malloc(10);
free( p );
}
[==========] Running 4 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 4 tests from BaseTestFixture
[ RUN ] BaseTestFixture.BaseTestFixtureTest
[ OK ] BaseTestFixture.BaseTestFixtureTest (0 ms)
[ RUN ] BaseTestFixture.BaseTestFixtureMallocTest
../main.cpp:860: Failure
Failed
Memory Leak(s) Detected!
[ FAILED ] BaseTestFixture.BaseTestFixtureMallocTest (0 ms)
[ RUN ] BaseTestFixture.BaseTestFixtureNewFailTest
../main.cpp:860: Failure
Failed
Memory Leak(s) Detected!
[ FAILED ] BaseTestFixture.BaseTestFixtureNewFailTest (0 ms)
[ RUN ] BaseTestFixture.BaseTestFixtureNewTest
[ OK ] BaseTestFixture.BaseTestFixtureNewTest (0 ms)
[----------] 4 tests from BaseTestFixture (0 ms total)
[----------] Global test environment tear-down
[==========] 4 tests from 1 test case ran. (0 ms total)
[ PASSED ] 2 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] BaseTestFixture.BaseTestFixtureMallocTest
[ FAILED ] BaseTestFixture.BaseTestFixtureNewFailTest
2 FAILED TESTS
Zhanyong Wan (λx.x x)
Thanks for sharing your solution! I'm sure it will help other people.
On Mon, Aug 10, 2009 at 7:41 AM, Justin Holzer<jsho...@gmail.com> wrote:
>
>
> I have posted the code for a test fixture base class that will perform
> memory leak checking in the constructor and destructor. It can be
> found here:
>
> http://dev.opaquepixel.com/code/LeakCheckingTestFixture.h
Some quick comments:
- For maximum reusability, it's better to define the leak checker as a
stand-alone class (not derived from testing::Test). If someone
wants leak checking for a test case, he just needs to add an
instance of the leak checker as a data member of his test fixture
class, instead of inheriting from LeakCheckingTestFixture. This
allows different enhancements to be easily combined in the same
fixture. (Suppose someone else develops another enchancement also
as a subclass of testing::Test (say, BarFixture), you cannot derive
your fixture from both LeakCheckingTestFixture and BarFixture, as
both classes share the same, duplicated base class testing::Test.
In contrast, you can easily use both enhancements in the same
fixture if they aren't subclasses of testing::Test.)
- You can replace the call to failTest(...) with ADD_FAILURE() << ....
Then you don't need the failTest() function.
--
Zhanyong
Zhanyong Wan (λx.x x)
On Mon, Aug 10, 2009 at 7:57 PM, Jay Sprenkle<jspr...@gmail.com> wrote:
>
>>
>> I have posted the code for a test fixture base class that will perform
>> memory leak checking in the constructor and destructor. It can be
>> found here:
>>
>> http://dev.opaquepixel.com/code/LeakCheckingTestFixture.h
>
> Hey Justin,
> I did some poking around and found the gnu malloc system includes a call to
> determine how much memory is currently allocated. The following is able to
> find failures in both malloc and new, at least with my trivial testing:
>
> #include <malloc.h>
>
> /**
> * A base GTest (GoogleTest) text fixture class that supports memory leak
> checking.
> *
> * NOTE: when using this class as a base class for a test fixture,
> * the derived class should not create any local member variables, as
> * they can cause memory leaks to be improperly reported.
> */
> class BaseTestFixture : public ::testing::Test
It's better to define this as a stand-alone class (i.e. not a subclass
of testing::Test), as I pointed out in my reply to Justin.
> {
> public:
> virtual void SetUp()
It's better to do the leak checking in the ctor/dtor as opposed to
SetUp/TearDown:
- It can catch leaks caused in the ctor/dtor of a fixture derived from
BaseTestFixture.
- When the user needs to define his own SetUp/TearDown in a derived
fixture class, he doesn't need to remember to call SetUp/TearDown in
BaseTestFixture.
--
Zhanyong
Justin Holzer
If you look at one of Zhanyong's earlier posts, he points out how to
use the FAIL() macro from within a destructor, and also points to
where this is written in the Google Test Advanced Guide. I originally
did not think it was possible to call FAIL() within the context of the
destructor, but since it is, you would definitely want to use the ctor/
dtor for your leak checking. This will allow you to define member
variables in your test fixtures without generating false positives.
Zhanyong,
If the class that provides the memory leak checking did not extend
testing::Test, would the FAIL macro still work properly? I would think
the answer would be yes, but I haven't had the chance to try it yet.
Also, wouldn't it also work to just define the leak checking class as
an additional base class of the test fixture?
For instance:
class MyTestFixture : public testing::Test, public LeakChecker
Or perhaps there could still be a base test class that would have the
same signature as above and test fixtures could just extend it:
class LeakCheckingTest : public testing::Test, public LeakChecker
{};
class MyTestFixture :: public LeakCheckingTest {};
I see the point about portability, in that a class that is doing
memory leak checking certainly has applications outside of test cases.
I just don't feel comfortable about declaring a member variable to
provide the memory leak checking. I guess I just always felt that
inheritance was the "proper" way in OOD to inject additional
functionality in to a class hierarchy, though I suppose it's really a
matter of personal preference.
I will try a minor refactoring with a separate LeakChecker class and
see how that goes. Thanks for all the great feedback.
- Justin
Zhanyong Wan (λx.x x)
> Zhanyong,
>
> If the class that provides the memory leak checking did not extend
> testing::Test, would the FAIL macro still work properly? I would think
> the answer would be yes, but I haven't had the chance to try it yet.
The answer is yes.
http://code.google.com/p/googletest/wiki/GoogleTestAdvancedGuide#Assertion_Placement
says:
"You can include assertions in any C++ function's code. The one
constraint is that assertions that generate a fatal failure (FAIL* and
ASSERT_*) can only be used by void-returning functions."
Perhaps we should make it more obvious that by "any" we mean the
funciton doesn't have to be a member of the test fixture class.
> Also, wouldn't it also work to just define the leak checking class as
> an additional base class of the test fixture?
>
> For instance:
> class MyTestFixture : public testing::Test, public LeakChecker
>
> Or perhaps there could still be a base test class that would have the
> same signature as above and test fixtures could just extend it:
> class LeakCheckingTest : public testing::Test, public LeakChecker
> {};
> class MyTestFixture :: public LeakCheckingTest {};
>
> I see the point about portability, in that a class that is doing
> memory leak checking certainly has applications outside of test cases.
This is a good point, but not the same one I was making. I was saying
that a test fixture class may want to use both leak checking and some
other utility. This wouldn't be possible if both leak checking and
the other utility are implemented as subclasses of testing::Test.
> I just don't feel comfortable about declaring a member variable to
> provide the memory leak checking. I guess I just always felt that
> inheritance was the "proper" way in OOD to inject additional
> functionality in to a class hierarchy, though I suppose it's really a
> matter of personal preference.
Not entirely. Inheritance is for the "is-a" relationship, while
composition is for the "uses" relationship. In this case, the test
fixtures *uses* a leak checker, but itself is not a leak checker.
Therefore it's more appropriate to make the leak checker a member
variable.
For more info on why implementation inheritance and multi-inheritance
are suspicious, you can read
http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml#Inheritance
> I will try a minor refactoring with a separate LeakChecker class and
> see how that goes. Thanks for all the great feedback.
>
> - Justin
--
Zhanyong
Justin Holzer
You bring up some excellent points. Even with a separate LeakChecker
object, wouldn't a base test fixture class still be needed to avoid
getting false positives from test fixture member variables? The memory
leak checking needs to be done prior the initialization of test
fixture member variables, and after their destruction, and the only
way I know of that will do this is by using the ctor/dtor of a base
class.
So you might end up with something like...
class BaseTestFixture : public testing::Test
public:
virtual BaseTestFixture()
{
leakChecker.Start(); // have the LeakChecker capture initial
memory state
}
virtual ~MyTestFixture()
{
leakChecker.Finish(); // tell the leak checker to compare current
mem state with the initial one
if (leakChecker.FoundLeaks())
failTest("Memory leak detected!");
}
private:
void failTest(const char* message)
{
FAIL() << message;
}
LeakChecker leakChecker;
};
Down the road, if I needed additional features in my test cases, I
could either enhance the BaseTestFixture class or create a new
subclass that my test fixtures requring the added functionality could
themselves inherit from. Am I still missing the point here?
- Justin
On Aug 11, 1:48 pm, Zhanyong Wan (λx.x x) <w...@google.com> wrote:
> > Zhanyong,
>
> > If the class that provides the memory leak checking did not extend
> > testing::Test, would the FAIL macro still work properly? I would think
> > the answer would be yes, but I haven't had the chance to try it yet.
>
Zhanyong Wan (λx.x x)
>
> Zhanyong,
>
> You bring up some excellent points. Even with a separate LeakChecker
> object, wouldn't a base test fixture class still be needed to avoid
> getting false positives from test fixture member variables?
No. You just need to declare the LeakChecker object as the *first*
member variable of the fixture class. C++ guarantees that it's
constructed before and destructed after other member variables.
> The memory
> leak checking needs to be done prior the initialization of test
> fixture member variables, and after their destruction, and the only
> way I know of that will do this is by using the ctor/dtor of a base
> class.
>
> So you might end up with something like...
>
> class BaseTestFixture : public testing::Test
> {
> public:
> virtual BaseTestFixture()
> {
> leakChecker.Start(); // have the LeakChecker capture initial
> memory state
> }
>
> virtual ~MyTestFixture()
> {
> leakChecker.Finish(); // tell the leak checker to compare current
> mem state with the initial one
>
> if (leakChecker.FoundLeaks())
> failTest("Memory leak detected!");
> }
>
> private:
> void failTest(const char* message)
> {
> FAIL() << message;
> }
>
> LeakChecker leakChecker;
> };
This wouldn't be necessary.
> Down the road, if I needed additional features in my test cases, I
> could either enhance the BaseTestFixture class or create a new
> subclass that my test fixtures requring the added functionality could
> themselves inherit from. Am I still missing the point here?
If you have N enhancements, how many combinations do you want to
provide? You'd end up with 2 to the power of N variations, which is
infeasible. It's best to define the N enhancements separately and
allow them to be freely combined by the user.
--
Zhanyong