Re: [googletest] whats the order of the test cases executed

[Greg Miller]

The order of test execution is undefined. It's best to not write tests that depend on the order of execution.

HTH,

Greg

On Wed, Sep 19, 2012 at 3:45 PM, prafullakumar palwe <prafull...@gmail.com> wrote:

I have defined 2 source files with different test cases.

When I run the executables it runs test cases in specific order?

 

What if I want to change the order of the test cases executed?

By default how it choses the order of the test cases executed?

[Matthew Woehlke]

Also keep in mind that the user can specify what tests to run. If TestA
depends on TestB running first, and the user asks to only run TestA, you
are going to have problems...

In some cases, tests might even run concurrently (e.g. running gtest
tests through CTest; running multiple instances, especially with repeat
count; etc.).

I would expand on what Greg said, and say it is best not to write tests
that depend on other tests in any fashion.

Matthew

[Evan Driscoll]

On Wednesday, September 19, 2012 2:45:03 PM UTC-5, prafullakumar palwe wrote:

I have defined 2 source files with different test cases.

When I run the executables it runs test cases in specific order?

 

What if I want to change the order of the test cases executed?

By default how it choses the order of the test cases executed?

I'm going to resurrect this thread and actually request this as a feature which I would consider writing. (In particular, it'd be nice if saying --gtest_filter=A:B:C ran them in that order if --gtest_shuffle was not provided.)

The reason is not that I want to write tests that depend on the order they're executed in (as Greg and Matthew wisely advise against), but rather that I have already done so *accidentally* and would like more help in diagnosing the cause. For instance, if I know that I get a failure with A then B then C, do I get the same failure with B then A then C?

(I'm actually in the unfortunate situation right now of having *seven* tests for which the last one fails when run in sequence, and if I remove any single test it passes. It Valgrinds clean.)

Any idea how difficult this would be to implement?

Evan

[Zhanyong Wan (λx.x x)]

On Tue, Feb 5, 2013 at 12:03 PM, Evan Driscoll <evaned...@gmail.com> wrote:

On Wednesday, September 19, 2012 2:45:03 PM UTC-5, prafullakumar palwe wrote:

I have defined 2 source files with different test cases.

When I run the executables it runs test cases in specific order?

 

What if I want to change the order of the test cases executed?

By default how it choses the order of the test cases executed?

I'm going to resurrect this thread and actually request this as a feature which I would consider writing. (In particular, it'd be nice if saying --gtest_filter=A:B:C ran them in that order if --gtest_shuffle was not provided.)

I like the idea.

Note that in general, the filter may contain wildcard and negative components.  We can rank the tests by the *first* positive filter that matches it, and promise to run the tests in the order of their ranks (tests with the same rank will be run in an unspecified order).

 

The reason is not that I want to write tests that depend on the order they're executed in (as Greg and Matthew wisely advise against), but rather that I have already done so *accidentally* and would like more help in diagnosing the cause. For instance, if I know that I get a failure with A then B then C, do I get the same failure with B then A then C?

(I'm actually in the unfortunate situation right now of having *seven* tests for which the last one fails when run in sequence, and if I remove any single test it passes. It Valgrinds clean.)

Any idea how difficult this would be to implement?

It's not trivial but shouldn't be very hard either.  The key is to implement the ranking correctly.  Thanks,

 

Evan

--
 
---
You received this message because you are subscribed to the Google Groups "Google C++ Testing Framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to googletestframe...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Zhanyong

[Evan Driscoll]

On Tuesday, February 5, 2013 2:18:31 PM UTC-6, Zhanyong Wan wrote:

I like the idea.

Note that in general, the filter may contain wildcard and negative components.  We can rank the tests by the *first* positive filter that matches it, and promise to run the tests in the order of their ranks (tests with the same rank will be run in an unspecified order).

Is there a Git (preferably) or Hg version of the repository somewhere? Using a DVCS dramatically lowers the overhead of working on a third-party project without commit access.

I found what seems to be a Chromium mirror: http://git.chromium.org/gitweb/?p=external/googletest.git;a=summary ; do you know if I can hijack that? The current tips are identical.

Evan

[Zhanyong Wan (λx.x x)]

On Tue, Feb 5, 2013 at 1:14 PM, Evan Driscoll <evaned...@gmail.com> wrote:

On Tuesday, February 5, 2013 2:18:31 PM UTC-6, Zhanyong Wan wrote:

I like the idea.

Note that in general, the filter may contain wildcard and negative components.  We can rank the tests by the *first* positive filter that matches it, and promise to run the tests in the order of their ranks (tests with the same rank will be run in an unspecified order).

Is there a Git (preferably) or Hg version of the repository somewhere? Using a DVCS dramatically lowers the overhead of working on a third-party project without commit access.

Sorry, gtest only has an svn repo as that was the only option on Google Code when gtest was first released.

 

I found what seems to be a Chromium mirror: http://git.chromium.org/gitweb/?p=external/googletest.git;a=summary ; do you know if I can hijack that? The current tips are identical.

I don't know - haven't used that myself.  I think it should be fine as long as you can produce a standard patch file from it.

If you are serious about this, please read the gtest dev guide (https://code.google.com/p/googletest/wiki/DevGuide).  In particular, the part about signing the CLA.  Thanks!

 

Evan

--
 
---
You received this message because you are subscribed to the Google Groups "Google C++ Testing Framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to googletestframe...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Zhanyong

[Evan Driscoll]

On Tuesday, February 5, 2013 3:26:30 PM UTC-6, Zhanyong Wan wrote:

> Sorry, gtest only has an svn repo as that was the only option on Google Code when gtest was first released.

Any chance of a switch at some point in the future?

 

If you are serious about this, please read the gtest dev guide (https://code.google.com/p/googletest/wiki/DevGuide).  In particular, the part about signing the CLA.  Thanks!

 

OK, FWIW I have a patch that seems to work in a way that was good enough for me to figure out my problem. I'm not formally submitting it yet for the reasons discussed below, but if you want to pick it up and run with it feel free (I've submitted a CLA under a different email address). Otherwise I'll try to come back to it in... a while (probably not for another month or so, unfortunately) and tighten things up. Feel free to comment on the overall approach, request identifier changes, say you don't like the index/-1 scheme, etc.

Featurewise, my explanation above was a bit too simple, because I didn't appreciate the interaction of the hierarchy of test cases with the ordering. If the user says --gtest_filter=A.2:B.1:A.1, I can't run them in that order because it's splitting up the A test case. What I do is use the following rule: each individual test is assigned a rank (as you suggest) which is the first pattern in the filter that it matches. Then I assign a rank to each test case which is the lowest (highest-priority) rank of any of its individual tests. So for that --gtest_filter line, it would run A.2, then A.1, then B.1. (At least that's the idea. :-))

Incomplete parts/problems. First, right now I don't do anything special with death tests, so I'm not sure if they'll still run first. Second, in the test suite that I was having problems with, it seems to *sometimes* ignore ordering of tests in a test case and sometimes not. If I say --gtest_filter=A.3:A.2:A.1 it runs A.1 then A.3 then A.2 (so A.1 is at the start instead of end), but if I say --gtest_filter=A.2:A.3:A.1 it runs A.1 then A.2 then A.3 (so switching A.2 and A.3 in the filter changed their order, but A.1 is still misplaced). That seems very strange to me but I don't really have time to debug it at the moment. Finally, there are no tests of the new functionality.

Implementation wise, at a high level I replaced Boolean match/no match queries with ones that return an index into the filter list or -1 on a non-match. (So with --gtest_filter=A.2:B.1:A.1, the TestInfo for B.1 would have rank 1.) I started off calling this "index" but decided I like "rank" better so switched midway through. I'll change them later if someone else doesn't get to them first. So for instance, TestInfo's 'bool matches_filter_' field was replaced by 'int filter_match_rank_', and the 'bool FilterMatchIndex(...)' function was replaced by an 'int FilterMatchIndex(...)'. Places that used the actual result as a Boolean got a '... != -1' at the end. TestCase gets a 'int first_rank()' function that returns the lowest rank of any TestInfo in it (-1 doesn't count as a rank; first_rank() only returns -1 if all TestInfos have "rank" -1); right now, it just iterates through instead of storing the information, but it's possible that this it should be changed to store the informaton. Finally, parallel to UnitTestImpl's and TestCase's 'ShuffleTests' functions I introduced a new 'OrderTests' function which sorts the test (case) indices based on their rank. Either ShuffleTests or OrderTests is always called; ShuffleTests has priority.

Evan

[Zhanyong Wan (λx.x x)]

On Wed, Feb 6, 2013 at 9:57 AM, Evan Driscoll <evaned...@gmail.com> wrote:

On Tuesday, February 5, 2013 3:26:30 PM UTC-6, Zhanyong Wan wrote:

> Sorry, gtest only has an svn repo as that was the only option on Google Code when gtest was first released.

Any chance of a switch at some point in the future?

Good question.  This is technically possible but not trivial.  Google Code allows switching from svn to git, but doesn't seem to allow both to co-exist (with one being a mirror).  We have clients that depend on gtest via svn:externals, so we cannot just simply pull the plug.  My guess is that this won't happen unless someone really motivated and experienced with git drives it.

 

 

If you are serious about this, please read the gtest dev guide (https://code.google.com/p/googletest/wiki/DevGuide).  In particular, the part about signing the CLA.  Thanks!

 

OK, FWIW I have a patch that seems to work in a way that was good enough for me to figure out my problem. I'm not formally submitting it yet for the reasons discussed below, but if you want to pick it up and run with it feel free (I've submitted a CLA under a different email address). Otherwise I'll try to come back to it in... a while (probably not for another month or so, unfortunately) and tighten things up. Feel free to comment on the overall approach, request identifier changes, say you don't like the index/-1 scheme, etc.

Great!  Thanks, Evan.

 

Featurewise, my explanation above was a bit too simple, because I didn't appreciate the interaction of the hierarchy of test cases with the ordering. If the user says --gtest_filter=A.2:B.1:A.1, I can't run them in that order because it's splitting up the A test case. What I do is use the following rule: each individual test is assigned a rank (as you suggest) which is the first pattern in the filter that it matches. Then I assign a rank to each test case which is the lowest (highest-priority) rank of any of its individual tests. So for that --gtest_filter line, it would run A.2, then A.1, then B.1. (At least that's the idea. :-))

Good point.  I didn't think about that either.  Your new rule makes sense.

 

Incomplete parts/problems. First, right now I don't do anything special with death tests, so I'm not sure if they'll still run first.

Thanks for noting this.  (For those of you who didn't know, gtest runs death tests before other tests for safety - https://code.google.com/p/googletest/wiki/AdvancedGuide#Death_Tests_And_Threads)

My thinking is that when gtest_filter is specified, we should respect the ordering specified by the flag (e.g. the user might want to debug what happens when a non-death test runs before a death test).  So my suggestion is to not give death tests any favor in ranking; however, for death tests and non-death tests that have the same ranking, we'll run the death tests first.  Makes sense?

 

Second, in the test suite that I was having problems with, it seems to *sometimes* ignore ordering of tests in a test case and sometimes not. If I say --gtest_filter=A.3:A.2:A.1 it runs A.1 then A.3 then A.2 (so A.1 is at the start instead of end), but if I say --gtest_filter=A.2:A.3:A.1 it runs A.1 then A.2 then A.3 (so switching A.2 and A.3 in the filter changed their order, but A.1 is still misplaced). That seems very strange to me but I don't really have time to debug it at the moment. Finally, there are no tests of the new functionality.

Implementation wise, at a high level I replaced Boolean match/no match queries with ones that return an index into the filter list or -1 on a non-match. (So with --gtest_filter=A.2:B.1:A.1, the TestInfo for B.1 would have rank 1.) I started off calling this "index" but decided I like "rank" better so switched midway through. I'll change them later if someone else doesn't get to them first. So for instance, TestInfo's 'bool matches_filter_' field was replaced by 'int filter_match_rank_', and the 'bool FilterMatchIndex(...)' function was replaced by an 'int FilterMatchIndex(...)'. Places that used the actual result as a Boolean got a '... != -1' at the end. TestCase gets a 'int first_rank()' function that returns the lowest rank of any TestInfo in it (-1 doesn't count as a rank; first_rank() only returns -1 if all TestInfos have "rank" -1); right now, it just iterates through instead of storing the information, but it's possible that this it should be changed to store the informaton. Finally, parallel to UnitTestImpl's and TestCase's 'ShuffleTests' functions I introduced a new 'OrderTests' function which sorts the test (case) indices based on their rank. Either ShuffleTests or OrderTests is always called; ShuffleTests has priority.

What you described makes sense, at least at the high level.  Thanks!

 

Evan

--
 
---
You received this message because you are subscribed to the Google Groups "Google C++ Testing Framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to googletestframe...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Zhanyong

[Wojtek Mamrak]

Hello,

are there any updates to this topic? Did anyone work on this feature? Is there any possibiliy of adding native support for this kind of functionality?

The patch by Evan looks to fulfill my needs, yet I would like to see more mature solution :)

regards