Test Event Listener and list-tests flag

[Paolo Polce]

Hello,

I would like to output a list of all the tests in my own format.
I thought I'd just write a listener to do that, the run all tests with
--gtest_list_tests (to avoid running the tests), but the listener
doesn't get called.

The alternative could be to not use gtest_list_tests and then be able
to cancel a test on the OnTestStarted event.
But I didn't find a test_info.Cancel().

Any hint?

thanks a lot
P

[Vlad Losev]

There is no TestInfo::Cancel(). But you can listen to OnTestProgramStart() and then enumerate all the tests from there, similarly to the way UnitTestImpl::ListTestsMatchingFilter does. Unfortunately, TestInfo doesn't have a public method to find out whether it matches a filter. Your best choice is public TestInfo::should_run which determines whether Google Test will run the test under current circumstances. Once you print your test names you can just call exit().

HTH,

Vlad

 

thanks a lot
P

[Paolo Polce]

Thanks Vlad,

on the same topic...
I will probably have to write a small GUI, with a progress bar to
display the results and maybe a tree to select one or more tests to
run.
Actually I don't know if there is one already out there (I've heard
there is something for eclipse, but I'm on VS2005 these days).
Anyway, I could use a listener as you suggested to build my list, then
show the list in a tree-view to let the user select what to run but
then, since there is no way to skip a test in the listener, I would
probably have to pass the list I created to a colon separated filter
and finally call run_tests_again?

something like:

1) init gtest
2) add listener ( myTestListBuilder )
3) run_all_tests() (the listener will exit() on program start as we
said)
4) myTreeView.Show( ... ); // the user selects what to run...
5) ::testing::FLAGS_gtest_filter = userSelectedList.ToColonSeparated
();
6) add listener ( myProgressBarAndStuff..)
7) run_all_tests()

That would probably work, but it doesn't look very pretty :)

I guess I'm looking for a way to add/remove elements from
*UnitTest::GetInstance().

Cheers
P

> > P- Nascondi testo citato
>
> - Mostra testo citato -- Nascondi testo citato
>
> - Mostra testo citato -

[Vlad Losev]

Hi Paolo,

On Mon, Nov 9, 2009 at 12:56 AM, Paolo Polce <pa...@webshell.it> wrote:

Thanks Vlad,

on the same topic...
I will probably have to write a small GUI, with a progress bar to
display the results and maybe a tree to select one or more tests to
run.
Actually I don't know if there is one already out there (I've heard
there is something for eclipse, but I'm on VS2005 these days).
Anyway, I could use a listener as you suggested to build my list, then
show the list in a tree-view to let the user select what to run but
then, since there is no way to skip a test in the listener, I would
probably have to pass the list I created to a colon separated filter
and finally call run_tests_again?

something like:

1) init gtest
2) add listener ( myTestListBuilder )
3) run_all_tests()  (the listener will exit() on program start as we
said)
4) myTreeView.Show( ... ); // the user selects what to run...
5) ::testing::FLAGS_gtest_filter = userSelectedList.ToColonSeparated
();
6) add listener ( myProgressBarAndStuff..)
7) run_all_tests()

That would probably work, but it doesn't look very pretty :)

Yes, this is the way to work with it now.
 

I guess I'm looking for a way to add/remove elements from
*UnitTest::GetInstance().

Currently, we do not support it. During many events, Google Test is iterating over the collections of test cases and tests in them. Clients changing those collections during the iteration would lead to problems. However, it would be possible to implement an OnTestListAvailable event. Care to try your hand at this? :-)
 

Regards,
Vlad

[Paolo Polce]

Hi Vlad,

> it would be possible to implement an OnTestListAvailable event.

Would it be called before OnTestProgramStart, and expose a modifiable
list?
I still think that a UnitTest.Ignore(this) inside the OnTestStart(...)
would be better, because I would show the whole list in the tree-view
but then just ignore what the user doesn't select.

> Care to try your hand at this? :-)

At least it'll keep me quiet for a while :)

P

> Vlad- Nascondi testo citato

[Vlad Losev]

On Wed, Nov 11, 2009 at 6:32 AM, Paolo Polce <pa...@webshell.it> wrote:

Hi Vlad,

> it would be possible to implement an OnTestListAvailable event.

Would it be called before OnTestProgramStart, and expose a modifiable
list?
I still think that a UnitTest.Ignore(this) inside the OnTestStart(...)
would be better, because I would show the whole list in the tree-view
but then just ignore what the user doesn't select.

My idea for OnTestListAvailable was to allow custom output of the test set. But you are right, an opportunity to select tests interactively is a natural feature for a GUI. I think it is possible to provide a place where the user can adjust the set of tests to run, one way or another. OnTestListAvailable can be such a place. But the exact way to select/unselect tests should be discussed further.

> Care to try your hand at this? :-)

At least it'll keep me quiet for a while :)

Upon further consideration I would like this idea to discussed further before any work is done. One of the issues is that if we move the test list display to the standard output listener to allow customization then shutting down the standard output will also block printing the list of tests. This changes the existing behavior, which is to print the test list whether the standard output listener is shut down or not. Does this worry anyone?
 

Regards,
Vlad

[Paolo Polce]

On Nov 11, 7:01 pm, Vlad Losev <vl...@google.com> wrote:
> On Wed, Nov 11, 2009 at 6:32 AM, Paolo Polce <pa...@webshell.it> wrote:
>
> > Hi Vlad,
>
> > > it would be possible to implement an OnTestListAvailable event.
>
> > Would it be called before OnTestProgramStart, and expose a modifiable
> > list?
> > I still think that a UnitTest.Ignore(this) inside the OnTestStart(...)
> > would be better, because I would show the whole list in the tree-view
> > but then just ignore what the user doesn't select.
>
> > My idea for OnTestListAvailable was to allow custom output of the test set.
>
> But you are right, an opportunity to select tests interactively is a natural
> feature for a GUI. I think it is possible to provide a place where the user
> can adjust the set of tests to run, one way or another. OnTestListAvailable
> can be such a place. But the exact way to select/unselect tests should be
> discussed further.

OnTestListAvailable is a good idea to let the user customize the
output of the set.
For building a gui, tho', I'm not sure it would be a great help.
It would only spare you from having to call run_all_tests() twice (the
first time with a listener that builds up the list and exit(), the
second time to actually run the tests once the list has been filtered
by the user).

>
> > > Care to try your hand at this? :-)
>
> > At least it'll keep me quiet for a while :)
>
> Upon further consideration I would like this idea to discussed further
> before any work is done.

Sure.

Thanks
P

[Vlad Losev]

Hi,

On Fri, Nov 13, 2009 at 11:32 AM, Paolo Polce <pa...@webshell.it> wrote:

On Nov 11, 7:01 pm, Vlad Losev <vl...@google.com> wrote:

> On Wed, Nov 11, 2009 at 6:32 AM, Paolo Polce <pa...@webshell.it> wrote:
>
> > Hi Vlad,
>
> > > it would be possible to implement an OnTestListAvailable event.
>
> > Would it be called before OnTestProgramStart, and expose a modifiable
> > list?
> > I still think that a UnitTest.Ignore(this) inside the OnTestStart(...)
> > would be better, because I would show the whole list in the tree-view
> > but then just ignore what the user doesn't select.
>
> > My idea for OnTestListAvailable was to allow custom output of the test set.
>
> But you are right, an opportunity to select tests interactively is a natural
> feature for a GUI. I think it is possible to provide a place where the user
> can adjust the set of tests to run, one way or another. OnTestListAvailable
> can be such a place. But the exact way to select/unselect tests should be
> discussed further.

OnTestListAvailable is a good idea to let the user customize the
output of the set.
For building a gui, tho', I'm not sure it would be a great help.
It would only spare you from having to call run_all_tests() twice (the
first time with a listener that builds up the list and exit(), the
second time to actually run the tests once the list has been filtered
by the user).

Here is a slightly more detailed proposal:

• Google Test introduces the new event. The proposed signature is void OnTestListAvailable(TestList test_cases, bool* continue);
• The event is fired after all the tests are registered and marked as should run/should not run.
• The event listeners have chance to adjust the set of tests to run. There are two alternatives for doing this:
1. Listeners modify GTEST_FLAG(filter). In this case the the list of test cases can be obtained via the UnitTest class.
2. Listeners set the should_run flag on individual TestInfo objects. This will require us to somehow pass non-const TestInfos into the event. For example, we can introduce TestList that will give write access to TestInfos contained in it.
• After the listeners have finished, Google Test does all processing necessary to adjust the set of tests to run.
• Listeners will have an option of setting the parameter continue to false to prevent Google Test from actually running the tests.

I would like everyone interested comment on the proposal. Maybe someone has concerns about bad interactions with disabled tests, or with sharding. Maybe someone has a better solution?

Thanks,

Vlad

[Paolo Polce]

Hi Vlad,

>    - Google Test introduces the new event. The proposed signature is

>    void OnTestListAvailable(TestList test_cases, bool* continue);

>    - The event is fired after all the tests are registered and marked as
>    should run/should not run.
>    - The event listeners have chance to adjust the set of tests to run.

>    There are two alternatives for doing this:

>       1. Listeners modify GTEST_FLAG(filter). In this case the the list of

>       test cases can be obtained via the UnitTest class.

>       2. Listeners set the should_run flag on individual TestInfo objects.

>       This will require us to somehow pass non-const TestInfos into
> the event. For
>       example, we can introduce TestList that will give write access
> to TestInfos
>       contained in it.

>    - After the listeners have finished, Google Test does all

>    processing necessary to adjust the set of tests to run.

>    - Listeners will have an option of setting the parameter continue to

>    false to prevent Google Test from actually running the tests.

The *continue in this case would not be much different from exit(),
except - I guess - it would let other listeners get called (which is
not a bad idea, anyway).
But the point is how to modify the test set.
Maybe an event fired before the test is added to the list would help:
OnAddToTestList(TestInfo ..., bool *shouldAdd)

In this scenario I could:

1) build a list of all the tests by listening to AddToTestList(... ,
true)
2) show the list to the user, let him select what to run
3) run the tests listening to AddToTestList again, but this time
*shouldAdd = myWhatShouldRunList.Contains(testinfo)

>
> I would like everyone interested comment on the proposal. Maybe someone has
> concerns about bad interactions with disabled tests, or with sharding. Maybe
> someone has a better solution?

thanks,
P

[Vlad Losev]

Doesn't enumerating over all test case/tests work for you?

Thanks,

Vlad