Event listener interface design proposal

[Vlad Losev]

Greetings,

Currently, Google Test can output its results in two ways: to the process's console and to an XML file. We've had requests to provide custom formats both for console output and XML output. Now we have a design proposal that lets you intercept Google Test events and generate any output you want. It is simple enough to be easily implemented and flexible enough to allow any type of output.

The design document is available at http://docs.google.com/Doc?id=dhspb9bn_6gqdknqhp and is also pasted at the end of this email.

We are asking for your comments and suggestions. If you ever wanted to have your test output in a custom format, or to implement a GUI-based test runner, or to integrate your tests with third party test tracking system, this is your chance to have contribute your ideas and opinions. Especially if you think this design is lacking for your needs. ;-)

Best regards,
Vlad Losev on behalf of Google Test dev team.

Event listener interface design

Internally, Google Test uses an event listener interface as a conduit for reporting a number of important internal events:

• Start and end of a test case
• Start an end of a test
• Global environment set-up and tear-down
• A failure occurring

These events are used for the following purposes:

• Implementing the regular console output
• Implementing the XML file output

Opening the event listener interface to Google Test users will allow easy implementation of alternative output formats including GUIs, alternative file output formats, reporting to database, etc.

Design

Google Test will publish the interface itself:

class TestEventListener {
 public:
  virtual void OnUnitTestStart(const UnitTest& unit_test) = 0;
  virtual void OnGlobalSetUpStart(const UnitTest& unit_test) = 0;
  virtual void OnGlobalSetUpEnd(const UnitTest& unit_test) = 0;
  virtual void OnTestCaseStart(const TestCase& test_case) = 0;
  virtual void OnTestStart(const TestInfo& test_info) = 0;
  virtual void OnNewTestPartResult(const TestPartResult& test_part_result) = 0;
  virtual void OnTestEnd(const TestInfo& test_info) = 0;
  virtual void OnTestCaseEnd(const TestCase& test_case) = 0;
  virtual void OnGlobalTearDownStart(const UnitTest& unit_test) = 0;
  virtual void OnGlobalTearDownEnd(const UnitTest& unit_test) = 0;
  virtual void OnUnitTestEnd(const UnitTest& unit_test) = 0;
};

The UnitTest class will expose an instance of a class EventListeners via the event_listeners() method. EventListeners will maintain a list of event listeners. When an event happens, listeners are notified in the order they appear in the list. UnitTest automatically adds a listener responsible for the console output and a listener responsible for the XML output (when necessary) to the list. In order for users to manipulate the list, EventListeners  publishes the following methods:

void Append(TestEventListener* listener)

Appends an event listener to the end of the list. Google Test assumes the ownership of the listener pointer (i.e. it will delete the listener when the test program finishes).

TestEventListener* Release(TestEventListener* listener)

Removes the given event listener from the list and returns it. It then becomes the user's responsibility to delete the listener.If the code that Release's a obtained via GetDefaultResultPrinter() or GetDefaultXmlGenerator() resides in a shared object (DLL) different from the one where Google Test resides, it may be impossible to delete this listener without corrupting memory. In such case that pointer has to be leaked.  

Google Test will be adding its internal listeners to this list alongside the user-added ones. To facilitate the modification of the standard output behavior, the UnitTest class will  also expose two of its internal listeners via these methods:

TestEventListener* default_result_printer() 

Returns the internal listener responsible for the output to the console. This listener can be removed from the list using Release() to shut down console output or another listener can be inserted after it to provide complementary output.

TestEventListener* default_xml_generator()

Returns the internal listener responsible for the output of XML reports. This listener can be removed from the list using Release() to shut down XML output in order to replace it with custom one.

EmptyTestEventListener

We may change TestEventLIstener in the future, adding more methods, or overloading existing methods with more parameters. Some people prefer such changes not to break their builds. For them, Google Test will also publish an interface stub (class EmptyTestEventListener) that will provide empty implementations for all the methods in TestEventListener and will be kept in sync with it. Inheriting from EmptyTestEventListener instead of TestEventListener will make the client code insensitive to the changes TestEventListener. Users will have an option of inheriting either from TestEventListener or from EmptyTestEventListener.

Appendix A. Classes providing information about running tests.

These are the classes of parameters to the TestEventListener's methods. They provide information that listeners may require.

UnitTest

UnitTest describes the status of the executing program as a whole. It will contain the following methods:

int successful_test_case_count() const;

Returns the number of test cases that don't have failed tests so far.

int failed_test_case_count() const;

Returns the number of test cases that have tests that have already failed.

int total_test_case_count() const;

Returns the number of all test cases.

int test_case_to_run_count() const;

Returns the number of all test cases that contain at least one test that should run.

int successful_test_count() const;

Returns the number of tests that have not yet failed (across all test cases).

int failed_test_count() const;

Returns the number of tests that have already failed (across all test cases).

int disabled_test_count() const;

Returns the number of disabled tests (across all test cases).

int total_test_count() const;

Returns the number of all tests (across all test cases).

int test_to_run_count() const;

Returns the number of tests that should run (across all test cases).

TimeInMillis elapsed_time() const;

Reurns the time, in milliseconds, elapsed since the start of RUN_ALL_TESTS().

bool Passed() const;

Returns true iff the unit test passed (i.e. all test cases passed). Valid only in OnUnitTestEnd.

bool Failed() const;

Returns true iff the unit test failed (i.e. some test case failed or something outside of all tests failed). Valid only in OnUnitTestEnd.

const TestCaseInterface* GetTestCase(int i) const;

Gets the i-th test case among all the test cases. i can range from 0 to total_test_case_count() - 1. If i is not in that range, returns NULL.

TestCase

TestCase describes execution of a single test case. It will contain the following methods:

const char* name() const;

Returns the name of the test case.

bool should_run() const;

Returns true if any test in this test case should run.

int successful_test_count() const;

Returns the number of tests in this test case that has not yet failed.

int failed_test_count() const;

Returns the number of tests in this test case that has failed by the moment of the call.

int disabled_test_count() const;

Returns the number of disabled tests in this test case.

int test_to_run_count() const;

Returns the number of tests in this test case that should run. The test will not run if either it or its test case are disabled or if it has been blocked from running in this invocation by the filter setting or by sharding selection.

int total_test_count() const;

Returns the number of all tests in this test case.

bool Passed() const;

Returns true iff the test case has passed. Valid only in and after this test case's OnTestCaseEnd.

bool Failed() const;

Returns true iff the test case failed. Valid only in and after this test case's OnTestCaseEnd.

TimeInMillis elapsed_time() const;

Returns the time, in milliseconds, that passed since the test case started.

const TestInfoInterface* GetTestInfo(int i) const;

Returns the i-th test among all the tests. i can range from 0 to total_test_count() - 1. If i is not in that range, returns NULL.

TestInfo

TestInfo describes a single test. It will have the following methods:

const char* test_case_name() const;

Returns the test case name.

const char* name() const;

Returns the test name.

bool should_run() const;

Returns true if this test should run.

const TestResultInterface* result() const;

Returns the pointer to object reporting results of this test.

TestResult

TestResult describes results of a single test. It will have the following methods:

int successful_part_count() const;

Returns the number test parts (EXPECT/ASSERT test assertions) that has succeeded so far.

int failed_part_count() const;

Returns the number of test parts that has failed so far.

int total_part_count() const;

Returns the number of all test parts that has been executed so far.  This is the sum of the number of successful test parts and the number of failed test parts.

int property_count() const;

Returns the number of test properties.

bool Passed() const;

Returns true iff the test has passed (i.e. no test part failed). Valid only in and after the test's OnTestEnd.

bool Failed() const;

Returns true iff the test has failed.

bool HasFatalFailure() const;

Returns true iff the test has fatally failed.

bool HasNonfatalFailure() const;

Returns true iff the test has a non-fatal failure.

TimeInMillis elapsed_time() const;

Returns the time, in milliseconds, that passed since the test started.

const TestPartResult* GetTestPartResult(int i) const;

Returns the i-th test part result. i can range from 0 to test_part_count() - 1. If i is not in that range, returns NULL.

const TestProperty* GetTestProperty(int i) const;

Returns the i-th test property. i can range from 0 to test_property_count() - 1. If i is not in that range, returns NULL.

TestPartResult

TestPartResult describes results of a single Google Test assertion (EXPECT_XXX/ASSERT_XXX). It will have the following methods:

TestPartResultType type() const;

Gets the outcome of the test part.

const char* file_name() const;

Gets the name of the source file where the test part took place, or NULL if it's unknown.

int line_number() const;

Gets the line in the source file where the test part took place, or -1 if it's unknown.

const char* summary() const;

Gets the summary of the failure message.

const char* message() const;

Gets the message associated with the test part.

bool passed() const;

Returns true iff the test part passed.

bool failed() const;

Returns true iff the test part failed.

bool nonfatally_failed() const;

Returns true iff the test part non-fatally failed. Will never be true for ASSERT_-type test parts.

bool fatally_failed() const;

Returns true iff the test part fatally failed. Will never be true for EXPECT_-type test parts.

TestProperty

TestProperty describes custom properties of the test. It will have the following methods:

const char* key() const;

Returns the property name.

const char* value() const;

Returns the property value.

TestPartResulType is define as an enum with values kTestPartSuccess, kTestPartFailure, and kTestPartFatalFailure.

[Zhanyong Wan (λx.x x)]

Thanks Vlad.

Users of Google Test, you'll probably want to skip the appendix for
now. The main interesting stuff is in the Design section. Please
focus your feedback on that part of the design doc at this time.
Thanks!

--
Zhanyong

[Paweł Hajdan Jr.]

2009/6/2 Zhanyong Wan (λx.x x) <w...@google.com>:

> Users of Google Test, you'll probably want to skip the appendix for
> now. The main interesting stuff is in the Design section. Please
> focus your feedback on that part of the design doc at this time.

I think it makes it possible to do what I wanted in Chromium and looks
very nice. It will be possible to add a test failure from an event
listener, right?

Paweł

[Vlad Losev]

2009/6/2 Paweł Hajdan Jr. <phajd...@chromium.org>

Ugh, it could lead to infinite recursion under certain circumstances; I would argue against that. :-)

Pray tell, why do you need to do that? The listeners are not code under test, after all.

Paweł

Regards,
Vlad

[Zhanyong Wan (λx.x x)]

2009/6/2 Vlad Losev <vl...@google.com>:

>
>
> 2009/6/2 Paweł Hajdan Jr. <phajd...@chromium.org>
>>
>>
>> 2009/6/2 Zhanyong Wan (λx.x x) <w...@google.com>:
>> > Users of Google Test, you'll probably want to skip the appendix for
>> > now.  The main interesting stuff is in the Design section.  Please
>> > focus your feedback on that part of the design doc at this time.
>>
>> I think it makes it possible to do what I wanted in Chromium and looks
>> very nice. It will be possible to add a test failure from an event
>> listener, right?
>
> Ugh, it could lead to infinite recursion under certain circumstances; I
> would argue against that. :-)

We don't need to support reporting failures in OnNewTestPartResult(),
but reporting failures in other events should be fine.

> Pray tell, why do you need to do that? The listeners are not code under
> test, after all.

If people use the listener to check resource leaks, they need to be
able to report failures.

>> Paweł
>
> Regards,
> Vlad
>

--
Zhanyong

[Vlad Losev]

Any other comments?

I am planning to commit an implementation within one week if no one has issues with the design.

Cheers,
Vlad

[Paweł Hajdan Jr.]

I have no other comments. Thanks for great work.

Paweł

[didrik]

hi vlad

this seems to be exactly what I need to make googletest output TAP
(http://testanything.org/wiki/index.php/Main_Page)
compatible.

On Jun 6, 2:13 am, Vlad Losev <vl...@google.com> wrote:
> Any other comments?
>
> I am planning to commit an implementation within one week if no one has
> issues with the design.

Any progress on this?
I'm very eager to get this functionality.

>
> Cheers,
> Vlad

regards,

-- didrik

>
> 2009/6/2 Zhanyong Wan (λx.x x) <w...@google.com>
>
> > 2009/6/2 Vlad Losev <vl...@google.com>:
>

> > > 2009/6/2 Paweł Hajdan Jr. <phajdan...@chromium.org>

[Vlad Losev]

Hi didrik,

On Mon, Jul 6, 2009 at 12:49 AM, didrik <torm...@gmail.com> wrote:

hi vlad

this seems to be exactly what I need to make googletest output TAP
(http://testanything.org/wiki/index.php/Main_Page)
compatible.

On Jun 6, 2:13 am, Vlad Losev <vl...@google.com> wrote:
> Any other comments?
>
> I am planning to commit an implementation within one week if no one has
> issues with the design.

Any progress on this?
I'm very eager to get this functionality.

Yes, we have performed a sizable re-factoring needed to publish the listener interface. But the further progress was slowed down as on of us had to go on leave. I expect the change to hit svn by the end of the next week.

Sorry for the delay. :-(

Regards,
Vlad

[Tor Didriksen]

On Mon, Jul 6, 2009 at 10:01 PM, Vlad Losev <vl...@google.com> wrote:

Hi didrik,

On Mon, Jul 6, 2009 at 12:49 AM, didrik <torm...@gmail.com> wrote:

hi vlad

this seems to be exactly what I need to make googletest output TAP
(http://testanything.org/wiki/index.php/Main_Page)
compatible.

On Jun 6, 2:13 am, Vlad Losev <vl...@google.com> wrote:
> Any other comments?
>
> I am planning to commit an implementation within one week if no one has
> issues with the design.

Any progress on this?
I'm very eager to get this functionality.

Yes, we have performed a sizable re-factoring needed to publish the listener interface. But the further progress was slowed down as on of us had to go on leave. I expect the change to hit svn by the end of the next week.

Sorry for the delay. :-(

Hi vlad

I'm still relying on a hack to get the output I want (see attachment)
Will you make UnitTestEventListenerInterface public?

-- didrik
 

[Zhanyong Wan (λx.x x)]

Hi Tor,

On Thu, Aug 20, 2009 at 7:21 AM, Tor Didriksen<torm...@gmail.com> wrote:
>
>
> On Mon, Jul 6, 2009 at 10:01 PM, Vlad Losev <vl...@google.com> wrote:
>>
>> Hi didrik,
>>
>> On Mon, Jul 6, 2009 at 12:49 AM, didrik <torm...@gmail.com> wrote:
>>>
>>>
>>> hi vlad
>>>
>>> this seems to be exactly what I need to make googletest output TAP
>>> (http://testanything.org/wiki/index.php/Main_Page)
>>> compatible.
>>>
>>> On Jun 6, 2:13 am, Vlad Losev <vl...@google.com> wrote:
>>> > Any other comments?
>>> >
>>> > I am planning to commit an implementation within one week if no one has
>>> > issues with the design.
>>>
>>> Any progress on this?
>>> I'm very eager to get this functionality.
>>
>> Yes, we have performed a sizable re-factoring needed to publish the
>> listener interface. But the further progress was slowed down as on of us had
>> to go on leave. I expect the change to hit svn by the end of the next week.
>>
>> Sorry for the delay. :-(
>
> Hi vlad
>
> I'm still relying on a hack to get the output I want (see attachment)
> Will you make UnitTestEventListenerInterface public?

We will publish it. However, Vlad and I have been on vacation lately
(Vlad still is) and thus the work stalled. We'll resume working on it
once Vlad returns next week. Thanks for your patience!

--
Zhanyong