Experimenting with the Verbose Text Output

[Erik Bernoth]

Hi,

did you ever try to manipulate the nosetests/unittest output on a more miniscule level? In our company we have some test cases that take quite a long time which makes it quite important which information is printed at which time. Therefore I have to be able to change the default nose output quite a bit to experiment which output might be the best for our test developers. Either I haven't found the right place to look at, or nobody else has done that yet, because it seems to be quite hard to do.

First I had a look at the example plugin for html generation (examples/html_plugin/htmlplug.py). The process it uses is it catches all instances of test results, stores them in memory and in the end it prints all of them at the end. This is already a nice approach, but not feasible for our situation. As I said the suite might run a few hours. Getting "instant" feedback from one test case to the next test case is quite important.

Second I tried to find other plugins that might print instant feedback, but I haven't found any.

At last I decided to try as a first experiment to manipulate the normal nose output as shown in this commit: https://github.com/erikb85/nose/commit/c73e3a3d1542744d519f56bfc686a335c36b0ef4

This is already a step in the right direction. Every time a verbose output is generated it doesn't just say ERROR or FAIL but it also prints some hint (the first line of the error message) about what might have went wrong.

Now I can't find a way to put that into the plugin scheme. Maybe if I go the html plugin approach and write in a log file in my own format instead of just storing the data in stream. That might already be useful, but it still feels like a workaround. I would really love if someone has some more insights and can help me get this going. Later on I also want to do more things like reverting the order of the detailed error messages (because usually you have to scroll up, not down, when writing tests), etc.

Best
Erik

PS: Reading the source code feels really logical and well structured. At first I felt quite lost, but it's normal in a complex project. After I understood what runners, test cases, plugins etc were it was a breeze to browse the code base. Kudos at the maintainers for the code quality!

[John Szakmeister]

On Mon, Jan 19, 2015 at 8:57 AM, Erik Bernoth <erik.b...@gmail.com> wrote:
> Hi,
>
> did you ever try to manipulate the nosetests/unittest output on a more
> miniscule level? In our company we have some test cases that take quite a
> long time which makes it quite important which information is printed at
> which time. Therefore I have to be able to change the default nose output
> quite a bit to experiment which output might be the best for our test
> developers. Either I haven't found the right place to look at, or nobody
> else has done that yet, because it seems to be quite hard to do.

I guess that depends on the output you're looking for. Nose already
prints the test name or the docstring when run with the `-v` option (I
personally find the former more useful than the latter). Usually
that's enough for most folks as it is emitted at the start of test
execution, with success or fail being emitted after it's completed.

> First I had a look at the example plugin for html generation
> (examples/html_plugin/htmlplug.py). The process it uses is it catches all
> instances of test results, stores them in memory and in the end it prints
> all of them at the end. This is already a nice approach, but not feasible
> for our situation. As I said the suite might run a few hours. Getting
> "instant" feedback from one test case to the next test case is quite
> important.

Most plugins that produce a report wait until the report() step to
write the report to disk. It reduces disk I/O, and they don't have to
worry about making sure to write a well-formed document each time.

You could switch that around a bit and re-write the report file with
each addError(), addSuccess(), and addFailure() (without the final run
statistics).

> Second I tried to find other plugins that might print instant feedback, but
> I haven't found any.
>
> At last I decided to try as a first experiment to manipulate the normal nose
> output as shown in this commit:
> https://github.com/erikb85/nose/commit/c73e3a3d1542744d519f56bfc686a335c36b0ef4
>
> This is already a step in the right direction. Every time a verbose output
> is generated it doesn't just say ERROR or FAIL but it also prints some hint
> (the first line of the error message) about what might have went wrong.

I think you should be able to do something similar in a plugin too.
You may need to adjust where you are in the plugin stack (with the
score class variable) to make it work for you though. Are you just
trying to get that information on the screen, or is it something else
you are trying to do?

> Now I can't find a way to put that into the plugin scheme. Maybe if I go the
> html plugin approach and write in a log file in my own format instead of
> just storing the data in stream. That might already be useful, but it still
> feels like a workaround. I would really love if someone has some more
> insights and can help me get this going. Later on I also want to do more
> things like reverting the order of the detailed error messages (because
> usually you have to scroll up, not down, when writing tests), etc.

I can't promise that I can lead you to the answer (my time is awfully
limited right now with work stuff), but I can at least try to provide
some insights.

One thing I've found useful in the past is to provide my own test
runner. You have a little more control over the output, and that's
sometimes useful. The downside of this approach is that you can no
longer use the `nosetests` command to drive the tests (you have to use
your own command that sets up your runner and calls Nose's
TestProgram() with the new test runner.

> Best
> Erik
>
> PS: Reading the source code feels really logical and well structured. At
> first I felt quite lost, but it's normal in a complex project. After I
> understood what runners, test cases, plugins etc were it was a breeze to
> browse the code base. Kudos at the maintainers for the code quality!

Thanks for the kind words (though most of it Jason's doing). It could
definitely use some improvements though. Trying to make things work
for all versions of Python has been very challenging.

-John