Visualization of error

3 views
Skip to first unread message

Olof Bjarnason

unread,
Nov 15, 2011, 3:32:51 PM11/15/11
to pytd...@googlegroups.com
A couple of weeks ago, any syntax error in pytddmon resulted in a gray
background with 0/34 displayed in the center. Really un-informative
and frankly quite magic (where did that 34 come from?)

So I really like that the situation has improved quite a lot since
then. Nevertheless, I do think the current situation could be
improved.

I attached an image of what happens when there is a syntax error in
one of the unit tests of pytddmon itself. The first "28" means there
are 28 unit tests passing, the second "28" I'm not sure what it means,
and the last "2j" is something very creative to my mind. Two unit
tests are "imaginary" (j is used instead of i to describe the
imaginary part of a complex number; for some reason it is in pytddmon
now).

Obvious improvements:

1. Don't use complex numbers, at least not in the main window
2. Since syntax errors rules out discovering the number of errors in
total, I think a question mark would be good somewhere
3. Discuss what color should be used for syntax errors; it should be
more "severe" than unit test errors, but red feels more severe than
orange.

pytddmon_syntax_error_display.png

Samuel Ytterbrink

unread,
Nov 15, 2011, 3:40:40 PM11/15/11
to pytd...@googlegroups.com
The 2j here means that 2 tests (at least) failed hard. It is the
number of test files that did throw a error that the test framework
did not catch.

Colors?
So you sugest
green = ok,
orange = tests failed.
red = you got un caught exceptions.

hmm should there be a new way of writning the output?

28/28/2 ?
or only:
*2*

or something other?

2011/11/15 Olof Bjarnason <olof.bj...@gmail.com>:

--
//Samuel Ytterbrink

Olof Bjarnason

unread,
Nov 15, 2011, 3:45:22 PM11/15/11
to pytd...@googlegroups.com
2011/11/15 Samuel Ytterbrink <nep...@gmail.com>:

> The 2j here means that 2 tests (at least) failed hard. It is the
> number of test files that did throw a error that the test framework
> did not catch.
>
> Colors?
> So you sugest
> green = ok,
> orange = tests failed.
> red = you got un caught exceptions.

Well, that doesn't suit the basic colors of TDD: Red, Green, Refactor.

So I'm not sure what we should do.

>
> hmm should there be a new way of writning the output?
>
> 28/28/2 ?
> or only:
> *2*

Yeah, maybe simplify and not complicate? If there is a syntax error,
it is something severe. Why not stick with the red color, and change
the text from 34/34 to something like *2*. After all, it is not
possible to calculate the real total number of tests anymore, because
of the syntax error.

Some attempts at symbols to show this:

2?
2!
*2*
2"#%¤% (like in the comics, some one swearing haha)

Samuel Ytterbrink

unread,
Nov 15, 2011, 3:47:35 PM11/15/11
to pytd...@googlegroups.com
I like the last one, but we need some mor information....

So that a new user feels at home with the error, that is the problem
in the first place:

<push me!> is another great think to show.

2011/11/15 Olof Bjarnason <olof.bj...@gmail.com>:

--
//Samuel Ytterbrink

Olof Bjarnason

unread,
Nov 15, 2011, 3:53:23 PM11/15/11
to pytd...@googlegroups.com
2011/11/15 Samuel Ytterbrink <nep...@gmail.com>:

> I like the last one, but we need some mor information....
>
> So that a new user feels at home with the error, that is the problem
> in the first place:
>
> <push me!> is another great think to show.

Haha, yeah that's one idea. What about this:

[Push me. It's complicated]

:)

Maybe we should just type what we know: it is a compile-time error in
the code of the user:

<Error>

or even

Error

Olof Bjarnason

unread,
Nov 15, 2011, 3:55:51 PM11/15/11
to pytd...@googlegroups.com
2011/11/15 Olof Bjarnason <olof.bj...@gmail.com>:
> 2011/11/15 Samuel Ytterbrink <nep...@gmail.com>:
>> I like the last one, but we need some mor information....
>>
>> So that a new user feels at home with the error, that is the problem
>> in the first place:
>>
>> <push me!> is another great think to show.
>
> Haha, yeah that's one idea. What about this:
>
> [Push me. It's complicated]
>
> :)
>
> Maybe we should just type what we know: it is a compile-time error in
> the code of the user:
>
> <Error>
>
> or even
>
> Error

Hmm. Since we know the number of modules that have syntax errors, we
could put that into the mix too:

2 error(s)

Samuel Ytterbrink

unread,
Nov 15, 2011, 3:56:12 PM11/15/11
to pytd...@googlegroups.com
well it doesn't have to be a compile error, can be that some there
thing went wrong. I dont know exactly what types of errors the
unittest lib dont catch.

--
//Samuel Ytterbrink

Samuel Ytterbrink

unread,
Nov 15, 2011, 3:58:24 PM11/15/11
to pytd...@googlegroups.com
well i like the:

[Push me. It's complicated]

how ever if you are fixing failing tests, you may still want to know
which tests fail and which dont. so removing the 28/28 may be a bad
ide. Cause you may want to fix the errors last.

--
//Samuel Ytterbrink

Olof Bjarnason

unread,
Nov 15, 2011, 4:14:27 PM11/15/11
to pytd...@googlegroups.com
2011/11/15 Samuel Ytterbrink <nep...@gmail.com>:

> well i like the:
>
> [Push me. It's complicated]
>
> how ever if you are fixing failing tests, you may still want to know
> which tests fail and which dont. so removing the 28/28 may be a bad
> ide. Cause you may want to fix the errors last.

I found out what distinction unittest does between "error" and "failure":

"Note that in order to test something, we use the one of the assert*()
methods provided by the TestCase base class. If the test fails, an
exception will be raised, and unittest will identify the test case as
a failure. Any other exceptions will be treated as errors. This helps
you identify where the problem is: failures are caused by incorrect
results - a 5 where you expected a 6. Errors are caused by incorrect
code - e.g., a TypeError caused by an incorrect function call."

So any exception (including syntax errors and a DivideByZero) in user
code (application or unit test) is counted as an error, while a failed
self.assertEqual+friends assertion is counted as a failure. [Sadly,
that also means using a built-in assert statement would probably also
count as an error and not a failure in unittest :( ]

Samuel Ytterbrink

unread,
Nov 15, 2011, 4:18:47 PM11/15/11
to pytd...@googlegroups.com
dont think the assert statement givs a error.
Im 99% sure i have tested this.

--
//Samuel Ytterbrink

Olof Bjarnason

unread,
Nov 15, 2011, 4:27:45 PM11/15/11
to pytd...@googlegroups.com
2011/11/15 Samuel Ytterbrink <nep...@gmail.com>:

> dont think the assert statement givs a error.
> Im 99% sure i have tested this.

I just tested it, if gets treated as a failure! Horray! :)

Anyway, back on topic. I think we really shouldn't strive to use all
information we have in the main GUI. A syntax error situation (which
is the main point of this discussion, and a critical use case) is
exceptional and should be handled first (before fixing failures).

If the user really wants to fix the failures before the syntax errors,
he/should could rename those error-files temporarily to something not
of pattern test_*.py, to exclude them from pytddmons "evil eye" :)

So in summary, please go for "2 error(s)". Too much information is not
somewhere we want to return too in the minimalistic main GUI.

Samuel Ytterbrink

unread,
Nov 15, 2011, 4:40:42 PM11/15/11
to pytd...@googlegroups.com
Ok, but jst so you understand first.

if you would rename a class that 5 test imports, then you would get 5 errors.

about the assert statement:
i was looking in the code and foudn out that we actualy do have
different ways of finding syntax/loading erros and errors. but we
translates unittest errors as failures.

313 return (
314 result.testsRun - len(result.failures) - len(result.errors),
315 result.testsRun,
316 err_log.getvalue()
317 )

so its only when a test runner fails, that we get a error:

279 try:
280 return func(*a, **k)
281 except:
282 import traceback
283 return (0, 1j, traceback.format_exc())

and as in another thread i think we should change the tuple to the form:
(total, greens, errors, loading_erros, log)

lett the gui make the decision on what is wat and lett the pytddmon
class have the full picture.

--
//Samuel Ytterbrink

Olof Bjarnason

unread,
Nov 15, 2011, 4:50:48 PM11/15/11
to pytd...@googlegroups.com
2011/11/15 Samuel Ytterbrink <nep...@gmail.com>:

> Ok, but jst so you understand first.
>
> if you would rename a class that 5 test imports, then you would get 5 errors.

Yes, that is alright. The point is the user has to look into the
Details window to understand what is happening.

If you think this is a problem, I'd be satisfied with skipping that
number altogether and just writing "Error" in the main window. The
user has to read the log anyway.

>
> about the assert statement:
> i was looking in the code and foudn out that we actualy do have
> different ways of finding syntax/loading erros and errors. but we
> translates unittest errors as failures.
>
> 313     return (
> 314         result.testsRun - len(result.failures) - len(result.errors),
> 315         result.testsRun,
> 316         err_log.getvalue()
> 317     )
>
> so its only when a test runner fails, that we get a error:
>
> 279         try:
> 280             return func(*a, **k)
> 281         except:
> 282             import traceback
> 283             return (0, 1j, traceback.format_exc())
>
> and as in another thread i think we should change the tuple to the form:
> (total, greens, errors, loading_erros, log)
>
> lett the gui make the decision on what is wat and lett the pytddmon
> class have the full picture.

+1.

And it's great we already have that distinction between load-errors,
exception-errors and assertion-errors.

But if there are load-errors, I still think that should "take
priority" over everything else in the main GUI. But we might want to
call them just that, load-errors, instead of the now terribly ambigous
"errors":

2 load error(s)

Reply all
Reply to author
Forward
0 new messages