Visualization of error
Olof Bjarnason
background with 0/34 displayed in the center. Really un-informative
and frankly quite magic (where did that 34 come from?)
So I really like that the situation has improved quite a lot since
then. Nevertheless, I do think the current situation could be
improved.
I attached an image of what happens when there is a syntax error in
one of the unit tests of pytddmon itself. The first "28" means there
are 28 unit tests passing, the second "28" I'm not sure what it means,
and the last "2j" is something very creative to my mind. Two unit
tests are "imaginary" (j is used instead of i to describe the
imaginary part of a complex number; for some reason it is in pytddmon
now).
Obvious improvements:
1. Don't use complex numbers, at least not in the main window
2. Since syntax errors rules out discovering the number of errors in
total, I think a question mark would be good somewhere
3. Discuss what color should be used for syntax errors; it should be
more "severe" than unit test errors, but red feels more severe than
orange.
Samuel Ytterbrink
number of test files that did throw a error that the test framework
did not catch.
Colors?
So you sugest
green = ok,
orange = tests failed.
red = you got un caught exceptions.
hmm should there be a new way of writning the output?
28/28/2 ?
or only:
*2*
or something other?
2011/11/15 Olof Bjarnason <olof.bj...@gmail.com>:
--
//Samuel Ytterbrink
Olof Bjarnason
> The 2j here means that 2 tests (at least) failed hard. It is the
> number of test files that did throw a error that the test framework
> did not catch.
>
> Colors?
> So you sugest
> green = ok,
> orange = tests failed.
> red = you got un caught exceptions.
Well, that doesn't suit the basic colors of TDD: Red, Green, Refactor.
So I'm not sure what we should do.
>
> hmm should there be a new way of writning the output?
>
> 28/28/2 ?
> or only:
> *2*
Yeah, maybe simplify and not complicate? If there is a syntax error,
it is something severe. Why not stick with the red color, and change
the text from 34/34 to something like *2*. After all, it is not
possible to calculate the real total number of tests anymore, because
of the syntax error.
Some attempts at symbols to show this:
2?
2!
*2*
2"#%¤% (like in the comics, some one swearing haha)
Samuel Ytterbrink
So that a new user feels at home with the error, that is the problem
in the first place:
<push me!> is another great think to show.
2011/11/15 Olof Bjarnason <olof.bj...@gmail.com>:
--
//Samuel Ytterbrink
Olof Bjarnason
> I like the last one, but we need some mor information....
>
> So that a new user feels at home with the error, that is the problem
> in the first place:
>
> <push me!> is another great think to show.
Haha, yeah that's one idea. What about this:
[Push me. It's complicated]
:)
Maybe we should just type what we know: it is a compile-time error in
the code of the user:
<Error>
or even
Error
Olof Bjarnason
> 2011/11/15 Samuel Ytterbrink <nep...@gmail.com>:
>> I like the last one, but we need some mor information....
>>
>> So that a new user feels at home with the error, that is the problem
>> in the first place:
>>
>> <push me!> is another great think to show.
>
> Haha, yeah that's one idea. What about this:
>
> [Push me. It's complicated]
>
> :)
>
> Maybe we should just type what we know: it is a compile-time error in
> the code of the user:
>
> <Error>
>
> or even
>
> Error
Hmm. Since we know the number of modules that have syntax errors, we
could put that into the mix too:
2 error(s)
Samuel Ytterbrink
thing went wrong. I dont know exactly what types of errors the
unittest lib dont catch.
--
//Samuel Ytterbrink
Samuel Ytterbrink
[Push me. It's complicated]
how ever if you are fixing failing tests, you may still want to know
which tests fail and which dont. so removing the 28/28 may be a bad
ide. Cause you may want to fix the errors last.
--
//Samuel Ytterbrink
Olof Bjarnason
> well i like the:
>
> [Push me. It's complicated]
>
> how ever if you are fixing failing tests, you may still want to know
> which tests fail and which dont. so removing the 28/28 may be a bad
> ide. Cause you may want to fix the errors last.
I found out what distinction unittest does between "error" and "failure":
"Note that in order to test something, we use the one of the assert*()
methods provided by the TestCase base class. If the test fails, an
exception will be raised, and unittest will identify the test case as
a failure. Any other exceptions will be treated as errors. This helps
you identify where the problem is: failures are caused by incorrect
results - a 5 where you expected a 6. Errors are caused by incorrect
code - e.g., a TypeError caused by an incorrect function call."
So any exception (including syntax errors and a DivideByZero) in user
code (application or unit test) is counted as an error, while a failed
self.assertEqual+friends assertion is counted as a failure. [Sadly,
that also means using a built-in assert statement would probably also
count as an error and not a failure in unittest :( ]
Samuel Ytterbrink
Im 99% sure i have tested this.
--
//Samuel Ytterbrink
Olof Bjarnason
> dont think the assert statement givs a error.
> Im 99% sure i have tested this.
I just tested it, if gets treated as a failure! Horray! :)
Anyway, back on topic. I think we really shouldn't strive to use all
information we have in the main GUI. A syntax error situation (which
is the main point of this discussion, and a critical use case) is
exceptional and should be handled first (before fixing failures).
If the user really wants to fix the failures before the syntax errors,
he/should could rename those error-files temporarily to something not
of pattern test_*.py, to exclude them from pytddmons "evil eye" :)
So in summary, please go for "2 error(s)". Too much information is not
somewhere we want to return too in the minimalistic main GUI.
Samuel Ytterbrink
if you would rename a class that 5 test imports, then you would get 5 errors.
about the assert statement:
i was looking in the code and foudn out that we actualy do have
different ways of finding syntax/loading erros and errors. but we
translates unittest errors as failures.
313 return (
314 result.testsRun - len(result.failures) - len(result.errors),
315 result.testsRun,
316 err_log.getvalue()
317 )
so its only when a test runner fails, that we get a error:
279 try:
280 return func(*a, **k)
281 except:
282 import traceback
283 return (0, 1j, traceback.format_exc())
and as in another thread i think we should change the tuple to the form:
(total, greens, errors, loading_erros, log)
lett the gui make the decision on what is wat and lett the pytddmon
class have the full picture.
--
//Samuel Ytterbrink
Olof Bjarnason
> Ok, but jst so you understand first.
>
> if you would rename a class that 5 test imports, then you would get 5 errors.
Yes, that is alright. The point is the user has to look into the
Details window to understand what is happening.
If you think this is a problem, I'd be satisfied with skipping that
number altogether and just writing "Error" in the main window. The
user has to read the log anyway.
>
> about the assert statement:
> i was looking in the code and foudn out that we actualy do have
> different ways of finding syntax/loading erros and errors. but we
> translates unittest errors as failures.
>
> 313 return (
> 314 result.testsRun - len(result.failures) - len(result.errors),
> 315 result.testsRun,
> 316 err_log.getvalue()
> 317 )
>
> so its only when a test runner fails, that we get a error:
>
> 279 try:
> 280 return func(*a, **k)
> 281 except:
> 282 import traceback
> 283 return (0, 1j, traceback.format_exc())
>
> and as in another thread i think we should change the tuple to the form:
> (total, greens, errors, loading_erros, log)
>
> lett the gui make the decision on what is wat and lett the pytddmon
> class have the full picture.
+1.
And it's great we already have that distinction between load-errors,
exception-errors and assertion-errors.
But if there are load-errors, I still think that should "take
priority" over everything else in the main GUI. But we might want to
call them just that, load-errors, instead of the now terribly ambigous
"errors":
2 load error(s)