Coverage testing of exception-branches?

[Major Tal]

I posted this question at Stackoverflow:
http://stackoverflow.com/questions/2647790/python-pre-testing-for-exceptions-when-coverage-fails

Bottom line: can we extend coverage.py (just like the branch coverage
feature) so that when a statement **may** generate an exception (or
several exceptions) it will only be marked as Covered if we pass
through it and it actually does raise the exception?

This is a very important feature (IMHO), and the information is even
more important to me than lines of code I did not cover, since it has
the potential to completely crash my application.

How may I help?

[Ned Batchelder]

I'd thought about this as a possibility, and here's the problem: in
Python, we don't know which lines might throw an exception, or
alternately, we know that every line might throw an exception.

Really what you're looking for is a Java-like environment where every
function call declares what exception types it might throw. Then you
could insist that each function call actually be observed throwing that
exception. Unfortunately for us tool-makers (but fortunately for the
rest of the world using Python), Python doesn't require functions to
declare exceptions that way.

Coverage measurement fundamentally needs two phases: 1) observe
interesting events in a running program, and 2) determine where those
events could have occurred but didn't. I know how to measure exceptions
being thrown (phase 1). I don't know how to do phase 2. If you have
some ideas, I'd love to hear them, and maybe we can get something built.

--Ned.

[Tal Weiss]

I completely understand (and glad I'm not working in Java...).

I think a simple (yet mind-boggling boring) review of the Python documentation will reveal all the exceptions that are "part of the interface". getattr may raise AttributeError (see http://docs.python.org/library/functions.html#getattr), isinstance may raise TypeError (See http://docs.python.org/library/functions.html#isinstance ) etc.
Those are really all I'm currently losing sleep about. I understand that it is not a complete list, but it is a very big jump, quality wise.
What do you think?

Thanks,

Tål
http://www.google.com/profiles/major.tal

[Ned Batchelder]

But "obj.attr" can also raise AttributeError, and I certainly don't want to insist that every attribute access somehow raise AttributeError before the code is considered well-covered.  And as you point on on Stack Overflow, the ValueError you got from min() isn't even documented as a possibility.

I just don't see how to get a good handle on "everything that can happen".  With statement coverage, it's "every line can execute", with branch coverage it's "every branch takes all its outcomes".  But I don't expect every attribute access to raise AttributeError at some point, I don't even expect every getattr() to raise it.

My philosophy with coverage is that false negatives are the worst outcome, because they distract the programmer, and lower confidence in the tool.  Trying to imagine what an exception coverage report would look like, it seems to me that it would be full of alarms about lines that should have raised exceptions but didn't.  The developer wouldn't get far into that report before simply turning off the feature.

Another line of exploration for your original problem might be to get at the empty list rather than at the exception.  I can see real value in a developer being able to require certain predicates to be true of values.  In your case, perhaps we want to make sure the code is run with an empty list, a list of one element, lists with both even and odd numbers of elements, and a list with more than 1000 elements, and so on.

Unfortunately, this would probably require annotation by the developer, and if it doesn't occur to the developer to try an empty list, then how would it occur to him to annotate the function to get coverage to insist on it?

--Ned.

[Tal Weiss]

I agree with you that an exception report covering all possible exceptions would be too verbose, perhaps on the verge of making it useless. That said, to follow your example, I do think that people using getattr implicitly without supplying a default need to account for the possible AttributeError in the code, and if they forget to do so I would love it for there to be a tool to help remind them (before their application crashes in production).

I don't think that becuase we cannot develop the perfect exception coverage report we should not develop anything. Anything would still be better than nothing.

And, of course, like the branch coverage feature, it should be configurable.

I agree that user annotations are not a good approach (even though they cannot be avoided sometimes. My code has many sprinkles of pylint and pydev "remarks").

Thanks,

Tål
http://www.google.com/profiles/major.tal

[Ned Batchelder]

It would be very interesting to explore this vein.  It will require significant new functionality in coverage.py, since it currently has very little understanding of what code is being executed.  For example, it doesn't know where the getattr() calls are, much less which ones have a default value and which do not.  This is edging into pylint territory.

Others have experimented with this sort of thing.  Andrew Dalke wrote up AST-based instrumentation, he used it to find errors in %-formatting: http://www.dalkescientific.com/writings/diary/archive/2010/02/22/instrumenting_the_ast.html.  Matthew Desmarais started something similar called canopy (fantastic name!): http://bitbucket.org/desmaj/canopy/wiki/Home

One possibility that would be more in keeping with coverage.py's current skillset is to record where exceptions happened, and what exception they were.  This could be used to annotate the HTML listing.  Then a developer could at least review the listing and get a sense of where exceptions had happened and where they hadn't.  There wouldn't be any summary about where exceptions should have happened, but it would be a start.

--Ned.

[Tal Weiss]

Another thought re your previous argument: 

When the software contains obj.attr expressions, it (usually) means the attribute is static, so regular Line Coverage of that statement is sufficient to protect against surprise AttributeErrors. When people use getattr it usually means they are implementing a dynamic query and they must^H^H^H^H should implement an exception handler.

Pylint is an interesting idea.... Theoretically, I could add a regular expression to it that warns against "unprotected" built-in function calls. It's static analysis and not actual coverage, but it is a good starting point as well.

Thanks,

Tål
http://www.google.com/profiles/major.tal

[Zooko O'Whielacronx]

On Mon, Jul 26, 2010 at 5:36 AM, Ned Batchelder <n...@nedbatchelder.com> wrote:
>
> One possibility that would be more in keeping with coverage.py's current
> skillset is to record where exceptions happened, and what exception they
> were.  This could be used to annotate the HTML listing.  Then a developer
> could at least review the listing and get a sense of where exceptions had
> happened and where they hadn't.

I was recently wishing for this, in part to understand control flow
and in part to improve performance.

Regards,

Zooko