exception handling in complex Python programs
eliben
mechanism for its programmers. This is good. But exceptions, like any
complex construct, are difficult to use correctly, especially as
programs get large.
Most of the issues of exceptions are not specific to Python, but I
sometimes feel that Python makes them more acute because of the free-n-
easy manner in which it employs exceptions for its own uses and allows
users to do the same.
Now, what do I mean more specifically... When a program starts growing
large, I find myself a bit scared of all the exceptions that might be
thrown: Python's exceptions as a result of runtime-detection of errors
(Python's dynamic typing also comes into play here), exceptions from
libraries used by the code, and exceptions from my lower-level
classes.
Python doesn't allow to specify which exceptions are thrown (C++'s
feature adding 'throw' after a function/method declaration specifying
the exceptions that can be thrown), and this leaves me at loss - what
should be caught and where ? Which errors should be left to
propagate ?
I've tried looking around the Python blogosphere, but there doesn't
seem to be much concern with this topic.
Apologies for the not-too-coherent post, but I suspect you feel the
pain too and can understand my meaning.
Eli
P.S. There's a common case where a method is passed a filename, to do
something with a file (say, read data). Should the method catch the
errors possibly thrown by open(), or leave it to the caller ?
P.P.S. There's a great post on conditions (Common Lisp's exceptions)
here:
http://dlweinreb.wordpress.com/2008/03/24/what-conditions-exceptions-are-really-about/
Not really CL specific, and can apply to Python's exceptions.
Rafe
> Not really CL specific, and can apply to Python's exceptions.
Maybe I am oversimplifying (and I am here to learn), but I catch all
exceptions which otherwise would be hard to understand as a user. In
other words, when a better error message is useful.
Again, this is probably too simple to help, but the only way to ignore
certain types of exceptions, as far as I know, is to catch them and
pass.
e.g. this ignores type errors...
try:
somethingBad()
except TypeError, err:
pass
except Exception, err:
raise TypeError(err)
I suppose you could write a decorator to do this if you want it at the
function level, but that seems a bit to broad. Shouldn't exceptions be
on a case-by-case basis to add protection and return information
exactly where it is needed?
- Rafe
Chris Mellon
> Python provides a quite good and feature-complete exception handling
> mechanism for its programmers. This is good. But exceptions, like any
> complex construct, are difficult to use correctly, especially as
> programs get large.
>
> Most of the issues of exceptions are not specific to Python, but I
> sometimes feel that Python makes them more acute because of the free-n-
> easy manner in which it employs exceptions for its own uses and allows
> users to do the same.
>
Lots of people seem to have this fear. They treat exceptions like they
would treat error codes, trying to handle any possible case around any
particular call.
This is the wrong thing to do, and it only leads to more fragile code.
There are only 2 reasonable things to do with an exception:
1) handle it, by which I mean catch the exception knowing what error
condition it signifies, and take an appropriate action to correct the
error and
2) pass it up so something else has a chance at it.
Catching an exception when you don't know exactly what to do to fix it
is an error. At best, it will make debugging a program harder (because
you're losing context information about the error) and at worst it
adds bugs to your program. The way Javas checked exceptions encourage
empty or otherwise useless exception handlers is a major problem with
them.
There's some fear about presenting exceptions to the end user. That's
a user interface issues, not a software quality or engineering issue,
and it's resolvable with top-level handlers that log tracebacks
somewhere a user can't see them if desired.
> Now, what do I mean more specifically... When a program starts growing
> large, I find myself a bit scared of all the exceptions that might be
> thrown: Python's exceptions as a result of runtime-detection of errors
> (Python's dynamic typing also comes into play here), exceptions from
> libraries used by the code, and exceptions from my lower-level
> classes.
> Python doesn't allow to specify which exceptions are thrown (C++'s
> feature adding 'throw' after a function/method declaration specifying
> the exceptions that can be thrown), and this leaves me at loss - what
> should be caught and where ? Which errors should be left to
> propagate ?
>
You should catch anything that you can correct. If you don't have a
specific answer for a specific exception, don't catch it.
> I've tried looking around the Python blogosphere, but there doesn't
> seem to be much concern with this topic.
>
> Apologies for the not-too-coherent post, but I suspect you feel the
> pain too and can understand my meaning.
>
> Eli
>
> P.S. There's a common case where a method is passed a filename, to do
> something with a file (say, read data). Should the method catch the
> errors possibly thrown by open(), or leave it to the caller ?
>
Same rules apply. The only sort-of exception (no pun intended) is that
sometimes you want to re-raise as a different type of exception. Make
sure that you preserve all of the original information (including the
original traceback) if you do this.
> P.P.S. There's a great post on conditions (Common Lisp's exceptions)
> here:
> http://dlweinreb.wordpress.com/2008/03/24/what-conditions-exceptions-are-really-about/
> Not really CL specific, and can apply to Python's exceptions.
Fredrik Lundh
> Again, this is probably too simple to help, but the only way to ignore
> certain types of exceptions, as far as I know, is to catch them and
> pass.
> e.g. this ignores type errors...
>
> try:
> somethingBad()
> except TypeError, err:
> pass
> except Exception, err:
> raise TypeError(err)
so what kind of code are you writing where *type errors* are not
considered programming errors? (catching them and proceeding is one
thing, but catching them and ignoring them?)
I'd be really worried if I found that in a piece of source code I had to
maintain.
</F>
dbpo...@gmail.com
> P.S. There's a common case where a method is passed a filename, to do
> something with a file (say, read data). Should the method catch the
> errors possibly thrown by open(), or leave it to the caller ?
You want to look up Easier to Ask Forgivness than Permission (EAFP)
which is touted as the "canonical" error-handling paradigm for Python.
This would give rise to the following function:
def do_something(filename):
try:
f = open(filename)
except IOError:
return err("File %s not found" % filename)
...
where err is a function that generates an error object that your
application understands. I personally think this is sloppy because you
have to couple the exception type with the function --- between file()
and open() in Python 2 and 3, a NameError is thrown with open() in
Python 3 and an IOError is thrown in the other three cases <bashes
head against keyboard>. The alternative is
def do_something(filename):
if not os.access(filename,os.R_OK):
return err(...)
f = open(filename)
...
or, (and this last one I actually used for a web application)
def do_something(filename):
if not os.access(filename,os.R_OK):
raise MyApplicationsExceptionType("File not found...")
f = open(filename)
...
The last one has the advantage that you can write a request handler
like this
def handle_http_request(...):
func = specific_handler_func(...)
try:
response = func(...)
return response
except MyApplicationsExceptionType as exc: #3.0 syntax
return error_response(exc,...)
Exceptions you don't expect (i.e. bugs) will get handled by the web
app framework, but you get to handle your own exceptions. Raising your
own exception type can also be employed with the EAFP approach like
this:
def do_something(filename):
try:
f = open(filename)
except IOError:
raise MyApplicationsExceptionType("File %s not found" %
filename)
...
If you are writing a library (for instance using a file for persistent
storage), then the answer to your question is "don't catch the
exception." Clients will expect the usual exception to be thrown when
a bad file name is passed.
David
Steven D'Aprano
> def do_something(filename):
> if not os.access(filename,os.R_OK):
> return err(...)
> f = open(filename)
> ...
You're running on a multitasking modern machine, right? What happens when
some other process deletes filename, or changes its permissions, in the
time after you check for access but before you actually open it?
This isn't just a theoretical risk. There's a whole class of errors and
security holes based on similar race conditions. I find it amusing that
you consider it "sloppy" to deal with errors raised when actually opening
a file, but then recommend a technique that has a well-known failure mode.
That's not to say that I never use such techniques myself. For quick and
dirty scripts, where I can tolerate the risk of some other process moving
a file behind my back, I've been known to do something similar.
--
Steven
eliben
and open() in Python 2 and 3, a NameError is thrown with open() in
Python 3 and an IOError is thrown in the other three cases <bashes
head against keyboard>.
"""
This is *exactly* my concern with Python exceptions. You just never
know what can be thrown at you.
> You want to look up Easier to Ask Forgivness than Permission (EAFP)
> which is touted as the "canonical" error-handling paradigm for Python.
Any (semi)complete guides on this canonical paradigm online ? I've
only found some references in maillist discussions.
> def do_something(filename):
> if not os.access(filename,os.R_OK):
> return err(...)
> f = open(filename)
> ...
>
But does os.access cover absolutely all the errors that can happen
during open() ? What guarantees it, and how can I know without you
teaching me, just from the docs ?
Marc 'BlackJack' Rintsch
>> You want to look up Easier to Ask Forgivness than Permission (EAFP)
>> which is touted as the "canonical" error-handling paradigm for Python.
>
> Any (semi)complete guides on this canonical paradigm online ? I've only
> found some references in maillist discussions.
There's the glossary in the documentation:
http://docs.python.org/tut/node18.html
Look under 'duck-typing', 'EAFP', and 'LBYL'.
Ciao,
Marc 'BlackJack' Rintsch
Steven D'Aprano
> """ between file()
> and open() in Python 2 and 3, a NameError is thrown with open() in
> Python 3 and an IOError is thrown in the other three cases <bashes head
> against keyboard>.
> """
I'm curious about the claim that open() will raise NameError in Python3.
I find it hard to credit that claim, but if it is correct, what's the
justification for that?
> This is *exactly* my concern with Python exceptions. You just never know
> what can be thrown at you.
It's true that documentation of exceptions is relatively weak in Python.
And some functions can raise a bewildering array of exceptions. See for
example this thread where somebody notes that urllib2.urlopen() can raise
any of six different exceptions:
http://mail.python.org/pipermail/baypiggies/2008-April/003187.html
And I've had it raise socket.error, which makes seven. And the
documentation only mentions one of those exceptions.
However, as Gregory Smith describes, some of those seven exceptions are
subclasses of others, so it is possible to reduce it down to three cases
-- and arguably one of those cases (ValueError) is a bug that needs
fixing, not an exception that needs catching.
That's probably as bad as it gets in Python, at least for the standard
library. Most functions don't raise arbitrary exceptions for sensible
data, and if you pass non-sensible data then you should treat the
exception as a bug in your code and fix it.
--
Steven
dbpo...@gmail.com
cybersource.com.au> wrote:
> On Tue, 19 Aug 2008 11:07:39 -0700, dbpoko...@gmail.com wrote:
> > def do_something(filename):
> > if not os.access(filename,os.R_OK):
> > return err(...)
> > f = open(filename)
> > ...
>
> You're running on a multitasking modern machine, right? What happens when
> some other process deletes filename, or changes its permissions, in the
> time after you check for access but before you actually open it?
This is a good point - if you want to use the correct way of opening
files, and
you don't want to worry about tracking down exception types, then we
can probably
agree that the following is the simplest, easiest-to-remember way:
def do_something(filename):
try:
f = open(filename)
except:
<handle exception>
...
Opening files is a special case where EAFP is the only correct
solution (AFAIK). I still liberally sprinkle LBYL-style "assert
isinstance(...)" and other similar assertions in routines. The point
is that EAFP conflicts with the interest of reporting errors as soon
as possible (on which much has been written see, for instance Ch. 8 -
Defensive Programming in Code Complete), but LBYL conflicts with
correctness when objects can be shared.
Also, look at the man page for access. I have found at least two (one
on my Linux box, another online) that essentially say "never use it."
I completely forgot about this in my last post...
David
Bruno Desthuilliers
> On Aug 19, 10:19 am, eliben <eli...@gmail.com> wrote:
>
>> P.S. There's a common case where a method is passed a filename, to do
>> something with a file (say, read data). Should the method catch the
>> errors possibly thrown by open(), or leave it to the caller ?
>
> You want to look up Easier to Ask Forgivness than Permission (EAFP)
> which is touted as the "canonical" error-handling paradigm for Python.
> This would give rise to the following function:
>
> def do_something(filename):
> try:
> f = open(filename)
> except IOError:
> return err("File %s not found" % filename)
> ...
>
> where err is a function that generates an error object that your
> application understands.
Sorry but that's IMHO totally broken.
This "error object" is useless (heck, we *do* have exceptions, don't we
???), *returning* it ruins the whole point of structured exception
handling and take us back to infamous C error code checking (which are
almost never checked), and - icing on the cake - the error message is
very possibly wrong and misleading (IOError dont necessarily mean 'file
not found'). This kind of exception "handling" manages to be worse than
no exception handling at all.
> I personally think this is sloppy because you
> have to couple the exception type with the function --- between file()
> and open() in Python 2 and 3, a NameError is thrown with open() in
> Python 3
??? I suspect this has nothing to do with any error happening while
opening the file. NameError means the name doesn't exists in the current
namespace nor it's enclosing namespaces. Could it be possible that
open() has been removed from Py3k ?
> and an IOError is thrown in the other three cases <bashes
> head against keyboard>. The alternative is
>
> def do_something(filename):
> if not os.access(filename,os.R_OK):
> return err(...)
> f = open(filename)
> ...
This gets even worse. race condition... Things can change between the
call to os.access and the call to open. Well-known antipattern.
> or, (and this last one I actually used for a web application)
>
> def do_something(filename):
> if not os.access(filename,os.R_OK):
> raise MyApplicationsExceptionType("File not found...")
You loose all the useful information you'd have from an IOError raised
by a direct call to open...
> f = open(filename)
> ...
... IOError that you're still likely to see happen anyway.
> The last one has the advantage that you can write a request handler
> like this
>
> def handle_http_request(...):
> func = specific_handler_func(...)
> try:
> response = func(...)
> return response
> except MyApplicationsExceptionType as exc: #3.0 syntax
> return error_response(exc,...)
If you want to raise a different exception type - which can indeed be a
sensible thing to do, depending on the context -, you can do it safely
and keep accurate informations:
def do_something(filename):
try:
f = open(filename)
except IOError, e
raise MyApplicationsExceptionType(e.msg)
# could even pass whole traceback etc
# etc...
> Exceptions you don't expect (i.e. bugs)
An exception you don't expect is not necessarily a bug. Try unplugging
your lan cable while writing to a socket connected to another computer...
(snip)
> If you are writing a library (for instance using a file for persistent
> storage), then the answer to your question is "don't catch the
> exception." Clients will expect the usual exception to be thrown when
> a bad file name is passed.
Indeed.
Bruno Desthuilliers
>
> This is *exactly* my concern with Python exceptions. You just never
> know what can be thrown at you.
This rarely happen to be a problem in real life. At least not in mine.
Exception that can be expected (ie : IOError when dealing with files)
are usually obvious and more or less documented - or easy to figure out
(like TypeError and ValueError when trying to build an int from an
arbitrary object, KeyError when working with dicts, AttributeError when
inspecting an object, etc) from concrete use.
IOW, it's usually easy to know which exceptions you're able to deal with
at the lower level.
Any other exception is either a programming error - which needs to be
fixed, not hidden - or nothing you can deal with at the lower level - in
which case just let it propagate until some other layer above deal with
it (eventually just logging the error, displaying a user-friendly
message, and crashing if nothing else is possible).
>> def do_something(filename):
>> if not os.access(filename,os.R_OK):
>> return err(...)
>> f = open(filename)
>> ...
>>
>
> But does os.access cover absolutely all the errors that can happen
> during open() ? What guarantees it, and how can I know without you
> teaching me, just from the docs ?
The above code is a perfect antipattern. It's useless (if you can't
access the file, you'll get an IOError when trying to open it anyway),
it's wrong (things may change between the call to os.access and the call
to open), and it defeats the whole point of exception handling (by
returning some kind of error object instead of using exception handling).
Rafe
I'm not it was just the first exception that came to mind... It is
pretty rare that I would pass an exception in fact. Maybe as a last-
resort test in some cases.
- Rafe
Steven D'Aprano
> On Aug 19, 4:12 pm, Steven D'Aprano <st...@REMOVE-THIS-
> cybersource.com.au> wrote:
>> On Tue, 19 Aug 2008 11:07:39 -0700, dbpoko...@gmail.com wrote:
>> > def do_something(filename):
>> > if not os.access(filename,os.R_OK):
>> > return err(...)
>> > f = open(filename)
>> > ...
>>
>> You're running on a multitasking modern machine, right? What happens
>> when some other process deletes filename, or changes its permissions,
>> in the time after you check for access but before you actually open it?
>
> This is a good point - if you want to use the correct way of opening
> files, and
> you don't want to worry about tracking down exception types, then we can
> probably
> agree that the following is the simplest, easiest-to-remember way:
>
> def do_something(filename):
> try:
> f = open(filename)
> except:
> <handle exception>
No, we don't agree that that is the correct way of opening files. Simple
it might be, but correct it is not.
If you're using Python 2.6 or greater, then you should be using a with
block to handle file opening.
And regardless of which version of Python, you shouldn't use a bare
except. It will mask exceptions you *don't* want to catch, including
programming errors, typos and keyboard interrupts.
> Opening files is a special case where EAFP is the only correct solution
> (AFAIK). I still liberally sprinkle LBYL-style "assert isinstance(...)"
Oh goodie. Another programmer who goes out of his way to make it hard for
other programmers, by destroying duck-typing.
BTW, assertions aren't meant for checking data, because assertions can be
turned off. Outside of test frameworks (e.g. unit tests), assertions are
meant for verifying program logic:
def foo(x):
# This is bad, because it can be turned off at runtime,
# destroying your argument checking.
assert isinstance(x, int)
# And it raises the wrong sort of exception.
# This is better (but not as good as duck-typing).
if not isinstance(x, int):
raise TypeError('x not an int')
# And it raises the right sort of error.
y = some_function(x)
# y should now be between -1 and 1.
assert -1 < y < 1
do_something_with(y)
> and other similar assertions in routines. The point is that EAFP
> conflicts with the interest of reporting errors as soon as possible
Not necessarily. Tell me how this conflicts with reporting errors as soon
as possible:
def do_something(filename):
try:
f = open(filename)
except IOError, e:
report_exception(e) # use a GUI, log to a file, whatever...
How could you report the exception any earlier than immediately?
--
Steven
Marc 'BlackJack' Rintsch
>> I personally think this is sloppy because you have to couple the
>> exception type with the function --- between file() and open() in
>> Python 2 and 3, a NameError is thrown with open() in Python 3
>
> ??? I suspect this has nothing to do with any error happening while
> opening the file. NameError means the name doesn't exists in the current
> namespace nor it's enclosing namespaces. Could it be possible that
> open() has been removed from Py3k ?
No it's moved/changed but there's still a name for it in the builtin
namespace. `file` on the other hand is gone:
Python 3.0b2 (r30b2:65080, Aug 20 2008, 20:41:17)
[GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> open
<class 'io.OpenWrapper'>
>>> file
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'file' is not defined
Ciao,
Marc 'BlackJack' Rintsch
dbpo...@gmail.com
cybersource.com.au> wrote:
> Oh goodie. Another programmer who goes out of his way to make it hard for
> other programmers, by destroying duck-typing.
Remember kids: personal attacks are cruise control for cool.
So this was a simplification - most of the asserts I've written don't
actually use isinstance, partly because typing isinstance takes too
long. The point is to create a barricade so that when something goes
wrong, you get an assertion error against the code you wrote, not an
exception against doing something like
print("blah blah %s" % message)
where message turns out to be None. This is simply a way to make
debugging a more pleasant experience (quite valuable IMHO since
debugging is inherently difficult and can be quite aggravating). Here
is a sampling:
assert statelt.tag == 'stat'
assert len(path) > 0 and path[0] == '/'
assert self.__expr != None
So here asserts are used to made distinctions that are more fine-
grained than type.
> > and other similar assertions in routines. The point is that EAFP
> > conflicts with the interest of reporting errors as soon as possible
>
> Not necessarily. Tell me how this conflicts with reporting errors as soon
> as possible:
>
> def do_something(filename):
> try:
> f = open(filename)
> except IOError, e:
> report_exception(e) # use a GUI, log to a file, whatever...
>
> How could you report the exception any earlier than immediately?
Here is an example: a simple query tool for a tiny "mock SQL"
relational database. With a method (called "select_all") you can
perform the equivalent of a select query on the database. The contents
of the query are specified with triples of the form
[field, comparison_operator, value]
for instance ['name', operator.equals, cmd_name]. You can also specify
an "order by" field which is None by default. In the code written,
there is an assertion that the order-by field is either None or a
valid field name (we can't order by a nonexistent field!). If the
assertion isn't there, then I will get an error on this line:
key_extractor = KeyExtractor(q_column_names.index(order_by_column))
In this particular case, I will get a ValueError (what does ValueError
mean again? And what is this KeyExtractor?) since the index method
will fail. I wrote the tiny relational database a long time ago, and I
really don't want to put pressure on my mental cache by thinking about
the internal logic of this chunk of code. After scratching my head for
a while, I'll probably figure it out. Now imagine that you instead get
an error on this line:
assert order_by_column in q_column_names
Now the programming error slaps me with a fish and yells "STOP! YOU
CAN'T ORDER BY A FIELD THAT DOESN'T EXIST!!!". It will take about 2
seconds to figure out what went wrong. I just saved a minute figuring
out what the problem is. Multiply that by ten, and you've just
eliminated work in a potentially laborious debugging session.
If you look at the history of the EAFP concept in Python, then you see
that it comes from Alex Martelli's Python in a Nutshell around pages
113-114. I don't think the code examples make the case for EAFP very
well (not that I know what EAFP is in the first place, given that it
is barely explained. I interpret it as "wrap questionable stuff in try/
except blocks"), and in any case there is practically no support for
using EAFP as the dominant error-handling paradigm. If you look at
Code Complete, then you'll see the opposite suggestion, namely that
exceptions should only be used for truly exceptional circumstances
(i.e. bugs). McConnell argues that try/except is an inherently complex
control structure so it should be used sparingly (just like balsamic
vinegar!). I happen to think that truth lies between these extremes,
but I'd err on using fewer try/except structures, not more. Using
several try/except blocks across multiple activation records sounds
like unreadable code to me.
If shared objects are used pervasively, then I would predict that EAFP
will not provide adequate abstractions to control program complexity
(see http://research.microsoft.com/Users/simonpj/papers/stm/stm.pdf
and the Wikipedia article on software transactional memory). These
days you can switch to Stackless and use tasklets and atomic
operations (See http://www.stackless.com/wiki/Tasklets). There is a
debate between EAFP and LBYL here: http://mail.python.org/pipermail/python-list/2003-May/205182.html.
Martelli's posts in support of EAFP are heavily skewed towards a
multithreaded scenario and avoiding race conditions. IMHO, letting
locking and race condition concerns dictate your error-handling
paradigm is a case of the tail wagging the dog, especially when there
are alternatives to this particular tar pit: pipes or a shared nothing
architecture.
David
Steven D'Aprano
> On Aug 20, 10:59 am, Steven D'Aprano <st...@REMOVE-THIS-
> cybersource.com.au> wrote:
>> Oh goodie. Another programmer who goes out of his way to make it hard
>> for other programmers, by destroying duck-typing.
>
> Remember kids: personal attacks are cruise control for cool.
It might not be enjoyable to have a sarcastic remark directed your way,
but it isn't a personal attack. Just because a comment is about something
you do doesn't make it a personal attack. Personal attacks are about who
you are rather than what you do.
> So this was a simplification - most of the asserts I've written don't
> actually use isinstance, partly because typing isinstance takes too
> long.
You say that you liberally sprinkle isinstance() checks through your
code, then you say that you don't. That confuses me.
There's an apparent contradiction in your argument. You seem to be
arguing against EAFP and in favour of LBYL, but now you're suggesting
that you don't use type-checking. As near as I can tell, you don't do
type-checking, you don't do duck typing, you don't like catching
exceptions. So what do you actually do to deal with invalid data?
By the way, if you're worried that isinstance() is too long to type, you
can do this:
isin = isinstance
isin(123, int)
> The point is to create a barricade so that when something goes
> wrong, you get an assertion error against the code you wrote
...
> assert statelt.tag == 'stat'
> assert len(path) > 0 and path[0] == '/'
> assert self.__expr != None
All of those examples seem to be reasonably straight forward tests of
program logic, which would make them good cases for assertions. Assuming
that statelt, path and self.__expr are internally generated and not user-
supplied arguments.
I'll note that testing for (non-)equality against None is possibly a
mistake. It won't catch the case where self.__expr is an object that, for
some reason, compares equal to None but isn't None. If that's your
intention, then it's fine, but given that you don't seem to be a big fan
of duck typing I guess you'd probably be better off with:
assert self.__expr is not None
It runs faster too, although a micro-optimization of that tiny size isn't
in itself sufficient reason for preferring "is not" over "!=". The real
reason is to avoid accidental matches.
> So here asserts are used to made distinctions that are more fine-
> grained than type.
The problem with your earlier example isn't that isinstance() is too
coarse-grained (although I try to avoid it as much as possible), but that
assert is not meant for argument testing. In principle, the end user
should never see an AssertionError. As I said earlier, assert is for
testing program logic.
>> > and other similar assertions in routines. The point is that EAFP
>> > conflicts with the interest of reporting errors as soon as possible
>>
>> Not necessarily. Tell me how this conflicts with reporting errors as
>> soon as possible:
>>
>> def do_something(filename):
>> try:
>> f = open(filename)
>> except IOError, e:
>> report_exception(e) # use a GUI, log to a file, whatever...
>>
>> How could you report the exception any earlier than immediately?
>
> Here is an example:
I gather by your lack of answer to my question that you now accept that
exceptions don't necessarily delay reporting errors as early as possible.
> a simple query tool for a tiny "mock SQL" relational
> database. With a method (called "select_all") you can perform the
> equivalent of a select query on the database. The contents of the query
> are specified with triples of the form
>
> [field, comparison_operator, value]
>
> for instance ['name', operator.equals, cmd_name]. You can also specify
> an "order by" field which is None by default. In the code written, there
> is an assertion that the order-by field is either None or a valid field
> name (we can't order by a nonexistent field!). If the assertion isn't
> there, then I will get an error on this line:
>
> key_extractor = KeyExtractor(q_column_names.index(order_by_column))
>
> In this particular case, I will get a ValueError (what does ValueError
> mean again?
It means you've supplied an invalid value. What did you think it meant?
In fact, what you get is:
ValueError: list.index(x): x not in list
which tells you exactly what went wrong and why. It's concise and self-
explanatory.
> And what is this KeyExtractor?)
Irrelevant. That's not part of the exception. It just happens to be on
the same line of source code as the expression that raises an exception.
> since the index method will
> fail. I wrote the tiny relational database a long time ago, and I really
> don't want to put pressure on my mental cache by thinking about the
> internal logic of this chunk of code. After scratching my head for a
> while, I'll probably figure it out. Now imagine that you instead get an
> error on this line:
>
> assert order_by_column in q_column_names
>
> Now the programming error slaps me with a fish and yells "STOP! YOU
> CAN'T ORDER BY A FIELD THAT DOESN'T EXIST!!!". It will take about 2
> seconds to figure out what went wrong.
You're assuming that, six months from now, you'll see this error:
>>> q_column_names = ['fee', 'fi', 'fo', 'fum']
>>> order_by_column = 'floop'
>>> assert order_by_column in q_column_names
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AssertionError
and remember that the reason you were testing the assertion is that
*later on* you intend to order a field by something named
"order_by_column". Or maybe order_by_column is what you've *already*
ordered the field by, and now you're about to do something else? Who
knows what it means? It could mean anything.
To make your code actually useful, you need to supply an assertion string
that explains *why you care* about the assertion, and even then you
better not rely on it because asserts can be turned off at runtime and
the test might not even be executed.
> I just saved a minute figuring
> out what the problem is. Multiply that by ten, and you've just
> eliminated work in a potentially laborious debugging session.
I'm not convinced by your example. Your example seems to actually make
debugging harder, not easier.
> If you look at the history of the EAFP concept in Python, then you see
> that it comes from Alex Martelli's Python in a Nutshell around pages
> 113-114.
I think it actually comes from a lot earlier than Alex's book.
> I don't think the code examples make the case for EAFP very
> well (not that I know what EAFP is in the first place, given that it is
> barely explained.
Deary me. Perhaps you should find out what it is before critiquing it?
> I interpret it as "wrap questionable stuff in try/
> except blocks"),
As a one-line summary, that's not bad.
> and in any case there is practically no support for
> using EAFP as the dominant error-handling paradigm.
You're joking, right? You're trying to wind me up for a laugh?
What exactly do you think try...except... blocks are for, if not EAFP?
> If you look at Code
> Complete, then you'll see the opposite suggestion, namely that
> exceptions should only be used for truly exceptional circumstances (i.e.
> bugs).
No no no, exceptions are not necessarily bugs!!! A bug is an exceptional
circumstance, but not all exceptional circumstances are bugs.
>>> list_of_cheeses = "Wensleydale Mozzarella Stilton Edam Feta"
>>> list_of_cheeses.index('Cheddar')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: substring not found
That's not a bug, that just means that Cheddar isn't found in the list of
cheeses. The bug is *failing to deal with the exceptional circumstances*.
>>> try:
... list_of_cheeses.index('Cheddar')
... except ValueError:
... print "Sorry sir, we don't sell cheddar. It's not very popular."
... else:
... print "Yes, we have some cheddar."
...
Sorry sir, we don't sell cheddar. It's not very popular.
I think I can summarize the rest of your post as:
* EAFP isn't a panacea for everything.
* badly written code can abuse try...except blocks.
* if you run a single-tasking architecture where nothing is shared, then
race conditions aren't a problem.
I certainly won't argue with the first two.
--
Steven
Gabriel Genellina
> On Aug 20, 10:59 am, Steven D'Aprano <st...@REMOVE-THIS-
> cybersource.com.au> wrote:
>> Oh goodie. Another programmer who goes out of his way to make it hard for
>> other programmers, by destroying duck-typing.
>
> Remember kids: personal attacks are cruise control for cool.
>
> So this was a simplification - most of the asserts I've written don't
> actually use isinstance, partly because typing isinstance takes too
> long. The point is to create a barricade so that when something goes
> wrong, you get an assertion error against the code you wrote, not an
> exception against doing something like
>
> print("blah blah %s" % message)
>
> where message turns out to be None. This is simply a way to make
> debugging a more pleasant experience (quite valuable IMHO since
> debugging is inherently difficult and can be quite aggravating). Here
> is a sampling:
>
> assert statelt.tag == 'stat'
> assert len(path) > 0 and path[0] == '/'
> assert self.__expr != None
>
> So here asserts are used to made distinctions that are more fine-
> grained than type.
I think you missed the point. All of those look like program logic verification, and that's fine. But using assert to check user-supplied data is wrong (here "user" may be another programmer if you're developing a library). Assertions may be turned off at runtime.
> If you look at the history of the EAFP concept in Python, then you see
> that it comes from Alex Martelli's Python in a Nutshell around pages
> 113-114.
Mmm, I think it's older than that.
> I don't think the code examples make the case for EAFP very
> well (not that I know what EAFP is in the first place, given that it
> is barely explained.
Ok, so you don't know what it is, but dislike it anyway?
> I interpret it as "wrap questionable stuff in try/
> except blocks"), and in any case there is practically no support for
> using EAFP as the dominant error-handling paradigm.
Uh? I think that using try/except IS the "dominant error-handling paradigm" and that's just EAFP.
> Martelli's posts in support of EAFP are heavily skewed towards a
> multithreaded scenario and avoiding race conditions. IMHO, letting
> locking and race condition concerns dictate your error-handling
> paradigm is a case of the tail wagging the dog, especially when there
> are alternatives to this particular tar pit: pipes or a shared nothing
> architecture.
There are race conditions since multiprogramming existed, around '70, and I'm afraid they'll stay for a long time...
--
Gabriel Genellina
Richard Levasseur
every package or module has a main Error class, and all sub-types
inherit from that class. So you just catch mylib.Error, and you're
going to catch all the exceptions that package generates directly.
There seems to be a lot of concern about what exceptions a functions
might throw, and what to do when it happens. The way I see it, there
are only two types of exceptions: those you can recover from, and
those you can't.
The only ones -really- worth catching are those you can recover from.
There's a middle-ground type of 'cleanup', to free any resources, but
those generally go in a finally block, not an except block.
For the ones you can't handle, it doesn't matter if you catch them or
not. If you catch them, what do you do? Log an error message, then
rethrow it. You're still throwing an exception, so you haven't really
gained anything. You might repackage it and put additional
information in the exception so you can do something at a higher
level. What that is, I don't know. I don't think I've ever passed
information up in an exception that was of use to the program, and I'm
hard pressed to think of any information you could provide that could -
fix- the problem.
If you can derive recoverable information, then why rethrow? Thats
pretty much a recoverable situation, so there's not need to rethrow.
In java, there are checked exceptions, which are nice - they tell you
what a function might throw, so you know what to catch. I don't think
this improves the quality of anything, though. It just annoys the
developer. What they end up doing is writing an application-specific
exception class, and -everything- gets rethrown as that, and
everything begins to declare it throws AppError. Whats worse is that
you have heavily, heavily repackaged exceptions: SQLError -> AppError
-> MessageError -> AppError -> MessageError (yes, i've seen this
before).
That is almost completely useless. Sure, you could dig down to
SQLError, but how do you know to do that? If you knew how far you
should dig down, that means you know what the problem was, in which
case, you could have prevented it or aborted early. Whats worse, if
you have all these re-packaging catch blocks and they just log
something generic, which becomes common with all the catching going
on. "Couldn't do foo!", "Bar operation failed!", or "Couldn't fetch
filters from database" (why? We don't know, its catching an AppError
instead of something more specific), and then they rethrow the
exception. While trying to debug something not-during development,
those messages are completely useless, in fact, they're more than
useless. They're just more cruft to sift through in a log file.
Additionally, most root causes of an error are going to originate
where the input comes from the user. Handling anything at levels
deeper than that isn't going to gain you much. PrepareQuery threw an
error because of a missing field? Thats great. Where'd it come
from? There are 100 calls to PrepareQuery. There's only a few calls
to ReadUserInput(), and a single informative log message of "Query
failed, unknown field; fields=a, b, c" is much better than 100 lines
of traceback smattered with "Unknown field" and "Unable to prepare
query".
Finally, they give a false sense of security. "I'm catching
everything it could throw, so everything will be ok if an exception is
thrown!" I guess thats true. I guess. The only real advantage is
the whole program won't crash with the ever-helpful, single line of
"Segmentation fault." An improvement, but it doesn't prevent
anything.
In a complex system, an error can occur in any function at anytime.
Adding 'throws' to a method definition doesn't change that.
I guess my point is: everything Chris Mellon said was spot on.
eliben
> Python provides a quite good and feature-complete exception handling
Thanks for the interesting discussion. Armed by the new information
and few online sources, I blogged a summary for myself on the topic of
robust exception handling in Python:
http://eli.thegreenplace.net/2008/08/21/robust-exception-handling/
Bruno Desthuilliers
> On Aug 19, 4:12 pm, Steven D'Aprano <st...@REMOVE-THIS-
> cybersource.com.au> wrote:
>> On Tue, 19 Aug 2008 11:07:39 -0700, dbpoko...@gmail.com wrote:
>>> def do_something(filename):
>>> if not os.access(filename,os.R_OK):
>>> return err(...)
>>> f = open(filename)
>>> ...
>> You're running on a multitasking modern machine, right? What happens when
>> some other process deletes filename, or changes its permissions, in the
>> time after you check for access but before you actually open it?
>
> This is a good point - if you want to use the correct way of opening
> files, and
> you don't want to worry about tracking down exception types, then we
> can probably
> agree that the following is the simplest, easiest-to-remember way:
>
> def do_something(filename):
> try:
> f = open(filename)
> except:
> <handle exception>
> ...
Still not correct IMHO - bare except clauses are BAD. You want:
try:
f = open(filename)
except IOError, e:
<handle exception>
> Opening files is a special case where EAFP is the only correct
> solution (AFAIK). I still liberally sprinkle LBYL-style "assert
> isinstance(...)"
Which defeats the whole point of dynamic typing...
> and other similar assertions in routines.
> The point
> is that EAFP conflicts with the interest of reporting errors as soon
> as possible (on which much has been written see, for instance Ch. 8 -
> Defensive Programming in Code Complete),
Defensive programming makes sense in the context of a low-level language
like C where errors can lead to dramatic results. In high-level
languages like Python, the worse thing that an unhandled exception can
cause is an abrupt termination of the process and a nice traceback on
screen. In this context, defensive programming is mostly a waste of time
- if you can't *correctly* handle the exception where it happens, then
doing nothing is the better solution.
My 2 cents...
Bruno Desthuilliers
(snip)
> Here is an example: a simple query tool for a tiny "mock SQL"
> relational database. With a method (called "select_all") you can
> perform the equivalent of a select query on the database. The contents
> of the query are specified with triples of the form
>
> [field, comparison_operator, value]
>
> for instance ['name', operator.equals, cmd_name]. You can also specify
> an "order by" field which is None by default. In the code written,
> there is an assertion that the order-by field is either None or a
> valid field name (we can't order by a nonexistent field!). If the
> assertion isn't there, then I will get an error on this line:
>
> key_extractor = KeyExtractor(q_column_names.index(order_by_column))
>
> In this particular case, I will get a ValueError (what does ValueError
> mean again? And what is this KeyExtractor?) since the index method
> will fail. I wrote the tiny relational database a long time ago, and I
> really don't want to put pressure on my mental cache by thinking about
> the internal logic of this chunk of code. After scratching my head for
> a while, I'll probably figure it out. Now imagine that you instead get
> an error on this line:
>
> assert order_by_column in q_column_names
>
> Now the programming error slaps me with a fish and yells "STOP! YOU
> CAN'T ORDER BY A FIELD THAT DOESN'T EXIST!!!".
As far as I'm concerned, this is a case where I would explicitely raise
an exception (either a ValueError with an explicit message or a
library-defined exception type).
Steven D'Aprano
Just a few random points. You say:
"Exceptions are better than returning error status codes. Some languages
(like Python) leave you with no choice as the whole language core and
standard libraries throw exceptions."
Of course you have a choice. Your function can return anything you want:
def mysqrt(x):
try:
return math.sqrt(x)
except ValueError:
return "Code 37"
I've written functions that return an object on success and None if the
function failed. In the context of what I was doing, that made more sense
than raising an exception.
Furthermore, the str.find() method returns -1 for not found instead of
raising an exception. There are probably other examples as well.
You also wrote:
"Exceptions exist for exceptional situations: unanticipated events that
are not a part of normal execution."
Exceptions can and often are anticipated. E.g. if you write code that
opens a URL, you better anticipate that the server might reject your
connection. You better expect to be asked for a cookie, or
authentication. If you check for robots.txt, you better expect that it
might not exist. That's all normal execution.
"When a programmer calls str.find('substring') he doesn’t expect an
exception to be thrown if the substring isn’t found."
But if he called str.index() then he does expect an exception to be
thrown, just like for list.index() and dict[key] can raise exceptions.
They are neither bugs nor unexpected.
"This is what he called find for. A better approach is to return a
special value like None or -1."
Sometimes, maybe. But usually not, because that just complicates the
calling code. You end up writing code that repeatedly checks that the
result isn't a special value before doing anything.
Often a better tactic is to write your code assuming that the result is
the unexceptional case, and then wrap it in a try...except block to catch
the exceptional cases.
"When used for flow-control, exceptions are like goto. There might be a
few esoteric cases in which they’re appropriate, but 99.99% of the time
they are not."
I strongly disagree. try...except is like break or continue. Yes, it
breaks the linear flow of control, but not in a wild, dangerous way like
goto.
It is possible to write bad code with exceptions, but you can write bad
code with anything.
--
Steven
Bruno Desthuilliers
"""
When used for flow-control, exceptions are like goto. There might be a
few esoteric cases in which they’re appropriate, but 99.99% of the time
they are not.
"""
Python itself uses exceptions for flow control in iterators.
"""
For some exceptions, like programming errors (e.g. IndexError,
TypeError, NameError etc.) exceptions are best left to the programmer /
user, because “handling” them will just hide real bugs.
"""
Depends on the context. There are cases where you expect these kind of
errors - like when dealing with program inputs, inspecting objects etc.
As a Q&D example:
while True:
raw_num = raw_input("enter a number")
try:
num = float(raw_num)
except TypeError, ValueError:
print "sorry, '%s' is not a valid number" % raw_num
else:
# ok
break
"""
This is also the reason why you should be extremely careful with except:
clauses that catch everything. These will not only catch the exceptions
you intended, but all of them.
"""
And remember that SysExit and KeyboardInterrupt *are* exceptions too...
"""
Document the exceptions thrown by your code
"""
If you mean "the exceptions *explicitely raised* by your code", then I
agree. But with any generic enough code, documenting any possible
exception that could be raised by lower layers, objects passed in as
arguments etc is just plain impossible. Like, if you have a function
that takes a file-like object as arg, you just cannot know in advance
what exceptions this object might raise.
My 2 cents.
eliben
>
> Just a few random points. You say:
>
> "Exceptions are better than returning error status codes. Some languages
> (like Python) leave you with no choice as the whole language core and
> standard libraries throw exceptions."
>
> Of course you have a choice. Your function can return anything you want:
>
Of course. I didn't mean that the language prohibits returning error
codes, just that you can't use it without employing exception
handling. I've fixed the wording to make it clearer.
> You also wrote:
>
> "Exceptions exist for exceptional situations: unanticipated events that
> are not a part of normal execution."
>
> Exceptions can and often are anticipated. E.g. if you write code that
> opens a URL, you better anticipate that the server might reject your
> connection. You better expect to be asked for a cookie, or
> authentication. If you check for robots.txt, you better expect that it
> might not exist. That's all normal execution.
This is a point I'm not 100% in accord with. I still think that
exceptions are for exceptional situations. I've removed the word
"unanticipated" though, because it probably has no place in that
sentence. However, I think that if one of your valid execution paths
is w/o robots.txt, you should not use an exception to check whether
it's there. This indeed uses the "bad side" of exceptions, splitting
the exetution to two paths.
Check if robots.txt is there. If it is, open it. If you can't open it,
*that* is an exception, but if it's just not there, well it's part of
your application logic. I believe this isn't against EAFP.
I'm not sure I'm making the distinction clear here, it's a fine point.
>
> "When a programmer calls str.find('substring') he doesn’t expect an
> exception to be thrown if the substring isn’t found."
>
> But if he called str.index() then he does expect an exception to be
> thrown, just like for list.index() and dict[key] can raise exceptions.
> They are neither bugs nor unexpected.
>
But why are there two versions that are the same except for the
behavior in case it wasn't found ? My wishful imagination is precisely
because of the reasons I've named. If you *know* it's there,
use .index() - then, if it fails, it's an exception, but if a part of
your logic is finding an item that might be missing, use a special
value because you want to keep the logic in a single path.
> "When used for flow-control, exceptions are like goto. There might be a
> few esoteric cases in which they’re appropriate, but 99.99% of the time
> they are not."
>
> I strongly disagree. try...except is like break or continue. Yes, it
> breaks the linear flow of control, but not in a wild, dangerous way like
> goto.
>
try...except can 'exit' to several 'catch points', unlike break/
continue. Furthermore, try...except can bring execution to another
hierarchy level if it's not caught where it's thrown, so it's much
more like goto in these senses. To find where the execution may go
you'll find yourself searhching for the exception name over your
source files, looking for the exception class name in some "except"
clause. Sounds like looking for a goto label.
P.S. Thanks a lot for taking the time to comment
Eli
eliben
42.desthuilli...@websiteburo.invalid> wrote:
> eliben a écrit :> On Aug 19, 7:19 pm, eliben <eli...@gmail.com> wrote:
> >> Python provides a quite good and feature-complete exception handling
> > <snip>
>
> > Thanks for the interesting discussion. Armed by the new information
> > and few online sources, I blogged a summary for myself on the topic of
> > robust exception handling in Python:
>
> >http://eli.thegreenplace.net/2008/08/21/robust-exception-handling/
>
> A couple comments (mostly python-specific, so I post them here):
>
Thanks for the feedback. My comments below:
> """
> When used for flow-control, exceptions are like goto. There might be a
> few esoteric cases in which they’re appropriate, but 99.99% of the time
> they are not.
> """
>
> Python itself uses exceptions for flow control in iterators.
>
Yep, I'm aware of StopIteration, but I'm not sure whether it's a good
or a bad feature. I'm a bit wary of the programming style it might
encourage in inexperienced programmers. When this behavior is hidden
inside the implementation of 'for', fair enough. But when you have to
catch exceptions just to walk over some iterable explicitly, I'm not
sure the designers of this Python feature made the correct choices
here.
> """
> For some exceptions, like programming errors (e.g. IndexError,
> TypeError, NameError etc.) exceptions are best left to the programmer /
> user, because “handling” them will just hide real bugs.
> """
>
> Depends on the context. There are cases where you expect these kind of
> errors - like when dealing with program inputs, inspecting objects etc.
> As a Q&D example:
>
> while True:
> raw_num = raw_input("enter a number")
> try:
> num = float(raw_num)
> except TypeError, ValueError:
> print "sorry, '%s' is not a valid number" % raw_num
> else:
> # ok
> break
>
I agree.
> """
> This is also the reason why you should be extremely careful with except:
> clauses that catch everything. These will not only catch the exceptions
> you intended, but all of them.
> """
>
> And remember that SysExit and KeyboardInterrupt *are* exceptions too...
>
> """
> Document the exceptions thrown by your code
> """
>
> If you mean "the exceptions *explicitely raised* by your code", then I
> agree. But with any generic enough code, documenting any possible
> exception that could be raised by lower layers, objects passed in as
> arguments etc is just plain impossible. Like, if you have a function
> that takes a file-like object as arg, you just cannot know in advance
> what exceptions this object might raise.
>
This is one of the main concerns with which I started this c.l.py
thread ! I think it's a pity that we have no way of anticipating and
constraining the exceptions thrown by our code, and that we should
strive to make it more explicit. The "function accepting a file" is a
case in point. You know what you do with this file, so why can't you
know what exceptions might be thrown ? If you're trying to open it,
IOError (and OSError ?), etc. Besides, as I noted in the article,
perhaps you want to hide some of inner-level exceptions in your own,
to keep encapsulation.
Eli
Bruno Desthuilliers
> On Aug 21, 12:40 pm, Bruno Desthuilliers <bruno.
> 42.desthuilli...@websiteburo.invalid> wrote:
>> eliben a écrit :> On Aug 19, 7:19 pm, eliben <eli...@gmail.com> wrote:
>>> """
>>> Document the exceptions thrown by your code
>>> """
>>
>> If you mean "the exceptions *explicitely raised* by your code", then I
>> agree. But with any generic enough code, documenting any possible
>> exception that could be raised by lower layers, objects passed in as
>> arguments etc is just plain impossible. Like, if you have a function
>> that takes a file-like object as arg, you just cannot know in advance
>> what exceptions this object might raise.
>>
>
> This is one of the main concerns with which I started this c.l.py
> thread ! I think it's a pity that we have no way of anticipating and
> constraining the exceptions thrown by our code,
Java's "checked exception" system has proven to be a total disaster.
> and that we should
> strive to make it more explicit. The "function accepting a file" is a
> case in point. You know what you do with this file, so why can't you
> know what exceptions might be thrown ?
Reread more carefully. I wrote "a *file-like* object", not "a file".
This is the whole point of duck typing. Given that any file-like object
will work ok with my function, I have *no way* to know what exceptions
this object may raise.
> If you're trying to open it,
Trying to open a file *object* ? heck, it's supposed to be already
opened at this stage. And yes, it's a pretty common pattern in Python
(which is why I choose this example).
magloca
>>> If you mean "the exceptions *explicitely raised* by your code", then
>>> I agree. But with any generic enough code, documenting any possible
>>> exception that could be raised by lower layers, objects passed in as
>>> arguments etc is just plain impossible. Like, if you have a function
>>> that takes a file-like object as arg, you just cannot know in
>>> advance what exceptions this object might raise.
>>>
>>
>> This is one of the main concerns with which I started this c.l.py
>> thread ! I think it's a pity that we have no way of anticipating and
>> constraining the exceptions thrown by our code,
>
> Java's "checked exception" system has proven to be a total disaster.
Could you elaborate on that? I'm not disagreeing with you (or agreeing,
for that matter); I'd just really like to know what you mean by
a "total disaster."
m.
Bruno Desthuilliers
One of the most (in)famous Java coding pattern is the empty catchall
clause. Read Chris Mellon and Richard Levasseur posts in this thread for
more details - they already covered the whole point.
dbpo...@gmail.com
cybersource.com.au> wrote:
> It might not be enjoyable to have a sarcastic remark directed your way,
> but it isn't a personal attack. Just because a comment is about something
> you do doesn't make it a personal attack. Personal attacks are about who
> you are rather than what you do.
If you type in "Personal Attack" in Wikipedia (not an authoritative
source, I know) it takes you to the page on ad hominem arguments.
There you can find the following example of the fallacious ad hominem
argument:
Person A makes claim X
There is something objectionable about Person A
Therefore claim X is false
It is, ultimately, a matter of opinion, but "going out of one's way to
make it hard for other programmers" sounds objectionable to me. I
mean, I wouldn't want to work with anyone like that!
> There's an apparent contradiction in your argument. You seem to be
> arguing against EAFP and in favour of LBYL, but now you're suggesting
> that you don't use type-checking. As near as I can tell, you don't do
> type-checking, you don't do duck typing, you don't like catching
> exceptions. So what do you actually do to deal with invalid data?
Here is an example from a Django web app: when there is a bug, a
generic Exception is thrown and Django catches it and reports a
beautifully formatted stack trace. When something must be reported to
the user, a MyAppException is thrown (not the real name). The HTTP
request handler for the application is wrapped in a single try:...
except MyAppException:.... the big idea is that there should be a
maximum of two try/except blocks on the stack at any particular point
in time [1]: at a high level (already mentioned) and for wrapping
primitive "execute" operations against Rpy and MySQLdb. In practice,
this doesn't always happen - there is one place where an EAFP-style
construct is used (the "operation" in this case is to generate some
HTML and cache it based on some source XML, but the source may have
"syntax errors", so if the HTML can't be generated, then cleanup is
performed and an error message returned).
So to summarize:
try/except blocks at boundaries between system components: good
try/except blocks within a single component: slightly concerning
I think I may have overstated the case against EAFP. There are
certainly cases where EAFP makes a lot of sense; I would object to
portraying EAFP as an alternative to defensive programming. [Side
note: defensive programming serves much the same purpose in Python as
it does in C, but I agree that in C there is extra motivation such as
avoiding buffer overruns. I think of defensive programming simply as
"taking proactive steps to reduce the expected time to debug a program
if a programming error should arise".]
> By the way, if you're worried that isinstance() is too long to type, you
> can do this:
>
> isin = isinstance
> isin(123, int)
Actually I'm holding out for type objects to grow __lt__ and __le__
methods so you can do something like
from abstract_base_classes import sequence
if type(my_obj) <= sequence:
...
This is borrowed from the <= notation for subgroups in math (there are
probably other cases too). I don't use abstract base classes, so I
don't even know if this is right, but hopefully you get the idea.
> No no no, exceptions are not necessarily bugs!!! A bug is an exceptional
> circumstance, but not all exceptional circumstances are bugs.
I tend to agree, but I have found that thinking about these issues
makes me question the wisdom of Python's built-ins throwing exceptions
in non-exceptional circumstances (for instance you try to open a file
that doesn't exist - IMHO this is about as exceptional as trying a no-
wait acquire on a busy lock, in other words it isn't exceptional at
all). As long as we are in fantasy realm, one could argue that open()
should return a pair like this:
f, open_ok = open(...)
where open_ok is a status object whose __nonzero__ (in 3.0 __bool__ is
used) is true on success and false on an error, and also has an error
code and error message field. The idea is from Django's get_or_create
method in the db API.
[1] That I have to think about. I don't particularly care about try/
except blocks in Django's, Rpy's, or MySQLdb's activation records.
David
eliben
> generic Exception is thrown and Django catches it and reports a
> beautifully formatted stack trace. When something must be reported to
> the user, a MyAppException is thrown (not the real name). The HTTP
> request handler for the application is wrapped in a single try:...
> except MyAppException:.... the big idea is that there should be a
> maximum of two try/except blocks on the stack at any particular point
> in time [1]: at a high level (already mentioned) and for wrapping
> primitive "execute" operations against Rpy and MySQLdb.
This actually makes lots of sense as is exactly what Ned Batchelder
wrote here:
http://nedbatchelder.com/text/exceptions-in-the-rainforest.html
* A-layer generates exceptions,
* B-layer can often ignore the whole issue, and
* C-layer decides what to do
It's worth a read.
eliben
> On Tue, Aug 19, 2008 at 12:19 PM, eliben <eli...@gmail.com> wrote:
> > Python provides a quite good and feature-complete exception handling
> > complex construct, are difficult to use correctly, especially as
> > programs get large.
>
> > Most of the issues of exceptions are not specific to Python, but I
> > sometimes feel that Python makes them more acute because of the free-n-
> > easy manner in which it employs exceptions for its own uses and allows
> > users to do the same.
>
> Lots of people seem to have this fear. They treat exceptions like they
> would treat error codes, trying to handle any possible case around any
> particular call.
>
> This is the wrong thing to do, and it only leads to more fragile code.
> There are only 2 reasonable things to do with an exception:
> 1) handle it, by which I mean catch the exception knowing what error
> condition it signifies, and take an appropriate action to correct the
> error and
> 2) pass it up so something else has a chance at it.
>
But by 'handling', do you also mean "rethrow with better
information" ?
I feel there's an inherent clash between two 'good practices' in
exception handling:
1) Using EAFP over LBYL
2) Hiding implementation details
Consider this code, which I wrote just yesterday:
elif type in ('LinearStartAddr', 'SegmentStartAddr'):
if len(data) != 4:
line_error('expecting a 4-byte data field for this record type,
got %s' % len(data))
self.data.start_address = unpack('>L', data)
This is part of a method in a class that parses a data file. I've
ended up using LBYL here, to hide an implementation detail. I could've
let the Exception from unpack propagate, but that doesn't make much
sense with "hiding implementation". So I'm throwing a more useful
exception myself.
Was wrapping the call to unpack with try/except that throws my
exception a better idea, in your opinion ? Because that makes the code
somewhat more convoluted.
Eli
magloca
Thanks, I missed those. Having read them, I *am* agreeing with you.
Personally, I also dislike the predominance in the Java world of only
giving type information -- IllegalValueException,
ObjectRetrievalFailureException, and whatever. What was the illegal
value? What object couldn't be retrieved? How hard is it to use the
with-error-message version of the Exception constructor, and include
something that might actually be helpful in debugging?
m.
Maric Michaud
> > The point
> > is that EAFP conflicts with the interest of reporting errors as soon
> > as possible (on which much has been written see, for instance Ch. 8 -
> > Defensive Programming in Code Complete),
>
> Defensive programming makes sense in the context of a low-level language
> like C where errors can lead to dramatic results. In high-level
> languages like Python, the worse thing that an unhandled exception can
> cause is an abrupt termination of the process and a nice traceback on
> screen.
... and leave your datas in inconsistent state. So, what C or any other
language could do worse to your application ?
> In this context, defensive programming is mostly a waste of time
> - if you can't *correctly* handle the exception where it happens, then
> doing nothing is the better solution.
If I don't buy the argument I actually agree with the conclusion. Each
component of a program should try to manage only errors tied to their own
logic and let pass others up to the gui logic for rendering errors the good
way, persistence logic to rollback unwanted changes, and application logic to
continue execution the right way. This is hard to do in C because you have no
way to trap an error which happen randomly in the program, ie. a segfault
will interrupt the execution anyway.
--
_____________
Maric Michaud
Lie
cybersource.com.au> wrote:
I'm sure different people would have different view on what
immediately means. The LBYL supporters define immediately as
immediately after a definite potential problem is detected (by ifs or
assertion), while the EAFP supporters define immediately as
immediately after a problem arises. No side is right or wrong, both
have weakness and strength, but python-style programs are encouraged
to use EAFP-style exception handling whenever feasible, but with the
spirit of the Zen: "Special cases aren't special enough to break the
rules, although practicality beats purity", if LBYL makes things much
easier in that case, and doing EAFP would just convolute the code,
then practicality beats purity.
> --
> Steven
Bruno Desthuilliers
> Le Thursday 21 August 2008 09:34:47 Bruno Desthuilliers, vous avez écrit :
>>> The point
>>> is that EAFP conflicts with the interest of reporting errors as soon
>>> as possible (on which much has been written see, for instance Ch. 8 -
>>> Defensive Programming in Code Complete),
>> Defensive programming makes sense in the context of a low-level language
>> like C where errors can lead to dramatic results. In high-level
>> languages like Python, the worse thing that an unhandled exception can
>> cause is an abrupt termination of the process and a nice traceback on
>> screen.
>
> ... and leave your datas in inconsistent state.
Not all applications persist data, so this is an application-specific
problem, to be solved at the application level - IOW, there's no
one-size-fits-all solution here. Anyway: transactions management is not
what I meant when talking about defensive programming.
As far as I'm concerned, the only place where the defensive approach
really makes sense whatever the language is when dealing with external
datas (user inputs etc).
> So, what C or any other
> language could do worse to your application ?
An error in a C program can do *way* worse than leave an application's
data in inconsistent state. See ART for more on this:
http://dialspace.dial.pipex.com/prod/dialspace/town/green/gfd34/art/
>> In this context, defensive programming is mostly a waste of time
>> - if you can't *correctly* handle the exception where it happens, then
>> doing nothing is the better solution.
>
> If I don't buy the argument
cf above - maybe you buy it after all ?-)
> I actually agree with the conclusion. Each
> component of a program should try to manage only errors tied to their own
> logic and let pass others up to the gui logic for rendering errors the good
> way, persistence logic to rollback unwanted changes, and application logic to
> continue execution the right way. This is hard to do in C because you have no
> way to trap an error which happen randomly in the program, ie. a segfault
> will interrupt the execution anyway.
Indeed.
Lie
42.desthuilli...@websiteburo.invalid> wrote:
> dbpoko...@gmail.com a écrit :
Ah... now I understand what the Zen is talking about when it said:
"Now is better then never, although never is often better than *right*
now." If you don't have all the necessary resources to fix an
exception right now, don't try to fix it, instead let it propagate,
and allow it to be handled in a level where there is enough
information how to fix it.
> My 2 cents...
Steven D'Aprano says:
> Exceptions can and often are anticipated. E.g. if you write code that
> opens a URL, you better anticipate that the server might reject your
> connection. You better expect to be asked for a cookie, or
> authentication. If you check for robots.txt, you better expect that it
> might not exist. That's all normal execution.
I think we should change except: into expect:, it would confuse less,
would it? It signifies that the program expects so and so kinds of
exceptional situations. The try: should also be changed to... perhaps
in:, block:, onthiscode:, etc (this paragraph is written with my taste
buds on the part of my face below my eye and above my jaw)
Wojtek Walczak
> I think we should change except: into expect:, it would confuse less,
> would it? It signifies that the program expects so and so kinds of
> exceptional situations. The try: should also be changed to... perhaps
> in:, block:, onthiscode:, etc (this paragraph is written with my taste
> buds on the part of my face below my eye and above my jaw)
IMO it's not even worth considering. Breaking all the python
software that exists and moreover breaking the habits of programmists
to gain *nothing* is a waste of time of so many people, that
we should just forget it.
--
Regards,
Wojtek Walczak,
http://tosh.pl/gminick/
Maric Michaud
> Maric Michaud a écrit :
> > Le Thursday 21 August 2008 09:34:47 Bruno Desthuilliers, vous avez écrit :
> >>> The point
> >>> is that EAFP conflicts with the interest of reporting errors as soon
> >>> as possible (on which much has been written see, for instance Ch. 8 -
> >>> Defensive Programming in Code Complete),
> >>
> >> Defensive programming makes sense in the context of a low-level language
> >> like C where errors can lead to dramatic results. In high-level
> >> languages like Python, the worse thing that an unhandled exception can
> >> cause is an abrupt termination of the process and a nice traceback on
> >> screen.
> >
> > ... and leave your datas in inconsistent state.
>
> Not all applications persist data, so this is an application-specific
> problem, to be solved at the application level - IOW, there's no
> one-size-fits-all solution here.
... or lose open connection, reset sessions, etc.. it doesn't really matter
what is lost when a program crash, if it can be restarted without changes it
is hardly what I'd qualify a "dramatic result". It doesn't depend on the
language what implications of an unhandled error are, It is always
application-specific.
> Anyway: transactions management is not
> what I meant when talking about defensive programming.
>
> As far as I'm concerned, the only place where the defensive approach
> really makes sense whatever the language is when dealing with external
> datas (user inputs etc).
>
I agree, this is my whole point, "whatever the language is".
> > So, what C or any other
> > language could do worse to your application ?
>
> An error in a C program can do *way* worse than leave an application's
> data in inconsistent state. See ART for more on this:
> http://dialspace.dial.pipex.com/prod/dialspace/town/green/gfd34/art/
>
I didn't read "dramatic results" in that sense, but with the meaning of a
result that the program itself cannot handle.
If the whole system crash due to an unhandled error in a program, once
missile's dropped it doesn't really matter in which language it was written.
Reliability of a system, as high-availability of an application, is mostly a
problem beyond the scope of application level error checking.
> >> In this context, defensive programming is mostly a waste of time
> >> - if you can't *correctly* handle the exception where it happens, then
> >> doing nothing is the better solution.
> >
> > If I don't buy the argument
>
> cf above - maybe you buy it after all ?-)
>
> > I actually agree with the conclusion. Each
> > component of a program should try to manage only errors tied to their own
> > logic and let pass others up to the gui logic for rendering errors the
> > good way, persistence logic to rollback unwanted changes, and application
> > logic to continue execution the right way. This is hard to do in C
> > because you have no way to trap an error which happen randomly in the
> > program, ie. a segfault will interrupt the execution anyway.
>
> Indeed.
> --
> http://mail.python.org/mailman/listinfo/python-list
--
_____________
Maric Michaud
Gabriel Genellina
escribió:
> Bruno Desthuilliers @ Thursday 21 August 2008 17:31:
>
>> Java's "checked exception" system has proven to be a total disaster.
>
> Could you elaborate on that? I'm not disagreeing with you (or agreeing,
> for that matter); I'd just really like to know what you mean by
> a "total disaster."
Please allow me to share a few links:
http://www.mindview.net/Etc/Discussions/CheckedExceptions
http://radio.weblogs.com/0122027/stories/2003/04/01/JavasCheckedExceptionsWereAMistake.html
and see also a message from Richard Levasseur today in this very thread
explaining why checked exceptions are just an annoyance that buys you
nothing.
--
Gabriel Genellina
Delaney, Timothy (Tim)
> Ah... now I understand what the Zen is talking about when it said:
> "Now is better then never, although never is often better than *right*
> now." If you don't have all the necessary resources to fix an
> exception right now, don't try to fix it, instead let it propagate,
> and allow it to be handled in a level where there is enough
> information how to fix it.
Well, I believe the original intent was more along the lines of adding
features, etc to Python, but it's apropos here as well.
> I think we should change except: into expect:, it would confuse less,
> would it? It signifies that the program expects so and so kinds of
> exceptional situations.
Whilst the connotations are good, and I think create the right mindset,
it ain't gonna happen.
Cheers,
Tim Delaney
Bruno Desthuilliers
Hear hear...