Readings on Exceptions

[Peter Froehlich]

Hi all,

Andrew Black got his PhD with a thesis on exception handling. Or
rather, with a thesis on why exceptions suck. I still disagree (even
after writing some Go code), but it's interesting reading anyway:

http://web.cecs.pdx.edu/~black/publications/Black%20D.%20Phil%20Thesis.pdf

Maybe others have some more background reading on this whole topic?
Don't discuss, just post your favorite exception-related links. :-D

Cheers,
Peter
--
Peter H. Froehlich <http://www.cs.jhu.edu/~phf/>
Senior Lecturer | Director, Johns Hopkins Gaming Lab

[Vincent Risi]

Currently the thinking in go is that a called function can fail only
in one way and the error is being sent as a return value. I seriously
dislike the mixing of normal return values with error return values. I
think that called function should return error states but these should
be done if an error is thrown or passed through. All the possible
errors that a function returns that can be caught should be handled by
the calling function with compile errors raised if they are not
handled. The calling function could handle the error or pass it
through as is or as an execution failure, resulting in the call stack
being traversed and all defers action before terminating.

On Dec 8, 1:46 am, Peter Froehlich <peter.hans.froehl...@gmail.com>
wrote:

> Hi all,
>
> Andrew Black got his PhD with a thesis on exception handling. Or
> rather, with a thesis on why exceptions suck. I still disagree (even
> after writing some Go code), but it's interesting reading anyway:
>

> http://web.cecs.pdx.edu/~black/publications/Black%20D.%20Phil%20Thesi...

[Miek Gieben]

[ Quoting Vincent Risi in "[go-nuts] Re: Readings on Exception"... ]

> Currently the thinking in go is that a called function can fail only
> in one way and the error is being sent as a return value. I seriously
> dislike the mixing of normal return values with error return values. I

As we are now voicing personal preferences. I really like these
multiple return values. Much better than doing them inbound as in
C.

grtz Miek

[Vincent Risi]

I do not wish to remove multiple return values, I really like these,
it would be nice if they could be treated like a tuple as in python.
The called function could fail is many ways, it could have a memory
allocation failure, a memory access failure or an index out of bounds
error other than its nominal errors. Mixing normal function returns
with error returns is not great for programming. If the application is
required to be robust however some of these errors need to be dealt
with by the caller. I am just suggesting that exception handling
should not be interspersed with normal program function, but should be
dealt with cleanly. The current if erc == true { return erc; } model
that was prevalent in C should be discouraged. Having error handling
as library without compiler support is also not acceptable.

>  signature.asc
> < 1KViewDownload

[Jonathan Amsterdam]

So much for saying "don't discuss" in a discussion board...

But if you folks must discuss, please make sure that you understand
that Go already supports a form of exceptions. Read the code at
http://www.google.com/url?sa=D&q=http://code.google.com/p/go/source/browse/src/pkg/exp/exception/exception.go%3Fspec%3Dsvnaff697587839774c069bbf35bc120257d4a83865%26r%3Daff697587839774c069bbf35bc120257d4a83865&usg=AFQjCNED5zny5ckMaZaVteMbnpeIn453fg.

[Jonathan Amsterdam]

If that link is broken, try this one: http://code.google.com/p/go/source/browse#hg/src/pkg/exp/exception

On Dec 8, 10:55 am, Jonathan Amsterdam <jbamster...@gmail.com> wrote:
> So much for saying "don't discuss" in a discussion board...
>
> But if you folks must discuss, please make sure that you understand

> that Go already supports a form of exceptions. Read the code athttp://www.google.com/url?sa=D&q=http://code.google.com/p/go/source/b....

[Jonathan Amsterdam]

> http://web.cecs.pdx.edu/~black/publications/Black%20D.%20Phil%20Thesi...

I'm also going to break the "don't discuss" rule, but at least I'll be
talking about the paper.

I haven't finished it yet, but I think the guy's blind spot is
summarized nicely by this quotation from p. 137 (PDF p. 145):

Horning's own response was t o admit t h a t it is very d i f f i c u
l t t o give a
convincing s p e c i f i c example t h a t demonstrates the u t i l i
t y of s i g n a l s . He
a t t r i b u t e d t h i s t o the f a c t t h a t s i g n a l s a r
e a mechanism f o r the control of
complexity. In simple examples there is not enough complexity, and
more
complex systems are too big t o be good examples. This seems t o be a
weak
argument. Procedures and a b s t r a c t data types both e x i s t t o
control
complexity: the l i t e r a t u r e is nevertheless f u l l of simple
examples i l l u s t r a t i n g
t h e i r usefulness. My own f e e l i n g is t h a t a f a c i l i t
y w i l l be of l i t t l e use
i n a r e a l system if it is not even of use in a specially
constructed example.

(Sorry for the gap-toothed look, that's what Adobe copy 'n' paste
gives me.) The utility of exceptions comes in when there's long-
distance control flow. If there isn't, then the approach he advocates,
returning discriminated unions, is probably better. If he's not
willing to look at real examples, then his conclusion is foregone.

I should also point out that most of his attacks refer to styles of
exception use that would be considered poor today (e.g. exceptions as
alternate ways of returning values, like EOF). His criticisms of those
techniques are dead on.

[Ben Tilly]

On Tue, Dec 8, 2009 at 11:22 AM, Jonathan Amsterdam
<jbams...@gmail.com> wrote:
>> http://web.cecs.pdx.edu/~black/publications/Black%20D.%20Phil%20Thesi...
>
> I'm also going to break the "don't discuss" rule, but at least I'll be
> talking about the paper.
>
> I haven't finished it yet, but I think the guy's blind spot is
> summarized nicely by this quotation from p. 137 (PDF p. 145):
>

> Horning's own response was to admint that it is very
> difficult to give a convincing specific example that
> demonstrates the utility of signals. He attributed
> this to the fact that signals are a mechanism for the

> control of complexity. In simple examples there is
> not enough complexity, and more complex systems are

> too big to be good example. This seems to be a weak

> argument. Procedures and abstract data types both
> exist to control complexity: the literature is
> nevertheless full of simple examples illustrating
> their usefulness. My own feeling is that a facility

> will be of little use in a real system if it is not

> even of use in a specially constructed example.
>
> (Sorry for the gap-toothed look, that's what Adobe copy 'n' paste
> gives me.) The utility of exceptions comes in when there's long-
> distance control flow. If there isn't, then the approach he advocates,
> returning discriminated unions, is probably better. If he's not
> willing to look at real examples, then his conclusion is foregone.

I fixed the gap toothed look by retyping it...

I agree that this is a critical point. My experience is that
exceptions can be a great feature in the right environment.

For example if you encounter a serious error while generating a web
page, what you really want to do is write out an informative
explanation of what went wrong, abandon creating the web page, and
then move on to the next. Then you really want people to look at the
error logs, see the errors that come up, and fix the underlying bugs.
Make your errors big, fat, and quickly fixed.

Doing anything else, such as trying to recover from the error, is
generally a bad idea. Attempted cleverness can make a minor problem
much more serious. It is very easy to wind up with a logic mistake
that causes your program to go into an endless loop, possibly with a
fast memory leak or disk. (I just fixed a bug someone else wrote
which would cause the wrong URL to fill up the hard drive by endlessly
spewing to the error log.) People are tolerant enough of errors on
web pages that it really makes sense to just let them see an internal
server error. (Many environments will echo the error to the browser.
That is fine for a dev environment, but doing that in production lets
an attacker debug any attacks they discover against you.)

In other areas of programming, exceptions make much less sense. Does
anyone really want to see their desktop computer spew a bunch of
debugging information and wait for you to attach a debugger you don't
have access to to it? (Useless trivia, Windows computers do not
technically crash when they show a BSOD.) Absolutely nothing
desirable can be done with an exception, so you shouldn't throw one.

Cheers,
Ben

[Joubin Houshyar]

On Dec 8, 4:35 am, Vincent Risi <vincent.r...@gmail.com> wrote:
> Currently the thinking in go is that a called function can fail only
> in one way and the error is being sent as a return value.

I'm not sure why you think this is the case.

Given

func foo (...) (R1, R2, ...., Rn, E) {
}

where type E supports os.Error, and could have additional methods to
report fault category, type, beyond the general basic message, you
would test for !=nil, and then branch on the available
differentiators.

You could ignore the returned E using _ (just as you could use a no op
try/catch block), and testing the error differentiators would entail
unsightly test and branch blocks (just as differentiating between
thrown exceptions in fine grained catch blocks).

The only thing that is functionally missing is the transparent
propagation of thrown exceptions. Perhaps introducing a special
marker (similar to _) would tell the compiler to autogenerate the code
that would test for non nil returned variable and (if the error type
matches a return parameter) to propagate up the stack.

// just a sketch - assume Exception is a core interface type known to
the compiler)

func caller (/* ... params */) (X1, X2, ..., Exception) {

// compiler will not complain about the missing receiver
// for the last item on return list. Similar to _, / means return
value
// is ignored here, but, we want the compiler to insert the test
and return error
// if it is not.
a, b, c, ..., / := foobar (.....);

//
}

func foobar (/* ... params */) (R1, R2, ...., Exception) {
..
// fatal error - return Exception
return newException ("foobar exception", category, type, ...);
}

/R

[Vincent Risi]

But you are returning an E. Let us say

func Called() int {
catch MemoryError { throw }
... error case
e := newE();
throw e;
...
return 1234;
}

func Caller() {
catch error *E {
// code to handle error - say it works like a defer
}
catch error2 *MemoryError { fail }
no := Called(); // better still I could use the result as a
parameter value
}

In this example the catches could be treated as function literals.

The important thing is the error cases are different from the normal
return cases and they should be treated differently not mixed.

The actual underlying code the compiler generates could package them
together as a call state and return values but the programmer does not
have to give use two part logic to deal with normal application and
the exception case.

[atomly]

On Tue, Dec 8, 2009 at 3:59 PM, Joubin Houshyar <sun...@gmail.com> wrote:
> You could ignore the returned E using _ (just as you could use a no op
> try/catch block), and testing the error differentiators would entail
> unsightly test and branch blocks (just as differentiating between
> thrown exceptions in fine grained catch blocks).
>
> The only thing that is functionally missing is the transparent
> propagation of thrown exceptions.  Perhaps introducing a special
> marker (similar to _) would tell the compiler to autogenerate the code
> that would test for non nil returned variable and (if the error type
> matches a return parameter) to propagate up the stack.

This is very similar to something I proposed on another thread
yesterday and I still think it's an OK idea. I find it analogous to
the optional comma ok syntax for maps, channels, etc... The ability
to define an optional exception return type that, if checked, is
treated just like a return type but, if omitted, creates a runtime
error. The other thing I would want to see is the ability to catch
this somewhere up the stack (perhaps with a catch() language builtin),
much like I'd like to be able to catch runtime errors in my main() so
that somebody else's library trying to get a non-existent key from a
map doesn't halt my entire application.

--
:: atomly ::

[ ato...@atomly.com : www.atomly.com : http://blog.atomly.com/ ...
[ atomiq records : new york city : +1.917.442.9450 ...
[ e-mail atomly-new...@atomly.com for atomly info and updates ...

[Joubin Houshyar]

On Dec 8, 4:33 pm, Vincent Risi <vincent.r...@gmail.com> wrote:
> But you are returning an E. Let us say
>
> func Called() int {
>   catch MemoryError { throw }
>   ... error case
>   e := newE();
>   throw e;
>   ...
>   return 1234;
>
> }

Right, an exception.

I was just addressing the statement that "Currently the thinking in go
is that a called function can fail only in one way" which is not the
case. I think the current thinking in Go is that we have enough built
in features to aggregate to achieve quite a lot of "missing" semantics
(generics notwithstanding).

[Joubin Houshyar]

On Dec 8, 4:34 pm, atomly <ato...@atomly.com> wrote:
> On Tue, Dec 8, 2009 at 3:59 PM, Joubin Houshyar <suno...@gmail.com> wrote:
> [snip]

> This is very similar to something I proposed on another thread
> yesterday and I still think it's an OK idea.

(Missed it. Can you post a link?)

I'm not sure what I think about catch [I missed that ;)] beyond the
receive on call site.

There is a certain (perhaps illusory) sense of clarity to Go code (in
contrast to Java) that I like. And there is also the annoyance of
typing the same thing again and again (if e != nil { return e}) which
I do not. After all, even a catch type behavior can be (tediously)
achieved with goto and 'exceptions' label. (I vaguely recall seeing
something to that effect in the net package sources.)

/R

[atomly]

On Tue, Dec 8, 2009 at 4:59 PM, Joubin Houshyar <sun...@gmail.com> wrote:
> (Missed it.  Can you post a link?)

http://groups.google.com/group/golang-nuts/browse_thread/thread/2979b968005d5714

[mythz]

I'm really surprised there are still continued discussions on whether
Exceptions are better than return error codes?
It sounds like some people trying to make excuses for Go's immaturity
and lack of this fundamental feature and somehow turn it into a good
thing.

It isn't, lack of exceptions pollute the API and can have dangerous
consequences as any time you fail to check for an error and one
happens
your program continues down an un-trusted path basing logic on invalid
responses you failed to check for.

Error codes put the burden on the developer to write boiler plate code
to 'opt-in' and cater for the 'error path' every time.
Whilst with Exceptions you only 'opt-in' and cater for the 'error
path' if you know how to deal with it, otherwise it bubbles up.

Exceptions are for exceptional circumstances, i.e. when code execution
doesn't go as planned.
If you failed to cater for this execution path than the default
behaviour should be to stop the execution of the program.

Obviously, the longer Go continues without exceptions the harder it
will be to add them since most of the API will be bounded to error
codes.
So the future solution (if any) might have to be syntactic sugar that
looks like 'Exceptions' but work with error codes underneath, i.e.

try
{
var result = 0;
result = applyRule1(result);
result = applyRule2(result);
result = applyRule3(result);
return result;
}
catch (err lang.Error)
{
fmt.Fprintf(os.Stderr, "%s\n", err);
os.Exit(2);
}

//could be rewritten by the compiler to be something like:
var result = 0;
result, err = applyRule1(result);
if err != nil { goto ErrorHandler }

result, err = applyRule2(result);
if err != nil { goto ErrorHandler }

result, err = applyRule3(result);
if err != nil { goto ErrorHandler }

return result;

ErrorHandler :
fmt.Fprintf(os.Stderr, "%s\n", err);
os.Exit(2);



Basically, it would be nice if the compiler could provide syntactic
sugar for a try/catch/finally construct around the existing error
codes.
So to extend the above example further it would be nice if 'func a()'
below could be rewritten by the compiler.

func main()
{
try
{
result := a(1) + a(2) + a(3);
}
catch (err lang.Error)
{
fmt.Fprintf(os.Stderr, "%s\n", err);
os.Exit(2);
}
}

func a(i int) int
{
return b(i) * 2;
}

func b(i int) (result int, err lang.Error)
{
.. //impl that returns an error.
}

//could be rewritten by the compiler to look something like:
func main()
{
var a1, a2, a3 int;

a1, err = a(1);
if err != nil { goto ErrorHandler }

a2, err = a(2);
if err != nil { goto ErrorHandler }

a3, err = a(3);
if err != nil { goto ErrorHandler }

result = a1 + a2 + a3;

ErrorHandler:
fmt.Fprintf(os.Stderr, "%s\n", err);
os.Exit(2);
}

func a(i int) (result int, err lang.Error)
{
result, err := b(i);
if err !=nil { goto ErrorHandler }

return r * 2;

ErrorHandler:
return result, err;
}

func b(i int) (result int, err lang.Error)
{
.. //impl that returns an error.
}


Anyway the idea above is for the compiler to try ease the burden for
the programmer from having to write boilerplate
error handling code while still maintaining clarity as to the expected
behaviour of the source code.

[atomly]

Well put. I guess the big thing I keep trying to come back to is that
I want exceptions for _unexpected_ errors-- exceptional cases. Go's
error handling works OK for expected errors, but when working on
large, distributed systems, anything that can fail, will, and I don't
want to have to account for that in every single function call and
return statement in my entire system. I want to separate concerns and
have error handlers that know what to do when the FS, network, DB,
etc. fail, for example...

Your exception code reminded me of something strange I saw, so I
looked it up... As an example of strange error handling, take a look
at:

http://github.com/dustin/gomemcached/blob/master/mc_conn_handler.go#L72

[ollave]

On Dec 9, 12:32 pm, mythz <demis.bel...@gmail.com> wrote:
> I'm really surprised there are still continued discussions on whether
> Exceptions are better than return error codes?
> It sounds like some people trying to make excuses for Go's immaturity
> and lack of this fundamental feature and somehow turn it into a good
> thing.

Some of us believe it is a good thing.

> It isn't, lack of exceptions pollute the API and can have dangerous
> consequences as any time you fail to check for an error and one
> happens
> your program continues down an un-trusted path basing logic on invalid
> responses you failed to check for.

And having exceptions in a language positively encourages programmers
to ignore error codes at the point at which they are recoverable, and
defer their handling to higher level code which often:

o doesn't know as much about the situation that provoked the error as
is
desirable for a quality diagnostic message, much less to attempt
recovery

o invites programs to continue anyway, having leaked who-knows-what
resources (memory not such a problem with garbage collection, but
open files, database connections, temporary files, etc still
matter)

Equally bad (at least equally bad, possibly worse) is that for every
package function that my code calls, I have to know not just what
error
returns it may return, but what exceptions it may throw, and write
code to handle both.

And if (like Java) you allow "runtime exceptions" which don't have to
be declared, then there's no way to tell from the documentation what
exceptions your code might have to handle.

Calling os.Exit() is probably as useful as throwing an exception in
most cases. If you don't know how to handle an error, the chance
of someone else knowing higher up the code isn't good. Bearing
in mind all those resource leak possibilities. :-(

> Error codes put the burden on the developer to write boiler plate code
> to 'opt-in' and cater for the 'error path' every time.

Since you have to think about the error path, you might as well do
some typing at the same time.

> Whilst with Exceptions you only 'opt-in' and cater for the 'error
> path' if you know how to deal with it, otherwise it bubbles up.

Making it "someone else's" problem, but not solving it.

> Exceptions are for exceptional circumstances, i.e. when code execution
> doesn't go as planned.
> If you failed to cater for this execution path than the default
> behaviour should be to stop the execution of the program.

Yup. Hence my suggestion of os.Exit() above.

> Anyway the idea above is for the compiler to try ease the burden for
> the programmer from having to write boilerplate
> error handling code while still maintaining clarity as to the expected
> behaviour of the source code.

I'm skeptical that that is achievable. I grant it would be nice, but
the
quality of the error handling in code I've worked on has generally
been a good guide to the robustness of the product.

Possibly I've done too much kernel programming without support
for exceptions, and have had too much bad experience with poor
quality application programmers using C++ who were using
exceptions, but I'm far from convinced they're a good thing to
have.

[Vincent Risi]

Returning os.Error as the last parameter in the return statement
denies the fact that we could have multiple types of errors to deal
with. Handling of errors is not so much important as also helping the
maintenance programmer that has to deal with the post mortem of the
programs demise. os.Error does not offer enough information to assist
in these events. Kind of like the stupid errors we get in Windows like
"DLL Not Found" without the pertinent information as to why the
offendong DLL was not found, e.g. because the manifest of the
underlying runtime that you are employing has changed with a service
pack. As bad as exceptions may be the concept of an error code with a
description is far worse. Programmers have to have enough information
to be able to solve issues and the application programs that we are
now writing are huge behemoths by comparison to the one we wrote 10
years ago.

[Ben Tilly]

On Tue, Dec 8, 2009 at 7:38 PM, ollave <giles...@pobox.com> wrote:
> On Dec 9, 12:32 pm, mythz <demis.bel...@gmail.com> wrote:

[...]

> Calling os.Exit() is probably as useful as throwing an exception in
> most cases.  If you don't know how to handle an error, the chance
> of someone else knowing higher up the code isn't good.  Bearing
> in mind all those resource leak possibilities. :-(

Be careful of generalizing from experience with one area to other
areas. A lot of my experience is with web applications. In web
development throwing an exception is frequently useful, but os.Exit is
generally a Bad Idea. That provides a lot of practical cases where
os.Exit is the wrong thing to do.

>> Error codes put the burden on the developer to write boiler plate code
>> to 'opt-in' and cater for the 'error path' every time.
>
> Since you have to think about the error path, you might as well do
> some typing at the same time.

You shouldn't think about the error path in web development. It
should be a reflex. If something goes wrong, throw an exception.
You'll catch it at the top level. That handler will log a stack
backtrace, send off an internal server error, then move on to your
next request.

Then because you don't like showing crap to end users you make sure to
monitor your error log, fix the bugs that result in internal server
errors, and make sure that you don't wind up on that error path for
any problem within your control. (If your database goes down you may
have no reasonable choice...)

The critical fact about web development that makes it so is that it is
generally not a big deal if you abandon the current web page, but it
is usually a bad idea to take down the current process.

>> Whilst with Exceptions you only 'opt-in' and cater for the 'error
>> path' if you know how to deal with it, otherwise it bubbles up.
>
> Making it "someone else's" problem, but not solving it.

In many application spaces I agree that making it someone else's
problem is a useless thing to do. But not in web programming.
Because in web programming it is clear what someone else needs to do
about it. Namely log a message, return a 500 internal server error,
and let the server move on to the next request.

>> Exceptions are for exceptional circumstances, i.e. when code execution
>> doesn't go as planned.
>> If you failed to cater for this execution path than the default
>> behaviour should be to stop the execution of the program.
>
> Yup.  Hence my suggestion of os.Exit() above.

Try that in a multi-threaded web application and you wind up without a
webserver.

It isn't as bad in a pre-fork webserver (Apache is often set up this
way). But even then you'll cause a significant problem on your
website to cause cascading problems as new web processes get spawned,
grab database connections, go down unexpectedly, and leave the
database holding empty connections. Each connection takes database
resources, and once you've tied up enough of them your database goes
down. Been there, done that, it ain't pretty.

In both cases terminating the process needs to be avoided because that
causes collateral damage that makes an existing problem worse.

>> Anyway the idea above is for the compiler to try ease the burden for
>> the programmer from having to write boilerplate
>> error handling code while still maintaining clarity as to the expected
>> behaviour of the source code.
>
> I'm skeptical that that is achievable.  I grant it would be nice, but
> the
> quality of the error handling in code I've worked on has generally
> been a good guide to the robustness of the product.

My opinion is that the truth of your statement depends very strongly
on what application area you are in. As an extreme example, the best
web code that I've worked with doesn't even TRY to handle errors
carefully. Effort is put into avoiding errors in the first place, but
errors get sent to a top level error handler and that is that.

> Possibly I've done too much kernel programming without support
> for exceptions, and have had too much bad experience with poor
> quality application programmers using C++ who were using
> exceptions, but I'm far from convinced they're a good thing to
> have.

As I'm trying to make clear, this strongly depends on your application
area. For example let's compare kernel programming, desktop
applications, batch processing and web development. I believe that
exceptions are a horrible idea in the first two, are moderately good
for batch processing, and are an excellent idea within complex web
applications. Here is why I believe that.

- In kernel programming throwing an exception you can't immediately
resolve gives you no good options other than crashing the OS. Don't
do that.

- In a desktop application an exception has nothing particularly
useful to do beyond shutdown the application unless it has been very
carefully segmented internally. (Even then segmenting along process
boundaries makes sense - see Google's Chrome for an example.) So I'd
be inclined to avoid exceptions there, though I can see how it could
be used in the right case.

- I personally have good experience using exceptions with batch
processing. The idea is that on an exceptional event you want to
alert a human, save off context, and then (depending on what the batch
process does) either move on to another piece of work or else halt.
Well chosen exception handlers make this easy to set up. Yes, you
could use goto for error handling in batch processing. But I've found
that not as pleasant. Particularly when you have libraries that are
used in multiple utilities that want to do different things after
errors. For instance if you can't execute a query in one utility you
want to kill the current process, while in another you want to wind up
moving on to the next job. Exception handlers make this easy to set
up.

- I've already discussed web applications in too much detail.

Cheers,
Ben

PS I can't believe that I just spent so long explaining why exceptions
can be a good thing, depending on application area, when I'm not
particularly bothered by the omission of them from Go. My attitude is
that Go is aiming for application areas where exceptions aren't a
particularly good idea, and I see how exceptions would cause issues
within the language. Therefore I'm OK with their omission.

[Russ Cox]

> Returning os.Error as the last parameter in the return statement
> denies the fact that we could have multiple types of errors to deal
> with. Handling of errors is not so much important as also helping the
> maintenance programmer that has to deal with the post mortem of the
> programs demise. os.Error does not offer enough information to assist
> in these events. Kind of like the stupid errors we get in Windows like
> "DLL Not Found" without the pertinent information as to why the
> offendong DLL was not found, e.g. because the manifest of the
> underlying runtime that you are employing has changed with a service
> pack. As bad as exceptions may be the concept of an error code with a
> description is far worse. Programmers have to have enough information
> to be able to solve issues and the application programs that we are
> now writing are huge behemoths by comparison to the one we wrote 10
> years ago.

Everything you said is correct except the part about os.Error.

os.Error can contain arbitrarily rich information about what happened,
the same way an exception object can in other languages.
You can use a type guard or a type switch to find out what kind
of os.Error it is.

http://golang.org/doc/effective_go.html#errors

Russ

[atomly]

On Wed, Dec 9, 2009 at 2:10 AM, Ben Tilly <bti...@gmail.com> wrote:
> Be careful of generalizing from experience with one area to other
> areas.  A lot of my experience is with web applications.  In web
> development throwing an exception is frequently useful, but os.Exit is
> generally a Bad Idea.  That provides a lot of practical cases where
> os.Exit is the wrong thing to do.

I would actually generalize it to say that in that space, it's never
the right thing to do. Most webapp code I've seen, in fact, only
really ever catches Exceptions in order to rollback database
transactions and the like. Otherwise, you want it to find its way up
the stack to the proper handler so that, as you said, the app can move

on to the next request.

I can think of a LOT of other situations where this is the case as
well-- pretty much any network service fits the bill. I definitely
don't want my database server, distributed cache, etc. to os.Exit
because there was a minor (possibly unforeseeable) error in one
request thread.

[Ben Tilly]

On Wed, Dec 9, 2009 at 12:14 AM, atomly <ato...@atomly.com> wrote:
> On Wed, Dec 9, 2009 at 2:10 AM, Ben Tilly <bti...@gmail.com> wrote:
>> Be careful of generalizing from experience with one area to other
>> areas.  A lot of my experience is with web applications.  In web
>> development throwing an exception is frequently useful, but os.Exit is
>> generally a Bad Idea.  That provides a lot of practical cases where
>> os.Exit is the wrong thing to do.
>
> I would actually generalize it to say that in that space, it's never
> the right thing to do.  Most webapp code I've seen, in fact, only
> really ever catches Exceptions in order to rollback database
> transactions and the like.  Otherwise, you want it to find its way up
> the stack to the proper handler so that, as you said, the app can move
> on to the next request.

[...]

Never say never. :-)

With a pre-fork webserver it can be a good idea to set soft and hard
ulimits on CPU and memory usage. Before accepting a new request you
check the soft limits and exit gracefully if necessary. And you do a
hard exit when you pass the hard ulimits.

This strategy prevents poorly designed code from spinning out of
control and tying up resources with potentially disastrous effect.

Cheers,
Ben

[Joubin Houshyar]

On Dec 9, 2:06 am, Vincent Risi <vincent.r...@gmail.com> wrote:
> Returning os.Error as the last parameter in the return statement [...]

"Given
func foo (...) (R1, R2, ...., Rn, E) {
}

where type E supports os.Error"

Supports, not "is".
Type "E", no "os.Error".

/R

[ollave]

On Dec 9, 6:10 pm, Ben Tilly <bti...@gmail.com> wrote:

> You shouldn't think about the error path in web development.  It
> should be a reflex.  If something goes wrong, throw an exception.
> You'll catch it at the top level.  That handler will log a stack
> backtrace, send off an internal server error, then move on to your
> next request.

Web applications are an instance where I've used exceptions, I
confess. :-)

There's still the problem of cleaning up allocated resources; this
becomes non-trivial when attempting to abandon any single
thread in a multithreaded process ... and having someone
*else's* buggy code tie up all the database connections is,
um, irritating.

Obviously I agree that calling os.Exit() isn't a decision to take
lightly when you've multiple threads in play, and I didn't state
that in my previous post, but should have. Early exit of a
goroutine would work sometimes ... except when you're unsure
about possibly allocated and soon-to-be-leaked resources. :-(

> - I personally have good experience using exceptions with batch
> processing.  The idea is that on an exceptional event you want to
> alert a human, save off context, and then (depending on what the batch
> process does) either move on to another piece of work or else halt.
> Well chosen exception handlers make this easy to set up.

So does an application monitor which restarts the application.
I thought everyone set things up that way, just like they put
automatic timeout routines in web code to prevent any
possibility of an infinite loop....

> PS I can't believe that I just spent so long explaining why exceptions
> can be a good thing, depending on application area, when I'm not
> particularly bothered by the omission of them from Go.  My attitude is
> that Go is aiming for application areas where exceptions aren't a
> particularly good idea, and I see how exceptions would cause issues
> within the language.  Therefore I'm OK with their omission.

I still think there are other ways to skin the particular cats that
exceptions can help with, and they're a temptation to abuse that
is better left out of the language.

Obviously I too am OK with go's (current) omission of exceptions. :-)

Thanks for your well reasoned and written post.

[Ben Tilly]

On Wed, Dec 9, 2009 at 1:44 AM, ollave <giles...@pobox.com> wrote:
> On Dec 9, 6:10 pm, Ben Tilly <bti...@gmail.com> wrote:
>
>> You shouldn't think about the error path in web development.  It
>> should be a reflex.  If something goes wrong, throw an exception.
>> You'll catch it at the top level.  That handler will log a stack
>> backtrace, send off an internal server error, then move on to your
>> next request.
>
> Web applications are an instance where I've used exceptions, I
> confess. :-)
>
> There's still the problem of cleaning up allocated resources; this
> becomes non-trivial when attempting to abandon any single
> thread in a multithreaded process ... and having someone
> *else's* buggy code tie up all the database connections is,
> um, irritating.

This is one of the reasons why I like the pre-fork model. Resources
are per process, and so it is much easier to keep track of the
allocation/deallocation thereof. (In general you want each process to
have one of everything, so you don't need to worry about it.) The one
detail that needs to be taken care of is keeping a slow child process
from consuming valuable resources, but a reverse proxy in httpd
accelerator mode takes care of that.

This design is rare in the Java world because everyone wants to wield
the thread sledgehammer. But every sane high-performance mod_perl
application uses this design.

> Obviously I agree that calling os.Exit() isn't a decision to take
> lightly when you've multiple threads in play, and I didn't state
> that in my previous post, but should have.  Early exit of a
> goroutine would work sometimes ... except when you're unsure
> about possibly allocated and soon-to-be-leaked resources. :-(

Whether early exit works depends strongly on application design. It
is more likely to be a problem with highly concurrent code. And Go,
of course, is intended for writing highly concurrent code.

>> - I personally have good experience using exceptions with batch
>> processing.  The idea is that on an exceptional event you want to
>> alert a human, save off context, and then (depending on what the batch
>> process does) either move on to another piece of work or else halt.
>> Well chosen exception handlers make this easy to set up.
>
> So does an application monitor which restarts the application.
> I thought everyone set things up that way, just like they put
> automatic timeout routines in web code to prevent any
> possibility of an infinite loop....

I'm not sure what good an application monitor that just restarts
things would be. My experience says that when a batch process fails
it fails because the case under consideration won't work no matter how
many times it is run, or because a critical resource (eg the machine
it is supposed to put things on) is not available. Neither type of
condition is going to work if you just try it again. That is why you
need a human to become involved.

>> PS I can't believe that I just spent so long explaining why exceptions
>> can be a good thing, depending on application area, when I'm not
>> particularly bothered by the omission of them from Go.  My attitude is
>> that Go is aiming for application areas where exceptions aren't a
>> particularly good idea, and I see how exceptions would cause issues
>> within the language.  Therefore I'm OK with their omission.
>
> I still think there are other ways to skin the particular cats that
> exceptions can help with, and they're a temptation to abuse that
> is better left out of the language.

There are other approaches that can be taken. But none is nearly as
simple and reliable for the cases where exceptions are a good fit.

> Obviously I too am OK with go's (current) omission of exceptions. :-)
>
> Thanks for your well reasoned and written post.

You're welcome.

Cheers,
Ben

[Mark Chu-Carroll]

>I'm really surprised there are still continued discussions on whether 
>Exceptions are better than return error codes? 
>It sounds like some people trying to make excuses for Go's immaturity 
>and lack of this fundamental feature and somehow turn it into a good 
>thing. 

>

>It isn't, lack of exceptions pollute the API and can have dangerous 

>consequences as any time you fail to check for an error and one 

>happens your program continues down an un-trusted path basing logic on 

>invalid responses you failed to check for. 

I don't think the issue is nearly so cut-and-dried.

I'm not exactly overwhelmed with joy at the return-value method

of checking for errors that's the current idiom in Go. But I'm

not sure that I really want go to go down the catch/throw route.

My experience is that the conventional exception handling mechanism

is vastly oversold, and frequently leads to sloppy code that does a terrible

job of actually handling errors, and which has a non-trivial impact on the

complexity of the compliler, the runtime, and the performance of generated

code.

Exceptions mean that you need to have an unwindable stack. Not just

a stack that can be printed out as an error, but a stack that can be unwound

precisely enough that you can resume execution ant any stack frame that

you want. That's a burden on the implementation.

But what's worse is the sloppiness that exception handling frequently

inspires. I can't tell you how much time I've wasted working on things where

someone basically just assumed that if they threw an exception, they'd

taken care of the error. I've seen memory leaks, deadlocks, data loss,

and data corruption all occur as a result of someone saying "Oh, an error.

I'll throw an exception, and whoever handles it will take care of the cleanup." -

and never considering that after when the stack unwinds, the catcher won't

have the necessary context to do the cleanup.

Conversely, I've dealt with tons of errors caused by someone who 

catches an exception without doing the necessary cleanup - they

assume that the thrower cleaned up their own context, and that

they're only responsible for resuming after the error. 

The catch/throw model creates a very wide disconnect between the error,

and the point where the error is handled. There's a lot of stuff in between

those two points. But when they're so disconnected, it's easy for things to

get lost in the gap.

The Go multiple value/result code hack is certainly ugly in a lot of ways - but

it forces programmers to actually take some responsibility for their errors.

They can't just write code assuming that any errors will turn into exceptions

that will be caught and handled elsewhere. They need to consider failure

when and where it can happen, and act appropriately. 

I can't say that I think that's automatically a bad thing. It's ugly, and it

frequently seems really awkward when I'm writing code. But I can't say

that I'm convinced it's actually worse than typical exception handlers.

>Error codes put the burden on the developer to write boiler plate code 

>to 'opt-in' and cater for the 'error path' every time. 

>Whilst with Exceptions you only 'opt-in' and cater for the 'error 

>path' if you know how to deal with it, otherwise it bubbles up. 

You say that as if it's a good thing. I'm not convinced that it is.

      -Mark

--
Mark Craig Chu-Carroll  
*** Software Engineer@Google, Software Tools/Programming Language/Math Geek
*** Email: mar...@gmail.com
*** Blog: http://scienceblogs.com/goodmath

[SnakE]

2009/12/9 Mark Chu-Carroll <mar...@gmail.com>

The Go multiple value/result code hack is certainly ugly in a lot of ways - but

it forces programmers to actually take some responsibility for their errors.

They can't just write code assuming that any errors will turn into exceptions

that will be caught and handled elsewhere. They need to consider failure

when and where it can happen, and act appropriately. 

I can't say that I think that's automatically a bad thing. It's ugly, and it

frequently seems really awkward when I'm writing code. But I can't say

that I'm convinced it's actually worse than typical exception handlers

One must always write error-safe code.  Exceptions do not make this responsibility vanish.  But I believe exceptions make it easier to write error-safe (exception-safe) code because you only spend time on it when it's truly necessary.

There are 3 typical reasons to handle an exception:
1. perform cleanup
2. add context
3. recover

With error returns you still have all those but get another one:
4. pass through

which adds code and therefore bugs.

I can understand that for some people error returns are preferable because they're in your face and harder to forget about.  But if you naturally care about the error path as much as you do about the main one then exceptions help you save complexity.

[Ben Tilly]

On Wed, Dec 9, 2009 at 11:15 AM, SnakE <snake...@gmail.com> wrote:
> 2009/12/9 Mark Chu-Carroll <mar...@gmail.com>
>>
>> The Go multiple value/result code hack is certainly ugly in a lot of ways
>> - but
>> it forces programmers to actually take some responsibility for their
>> errors.
>> They can't just write code assuming that any errors will turn into
>> exceptions
>> that will be caught and handled elsewhere. They need to consider failure
>> when and where it can happen, and act appropriately.
>> I can't say that I think that's automatically a bad thing. It's ugly, and
>> it
>> frequently seems really awkward when I'm writing code. But I can't say
>> that I'm convinced it's actually worse than typical exception handlers
>
> One must always write error-safe code.  Exceptions do not make this
> responsibility vanish.  But I believe exceptions make it easier to write
> error-safe (exception-safe) code because you only spend time on it when it's
> truly necessary.
>
> There are 3 typical reasons to handle an exception:
> 1. perform cleanup
> 2. add context
> 3. recover
>
> With error returns you still have all those but get another one:
> 4. pass through

You're missing one that is very important in Go, but is less common in
other languages.

5. make the exception go up the flow of control where that ISN'T the call stack.

The fact that in Go the call stack is often not the correct flow of
control means you're going to need to add a lot of explicit error
handling passed between goroutines. People who automatically reach
for exceptions tend to think "that's taken care of" and fail to think
those details through.

> which adds code and therefore bugs.

This is an important issue in cases where exceptions fit. The code
added with explicit error handling is seldom exercised and therefore
tends to be significantly more buggy than normal code. It is often
also difficult to write unit tests for. Any quality problems in your
overall development process therefore tend to show up in that part of
the code.

> I can understand that for some people error returns are preferable because
> they're in your face and harder to forget about.  But if you naturally care
> about the error path as much as you do about the main one then exceptions
> help you save complexity.

Exceptions only help you save complexity if two conditions are met.

1. The flow of control is represented by the call stack.

2. There is a reasonable generic thing that is useful to do when
things go wrong.

The truth of #2 depends strongly on your application area. But #1 is
unlikely to be true in interesting Go programs. I would go farther
than that and say that if #1 is true about a given program, then it
would probably make more sense to write that program in a more mature
language.

Cheers,
Ben

[SnakE]

2009/12/9 Ben Tilly <bti...@gmail.com>

You're missing one that is very important in Go, but is less common in
other languages.

5. make the exception go up the flow of control where that ISN'T the call stack.

The fact that in Go the call stack is often not the correct flow of
control means you're going to need to add a lot of explicit error
handling passed between goroutines.  People who automatically reach
for exceptions tend to think "that's taken care of" and fail to think
those details through.

I'm not missing it.  I think that exception handling mechanism in Go should take care of this as automatically as possible.  But existing stack-based exception handling schemes do not fit the bill.  Something significantly different is required.  If not this, I believe exceptions would have been implemented already.

[mythz]

Ok looks like there are tons of different misconceptions about
Exceptions which I believe .

>>And having exceptions in a language positively encourages programmers
>>to ignore error codes at the point at which they are recoverable,

Again, exceptions are for exceptional circumstances, usually this
means they are not recoverable.
What do you do if your network dies, you run out of memory or disk
space, etc?
You can't continue what you are doing if you need these resources, so
you stop processing that particular task
log it providing the best diagnostics and then provide the appropriate
alerts/notifications.
You should not have to write error handling code for every sub task,
db, network or io call within that task.
Once around the root of the task should suffice.

Your success path should not include relying on Exceptions being
thrown, as someone pointed out ideally you should
program defensively before, i.e. it doing the necessary checks and
assertions before making the call so an exception doesn't get thrown
at all.
Very rarely (i.e. I can't remember any) have I been able to gracefully
recover from an error and continue what I'm doing.
Usually I'll be rolling back the transaction and freeing resources and
logging the fact that it the task couldn't be completed.

>> defer their handling to higher level code which often:
>> o doesn't know as much about the situation that provoked the error as is
>> desirable for a quality diagnostic message, much less to attempt
>> recovery

Again the root task does not need to recover or know what exactly went
wrong,
it just needs to know what to do if an error was thrown and the task
didn't complete.
i.e. roll back, apply compensatory logic if necessary, record the
diagnostics and move onto the next work item.

Seriously, Exceptions with a StackTrace and the graph of child
InnerExceptions is sufficient alone in diagnosing a problem.
This is a lot more useful than a random error number with a 'one
liner' output that usually doesn't contain any information about the
error.

>> o invites programs to continue anyway, having leaked who-knows-what
>> resources (memory not such a problem with garbage collection, but
>> open files, database connections, temporary files, etc still matter)

Honestly I must be spoiled this problem rarely happens in C#. Almost
all access for
resources in an external process is wrapped in an IDisposable and
automatically cleaned up
thanks to the 'using{}' statement which is just a convenience wrapper
around try/finally.
Which is another example of why syntactic sugar is useful. the intent
of the code remains clear, terse, safe and robust.

> Equally bad (at least equally bad, possibly worse) is that for every
> package function that my code calls, I have to know not just what
> error
> returns it may return, but what exceptions it may throw, and write
> code to handle both.

Again you should only catch Exceptions you can handle.
If an Exception happened its usually unexpected and not recoverable
for the current operation, so you go into 'error handling mode', i.e.
rollback, release, record error.
If you want to write code to handle all exceptions use the base class
which is usually 'catch (Exception ex){}'

> And if (like Java) you allow "runtime exceptions" which don't have to
> be declared, then there's no way to tell from the documentation what
> exceptions your code might have to handle.

Funny, the documentation I'm used to reading usually tells you exactly
what it throws.

> Calling os.Exit() is probably as useful as throwing an exception in
> most cases.  If you don't know how to handle an error, the chance
> of someone else knowing higher up the code isn't good.  Bearing
> in mind all those resource leak possibilities. :-(

I agree os.Exit() is generally a bad idea. Normally it will be abandon
current task gracefully and move to the next work item.

[SnakE]

2009/12/10 inspector_jouve <kaush...@gmail.com>

I think the issue will be resolved almost automatically when we
realize the nature of a problem. Let's compare error propagation up
stack, and error propagation to parent goroutine.
1. Parent function has only 1 child function at a time. Goroutine
might have N children.
2. When child function executes, parent is idle. When child goroutine
executes, parent is busy.

3. When child function executes, state of the parent is known.  When child goroutine executes, state of the parent is arbitrary.

There's no more differences. Both get addressed by simple mechanism:
- each goroutine has one built-in channel to send errors to parent
(like "standard error")
- each goroutine has one built-in channel to receive errors from
children (like "standard input", but for errors only)

Please think of examples where this won't be enough.

Using channels for exception passing is an implementation detail.  The really important part is what happens when a goroutine receives a signal from its child that it terminated with an exception.  Is this error automatically propagated to grand parents?  I think it should.  Then should the other children of this goroutine be terminated as well?  Are there try-catch blocks?  If yes, and you receive a error from a child while in an absolutely unrelated try block, should it catch this error?  If no, do you handle children-related errors separately?  Or maybe limit catches to goroutine-scope only?

These are many difficult questions.  I don't know the answers yet.

[atomly]

This sort of situation is already very clearly solved by systems like Erlang.

In Erlang it is incredibly easy to spawn a new process (much like a
goroutine) and, in fact, even incredibly easy to do so on a remote
node, yet Erlang is the easiest system I've ever seen for building
robust, fault-tolerant services.

http://www.erlang.org/course/error_handling.html
http://www.erlang.org/doc/reference_manual/processes.html
http://www.erlang.org/doc/reference_manual/errors.html

I don't propose that Go come anywhere near the complexity of Erlang in
this regard, but the notion that it's difficult to propagate error
date to parents and children with lightweight processes (read:
goroutines) is definitely wrong.

[Ben Tilly]

On Wed, Dec 9, 2009 at 4:54 PM, inspector_jouve <kaush...@gmail.com> wrote:
> I think the issue will be resolved almost automatically when we
> realize the nature of a problem. Let's compare error propagation up
> stack, and error propagation to parent goroutine.
> 1. Parent function has only 1 child function at a time. Goroutine
> might have N children.
> 2. When child function executes, parent is idle. When child goroutine
> executes, parent is busy.
>

> There's no more differences. Both get addressed by simple mechanism:
> - each goroutine has one built-in channel to send errors to parent
> (like "standard error")
> - each goroutine has one built-in channel to receive errors from
> children (like "standard input", but for errors only)
>
> Please think of examples where this won't be enough.

Consider the standard example of a web server that uses a connection
pool of goroutines that have open database connections. There is a
problem with a database query. Your rules would propagate the
exception to the goroutine that spawned the goroutines in the
connection pool, when in fact you want the exception to go to the
goroutine that is handling the http request.

Variations on the problem in this example will occur any time you set
up your application as a set of cooperating services that are made
from long-lasting goroutines that communicate through channels.

Cheers,
Ben

[Jonathan Amsterdam]

> Consider the standard example of a web server that uses a connection
> pool of goroutines that have open database connections.  There is a
> problem with a database query.  Your rules would propagate the
> exception to the goroutine that spawned the goroutines in the
> connection pool, when in fact you want the exception to go to the
> goroutine that is handling the http request.

This is a great example for people to think about. It is easy to
handle in the existing language. In fact, it would fall out quite
naturally from the design. If your db actions are actually handled by
a separate goroutine, then your connection needs two channels: one to
write requests, one to read responses. The responses would include any
errors.

What does the goroutine that owns the raw connection do on an error?
It closes the raw DB connection, writes the error to its connection's
channel and then aborts with runtime.Goexit(). (It would also write to
a channel owned by the connection pool, so the pool would know the
goroutine had died.)

This can be simplified a bit, and builtin errors handled as well, if
goroutines had exit channels. See
http://groups.google.com/group/golang-nuts/browse_thread/thread/edc1e9897e5d33bb#.
I think that mechanism is all you need -- all other complicated
patterns of parents killing grandchildren and the like can be built
from that. I recommend you all keep your designs extremely simple. In
Erlang, for instance, everything can be done with this simple idea: if
one process requests to monitor another, then when the latter dies the
first gets an EXIT message. That's it.

[Ben Tilly]

On Thu, Dec 10, 2009 at 7:49 AM, inspector_jouve <kaush...@gmail.com> wrote:
>
>>Consider the standard example of a web server that uses a connection
>>pool of goroutines that have open database connections.
>

> This is really good example. You describe exactly the kind of
> communication that should never be allowed.

What is your realistic alternative? This is the standard solution to
connecting to a database in a multi-threaded environment. The
underlying issues are that databases like synchronous connections,
opening/closing those connections is expensive, and maintaining too
many open connections is expensive. So you need long-lasting
connections, connections can't be shared at any point, and you want to
limit how many connections you have. Connection pools are the natural
solution with those characteristics.

> You remember what problem we are trying to solve? It is this: for
> goroutines, simulate the same error handling mechanism as we have for
> exceptions in single-thread application. For this problem description
> to make any sense at all, we need to impose requirement: goroutine is
> allowed to communicate only with direct parent and direct children.
> Again, this requirement is there just to allow us to formulate a
> problem! (I assume that undefined problem cannot have a solution, by
> definition)

Who is "we"? _I_ never signed up to solve that problem. And I submit
that any proposed solution to that problem that throws away a large
part of Go's flexibility is a nonstarter.

> Now, if we agree on that, it follows that you cannot have independent
> iceland of goroutines handing connections. If you want to use
> connection, you have to go through common (grand)parent - I mean, the
> (grand)parent that spawned your process AND this pool of goroutines.
> And if it's really a grandparent, you have to go through parent.

We absolutely don't agree on that. Forcing messages to be relayed is
a significant restriction that provides no value that I can see.

> On the first glance, it looks like this grandaprent becomes a
> bottleneck. But it's not, that's the whole point.
> There's no analogy between network of N computers and network of
> goroutines. Computers exchange messages, which is serialized data,
> which consumes bandwidth, requires decoding etc. - in this case, one
> "main" computer cannot mediate all communication. But goroutines don't
> send serialized data - they send pointers to data. There's very small
> overhead introduced by this mediation.

I'm not concerned about performance overhead. Sure, I can tell my
parent to relay messages to my sibling. But that's a bunch of logic
with no gain. If I need to talk to my sibling, what is the point of
getting my parent involved?

> Does it make sense?

Sorry, but no.

> However stupid this mechanism might seem, it's the only thing I can
> think of that still makes it possible for the programmer to understand
> what's going on. Generic mechanism of Erlang demands supernatural
> intellectual abilities from a programmer - very few people have this
> kind of qualifications. I certainly don't.

If you want people to understand what is going on, it strikes me as a
really bad idea to introduce arbitrary restrictions that force people
to complicate what could be straightforward designs.

Ben

[Ian Lance Taylor]

inspector_jouve <kaush...@gmail.com> writes:

> You remember what problem we are trying to solve? It is this: for
> goroutines, simulate the same error handling mechanism as we have for
> exceptions in single-thread application. For this problem description
> to make any sense at all, we need to impose requirement: goroutine is
> allowed to communicate only with direct parent and direct children.
> Again, this requirement is there just to allow us to formulate a
> problem! (I assume that undefined problem cannot have a solution, by
> definition)

While I see what you are trying to do, I don't think that restriction
is reasonable. I do not think it makes sense to shackle the language
in support of a cross-goroutine error handling mechanism. Instead,
any cross-goroutine error handling mechanism should be expressed in
terms of channels, somehow.

Ian

[Ian Lance Taylor]

inspector_jouve <kaush...@gmail.com> writes:

> I'm thinking of this passage:

>
>>> If I need to talk to my sibling, what is the point of getting my parent involved?
>

> Let's recall how organizations work. Every Email you send to your
> colleague (especially if it requires some work to be done) has to be
> CC'd to a manager. Why? This CC is practically equivalent to
> communication via manager. Isn't it a correct analogy?

Are you suggesting that that is how your organization works? Because
mine sure doesn't work that way. Why would my manager want to waste
all his time reading e-mail between people on the team?

Ian

[jimmy frasche]

On Thu, Dec 10, 2009 at 9:27 AM, inspector_jouve <kaush...@gmail.com> wrote:
> If you find a solution to this, you will make a revolution in business
> administration.
> It all boils down to the question: what is the role of managers?
> Managers are often not qualified to make any technical decisions.
> They are often not qualified to make business decisions either.
> What is their role then?
> Here's my definition: High-paid communication hubs with added
> responsibility of exception handling.
> In organization, everything proceeds according to scenario I described
> above. If one doesn't make sense, same is for the other.

I don't see why this should be built into the language, causing
overhead for every goroutine, when it could be a library and you could
buy into the complexity when needed. I do not think Go needs
exceptions. It needs a library for a number of common error handling
schemes that can be bought-into piecemeal.

[Brian Stuart]

>>Why would my manager want to waste  all his time reading e-mail between people on the team?
>

> Because this is what he/she is paid for.
>
> Can you provide a different definition of a manager? I'm just curious
> what planet you are living in.

Apparently the same one I live on. I've worked for five companies in
my career (not counting colleges and consulting). They ranged in
sizes from 30 people to 250,000 people, from startups to muti-billion
dollar companies. I've never been expected to copy a manager on every
communication that took place. My role has generally been to produce
systems, investigate alternatives, and make recommendations. The
manager has pretty much always been the point of contact between the
project and the rest of the administrative organization, in terms of
budget, schedule, and people. That function would never have been
enhanced by seeing every discussion about what data to include in the
network protocol or where to do mutex locking or which team member
would be responsible for what sub-systems. For that matter, I've had
managers and directors who didn't even read all of messages addressed
directly to them. (In one case, we soon learned that he would read
the first sentence or two of each paragraph, so we made sure the
important points were in those sentences.)

[Ben Tilly]

On Thu, Dec 10, 2009 at 9:41 AM, inspector_jouve <kaush...@gmail.com> wrote:
>>Are you suggesting that that is how your organization
>>works?  Because mine sure doesn't work that way
>

> You are a lucky one.

Well Ian is a world-famous programmer working for Google. Obviously
he is fortunate in many ways. But in this he doesn't seem
particularly fortunate.

In the last decade I have only once had the misfortune to be managed
by someone who wanted to be CCed on every email. And that period
stands out as being exceptionally unpleasant, and one where I
underperformed significantly.

>>Why would my manager want to waste  all his time
>>reading e-mail between people on the team?
>
> Because this is what he/she is paid for.

Dysfunctional organizations are glad to pay people to be incompetent.
Sane organizations don't. Barring a discipline/conflict situation,
managers should not waste time being CCed on every email.

> Can you provide a different definition of a manager? I'm just curious
> what planet you are living in.

I like what _First, Break All The Rules_ has to say about this. Their
claim is that great managers are catalysts that speed the desired
reaction between the employee's talents and the company's goals, and
between the employee's talents and the customers' needs. The four
skills great managers need are:

- select a person
- set expectations
- motivate the person
- develop the person

This list was generated by first using various kinds of surveys to
identify managers whose teams were particularly effective, and then
doing more intense investigation on what made the best managers
different from the average. (What? A management book based on actual
data? Who would want to learn from such a thing!)

Being CCed on every email strikes me as a demotivator, and makes the
manager into a bottleneck.

Note that many common managerial activities are conspicuous by its
absence from that list. For example none of the following are
important:

- provide leadership
- plan strategy
- give good presentations

Let me backtrack. Those are *personally* important because being seen
doing those things can help your career advance. But none translate
into improved team performance.

Cheers,
Ben

[Ian Lance Taylor]

inspector_jouve <kaush...@gmail.com> writes:

> Now, even if we settle for the military-style design, it's still not
> 100% trivial to design protocol of error propagation.
> E.g., there should be standard error channel, these channels from
> children should be multiplexed into parent error-reading channel;
> orphans should be stopped somehow; all communication channels between
> parent and children have to be accounted to close them on error
> (preventing deadlocks), etc. etc. It's complicated, but doable. But
> without this restriction - sorry, I don't know even how to define a
> problem.

The language should permit all reasonable types of design. It should
not force all programs to be written in a "military-style" design.
The question to ask is: is there a general construct we can add to the
language which can support not only my desired style of programming,
but also other styles?

Ian

[atomly]

On Thu, Dec 10, 2009 at 11:58 AM, inspector_jouve <kaush...@gmail.com> wrote:
> Let's recall how organizations work. Every Email you send to your
> colleague (especially if it requires some work to be done) has to be
> CC'd to a manager. Why? This CC is practically equivalent to
> communication via manager. Isn't it a correct analogy?

Glad I don't work where you work! :P

[Ben Tilly]

On Thu, Dec 10, 2009 at 11:30 AM, inspector_jouve <kaush...@gmail.com> wrote:
[...]
> The totally hierarchical system I described is a well-established
> practice in the military.
> What I propose is: to implement same kind of system, because it's easy
> to understand, and it works.

You have not demonstrated that it works. You have categorically
rejected the idea of having database connection pools, and I have yet
to see a design proposal for the problem space that they address.

> Otherwise, exception handling is a nightmare.

Make things as simple as possible and no simpler. When communication
goes multiple ways, error messages also need to sometimes go multiple
ways.

Let's take the connection pool example. If the current transaction is
aborted due to deadlock that should be logged and should go to the
goroutine that sent the SQL request. If the connection goes away then
that should be logged, the current partner should be told, the parent
goroutine should be told, and the current goroutine should exit.

> Suppose you work for utopian organization where everybody just loves
> everybody else, and you never CC anything to anybody. But we are
> talking about exceptions, right? Suppose one of your coworkers fell
> ill, or took maternity leave, or her uncle died and left her a fortune
> and she doesn't have to work any more - one of those things. What will
> you do?
> Is it even your responsibility to deal with this? Do you have to
> inform your manager about the situation, or somehow he is supposed to
> know already?

You are arguing a straw man here. The fact that managers do not ask
to be CCed on everything doesn't mean that they don't expect to be
emailed when appropriate. They just expect employees to use good
judgment about when it is appropriate.

> Now, even if we settle for the military-style design, it's still not
> 100% trivial to design protocol of error propagation.
> E.g., there should be standard error channel, these channels from
> children should be multiplexed into parent error-reading channel;
> orphans should be stopped somehow; all communication channels between
> parent and children have to be accounted to close them on error
> (preventing deadlocks), etc. etc. It's complicated, but doable. But
> without this restriction - sorry, I don't know even how to define a
> problem.

The fundamental challenge is that some errors need to be passed on,
and some do not. The military has a principle called "need to know".
Sending information to a central person works exactly because that
person knows who to radiate information back down to.

> When you say you don't like this, it seems that you are focusing on
> scenario when everything works fine - and really in this case the
> restriction looks stupid. But if someone's uncle dies ... I'm talking
> about this possibility, actually.

Similar to my connection pool example, some problems need to involve
the manager, and on others you want to resolve it between the affected
people. Don't be afraid to bring management in for real problems, but
involving them shouldn't be your default first reaction.

Make things as simple as possible but no simpler. Human organizations
need to send different kinds of errors to different places. This is a
piece of necessary organizational complexity. Organized sets of
goroutines have the same issue.

Cheers,
Ben

[SnakE]

I think I'll leave the questions of perfect management aside, and will address some other points.

2009/12/10 inspector_jouve <kaush...@gmail.com>

3) To be sure we are on the same page: parent never receives EXCEPTION

from children; it has to retrieve their errors from "standard input
for errors" channel explicitly. This eliminates some of your issues.

Then this is the same as error returns, just one abstraction layer higher.  I'd like the errors to propagate implicitly whenever possible.
 

But for all this to work, it really should be a TREE of goroutines.
[...]
Can we agree on that?

I don't think so.  I think it's a perfectly valid design for a goroutine to spawn a hundred of other goroutines and then terminate, leaving kids to do the job.

Here are my thoughts.

Goroutines implement conventional exceptions, presumably with try/catch/whatever.  Additionally, every goroutine, including the main one, has an implicit error channel.  Any unhandled exception, presumably of type os.Error, terminates the goroutine and goes into this channel.

Now the interesting part.  If you start a goroutine using the current syntax, the goroutine *inherits* the parent's error channel.  I.e. the parent's error channel becomes the child's error channel as well.  If only "go foo()" syntax is used in the whole program, all the goroutines inherit the error channel from main().  This channel is supplied by the runtime, and the application terminates if anything is sent into this channel.

But you can build a complex hierarchy of observers if you wish.  To do this, you use a special goroutine invocation syntax:

err := make(chan os.Error)
go(err) foo()

Now foo() gets err for its error channel, and all the goroutines started from foo() inherit it.  Now it is your program's responsibility to monitor err and handle whatever falls out of it, but this should not necessarily be done in the goroutine which started foo().  Since err is just a regular channel you can pass it to a specialized error handler, or use one of the specialized global error channels, or implement any number of other interesting designs.

This solution has many qualities of regular exceptions:

* error handling doesn't get in the way when you don't care
* your program crashes with nice *go stacks* in addition to call stacks
* you can intercept error flow
* you can re-route error flow
* you can add context and re-throw

Now, what do you think?

[atomly]

Can I reiterate what I said before?

This sort of situation is already very clearly solved by systems like Erlang.

In Erlang it is incredibly easy to spawn a new process (much like a
goroutine) and, in fact, even incredibly easy to do so on a remote
node, yet Erlang is the easiest system I've ever seen for building
robust, fault-tolerant services.

http://www.erlang.org/course/error_handling.html
http://www.erlang.org/doc/reference_manual/processes.html
http://www.erlang.org/doc/reference_manual/errors.html

I don't propose that Go come anywhere near the complexity of Erlang in
this regard, but the notion that it's difficult to propagate error
date to parents and children with lightweight processes (read:
goroutines) is definitely wrong.

[SnakE]

2009/12/11 atomly <ato...@atomly.com>

Can I reiterate what I said before?

Sure you can.  You know, free speech and stuff.  :D

In Erlang, you have PIDs.  And you must link explicitly.  What happens if you don't link?  My familiarity with Erlang is cursory but I think only that process terminates if there are no listeners to it.  Also, in Erlang, any process must poll its queue, or otherwise error propagation won't work.  I think my approach is much less intrusive.  It could be also not that powerful, but I'm not sure.

[Jonathan Amsterdam]

> Goroutines implement conventional exceptions, presumably with
> try/catch/whatever.  

Already done. See exp/exception in the current release.

The rest of your proposal is very similar to the one I described in
"unifying exceptions and goroutine termination", except that
everything is explicit -- you must create your own error channel, and
you must pass it to each goroutine you spawn. I think being explicit
is more in the spirit of the language.

[Jonathan Amsterdam]

> This sort of situation is already very clearly solved by systems like Erlang.

Erlang is set up dually to Go -- processes are the things with
identity, and channels are second-class -- so the mechanisms there
don't quite carry over. It would be helpful if you tried to translate
the link and monitor concepts to Go.

[SnakE]

2009/12/11 inspector_jouve <kaush...@gmail.com>

The issue of orphans is the core of the problem. If you don't impose
any restrictions on the graph of connections between goroutines, this
is very hard question.

Even if it really is a problem, and even the core one, you'll have to learn to live with it.  There is no safe way to kill a running thread.  Here's an excerpt from MS Windows TerminateThread() function documentation [1]:

TerminateThread is a dangerous function that should only be used in the most extreme cases. You should call TerminateThread only if you know exactly what the target thread is doing, and you control all of the code that the target thread could possibly be running at the time of the termination. For example, TerminateThread can result in the following problems:

• If the target thread owns a critical section, the critical section will not be released.
• If the target thread is allocating memory from the heap, the heap lock will not be released.
• If the target thread is executing certain kernel32 calls when it is terminated, the kernel32 state for the thread's process could be inconsistent.
• If the target thread is manipulating the global state of a shared DLL, the state of the DLL could be destroyed, affecting other users of the DLL.

[1] http://msdn.microsoft.com/en-us/library/ms686717%28VS.85%29.aspx

[atomly]

On Thu, Dec 10, 2009 at 5:59 PM, Jonathan Amsterdam
<jbams...@gmail.com> wrote:
> Erlang is set up dually to Go -- processes are the things with
> identity, and channels are second-class -- so the mechanisms there
> don't quite carry over. It would be helpful if you tried to translate
> the link and monitor concepts to Go.

You also don't have pattern matching, etc. There are definitely
differences, but I think it could be looked to for inspiration.

I'm going to ponder this on my train ride home...

[SnakE]

2009/12/11 Jonathan Amsterdam <jbams...@gmail.com>

The rest of your proposal is very similar to the one I described in
"unifying exceptions and goroutine termination", except that
everything is explicit -- you must create your own error channel, and
you must pass it to each goroutine you spawn. I think being explicit
is more in the spirit of the language.

Yes your proposal is very similar, and you discuss several important implementation details which I omitted.  The main difference between our approaches is that, in my case, all the goroutines started from a checked goroutine also become checked, while in your case, only one goroutine in the middle of the flow gets a custom error channel.  It's like if try{} only worked on the outermost function within the try block, and exceptions thrown in functions called from there passed by uncaught.

[SnakE]

2009/12/11 inspector_jouve <kaush...@gmail.com>

> This sort of situation is already very clearly solved by systems like Erlang

I made a *concerted* effort to understand Erlang's concept. I don't
get it. What is signal? Is it an interrupt? How they manage to
terminate a process when it doesn't want to be terminated? Just by
killing it? (Document says so)

If I understand it correctly, Erlang error handling model is purely cooperative.

Every Erlang process has a message queue.  A process *must* read that queue for things to work.

You can link() to another process, doesn't matter if it's its child or not.  Then a hidden one-way channel is created and when a linked process terminates for whatever reason a message is sent to the parent process.  This message gets into the linking process's queue and gets handled when the process reads next message.

By default, a message about normal termination of a "child" process does nothing, and a message about abnormal termination causes the listening thread terminate as soon as it tries to read next message from its uber queue.

A process may request the runtime to disable default handling of those death messages and instead treat them as regular messages.

It's basically the same as I and Jonathan proposed.

[SnakE]

2009/12/11 inspector_jouve <kaush...@gmail.com>

> Even if it really is a problem, and even the core one, you'll have to learn to live with it.

The only way known to me to deal with this problem is java way. That
is, provide no mechanism at all, and do it yourself on case-by-case
basis. Then, each time when you say "go func()", you will have to
write a ton of quite complicated code around its errors.

Unfortunately yes, I think so.  But if the garbage collector learns how to kill a goroutine which is blocked on a channel which nobody references then hopefully it will be enough in most of the cases.  A goroutine blocked on a channel has a very deterministic state so it should be safe to assume it can be killed on any platform.

[atomly]

On Thu, Dec 10, 2009 at 9:28 PM, SnakE <snake...@gmail.com> wrote:
> If I understand it correctly, Erlang error handling model is purely
> cooperative.
>
> Every Erlang process has a message queue.  A process *must* read that queue
> for things to work.
>
> You can link() to another process, doesn't matter if it's its child or not.
> Then a hidden one-way channel is created and when a linked process
> terminates for whatever reason a message is sent to the parent process.
> This message gets into the linking process's queue and gets handled when the
> process reads next message.
>
> By default, a message about normal termination of a "child" process does
> nothing, and a message about abnormal termination causes the listening
> thread terminate as soon as it tries to read next message from its uber
> queue.
>
> A process may request the runtime to disable default handling of those death
> messages and instead treat them as regular messages.
>
> It's basically the same as I and Jonathan proposed.

Close.

In Erlang, every process has an unbounded mailbox (asynchronous
message queue). Sending a message to a process is as easy as writing

Pid ! message

The language also has the concept of "links." This is not one-way, as
you said, but bidirectional. Monitors are similar, but one-way. As
you said, any process can link to any other process. When a process
has an error, the runtime sends an exit signal to everybody in that
process's linked set with the reason. Normal process are killed when
they receive this signal, but system processes do not. In order to
become a system process, you do:

process_flag(trap_exit, true)

This tells the runtime that you're a system process and you want to
trap exit signals from your linked processes. You will just receive
those exit signals as normal messages in your inbox and you can decide
what to do with them (log an error, restart the process, etc).

[atomly]

On Thu, Dec 10, 2009 at 9:49 PM, inspector_jouve <kaush...@gmail.com> wrote:
> Erlang was developed in 1986. Java - in  1995. C# - in 2000. No
> attempt to borrow Erlang's concept was made.
> Probably, there was a reason for this. And it's just my guess that
> without severe restrictions on topology of communicating processes you
> cannot even formulate a problem. This is just a guess. I might be
> wrong.

I think you are. Erlang, despite being old, is very much a growing
language. In addition to this, it has influenced a lot of modern
languages (c.f. Scala)

Erlang's syntax, type model, etc. are the reasons most people don't use it...

[atomly]

As I said before, I don't want to graft the whole Erlang system onto
Go, but I do think it would be good to look at their model and see
what can be taken from it for Go.

I think it would be interesting, for example, if it were possible to
do something like this:

a := go foo()

Giving you a channel to your child process. If your child has an
exception, it's communicated on this channel. Then, your main can
just be a select on your children channels so that you can do logging,
error-handling, etc...

[Ian Lance Taylor]

atomly <ato...@atomly.com> writes:

> I think it would be interesting, for example, if it were possible to
> do something like this:
>
> a := go foo()
>
> Giving you a channel to your child process. If your child has an
> exception, it's communicated on this channel. Then, your main can
> just be a select on your children channels so that you can do logging,
> error-handling, etc...

One important question for this approach is: what is the the type of
a? That is, what kind of values may be sent on the channel?

Ian

[atomly]

On Thu, Dec 10, 2009 at 10:35 PM, Ian Lance Taylor <ia...@google.com> wrote:
> One important question for this approach is: what is the the type of
> a?  That is, what kind of values may be sent on the channel?

Since it's not as easy to send generic messages as Erlang, I was
thinking it'd probably have to either be a made up ProcInfo type
structure, or just an os.Error

[Ian Lance Taylor]

atomly <ato...@atomly.com> writes:

> On Thu, Dec 10, 2009 at 10:35 PM, Ian Lance Taylor <ia...@google.com> wrote:
>> One important question for this approach is: what is the the type of
>> a?  That is, what kind of values may be sent on the channel?
>
> Since it's not as easy to send generic messages as Erlang, I was
> thinking it'd probably have to either be a made up ProcInfo type
> structure, or just an os.Error

OK, but the next thing one would want is the ability for the goroutine
to explicitly send something on that channel.

Ian

[atomly]

On 12/10/09, Ian Lance Taylor <ia...@google.com> wrote:
> OK, but the next thing one would want is the ability for the goroutine
> to explicitly send something on that channel.

Well, if it's only as an exception-handling mechanism and we continue
down the Erlang approach, then the goroutine would only send on that
channel by calling something like exit(os.Error) or throw(os.Error)

[Jonathan Amsterdam]

> >> One important question for this approach is: what is the the type of
> >> a?  That is, what kind of values may be sent on the channel?

I propose:

type ExitInfo interface {
String() string;
Error() os.Error; // nil if terminated normally
OtherInfo() interface{}; // whatever the thrower feels might be
useful
}

[Jonathan Amsterdam]

> OK, but the next thing one would want is the ability for the goroutine
> to explicitly send something on that channel.

No, we're talking about a special channel that deals with things that
cannot be expressed in the language right now (e.g. built-in errors,
info communicated via runtime.Goexit without having to pass down a
special throw function). If you just want to send "something", you use
ordinary channels.

[Jonathan Amsterdam]

> The main difference between our
> approaches is that, in my case, all the goroutines started from a checked
> goroutine also become checked, while in your case, only one goroutine in the
> middle of the flow gets a custom error channel.  It's like if try{} only
> worked on the outermost function within the try block, and exceptions thrown
> in functions called from there passed by uncaught.

I like your way better. I think that the Go designers prefer
explicitness.

[atomly]

This is also how Erlang works-- relationships must be explicitly established.

Say process A spawns process B and links to it, then B spawns process
C. If C dies, nobody cares. If B dies, A dies (unless it traps the
signal) and C continues, none the wiser...

[SnakE]

2009/12/11 atomly <ato...@atomly.com>

On Fri, Dec 11, 2009 at 11:46 AM, Jonathan Amsterdam
<jbams...@gmail.com> wrote:
>> The main difference between our
>> approaches is that, in my case, all the goroutines started from a checked
>> goroutine also become checked, while in your case, only one goroutine in the
>> middle of the flow gets a custom error channel.  It's like if try{} only
>> worked on the outermost function within the try block, and exceptions thrown
>> in functions called from there passed by uncaught.
>
> I like your way better. I think that the Go designers prefer
> explicitness.
>
This is also how Erlang works-- relationships must be explicitly established.

Say process A spawns process B and links to it, then B spawns process
C.  If C dies, nobody cares.  If B dies, A dies (unless it traps the
signal) and C continues, none the wiser...

So, Erlang hides unhandled errors.  In my proposal, and Jonathan's, and Ryanne's, unhandled errors crash the application.  Choose your poison.  I prefer mine.

[Qtvali]

I am somewhat against exceptions.

Exceptions are extremely slow, which does not block their misuse even
by language designers - often, language designers use them to stop an
iteration, from example. This means that when I iterate over my array
with my own iterator class, it will throw an exception containing few
empty strings, a stack content as an array of methods, code files and
line numbers and maybe some more additional data. Then a receiver
catches this and stops the iteration.

I would like to just throw something to be catched, but not an
exception. When exception is needed, I would just collect a stack
content by my own class, and then, for example, send it to some
exception-collecting thread (as Russ mentioned, it's not good to have
exceptions for multithreaded applications as they must all be able to
handle them), then stop that thread. This can be achieved with
channels.

In this text about those block functions I thought, I did write about
those throw cases - I did it within context and this could be achieved
by pattern, which would take this specific bit out of what I proposed
(going from more complex to specific primitives) I would like such
way:

throwingFunction() {
stopIterator():
some code to stop iterator.
failedIterating():
some other code.
}

Where throw function would include those two throw cases as some
interface it "yields", both methods would be defined in that
interface. They could take params and return values. Inside catching
block, throwingFunction could be optionally stopped, but it would also
allowed to continue, throwing more things. This simple pattern would
also allow all other patterns I did describe - just in more verbose
way. throwingFunction should also be allowed to use "go" when
throwing, but this should not mess the caller by executing several
cases at once; using "go throwingFunction" should be also allowed, but
caller should be aware of dangers in such case.

i := throwingFunction() {
stopIterator():
return 1; // would stop the caller and set it to return 1
}