runtime.OnPanic(...)

[Alan Shreve]

I want to build automated crash-reporting into Go programs that I distribute to run on machines I do not control. There are a few approaches to doing this that I outlined in this thread to go-nuts: https://groups.google.com/forum/#!searchin/golang-nuts/crash$20reporting/golang-nuts/XLXXKkRqO1c/_SDPCIHvhaoJ I believe the best approach involves adding a new API to the runtime package, so I’d like to propose:

    runtime.OnPanic(handler func(reason interface{}))

When an unhanded panic in a Goroutine would cause the process crash, this function is called with the reason for the panic.

I’ve prototyped this change and have a working changeset. Adding new API surface area is a place where one treads delicately, so I wanted to get the Go team's input on whether this is a sane approach and if you would be open to including it in the language.

The more detailed semantics are as follows:

- When the handler completes, the program crashes with the same behavior as it does today.

- I am ambivalent about whether callers can only set a single handler or if multiple calls mean that there are multiple panic handlers that are run serially. My implementation currently allows only a single global one.

- A panic handler is called once, and only once. That means:

  1. if a panic happens while running the handler, the program crashes and does not recursively trigger the handler

  2. registering a new OnPanic handler while in a panic handler does not change this guarantee and is effectively a no-op

- alan

[Aram Hăvărneanu]

It's impossible to execute arbitrary code safely in panic context.

--
Aram Hăvărneanu

[Alan Shreve]

Is it impossible for all panics? Or is it only unsafe to do so in cases where the runtime internally caused the panic? If it is the former, what makes it impossible to do so?

> --
> You received this message because you are subscribed to the Google Groups "golang-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[minux]

On Aug 20, 2014 6:12 PM, "Alan Shreve" <al...@inconshreveable.com> wrote:
> Is it impossible for all panics? Or is it only unsafe to do so in cases where the runtime internally caused the panic? If it is the former, what makes it impossible to do so?

it might be holding a lock, or the memory might be corrupted. please consider how you're going to document the new feature: what is safe to run as a handler?

the only reliable way is to run your application as a child process and monitor its output.

[Alan Shreve]

Thanks for clarifying. Sorry I’m a little slow here but could you help me understand a little bit more?

Running another process is not 100% reliable either; the hardware could fail, the operating system could fail, or there could be a bug in your monitoring process. No crash reporting mechanism is ever 100%, one must choose some set of given primitives and assume them to be “safe”.

The proposed API would widen the scope of errors which could cause undefined behavior to include those caused by a broken runtime or misbehaving Cgo, but it seems like it could still safely cover the 99% case of panics caused by nil pointers, array out-of-bounds accesses, failed type conversions, and explicit calls to panic by user code. Am I misunderstanding the runtime code or is it still unsafe to run code after these types of panics?

Go provides escape mechanisms that allow the programmer to perform operations where safety is not guaranteed (Cgo and package unsafe, for example). This API would have to be documented with that explicit caveat as well.

Ideally, I only want to capture the panic reason and thread stacks and hand those off to another process, would an API more targeted to exactly that use case be more acceptable?

[minux]

On Aug 20, 2014 7:41 PM, "Alan Shreve" <al...@inconshreveable.com> wrote:
> Thanks for clarifying. Sorry I’m a little slow here but could you help me understand a little bit more?

how are you going to document this proposed feature? we can't just say it's a best effort service. you can't allocate memory, can't defer (it allocates memory), can't use network (might take lock), can't start goroutines (allocates memory), can't take lock (might deadlock), can't start child process (takes the ForkExec lock), ...(there are more that you can't safely do, these are just from the top of my head) when running the handler.

i've heard program that tries to flush the log just before the program exits goes wrong and the program never exits.

[Rob Pike]

This cannot go into the runtime. It is a disastrous idea.

When a program is panicking, nothing is guaranteed. The job of panic
is to write down what it can and get out, assuming it even can.
Allowing the user to run unspecified, uncontrolled code at that point
can trigger more panics, corrupt the heap, loop forever, hang forever,
on and on. I've seen all of these things happen with similar features
in other languages, like atexit in C.

If atexit is undependable, OnPanic is downright dangerous.

This cannot go into the runtime. It is a disastrous idea.

-rob

[Taru Karttunen]

On 20.08 12:26, Alan Shreve wrote:
> I want to build automated crash-reporting into Go programs that I distribute to run on machines I do not control. There are a few approaches to doing this that I outlined in this thread to go-nuts: https://groups.google.com/forum/#!searchin/golang-nuts/crash$20reporting/golang-nuts/XLXXKkRqO1c/_SDPCIHvhaoJ I believe the best approach involves adding a new API to the runtime package, so I'd like to propose:
>

We have stumbled upon the same thing. Ideally there would be an API
like:

somelib.RunThisSupervised() error

which would re-execute the current executable as a child-process
and return when it exited (+ possible heart-beat monitoring?).

However this is actually quite non-trivial with Go as all kinds of
initialization code will run before main. If you are not averse of
hacking things up one can have a custom startup and a C supervisor
and a Go program inside the same executable, but currently this is
very hacky.

This reverts essentially into the "how do we daemonize a Go process"
issue.

- Taru Karttunen

[Mateusz Czapliński]

On Thursday, August 21, 2014 1:41:04 AM UTC+2, Alan Shreve wrote:

Running another process is not 100% reliable either; the hardware could fail, the operating system could fail, or there could be a bug in your monitoring process. No crash reporting mechanism is ever 100%, one must choose some set of given primitives and assume them to be “safe”.

Not 100% reliable, but certainly gives a much bigger chance to do something reasonable than in a process which is already going down crashing and burning. You might want to have a look at:

  http://en.wikipedia.org/wiki/Crash_reporter

- that's apparently how Mozilla and Google approach such situations, among others.

/M.

[Alan Shreve]

Thanks all for your time and thoughts, I’ll explore these other options more deeply.

[Alan Donovan]

On 21 August 2014 11:33, Mateusz Czapliński <czap...@gmail.com> wrote:

Not 100% reliable, but certainly gives a much bigger chance to do something reasonable than in a process which is already going down crashing and burning. You might want to have a look at:

  http://en.wikipedia.org/wiki/Crash_reporter

- that's apparently how Mozilla and Google approach such situations, among others.

FWIW, I've heard advice from some of the most senior engineers at Google not to even attempt to shut down servers cleanly (and forget about panicking servers), because shutdown code is in general so poorly tested and fraught with bugs and race conditions.  _exit(2) is your friend.

[Mateusz Czaplinski]

This discussion is probably getting more and more OT here and should
be on golang-nuts I suppose, but sorry I couldn't resist:

http://en.wikipedia.org/wiki/Crash-only_software (quick intro)
http://lwn.net/Articles/191059/ (longer, but good writing and well
worth the read!)

/M.

[Niklas Schnelle]

How about using a proper init system like systemd that allows you to detect crashed and hung daemons and even supports

heartbeats killing and restarting a daemon that doesn't answer? Afaik it also supports executing another process on failure and/or exit.

[Gustavo Niemeyer]

Given that strong opinion on what's supposed to happen on a panic, I'm
curious about your opinion on how net/http doesn't actually stop the
application when a panic happens, but rather just logs it and hopes
the rest of the system continues working well. Isn't that an analogous
situation?

> --
> You received this message because you are subscribed to the Google Groups "golang-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--

gustavo @ http://niemeyer.net

[Russ Cox]

On Mon, Aug 25, 2014 at 3:45 PM, Gustavo Niemeyer <gus...@niemeyer.net> wrote:

Given that strong opinion on what's supposed to happen on a panic, I'm
curious about your opinion on how net/http doesn't actually stop the
application when a panic happens, but rather just logs it and hopes
the rest of the system continues working well. Isn't that an analogous
situation?

That is undeniably convenient, but it may have been a mistake. If I had it to do over again I might have made it opt-in. Oh well.

Russ

[Ethan Burns]

Does that mean we should consider it dangerous to recover a panic then continue on?

Ethan

[minux]

On Aug 25, 2014 6:18 PM, "Ethan Burns" <burns...@gmail.com> wrote:
> Does that mean we should consider it dangerous to recover a panic then continue on?

it's dangerous to recover _all_ panics and _always_ continue, esp. for the runtine panics.

[Andrew Gerrand]

In essence, you should only recover from panics that you are expecting (ie, you wrote the call to panic()), not to handle buggy code.  

I can count on one hand the number of times I have used recover in real code. 

On Tuesday, 26 August 2014, Ethan Burns <burns...@gmail.com> wrote:

Does that mean we should consider it dangerous to recover a panic then continue on?


Ethan

[gov...@ver.slu.is]

On Tuesday, August 26, 2014 12:25:43 AM UTC+2, Andrew Gerrand wrote:

In essence, you should only recover from panics that you are expecting (ie, you wrote the call to panic()), not to handle buggy code.  

I can count on one hand the number of times I have used recover in real code. 

Out of curiosity, what would be a situation in which such a thing is preferable over returning an error? 

[Daniel Bryan]

Callback for APIs that do not accept a function that returns an error.

[Andrew Gerrand]

As always, the standard library is instructive: http://golang.org/search?q=recover#Local

It is used in tests to ensure that panics happen when they are expected:

  http://golang.org/src/pkg/crypto/aes/aes_test.go?h=recover#L373

The bytes package uses it to trap the runtime error that occurs when allocating a slice that's too big:

  http://golang.org/src/pkg/bytes/buffer.go?h=recover#L182

The encoding/json package uses panic and recover to jump out of (potentially) deeply-nested call stacks, without doing a whole lot of "if err != nil { return err }".

  http://golang.org/src/pkg/encoding/json/decode.go?h=recover#L131

  http://golang.org/src/pkg/encoding/json/encode.go?h=recover#L256

It's worth noting that in all these cases, the recovered panic never crosses package boundaries. That is, the use of panic/recover is an implementation detail, not part of the package's API.

Andrew

[Dan Kortschak]

In what cases would that actually be recoverable; if it's an out of memory it's not recoverable AFAIK.

[Ian Lance Taylor]

On Tue, Aug 26, 2014 at 2:05 AM, Dan Kortschak
<dan.ko...@adelaide.edu.au> wrote:
>
> In what cases would that actually be recoverable; if it's an out of memory
> it's not recoverable AFAIK.

The bytes.Buffer code works by doubling the buffer size each time it
runs out of space (see the grow method). So if you get up to, say, a
1G buffer, then it is possible that it will be unable to allocate a 2G
buffer although the program is not out of memory in any meaningful
sense.

Ian

[Dan Kortschak]

Thanks Ian,

I was looking at makeSlice which just wraps make([]byte, n) with a
deferred recover. Are you saying that the failure would be due to an
non-availablility of an adequate chunk size?

Dan

[Dave Cheney]

Here is the original change, https://codereview.appspot.com/5563045

[Ethan Burns]

Thanks, Andrew. That makes me a bit sad. I suspect most of the catch-your-own-panic uses of recover are ugly conveniences for things like error handling in parsers. It always comforted me knowing that recover's true reason for existence was the net/http use case: avoiding a server crash when one client hits an inevitable bug. It's sad to crash an entire server when an off-by-one error causes an index out of bound (which I assume is perfectly safe to recover). However, given Rob's comment on how unstable the system is on a panic and Russ's regret about net/http's behavior, I'm now led to believe that I should just let the server crash. Or am I being overly pessimistic?

Ethan

[Ian Lance Taylor]

On Tue, Aug 26, 2014 at 4:57 AM, Dan Kortschak
<dan.ko...@adelaide.edu.au> wrote:
>
> I was looking at makeSlice which just wraps make([]byte, n) with a
> deferred recover. Are you saying that the failure would be due to an
> non-availablility of an adequate chunk size?

Non-availability of sufficient contiguous memory for a very large
slice, yes. Look at the callers of makeSlice.

Ian

[Dan Kortschak]

On Tue, 2014-08-26 at 22:00 +1000, Dave Cheney wrote:
> Here is the original change, https://codereview.appspot.com/5563045

It both relieves and unsettles me that reviewers were similarly
concerned. I'm afraid it doesn't yet explain to me the conflicting
information I've received about the incapacity to recover from a failed
allocation. Sorry for being dim.

[Dan Kortschak]

On Tue, 2014-08-26 at 05:20 -0700, Ian Lance Taylor wrote:
> Non-availability of sufficient contiguous memory for a very large
> slice, yes. Look at the callers of makeSlice.

OK, that makes sense now. Thanks.

[Ian Lance Taylor]

On Tue, Aug 26, 2014 at 5:11 AM, Ethan Burns <burns...@gmail.com> wrote:
>
> Thanks, Andrew. That makes me a bit sad. I suspect most of the
> catch-your-own-panic uses of recover are ugly conveniences for things like
> error handling in parsers. It always comforted me knowing that recover's
> true reason for existence was the net/http use case: avoiding a server crash
> when one client hits an inevitable bug. It's sad to crash an entire server
> when an off-by-one error causes an index out of bound (which I assume is
> perfectly safe to recover). However, given Rob's comment on how unstable the
> system is on a panic and Russ's regret about net/http's behavior, I'm now
> led to believe that I should just let the server crash. Or am I being overly
> pessimistic?

It's not crazy for something like an HTTP server to carry on if a
panic occurs. You can, of course, have various sorts of resource
leaks--files not closed and the like. Since Go when not using
cgo/SWIG is memory safe, massive corruption is unlikely.

Though I think that OnPanic is a bad idea, I think that Rob slightly
overstated the case in his e-mail message. When the program is
crashing, nothing is guaranteed. When the program is panicing, the
overall system should still work reasonably reliably.

All that said, I agree that it's definitely best to only catch your
own panics. I think the most sensible way to handle a panic in
something like an HTTP server, if you really don't want to just crash,
is to let existing connections cleanly shut down, exit the program,
and restart the server.

Ian

[gov...@ver.slu.is]

On Tuesday, August 26, 2014 2:11:05 PM UTC+2, Ethan Burns wrote:

It's sad to crash an entire server when an off-by-one error causes an index out of bound (which I assume is perfectly safe to recover). However, given Rob's comment on how unstable the system is on a panic and Russ's regret about net/http's behavior, I'm now led to believe that I should just let the server crash.

I'd rather crash the entire server and know there's an off-by-one error somewhere, those can be dangerous too. (see http://en.wikipedia.org/wiki/Fail-fast)

It might give a bit more trouble at the beginning but leads to a better program in the end.

I have a watchdog on my http servers so that if anything fails I'll get notified but the server process restarted.

Speaking of which. The current documentation of net/http doesn't even mention swallowing the panics.

Might be better of on a seperate thread, but how would people feel about making the swallowing of panics opt-out or at least documented?

I'd be willing to try that as a change to get my feet wet.

[Brad Fitzpatrick]

Filed https://code.google.com/p/go/issues/detail?id=8594 for net/http docs.

--

[Igor Gatis]

It's 2024 now. I wonder if anything has improved. I totally get the reasons why recovering from an unknown-panic is plain dangerous. However, think of the scenario of a service that handles tons of parallel requests. There must be a way to isolate misbehaving requests from the others.

Back in 2018, I posted this question: https://stackoverflow.com/questions/50418919/how-to-prevent-go-program-from-crashing-after-accidental-panic. My team was considering Go to write a credit card transaction service which heavily depended on third party (possibly buggy) code. Even if it was 100% in house code, we were just afraid that a bug in a single request could bring the whole service down, causing a lot of problems for the other non-related requests. We ended up picking C# (due to team's background, but it could have been Java too). We placed an outermost try-catch-all for each request. This isolation mechanism was good enough. We found bugs and fixed them. Over time, the service matured really well.

For the scenario, is there a recommended way of writing a Go service which isolates requests?

[Alan Donovan]

> It's 2024 now. I wonder if anything has improved. I totally get the reasons why recovering from an unknown-panic is plain dangerous. However, think of the scenario of a service that handles tons of parallel requests. There must be a way to isolate misbehaving requests from the others.

The problem is that panics are often not isolated. A panicking
goroutine may leave things like locks or data in a bad state. A
panicking worker goroutine must cause its waiter to either block
forever, causing the program to get stuck, or to unblock when the
subtask wasn't completed, causing it to explore ill-defined states.
There's no safe way to generalize what to do.

But you may be interested in the new (go1.23)
runtime/debug.SetCrashOutput feature, which allows a Go program to
write its unhandled panics and other crashes to another process so
that they can ultimately be reported to its developer. See
https://github.com/golang/go/issues/42888.

[Igor Gatis]

Thanks for the pointer, I'm going to check it out.

I believe that the same arguments could apply to other platforms such as Java and C#. I guess they're are either too permissive or Go is too strict.

In my use case, I'd be happy if panic caused my service to enter in lame duck mode, accepting no new requests but able to at least *try* to finish handling other requests for some short period of time and eventually terminating.

Sounds like one of those things it could be a feature flag which one needs to turn on by own risk.

[Russ Cox]

I am not sure that we are talking about the same things. Specifically, to first approximation panic and recover in Go are identical to throw and try...catch in C# or Java. So if you were happy using try...catch in C# for isolation, then you should have been equally happy with recover, had you chosen Go instead.

Best,

Russ

[Igor Cananea]

Having library code that creates goroutines without handling panics is an anti-pattern.

Better designed/maintained libraries will avoid this scenario and if you find one that does this, switching to a different library or patching it in your fork could be done.

For goroutines in your code, you should be using something to manage them, like waitgroup or even better errgroup. Neither of them solve your problem directly, as you can still panic in your goroutine and would have to handle the panic in each goroutine.

There is an accepted proposal to add panic handling to errgroup: https://github.com/golang/go/issues/53757

But until that lands, a developer has created a drop-iin replacement for errgroup that handles panics in goroutines: https://github.com/kucherenkovova/safegroup

    Igor Chaves Cananéa

"Designing grand concepts is fun; finding nitty little bugs is work." (Frederick P. Brooks Jr - The Mythical Man-Month)

http://correntequebrada.blogspot.com.com

--

You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-dev/CAPBqLKG7tYFSvjEua6Hcpk19u%2BL4hcNCwDxjV8B3vBzd5vEqbg%40mail.gmail.com.

[Robert Engels]

Panic recover are not the same as try/catch. Exception based languages have concepts of finally as well so it is much easier to ensure a valid state after an exception occurs. Go cannot do this except in very localized ways. 

On Jan 31, 2024, at 2:59 PM, Igor Cananea <icca...@gmail.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/golang-dev/CAEX6X_CzmK4fY-sGSt%2Bj_yv7euSPGyM%2BL_OhHWdvhLk5VAnxrw%40mail.gmail.com.

[Igor Gatis]

Russ, by default, Java will not kill the process if a child thread throws an unhandled exception. That's not the case in Go. In order to have the same behavior in Go, I'd have to modify every single goroutine creation to have a deferred recover(). That is essentially Igor Cananea's recommendation. The only problem I see with that recommendation is the fact it is a *manual process*: it needs to be enforced by code reviews and keeping third party lib forks around can be laborious/tricky.

Until this very morning, I thought there was no programmatic way of doing so. Then I came across Dominic Black's GopherCon UK presentation: https://www.youtube.com/watch?v=MRZU5J29Rys. His presentation talks about Go forks and the example he shows early on, pin points the exact location where an outermost recover() call could be placed. I'd probably have picked Go if I knew this tip back then. In my fork, I'd introduce a mechanism like GOPANIC=timeout=10 which upon unhandled panic, it would call a callback and wait up to 10 seconds before exiting the process.

[Robert Engels]

As I mentioned, this wouldn’t really work. You have no way of cleaning up the data structures and resources that may be partially mutated when the panic occurs. 

The reason Go terminates is that this could lead to disastrous results. 

On Jan 31, 2024, at 8:01 PM, Igor Gatis <igor...@gmail.com> wrote:



[Igor Gatis]

Robert, I'm trying to understand why it is "more disastrous" in Go than it is in Java or C#. Is it?

[Robert Engels]

Yes, because each function in the call stack can be written in a way to trap the exception and handle or at least in the finally clean up. Unless it is an expected panic like some of the stdlib uses this would require writing every function in Go with panic/recover - including the stdlib - to ensure a consistent state. Which is certainly not idiomatic Go nor I think feasible - because the panic does not have enough detail to handle. 

On Jan 31, 2024, at 8:59 PM, Igor Gatis <igor...@gmail.com> wrote:



[Robert Engels]

I take it back. I did not realize Go panic runs defer statements as it goes back up the stack. So if resources are properly handled with defer it will work. 

On Jan 31, 2024, at 9:50 PM, Robert Engels <ren...@ix.netcom.com> wrote:



[Steven Hartland]

Could the equivalent of finally be achieved by using a defer, which unwinds as it goes back up the stack hence could provide a similar concept?

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-dev/15490FA4-85CA-460E-81DA-582B52471C82%40ix.netcom.com.