A global panic handler for crash reporting

[Alan Shreve]

I’d like to do crash-reporting for programs that run in environments I don’t control (e.g. your laptop). The behavior I want is similar to what many production-grade desktop applications do when they crash: capture process state information, optionally prompt the user for permission, and send the crash report to a secure server.

How one would implement such a function for Go programs is tricky without cooperation from the runtime. The options I’m considering:

1. The strawman is to wrap every goroutine that I spawn with a function that defers and calls the panic handler. It would have no effect on third party libraries which spawn goroutines (or the standard library) which makes it pretty much a non-starter. It’s also extremely onerous and unidiomatic to write all of your code this way.

2. Automate the above behavior by parsing all of the Go code (including 3rd party libs) to rewrite all statements which spawn goroutines to wrap each goroutine with a panic handler. It’s messy, adds another stage to my build process, but could work well for all of my code and 3rd party code and possibly for the

3. Set GOTRACEBACK=crash and then use the operating-system native interfaces to recover the state of the program. This is a lot of work. This interface is defined differently on each OS. Recovering the state from these crash handlers would be challenging because it would happen outside the runtime and the existing tools for this like google’s breakpad are built for C applications. A minor point, but also GOTRACEBACK=crash isn’t implemented on some OS’s yet (notably Windows).

4. Fork immediately after startup and use the parent process to monitor the child for exit code 2 and a panic traceback on stderr. This is the approach taken by panicwrap[0] which is known to work, but has two issues. Dealing with signals becomes especially tricky. Any number of supervisor programs and system administration tools rely on sending signals to manipulate processes in production. The crash-handling parent process would need to handle these signals appropriately. Should it forward them to the children? Or rely on the signaling process to signal the whole process tree? Signal handling behavior is not consistent across platforms, which makes this difficult to get right. For example, Windows apparently sends CTRL+BREAK to the whole tree, but not CTRL+C. As a final point, this approach also fails on systems that disallow spawning additional processes (NaCl, maybe AppEngine, I’m unsure).

5. Fork immediately after startup and dup stderr through to the child process. This avoids all of the signal handling conundrums of approach #4 but does mean that you can no longer check the exit status of the program and would have to fall back just to looking for the ‘panic:’ header only. Still doesn’t work if you can’t spawn processes.

6. I’d like to modify the Go runtime to simply add an API which allows a developer to intercept runtime panics and choose how to handle them. Ideally, I would like to do this with an API like:

runtime.OnPanic(func(state *ProcessState) {
// send to crash report server
})

I’d even settle for:

runtime.OnPanic(func () {
// send to crash report server
runtime.Stack()
})

What is the best option among these? Would this be an API the Go team would consider adding to the language?

[0] github.com/mitchellh/panicwrap

[Nikolay Bryskin]

Hey, Alan. I'm relatively new to Go, and faced a similar issue - I writing tests for a legacy codebase and want to fail the test if the tested code panics somewhere inside.

Almost ten years passed - did you find or create a solution?

[Gergely Brautigam]

If you have a top level recover in you main, it doesn't matter where the panic happens, you'll capture it. Even in third party library.

package main

import (
"fmt"

"github.com/Skarlso/panic"
)

func main() {
defer func() {
if r := recover(); r != nil {
fmt.Println("Recovered in f", r)
}
}()
fmt.Println("testing")
panic.MakePanic("testing this now")
}

This will output:

go run main.go
testing
Recovered in f testing this now

Is that what you're looking for?

[Jan Mercl]

On Tue, Dec 26, 2023 at 9:12 AM Gergely Brautigam <skarl...@gmail.com> wrote:

> If you have a top level recover in you main, it doesn't matter where the panic happens, you'll capture it. Even in third party library.

Iff the panic occurs in the same goroutine where the defer is. Every
go statement starts a new goroutine that is not affected by defer
statement(s) in other goroutines.

[Gergely Brautigam]

Yah, that's true, I completely forgot about that. :/

I guess you could do what some others have done which is implement a Call function or a framework that runs all function calls in a middleware-esk style. And middleware has a recovery. So every call will have a recovery.

[Johan Liebert]

A global panic handler for crash reporting is a critical component in software development. It serves as a robust mechanism to capture, analyze, and respond to unexpected errors or crashes in applications. This handler acts as a safety net, allowing developers to receive real-time notifications, gather essential data, and implement timely fixes, contributing to improved software reliability and user experience.

The incorporation of a life2vec calculator into this system could further enhance its capabilities by providing a tool for assessing the impact of errors on the overall life and performance of the application. This innovative approach aligns with the industry's commitment to continuous improvement and proactive error management, ensuring smoother and more resilient software operation.