Linux core dumper tool

[Brad Fitzpatrick]

I wanted to use https://github.com/cloudwego/goref/ (an amazing tool!) on a large server with ~250 GB of heap to figure out where/why memory was being retained.

But using "goref attach" on that would stop the server for unacceptably long. Effectively an outage from a user's point of view.

I tried to use criu (https://github.com/checkpoint-restore/criu) with its pre-dump/dump but it's so focused on restoring, it doesn't provide knobs to let you ignore things it can't dump. And its coredump generation is in unpackage Python code, even if I could get a complete dump out of it.

I then thought how I really just wanted a lightweight criu that did the iterative memory copying part without caring about restoring, and how I want it to do the core dump writing after the processes were unpaused.

So, I "made" this:

https://github.com/bradfitz/livecore/

... my first ever vibe coding experience. Maybe it even works! It seems to.

I'd love a review from somebody who actually knows ELF & core dumps, though.

(But really I'd love it if Go had support for doing this on its own somehow in the runtime. ;))

Alternatively, anybody have other tool recommendations for something that'd let me make a core dump concurrently without my application serving traffic? Or really a way for me to do leak analysis on a 250 GB heap without pauses.

- Brad

[Robert Engels]

I suspect for heaps that size and you want to do it live, you would need to implement fork() somehow, and then have the child process the heap. Since fork() does copy on write from either side, it should be fairly instantaneous.

Other than fork() you might be able to manually set up a copy on write of the heap address space, and then process that privately in the background.

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/golang-dev/CAFzRk01SOx4a%3DGDypsp-c5wOdU_xmdmBcv1f7b_bmFzNcW3iCQ%40mail.gmail.com.

[Devon H. O'Dell]

TL;DR:

 * Maybe you want to tag allocations in NewFoo if you want to know in aggregate what types of things are being leaked in analysis

 * Double check all threads are paused. You can spawn threads while attaching to all the threads.

 * If the approach you have is working "fast enough" for you, the approach seems mostly correct modulo the above.

 * I can't comment on the ELF stuff.

 * Analyzers will benefit from having a detailed understanding of Go's allocator regardless.

It's been about 10 years so I'm fuzzy on all the details, but I worked on a tool to do this at Fastly where the server processes were hundreds of gigs resident memory and tens of thousands of threads, though the system was C. We already had tagged allocations, so it was easy to figure out what types allocations were, and even what was allocated, even without understanding allocator internals. I imagine this could be useful to do if you want to figure out what kinds of things are being leaked, even if you have some more detailed knowledge of Go's allocator. In this project, I wrote a custom output format for analysis, because I was looking specifically to analyze leaks and we had thousands of maps that were irrelevant for holding pointers. I also have a fairly limited understanding of ELF, so I can't speak to that aspect of your tool.

I'm not clear on what classifies as "outage" for you; we drained traffic from machines under analysis so that it wouldn't cause outages. This kind of thing can take a long time, though if this program is falling within an acceptable time, maybe you can ignore some of my cautions about these aspects. Problematically, for memory leak analysis, you need to pause allocators for the duration of instrumentation. It's usually easier to just pause everything for the duration of copying maps, which seems like what you're doing.

A cursory look over your project seems like most issues have been identified. I've seen tools forget to collect register state before, and that often holds references to huge live arenas that otherwise might go unreported.

Whatever method you use, you'll need to leave the whole process, and all threads, paused for a significant period of time to traverse and write the maps to disk. If you have any in-process liveness checks, turn those off so that when ptrace detaches, your process doesn't automatically die. (Or that attaching doesn't take so long that the health check thread lives long enough to kill the system while you're waiting to attach to everything.)

You might want to do an extra pass after running FreezeAllThreads to check if any threads were created while you were busy pausing threads. I haven't done a detailed review so I'm not sure if this means you'll also have to re-scan maps to get those thread stacks etc.

Not sure how you're doing core analysis but of course having some analyzer that understands how to read go's allocator state will be helpful.

Hope that's helpful in some capacity. But overall, this looks like a great project, and mostly correct. It really doesn't take that much code to do this kind of thing!

Kind regards,

--dho

[Brad Fitzpatrick]

Thanks for the reply! Doing another pass looking for threads started during the thread pausing sounds like a great idea.

Oh, and my email wasn't clear. I left out a key detail: this tool was made for use by the "goref" tool. goref can run in one of two modes: 1) attach to a PID, or 2) on a core file. The attach mode is too slow, so I wanted a better way to get a core file so I could use goref in that other mode.

Because I only care about goref as a consumer of the output, I skipped some things like getting floating point registers in the core dump, since they can't hold pointers.

[Robert Engels]

It seems like google-coredumper might be a good start - it seems to monitor threads etc - I am sure it will take some work to integrate it.

[Brad Fitzpatrick]

I started down the road of modifying the runtime to have new API to do a fork and then all in a nosplit assembly function mask signals and sigstop the pid, just to have a place to core dump from externally, but then I realized that lost all the thread info. And dealing with forks in a multi-threaded program (or trying to bring in the abandoned Google C library to do it in Go's universe) seemed dicey.

Doing it from the outside was much more within my reach.

[Devon H. O'Dell]

gotcha! that all makes sense.

the thread re-scan thing is a bit of a "halting problem" kind of problem: if you do a scan and pause a thread, you ought to try again.

i took a bit of a deeper look into the copy strategy and that seems pretty efficient as well. overall, again, looks like a cool utility. thanks for making and sharing it!

--dho

[Austin Clements]

We've considered doing this in the runtime before, mostly to replace its custom and aging "heap dump" format, but it's never been pressing or valuable enough for us to actually tackle.

One approach if you're in the runtime is to stop the world, capture just the thread state (which will be quite minimal at that point; for goref you might only care about threads in syscalls), fork, start the world, and then dump memory from the child.

To view this discussion visit https://groups.google.com/d/msgid/golang-dev/CAFzRk02DPS2bin44u0D%2B7RQLGTdhBB%2BQ61pObh7RzPVaAJuC3Q%40mail.gmail.com.

[Brad Fitzpatrick]

Now that we were able to run this in prod, it appears we have a timer leak somewhere. The hunt is underway :)

I see no metrics in runtime/metrics about the size of the heap timers. I'll probably add something like this in our Go fork:

package runtime

func TailscaleNumTimers() int {
  var sum uint32
  lock(&allpLock)
  for _, pp := range allp {
     if pp != nil {
        sum += pp.timers.len.Load()
     }
  }
  unlock(&allpLock)
  return int(sum)
}

Or does that exist somewhere? If not, maybe that's something that could be added to runtime/metrics?

[Michael Knyszek]

Yeah, we could totally add this.

My (slight) preference would be the sched metrics since the grouping generally makes sense, and we take allpLock to compute goroutine counts and such already.

I can file a proposal.

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/golang-dev/CAFzRk01q-iR%3DbYv8-QyCuce-yOnGLru8mR-yNRBP7S%2Bkyu%2BkPg%40mail.gmail.com.

[Michael Knyszek]

Filed https://github.com/golang/go/issues/75956.