PPROF weirdness with goroutine status

[Blake Caldwell]

TLDR: PPROF shows the count of goroutines to be at 5,000, but the goroutine stack trace page only shows 20 or so - WHY???

Details:

I'm using the pprof web endpoints. It's been insanely valuable to see the stack traces for all my goroutines. I recently reported that it looks like my goroutines aren't being in some weird case here:

https://groups.google.com/forum/#!searchin/golang-nuts/blake$20caldwell/golang-nuts/uhqGZYM2LJw/8KzuW5O7IJoJ

Well, I have a lot more info now. The reason I thought the deferred statements weren't being called is because they didn't show up in the goroutine stack trace page at /debug/pprof/goroutine?debug=1. There, I only see about 20 goroutines, which is what I'd expect. However... I just noticed that they *are* showing up on the /debug/pprof/ page in the goroutine count:

/debug/pprof/

profiles:

0 block

5544 goroutine

2716 heap

12 threadcreate

Disregard the "block" count - I don't have that enabled.

So, in what state is a goroutine where it's being counted in the total count, but not showing up in the stack trace page?

Thanks!

[Blake Caldwell]

Should:

/debug/pprof/goroutine?debug=2

show more goroutines than:

/debug/pprof/goroutine?debug=1

??

At quick glance, that seems to be the case, but I'm going to have to check later on to be sure. I think the debug=2 page is showing my hanging routines. I notice at least one looks like this:

"goroutine 24131766 [select, 55 minutes]:"

I'm guessing this will show me the problem, but need to understand how this is happening better.

[Benjamin Measures]

On Thursday, 29 January 2015 23:02:24 UTC, Blake Caldwell wrote:

So, in what state is a goroutine where it's being counted in the total count, but not showing up in the stack trace page?

TLDR; One shows active goroutines (ie. runnable, and not in other states, such as syscall or wait), and the other dumps all goroutines (in any state).

<http://golang.org/pkg/net/http/pprof/>:

Package pprof serves via its HTTP server runtime profiling data

<http://golang.org/pkg/runtime/pprof/>:

goroutine    - stack traces of all current goroutines
[...]
The predefined profiles may assign meaning to other debug values; for example, when printing the "goroutine" profile, debug=2 means to print the goroutine stacks in the same form that a Go program uses when dying due to an unrecovered panic.

<http://golang.org/pkg/runtime/#GoroutineProfile>:

GoroutineProfile returns n, the number of records in the active goroutine stack profile.