Output of net/http/pprof (goroutines are constantly increasing)

[michael...@googlemail.com]

I'm porting a website from php to go, section after section using my go app behind a reverse proxy (apache). Everything works fine and as expected.

Now I have ported a section that makes heavy use of a mysql database (using github.com\go-sql-driver\mysql) and multiple goroutines when selecting the data. This is also the first section that has decent traffic. GOMAXPROCS is set to runtime.NumCPU(), which is 2.

I've imported the "net/http/pprof" handler to analyze my app looking at the values of http://localhost:9000/debug/pprof/. I discovered a behavior that I don't understand. If I send traffic to the new section with "boom" on my local (windows) development machine and our linux testing server, the number of goroutines increases and goes down again after the load to about 10.

On our production server, the number of goroutines constantly increases and doesn't go down again (without using "boom"). The number is now about 4000 goroutines and I'm a little concerned. So my question is: what could be the reason for this and what exactly does the number of goroutines in the debug/pprof handler mean? Is this really the number of active goroutines or is this kind of an accumulated value?

[Dmitry Vyukov]

It is the number of currently live goroutine.
Just look at the stacks, that must explain the issue. What are the
goroutines that you expect to not be in the dump doing?

[Dave Cheney]

This should be pretty easy to figure out from the stack dump. Kill your process with SIGQUIT and paste the entire output, it'll be easy to tell what is going.

My bet is it is either all the goroutines are blocked waiting on the database, or serving something unexpected like the fav icon is somehow related.

[michael...@googlemail.com]

Thanks for your replies!

There is indeed one handler, where I'm using the following pattern (and currently only there in production) and I always asked myself if this is correct. Here is the (pseudo) code:

http://play.golang.org/p/gDl7tOo7Z6

It would be nice if someone can have a look and give me some feedback, because this is the way I usually retrieve data from the database. Until now without any problems.

Dave, maybe the mysql database of our production server has different settings than on my local machine and can't handle the concurrency. I don't know what this should be, but maybe I set the GOMAXPROCS back to 1 to find out.

Here is a snippet of the "full goroutine stack dump":

goroutine 104 [chan send, 1455 minutes]:
domain.com/myproject/app.(*details).getData1(0xc208325b00, 0xc208334a20)
	D:/Go/src/domain.com/myproject/app/details.go:302 +0x8c0
created by domain.com/myproject/app/newDetails
	D:/Go/src/domain.com/myproject/app/details.go:121 +0x150

goroutine 311 [chan send, 1453 minutes]:
domain.com/myproject/app.(*details).getData2(0xc2085566c0, 0xc2084c7ce0)
	D:/Go/src/domain.com/myproject/app/details.go:393 +0xf61
created by domain.com/myproject/app/newDetails
	D:/Go/src/domain.com/myproject/app/details.go:122 +0x172

[Dave Cheney]

Please post the entirety of the stack dump as requested. 

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/YJPPpTQhBpo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dmitry Vyukov]

On Tue, Oct 7, 2014 at 3:43 PM, <michael...@googlemail.com> wrote:
> Thanks for your replies!
>
>
> There is indeed one handler, where I'm using the following pattern (and
> currently only there in production) and I always asked myself if this is
> correct. Here is the (pseudo) code:
>
>
> http://play.golang.org/p/gDl7tOo7Z6
>
>
> It would be nice if someone can have a look and give me some feedback,
> because this is the way I usually retrieve data from the database. Until now
> without any problems.
>
>
> Dave, maybe the mysql database of our production server has different
> settings than on my local machine and can't handle the concurrency. I don't
> know what this should be, but maybe I set the GOMAXPROCS back to 1 to find
> out.

No, this is not the case. The goroutine are blocked on chan send. So
it's something in your app. Find out who is intended to read from the
other end of the chan, and why it is not reading.

[michael...@googlemail.com]

Dave, sorry for the dumb question, but how should I get the stack dump after I quit my app? I didn't implement something like https://gist.github.com/Mistobaan/2892704 yet. So here is again the almost full stack dump from the pprof handler while running my app (I had to delete a few lines to not exceed the limit):

http://pastebin.com/vxM8zQQ8

Thanks Dmitry, I'll try to investigate this issue, but I don't know why the behavior is different on the production server.

[Dave Cheney]

I believe the error is on line 56 of your pseudo code. If one error is returned the the handler will return without receiving the second value, blocking the sender from exiting.

Tbh, I would make your life easier by not using goroutines and channels here. There is enough concurrency in handling each request in its own goroutine, which the http server does for you automatically.

You're going to be limited by the database server anyway.

[michael...@googlemail.com]

You are right, Dave! This is the problem. And the errors occuring were sql.ErrNoRows errors which I treated somehow differently. Now I can try to fix it.
And yes, maybe I overdid it here with goroutines and channels, but I wanted to test things out. ;-)

Thank you both for your help!!

Michael