Go scheduler tracing questions

[envee]

Hi All,

I have an application which always has about 2000 goroutines running. (I checked this by querying prometheus metrics at regular intervals.) This number sounds reasonably correct to me because at the start I launch 1000 goroutines and each of these 1000 goroutines spawns 1 goroutine each to make an HTTP/2 request to a server.

I run this application on a 40 vCPU machine which is quite lightly loaded. Using atop or top, I can see that the overall CPU utilization is very low before I start my application. Infact, it is idle for 99% of the time before I run my application.

top - 11:01:03 up 255 days, 23:33,  2 users,  load average: 0.04, 0.05, 0.05
Tasks: 525 total,   1 running, 524 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  0.1 sy,  0.0 ni, 99.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 52815753+total, 47459616+free,  4746712 used, 48814668 buff/cache
KiB Swap:  4194300 total,  4194300 free,        0 used. 51626032+avail Mem 

Now  I turned on the go scheduler trace as below and run my application.

GODEBUG=schedtrace=10 ./myapp start -c config.yaml

I always see all of the local runqueues of the procs with a value of 0. Also, my idleprocs is always equal to the total number of logical procs on my machine (i.e. 40).

SCHED 11639ms: gomaxprocs=40 idleprocs=40 threads=46 spinningthreads=0 idlethreads=41 runqueue=0 [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]

Does this mean that the go scheduler is not scheduling the goroutines fast enough ?  I expected to see all the local run queues to conain a value of atleast 1 when my application is running.

In my application I make a lot of HTTP/2 calls at the rate of say 1000 rps or 2000 rps etc. so it is a pretty busy application I would imagine.

I noticed only 1 instance where a lot of goroutines assigned to the local run queue of some processors. And in this instance I can see that there are very few idleprocs.

SCHED 11926ms: gomaxprocs=40 idleprocs=2 threads=46 spinningthreads=1 idlethreads=3 runqueue=80 [2 0 0 0 2 1 0 0 2 0 0 0 4 0 0 0 0 0 0 0 0 0 0 0 0 162 0 0 0 0 0 0 0 0 0 0 0 0 0 0]

Is there  some other diagnostic to look at to explain why this is happening ?

[Ian Lance Taylor]

A goroutine that is waiting for network I/O will not show up in the
schedtrace list. Whether this is expected or not really depends on
what your goroutines are doing. If they are doing a lot of
computation then they should be there. But if they are just putting
together a simple network request, sending it off, and waiting for a
reply, then it is normal for your goroutines to be mostly idle.

Ian