too many runtime.gcBgMarkStartWorkers ?

[Ganesh Sangle]

Hi,

While examining the stack trace from a crash where the process seems to have become extremely slow, I see a lot of these goroutines:

goroutine 66 [mark worker (idle)]:

runtime.gopark(0x2cf6cf0, 0xc820d10000, 0x29268b0, 0x12, 0x14, 0x0)

        /usr/local/go/src/runtime/proc.go:185 +0x163 fp=0xc82006ff50 sp=0xc82006ff28

runtime.gcBgMarkWorker(0xc820016000)

        /usr/local/go/src/runtime/mgc.go:1289 +0xf7 fp=0xc82006ffb8 sp=0xc82006ff50

runtime.goexit()

        /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc82006ffc0 sp=0xc82006ffb8

created by runtime.gcBgMarkStartWorkers

        /usr/local/go/src/runtime/mgc.go:1239 +0x93

There are 65 such goroutines according to stack trace. Can someone help me understand what could be going on ?

1. Why does go need so many routines gcBgMarkWorkers ?

2. Does it indicate that the process is thrashing memory somehow ? Does it indicate to any specific symtoms ?

3. Can it make the process super slow ?

Thanks for the help ! 

[Dave Cheney]

On Wednesday, 28 December 2016 19:06:42 UTC+11, Ganesh Sangle wrote:

Hi,

While examining the stack trace from a crash where the process seems to have become extremely slow, I see a lot of these goroutines:

goroutine 66 [mark worker (idle)]:

runtime.gopark(0x2cf6cf0, 0xc820d10000, 0x29268b0, 0x12, 0x14, 0x0)

        /usr/local/go/src/runtime/proc.go:185 +0x163 fp=0xc82006ff50 sp=0xc82006ff28

runtime.gcBgMarkWorker(0xc820016000)

        /usr/local/go/src/runtime/mgc.go:1289 +0xf7 fp=0xc82006ffb8 sp=0xc82006ff50

runtime.goexit()

        /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc82006ffc0 sp=0xc82006ffb8

created by runtime.gcBgMarkStartWorkers

        /usr/local/go/src/runtime/mgc.go:1239 +0x93

There are 65 such goroutines according to stack trace. Can someone help me understand what could be going on ?

1. Why does go need so many routines gcBgMarkWorkers ?

There should be only as many background workers as CPUs in your host. Have you, or a library you use, altered GOMAXPROCS? Does your machine have 64 or more cpus?

 

2. Does it indicate that the process is thrashing memory somehow ? Does it indicate to any specific symtoms ?

No, these workers are idle

 

3. Can it make the process super slow ?

If there were all running, maybe, but they are idle. These are just normal goroutines and the runtime is built to handle hundreds of thousands of idle goroutines.

[Ganesh Sangle]

Thanks Dave, we do set the GOMAXPROCS to 64. thanks for explanation. 

[Dave Cheney]

There may be a bug here. IMO the runtime should never try to start more that numCPUs background workers regardless of the size of GOMAXPROCS as that'll just cause contention during the GC cycle.

[r...@golang.org]

The default is GOMAXPROCS == numCPU and the runtime is optimized and tested for this. There are use cases involving co-tenancy where setting GOMAXPROCS < numCPU tells the OS to limit HW allocation and improves overall throughput when several programs are running concurrently. 

Setting GOMAXPROCS > numCPU seems to indicate that the Go scheduler and the OS scheduler are out of sync. Perhaps the delay between the OS knowing a call is blocked and the Go scheduler knowing it is blocked is the root cause. Any insight into why setting GOMAXPROCS > numCPU is a win might lead to improving the Go scheduler or perhaps Go / OS interaction.

[John Souvestre]

Ø  Any insight into why setting GOMAXPROCS > numCPU is a win might lead to improving the Go scheduler or perhaps Go / OS interaction.

 

I’ve found it useful if you happen to have a number of compute-bound goroutines which don’t reschedule.  Setting more processes than CPUs causes the kernel to do some scheduling for them.

 

John

    John Souvestre - New Orleans LA

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[r...@golang.org]

Will making tight loops preemptable (CL 10958) resolve this use case?

Is it true that some of the goroutines are compute bound while others have to respond to some sort of stimulus?  What response time do these  goroutines required? 1ms, 10ms, 100ms, 1second?

Thanks