goroutine priority

[Zhao Weng]

Hi gophers, 

I'm doing a research on how to prioritise some goroutines over others.So I can allocate more CPU resource to more important part of the program.

I try to do it by calling runtime.LockOSThread to assign the goroutine needs to be prioritise to a designated OS thread, and unix.SchedSetAttr to prioritise that OS thread. Here is the code:

```go

package main

import (

"flag"

"fmt"

"runtime"

"sync"

"time"

"golang.org/x/sys/unix"

)

var (

nPrioritizedGoroutines = flag.Int("p", 1, "# of prioritized goroutines")

nNormalGoroutines = flag.Int("n", 1, "# of normal goroutines")

restDuration = flag.Duration("r", 0, "rest for a certain amount of time between works")

)

func prioritizeThread() {

// set thread priority to the highest

a := unix.SchedAttr{

Size: unix.SizeofSchedAttr,

Policy: 1,

Priority: 99,

}

if err := unix.SchedSetAttr(0, &a, 0); err != nil {

panic(err)

}

}

func doWorks(workerId int) {

t := time.Now()

for i := 0; i < 100; i++ {

st := time.Now()

res := 0

for ii := 0; ii < 1e9; ii++ {

res += ii

}

fmt.Printf("%d@%d, timecost: %s, res: %d \n", workerId, unix.Gettid(), time.Since(st), res)

// sleep for a while to simulate gaps between requests.

if *restDuration > 0 {

time.Sleep(*restDuration)

}

}

fmt.Printf("total execute time for worker: %d is %s\n", workerId, time.Since(t))

}

func main() {

flag.Parse()

runtime.GOMAXPROCS(*nPrioritizedGoroutines)

var wg sync.WaitGroup

workerId := 0

for i := 0; i < *nPrioritizedGoroutines; i++ {

wg.Add(1)

go func(workerId int) {

// assign goroutine to a designated thread

runtime.LockOSThread()

// prioritize this thread

prioritizeThread()

defer wg.Done()

doWorks(workerId)

}(workerId)

workerId++

}

for i := 0; i < *nNormalGoroutines; i++ {

wg.Add(1)

go func(workerId int) {

defer wg.Done()

doWorks(workerId)

}(workerId)

workerId++

}

wg.Wait()

}

```

compile on linux, and run with command `sudo ./sche`, it seems not working, CPU resource is shared by two thread, and two goroutine execute `doWorks` in similar timecost(1.5 seconds).

sudo ./sche

1@255429, timecost: 1.475347182s, res: 499999999500000000
0@255425, timecost: 1.517077413s, res: 499999999500000000
1@255429, timecost: 1.473167148s, res: 499999999500000000
0@255425, timecost: 1.515322146s, res: 499999999500000000
1@255429, timecost: 1.494751901s, res: 499999999500000000
0@255425, timecost: 1.532692691s, res: 499999999500000000

while with 1 goroutine only, the timecost will be 0.75 seconds

sudo ./sche -n 0
0@257072, timecost: 751.18938ms, res: 499999999500000000
0@257072, timecost: 747.364725ms, res: 499999999500000000
0@257072, timecost: 745.362553ms, res: 499999999500000000
0@257072, timecost: 748.353778ms, res: 499999999500000000

Am I doing anything wrong here?

Zhao Weng

[TheDiveO]

Are you running this on a multi-core? Your non-fifo tasks can be scheduled to other cores, et cetera. BTW, fifo 99 is a recipe for desaster as it can starve the kernel on a core, preventing necessary kernel house-keeping. Don't ask me how I know...

What is the reason for setting GOMAXPROCS, I'm at a loss here, so appreciating help! IIRC, OS-thread locked goroutines don't count against this limit.

This looks like a situation where I personally would fire off recording kernel scheduler tracepoint events and then having a close look at in kernelshark at what really is going on. There are syscalls in the FIFO scheduled path that might cause preemption. I would look at sched_switch here, as well as sched_switch_new. Maybe print the TIDs so you can find your go-routine related tasks in all the scheduler event noise, but maybe the binary's comm (name) might be already sufficient if you give it a catchy name.

A steeper learning curve would be to prepare a bpftrace script matching on the comm/name on the scheduler trace events, to get an idea of whats going on.

[robert engels]

What you want to do is start the process with a cset to restrict the cores it can use - then use the setsched to move certain threads to cores that have been excluded.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/27ad2fb1-97df-43eb-84a9-1c83669ad2e2n%40googlegroups.com.

[robert engels]

Also, you can configure the kernel at boot to exclude certain cpus from the OS completely so no process can interfere with them - BUT - processes/threads can still be moved to these isolated cpus using setsched().

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/1EA11541-07A1-4A85-8775-F256EC2338EB%40ix.netcom.com.

[TheDiveO]

you need to keep a bunch of per-cpu ("per-core") kernel threads and you need to make sure not to starve them, but for a short test that's okay. oh, and don't have any processes running that use io uring...

[robert engels]

I am not sure that is true unless things have changed. You can configure the interrupt/kernel handlers to run on specific cpus and keep other cpus and their caches completely isolated. There was an issue at one point with clock timings but as I recall it was resolved (it’s been several years so I don’t remember the specifics).

We were using the Linux with real-time kernel.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/131cb640-3de2-4ae0-b833-e3fd75c865c3n%40googlegroups.com.

[TheDiveO]

Seems to have changed, as there have been quite some more kthread-types sprung up. Oh, you're reminding me ask the OP a question just to be sure...

[TheDiveO]

just to be sure: you do actually use an RT kernel? I know you can set ff on stock non-RT kernels, but the results can be quite different.

On Thursday, September 26, 2024 at 5:49:53 AM UTC+2 Zhao Weng wrote: