help with thread limit

[Steve Roth]

I am running Go code on a shared web hosting server from a major hosting company.  Using cgroups, they limit the number of threads any user can create to 25.  Up until a week ago, they had me running on a server with 24 cores, and everything worked fine.  Now they've moved me to a new server with 128 cores.  The Go runtime thinks it should be able to create that many threads, and it can't, and all of my Go programs crash with, "runtime: failed to create new OS thread".

Setting GOMAXPROCS doesn't help this, since it only controls the number of active threads, not the total number of threads.  The Go runtime still thinks it can create as many threads as there are cores, and crashes the program when it's blocked from doing so.

Is there any way around this?  Or is Go just completely unusable in an environment where the per-user process limit is less than the number of cores?

Many thanks,

Steve

[Robert Engels]

You can use cpuctrl or similar to limit the number of visible cores when you start the process. 

On Jan 31, 2024, at 8:43 PM, Steve Roth <st...@rothskeller.net> wrote:



--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAAnpqKHvfni_U%2BkCHp0hhUryDytPzQ%2BSqHZTcAFH9HNXgRCy_A%40mail.gmail.com.

[Steve Roth]

Thanks to the people who suggested how to limit the number of apparent cores.  Unfortunately, doing that didn't solve the problem.  I have the number of cores now limited to two — confirmed by checking runtime.NumCPU() — but the program is still trying to allocate more than 25 threads, and crashing when cgroups won't let it.

Surely there must be some way to get a Go program to run successfully in an environment with process limits?  Any further suggestions would be greatly appreciated.

Regards,

Steve

[Steven Hartland]

To be honest I would question if its a usable solution if they are limiting that low on threads, so is the fix to switch or have the provider increase the limit to a sensible number?

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAAnpqKEzZcFxDkG2Qtzf25hPwkZq2EstbDbYtBz4CtTaBQxqWA%40mail.gmail.com.

[Steve Roth]

Fair question.  In their view, 25 threads per user is a sensible limit.  Their mindset is based around single-threaded PHP.  And for that reason among others, yes, switching providers is definitely something I will pursue.  But that will be a long-term effort.  Right now I'm looking for a short-term solution to get my site back up.

Thanks,

Steve

[Bruno Albuquerque]

In Go, you can not really limit the maximum number of threads that will be created. The runtime itself will create threads whenever needed for things like blocking CGO calls, garbage collection, network polling, etc. As I understand it, GOMAXPROCS only limits the number of threads that would be spawned to run your actual Go code (i.e. if you create 10 goroutines and you have GOMAXPROCS set to 4, those will all be multiplexed in 4 OS threads at most).

-Bruno

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAAnpqKH%2BpawFZ2uqC0-wZm2vidngmePhS_%3Dgqfh9iwKPyN%2Bdcg%40mail.gmail.com.

[Nagaev Boris]

Steve,

I checked with strace and top -p (press H to see threads).

It seems that Go can create few more threads than cores are given by taskset.

For example, on an 8 core ARM machine in the cloud, I got these
results on a program launching 100 goroutines utilizing a lot of CPU
each.

$ strace --follow-forks -o /tmp/m2.log taskset --cpu-list 0 ./main
1
^C
$ strace --follow-forks -o /tmp/m2.log taskset --cpu-list 0-7 ./main
8
^C
$ grep -c clone /tmp/m.log
3
$ grep -c clone /tmp/m2.log
13

So for 8 cores it launched 13 threads, for 1 core - 3 threads.

You can try running your tool under "strace --follow-forks" and count
how many "clone" calls there are in the log.

Try to set just 1 core in taskset (taskset --cpu-list 0) and see if it
works and how many threads it launches. If it works, try to increase
the number of cores in taskset until it breaks.

Here is my program:

package main

import (
"sync/atomic"
"fmt"
"runtime"
"time"
)

func main() {
fmt.Println(runtime.NumCPU())
var b atomic.Int64
for i := 0; i < 100; i++ {
go func() {
for {
b.Add(1)
time.Sleep(1000)
}
}()
}
time.Sleep(1000*time.Second)

> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAAnpqKEzZcFxDkG2Qtzf25hPwkZq2EstbDbYtBz4CtTaBQxqWA%40mail.gmail.com.



--

Best regards,
Boris Nagaev

[Steven Hartland]

Sounds sensible to me. 

I haven't read the entire thread, pun intended ;-), so just checking if you tried debug.SetMaxThreads?

[Steven Hartland]

Ignore that as it just triggers a panic if the limit is exceeded, however based on that I'm guessing it's not going to be possible.