Go's 10000 thread limit

[lab...@linux.vnet.ibm.com]

Hi all,

I know this topic has been brought up in the past but I haven't seen it lately.

In the Go programming language there is a limit of 10000 threads (pthreads not goroutines).  Once this limit is hit an error is issue and the program dies with a fatal error.

When running Docker and creating containers, each uses 3-4 threads so that when trying to scale to a large number of containers this limit is hit.

Seems like this thread limit should be based on the characteristics of the system or changeable.  I know it can be increased with a change to source code to call SetMaxThreads but sometimes changing the source and rebuilding is not always possible. 

- Lynn

[Brad Fitzpatrick]

On Fri, Apr 1, 2016 at 7:05 AM, <lab...@linux.vnet.ibm.com> wrote:

When running Docker and creating containers, each uses 3-4 threads so that when trying to scale to a large number of containers this limit is hit.

How so?

What's the Go code in question?

 

[lab...@linux.vnet.ibm.com]

The go code is in runtime/proc.go:

func schedinit() {
        // raceinit must be the first call to race detector.
        // In particular, it must be done before mallocinit below calls racemapshadow.
        _g_ := getg()
        if raceenabled {
                _g_.racectx = raceinit()
        }

        sched.maxmcount = 10000  <---- this is the limit

Then in this function, if the limit is hit it gets a fatal error:

func checkmcount() {
        // sched lock is held
        if sched.mcount > sched.maxmcount {
                print("runtime: program exceeds ", sched.maxmcount, "-thread limit\n")
                throw("thread exhaustion")
        }
}

Output looks like this:

Mar 31 14:53:28 docker5 docker[2300]: runtime: program exceeds 10000-thread limit
Mar 31 14:53:28 docker5 docker[2300]: fatal error: thread exhaustion

I've seen this discussed before and the response was something like your app shouldn't be creating so many threads. 
But I don't think Go should be making the thread limit decision, let the failure happen when the system resources run out.

I can open an issue on this if that is preferred.

[Brad Fitzpatrick]

No, I meant the Go user code. Is the code in question in Docker itself, or are you talking about some code that interfaces with Docker?

Is this something that Docker cares about? Is there an upstream Docker bug?

--

You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[lab...@linux.vnet.ibm.com]

If someone tries to create more than 2500 containers with Docker they will hit this limit.  Since it is a limit in the Go language, it doesn't matter what platform you are running on or what Docker version.

I'm not aware there is a Docker issue on it yet but if not we will create one.

[Brad Fitzpatrick]

But why is Docker using 3-4 threads per container?

This conversation really seems best suited for a Docker mailing list or bug tracker. Quick searches suggest Docker falls apart around 1,000 containers anyway for unrelated reasons. Maybe they don't care about the thread limit yet, or optimizing their threads/container.

[minux]

On Apr 1, 2016 2:22 PM, <lab...@linux.vnet.ibm.com> wrote:
> If someone tries to create more than 2500 containers with Docker they will hit this limit.  Since it is a limit in the Go language, it doesn't matter what platform you are running on or what Docker version.

I think Brad's question is more about why Docker needs 4 threads per container.

[lab...@linux.vnet.ibm.com]

Agreed, we'll post this to Docker mailing list.  Thanks for the responses.

[Pierre Durand]

Do you think that running Docker in Docker could solve the issue?

You would have multiple Docker/Go processes and it increases the threads limit.

[Brad Fitzpatrick]

On Sat, Apr 2, 2016 at 6:27 AM, Pierre Durand <pierre...@gmail.com> wrote:

Do you think that running Docker in Docker could solve the issue?

You would have multiple Docker/Go processes and it increases the threads limit.

Off by one error: April Fools' was yesterday.

[lab...@linux.vnet.ibm.com]

I found this issue on the topic.  https://github.com/docker/docker/issues/15256.

[Ged Wed]

Found a way to increase the 10,000 thread limit:https://golang.org/pkg/runtime/debug/#SetMaxThreads

debug.SetMaxThreads(20000)