Where does the maximum number of threads comes from?

[Paulo Moura]

Hi,

I'm wondering where does the maximum number of threads comes from? On my Mac OS X system, I can create ~2K threads. Is this limit inherited from the operating-system or is it defined somewhere in the SWI-Prolog sources? Sorry if I missed the explanation somewhere in the documentation.

Cheers,
Paulo

-----------------------------------------------------------------
Paulo Moura
Logtalk developer

Email: <mailto:pmo...@logtalk.org>
Web: <http://logtalk.org/>
-----------------------------------------------------------------

[Jan Wielemaker]

On 06/03/2016 03:46 PM, Paulo Moura wrote:
> Hi,
>
> I'm wondering where does the maximum number of threads comes from? On
> my Mac OS X system, I can create ~2K threads. Is this limit inherited
> from the operating-system or is it defined somewhere in the
> SWI-Prolog sources? Sorry if I missed the explanation somewhere in
> the documentation.

There is no limit in SWI-Prolog. On my Linux box (64 bit, 32Gb main
mem), 30,000 works fine. Above that I get a memory resource error
exception. Not sure you want that many though. The main resource cost is
the C-stack that gets reserved for the thread. I don't know how well
modern OSes handle really large number of threads.

Note that since two releases we have delimited continuations. These
allow you to make `threads' as well. Unlike system threads however these
run cooperatively on the same stacks and thus can share variables. They
are way cheaper as no C stack needs to be reserved, nor is there a need
for the overall state involved with a thread (stacks, I/O, local
predicates, etc.) I wonder how suitable they would be for simulations.

So, basically the scheduler maintains a database of continuations. It
selects the next task to run and calls the continuation. The task
continues and if it has done its job it calls shift/1 to return control
to the scheduler.


Cheers --- Jan

[Paulo Moura]

Hi Jan,

Thanks for the quick reply and the tip on using continuations as an alternative. Meanwhile, regarding Mac OS X, I found relevant information here:

https://support.apple.com/en-us/HT3854

Creating a large number of threads can, of course, be problematic but in the scenarios I'm considering only a small percentage of them would be active at any given time.

Cheers,

Paulo

[Jan Wielemaker]

On 06/03/2016 05:22 PM, Paulo Moura wrote:
> Hi Jan,
>
> Thanks for the quick reply and the tip on using continuations as an
> alternative. Meanwhile, regarding Mac OS X, I found relevant
> information here:
>
> https://support.apple.com/en-us/HT3854
>
> Creating a large number of threads can, of course, be problematic but
> in the scenarios I'm considering only a small percentage of them
> would be active at any given time.

That might be precisely the scenario where continuations may come
handy. Of course, you only use one CPU if you do it that way and
it is _cooperative_: the coroutine must yield control explicitly
to the scheduler.

Curious ...

Cheers --- Jan

[Paulo Moura]

Again in the specific case of Mac OS X, I found the answer:

$ sysctl kern.num_threads
kern.num_threads: 10240

$ sysctl kern.num_taskthreads
kern.num_taskthreads: 2048

The second value, kern.num_taskthreads, is the maximum number of threads per process and it's a read only value.