Re: [akka-user] ForkJoin pool spawns and kill threads very rapidly

[√iktor Ҡlang]

How many actors are using that dispatcher, for how long do they live, how what's your average messages per second per actor and how many processors does your machine have (HT included)?

On Wed, May 29, 2013 at 4:05 PM, Jakub Kozlowski <ma...@jakub-kozlowski.com> wrote:

Hi,

I am experiencing weird ForkJoin pool behaviour, I was hoping if someone could explain to me if it's expected. 

Essentially, the ForkJoin pool is killing and creating worker threads very rapidly (about every second), if the app experiences a period of relative inactivity. As an example, after about an hour of running the app, the thread count (i.e. what shows up in logs as thread names) can go as high as ~700, which sounds rather excessive.

Is there any way to configure the inactivity period after which a worker thread should die, or is it the expected behaviour?

Below you have the config.

I would appreciate any help.

Thanks,

Jakub

fork-join-executor {
                # Min number of threads to cap factor-based parallelism number to

                parallelism-min = 8

                # The parallelism factor is used to determine thread pool size using the

                # following formula: ceil(available processors * factor). Resulting size

                # is then bounded by the parallelism-min and parallelism-max values.

                parallelism-factor = 1.0

                # Max number of threads to cap factor-based parallelism number to

                parallelism-max = 24
            }

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Viktor Klang
Director of Engineering

Typesafe

Twitter: @viktorklang

[√iktor Ҡlang]

Hi!

On Thu, May 30, 2013 at 11:42 AM, Jakub Kozlowski <ma...@jakub-kozlowski.com> wrote:

Hi,

Thanks for the prompt reply.

How many actors are using that dispatcher, for how long do they live

So far we have only done QA testing (so nowhere near production load), but essentially there is a pool of "global actors" (about 3, and that number is not going to increase) and (for lack of better name) "session" actors, which are spawned when a "session" is started and long lived (they currently never die). So for your testing we had a few of those, so I would say ~10 actors total.

And they are all configured to use that dispatcher?

 

how what's your average messages per second

For this testing it was probably ~10, so low

And how many actors received messages how frequently?

 

how many processors does your machine have (HT included)

4 cores

Would you suggest that the load was too small for this configuration? 

No, my suspicion is/was that threads are idling for too long (FJ worker threads automatically shut down after 4 seconds of nothing to do)

OR, you are generating fatal errors which cause the threads to die, but that should manifest in your logs.

Why is you max pool size 24 if you only have 4 cores?

Cheers,

√

[√iktor Ҡlang]

On Thu, May 30, 2013 at 2:18 PM, Jakub Kozlowski <ma...@jakub-kozlowski.com> wrote:

And they are all configured to use that dispatcher?

Yep.

And how many actors received messages how frequently?

Hard to say exactly, but essentially the activity is very bursty, as the engine that connects to this actor system might get an event and as a result of this send multiple messages.

(FJ worker threads automatically shut down after 4 seconds

Ah, that's probably why. Is there any way to configure this parameter, or not really?

No, that's hardcoded in FJP IIRC.

 

Why is you max pool size 24 if you only have 4 cores?

Answer is "I probably found if on some blog...", I didn't really tinker with those settings yet, but your "(FJ worker threads automatically shut down after 4 seconds" answers resolves my suspicions.

Yep, you might want to have a lower number of parallelism or switch to a thread-pool-executor that will keep threads alive.

Cheers,

√

 

Thanks,

Jakub 

[Patrik Nordwall]

On Thu, May 30, 2013 at 2:25 PM, √iktor Ҡlang <viktor...@gmail.com> wrote:

On Thu, May 30, 2013 at 2:18 PM, Jakub Kozlowski <ma...@jakub-kozlowski.com> wrote:

And they are all configured to use that dispatcher?

Yep.

And how many actors received messages how frequently?

Hard to say exactly, but essentially the activity is very bursty, as the engine that connects to this actor system might get an event and as a result of this send multiple messages.

(FJ worker threads automatically shut down after 4 seconds

Ah, that's probably why. Is there any way to configure this parameter, or not really?

No, that's hardcoded in FJP IIRC.

 

Why is you max pool size 24 if you only have 4 cores?

Answer is "I probably found if on some blog...", I didn't really tinker with those settings yet, but your "(FJ worker threads automatically shut down after 4 seconds" answers resolves my suspicions.

Yep, you might want to have a lower number of parallelism or switch to a thread-pool-executor that will keep threads alive.

or schedule a dummy message every 3rd second

--

Patrik Nordwall
Typesafe -  The software stack for applications that scale
Twitter: @patriknw

[√iktor Ҡlang]

On Thu, May 30, 2013 at 2:27 PM, Patrik Nordwall <patrik....@gmail.com> wrote:

On Thu, May 30, 2013 at 2:25 PM, √iktor Ҡlang <viktor...@gmail.com> wrote:

On Thu, May 30, 2013 at 2:18 PM, Jakub Kozlowski <ma...@jakub-kozlowski.com> wrote:

And they are all configured to use that dispatcher?

Yep.

And how many actors received messages how frequently?

Hard to say exactly, but essentially the activity is very bursty, as the engine that connects to this actor system might get an event and as a result of this send multiple messages.

(FJ worker threads automatically shut down after 4 seconds

Ah, that's probably why. Is there any way to configure this parameter, or not really?

No, that's hardcoded in FJP IIRC.

 

Why is you max pool size 24 if you only have 4 cores?

Answer is "I probably found if on some blog...", I didn't really tinker with those settings yet, but your "(FJ worker threads automatically shut down after 4 seconds" answers resolves my suspicions.

Yep, you might want to have a lower number of parallelism or switch to a thread-pool-executor that will keep threads alive.

or schedule a dummy message every 3rd second

per worker, and be sure that every worker gets one of them (workstealing y'know)

:)

[Patrik Nordwall]

On Thu, May 30, 2013 at 2:30 PM, √iktor Ҡlang <viktor...@gmail.com> wrote:

On Thu, May 30, 2013 at 2:27 PM, Patrik Nordwall <patrik....@gmail.com> wrote:

On Thu, May 30, 2013 at 2:25 PM, √iktor Ҡlang <viktor...@gmail.com> wrote:

On Thu, May 30, 2013 at 2:18 PM, Jakub Kozlowski <ma...@jakub-kozlowski.com> wrote:

And they are all configured to use that dispatcher?

Yep.

And how many actors received messages how frequently?

Hard to say exactly, but essentially the activity is very bursty, as the engine that connects to this actor system might get an event and as a result of this send multiple messages.

(FJ worker threads automatically shut down after 4 seconds

Ah, that's probably why. Is there any way to configure this parameter, or not really?

No, that's hardcoded in FJP IIRC.

 

Why is you max pool size 24 if you only have 4 cores?

Answer is "I probably found if on some blog...", I didn't really tinker with those settings yet, but your "(FJ worker threads automatically shut down after 4 seconds" answers resolves my suspicions.

Yep, you might want to have a lower number of parallelism or switch to a thread-pool-executor that will keep threads alive.

or schedule a dummy message every 3rd second

per worker, and be sure that every worker gets one of them (workstealing y'know)

:)

We had exactly this problem in Atmos where we had something scheduled at 5 seconds, and number of stopped/created threads improved a lot by scheduling a dummy message. I might be wrong, but it could easily be tried if you have this problem.

/Patrik

[√iktor Ҡlang]

On Thu, May 30, 2013 at 3:39 PM, Jakub Kozlowski <ma...@jakub-kozlowski.com> wrote:

Thanks a lot guys for your help.

For now we switched to thread-pool-executor, but as we enter production we'll consider our options.

It would be interesting to know if the algorithm in FJP depends on that particular timeout, or does it just happen to be non-configurable because "reasons"...

An interesting read: http://gee.cs.oswego.edu/cgi-bin/viewcvs.cgi/jsr166/src/jsr166e/ForkJoinPool.java?view=markup

(in the most recent version the IDLE_TIMEOUT is 2 seconds)

 

From what I've seen you use scala's copy of the JDK7 FJP, so it shouldn't be difficult to add a config for this, if it's possible...

We use the version for Java8, and if you want to make it configurable, that should go via jsr166(x), Scala does not wish to fork the FJP & have it diverge from upstream.

Cheers,

√

 

I'll look around

Thanks a lot,

Jakub