Suddenly increasing inter-actor message delivery time

[Alexey Shuksto]

Hello hAkkers!

We've got some very strange message delivery time pattern between actors:

We have system with ~2000 type "A" working actors, each of whom have 1 to 50 type "B" sub-workers (who do actual work, but do it very fast -- >1ms between request and response).

Every type A actor every second receives 1 to 50 (equal to number of sub-workers) payloads of 1 to 10 messages (1 to 500 messages total), chooses one type B actor per payload (1 to 10 messages), forwards them and interpreting B-actor result.

Number of "in-payload" messages is dependent of 'second-per-minute' -- most messages are received in 29th and 59th seconds.

Usually a total time of message processing is around 1 to 5ms for a full circle: in -> A -> B -> A -> out.

But in the "high load" times processing time quickly escalates up to 2.5 _seconds_.

After some investigation and providing of separate dispatchers for type A (FJE, parallelism 8 min, 64 max, 3.0 factor) and type B (same configuration) actors, we were able to determine that type A actor still receive messages at very fast rate, but type B actors...

At the start of processing they receive payload almost momentarily (0 to 1ms latency), but as processing continues, time to deliver message from A to B starts increasing up to 2.5 seconds mentioned above.

We tried to tweak type B dispatcher and set SingleConsumerOnlyUnboundedMailbox for them to no effect at all.

From hardware side we have dual 6 core Intel server class processors (24 cores total), JVM has 32GB of ram (no swapping), G1GC, gc pauses do not exceed 100ms and happens usually every 10-15 seconds.

Is there anything else that we can tweak or use to pinpoint the problem? May be some metric for average actor queue size and per-actor dispatcher time?

[Viktor Klang]

Version?

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscribe@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Cheers,

√

[Алексей Шуксто]

2.4.10.

Чт, 8 сент. 2016 г. в 19:49, Viktor Klang <viktor...@gmail.com>:

To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.

To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Cheers,

√

--

>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---

You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/baZ8uatd3IA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to akka-user+...@googlegroups.com.

[Rob Crawford]

Kamon (kamon.io) has some of those metrics.

http://kamon.io/integrations/akka/actor-router-and-dispatcher-metrics/

Can you reproduce the problem easily? If so, try it with a Kamon-instrumented build and see what you get. 

[Patrik Nordwall]

If it's cpu bound work I would recommend one dispatcher for all with slightly less threads than cores. Set max to cores - 1. Too many threads will just make things slower for cpu bound work.

The reason for not using all cores is that it can be good to have some spare capacity for other manegement tasks. You have to experiment with what is best for your system.

/Patrik

--