Akka Streams 2.5 sudden throughput drop

[Dmitry Golubets]

Hi,

We have a custom data processing pipeline in the company built on Akka Streams, that has been happily working for a year on 2.4 version.
Recently I've started migrating to 2.5 and it went very smooth, but.. There is a huge throughput drop at some random time point.

So, we measure it in messages per second.
With 2.4 we always had 21-22k with some small fluctuations.
With 2.5 it starts well, even faster, but then at some point (6 hours or 24 hours, kinda random) it drops to ~15k, which is quite significant 25% drop.

User code is the same between versions.
The only thing that had to be added is .async boundaries in many places - cos in 2.4 we had fusing disabled.
I tried both default ForkJoinPool and new AffinityPool and both have the same issues.

Profiling with JvisualVM pretty much gives me nothing.
I compared snapshots between application running fast and slow. The only difference I see is CPU load ( faster app has higher load).
Memory consumption is the same.
Sampling shows nothing suspicious.
Threads usage looks the same.

So, the only assumption I could make is that it is has to do something with internal akka dispatching\scheduling of tasks or some deeper internal change.

So far I failed to reproduce it in some simple isolated example.
I can't even reliably do it in the application itself.
However I noticed that sometimes I can trigger it by forcing full GC, but again, not every time, still random..

I hope that either someone has experienced similar issues or has more idea of what has changed internally in 2.5 that could lead to this problem?
I will appreciate any ideas! :)

[Dmitry Golubets]

One thing I want to add, is that sometimes there is actually one difference that I see in sampling times:
fast: akka.dispatch.forkjoin.ForkJoinPool.runWorker()    1.592451    69,724 ms (1.6%)    69,724 ms    1,989,681 ms    864,210 ms
slow: akka.dispatch.forkjoin.ForkJoinPool.runWorker()    0.6935879    5,440 ms (0.7%)    5,440 ms    1,593,446 ms    456,197 ms

[Patrik Nordwall]

Difficult to say what it can be without knowing what your code looks like. I suggest that you try to minimize it and share a reproducer or example.

/Patrik

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.