Memory leak with akka-streams 2.0.1?

[Clelio De Souza]

Hi,

We use Akka Streams for streaming live data points via Websocket client connections in our project. We have this in production and it has been working fine for some time.

I have been working on upgrading to Akka Streams to 2.0.1 and we noticed a significantly downgrade of resource management and/or potential memory leak.

To demonstrate that I have pushed a small project called "websocket-stream" to my Github account https://github.com/cleliofs/websocket-stream - I have also taken some screen shots showing my findings.

The scenario I ran was 3 sequential executions for 500 requests each first with Akka Streams 2.0.1 and then I repeated the test with Akka Streams 1.0:


1) Akka Streams 2.0.1

a) See attachment "akka-streams-2.0.1-mem.png" (JMX Heap graph): Here we can see humps showing the memory allocation. Also after each execution (~ 2mins) I trigger GC to see how much of memory would be recollected. On this image we can see that progressively the amount of memory re-collected by GC was less and less.

b) See attachment "akka-streams-2.0.1-histogram.png" (JMX memory histogram): Here we can see a memory histogram of the allocated memory at the end of all 3 executions (~ 6mins). I would like to draw your attention to "akka.stream.FlowShape" class. For this class there are > 64k allocated objects occupying 1.7% of the whole heap size (and of course, this is also cascading to other objects, such as "akka.stream.GraphStageModule", "akka.stream.Outlet", "akka.stream.AbstractStage", "akka.stream.Inlet")


2) Akka Streams 1.0

a) See attachment "akka-streams-1.0-mem.png" (JMX Heap graph): Here at the start of the test, a similar pattern happens with some memory being allocated, but the huge difference occurs when I trigger GC (I did 3 times by clicking "Perform GC"). Visually we can see that all the time the memory being freed up via GC is significantly better with Akka Streams 1.0 in comparison with Akka Streams 2.0.1

b) See attachment "akka-streams-1.0-histogram.png" (JMX memory histogram): The same improvement can be seen for the memory histogram when the number of live objects for "akka.stream.FlowShape" at the end of the test (after performing 3 GCs) was reduced to only 3 (with only 96 bytes allocated - 0.0%).



So, clearly based on my tests Akka Streams 1.0 manages better the resources, by freeing them up when the websocket connections are closed, whereas the same does not apply for 2.0.1.

I added on my flow a PushStage to hook up the onPush and other methods such as onDownstreamFinish and postStop. Interestingly, for my Flow with 2.0.1 I can not see the "onDownstreamFinish" nor "postStop" being printed, only for version 1.0 of Akka Streams I can see those methods being invoked after a websocket client disconnection.

At the moment, unfortunately, we postponed the task for upgrading Akka Streams to 2.0 until we are confident this potential memory leak issue is fixed.

The source code is available at my Github repository (https://github.com/cleliofs/websocket-stream) if anyone would like to try that out.


Thanks,
Leo

[Johannes Rudolph]

Hi Leo,

when do you want the WS connections to be closed? I probably miss something but I don't see code on either side which would attempt to shutdown the connection.

Johannes

[Clelio De Souza]

Hi Johannes,

The WS connection can be closed directly via the client itself, as I am doing for this little programme I wrote (when the client app process gets killed).

For Akka Streams 1.0 that is happening fine (I can see a call to "onDownstreamFinish" and "postStop". However, the same is not happening for 2.0.1 and probably that is causing the memory leak.

Any ideas?

Thanks,
Leo

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/8tozN2SX--g/unsubscribe.
To unsubscribe from this group and all its topics, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[Andrey Kuznetsov]

We probably encountered the same problem. After upgrading to Streams 2.0.0, our server started to go down every day because OutOfMemory.

[Andrey Kuznetsov]

> I would like to draw your attention to "akka.stream.FlowShape" class. For this class there are > 64k allocated objects occupying 1.7% of the whole heap size (and of course, this is also cascading to other objects, such as "akka.stream.GraphStageModule", "akka.stream.Outlet", "akka.stream.AbstractStage", "akka.stream.Inlet")

In heap dump we saw the same types occupying unusually lots of memory. 

[Johannes Rudolph]

Hi,

after having a quick look into it (the fully working reproducer was a
great help btw), it turned out that this is indeed a regression
introduced shortly before the release. It is tracked here:
https://github.com/akka/akka/issues/19398

Johannes

> --
>>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>>> Check the FAQ:
>>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "Akka User List" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/akka-user/8tozN2SX--g/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> akka-user+...@googlegroups.com.
> To post to this group, send email to akka...@googlegroups.com.
> Visit this group at https://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.

--
Johannes

-----------------------------------------------
Johannes Rudolph
http://virtual-void.net

[Roland Kuhn]

Thanks for the quick response, Johannes!

> You received this message because you are subscribed to the Google Groups "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.

> To post to this group, send email to akka...@googlegroups.com.
> Visit this group at https://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.

Dr. Roland Kuhn
Akka Tech Lead
Typesafe – Reactive apps on the JVM.
twitter: @rolandkuhn