Akka SSLSession leak when running Akka with native TLS

[Sean Gibbons]

Hi all,

I have been working with a native TLS Akka HTTP service deployed to Production. We have noticed memory increasing consistently throughout the week until our Akka service died due to memory constraints. 

Running a JProfiler locally I've managed to reproduce what I believe to be a leak in SSL related classes just using Akka code. 

 

I am using Akka HTTP version 10.1.5 and Akka version 2.5.18.

I ran a comparison of a non native TLS Akka server vs a native TLS Akka server. The load test consisted of sending around 20 req/s to a dummy endpoint that simply just returns a hardcoded "hello" string.

A complete runnable example of the Akka code used to produce this leak can be found as dummy-app.zip along with JProfiler snapshots in this Google Drive link - https://drive.google.com/drive/folders/1Q1zgN4m5J4oI_S0TupMs1LO1UVlJfh87?usp=sharing

With Native TLS:

JProfiler snapshots associated with this image are included in the Google Drive link above and named:
 - TLSLocal1.jps
 - TLSLocal2.jps
 - TLSLocal3.jps

Each blue line above is when a snapshot was taken.

Without Native TLS:

JProfiler snapshots associated with this image are included in the Google Drive link above and named:
 - NoTLSLocal1.jps
 - NoTLSLocal2.jps
 - NoTLSLocal3.jps

Each blue line above is when a snapshot was taken.

Any help is much appreciated on this topic thanks.

Séanadh Ríomhphoist/
Email Disclaimer
Tá an ríomhphost seo agus aon chomhad a sheoltar leis faoi rún agus is lena úsáid ag an seolaí agus sin amháin é. Is féidir tuilleadh a léamh anseo. 
This e-mail and any files transmitted with it are confidential and are intended solely for use by the addressee. Read more here. 

[johannes...@lightbend.com]

Hi Sean,

thanks for the comprehensive report. What do you mean with a native vs non-native TLS server? Is the example app for the "native TLS" server?

Johannes

[Sean Gibbons]

Hi Johannes thanks for the reply,

I used the example code for both the native and non-native TLS Akka servers. In the case of the non-native TLS server setup I simply changed the Http.bind() method to not include the setupTls parameter e.g. Http().bindAndHandle(routes, "0.0.0.0", 17715)

[Sean Gibbons]

Just to clarify by non-native TLS I meant to say "no TLS" i.e. simply an insecure AKKA server.

[Sean Gibbons]

And by native TLS I just mean a standard TLS AKKA Server, apologies for any confusion.

[Johannes Rudolph]

I see. Thanks.

With the provided code I couldn't reproduce the issue at least in the quick tests I did. Could you run

jmap -histo:live <pid> on the command line when some memory has accrued and send the output here (or in private)?

Johannes

On Thu, Nov 22, 2018 at 1:16 PM Sean Gibbons <sean.g...@mail.dcu.ie> wrote:

And by native TLS I just mean a standard TLS AKKA Server, apologies for any confusion.

Séanadh Ríomhphoist/
Email Disclaimer
Tá an ríomhphost seo agus aon chomhad a sheoltar leis faoi rún agus is lena úsáid ag an seolaí agus sin amháin é. Is féidir tuilleadh a léamh anseo. 
This e-mail and any files transmitted with it are confidential and are intended solely for use by the addressee. Read more here. 

--
*****************************************************************************************************
** New discussion forum: https://discuss.akka.io/ replacing akka-user google-group soon.
** This group will soon be put into read-only mode, and replaced by discuss.akka.io
** More details: https://akka.io/blog/news/2018/03/13/discuss.akka.io-announced
*****************************************************************************************************
>>>>>>>>>>
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/b6VtlNFLsr8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[Sean Gibbons]

I've ran the test again locally using the example server and attached is my output for 3 heap dumps over a 40 minutes time period

Notice the growth in SSLSessionImpl count between 'grep ssl snapShot1', 'grep ssl snapShot2' and 'grep ssl snapShot3' 
snapShot1 was taken at the start of the process, snapShot2 was taken after about 10 minutes and snapShot3 was taken about 30 minutes after snapShot2

How my test is setup:
- Start the dummy Akka service supplied in the google drive link with TLS enabled
- Run the following script testEndpoint.sh 20 times so as to have 20 req/s incoming

#!/bin/bash

set -B                  # enable brace expansion

for i in {1..100000}; do

  curl -s -k 'GET' 'https://localhost:17715/test'

  sleep 1

done

- I ran this script using './testEndpoint.sh &'


In the google drive link from the opening post I have supplied JProfiler snapshots that also show the heap usage which I would recommend viewing for additional verification.

[Patrik Nordwall]

Hi Sean,

I tried your sample app and ran 20 clients as you described. Attached profiler and looked at memory snapshot. There are a few thousand SSLSessionImpl objects but most of them are not strong reachable, so they will be garbage collected when needed.

To be convinced you can try to run with a smaller maximum heap, such as -Xmx128m, which will trigger garbage collection more aggressively. If there is a real memory leak you will eventually see OutOfMemoryError.

Regards,

Patrik

Séanadh Ríomhphoist/
Email Disclaimer
Tá an ríomhphost seo agus aon chomhad a sheoltar leis faoi rún agus is lena úsáid ag an seolaí agus sin amháin é. Is féidir tuilleadh a léamh anseo. 
This e-mail and any files transmitted with it are confidential and are intended solely for use by the addressee. Read more here. 

--
*****************************************************************************************************
** New discussion forum: https://discuss.akka.io/ replacing akka-user google-group soon.
** This group will soon be put into read-only mode, and replaced by discuss.akka.io
** More details: https://akka.io/blog/news/2018/03/13/discuss.akka.io-announced
*****************************************************************************************************
>>>>>>>>>>
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---

You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.

To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Patrik Nordwall
Akka Tech Lead
Lightbend -  Reactive apps on the JVM
Twitter: @patriknw

[Sean Gibbons]

Thanks for the response Patrik, we have tested with a smaller heap size and it appears our applications memory does remain stable with no appearance of a leak.  

I believe the reason we were seeing our instances fail was actually due to setting the heap size too large and our instances were using up all available memory on the box they were running on.

Appreciate the help debugging this issue.

All the best,

Sean

[Patrik Nordwall]

You're welcome.

Séanadh Ríomhphoist/
Email Disclaimer
Tá an ríomhphost seo agus aon chomhad a sheoltar leis faoi rún agus is lena úsáid ag an seolaí agus sin amháin é. Is féidir tuilleadh a léamh anseo. 
This e-mail and any files transmitted with it are confidential and are intended solely for use by the addressee. Read more here. 

--
*****************************************************************************************************
** New discussion forum: https://discuss.akka.io/ replacing akka-user google-group soon.
** This group will soon be put into read-only mode, and replaced by discuss.akka.io
** More details: https://akka.io/blog/news/2018/03/13/discuss.akka.io-announced
*****************************************************************************************************
>>>>>>>>>>
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.