Akka Http memory leak suspect

[Bartosz Jankiewicz]

I have been running an app with Akka Http 1.0.9.

It had only single endpoint respoding with JSON. The service returned the value as future therefore I used onComlete semantics.

The app was consistently running into OoM issues. Heap dump analysis has led me to 1,536,693 instances of akka.actor.ActorCell. Along with accompanying objects (scala.collection.immutable.RedBlackTree$BlackTree)  it saturated the heap. All ActorCell objects seem to be related to Akka-Streams - their names are: StreamSupervisor-xxxxxx

Has anyone fallen into a similar issue?

[Konrad “ktoso” Malawski]

What are you doing in the app ;-)

Sounds to me like you’re starting tons of things that are never stopped.

—

Konrad `kto.so` Malawski

Akka @ Lightbend

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[Bartosz Jankiewicz]

Indeed, this is what it looks like :)

The only place where I use actors is in Akka Http routes. The service I'm using returns a future. The only clue I had was that the Future results returned from my service never evaluate could cause the stream workers to stack. But in such case I should also spot large number of Future objects on the heap but I don't observe that.

def campaign(): Route =
    path("campaign") {
      get {
        withRequestTimeout(1 second) {
          parameters('cookieId, 'ip, 'debug.?) { (cookieId, ip, debug) =>
            val verbose = debug match {
              case Some("missingCampaigns") => true
              case _ => false
            }
            val resultF: Future[EvaluationResult] = Free.liftF(EvaluateCampaign(cookieId, ip, verbose))
              .foldMap(evaluationService)
            onSuccess(resultF) { result =>
              complete(result)
            }
          }
        }
      }
    }

[Konrad Malawski]

This code seems unlikely to be the cause, likely somewhere else - run your app while profiling to find the leak.

To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscribe@googlegroups.com.

To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Cheers,

Konrad 'ktoso' Malawski

Akka @ Typesafe

[Bartosz Jankiewicz]

Precisely - I simplified the solution to eliminate all possible causes. As mentioned, I don't use actors anywhere except for this endpoint and instances of ActorCell pollute the heap. I've tried profiling and the observation is the same.

--

Cheers,

Konrad 'ktoso' Malawski

Akka @ Typesafe

--

>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---

You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/GSsa1akTdjQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to akka-user+...@googlegroups.com.

[Konrad “ktoso” Malawski]

Please share a reproducer if you indeed think it’s a leak in here.

Investigate *which* cells are, and *what* they contain; very likely it’s not “akka leaks”, but somewhere you’re starting and not stopping things.

—

Konrad `kto.so` Malawski

Akka @ Lightbend

[johannes...@lightbend.com]

Hi Bartosz,

I can look into the heap dump. You can send it to me privately. If that's not possible could you post an histogram? It would be great if that could be filtered once for subclasses of `Actor` (which will probably be dominated by `ActorGraphInterpreter`) and once filtered by `GraphStage` which might show which streams stay open.

Johannes

[Bartosz Jankiewicz]

Hi Johannes,

I can share the heap dump. I will try uploading it to Google Drive.

I'm still suspecting Akka Http. A few days ago I was forced to migrate to HTTP4S in emergency. After the migration of endpoints (no single other change applied) the heap dump utilisation is stable.

Bartosz

[Patrik Nordwall]

If the names are StreamSupervisor- I think it can be that a new Materializer is created for each request. I don’t know if that is done by your application or by Akka Http. Does that ring any bells? Do you have any creation of stream materializers in your code?

/Patrik

--

[Johannes Rudolph]

On Tue, Sep 26, 2017 at 7:18 AM, Patrik Nordwall <patrik....@gmail.com> wrote:

If the names are StreamSupervisor- I think it can be that a new Materializer is created for each request. I don’t know if that is done by your application or by Akka Http. Does that ring any bells? Do you have any creation of stream materializers in your code?

Ah good point. I just assumed that it would be child actors of the supervisor since they also have that in the name but if it's the supervisor itself, creating too many Materializers could really be the cause.

Johannes

 

[Bartosz Jankiewicz]

I have verified that but there are 2 places where declare the materializers. Both are declared as vals. I will verify the number of materializer instances on my heap-dump to confirm.

--

>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---

You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/GSsa1akTdjQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to akka-user+...@googlegroups.com.

[Johannes Rudolph]

Hi Bartosz,

I had a quick look into the dump. It contains >317000 StreamSupervisors, so creating too many materializers is really the issue. Note, that the materializer itself might go out of scope but the engine still stays alive if the materializer has not been shutdown manually.

I created https://github.com/akka/akka/issues/23736 to discuss if we could warn if the `Materializer` reference is not referenced any more but the infrastructure is still alive.

Johannes

On Tue, Sep 26, 2017 at 1:59 PM, Bartosz Jankiewicz <bartosz.j...@gmail.com> wrote:

I have verified that but there are 2 places where declare the materializers. Both are declared as vals. I will verify the number of materializer instances on my heap-dump to confirm.

On Tue, 26 Sep 2017 at 13:24 Johannes Rudolph <johannes.rudolph@lightbend.com> wrote:

On Tue, Sep 26, 2017 at 7:18 AM, Patrik Nordwall <patrik....@gmail.com> wrote:

If the names are StreamSupervisor- I think it can be that a new Materializer is created for each request. I don’t know if that is done by your application or by Akka Http. Does that ring any bells? Do you have any creation of stream materializers in your code?

Ah good point. I just assumed that it would be child actors of the supervisor since they also have that in the name but if it's the supervisor itself, creating too many Materializers could really be the cause.

Johannes

 

--

>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/GSsa1akTdjQ/unsubscribe.

To unsubscribe from this group and all its topics, send an email to akka-user+unsubscribe@googlegroups.com.

To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/GSsa1akTdjQ/unsubscribe.

To unsubscribe from this group and all its topics, send an email to akka-user+unsubscribe@googlegroups.com.

[Bartosz Jankiewicz]

Hi Johannes,

Thank you for looking into that.

This is really surprising finding. I cannot find anything wrong with my code though in this regards. There is one more materializer beining initialized, but for an endpoint that is rarely used (Swagger docs - I have verified in the logs, that there are only a few requests per day to that one).

implicit val materializer = ActorMaterializer()

val bindingFuture: Future[Http.ServerBinding] = Http().bindAndHandle(routes, host, port)
bindingFuture.onFailure {
  case ex: Exception =>
    logger.error(s"Failed to bind to $host, $port", ex)
}

On Wed, 27 Sep 2017 at 11:39 Johannes Rudolph <johannes...@lightbend.com> wrote:

Hi Bartosz,

I had a quick look into the dump. It contains >317000 StreamSupervisors, so creating too many materializers is really the issue. Note, that the materializer itself might go out of scope but the engine still stays alive if the materializer has not been shutdown manually.

I created https://github.com/akka/akka/issues/23736 to discuss if we could warn if the `Materializer` reference is not referenced any more but the infrastructure is still alive.

Johannes

On Tue, Sep 26, 2017 at 1:59 PM, Bartosz Jankiewicz <bartosz.j...@gmail.com> wrote:

I have verified that but there are 2 places where declare the materializers. Both are declared as vals. I will verify the number of materializer instances on my heap-dump to confirm.

On Tue, 26 Sep 2017 at 13:24 Johannes Rudolph <johannes...@lightbend.com> wrote:

On Tue, Sep 26, 2017 at 7:18 AM, Patrik Nordwall <patrik....@gmail.com> wrote:

If the names are StreamSupervisor- I think it can be that a new Materializer is created for each request. I don’t know if that is done by your application or by Akka Http. Does that ring any bells? Do you have any creation of stream materializers in your code?

Ah good point. I just assumed that it would be child actors of the supervisor since they also have that in the name but if it's the supervisor itself, creating too many Materializers could really be the cause.

Johannes

 

--

>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/GSsa1akTdjQ/unsubscribe.

To unsubscribe from this group and all its topics, send an email to akka-user+...@googlegroups.com.

To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/GSsa1akTdjQ/unsubscribe.

To unsubscribe from this group and all its topics, send an email to akka-user+...@googlegroups.com.

To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/GSsa1akTdjQ/unsubscribe.

To unsubscribe from this group and all its topics, send an email to akka-user+...@googlegroups.com.