Play Framework 2.5 WSClient OutOfMemoryError ?

[Clément Poissonnier]

My API crash when it is used by many users, after a several minutes. This API uses other REST WS, using the WSClient of Play Framework.

The root cause is not provided (cf. https://groups.google.com/forum/#!topic/play-framework/GBYt71GAFFg), but I strongly suspect the WSClient to be the cause of the crash (that's the only akka related part of the application).

Anyway, is there any configuration to make sure that there is no memory leak while using WSClient? Any known issues about using WSClient when you have a lot of requests?

Thanks ! 

[Will Sargent]

There's a note if you're creating it manually: https://www.playframework.com/documentation/2.5.x/JavaWS#Directly-creating-WSClient

Note: If you create a WSClient manually then you must call client.close() to clean it up when you’ve finished with it. Each client creates its own thread pool. If you fail to close the client or if you create too many clients then you will run out of threads or file handles -— you’ll get errors like “Unable to create new native thread” or “too many open files” as the underlying resources are consumed.

--

Will Sargent

Engineer, Lightbend, Inc.

will.s...@lightbend.com

--
You received this message because you are subscribed to the Google Groups "Play Framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to play-framework+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/play-framework/72a44d7e-2595-4740-a5ea-bb77a8875c9d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Clément Poissonnier]

I saw that, but WSClient is injected ...

Le jeudi 23 mars 2017 00:56:16 UTC+1, Will Sargent a écrit :

There's a note if you're creating it manually: https://www.playframework.com/documentation/2.5.x/JavaWS#Directly-creating-WSClient

Note: If you create a WSClient manually then you must call client.close() to clean it up when you’ve finished with it. Each client creates its own thread pool. If you fail to close the client or if you create too many clients then you will run out of threads or file handles -— you’ll get errors like “Unable to create new native thread” or “too many open files” as the underlying resources are consumed.

--

Will Sargent

Engineer, Lightbend, Inc.

will.s...@lightbend.com

On Wed, Mar 22, 2017 at 9:50 AM, Clément Poissonnier <clement.p...@gmail.com> wrote:

My API crash when it is used by many users, after a several minutes. This API uses other REST WS, using the WSClient of Play Framework.

The root cause is not provided (cf. https://groups.google.com/forum/#!topic/play-framework/GBYt71GAFFg), but I strongly suspect the WSClient to be the cause of the crash (that's the only akka related part of the application).

Anyway, is there any configuration to make sure that there is no memory leak while using WSClient? Any known issues about using WSClient when you have a lot of requests?

Thanks ! 

--
You received this message because you are subscribed to the Google Groups "Play Framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to play-framewor...@googlegroups.com.

[Will Sargent]

The only other thing I can think of is that you haven't specified a maximum number of connections in AHC and the default settings are too high.

Someone tweeted about this recently: https://twitter.com/alexelcu/status/844594076431368192

but didn't file a bug or reproducing case, so I don't know if it was just specific to his situation.

--

Will Sargent

Engineer, Lightbend, Inc.

will.s...@lightbend.com

On Wed, Mar 22, 2017 at 9:50 AM, Clément Poissonnier <clement.p...@gmail.com> wrote:

--

You received this message because you are subscribed to the Google Groups "Play Framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to play-framework+unsubscribe@googlegroups.com.

[Clément Poissonnier]

That's not the first time I've got this kind of issue, few requests are enough to make the application crash ... (it did happened in a 2.2.5 app as well).

The most annoying thing is that I don't even know what kind of error makes the application crash since the log only say

2017-03-22 16:47:15,637 - [[error]] - akka.actor.ActorSystemImpl - application-akka.actor.default-dispatcher-79 - Uncaught error from thread [application-akka.actor.default-dispatcher-77] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled -  

(cf. https://groups.google.com/forum/#!topic/play-framework/GBYt71GAFFg)

It seems that when a make a WSClient call, threads are created, but some of then are never destroyed. I'm going to try to find the time to file an issue and create a project to reproduce this issue.

(I'm seriously thinking about using another http client lib as well, because the application is in production and crash every 5-60 minutes)

[Igmar Palsenberg]

I doubt it's the client lib, it's most likely your code. We have a service that does a lot of callouts, and it survives a constant stream of callout a a high rate (consuming > 80% CPU) for at least half a day.

I suggest you enable proper logging, to see at what point the actor system decides to kill the JVM.

Igmar

 

[Clément Poissonnier]

About the threads, my bad, that was completely false. That true for the first requests, but that's because there is pooling involved.

I just figured out that my logback conf was bad (I though that akka activity was logged as "play" ... no comment) ... I just have to wait for the application to crash again and see akka logs.

[Clément Poissonnier]

Still no logs with a configuration that seems good for akka ...

<configuration> <conversionRule conversionWord="coloredLevel" converterClass="play.api.Logger$ColoredLevel" />

  <appender name="FILE" class="ch.qos.logback.core.FileAppender">
     <file>/home1/log/application.log</file>
     <encoder>
       <pattern>%date - [%coloredLevel] - %logger - %thread - %message - %xException%n</pattern>
     </encoder>
   </appender>

  <logger name="play" level="DEBUG" />
  <logger name="application" level="DEBUG" />
  <logger name="akka" level="DEBUG" />

  <root level="DEBUG">
    <appender-ref ref="FILE" />
  </root>

</configuration>

2017-03-24 14:14:57,659 - [[error]] - akka.actor.ActorSystemImpl - application-akka.actor.default-dispatcher-104 - Uncaught error from thread [application-akka.actor.default-dispatcher-100] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled -

[Clément Poissonnier]

I managed to get the real exception by running the application in development mode. The cause exception was displayed in console ... The issue was in our code.

Unfortunately, the cause exception is not logged at all, I'm going to propose an enhancement to Akka.

Thanks Will and Igmar !

[Hakim Hauston]

Hi,

Would you mind sharing your problem and fixes?

I am having just 1000+ request/sec here with the injected WSClient, but it hangs after just a few seconds and sometimes the JVM crashed with OutOfMemory Error. I have tried playing around with play.ahc.maxConnectionPerHost too with no avail...