Play 2.6.2, web server becomes unresponsive, possible deadlock?
473 views
Skip to first unread message
harv...@cse.ohio-state.edu
Aug 17, 2017, 5:16:18 PM8/17/17
to Play Framework
Hi All,
I have a basic CRUD app that is implemented in Scala using Play 2.6.2. Every once in a while, the application will just stop responding and I have to kill SBT using CTRL+C. Here is a thread dump with the thread that I think may be the most likely culprit.
There does not appear to be anything in the stack trace that would help me trace the problem to my own code. Is there anything else I should send to help diagnose? Any workarounds?
Thanks!
William
"application-akka.actor.default-dispatcher-2" #88 prio=5 os_prio=0 tid=0x000000001ff21800 nid=0x3744 waiting on condition [0x000000002cf8d000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000dd5b4618> (a scala.concurrent.impl.Promise$CompletionLatch)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at scala.concurrent.impl.Promise$DefaultPromise.tryAwait(Promise.scala:238)
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:254)
at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:259)
at scala.concurrent.Await$.$anonfun$result$1(package.scala:215)
at scala.concurrent.Await$$$Lambda$150/1013255889.apply(Unknown Source)
at akka.dispatch.MonitorableThreadFactory$AkkaForkJoinWorkerThread$$anon$3.block(ThreadPoolBuilder.scala:167)
at akka.dispatch.forkjoin.ForkJoinPool.managedBlock(ForkJoinPool.java:3641)
at akka.dispatch.MonitorableThreadFactory$AkkaForkJoinWorkerThread.blockOn(ThreadPoolBuilder.scala:165)
at akka.dispatch.BatchingExecutor$BlockableBatch.blockOn(BatchingExecutor.scala:106)
at scala.concurrent.BatchingExecutor$Batch.blockOn(BatchingExecutor.scala:97)
at scala.concurrent.Await$.result(package.scala:142)
at play.core.server.DevServerStart$$anon$1.get(DevServerStart.scala:200)
- locked <0x00000000828dcdb8> (a play.core.server.DevServerStart$$anon$1)
at play.core.server.netty.PlayRequestHandler.resultUtils(PlayRequestHandler.scala:68)
at play.core.server.netty.PlayRequestHandler.$anonfun$handleAction$5(PlayRequestHandler.scala:290)
at play.core.server.netty.PlayRequestHandler$$Lambda$1043/1712309872.apply(Unknown Source)
at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:302)
at scala.concurrent.Future$$Lambda$943/326620663.apply(Unknown Source)
at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:37)
at scala.concurrent.impl.Promise$$Lambda$944/1587777865.apply(Unknown Source)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
at play.api.libs.streams.Execution$trampoline$.executeScheduled(Execution.scala:109)
at play.api.libs.streams.Execution$trampoline$.execute(Execution.scala:71)
at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68)
at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:284)
at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:284)
at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284)
at scala.concurrent.Promise.$anonfun$tryCompleteWith$1(Promise.scala:71)
at scala.concurrent.Promise.$anonfun$tryCompleteWith$1$adapted(Promise.scala:71)
at scala.concurrent.Promise$$Lambda$127/1004349422.apply(Unknown Source)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
at scala.concurrent.BatchingExecutor$Batch.processBatch$1(BatchingExecutor.scala:63)
at scala.concurrent.BatchingExecutor$Batch.$anonfun$run$1(BatchingExecutor.scala:78)
at scala.concurrent.BatchingExecutor$Batch$$Lambda$151/2133659219.apply$mcV$sp(Unknown Source)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
at scala.concurrent.BatchingExecutor$Batch.run(BatchingExecutor.scala:55)
at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:864)
at scala.concurrent.BatchingExecutor.execute(BatchingExecutor.scala:106)
at scala.concurrent.BatchingExecutor.execute$(BatchingExecutor.scala:103)
at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:862)
at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68)
at scala.concurrent.impl.Promise$KeptPromise$Kept.onComplete(Promise.scala:368)
at scala.concurrent.impl.Promise$KeptPromise$Kept.onComplete$(Promise.scala:367)
at scala.concurrent.impl.Promise$KeptPromise$Successful.onComplete(Promise.scala:375)
at scala.concurrent.Promise.tryCompleteWith(Promise.scala:71)
at scala.concurrent.Promise.tryCompleteWith$(Promise.scala:69)
at scala.concurrent.impl.Promise$DefaultPromise.tryCompleteWith(Promise.scala:183)
at scala.concurrent.Promise.completeWith(Promise.scala:63)
at scala.concurrent.Promise.completeWith$(Promise.scala:63)
at scala.concurrent.impl.Promise$DefaultPromise.completeWith(Promise.scala:183)
at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:40)
at scala.concurrent.impl.Promise$$Lambda$944/1587777865.apply(Unknown Source)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$Lambda$942/2002232552.apply$mcV$sp(Unknown Source)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:38)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:43)
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Locked ownable synchronizers:
- None
I have a basic CRUD app that is implemented in Scala using Play 2.6.2. Every once in a while, the application will just stop responding and I have to kill SBT using CTRL+C. Here is a thread dump with the thread that I think may be the most likely culprit.
There does not appear to be anything in the stack trace that would help me trace the problem to my own code. Is there anything else I should send to help diagnose? Any workarounds?
Thanks!
William
"application-akka.actor.default-dispatcher-2" #88 prio=5 os_prio=0 tid=0x000000001ff21800 nid=0x3744 waiting on condition [0x000000002cf8d000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000dd5b4618> (a scala.concurrent.impl.Promise$CompletionLatch)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at scala.concurrent.impl.Promise$DefaultPromise.tryAwait(Promise.scala:238)
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:254)
at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:259)
at scala.concurrent.Await$.$anonfun$result$1(package.scala:215)
at scala.concurrent.Await$$$Lambda$150/1013255889.apply(Unknown Source)
at akka.dispatch.MonitorableThreadFactory$AkkaForkJoinWorkerThread$$anon$3.block(ThreadPoolBuilder.scala:167)
at akka.dispatch.forkjoin.ForkJoinPool.managedBlock(ForkJoinPool.java:3641)
at akka.dispatch.MonitorableThreadFactory$AkkaForkJoinWorkerThread.blockOn(ThreadPoolBuilder.scala:165)
at akka.dispatch.BatchingExecutor$BlockableBatch.blockOn(BatchingExecutor.scala:106)
at scala.concurrent.BatchingExecutor$Batch.blockOn(BatchingExecutor.scala:97)
at scala.concurrent.Await$.result(package.scala:142)
at play.core.server.DevServerStart$$anon$1.get(DevServerStart.scala:200)
- locked <0x00000000828dcdb8> (a play.core.server.DevServerStart$$anon$1)
at play.core.server.netty.PlayRequestHandler.resultUtils(PlayRequestHandler.scala:68)
at play.core.server.netty.PlayRequestHandler.$anonfun$handleAction$5(PlayRequestHandler.scala:290)
at play.core.server.netty.PlayRequestHandler$$Lambda$1043/1712309872.apply(Unknown Source)
at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:302)
at scala.concurrent.Future$$Lambda$943/326620663.apply(Unknown Source)
at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:37)
at scala.concurrent.impl.Promise$$Lambda$944/1587777865.apply(Unknown Source)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
at play.api.libs.streams.Execution$trampoline$.executeScheduled(Execution.scala:109)
at play.api.libs.streams.Execution$trampoline$.execute(Execution.scala:71)
at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68)
at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:284)
at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:284)
at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284)
at scala.concurrent.Promise.$anonfun$tryCompleteWith$1(Promise.scala:71)
at scala.concurrent.Promise.$anonfun$tryCompleteWith$1$adapted(Promise.scala:71)
at scala.concurrent.Promise$$Lambda$127/1004349422.apply(Unknown Source)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
at scala.concurrent.BatchingExecutor$Batch.processBatch$1(BatchingExecutor.scala:63)
at scala.concurrent.BatchingExecutor$Batch.$anonfun$run$1(BatchingExecutor.scala:78)
at scala.concurrent.BatchingExecutor$Batch$$Lambda$151/2133659219.apply$mcV$sp(Unknown Source)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
at scala.concurrent.BatchingExecutor$Batch.run(BatchingExecutor.scala:55)
at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:864)
at scala.concurrent.BatchingExecutor.execute(BatchingExecutor.scala:106)
at scala.concurrent.BatchingExecutor.execute$(BatchingExecutor.scala:103)
at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:862)
at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68)
at scala.concurrent.impl.Promise$KeptPromise$Kept.onComplete(Promise.scala:368)
at scala.concurrent.impl.Promise$KeptPromise$Kept.onComplete$(Promise.scala:367)
at scala.concurrent.impl.Promise$KeptPromise$Successful.onComplete(Promise.scala:375)
at scala.concurrent.Promise.tryCompleteWith(Promise.scala:71)
at scala.concurrent.Promise.tryCompleteWith$(Promise.scala:69)
at scala.concurrent.impl.Promise$DefaultPromise.tryCompleteWith(Promise.scala:183)
at scala.concurrent.Promise.completeWith(Promise.scala:63)
at scala.concurrent.Promise.completeWith$(Promise.scala:63)
at scala.concurrent.impl.Promise$DefaultPromise.completeWith(Promise.scala:183)
at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:40)
at scala.concurrent.impl.Promise$$Lambda$944/1587777865.apply(Unknown Source)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$Lambda$942/2002232552.apply$mcV$sp(Unknown Source)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:38)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:43)
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Locked ownable synchronizers:
- None
Igmar Palsenberg
Aug 18, 2017, 4:04:07 AM8/18/17
to Play Framework
Hi All,
I have a basic CRUD app that is implemented in Scala using Play 2.6.2. Every once in a while, the application will just stop responding and I have to kill SBT using CTRL+C. Here is a thread dump with the thread that I think may be the most likely culprit.
There does not appear to be anything in the stack trace that would help me trace the problem to my own code. Is there anything else I should send to help diagnose? Any workarounds?
at play.core.server.DevServerStart$$anon$1.get(DevServerStart.scala:200)
Why are you running it in dev mode ?
Igmar
harv...@cse.ohio-state.edu
Aug 18, 2017, 9:22:57 AM8/18/17
to Play Framework
Good question - we run it in dev mode while we are doing development. For production deployment we use Play Framework's SBT "dist" task to create a binary that we run on the production server.
Having the application hang during dev is a productivity killer because web server has to be hard-killed (along with SBT), then SBT restarted, then the web server started again, then the application loaded, then the page-under-development navigated to, etc. That amounts to several worker-hours per day, day in, day out.
-William
Having the application hang during dev is a productivity killer because web server has to be hard-killed (along with SBT), then SBT restarted, then the web server started again, then the application loaded, then the page-under-development navigated to, etc. That amounts to several worker-hours per day, day in, day out.
-William
Igmar Palsenberg
Aug 18, 2017, 11:10:44 AM8/18/17
to Play Framework
Good question - we run it in dev mode while we are doing development. For production deployment we use Play Framework's SBT "dist" task to create a binary that we run on the production server.
Good.
Having the application hang during dev is a productivity killer because web server has to be hard-killed (along with SBT), then SBT restarted, then the web server started again, then the application loaded, then the page-under-development navigated to, etc. That amounts to several worker-hours per day, day in, day out.
Can you put a complete stackdump on a gist ? I'm not convinced that the thread mentioned is actually the cause (I t hink it's the change monitor thread). I've also never experienced this, and we have a pretty large Play stack. Can you also post info on
the environment used (OS, JVM version) ? That might be useful.
Igmar
harv...@cse.ohio-state.edu
Aug 18, 2017, 4:10:07 PM8/18/17
to Play Framework
Complete stack dump is available here: https://gist.github.com/harveywi/f6376eaf2f4f1ab6acdca9da519ac7f7
OS: Windows 7 Enterprise
JVM: Oracle 1.8.0_141
Thanks!
OS: Windows 7 Enterprise
JVM: Oracle 1.8.0_141
Thanks!
Will Sargent
Aug 19, 2017, 12:56:44 AM8/19/17
to play-fr...@googlegroups.com
This could be fixed in 2.6.3: https://github.com/playframework/playframework/pull/7644
--
You received this message because you are subscribed to the Google Groups "Play Framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to play-framework+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/play-framework/48eb9aa7-1d25-45eb-9f12-6d4c0450b1d5%40googlegroups.com.
Rich Dougherty
Aug 20, 2017, 11:14:19 PM8/20/17
to play-framework
I believe this was fixed in Play 2.6.2. It doesn't mean there's not another deadlock somewhere else though! ;)
Can you reproduce this issue with a minimal Play app?
– Rich
To view this discussion on the web visit https://groups.google.com/d/msgid/play-framework/CAAvUidMh_0Xu21y1oivVAyiQtxD-5whqG-dPBUrZz--woFeKaQ%40mail.gmail.com.
Reply all
Reply to author
Forward
0 new messages