Akka Cluster is shutting down after upgrading to Play 2.3.2

[Manikandan Kaliyaperumal]

Hi,

I have been running my Jobs using distributed Worker with Master in PLay 2.2.3 without any issues until now.

But, it's started causing issues after upgrading the Play version to 2.3.2.

It is starting up initially and after a while it is not accepting any work and later it's shutting down automatically.

Not sure what is causing this issue. I have just changed only the play version. All other stack (and code base) remains same - Akka 2.3.4 with Persistance and remoting (2.3.4).

[INFO] [08/06/2014 23:03:48.353] [QuoteClusterSystem-akka.actor.default-dispatcher-18] [akka://QuoteClusterSystem/user/producer] Produce.onReceive is called with UserTick:

[INFO] [08/06/2014 23:03:48.353] [QuoteClusterSystem-akka.actor.default-dispatcher-18] [akka://QuoteClusterSystem/user/producer] Produced work: 755

[INFO] [08/06/2014 23:03:53.375] [WorkersSystem-akka.actor.default-dispatcher-7] [akka://WorkersSystem/user/worker] No ack from master, retrying (00b5d3fe-d906-4b05-a9e9-74fbd248b630 -> 323649e0-33b5-49b4-8734-ce02eb383ad1)

[INFO] [08/06/2014 23:03:53.376] [QuoteClusterSystem-akka.actor.default-dispatcher-3] [akka://QuoteClusterSystem/user/master/active] Work 323649e0-33b5-49b4-8734-ce02eb383ad1 not in progress, reported as done by worker 00b5d3fe-d906-4b05-a9e9-74fbd248b630

[INFO] [08/06/2014 23:03:53.378] [QuoteClusterSystem-akka.actor.default-dispatcher-18] [akka://QuoteClusterSystem/user/master/active] Work 9d0376cb-b65e-499d-8787-4cc9ef51afbc not in progress, reported as done by worker 7ba10a7b-c7d8-41ad-a291-5f740fa3e422

[INFO] [08/06/2014 23:03:53.381] [QuoteClusterSystem-akka.actor.default-dispatcher-14] [akka://QuoteClusterSystem/user/master/active] Work 635ea450-e1f0-47dd-8ef3-5bca9069b217 not in progress, reported as done by worker 831637b1-e3a2-4cd1-bb70-0db0f4745156

[INFO] [08/06/2014 23:03:53.421] [WorkersSystem-akka.actor.default-dispatcher-6] [akka://WorkersSystem/user/worker] No ack from master, retrying (33dab9fa-a8b8-44af-b637-5c9ea89cd174 -> 2eb29b49-953f-40ac-a70f-d10b810da6c6)

[INFO] [08/06/2014 23:03:53.422] [QuoteClusterSystem-akka.actor.default-dispatcher-3] [akka://QuoteClusterSystem/user/master/active] Work 2eb29b49-953f-40ac-a70f-d10b810da6c6 not in progress, reported as done by worker 33dab9fa-a8b8-44af-b637-5c9ea89cd174

User  ==> 1000272547

User  ==> 1000272120

User  ==> 1000273974

User  ==> 1000275489

User  ==> 1000270351

User  ==> 1000273884

User  ==> 1000270803

User  ==> 1000273539

[INFO] [08/06/2014 23:03:56.384] [QuoteClusterSystem-akka.actor.default-dispatcher-7] [akka://QuoteClusterSystem/user/producer] Produce.onReceive is called with UserTick:

[INFO] [08/06/2014 23:03:56.384] [QuoteClusterSystem-akka.actor.default-dispatcher-7] [akka://QuoteClusterSystem/user/producer] Produced work: 756

[ERROR] [08/06/2014 23:03:56.400] [QuoteClusterSystem-akka.actor.default-dispatcher-16] [ActorSystem(QuoteClusterSystem)] Uncaught error from thread [QuoteClusterSystem-akka.actor.default-dispatcher-16] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled

java.lang.NoSuchMethodError: com.google.common.io.Closeables.closeQuietly(Ljava/io/Closeable;)V

        at org.iq80.leveldb.impl.MMapLogWriter.close(MMapLogWriter.java:83)

        at org.iq80.leveldb.impl.DbImpl.makeRoomForWrite(DbImpl.java:832)

        at org.iq80.leveldb.impl.DbImpl.writeInternal(DbImpl.java:658)

        at org.iq80.leveldb.impl.DbImpl.write(DbImpl.java:647)

        at akka.persistence.journal.leveldb.LeveldbStore$class.withBatch(LeveldbStore.scala:89)

        at akka.persistence.journal.leveldb.SharedLeveldbStore.withBatch(LeveldbStore.scala:127)

        at akka.persistence.journal.leveldb.LeveldbStore$class.writeMessages(LeveldbStore.scala:44)

        at akka.persistence.journal.leveldb.SharedLeveldbStore.writeMessages(LeveldbStore.scala:127)

        at akka.persistence.journal.leveldb.SharedLeveldbStore$$anonfun$receive$1.applyOrElse(LeveldbStore.scala:131)

        at akka.actor.Actor$class.aroundReceive(Actor.scala:465)

        at akka.persistence.journal.leveldb.SharedLeveldbStore.aroundReceive(LeveldbStore.scala:127)

        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)

        at akka.actor.ActorCell.invoke(ActorCell.scala:487)

        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)

        at akka.dispatch.Mailbox.run(Mailbox.scala:220)

        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)

        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)

        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)

        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)

        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

[INFO] [08/06/2014 23:03:56.607] [application-akka.remote.default-remote-dispatcher-9] [akka.tcp://appli...@sv2lxfascsint01.corp.equinix.com:59490/system/remoting-terminator] Shutting down remote daemon.

[INFO] [08/06/2014 23:03:56.620] [application-akka.remote.default-remote-dispatcher-9] [akka.tcp://appli...@sv2lxfascsint01.corp.equinix.com:59490/system/remoting-terminator] Remote daemon shut down; proceeding with flushing remote transports.

Any pointers on this to fix this would be very much helpful here.

Thanks,

Mani

[√iktor Ҡlang]

java.lang.NoSuchMethodError: com.google.common.io.Closeables.closeQuietly(Ljava/io/Closeable;)V

NSME is essentially always a sign of classpath issues, either having the wrong version of the lib on the classpath, the wrong version first on the classpath or not having the dependency on the classpath.

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Cheers,

√

[Manikandan Kaliyaperumal]

Hi,

Thanks for the details. I am looking into this. But the shut down starts well ahead of this and running with "no ack" for sometime before Closing it. Definitely i will look into this NSME issue, but this cluster shutdown is caused by some thing else? The same code is running fine in Play 2.2.3.

Thanks,

Mani

[INFO] [08/06/2014 23:03:56.607] [application-akka.remote.default-remote-dispatcher-9] [akka.tcp://application@sv2lxfascsint01.corp.equinix.com:59490/system/remoting-terminator] Shutting down remote daemon.

[INFO] [08/06/2014 23:03:56.620] [application-akka.remote.default-remote-dispatcher-9] [akka.tcp://application@sv2lxfascsint01.corp.equinix.com:59490/system/remoting-terminator] Remote daemon shut down; proceeding with flushing remote transports.

Any pointers on this to fix this would be very much helpful here.

Thanks,

Mani

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Cheers,

√

[Michael Pisula]

Hi Mani,

I had the same issue (NSME in the leveldb code) when I bumped an application to akka 2.3.4 and guava 17. As soon as I dropped the guava dependency everything was fine again. 

I just checked the pom and Play 2.3 seems to have a dependency on Guava 16, while the leveldb in akka-persistence 2.3.4 has a dependency on Guava 12 (?). 

There seems to be an issue with backwards compatibility in guava...

I did not find a direct dependency on Guava in Play 2.2.3, so that might explain why everything goes haywire after the version bump. As for a possible solution, perhaps someone with more sbt knowledge than myself can suggest something... ;-) Worst case, I guess you could switch to a different journal implementation that does not depend on such an outdated guava version.

Cheers,

Michael

[INFO] [08/06/2014 23:03:56.607] [application-akka.remote.default-remote-dispatcher-9] [akka.tcp://application@sv2lxfascsint01.corp.equinix.com:59490/system/remoting-terminator] Shutting down remote daemon.

[INFO] [08/06/2014 23:03:56.620] [application-akka.remote.default-remote-dispatcher-9] [akka.tcp://application@sv2lxfascsint01.corp.equinix.com:59490/system/remoting-terminator] Remote daemon shut down; proceeding with flushing remote transports.

[Michael Pisula]

Just verified, the method Closable.closeQuietly was deprecated since at least guava 14, and was removed in guava 16. Guava 17 then introduced two closeQuietly methods, which have different parameter types though...

Cheers,

Michael

[Manikandan Kaliyaperumal]

Thanks Michael - your suggestions worked well! And sorry for the delayed response - got stuck in BTN.

Thanks,

Mani