ActorSystem Uncaught fatal error shutting down ActorSystem !

3,324 views
Skip to first unread message

cie...@gmail.com

unread,
Jun 5, 2017, 3:16:44 AM6/5/17
to Akka User List

Hi all, this is my first post, i am poor in english, sorry~



[ERROR] [SECURITY][05/24/2017 16:41:54.422] [wssystem-akka.remote.default-remote-dispatcher-26] [akka.actor.ActorSystemImpl(wssystem)] Uncaught fatal error from thread [wssystem-akka.remote.default-remote-dispatcher-26] shutting down ActorSystem [wssystem]
                      [INFO] [05/24/2017 16:41:54.434] [wssystem-akka.remote.default-remote-dispatcher-5] [akka.tcp://wssy...@192.168.0.56:20300/system/remoting-terminator] Shutting down remote daemon.
                      [INFO] [05/24/2017 16:41:54.434] [wssystem-akka.remote.default-remote-dispatcher-5] [akka.tcp://wssy...@192.168.0.56:20300/system/remoting-terminator] Remote daemon shut down; proceeding with flushing remote transports.
                    [WARN] [05/24/2017 16:41:56.464] [wssystem-akka.actor.default-dispatcher-34] [akka.remote.Remoting] Shutdown finished, but flushing might not have been successful and some messages might have been dropped. Increase akka.remote.flush-wait-on-shutdown to a larger value to avoid this.
                      [INFO] [05/24/2017 16:41:56.464] [wssystem-akka.remote.default-remote-dispatcher-5] [akka.tcp://wssy...@192.168.0.56:20300/system/remoting-terminator] Remoting shut down.


i am got an error when using akka , i have no idea , the logs do not show detail whats error, please help me, thanks.



johannes...@lightbend.com

unread,
Jun 6, 2017, 6:55:27 AM6/6/17
to Akka User List
Hi,

in cases of fatal errors, the error and stack trace is logged to stderr (not using the logging framework). Note that in some cases, the logging itself may fail (that's why the error is fatal: after it happens, the state of the JVM might be corrupted and operations like logging may fail for various reasons).

Johannes

johannes...@lightbend.com

unread,
Jun 6, 2017, 7:56:03 AM6/6/17
to Akka User List
Hi,

my colleague Arnout just found out that the error will only be logged to stderr if you enable the `akka.jvm-exit-on-fatal-error` setting. Can you try enabling this setting and then run again?  

I also filed an issue to improve the logging of fatal errors: https://github.com/akka/akka/issues/23107

Johannes

cie...@gmail.com

unread,
Jun 7, 2017, 4:40:14 AM6/7/17
to Akka User List

thanks for your help,below is part of my app config .

akka {
  jvm-exit-on-fatal-error = off
  actor {
    provider = "akka.cluster.ClusterActorRefProvider"
  }
  remote {
    log-remote-lifecycle-events = off
    enabled-transports = ["akka.remote.netty.tcp"]
    netty.tcp {
      port = 20300
    }
  }
  cluster {
    roles = ["WS-MongodbRedisServer", "<DEV>"]
    join-point = "akka.tcp://wssy...@192.168.0.56:20000"

    auto-down-unreachable-after = off
  }
}

i have meet serveral situation : 

 situation 1 : ( has tell me the stack, and tell me Shutdown finished   )
 the setting 'akka.actor.warn-about-java-serializer-usage'
INFO   | jvm 1    | 2017/04/07 15:49:52 | [ERROR] [04/07/2017 15:49:52.227] [wssystem-akka.remote.default-remote-dispatcher-13] [akka.actor.ActorSystemImpl(wssystem)] Uncaught fatal error from thread [wssystem-akka.remote.default-remote-dispatcher-13] shutting down ActorSystem [wssystem]
INFO   | jvm 1    | 2017/04/07 15:49:52 | java.lang.NoClassDefFoundError: Lws/protos/EnumsProtos$HardTypeEnum;
INFO   | jvm 1    | 2017/04/07 15:49:52 |       at java.lang.Class.getDeclaredFields0(Native Method)
INFO   | jvm 1    | 2017/04/07 15:49:52 |       at java.lang.Class.privateGetDeclaredFields(Class.java:2583)
INFO   | jvm 1    | 2017/04/07 15:49:52 |       at java.lang.Class.getDeclaredField(Class.java:2068)
INFO   | jvm 1    | 2017/04/07 15:49:52 |       at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1703)
INFO   | jvm 1    | 2017/04/07 15:49:52 |       at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:72)
INFO   | jvm 1    | 2017/04/07 15:49:52 |       at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:484)
INFO   | jvm 1    | 2017/04/07 15:49:52 |       at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472)
INFO   | jvm 1    | 2017/04/07 15:49:52 |       at java.security.AccessController.doPrivileged(Native Method)
INFO   | jvm 1    | 2017/04/07 15:49:52 |       at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:472)
......
INFO   | jvm 1    | 2017/04/07 15:49:52 | [INFO] [04/07/2017 15:49:52.232] [wssystem-akka.remote.default-remote-dispatcher-5] [akka.tcp://wssy...@192.168.0.56:20300/system/remoting-terminator] Shutting down remote daemon.
INFO   | jvm 1    | 2017/04/07 15:49:52 | [INFO] [04/07/2017 15:49:52.233] [wssystem-akka.remote.default-remote-dispatcher-5] [akka.tcp://wssy...@192.168.0.56:20300/system/remoting-terminator] Remote daemon shut down; proceeding with flushing remote transports.
INFO   | jvm 1    | 2017/04/07 15:49:54 | [WARN] [04/07/2017 15:49:54.256] [wssystem-akka.actor.default-dispatcher-33] [akka.remote.Remoting] Shutdown finished, but flushing might not have been successful and some messages might have been dropped. Increase akka.remote.flush-wait-on-shutdown to a larger value to avoid this.
INFO   | jvm 1    | 2017/04/07 15:49:54 | [INFO] [04/07/2017 15:49:54.256] [wssystem-akka.remote.default-remote-dispatcher-5] [akka.tcp://wssy...@192.168.0.56:20300/system/remoting-terminator] Remoting shut down.



 situation 2 :( do not tell me the stack, and tell me Shutdown finished   )
INFO   | jvm 1    | 2017/05/24 16:41:36 |
INFO   | jvm 1    | 2017/05/24 16:41:45 | 16:41:45.044 [cluster-ClusterId{value='59252402784ba3565cc5ded1', description='null'}-192.168.0.56:29077] DEBUG org.mongodb.driver.cluster - Checking status of 192.168.0.56:29077
INFO   | jvm 1    | 2017/05/24 16:41:45 | 16:41:45.044 [cluster-ClusterId{value='59252402784ba3565cc5ded1', description='null'}-192.168.0.56:29077] DEBUG org.mongodb.driver.cluster - Updating cluster description to  {type=STANDALONE, servers=[{address=192.168.0.56:29077, type=STANDALONE, roundTripTime=0.2 ms, state=CONNECTED}]
INFO   | jvm 1    | 2017/05/24 16:41:54 | [ERROR] [SECURITY][05/24/2017 16:41:54.422] [wssystem-akka.remote.default-remote-dispatcher-26] [akka.actor.ActorSystemImpl(wssystem)] Uncaught fatal error from thread [wssystem-akka.remote.default-remote-dispatcher-26] shutting down ActorSystem [wssystem]
INFO   | jvm 1    | 2017/05/24 16:41:54 | [INFO] [05/24/2017 16:41:54.434] [wssystem-akka.remote.default-remote-dispatcher-5] [akka.tcp://wssy...@192.168.0.56:20300/system/remoting-terminator] Shutting down remote daemon.
INFO   | jvm 1    | 2017/05/24 16:41:54 | [INFO] [05/24/2017 16:41:54.434] [wssystem-akka.remote.default-remote-dispatcher-5] [akka.tcp://wssy...@192.168.0.56:20300/system/remoting-terminator] Remote daemon shut down; proceeding with flushing remote transports.
INFO   | jvm 1    | 2017/05/24 16:41:55 | 16:41:55.044 [cluster-ClusterId{value='59252402784ba3565cc5ded1', description='null'}-192.168.0.56:29077] DEBUG org.mongodb.driver.cluster - Checking status of 192.168.0.56:29077
INFO   | jvm 1    | 2017/05/24 16:41:55 | 16:41:55.045 [cluster-ClusterId{value='59252402784ba3565cc5ded1', description='null'}-192.168.0.56:29077] DEBUG org.mongodb.driver.cluster - Updating cluster description to  {type=STANDALONE, servers=[{address=192.168.0.56:29077, type=STANDALONE, roundTripTime=0.2 ms, state=CONNECTED}]


 situation 3 : ( do not tell me the stack, and tell me Shutdown finished  ,different from  situation 2 )
INFO   | jvm 1    | 2017/06/02 18:29:14 | 18:29:14.485 [cluster-ClusterId{value='5930c85d784ba304586d41ec', description='null'}-192.168.0.56:29077] DEBUG org.mongodb.driver.cluster - Updating cluster description to  {type=UNKNOWN, servers=[{address=192.168.0.56:29077, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused}}]
INFO   | jvm 1    | 2017/06/02 18:29:14 | [ERROR] [SECURITY][06/02/2017 18:29:14.501] [wssystem-akka.actor.default-dispatcher-15] [akka.actor.ActorSystemImpl(wssystem)] Uncaught fatal error from thread [wssystem-akka.actor.default-dispatcher-15] shutting down ActorSystem [wssystem]
INFO   | jvm 1    | 2017/06/02 18:29:16 | [INFO] [06/02/2017 18:29:16.326] [wssystem-akka.actor.default-dispatcher-28] [akka.cluster.Cluster(akka://wssystem)] Cluster Node [akka.tcp://wssy...@192.168.0.56:20300] - Exiting, starting coordinated shutdown
INFO   | jvm 1    | 2017/06/02 18:29:16 | [INFO] [06/02/2017 18:29:16.328] [wssystem-akka.actor.default-dispatcher-28] [akka.cluster.Cluster(akka://wssystem)] Cluster Node [akka.tcp://wssy...@192.168.0.56:20300] - Exiting completed
INFO   | jvm 1    | 2017/06/02 18:29:16 | [INFO] [06/02/2017 18:29:16.344] [wssystem-akka.actor.default-dispatcher-28] [akka.cluster.Cluster(akka://wssystem)] Cluster Node [akka.tcp://wssy...@192.168.0.56:20300] - Shutting down...
INFO   | jvm 1    | 2017/06/02 18:29:16 | [INFO] [06/02/2017 18:29:16.349] [wssystem-akka.actor.default-dispatcher-28] [akka.cluster.Cluster(akka://wssystem)] Cluster Node [akka.tcp://wssy...@192.168.0.56:20300] - Successfully shut down
INFO   | jvm 1    | 2017/06/02 18:29:24 | 18:29:24.486 [cluster-ClusterId{value='5930c85d784ba304586d41ec', description='null'}-192.168.0.56:29077] DEBUG org.mongodb.driver.connection - Closing connection connectionId{localValue:4}
INFO   | jvm 1    | 2017/06/02 18:29:26 | [WARN] [06/02/2017 18:29:26.389] [wssystem-akka.actor.default-dispatcher-31] [CoordinatedShutdown(akka://wssystem)] Coordinated shutdown phase [actor-system-terminate] timed out after 10000 milliseconds


 situation 4 : ( do not tell me the stack, and do not tell me Shutdown finished   )
INFO   | jvm 1    | 2017/06/07 15:26:47 | 15:26:47.216 [wssystem-akka.actor.default-dispatcher-3] DEBUG org.mongodb.driver.protocol.command - Sending command {findandmodify : BsonString{value='NewGuildCenter'}} to database Game_1 on connection [connectionId{localValue:2, serverValue:68}] to server 192.168.0.56:29077
INFO   | jvm 1    | 2017/06/07 15:26:47 | 15:26:47.221 [wssystem-akka.actor.default-dispatcher-3] DEBUG org.mongodb.driver.protocol.command - Command execution completed
INFO   | jvm 1    | 2017/06/07 15:26:47 | 15:26:47.244 [wssystem-akka.actor.default-dispatcher-25] ERROR akka.actor.ActorSystemImpl - Uncaught fatal error from thread [wssystem-akka.actor.default-dispatcher-3] shutting down ActorSystem [wssystem]
INFO   | jvm 1    | 2017/06/07 15:26:49 | 15:26:49.777 [cluster-ClusterId{value='59367010784ba375f91a4fec', description='null'}-192.168.0.56:29077] DEBUG org.mongodb.driver.cluster - Checking status of 192.168.0.56:29077
INFO   | jvm 1    | 2017/06/07 15:26:49 | 15:26:49.777 [cluster-ClusterId{value='59367010784ba375f91a4fec', description='null'}-192.168.0.56:29077] DEBUG org.mongodb.driver.cluster - Updating cluster description to  {type=STANDALONE, servers=[{address=192.168.0.56:29077, type=STANDALONE, roundTripTime=0.2 ms, state=CONNECTED}]
INFO   | jvm 1    | 2017/06/07 15:26:59 | 15:26:59.777 [cluster-ClusterId{value='59367010784ba375f91a4fec', description='null'}-192.168.0.56:29077] DEBUG org.mongodb.driver.cluster - Checking status of 192.168.0.56:29077
INFO   | jvm 1    | 2017/06/07 15:26:59 | 15:26:59.777 [cluster-ClusterId{value='59367010784ba375f91a4fec', description='null'}-192.168.0.56:29077] DEBUG org.mongodb.driver.cluster - Updating cluster description to  {type=STANDALONE, servers=[{address=192.168.0.56:29077, type=STANDALONE, roundTripTime=0.2 ms, state=CONNECTED}]
INFO   | jvm 1    | 2017/06/07 15:27:09 | 15:27:09.778 [cluster-ClusterId{value='59367010784ba375f91a4fec', description='null'}-192.168.0.56:29077] DEBUG org.mongodb.driver.cluster - Checking status of 192.168.0.56:29077
INFO   | jvm 1    | 2017/06/07 15:27:09 | 15:27:09.778 [cluster-ClusterId{value='59367010784ba375f91a4fec', description='null'}-192.168.0.56:29077] DEBUG org.mongodb.driver.cluster - Updating cluster description to  {type=STANDALONE, servers=[{address=192.168.0.56:29077, type=STANDALONE, roundTripTime=0.2 ms, state=CONNECTED}]
INFO   | jvm 1    | 2017/06/07 15:27:19 | 15:27:19.778 [cluster-ClusterId{value='59367010784ba375f91a4fec', description='null'}-192.168.0.56:29077] DEBUG org.mongodb.driver.cluster - Checking status of 192.168.0.56:29077
INFO   | jvm 1    | 2017/06/07 15:27:19 | 15:27:19.778 [cluster-ClusterId{value='59367010784ba375f91a4fec', description='null'}-192.168.0.56:29077] DEBUG org.mongodb.driver.cluster - Updating cluster description to  {type=STANDALONE, servers=[{address=192.168.0.56:29077, type=STANDALONE, roundTripTime=0.2 ms, state=CONNECTED}]
INFO   | jvm 1    | 2017/06/07 15:27:29 | 15:27:29.778 [cluster-ClusterId{value='59367010784ba375f91a4fec', description='null'}-192.168.0.56:29077] DEBUG org.mongodb.driver.cluster - Checking status of 192.168.0.56:29077








在 2017年6月6日星期二 UTC+8下午7:56:03，johannes...@lightbend.com写道：

Johannes Rudolph

unread,
Jun 7, 2017, 5:00:07 AM6/7/17
to akka...@googlegroups.com

The rest of the stack here will tell you where the problem comes from:

On Wed, Jun 7, 2017 at 10:40 AM, <cie...@gmail.com> wrote:
 situation 1 : ( has tell me the stack, and tell me Shutdown finished   )
INFO   | jvm 1    | 2017/04/07 15:49:52 | java.lang.NoClassDefFoundError: Lws/protos/EnumsProtos$HardTypeEnum;
INFO   | jvm 1    | 2017/04/07 15:49:52 |       at java.lang.Class.getDeclaredFields0(Native Method)
INFO   | jvm 1    | 2017/04/07 15:49:52 |       at java.lang.Class.privateGetDeclaredFields(Class.java:2583)
INFO   | jvm 1    | 2017/04/07 15:49:52 |       at java.lang.Class.getDeclaredField(Class.java:2068)
INFO   | jvm 1    | 2017/04/07 15:49:52 |       at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1703)
INFO   | jvm 1    | 2017/04/07 15:49:52 |       at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:72)
INFO   | jvm 1    | 2017/04/07 15:49:52 |       at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:484)
INFO   | jvm 1    | 2017/04/07 15:49:52 |       at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472)
INFO   | jvm 1    | 2017/04/07 15:49:52 |       at java.security.AccessController.doPrivileged(Native Method)
INFO   | jvm 1    | 2017/04/07 15:49:52 |       at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:472)
......

 
It might be related to actor messages received over remoting that cannot be deserialized because classpaths differ between the two nodes.

Johannes

cie...@gmail.com

unread,
Jun 7, 2017, 6:16:04 AM6/7/17
to Akka User List
yes, situation 1  , the log has tell me the reason cause the error , and i know how to solve situation 1  .
but situation 2,3,4  , din't tell me the reason ( the stack ), id dont hnow how solve this problem.

在 2017年6月7日星期三 UTC+8下午5:00:07，Johannes Rudolph写道：

Johannes Rudolph

unread,
Jun 7, 2017, 8:15:40 AM6/7/17
to akka...@googlegroups.com
If you set `akka.jvm-exit-on-fatal-error = on` it should log stack traces in all cases to stderr.

Johannes

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/R9utH4VIeXU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to akka-user+unsubscribe@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Akka Team

unread,
Jun 7, 2017, 8:17:43 AM6/7/17
to Akka User List
I think this is because of https://github.com/akka/akka/issues/22324 

It was marked as in progress but it seems there never was a PR submitted to fix that.

--
Johan
Akka Team

On Wed, Jun 7, 2017 at 12:16 PM, <cie...@gmail.com> wrote:

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscribe@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages