The Issues that caused by Kaa Cluster

[huawei1 xie]

I deployed 3 Kaa clusters follow the kaa Cluster setup guide document. The databases are Cassandra and MariaDB . When I run C Client(I make a code change that every client can login in three kaa servers in random) . a lot of questions are reported in the Server log file. I will upload the one Server logs in Kaa Cluster . I summed up a bit,there are  7 errors.The first error will appear when C Client login in kaa server firstly.After I stop C Client and run those secondly,the error didn't appear. I think that The second error is caused by Database cluster, maybe is Database consistency . Because I made a limit that use 10 MB HUB connected to  two Cassandra , the error often appears. I hava no idea about other errors.So I upload the errors and look forward to your help.

1 -------------------

2016-12-09 00:00:43,950 [EPS-endpoint-dispatcher-597] ERROR o.k.k.s.c.n.c.d.AbstractVersionableCassandraDao - [class org.kaaproject.kaa.server.common.nosql.cassandra.dao.model.CassandraEndpointProfile] Can't update entity with version 2. Entity already changed!

2016-12-09 00:00:43,952 [EPS-endpoint-dispatcher-597] WARN  o.k.k.s.c.d.s.EndpointServiceImpl - Optimistic lock detected in endpoint profile 

org.kaaproject.kaa.server.common.dao.exception.KaaOptimisticLockingFailureException: Can't update entity with version 2. Entity already changed!

at org.kaaproject.kaa.server.common.nosql.cassandra.dao.AbstractVersionableCassandraDao.updateLocked(AbstractVersionableCassandraDao.java:100) ~[cassandra-dao-0.10.0.jar:na]

at org.kaaproject.kaa.server.common.nosql.cassandra.dao.AbstractVersionableCassandraDao.save(AbstractVersionableCassandraDao.java:59) ~[cassandra-dao-0.10.0.jar:na]

at org.kaaproject.kaa.server.common.nosql.cassandra.dao.EndpointProfileCassandraDao.updateProfile(EndpointProfileCassandraDao.java:168) ~[cassandra-dao-0.10.0.jar:na]

at org.kaaproject.kaa.server.common.nosql.cassandra.dao.EndpointProfileCassandraDao.save(EndpointProfileCassandraDao.java:129) ~[cassandra-dao-0.10.0.jar:na]

at org.kaaproject.kaa.server.common.nosql.cassandra.dao.EndpointProfileCassandraDao.save(EndpointProfileCassandraDao.java:121) ~[cassandra-dao-0.10.0.jar:na]

at org.kaaproject.kaa.server.common.nosql.cassandra.dao.EndpointProfileCassandraDao.save(EndpointProfileCassandraDao.java:90) ~[cassandra-dao-0.10.0.jar:na]

at sun.reflect.GeneratedMethodAccessor134.invoke(Unknown Source) ~[na:na]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_111]

at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_111]

at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]

at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]

at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:136) ~[spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE]

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]

at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]

at com.sun.proxy.$Proxy64.save(Unknown Source) ~[na:na]

at org.kaaproject.kaa.server.common.dao.service.EndpointServiceImpl.saveEndpointProfile(EndpointServiceImpl.java:382) ~[dao-0.10.0.jar:na]

at org.kaaproject.kaa.server.common.dao.service.EndpointServiceImpl.attachEndpointToUser(EndpointServiceImpl.java:421) ~[dao-0.10.0.jar:na]

at sun.reflect.GeneratedMethodAccessor180.invoke(Unknown Source) ~[na:na]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_111]

at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_111]

at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]

at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]

at org.springframework.retry.annotation.AnnotationAwareRetryOperationsInterceptor.invoke(AnnotationAwareRetryOperationsInterceptor.java:121) [spring-retry-1.1.2.RELEASE.jar:na]

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]

at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]

at com.sun.proxy.$Proxy70.attachEndpointToUser(Unknown Source) [na:na]

at org.kaaproject.kaa.server.operations.service.user.DefaultEndpointUserService.attachEndpointToUser(DefaultEndpointUserService.java:93) [kaa-node-0.10.0.jar:na]

at org.kaaproject.kaa.server.operations.service.DefaultOperationsService.attachEndpointToUser(DefaultOperationsService.java:788) [kaa-node-0.10.0.jar:na]

at org.kaaproject.kaa.server.operations.service.akka.actors.core.endpoint.local.LocalEndpointActorMessageProcessor.processUserVerificationMessage(LocalEndpointActorMessageProcessor.java:1025) [kaa-node-0.10.0.jar:na]

at org.kaaproject.kaa.server.operations.service.akka.actors.core.endpoint.local.LocalEndpointActor.processUserVerificationMessage(LocalEndpointActor.java:123) [kaa-node-0.10.0.jar:na]

at org.kaaproject.kaa.server.operations.service.akka.actors.core.endpoint.local.LocalEndpointActor.onReceive(LocalEndpointActor.java:88) [kaa-node-0.10.0.jar:na]

at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167) [akka-actor_2.11-2.4.1.jar:na]

at akka.actor.Actor$class.aroundReceive(Actor.scala:480) [akka-actor_2.11-2.4.1.jar:na]

at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97) [akka-actor_2.11-2.4.1.jar:na]

at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526) [akka-actor_2.11-2.4.1.jar:na]

at akka.actor.ActorCell.invoke(ActorCell.scala:495) [akka-actor_2.11-2.4.1.jar:na]

at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) [akka-actor_2.11-2.4.1.jar:na]

at akka.dispatch.Mailbox.run(Mailbox.scala:224) [akka-actor_2.11-2.4.1.jar:na]

at akka.dispatch.Mailbox.exec(Mailbox.scala:234) [akka-actor_2.11-2.4.1.jar:na]

at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [scala-library-2.11.7.jar:na]

at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [scala-library-2.11.7.jar:na]

at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [scala-library-2.11.7.jar:na]

at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [scala-library-2.11.7.jar:na]

2 -------------------

2016-12-09 00:17:12,473 [EPS-endpoint-dispatcher-586] ERROR o.k.k.s.o.s.a.a.c.e.l.LocalEndpointActorMessageProcessor - [R18PpFaouxXSrHQVbOfyTtiGKBo=][cd72eab3ccec4135a6ef4f553add7bb6] processEndpointRequest

java.security.InvalidParameterException: Cache key can't be null

at org.kaaproject.kaa.server.operations.service.cache.concurrent.CacheTemporaryMemorizer.compute(CacheTemporaryMemorizer.java:77) ~[kaa-node-0.10.0.jar:na]

at org.kaaproject.kaa.server.operations.service.cache.concurrent.ConcurrentCacheService.getConfByHash(ConcurrentCacheService.java:581) ~[kaa-node-0.10.0.jar:na]

at sun.reflect.GeneratedMethodAccessor216.invoke(Unknown Source) ~[na:na]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_111]

at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_111]

at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302) ~[spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]

at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) ~[spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]

at org.springframework.cache.interceptor.CacheInterceptor$1.invoke(CacheInterceptor.java:52) ~[spring-context-4.2.5.RELEASE.jar:4.2.5.RELEASE]

at org.springframework.cache.interceptor.CacheAspectSupport.invokeOperation(CacheAspectSupport.java:320) ~[spring-context-4.2.5.RELEASE.jar:4.2.5.RELEASE]

at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:353) ~[spring-context-4.2.5.RELEASE.jar:4.2.5.RELEASE]

at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:302) ~[spring-context-4.2.5.RELEASE.jar:4.2.5.RELEASE]

at org.springframework.cache.interceptor.CacheInterceptor.invoke(CacheInterceptor.java:61) ~[spring-context-4.2.5.RELEASE.jar:4.2.5.RELEASE]

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]

at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208) ~[spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]

at com.sun.proxy.$Proxy96.getConfByHash(Unknown Source) ~[na:na]

at org.kaaproject.kaa.server.operations.service.delta.DefaultDeltaService.getDelta(DefaultDeltaService.java:155) ~[kaa-node-0.10.0.jar:na]

at org.kaaproject.kaa.server.operations.service.DefaultOperationsService.calculateConfigurationDelta(DefaultOperationsService.java:675) ~[kaa-node-0.10.0.jar:na]

at org.kaaproject.kaa.server.operations.service.DefaultOperationsService.syncConfiguration(DefaultOperationsService.java:426) ~[kaa-node-0.10.0.jar:na]

at org.kaaproject.kaa.server.operations.service.akka.actors.core.endpoint.local.LocalEndpointActorMessageProcessor.sync(LocalEndpointActorMessageProcessor.java:394) [kaa-node-0.10.0.jar:na]

at org.kaaproject.kaa.server.operations.service.akka.actors.core.endpoint.local.LocalEndpointActorMessageProcessor.sync(LocalEndpointActorMessageProcessor.java:310) [kaa-node-0.10.0.jar:na]

at org.kaaproject.kaa.server.operations.service.akka.actors.core.endpoint.local.LocalEndpointActorMessageProcessor.processEndpointSync(LocalEndpointActorMessageProcessor.java:135) [kaa-node-0.10.0.jar:na]

at org.kaaproject.kaa.server.operations.service.akka.actors.core.endpoint.local.LocalEndpointActor.processEndpointSync(LocalEndpointActor.java:131) [kaa-node-0.10.0.jar:na]

at org.kaaproject.kaa.server.operations.service.akka.actors.core.endpoint.local.LocalEndpointActor.onReceive(LocalEndpointActor.java:78) [kaa-node-0.10.0.jar:na]

at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167) [akka-actor_2.11-2.4.1.jar:na]

at akka.actor.Actor$class.aroundReceive(Actor.scala:480) [akka-actor_2.11-2.4.1.jar:na]

at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97) [akka-actor_2.11-2.4.1.jar:na]

at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526) [akka-actor_2.11-2.4.1.jar:na]

at akka.actor.ActorCell.invoke(ActorCell.scala:495) [akka-actor_2.11-2.4.1.jar:na]

at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) [akka-actor_2.11-2.4.1.jar:na]

at akka.dispatch.Mailbox.run(Mailbox.scala:224) [akka-actor_2.11-2.4.1.jar:na]

at akka.dispatch.Mailbox.exec(Mailbox.scala:234) [akka-actor_2.11-2.4.1.jar:na]

at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [scala-library-2.11.7.jar:na]

at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [scala-library-2.11.7.jar:na]

at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [scala-library-2.11.7.jar:na]

at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [scala-library-2.11.7.jar:na]

3  -----------------

2016-12-09 00:32:35,091 [EPS-core-dispatcher-1174] ERROR o.k.k.s.o.s.a.a.s.SupervisionStrategyFactory - Supervisor strategy got exception: null

java.lang.NullPointerException: null

2016-12-09 00:32:35,092 [EPS-akka.actor.default-dispatcher-4] ERROR akka.actor.OneForOneStrategy - null

java.lang.NullPointerException: null

2016-12-09 00:32:35,097 [EPS-endpoint-dispatcher-586] INFO  o.k.k.s.o.s.u.DefaultEndpointUserService - Can't find listeners for unassigned endpoint!

4 --------------------

2016-12-09 01:15:46,710 [pool-22-thread-1] ERROR o.k.k.s.thrift.NeighborConnection - EventWorker [192.168.10.14:9090:OPERATIONS_SERVICE:d9c71186-e45e-4c33-88b0-0ad5f26c6d17] error sending event messages pack. 

com.twitter.common.thrift.TResourceExhaustedException: java.util.concurrent.RejectedExecutionException: All 1 x 3 connections in use

at com.twitter.common.thrift.callers.DeadlineCaller.call(DeadlineCaller.java:91) ~[thrift-0.0.64.jar:na]

at com.twitter.common.thrift.Thrift$1.invoke(Thrift.java:363) ~[thrift-0.0.64.jar:na]

at com.sun.proxy.$Proxy141.sendMessages(Unknown Source) ~[na:na]

at org.kaaproject.kaa.server.operations.service.event.DefaultEventService$MessageTemplate.process(DefaultEventService.java:727) ~[kaa-node-0.10.0.jar:na]

at org.kaaproject.kaa.server.thrift.NeighborConnection$EventWorker.run(NeighborConnection.java:322) ~[server-shared-0.10.0.jar:na]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_111]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_111]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]

at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]

Caused by: java.util.concurrent.RejectedExecutionException: All 1 x 3 connections in use

at com.twitter.common.thrift.ThriftFactory$ManagedThreadPool$1.rejectedExecution(ThriftFactory.java:237) ~[thrift-0.0.64.jar:na]

at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) [na:1.8.0_111]

at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) [na:1.8.0_111]

at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) ~[na:1.8.0_111]

at com.twitter.common.util.concurrent.ForwardingExecutorService.submit(ForwardingExecutorService.java:66) ~[util-0.0.88.jar:na]

at com.twitter.common.thrift.callers.DeadlineCaller.call(DeadlineCaller.java:71) ~[thrift-0.0.64.jar:na]

... 9 common frames omitted

5 ---------------------

2016-12-09 02:15:50,233 [EPS-endpoint-dispatcher-598] ERROR o.k.k.s.o.s.a.a.c.e.l.LocalEndpointActorMessageProcessor - [Nje8C3i2b6K7ukh7tP8G/pDzdrE=][3f5e75f03bd34c30be183807486176ba] processEndpointRequest

java.lang.NullPointerException: null

6 -------------------------

2016-12-09 00:53:05,649 [EPS-core-dispatcher-1117] ERROR o.k.k.s.o.s.a.a.s.SupervisionStrategyFactory - Supervisor strategy got exception: null

java.lang.NullPointerException: null

2016-12-09 00:53:05,650 [EPS-akka.actor.default-dispatcher-4] ERROR akka.actor.OneForOneStrategy - null

java.lang.NullPointerException: null

7 ------------------------

2016-12-09 09:20:33,680 [qtp1981719029-883] ERROR o.k.k.s.admin.servlet.SdkServlet - Unexpected error in SdkServlet.doGet: 

org.eclipse.jetty.io.EofException: null

at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:192) ~[jetty-io-9.2.2.v20140723.jar:9.2.2.v20140723]

at org.eclipse.jetty.io.WriteFlusher.completeWrite(WriteFlusher.java:402) ~[jetty-io-9.2.2.v20140723.jar:9.2.2.v20140723]

at org.eclipse.jetty.io.SelectChannelEndPoint.onSelected(SelectChannelEndPoint.java:111) ~[jetty-io-9.2.2.v20140723.jar:9.2.2.v20140723]

at org.eclipse.jetty.io.SelectorManager$ManagedSelector.processKey(SelectorManager.java:577) ~[jetty-io-9.2.2.v20140723.jar:9.2.2.v20140723]

at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:548) ~[jetty-io-9.2.2.v20140723.jar:9.2.2.v20140723]

at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:488) ~[jetty-io-9.2.2.v20140723.jar:9.2.2.v20140723]

at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) ~[jetty-util-9.2.2.v20140723.jar:9.2.2.v20140723]

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:606) [jetty-util-9.2.2.v20140723.jar:9.2.2.v20140723]

at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:535) [jetty-util-9.2.2.v20140723.jar:9.2.2.v20140723]

at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]

Caused by: java.io.IOException: Broken pipe

at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[na:1.8.0_111]

at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[na:1.8.0_111]

at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[na:1.8.0_111]

at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[na:1.8.0_111]

at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) ~[na:1.8.0_111]

at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:170) ~[jetty-io-9.2.2.v20140723.jar:9.2.2.v20140723]

... 9 common frames omitted

[Taras Khalymon]

Question is answered here http://stackoverflow.com/questions/41057935/the-error-that-cant-send-and-receive-event-in-kaa-cluster.

[huawei1 xie]

Hi,

   anyone answered the question(http://stackoverflow.com/questions/41057935/the-error-that-cant-send-and-receive-event-in-kaa-cluster.) and the question in link is written by me .I sure the question maybe a bug on kaa server .I made a lot of tests about it.The C ,Java and Android Client can not send and receive event in kaa cluster sometimes when you login in server again and again .

Best regards 

在 2016年12月14日星期三 UTC+8上午2:29:43，Taras Khalymon写道：

[Taras Khalymon]

Hi,

Sorry, that's a wrong link, question is here (http://stackoverflow.com/questions/41096922/the-issues-that-are-caused-by-kaa-cluster/41100101) and I'm currently investigating it.

Best regards, Taras