Infinispan http-remoting-connector cache failure during upgrade to Keycloak 13.0.0
Do Hoang Khiem
We're upgrading our Keycloak in k8s cluster to latest 13.0.0 from 12.0.1, and facing the error related to Infinispan cache for http-remoting-connector below
05:38:03,397 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 61) MSC000001: Failed to start service org.wildfly.clustering.infinispan.cache.ejb.http-remoting-connector: org.jboss.msc.service.StartException in service org.wildfly.clustering.infinispan.cache.ejb.http-remoting-connector: org.infinispan.commons.CacheException: Initial state transfer timed out for cache http-remoting-connector on keycloak-1
at org.wildfly.clu...@23.0.2.Final//org.wildfly.clustering.service.FunctionalService.start(FunctionalService.java:66)
at org.wildfly.clu...@23.0.2.Final//org.wildfly.clustering.service.AsyncServiceConfigurator$AsyncService.lambda$start$0(AsyncServiceConfigurator.java:117)
at org.jbos...@2.4.0.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at org.jbos...@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
at org.jbos...@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
at org.jbos...@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
at java.base/java.lang.Thread.run(Thread.java:829)
at org.jbos...@2.4.0.Final//org.jboss.threads.JBossThread.run(JBossThread.java:513)
Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache http-remoting-connector on keycloak-1
at org.inf...@11.0.9.Final//org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:246)
at org.inf...@11.0.9.Final//org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1011)
at org.inf...@11.0.9.Final//org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:512)
at org.inf...@11.0.9.Final//org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:697)
at org.inf...@11.0.9.Final//org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:643)
at org.inf...@11.0.9.Final//org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:532)
at org.inf...@11.0.9.Final//org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:510)
at org.jboss.as.clus...@23.0.2.Final//org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:92)
at org.wildfly.cluste...@23.0.2.Final//org.wildfly.clustering.infinispan.spi.service.CacheServiceConfigurator.get(CacheServiceConfigurator.java:77)
at org.wildfly.cluste...@23.0.2.Final//org.wildfly.clustering.infinispan.spi.service.CacheServiceConfigurator.get(CacheServiceConfigurator.java:55)
at org.wildfly.clu...@23.0.2.Final//org.wildfly.clustering.service.FunctionalService.start(FunctionalService.java:63)
... 7 more
05:38:03,641 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([
("subsystem" => "ejb3"),
("service" => "remote")
]) - failure description: {"WFLYCTL0080: Failed services" => {"org.wildfly.clustering.infinispan.cache.ejb.http-remoting-connector" => "org.infinispan.commons.CacheException: Initial state transfer timed out for cache http-remoting-connector on keycloak-1
Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache http-remoting-connector on keycloak-1"}}
One more exception
05:34:03,390 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 61) WFLYCLINF0002: Started http-remoting-connector cache from ejb container
05:34:03,460 ERROR [org.infinispan.CLUSTER] (thread-11,ejb,keycloak-1) ISPN000474: Error processing request 25@keycloak-0: protostream.com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field. This could mean either that the input has been truncated or that an embedded message misreported its own length.
at org.infinispa...@4.3.5.Final//protostream.com.google.protobuf.InvalidProtocolBufferException.truncatedMessage(InvalidProtocolBufferException.java:84)
at org.infinispa...@4.3.5.Final//protostream.com.google.protobuf.CodedInputStream$StreamDecoder.refillBuffer(CodedInputStream.java:2737)
at org.infinispa...@4.3.5.Final//protostream.com.google.protobuf.CodedInputStream$StreamDecoder.readRawByte(CodedInputStream.java:2812)
at org.infinispa...@4.3.5.Final//protostream.com.google.protobuf.CodedInputStream$StreamDecoder.readRawVarint64SlowPath(CodedInputStream.java:2602)
at org.infinispa...@4.3.5.Final//protostream.com.google.protobuf.CodedInputStream$StreamDecoder.readRawVarint32(CodedInputStream.java:2496)
at org.infinispa...@4.3.5.Final//protostream.com.google.protobuf.CodedInputStream$StreamDecoder.readUInt32(CodedInputStream.java:2426)
at org.wildfly.clustering....@23.0.2.Final//org.wildfly.clustering.marshalling.protostream.DefaultProtoStreamReader.readUInt32(DefaultProtoStreamReader.java:167)
at org.wildfly.clustering....@23.0.2.Final//org.wildfly.clustering.marshalling.protostream.AnyField$2.readFrom(AnyField.java:93)
at org.wildfly.clustering....@23.0.2.Final//org.wildfly.clustering.marshalling.protostream.AnyField$2.readFrom(AnyField.java:81)
at org.wildfly.clustering....@23.0.2.Final//org.wildfly.clustering.marshalling.protostream.AnyMarshaller.readFrom(AnyMarshaller.java:55)
at org.wildfly.clustering....@23.0.2.Final//org.wildfly.clustering.marshalling.protostream.AnyMarshaller.readFrom(AnyMarshaller.java:37)
at org.wildfly.clustering....@23.0.2.Final//org.wildfly.clustering.marshalling.protostream.ProtoStreamMarshaller.readFrom(ProtoStreamMarshaller.java:48)
at org.infinispa...@4.3.5.Final//org.infinispan.protostream.impl.RawProtobufMarshallerDelegate.unmarshall(RawProtobufMarshallerDelegate.java:37)
at org.infinispa...@4.3.5.Final//org.infinispan.protostream.ProtobufUtil.readFrom(ProtobufUtil.java:128)
at org.infinispa...@4.3.5.Final//org.infinispan.protostream.ProtobufUtil.readFrom(ProtobufUtil.java:132)
at org.wildfly.clustering....@23.0.2.Final//org.wildfly.clustering.marshalling.protostream.ProtoStreamByteBufferMarshaller.readFrom(ProtoStreamByteBufferMarshaller.java:76)
at org.wildfly.clustering...@23.0.2.Final//org.wildfly.clustering.infinispan.marshalling.protostream.ProtoStreamMarshaller.readObject(ProtoStreamMarshaller.java:77)
at org.wildfly.clustering...@23.0.2.Final//org.wildfly.clustering.infinispan.marshalling.AbstractMarshaller.objectFromByteBuffer(AbstractMarshaller.java:67)
at org.wildfly.clustering...@23.0.2.Final//org.wildfly.clustering.infinispan.marshalling.AbstractMarshaller.objectFromByteBuffer(AbstractMarshaller.java:61)
at org.inf...@11.0.9.Final//org.infinispan.marshall.protostream.impl.MarshallableUserObject$Marshaller.readFrom(MarshallableUserObject.java:138)
at org.inf...@11.0.9.Final//org.infinispan.marshall.protostream.impl.MarshallableUserObject$Marshaller.readFrom(MarshallableUserObject.java:102)
at org.infinispa...@4.3.5.Final//org.infinispan.protostream.impl.RawProtobufMarshallerDelegate.unmarshall(RawProtobufMarshallerDelegate.java:37)
at org.infinispa...@4.3.5.Final//org.infinispan.protostream.WrappedMessage.readMessage(WrappedMessage.java:390)
at org.infinispa...@4.3.5.Final//org.infinispan.protostream.ProtobufUtil.fromWrappedByteArray(ProtobufUtil.java:162)
at org.inf...@11.0.9.Final//org.infinispan.marshall.persistence.impl.PersistenceMarshallerImpl.objectFromByteBuffer(PersistenceMarshallerImpl.java:155)
at org.inf...@11.0.9.Final//org.infinispan.marshall.persistence.impl.PersistenceMarshallerImpl.objectFromByteBuffer(PersistenceMarshallerImpl.java:150)
at org.inf...@11.0.9.Final//org.infinispan.marshall.core.GlobalMarshaller.readUnknown(GlobalMarshaller.java:906)
at org.inf...@11.0.9.Final//org.infinispan.marshall.core.GlobalMarshaller.readUnknown(GlobalMarshaller.java:891)
at org.inf...@11.0.9.Final//org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:715)
at org.inf...@11.0.9.Final//org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:358)
at org.inf...@11.0.9.Final//org.infinispan.marshall.core.BytesObjectInput.readObject(BytesObjectInput.java:32)
at org.inf...@11.0.9.Final//org.infinispan.container.entries.ImmortalCacheEntry$Externalizer.readObject(ImmortalCacheEntry.java:108)
at org.inf...@11.0.9.Final//org.infinispan.container.entries.ImmortalCacheEntry$Externalizer.readObject(ImmortalCacheEntry.java:97)
at org.inf...@11.0.9.Final//org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:728)
at org.inf...@11.0.9.Final//org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:709)
at org.inf...@11.0.9.Final//org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:358)
at org.inf...@11.0.9.Final//org.infinispan.marshall.core.BytesObjectInput.readObject(BytesObjectInput.java:32)
at org.infinis...@11.0.9.Final//org.infinispan.commons.marshall.MarshallUtil.lambda$unmarshallCollection$0(MarshallUtil.java:293)
at org.infinis...@11.0.9.Final//org.infinispan.commons.marshall.MarshallUtil.unmarshallCollection(MarshallUtil.java:276)
at org.infinis...@11.0.9.Final//org.infinispan.commons.marshall.MarshallUtil.unmarshallCollection(MarshallUtil.java:293)
at org.inf...@11.0.9.Final//org.infinispan.marshall.exts.CollectionExternalizer.readObject(CollectionExternalizer.java:103)
at org.inf...@11.0.9.Final//org.infinispan.marshall.exts.CollectionExternalizer.readObject(CollectionExternalizer.java:28)
at org.inf...@11.0.9.Final//org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:728)
at org.inf...@11.0.9.Final//org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:709)
at org.inf...@11.0.9.Final//org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:358)
at org.inf...@11.0.9.Final//org.infinispan.marshall.core.BytesObjectInput.readObject(BytesObjectInput.java:32)
at org.inf...@11.0.9.Final//org.infinispan.statetransfer.StateChunk$Externalizer.readObject(StateChunk.java:88)
at org.inf...@11.0.9.Final//org.infinispan.statetransfer.StateChunk$Externalizer.readObject(StateChunk.java:65)
at org.inf...@11.0.9.Final//org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:728)
at org.inf...@11.0.9.Final//org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:709)
at org.inf...@11.0.9.Final//org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:358)
at org.inf...@11.0.9.Final//org.infinispan.marshall.core.BytesObjectInput.readObject(BytesObjectInput.java:32)
at org.infinis...@11.0.9.Final//org.infinispan.commons.marshall.MarshallUtil.lambda$unmarshallCollection$0(MarshallUtil.java:293)
at org.infinis...@11.0.9.Final//org.infinispan.commons.marshall.MarshallUtil.unmarshallCollection(MarshallUtil.java:276)
at org.infinis...@11.0.9.Final//org.infinispan.commons.marshall.MarshallUtil.unmarshallCollection(MarshallUtil.java:293)
at org.inf...@11.0.9.Final//org.infinispan.commands.statetransfer.StateResponseCommand.readFrom(StateResponseCommand.java:125)
at org.inf...@11.0.9.Final//org.infinispan.marshall.exts.ReplicableCommandExternalizer.readCommandParameters(ReplicableCommandExternalizer.java:114)
at org.inf...@11.0.9.Final//org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:154)
at org.inf...@11.0.9.Final//org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:81)
at org.inf...@11.0.9.Final//org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:728)
at org.inf...@11.0.9.Final//org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:709)
at org.inf...@11.0.9.Final//org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:358)
at org.inf...@11.0.9.Final//org.infinispan.marshall.core.GlobalMarshaller.objectFromObjectInput(GlobalMarshaller.java:192)
at org.inf...@11.0.9.Final//org.infinispan.marshall.core.GlobalMarshaller.objectFromByteBuffer(GlobalMarshaller.java:221)
at org.inf...@11.0.9.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processRequest(JGroupsTransport.java:1362)
at org.inf...@11.0.9.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1302)
at org.inf...@11.0.9.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$300(JGroupsTransport.java:131)
at org.inf...@11.0.9.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1445)
at org.j...@4.2.11.Final//org.jgroups.JChannel.up(JChannel.java:784)
at org.j...@4.2.11.Final//org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:135)
at org.j...@4.2.11.Final//org.jgroups.stack.Protocol.up(Protocol.java:309)
at org.j...@4.2.11.Final//org.jgroups.protocols.FORK.up(FORK.java:142)
at org.j...@4.2.11.Final//org.jgroups.protocols.FRAG3.up(FRAG3.java:165)
at org.j...@4.2.11.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:351)
at org.j...@4.2.11.Final//org.jgroups.protocols.pbcast.GMS.up(GMS.java:876)
at org.j...@4.2.11.Final//org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:243)
at org.j...@4.2.11.Final//org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1049)
at org.j...@4.2.11.Final//org.jgroups.protocols.UNICAST3.addMessage(UNICAST3.java:772)
at org.j...@4.2.11.Final//org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:753)
at org.j...@4.2.11.Final//org.jgroups.protocols.UNICAST3.up(UNICAST3.java:405)
at org.j...@4.2.11.Final//org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:592)
at org.j...@4.2.11.Final//org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:132)
at org.j...@4.2.11.Final//org.jgroups.protocols.FailureDetection.up(FailureDetection.java:186)
at org.j...@4.2.11.Final//org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:254)
at org.j...@4.2.11.Final//org.jgroups.protocols.MERGE3.up(MERGE3.java:281)
at org.j...@4.2.11.Final//org.jgroups.protocols.Discovery.up(Discovery.java:300)
at org.j...@4.2.11.Final//org.jgroups.protocols.TP.passMessageUp(TP.java:1396)
at org.j...@4.2.11.Final//org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:87)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at org.jboss.as.cl...@23.0.2.Final//org.jboss.as.clustering.context.ContextReferenceExecutor.execute(ContextReferenceExecutor.java:49)
at org.jboss.as.cl...@23.0.2.Final//org.jboss.as.clustering.context.ContextualExecutor$1.run(ContextualExecutor.java:70)
at java.base/java.lang.Thread.run(Thread.java:829)
This stopped us from current upgrade process, any ideas what would be needed to overcome this. Thank you all!
Vlasta Ramik
Hello Hoang,
could you please create new issue at
https://issues.redhat.com/browse/KEYCLOAK and put all the
information there?
Thank you
--
You received this message because you are subscribed to the Google Groups "Keycloak User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to keycloak-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/keycloak-user/69a3d723-699b-4d6f-a013-18910bddb78bn%40googlegroups.com.
benjam...@gmail.com
The new Keycloak 14.0.x backed instance failed to start up with this exception within the cluster of 12.0.x backed instances.
Keycloak 12.0.x has Infinispan 11.0.4 Final whereas both 13.0.x and 14.0.0 have Inifinispan 11.0.9 Final - wonder if something changed with Inifinispan in between might be causing this. It's unusual though for something like this to occur for minor version update (ie, 11.0.4 to 11.0.9), probably an edge case scenario?
