Data-Index Thread blocked

50 views
Skip to first unread message

Adrian Apthorp

unread,
Feb 14, 2021, 3:48:11 AM2/14/21
to Kogito development mailing list
I keep hitting the following error reported by the data-index service:

2021-02-14 16:32:37,449 WARNING [io.ver.cor.imp.BlockedThreadChecker] (vertx-blocked-thread-checker) Thread Thread[vert.x-eventloop-thread-6,5,main]=Thread[vert.x-eventloop-thread-6,5,main] has been blocked for 249658 ms, time limit is 2000 ms: io.vertx.core.VertxException: Thread blocked
    at java...@11.0.10/jdk.internal.misc.Unsafe.park(Native Method)
    at java...@11.0.10/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
    at java...@11.0.10/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1798)
    at java...@11.0.10/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128)
    at java...@11.0.10/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1868)
    at java...@11.0.10/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021)
    at app//org.infinispan.client.hotrod.impl.Util.await(Util.java:51)
    at app//org.infinispan.client.hotrod.impl.RemoteCacheSupport.put(RemoteCacheSupport.java:196)
    at app//org.infinispan.client.hotrod.impl.RemoteCacheSupport.put(RemoteCacheSupport.java:186)
    at app//org.kie.kogito.persistence.infinispan.cache.StorageImpl.put(StorageImpl.java:68)
    at app//org.kie.kogito.index.service.IndexingService.indexModel(IndexingService.java:91)
    at app//org.kie.kogito.index.service.IndexingService_ClientProxy.indexModel(IndexingService_ClientProxy.zig:250)
    at app//org.kie.kogito.index.messaging.ReactiveMessagingEventConsumer.onDomainEvent(ReactiveMessagingEventConsumer.java:163)
    at app//org.kie.kogito.index.messaging.ReactiveMessagingEventConsumer.lambda$onDomainModelRegisteredEvent$0(ReactiveMessagingEventConsumer.java:79)
    at app//org.kie.kogito.index.messaging.ReactiveMessagingEventConsumer$$Lambda$828/0x0000000840501c40.handle(Unknown Source)
    at app//io.vertx.core.eventbus.impl.HandlerRegistration.deliver(HandlerRegistration.java:278)
    at app//io.vertx.core.eventbus.impl.HandlerRegistration.handle(HandlerRegistration.java:264)
    at app//io.vertx.core.eventbus.impl.EventBusImpl$InboundDeliveryContext.next(EventBusImpl.java:567)
    at app//io.vertx.core.eventbus.impl.EventBusImpl.lambda$deliverToHandler$4(EventBusImpl.java:527)
    at app//io.vertx.core.eventbus.impl.EventBusImpl$$Lambda$1061/0x00000008405f2440.handle(Unknown Source)
    at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
    at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
    at app//io.vertx.core.impl.EventLoopContext$$Lambda$578/0x00000008403c7440.run(Unknown Source)
    at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
    at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java...@11.0.10/java.lang.Thread.run(Thread.java:834)

This occur occasionally when I interact my process service via the web API or from the process manager.

I also found that this occurs when I open a task in the task manager when there are no assigned data attributes for the task.

I'm running 1.1.0.Final on ubuntu 20.04 in quarkus development mode.

Any help appreciated. I've already found a few errors in the documentation (logged in JIRA) to get this far.





















































Cristiano Nicolai

unread,
Feb 14, 2021, 10:05:41 PM2/14/21
to Kogito development mailing list
Hi Adrian,

This error usually happens when Data Index is receiving domain even messages via Kafka from your runtime but Data Index ( Infinispan ) is not aware of the object domain that needs to be persisted. The way to handle that is to copy the generated proto file(s) under ( target/classes/META-INF/resources/persistence/protobuf ) into a folder and pass that to Data Index via property -Dkogito.protobuf.folder=${FOLDER} to the Data Index.


--
You received this message because you are subscribed to the Google Groups "Kogito development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kogito-developm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kogito-development/dc521b46-e2a2-47e2-a53c-7b0960e85ffcn%40googlegroups.com.


--
Best regards,

Cristiano Nicolai

Adrian Apthorp

unread,
Feb 15, 2021, 6:59:40 AM2/15/21
to Kogito development mailing list
Cristiano,

That's exactly how I start the Data Index and from I what I can see everything looks correct in Infinispan. I guess I'll have to see if I can consistently recreate the error, as it is not every interaction that causes it. It definitely occurs when I open a user task in the task console where there is no data assignment (in or out) for the task, but this is not the only situation where it occurs.

BTW what is the difference between the kogito-application.proto and 'domain'.proto files?

Regards

Adrian

Cristiano Nicolai

unread,
Feb 15, 2021, 9:07:54 AM2/15/21
to Kogito development mailing list
Hi Adrian,

Ok, that might be another case, if you could try to have some way for us to reproduce the error, it would help a lot.

Regarding the difference between proto files, kogito-application.proto is used by the kogito application persistence itself, its how the process state date is marshalled into Infinispan database.
As for 'domain'.proto, these are the ones shared with the data index, this is so the data index can map that model in the Graphql API as well as enable the service to also persist it into Infinispan.

Pere Fernandez

unread,
Feb 16, 2021, 6:43:36 AM2/16/21
to kogito-de...@googlegroups.com
Thank you Adrian for raising this!

I'm taking a look at the Task Console now, it looks like the form breaks when the task has no data assignments. I'll open a JIRA to solve that ASAP.

However, I locally tried modifying one of our examples to remove task data assignments and I was not able to reproduce the Thread blocked issue, I think it would be really helpful for ust if you could provide a reproducer project that we could give a try.

Thank you!

Pere

Adrian Apthorp

unread,
Feb 16, 2021, 9:02:38 AM2/16/21
to Kogito development mailing list
I'm working on reproducing this error and found that I'm also getting the following from Infinispan. I didn't see it before. I assume the 'id' field indicated is a kogito meta data field as I don't have an 'id' field in my project. This error is hit when I start my process using the swagger UI. The strange thing is I didn't have the error to start with. Therefore I'm trying figure out what changed so that I can replicate.

infinispan_1  | 13:47:11,996 ERROR (blocking-thread--p3-t2) [org.jboss.threads.errors] Thread Thread[blocking-thread--p3-t2,5,ISPN-blocking-thread-group] threw an uncaught exception java.lang.IllegalStateException: The field 'id' was not found in the Protobuf schema
infinispan_1  |     at org.infinispan.protostream.ProtobufUtil.expectField(ProtobufUtil.java:332)
infinispan_1  |     at org.infinispan.protostream.ProtobufUtil.processObject(ProtobufUtil.java:436)
infinispan_1  |     at org.infinispan.protostream.ProtobufUtil.processJsonDocument(ProtobufUtil.java:357)
infinispan_1  |     at org.infinispan.protostream.ProtobufUtil.fromCanonicalJSON(ProtobufUtil.java:239)
infinispan_1  |     at org.infinispan.encoding.ProtostreamTranscoder.transcode(ProtostreamTranscoder.java:88)
infinispan_1  |     at org.infinispan.encoding.DataConversion.toStorage(DataConversion.java:223)
infinispan_1  |     at org.infinispan.cache.impl.EncoderCache.valueToStorage(EncoderCache.java:107)
infinispan_1  |     at org.infinispan.cache.impl.EncoderCache.putAsync(EncoderCache.java:456)
infinispan_1  |     at org.infinispan.server.hotrod.CacheRequestProcessor.putInternal(CacheRequestProcessor.java:178)
infinispan_1  |     at org.infinispan.server.hotrod.CacheRequestProcessor.lambda$put$4(CacheRequestProcessor.java:171)
infinispan_1  |     at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
infinispan_1  |     at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982)
infinispan_1  |     at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
infinispan_1  |     at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
infinispan_1  |     at java.base/java.lang.Thread.run(Thread.java:834)
infinispan_1  |

Adrian Apthorp

unread,
Feb 20, 2021, 3:48:26 AM2/20/21
to Kogito development mailing list
Problem fixed. My bpmn process id was the same as my domain object name except all lower case. e.g. process id ="thing" and domain object = "Thing". This resulted in a scrambled protobuf definition for the process.

I suggest that this at least be a documentation fix.

Cristiano Nicolai

unread,
Feb 22, 2021, 4:05:11 AM2/22/21
to Kogito development mailing list
Thanks for the feedback Adrian. Would you be able to open a JIRA to report the error? We can take a look if this is something that can be fixed or add validation at compile time.

Thanks

Adrian Apthorp

unread,
Feb 22, 2021, 7:49:37 AM2/22/21
to Kogito development mailing list
Done. Along with a couple more that I discovered along the way.

Ming Fang

unread,
Mar 16, 2021, 10:12:26 PM3/16/21
to Kogito development mailing list
I also notice this problem.
It appears to be some kind of stale connection problem and only happens when I first interact with Infinispan after a long idle time.
I haven't measured it but definitely happens overnight.
Reply all
Reply to author
Forward
0 new messages