Uh oh! Event loop context executing with wrong thread! Expected null got Thread

191 views
Skip to first unread message

Idob

unread,
Dec 19, 2017, 6:54:08 AM12/19/17
to vert.x
Hello,

We have implemented a SQS event bus. The consumer is creating an ExecutorService (java.util.concurrent) that creates a configurable amounts of threads.

While running with 3 threads, I get this error:
WARN  io.netty.util.concurrent.DefaultPromise - An exception was thrown by io.vertx.core.net.impl.ChannelProvider$$Lambda$219/1676533407.operationComplete()
java.lang.IllegalStateException: Uh oh! Event loop context executing with wrong thread! Expected null got Thread[globalEventExecutor-1-78,5,main]
at io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:316)
at io.vertx.core.impl.ContextImpl.executeFromIO(ContextImpl.java:193)
at io.vertx.core.http.impl.ConnectionManager$ConnQueue.connectionFailed(ConnectionManager.java:362)
at io.vertx.core.http.impl.ConnectionManager$ConnQueue.access$1900(ConnectionManager.java:194)
at io.vertx.core.http.impl.ConnectionManager$ChannelConnector.lambda$connect$2(ConnectionManager.java:528)
at io.vertx.core.net.impl.ChannelProvider.lambda$connect$0(ChannelProvider.java:42)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
at io.netty.util.concurrent.DefaultPromise.access$000(DefaultPromise.java:34)
at io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:431)
at io.netty.util.concurrent.GlobalEventExecutor$TaskRunner.run(GlobalEventExecutor.java:233)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
at java.lang.Thread.run(Thread.java:748)


What can this be?

Regards,
Ido


Alexander Lehmann

unread,
Dec 19, 2017, 7:12:47 AM12/19/17
to vert.x
not quite sure, but you may be missing a runOnContext when calling vert.x code from your ExecutorService thread.

Ido Barash

unread,
Dec 19, 2017, 7:18:04 AM12/19/17
to ve...@googlegroups.com
I do run on context:

```
executor.submit( () -> {
   
  // Get the message
sen
   vertx.getOrCreateContext().runOnContext(v -> callback.handle(message.getBody()));
.
});

```


--
You received this message because you are subscribed to a topic in the Google Groups "vert.x" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vertx/SSnbEG3AGJA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vertx+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/b39d58b9-d6b3-44b2-b966-027428d4fc7e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Idob

unread,
Dec 19, 2017, 7:19:09 AM12/19/17
to vert.x
I do run on context:



executor
.submit( () -> {
   
 
// Get the message
sen
   vertx
.getOrCreateContext().runOnContext(v -> callback.handle(message.getBody()));

.
});



Thomas SEGISMONT

unread,
Dec 19, 2017, 11:31:12 AM12/19/17
to ve...@googlegroups.com
If you call getOrCreateContext from an arbitrary thread (non Vert.x thead), it will always return a new context.

Change your code to:

Context context = vertx.getOrCreateContext();
executor.submit(() -> {
  // Get the message
  context.runOnContext(v -> callback.handle(message.getBody()));
});

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+unsubscribe@googlegroups.com.

Idob

unread,
Dec 19, 2017, 11:33:44 AM12/19/17
to vert.x
Done that earlier... It is still the same.

Can it happen because I have more than one instance?



On Tuesday, December 19, 2017 at 1:54:08 PM UTC+2, Idob wrote:

Thomas SEGISMONT

unread,
Dec 19, 2017, 11:34:45 AM12/19/17
to ve...@googlegroups.com
Hard to say without more information (larger snippet, reproducer)

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.

Idob

unread,
Dec 19, 2017, 11:42:28 AM12/19/17
to vert.x
This is a larger sinppet - I removed much boilerplater




private Map<String, ExecutorService> consumerThreadsMap = new ConcurrentHashMap<>();

private Context currentContext;

private numOfConsumerThreads;

public void init() {
  currentContext = vertx.getOrCreateContext();
}


public void consume(String address, int numOfConsumerThreads, Handler<String> callback) {

        ExecutorService executor = getOrCreateExecutor(address, numOfConsumerThreads);

        for (int i = 0; i < numOfConsumerThreads; i++) {
            Thread.currentThread().setName("vert.x-sqs-consumer-thread-" + i);
            executor.submit(() -> {
                
                while (!Thread.currentThread().isInterrupted()) {
                    try {
                        
                        List<Message> messages = gatherMessages(envAndQueueAddress);
                        if (messages != null && messages.size() > 0) {
                            messages.forEach(message -> currentContext.runOnContext(v -> callback.handle(message.getBody())););
                        }
                    } catch (Exception e) {
                        log.error("EventQueueSQS, Error inside consuming loop(Continuing to loop). ", e);
                    }
                }
            });
        }
    }


private ExecutorService getOrCreateExecutor(String address, int numOfConsumerThreads) {
        ExecutorService executor = consumerThreadsMap.get(address);
        if (executor == null) {
            executor = Executors.newFixedThreadPool(numOfConsumerThreads);
            consumerThreadsMap.put(address, executor);
        }
        return consumerThreadsMap.get(address);
    }






On Tuesday, December 19, 2017 at 1:54:08 PM UTC+2, Idob wrote:

Thomas SEGISMONT

unread,
Dec 19, 2017, 11:53:36 AM12/19/17
to ve...@googlegroups.com
Don't change the name of Vert.x threads. Also, make sure this class:
- is not invoked concurrently (otherwise you could create multiple ExecutorService for the same address
- is invoked from a Vert.x thread (otherwise getOrCreateContext will always return a new context

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.

Jez P

unread,
Dec 19, 2017, 12:16:53 PM12/19/17
to vert.x
Are you calling this code from within a verticle?
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.

id...@bluerbn.com

unread,
Dec 19, 2017, 12:18:35 PM12/19/17
to vert.x
Hi Jez, 
Yes. we call this from verticle. we inject this class(as spring bean) 

Jez P

unread,
Dec 19, 2017, 12:28:57 PM12/19/17
to vert.x
Do you have only one instance per verticle instance, with a guarantee that only that verticle instance will call into your bean instance? If not your init will store the context from one verticle instance but you could call it from another which could lead to what you're seeing. 

If you don't have that strong a coupling, I'd strongly recommend passing the context into the call rather than storing it in the bean.

Julien Viet

unread,
Dec 19, 2017, 4:05:58 PM12/19/17
to vert.x
Hi,

this happens because of a bug in Vert.x / Netty, i.e Netty is calling the Channel event loop with the wrong thread (the acceptor thread afair)

do you have a reproducer / which version are you using ?

Julien

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
Message has been deleted

hyginous neto

unread,
Feb 9, 2018, 2:31:44 PM2/9/18
to vert.x
Hi Julien,

We are getting same error with vertx-core 3.5.0, when we embed vertx based java component within jetty.

Julien Viet

unread,
Feb 10, 2018, 2:28:04 AM2/10/18
to ve...@googlegroups.com
Hi,

the http client pooling in 3.5.1 has been improved, you should check with this version.

do you have a reproducer for 3.5.0 ? that would help to check.

Julien

hyginous neto

unread,
Feb 17, 2018, 11:18:09 PM2/17/18
to vert.x
Hi Julien,

Actually, We are vertx NetClient (for tcp).

Julien Viet

unread,
Feb 19, 2018, 1:02:44 AM2/19/18
to ve...@googlegroups.com
do you have a strack trace and a reproducer ?

your case is different from the case listed below because the case below is using Vert.x HttpClient and you are using NetClient.

at least the stack trace would give us insights.

Message has been deleted

hyginous neto

unread,
Feb 19, 2018, 4:46:00 AM2/19/18
to vert.x
This error comes when using SLF4J with vertx

error
: java.lang.IllegalStateException: Uh oh! Event loop context executing with wrong thread! Expected null got Thread[globalEventExecutor-1-85,5,main] io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:318) io.vertx.core.impl.ContextImpl$$Lambda$28.0000000020E0A490.run(Unknown Source) io.vertx.core.impl.ContextImpl.executeFromIO(ContextImpl.java:195) io.vertx.core.net.impl.NetClientImpl.failed(NetClientImpl.java:264) io.vertx.core.net.impl.NetClientImpl.lambda$doConnect$7(NetClientImpl.java:225)

   fileName: io.netty.util.internal.logging.Slf4JLogger 

Julien Viet

unread,
Feb 19, 2018, 6:14:38 AM2/19/18
to ve...@googlegroups.com
can you provide a reproducer ?

i.e a small program with a build file that we can use to reproduce it

thanks

Julien

hyginous neto

unread,
Feb 20, 2018, 7:42:29 PM2/20/18
to vert.x
Hi Julien @julien,

Finally, we found the root cause.

It was due to conflict in netty jars between vertx and async-http-client

DefaultChannelPromise@669f9cbf(failure: io.netty.channel.ChannelException: Unable to create Channel from class class io.netty.channel.socket.nio.NioSocketChannel)

vertx 3.5.1 uses 4.1.19.Final and async-http-client 2.0.36 uses 4.0.51.Final.

Is there a vertx version which supports old version 4.0.51?

Jez P

unread,
Feb 21, 2018, 2:25:42 AM2/21/18
to vert.x
Async-http-client 2.1.x seem to use netty 4.1.x which should be compatible with that used by vert.x 3.5.1.
Also https://mvnrepository.com/artifact/io.vertx/vertx-core/3.2.1 suggests that vert.x 3.2.1 should use netty 4.0.x which means you should be able to use 4.0.51.

Julien Viet

unread,
Feb 21, 2018, 3:39:49 AM2/21/18
to ve...@googlegroups.com
no we don't.

but there are versions of AsyncHttpClient that use now Netty 4.1 so you should rather upgrade AsyncHttpClient

Julien

Reply all
Reply to author
Forward
0 new messages