WARNING Thread vert.x-worker-thread has been blocked

[Leonid Shamis]

Hello,

According to the documentation - http://vertx.io/docs/vertx-core/java/#worker_verticles - "Worker verticles are designed for calling blocking code, as they won’t block any event loops."

I have tried using Worker Verticle to execute potentially blocking code - subscribing to and receiving messages from ZeroMQ socket. The code works - see https://github.com/LeonidShamis/ZeroMQ_pub_sub/blob/master/sub_vertx/src/main/java/zeromqVertxSubJava/JeroMQSubscriberVerticle.java, but after 60 seconds I start getting "io.vertx.core.VertxException: Thread blocked" exception as shown below.

Could you please help me understanding why using Worker Verticle to execute potentially blocking code doesn't work or has adverse effect resulting in the following exception?

My Vert.x Worker Verticle code is located in https://github.com/LeonidShamis/ZeroMQ_pub_sub/tree/master/sub_vertx and the code that publishes messages to ZeroMQ is located at https://github.com/LeonidShamis/ZeroMQ_pub_sub/tree/master/pub_nodejs

Appreciate your help.

WARNING: Thread Thread[vert.x-worker-thread-0,5,io.vertx.core.Launcher] has been blocked for 60576 ms, time limit is 60000

io.vertx.core.VertxException: Thread blocked

        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)

        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)

        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)

        at zmq.Signaler.waitEvent(Signaler.java:132)

        at zmq.Mailbox.recv(Mailbox.java:90)

        at zmq.SocketBase.processCommands(SocketBase.java:906)

        at zmq.SocketBase.recv(SocketBase.java:829)

        at org.zeromq.ZMQ$Socket.recv(ZMQ.java:2501)

        at org.zeromq.ZMQ$Socket.recvStr(ZMQ.java:2573)

        at zeromqVertxSubJava.JeroMQSubscriberVerticle.start(JeroMQSubscriberVerticle.java:31)

        at io.vertx.core.AbstractVerticle.start(AbstractVerticle.java:111)

        at io.vertx.core.impl.DeploymentManager.lambda$doDeploy$8(DeploymentManager.java:434)

        at io.vertx.core.impl.DeploymentManager$$Lambda$27/22466669.handle(Unknown Source)

        at io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:337)

        at io.vertx.core.impl.ContextImpl$$Lambda$28/484849014.run(Unknown Source)

        at io.vertx.core.impl.TaskQueue.lambda$new$0(TaskQueue.java:60)

        at io.vertx.core.impl.TaskQueue$$Lambda$25/1951193928.run(Unknown Source)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

[Asher Tarnopolski]

this is not an error, but a warning. 

if i'm not mistaken you can re-configure it to a higher value, or disregard if such a long block is fine with you.  

[Leonid Shamis]

Thank you for the prompt response.

I tried "VertxOptions().setMaxWorkerExecuteTime(120000000000L)" to increase max worker execute time to 120 seconds, but that just increases the value of max worker execute time (DEFAULT_MAX_WORKER_EXECUTE_TIME) = 60000000000 ns (60 seconds)

All it achieves is that the WARNINGS start appearing after 120 seconds.

From more research I start suspecting that doing potentially blocking activities should not be done in start() - I see some comments about it here https://groups.google.com/forum/#!msg/vertx/xyvUdOeXL4A/0mDbfF2XCAAJ

So, maybe I will have to move the code that listens and receives messages from ZeroMQ socket to Vert.x periodic timer function...

[Julien Viet]

Hi,

worker threads can block but they should block for a reasonable amount of time, for instance flushing a file on the disk.

if you need to do some kind of polling you should your own thread, for instance polling a Kafka consumer (and that’s what vertx-kafka-client does).

Julien

-- 
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/d843178b-60ba-4a12-b775-9b9ef14858dd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Leonid Shamis]

Hi Julien,

Thank you for the explanation.

I have changed my Worker Verticle from long polling while loop 

while (!Thread.currentThread().isInterrupted())

to vertx.setPeriodic timer that executes lambda to consume messages from ZeroMQ socket - https://github.com/LeonidShamis/ZeroMQ_pub_sub/blob/master/sub_vertx/src/main/java/zeromqVertxSubJava/JeroMQSubscriberVerticle.java

It is working better now, without WARNING about blocked tread, BUT only if there are messages to consume!

If I stop publishing messages, then after 60 seconds I start getting the same WARNING messages about the blocked thread.

If I resume publishing messages, then they start being consumed again and the WARNING stops.

Q: Is this the expected behavior?

There is also another side effect - I cannot interrupt the running Vert.x program now using CTRL+C.

Appreciate your help.

Leonid

[Julien Viet]

On Aug 17, 2017, at 2:51 AM, Leonid Shamis <shamis...@gmail.com> wrote:

Hi Julien,

Thank you for the explanation.

I have changed my Worker Verticle from long polling while loop 

while (!Thread.currentThread().isInterrupted())

to vertx.setPeriodic timer that executes lambda to consume messages from ZeroMQ socket - https://github.com/LeonidShamis/ZeroMQ_pub_sub/blob/master/sub_vertx/src/main/java/zeromqVertxSubJava/JeroMQSubscriberVerticle.java

It is working better now, without WARNING about blocked tread, BUT only if there are messages to consume!

If I stop publishing messages, then after 60 seconds I start getting the same WARNING messages about the blocked thread.

If I resume publishing messages, then they start being consumed again and the WARNING stops.

Q: Is this the expected behavior?

There is also another side effect - I cannot interrupt the running Vert.x program now using CTRL+C.

yes this is expected, you should not use Vert.x worker thread for this and instead use your own thread.

look at Kafka consumer for Vert.x : https://github.com/vert-x3/vertx-kafka-client/blob/master/src/main/java/io/vertx/kafka/client/consumer/impl/KafkaReadStreamImpl.java#L101

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/d40f72dd-a047-4f2d-8b22-7e5fe020677d%40googlegroups.com.