[BlockedThreadChecker] Setting Max Worker Execute Time

[Agraj Mangal]

Hi, 

In my application, I've 2 normal verticles and 4 worker verticles. All of them extend the AbstractVerticle. And the application is deployed as a fat jar which uses "io.vertx.core.Starter" to deploy the main verticle, which then in turn deploys the other verticles reading their respective configuration from the disk. 

Now one of the Verticles uses periodic timers such as

vertx.setPeriodic(delay, handler -> {

// Talks to AWS SQS and uses Long Polling of 10 seconds

});

delay is 20 seconds and the Long polling interval is 10 seconds so it should not be blocked for more than 60 seconds ideally, which is the default maximum time a worker thread is allowed to be blocked. But somehow its taking more time and I'm still getting many exceptions of the following format:

Aug 05, 2015 3:19:50 PM io.vertx.core.impl.BlockedThreadChecker

WARNING: Thread Thread[vert.x-worker-thread-4,5,main] has been blocked for 128389 ms, time limit is 60000

io.vertx.core.VertxException: Thread blocked

at com.sun.org.apache.xerces.internal.util.SymbolTable.<init>(SymbolTable.java:79)

at com.sun.org.apache.xerces.internal.util.SymbolTable.<init>(SymbolTable.java:73)

at com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl.<init>(XMLStreamReaderImpl.java:91)

at com.sun.xml.internal.stream.XMLInputFactoryImpl.getXMLStreamReaderImpl(XMLInputFactoryImpl.java:277)

at com.sun.xml.internal.stream.XMLInputFactoryImpl.createXMLStreamReader(XMLInputFactoryImpl.java:129)

at com.sun.xml.internal.stream.XMLInputFactoryImpl.createXMLEventReader(XMLInputFactoryImpl.java:78)

at com.amazonaws.http.StaxResponseHandler.handle(StaxResponseHandler.java:89)

at com.amazonaws.http.StaxResponseHandler.handle(StaxResponseHandler.java:42)

at com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:1050)

at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:724)

at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:467)

at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:302)

at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2419)

at com.amazonaws.services.sqs.AmazonSQSClient.receiveMessage(AmazonSQSClient.java:1130)

at com.example.abc.notif.verticles.workers.PostProcessingVerticle.lambda$start$22(PostProcessingVerticle.java:48)

at com.example.abc.notif.verticles.workers.PostProcessingVerticle$$Lambda$34/1619894293.handle(Unknown Source)

at io.vertx.core.impl.VertxImpl$InternalTimerHandler.handle(VertxImpl.java:711)

at io.vertx.core.impl.VertxImpl$InternalTimerHandler.handle(VertxImpl.java:682)

at io.vertx.core.impl.ContextImpl.lambda$wrapTask$15(ContextImpl.java:314)

at io.vertx.core.impl.ContextImpl$$Lambda$7/667447085.run(Unknown Source)

at io.vertx.core.impl.OrderedExecutorFactory$OrderedExecutor.lambda$new$161(OrderedExecutorFactory.java:91)

at io.vertx.core.impl.OrderedExecutorFactory$OrderedExecutor$$Lambda$2/1259769769.run(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Solution : To increase the Max Worker Execute Time from 60 seconds to lets say 130 seconds 

BUT But but... This option is available on VertxOptions [1] which is when you create a Vertx Instance and cannot be used at the Verticle level ( like in DeploymentOptions or something like that ) 

The problem now comes is that as I'm extending AbstractVerticle which gives me the inherited vertx instance by default, when a Verticle is instantiated, so I'm not initializing the Vertx instance on my own and even if I do that in one of the verticles other verticles would be deployed in the default vertx and this verticle would be disconnected from them. 

So I need to share the same vertx instance across all the Verticles. Since I'm not aware of the nuisances of classloading etc w.r.t Vertx initialization and instantiation of Verticles, so my questions is How do I share my custom Vertx instance with all the Verticles ? 

Please let me know if something is not clear.

Regards,

Agraj

[1] - http://vertx.io/docs/apidocs/io/vertx/core/VertxOptions.html#setMaxWorkerExecuteTime-long-

[Agraj Mangal]

Hi Guys,

Any suggestions on how to tackle this problem ?

Regards,

Agraj

[Nat]

I would try to solve the problem with stuck thread first. Do you have a lot of connection to SQS? Can you take the entire stack dump when the problem happens? Can you also try to increase AWS maximum HTTP connections ( http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/ClientConfiguration.html#setMaxConnections(int) )?

However, if you want to specify the option, you can use system properties, vertx.options.maxWorkerExecuteTime to specify a higher timeout.

[Agraj Mangal]

Hi Nat,

Actually I don't have many connections to SQS - am maintaining just a single instance of AmazonSQSClient is being used for reading and writing to it. Took a thread dump but could not see anything useful than just that the Thread was always "waiting on condition"

Thanks for the command line option. Perhaps I will end up using that to get away from these exceptions

Regards,

Agraj

--
You received this message because you are subscribed to a topic in the Google Groups "vert.x" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vertx/zcAfHhveU1Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vertx+un...@googlegroups.com.
Visit this group at http://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/3e7ad0b5-2f07-4176-8b74-7a392bb17402%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks & Regards,

Agraj Mangal

[Tim Fox]

I have to say, I don't really understand the issue.... why not just increase the timeout in VertxOptions?

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.

Visit this group at http://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/64e3c258-5ef2-4b9c-ac97-c20fe8d0b174%40googlegroups.com.

[Tim Fox]

On 06/08/15 16:30, Nat wrote:

I would try to solve the problem with stuck thread first.

+1

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at http://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/3e7ad0b5-2f07-4176-8b74-7a392bb17402%40googlegroups.com.

[Russ Heibel]

I favor the comment "why not just increase the timeout in VertxOptions?" by Tim above. However, this does not seem to be as straightforward as the comment suggests. I have the same problem with my usage of Worker Verticles. I have a main vertx loop that deploys many Worker verticles to perform computationally intensive tasks on data, from a database, and then, those in turn deploy worker verticles to place the results into the same (external) database. The main vertx loop runs on a 5 second timer and its execution slices are very small (< a couple of milliseconds). I incorporated the following VertxOptions and I still get the same WARNING that Agraj shows above.

      VertxOptions vertxOptions = new VertxOptions();
      vertxOptions.setMaxWorkerExecuteTime(2147483647); // maximum number for 2^31 - 1
      Vertx.vertx().deployVerticle(Main.class.getName());

Further, the documentation for VertxOptions found here:
http://vertx.io/docs/apidocs/io/vertx/core/VertxOptions.html#setMaxWorkerExecuteTime-long-

presents the following:

          public VertxOptions setMaxWorkerExecuteTime(long maxWorkerExecuteTime)

                    Sets the value of max worker execute time, in ns.

                    Parameters:

          maxWorkerExecuteTime - the value of max worker execute time, in ms.

                    Returns:

                    a reference to this, so the API can be used fluently

Notice that the description of the method indicates that the parameter is in NANOSECONDS. But the description of the variable is identified as units of MILLISECONDS.
Since I am not quite up to Agraj's capabilities yet I am hopeful that one of you guys that are expert in this can help with resolving these issues. The value I use above  (2147483647) doesn't seem to have any effect. I am setting this VertxOptions immediately upon entering the constructor in the main loop and don't know if this carries to all Worker Verticles in this vertx instance. Any help, especially by some example(s) would be greatly appreciated.

Thanks,

Russ H.

[Clement Escoffier]

Hi,

On 11 août 2016, at 02:04, Russ Heibel <rsh1...@gmail.com> wrote:

I favor the comment "why not just increase the timeout in VertxOptions?" by Tim above. However, this does not seem to be as straightforward as the comment suggests. I have the same problem with my usage of Worker Verticles. I have a main vertx loop that deploys many Worker verticles to perform computationally intensive tasks on data, from a database, and then, those in turn deploy worker verticles to place the results into the same (external) database. The main vertx loop runs on a 5 second timer and its execution slices are very small (< a couple of milliseconds). I incorporated the following VertxOptions and I still get the same WARNING that Agraj shows above.

      VertxOptions vertxOptions = new VertxOptions();
      vertxOptions.setMaxWorkerExecuteTime(2147483647); // maximum number for 2^31 - 1
      Vertx.vertx().deployVerticle(Main.class.getName());

You don’t pass the options to Vert.x. Try with:

Vertx.vertx(vertxOptions).deployVerticle(Main.class.getName());

Clement

Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/fc72dfdc-7e0e-4677-8226-7c941c29d496%40googlegroups.com.

[Russ Heibel]

Thank you Clement, I did not previously recognize that option. Incorporating your suggestion had a very noticeable improvement but it still appears I receive the WARNING:

"WARNING: Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 5302 ms, time limit is 2000"

each at points that seem to suggest 5000ms even after .setMaxWorkerExecuteTime(2147483647) has been passed in as the VertxOptions. Can you confirm whether the number passed into .setMaxWorkerExecuteTime() is in ms or ns? And why the vertx-blocked-thead-checker might be checking against 5000ms rather than the timeframe passed in as the VertxOptions?

Thanks again,

Russ

[Clement Escoffier]

Hi,

You have configured the worker timeout (worker verticle, executeBlocking…), and it seems your are blocking the event loop. I don’t recommend blocking the event loop as you reduce (dramatically) the scalability. So, I would recommend moving the operation blocking the event loop to a worker verticle or inside an executeBlocking construct.

Clement

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/e7465957-9dbe-444e-86df-240d31643fe7%40googlegroups.com.