Vertx under load test - high worker thread queue delay

[Tim Lindgren]

Hi!

I am currently running load tests against my Vertx servers (running 3.3.0) and see a behaviour which I am a bit surprised of seeing. It's probably nothing strange but after spending quite some time on it, I still cannot figure out what the bottleneck is.

I have a setup where a server receives HTTP requests on the event loop (using Vertx web). It forwards the requests to my business layer running on the same verticle and then starts a blocking section of the code where it contacts a MySQL database (using the Vertx.executeBlockingObservable call). This works nicely until I increase the load and I see start seeing slow response times. By exposing several metrics I see that the Vertx worker thread queue delay is getting high while the CPU of the server is around 20%. The worker pool also seems to only use a few threads, it's a resource which I also monitor. The connection pool to the database also doesn't seem to reach its limit (Hikari).

The thing I am a bit surprised of is that the CPU utilization is low abd there are available workers but the worker thread queue delay is high. Is there a simple explanation or are there just too many things that could happen? I know this might be a too general question but I am hoping I am missing something simple =)

Cheers,

Tim

[Tim Lindgren]

It seems by changing the "ordered" flag in the Vertx.executeBlockingObservable call to true I got my expected behavior. I guess there was an actual queue for my worker threads. Now I just need to understand if that change is fine in my code base, even though after reading the documentation I am a bit unsure if it's a safe change!

[Nat]

What does the stacktrace look like?

[Julien Viet]

Hi,

you can safely use ordered=false if you acquire and release the connection in the execute blocking code block.

it would help if you share some code, because we don’t know if you use vertx-jdbc-client or not, etc...

-- 
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/bd2e9bbd-b3c0-467c-a5b8-9c8e3e7a159f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tim Lindgren]

Hello!

Here is some code that is very similiar to the one I am using currently. I am using Spring's JdbcTemplate for handling my connections against the database (and TransactionTemplate for transactions). Spring should give my connections back to the connection pool (Hikari in this case):

In my DAO class:

  public Observable<MyObject> getObjectAsObservable() {

return vertx.executeBlockingObservable(future -> {

MyObject o = null;

try {

o = getMyObject();

} catch (Exception e) {

logger.error(e.getMessage());

future.fail(e);

return;

}

future.complete(o);

return;

}, false);

}

private MyObject getMyObject(String id) {

String objectData = jdbcTemplate.queryForObject("select o from myObjects where id=?;"), 

new Object[]{id}, String.class);

return JsonUtil.toPojo(objectData, MyObject.class);

}

I added the ordered flag there and it seems to solve my problems. I started seeing more Vertx workers threads being active at the same for example.

Nat, there weren't really any exceptions in the log. I guess it was just a queue filling up.

Kind regards,

Tim

[Manish Kumar]

Tim - Did you figure out why true flag had solve the issue for you?