Deadlock in event loop threads

[Petar Partlov]

HI, we faced some deadlock in Vertx event loops. It happens in HttpClient, some time after we restart server side of connection. It actually happens couple of hours after it. I have JStack log of this deadlock:

"vert.x-eventloop-thread-0":

        at io.vertx.core.http.impl.HttpClientRequestImpl.getLock(HttpClientRequestImpl.java:554)

        - waiting to lock <0x00000006d43d04c8> (a io.vertx.core.http.impl.HttpClientRequestImpl)

        at io.vertx.core.http.impl.HttpClientRequestBase.handleResponse(HttpClientRequestBase.java:144)

        at io.vertx.core.http.impl.Http2ClientConnection$Http2ClientStream.handleHeaders(Http2ClientConnection.java:267)

        at io.vertx.core.http.impl.Http2ClientConnection.lambda$onHeadersRead$0(Http2ClientConnection.java:120)

        at io.vertx.core.http.impl.Http2ClientConnection$$Lambda$104/1578241090.run(Unknown Source)

        at io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:335)

        at io.vertx.core.impl.ContextImpl$$Lambda$100/543140181.run(Unknown Source)

        at io.vertx.core.impl.ContextImpl.executeFromIO(ContextImpl.java:193)

        at io.vertx.core.http.impl.Http2ClientConnection.onHeadersRead(Http2ClientConnection.java:119)

        - locked <0x00000006ed05dd88> (a io.vertx.core.http.impl.Http2ClientConnection)

        at io.vertx.core.http.impl.Http2ConnectionBase.onHeadersRead(Http2ConnectionBase.java:197)

        at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onHeadersRead(DefaultHttp2ConnectionDecoder.java:319)

        at io.netty.handler.codec.http2.DefaultHttp2FrameReader$1.processFragment(DefaultHttp2FrameReader.java:461)

        at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readHeadersFrame(DefaultHttp2FrameReader.java:468)

        at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:253)

        at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:160)

        at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:118)

        at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:341)

        at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:401)

        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411)

        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)

        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)

        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)

        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)

        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:129)

        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:642)

        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:565)

        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:479)

        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441)

        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)

        at java.lang.Thread.run(Thread.java:745)

"vert.x-eventloop-thread-1":

        at io.vertx.core.net.impl.ConnectionBase.metric(ConnectionBase.java:177)

        - waiting to lock <0x00000006ed05dd88> (a io.vertx.core.http.impl.Http2ClientConnection)

        at io.vertx.core.net.impl.ConnectionBase.reportBytesWritten(ConnectionBase.java:233)

        at io.vertx.core.http.impl.HttpClientRequestImpl.connected(HttpClientRequestImpl.java:773)

        - locked <0x00000006d43d04c8> (a io.vertx.core.http.impl.HttpClientRequestImpl)

        at io.vertx.core.http.impl.HttpClientRequestImpl.access$100(HttpClientRequestImpl.java:51)

        at io.vertx.core.http.impl.HttpClientRequestImpl$2.handleStream(HttpClientRequestImpl.java:719)

        at io.vertx.core.http.impl.ConnectionManager$ConnQueue.deliverStream(ConnectionManager.java:277)

        at io.vertx.core.http.impl.ConnectionManager$ConnQueue.lambda$getConnection$0(ConnectionManager.java:237)

        at io.vertx.core.http.impl.ConnectionManager$ConnQueue$$Lambda$153/318536498.handle(Unknown Source)

        at io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:337)

        at io.vertx.core.impl.ContextImpl$$Lambda$100/543140181.run(Unknown Source)

        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)

        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)

        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:445)

        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)

        at java.lang.Thread.run(Thread.java:745)

[Tim Fox]

You have two different event loops accessing the same http client request instance, this shouldn't happen.

Are you sharing objects between event loops? Hard to say much more without seeing any code.

[Petar Partlov]

I don't call HttpClient from event loop at all (as far as I understand). I don't execute anything on event loop because most of things are long running operations, so they are executed either from my own thread, or from worker thread pool. Does this mean I must have separate Http client from every thread it uses it? Also one of these deadlock is response (so probably callback). Is it always executed from same event loop as request, or it is not important?

[Tim Fox]

Again, very hard to say without seeing code, but I'd always recommend using the Vert.x threading model rather than trying to roll your own, it's the path of least resistance.

[Petar Partlov]

I would also suggest improved Javadoc comment in HttpClientImple class. In class comment it just says: This class is thread-safe. This is not completely true. Probably some more info should be provided regarding this. It is interesting that in regular work client works fine for long period of time, and we have application with multiple millions of daily active users so this is some special scenario.

[Petar Partlov]

Just to confirm, when you say Vert.x threading model it means that it should be safe to call client from worker or event loop thread? I can do that from context, available from Vertx.getOrCreateContext()?

[Julien Viet]

by safe we mean that you create an HttpClient within one Context (EventLoop or Worker) and you don’t share it with other Context.

we allow to use the HttpClient from a plain thread and it should work but you will not get the best performance with this model, this model may also run into deadlocks.

In your case it seems that you are using HTTP/2 which has a different behavior than HTTP/2 with respect to locking (the TCP connection can be shared between several HTTP concurrent requests) and perhaps there is an issue when used from plain thread.

So if you could provide a reproducer it would be a good great.

-- 
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/c5f32c29-78ae-4571-bf85-793c6f1dfb6d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Petar Partlov]

I will try to find some repo, but it will be hard I think. In the meantime I will provide top of the stack of two thread, executed from the regular java Scheduled Executor, which I think are callers of this client, it may help you:

"AuctionTimer-3":

        at io.vertx.core.http.impl.ConnectionManager$ConnQueue.getConnection(ConnectionManager.java:229)

        - waiting to lock <0x00000006ecfc79b8> (a io.vertx.core.http.impl.ConnectionManager$ConnQueue)

        at io.vertx.core.http.impl.ConnectionManager.getConnectionForRequest(ConnectionManager.java:177)

        at io.vertx.core.http.impl.HttpClientImpl.getConnectionForRequest(HttpClientImpl.java:915)

        at io.vertx.core.http.impl.HttpClientRequestImpl.connect(HttpClientRequestImpl.java:745)

        - locked <0x00000006d43d02f8> (a io.vertx.core.http.impl.HttpClientRequestImpl)

        at io.vertx.core.http.impl.HttpClientRequestImpl.write(HttpClientRequestImpl.java:867)

        at io.vertx.core.http.impl.HttpClientRequestImpl.end(HttpClientRequestImpl.java:343)

        - locked <0x00000006d43d02f8> (a io.vertx.core.http.impl.HttpClientRequestImpl)

"AuctionTimer-1":

        at io.vertx.core.http.impl.Http2ConnectionBase.isClosed(Http2ConnectionBase.java:134)

        - waiting to lock <0x00000006ed05dd88> (a io.vertx.core.http.impl.Http2ClientConnection)

        at io.vertx.core.http.impl.Http2ClientConnection.isValid(Http2ClientConnection.java:112)

        at io.vertx.core.http.impl.ConnectionManager$ConnQueue.getConnection(ConnectionManager.java:230)

        - locked <0x00000006ecfc79b8> (a io.vertx.core.http.impl.ConnectionManager$ConnQueue)

        at io.vertx.core.http.impl.ConnectionManager.getConnectionForRequest(ConnectionManager.java:177)

        at io.vertx.core.http.impl.HttpClientImpl.getConnectionForRequest(HttpClientImpl.java:915)

        at io.vertx.core.http.impl.HttpClientRequestImpl.connect(HttpClientRequestImpl.java:745)

        - locked <0x00000006d47414a0> (a io.vertx.core.http.impl.HttpClientRequestImpl)

        at io.vertx.core.http.impl.HttpClientRequestImpl.write(HttpClientRequestImpl.java:867)

        at io.vertx.core.http.impl.HttpClientRequestImpl.end(HttpClientRequestImpl.java:343)

        - locked <0x00000006d47414a0> (a io.vertx.core.http.impl.HttpClientRequestImpl)

[Julien Viet]

Hi,

that does not help much as these two threads don’t seem to be in deadlock situation.

Julien

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/a9bb7d9a-ac5c-45bf-a9c9-31a65742f8ec%40googlegroups.com.

[Petar Partlov]

Hi,

so, if I want to use HttpClient from multiple sources (event loop, worker pool and non-vertx threads) I should create separate HttpClient for each thread. What is best practice for this? Do you suggest I keep these clients in some ThreadLocal, or should I get Context and put this client in each context and reuse it this way? I also can force executing everything on Vertx (probably by calling executeOnContext or executeBlocking) if that is safer?

Sorry for additional questions, I just want to make it as stable as possible as it is very hard to find repro :)

Thanks.

[Tim Fox]

The path of least resistance is not to create your own threads. Let Vert.x do that for you. Deploy your code as verticles. Have a http client instance per verticle instance. That way everything is nice and simple and will work as designed with the best performance.

[Petar Partlov]

"Not creating our own threads" is not very easy in our case. We made communication lib, encapsulating things from Vertx we are using (Core, Web, Metrics, Circuit breaker...) and using that as our communication stack. We provide interface for easier creation of handlers, and for sending some HTTP requests. Our backend is very complex, have dozens of threads. Sending HTTP requests is sometimes scheduled action, sometimes depends of some DB changes, Cache notifications, files changes. HttpClient is used from at least 30 different use cases. If you try to make Vertx usable in Microservice architecture, for example, this is probably use-case: people will create some "communication" lib on top of it and use it in different services. Forcing use of just Vertx threads for this is kind of handicap. Besides that in most cases there is need for lot of different pools with different size and kind of works. Using same worker pool for all task will easily became nightmare.

[Jez P]

Then use the event bus rather than the http client from your own threads, but use a verticle-based model for the http client comms.

Sending to the eventbus is fine from your own threads and does not inherently complicate matters. Using things like the HttpClient will do so. Your mechanism for communication to the vert.x stuff should be via the eventbus, not direct calls elsewhere. 

[Petar Partlov]

Thank, that should be fine. I'll try that.

[Tim Fox]

Not sure if I agree with that. Vert.x has been designed from day 1 specifically for a microservice architecture. It provides a very simple threading model, which if you go with it, provides lots of advantages including easy scaling, virtual elimination of race conditions and great performance. If you fight against it and provide your own threading model... yes things will still work but your life will be a lot harder. I've seen this many times before. Also, btw, no one is forcing you to use a single pool. Vert.x provides functionality which allows you to safely use multiple worker pools in your app.

[Petar Partlov]

Yup I understand, it is great. It just requires little bit change in architecture, if I had it on my mind during development it would be much easier :), but that was long time ago.

@Tim so you suggest that I should have one or more Vertices responsible just for HttpClient calls and one or more Verticles responsible for deployment of server and handlers? If I want to call client I'll publish that call to event bus and client will pick that up and execute call?

[Tim Fox]

Usually you can just have a single verticle for the server and that's it. No need to split things up into lots of different verticles as that often just adds complexity.

[Petar Partlov]

I just need to be sure that I will use all available event loops for handling incoming requests. From what I see verticle is always processing events from the same thread. What this means exactly? Verticle has start and stop methods, and I will put creation of HTTP server and attach handler(s) in it. Once this start is executed HTTP server is up and I have for example 4 event loop threads are they all used for processing incoming requests or just one?

[Thomas SEGISMONT]

This document is worth reading to understand the relationship between verticles/event loops/contexts: https://github.com/vietj/vertx-materials/blob/master/src/main/asciidoc/Demystifying_the_event_loop.adoc

An event loop is assigned to each verticle instance. If you create a single instance of the verticle, you'll get one thread handling all the events (and this can be *a lot* already). If you need all cpus involved, deploy more instances of your verticle. This is explained in the Vert.x core documentation.

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/902cdfa6-1c54-4c91-996f-1e89a1e4e511%40googlegroups.com.

[Jez P]

And because I love self-promotion, a slightly different take on the same material http://vertx.io/blog/an-introduction-to-the-vert-x-context-object/

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.

Visit this group at https://groups.google.com/group/vertx.