Vertx HTTP client is blocking the event loop

[gideon caller]

Hi everyone,

I'm using Vertx for some sort of a real time scraper.

The flow goes something like this: 

- in the main program I'm reading some URLs (specifically in my current situation 300 URLs) and then I pass them to a CrawlerVerticle using the event bus. 

- When the CrawlerVerticle receives the URL it uses the Vertx HTTP client with the getAbs method to send the request to that URL. In the bodyhandler callback all I do is extract the HTML and send it using a JsonObject to the eventbus for another verticle to process (I create the JsonObject using Vertx Json class with the encode method using a POJO class). In addition to the getAbs method I also hang an exception handler which all it does is print an "error occurred". After hanging all the callbacks I invoke request.end()

The problem is that after a few seconds of running I start getting warnings about blocking the event loop with increasing delays (initially 2196 and afterwards 5197) until I start receiving "thread blocked" exceptions, the only part of my code that appears in the stacktrace is the crawler (the request.end())

Could anyone shed light on what may be blocking here?

Google didn't produce any interesting suggestions on how to solve this

Thanks in advance!

[Tim Fox]

The logs you get should tell you exactly where in your code the event loop is being blocked.

Without seeing that or your code it would be hard to provide further advice.

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/de4d90ef-0d51-487c-8c11-d6b95b18e44e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alexander Lehmann]

[Alexander Lehmann]

Sorry, clicked Post by accident.

When the time interval of the blocking warning is increasing that means that the execution is hanging in a specific place for longer, the warning and the exepection will repeat until blocking part is finished, this doesn't mean that there are different calls blocking that take longer and longer.

You should be able to see what is hanging in the stack trace, maybe something is using a blocking call like file write, it would be easiest if you could put up a small program that shows the issue.

[gideon caller]

The crawling verticle code is:

public class CrawlerVerticle extends AbstractVerticle {

    @Override

    public void start() throws Exception {

        MetricsService.create(vertx);

        HttpClientOptions options = new HttpClientOptions().setTcpKeepAlive(false).setConnectTimeout(1000);

        HttpClient httpClient = vertx.createHttpClient(options);

        EventBus eventBus = vertx.eventBus();

        MessageConsumer<String> consumer = eventBus.consumer("crawl");

        consumer.handler(event -> {

            String url = event.body();

            HttpClientRequest request = httpClient.getAbs(url, response -> {

                if (isSuccess(response)) {

                    response.bodyHandler(buffer -> {

                        String content = buffer.toString();

                        JsonObject jsonObject = new JsonObject(

                                Json.encode(new UrlInfo(url, content))

                        );

                        eventBus.send("parse", jsonObject);

                    });

                } else {

                    System.out.println("HTTP FAILURE IN CRAWLER. Status code: " + response.statusCode() + " for URL: " + url);

                }

            });

            request.exceptionHandler(throwable -> {

                System.out.println("CRAWLER HTTP CLIENT ERROR!");

                throwable.printStackTrace();

            });

            request.end();

        });

        consumer.exceptionHandler(throwable -> {

            System.out.println("CRAWLER ERROR!");

            throwable.printStackTrace();

        });

    }

    private boolean isSuccess(HttpClientResponse response) {

        int statusCode = response.statusCode();

        return 200 <= statusCode && statusCode < 400;

    }

}

The logs I receive are:

Feb 25, 2016 12:36:27 PM io.vertx.core.impl.BlockedThreadChecker

WARNING: Thread Thread[vert.x-eventloop-thread-3,5,main] has been blocked for 2196 ms, time limit is 2000

Feb 25, 2016 12:36:28 PM io.vertx.core.impl.BlockedThreadChecker

WARNING: Thread Thread[vert.x-eventloop-thread-3,5,main] has been blocked for 3196 ms, time limit is 2000

Feb 25, 2016 12:36:29 PM io.vertx.core.impl.BlockedThreadChecker

WARNING: Thread Thread[vert.x-eventloop-thread-3,5,main] has been blocked for 4196 ms, time limit is 2000

Feb 25, 2016 12:36:30 PM io.vertx.core.impl.BlockedThreadChecker

WARNING: Thread Thread[vert.x-eventloop-thread-3,5,main] has been blocked for 5197 ms, time limit is 2000

io.vertx.core.VertxException: Thread blocked

at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)

at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)

at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)

at java.net.InetAddress.getAllByName0(InetAddress.java:1276)

at java.net.InetAddress.getAllByName(InetAddress.java:1192)

at java.net.InetAddress.getAllByName(InetAddress.java:1126)

at java.net.InetAddress.getByName(InetAddress.java:1076)

at java.net.InetSocketAddress.<init>(InetSocketAddress.java:220)

at io.vertx.core.http.impl.HttpClientImpl.internalConnect(HttpClientImpl.java:749)

at io.vertx.core.http.impl.HttpClientImpl.access$000(HttpClientImpl.java:61)

at io.vertx.core.http.impl.HttpClientImpl$1.connect(HttpClientImpl.java:93)

at io.vertx.core.http.impl.ConnectionManager$ConnQueue.createNewConnection(ConnectionManager.java:173)

at io.vertx.core.http.impl.ConnectionManager$ConnQueue.getConnection(ConnectionManager.java:114)

at io.vertx.core.http.impl.ConnectionManager.getConnection(ConnectionManager.java:69)

at io.vertx.core.http.impl.HttpClientImpl.getConnection(HttpClientImpl.java:674)

at io.vertx.core.http.impl.HttpClientRequestImpl.connect(HttpClientRequestImpl.java:586)

at io.vertx.core.http.impl.HttpClientRequestImpl.write(HttpClientRequestImpl.java:735)

at io.vertx.core.http.impl.HttpClientRequestImpl.end(HttpClientRequestImpl.java:329)

at com.volcanodata.vertxpoc.CrawlerVerticle.lambda$start$3(CrawlerVerticle.java:45)

at com.volcanodata.vertxpoc.CrawlerVerticle$$Lambda$32/1091876389.handle(Unknown Source)

at io.vertx.core.eventbus.impl.HandlerRegistration.handleMessage(HandlerRegistration.java:207)

at io.vertx.core.eventbus.impl.HandlerRegistration.handle(HandlerRegistration.java:201)

at io.vertx.core.eventbus.impl.EventBusImpl.lambda$deliverToHandler$127(EventBusImpl.java:498)

at io.vertx.core.eventbus.impl.EventBusImpl$$Lambda$36/212890971.handle(Unknown Source)

at io.vertx.core.impl.ContextImpl.lambda$wrapTask$18(ContextImpl.java:335)

at io.vertx.core.impl.ContextImpl$$Lambda$21/1409160703.run(Unknown Source)

at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380)

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)

at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)

at java.lang.Thread.run(Thread.java:745)

I know the code is ugly but it's only a POC

I'll be happy to understand how to look at the logs better in order to identify the problem

Let me know if there's anything else I should add

Thanks

[Tim Fox]

Ah blocking DNS lookup. This is a known issue.

--

You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/58132c20-007d-449b-a53a-aa8c53867077%40googlegroups.com.

[gideon caller]

Could you tell that by seeing lookupAllHostAddr?

Could you elaborate on the problem a little bit more? Also, Is there anyway to solve this?

[Tim Fox]

This will be solved when Netty 4.1 comes out which has async DNS resolution.

Until then, you can work around the issue by resolving the IP address yourself either by running InetAddress.getByName() in an executeBlocking block or using the Vert.x async DNS client.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/b35a1839-01c6-47d0-814e-301a01d8ed84%40googlegroups.com.

[gideon caller]

I cannot wait for Netty 4.1

Could you share some code examples on how to do any of the 2 solutions you suggested? Is any of them preferable in some way?

Thanks!

[Clement Escoffier]

Hi,

Here is the executeBlocking solution:

https://github.com/vert-x3/vertx-mail-client/blob/386508287e91b20710633a54fbab9f8e4f524774/vertx-mail-client/src/main/java/io/vertx/ext/mail/impl/SMTPConnection.java#L183

Clement

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/b7b37bca-6f56-4dd2-9a1b-9f9f3e922837%40googlegroups.com.

[Alexander Lehmann]

Actually, it might not be as easy as a NetClient connect, since the hostname is contained in the Host: header which the server can evaluate and it might cause problems with https, but it certainly worth a try.

[gideon caller]

Thanks for the replies, however  as Alexander pointed out I need a lot of different hosts, is there a way to overcome this with Vertx? If I'll be using a different async client (besides Vertx HttpClient) can it solve my blocking DNS issue? thanks!

You received this message because you are subscribed to a topic in the Google Groups "vert.x" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vertx/FgI5sf-DAK0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vertx+un...@googlegroups.com.

Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/dc9f4767-a4e3-44eb-9123-0d504aa5044d%40googlegroups.com.

[clement escoffier]

Hi,

If you are in a worker verticle or execute the request in an executeBlocking call, you can use any Http client (okhttp, apache http client, ning http client...).

Clement

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/CAOJxNisEHjNgwyv-_ZN_TWUG608kQu5eSuD4Lb6LoPqR83O6uA%40mail.gmail.com.

[gideon caller]

Clement, however I am looking for an asynchronous solution since I'm expecting relatively large requests throughput

If I'll use Apache Async Client instead of the Vertx HttpClient, will it work?

And more generally, if I'm aiming for an async scraping solution - is there anyway Vertx can allow me to do that? Modifying configurations? using different async HTTP clients? downgrading version? Or any other solution possible?

Thanks

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/F65F3E61-4E6C-43F6-B79A-554615A958A5%40gmail.com.

[Clement Escoffier]

Hi,

If you use the async HTTP client (from Apache), you pass a callback notified when the request has been “completed”. This callback is called in a thread owned by the executor service from the client (not from Vert.x). So it will work, but you would need to be careful about this thread change. In the callback if you want to go back to the Vert.x event loop thread, you would need to do something like: context.runInContext(…);

Most Async HTTP client would work the same way, so non blocking - with callback, and so you will just need to be careful about the thread on which the callback is executed.

Clement

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/CAOJxNiupVkoEU6%3DM2Tc7zmbKnhMBK3pET5tC%3D7su-C44aza7YA%40mail.gmail.com.

[gideon caller]

but will it be able to overcome the blocking DNS issue?

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/4838A84E-94E8-4C11-A0EE-062EF37A124F%40gmail.com.

[clement escoffier]

It depends on the client library. If the lookup (address resolution) is done in the caller thread (event loop) or in background. Hopefully they are doing it in background but I don't known them enough to be sure at 100%.

Clement

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/CAOJxNisb8pVKvmRO6e1PfjHhpc-k5MaowhMEJfTKwaJAqJAKeQ%40mail.gmail.com.

[gideon caller]

So just to see if I get this right, you're basically saying that if I'm experiencing the blocking DNS issue with Vertx and going to different hosts then my only 2 options are: 1. use blocking http clients in an "executeBlocking" 2. use some async client which is not vertx httpclient and then run it in the vertx context?

Is there no better solution out there?

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/53C5D5AC-3641-4C14-8A54-68A576D51007%40gmail.com.

[ad...@cs.miami.edu]

I imagine that a DNS result is cached somewhere.  No?  And that every request to myDomain.com does not need to re-call the blocking DNS.

So, unless you have a huge number of different domains to look up, I would not imagine it will cause too many problems.  Yes, the first time it does an httpclient request to an non-cached domain, it might do a little blocking.  But after that I think it should be pretty smooth.  How long is it really taking to resolve the DNS for a domain?

You could probably just extend the time for the blocking timeout exception, and not worry about it.  Your app might be sluggish for a split second when it resolves a DNS the first time, but it should not take too long.  And then things should continue as expected.   With multiple verticles and multiple event loops, I imagine it would work fine.

Of course the above is moot if you have a a huge number of domains to resolve (like in the case for a general proxy), or if the DNS is not cached.

-Adam

[ad...@cs.miami.edu]

You could also keep a map of "resolved domains", that is domains that you have already sent a request to, and thus the DNS should be cached.

If you have never visited a particular domain, then do it first in an execute blocking (which should cache the domain), and then add the domain to the above map, and from then on whenever you see the same domain, just use non blocking goodness.

-Adam

[Tim Fox]

Option 3: Submit a PR with a fix.... if you can get it in before the core team fixes it :)

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/CAOJxNiss7%3Ds26P%2B9zx%2BbwE7BPxufHPYVrP4Mr0gMKHBzyALXGw%40mail.gmail.com.

[gideon caller]

If adam's suggestions work it would be great, otherwise I'll have to go for the 3rd option :)

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/56D4B4AE.9080203%40gmail.com.

[Tim Fox]

I'm not sure what Adam's suggestion is but doing the blocking DNS lookup in an executeBlocking block and setting the host header yourself should work fine and only be a few lines of code.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/CAOJxNit5_FFsv%3D8Sk8KpzLeUTuO7YPGiMhY8EZOwhz-SqaXc0A%40mail.gmail.com.

[Tim Fox]

E.g. hopefully something like this (haven't tested it)

HttpClient client = ...


vertx.executeBlocking(fut -> {
    // 1. resolve IP address - this can block
    fut.complete(InetAddress.getByName("foo.com").getHostAddress());
}, res -> {
    if (res.succeeded()) {
        String ipAddress = res.result();
        HttpClientRequest req = client.get(ipAddress, "/someuri");
        req.putHeader("Host", "foo.com").handler(resp -> System.out.println("Got response " + resp.statusCode()).end();
    } else {
        // Handle failure
    }
});

[Tim Fox]

On 01/03/16 10:41, Tim Fox wrote:

E.g. hopefully something like this (haven't tested it)

HttpClient client = ...


vertx.executeBlocking(fut -> {

You should also probably set the ordered param to false on executeBlocking as you probably don't care about order here and that should improve throughput

[gideon caller]

Tim, thanks for your reply.

The issue is that most of the URLs I'm trying to fetch are from different host, also the URLs are not just the hosts themself, should I perform this executeBlocking on every URL I encounter? Also, thanks for the ordered tip

[gideon caller]

Just to be a bit clearer on the issue of the hosts what I meant to say is that the URLs I'm scraping are full URLs and not just hosts but there are many different hosts for the different URLs

[Tim Fox]

You should just resolve the host part

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/CAOJxNit6M4W1oZ1L-1cL3sBi1Wm9i%2B%3DhsZ7hCVhn_WLeEZrseg%40mail.gmail.com.

[gideon caller]

Is it ok if I'm doing that for 250K different hosts? Or is there a hosts number limit?

[Tim Fox]

Hopefully your DNS server is caching results.. if you're seeing large resolution times for the same host after the first time... try a better DNS server or perhaps you have a network misconfiguration that is resulting in lost packets (DNS often uses UDP which can lose packets).

You can also try caching lookups locally, but be careful not to hold on to the lookup for too long in case the DNS records change for the host (e.g. host changes IP(s))

[gideon caller]

Got it, thanks a lot