Vertx3 HttpClient blocking bug?

[Oliver Rolle]

Hi,

atm I am porting my vertx2 to vertx3 and discovered a strange issue. Vertx3 seems to block the event loop during host name resolution. If I do these requests against localhost, its not an issue.

Code
public static void request(final Vertx vertx, final String url, final Handler<String> result) throws IOException {
    HttpClient client = vertx.createHttpClient();

    System.out.println("HTTP GET");
    HttpClientRequest req = client.get(80, "<my-domain.com>", url, res -> {
      res.bodyHandler(body -> {
        result.handle(body.toString());
      });
    });
   
    System.out.println("before end()");
   
    req.end();
   
    System.out.println("after end()");
  }

Output
HTTP GET
before end()
Jul 08, 2015 11:33:54 AM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-2,5,main] has been blocked for 2831 ms, time limit is 2000
Jul 08, 2015 11:33:55 AM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-2,5,main] has been blocked for 3832 ms, time limit is 2000
Jul 08, 2015 11:33:56 AM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-2,5,main] has been blocked for 4831 ms, time limit is 2000
Jul 08, 2015 11:33:57 AM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-eventloop-thread-2,5,main] has been blocked for 5831 ms, time limit is 2000
io.vertx.core.VertxException: Thread blocked
    at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
    at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:907)
    at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1302)
    at java.net.InetAddress.getAllByName0(InetAddress.java:1255)
    at java.net.InetAddress.getAllByName(InetAddress.java:1171)
    at java.net.InetAddress.getAllByName(InetAddress.java:1105)
    at java.net.InetAddress.getByName(InetAddress.java:1055)
    at java.net.InetSocketAddress.<init>(InetSocketAddress.java:220)
    at io.vertx.core.http.impl.HttpClientImpl.internalConnect(HttpClientImpl.java:653)
    at io.vertx.core.http.impl.HttpClientImpl.access$000(HttpClientImpl.java:59)
    at io.vertx.core.http.impl.HttpClientImpl$1.connect(HttpClientImpl.java:91)
    at io.vertx.core.http.impl.ConnectionManager$ConnQueue.createNewConnection(ConnectionManager.java:154)
    at io.vertx.core.http.impl.ConnectionManager$ConnQueue.getConnection(ConnectionManager.java:94)
    at io.vertx.core.http.impl.ConnectionManager.getConnection(ConnectionManager.java:63)
    at io.vertx.core.http.impl.HttpClientImpl.getConnection(HttpClientImpl.java:587)
    at io.vertx.core.http.impl.HttpClientRequestImpl.connect(HttpClientRequestImpl.java:500)
    at io.vertx.core.http.impl.HttpClientRequestImpl.write(HttpClientRequestImpl.java:640)
    at io.vertx.core.http.impl.HttpClientRequestImpl.end(HttpClientRequestImpl.java:286)
    at ft.crawler.crawl.RequestProcessor.request(RequestProcessor.java:40)
    at ft.crawler.crawl.RequestCoordinator.process(RequestCoordinator.java:37)
    at ft.crawler.crawl.RequestCoordinator.add(RequestCoordinator.java:25)
    at ft.crawler.country.CountryCrawler.recursive(CountryCrawler.java:31)
    at ft.crawler.country.CountryCrawler.lambda$0(CountryCrawler.java:19)
    at ft.crawler.country.CountryCrawler$$Lambda$10/1993776452.accept(Unknown Source)
    at java.util.Arrays$ArrayList.forEach(Arrays.java:3880)
    at ft.crawler.country.CountryCrawler.<init>(CountryCrawler.java:18)
    at ft.crawler.Main.lambda$0(Main.java:19)
    at ft.crawler.Main$$Lambda$6/99828496.handle(Unknown Source)
    at io.vertx.core.impl.VertxImpl$InternalTimerHandler.handle(VertxImpl.java:711)
    at io.vertx.core.impl.VertxImpl$InternalTimerHandler.handle(VertxImpl.java:682)
    at io.vertx.core.impl.ContextImpl.lambda$wrapTask$15(ContextImpl.java:314)
    at io.vertx.core.impl.ContextImpl$$Lambda$5/706277948.run(Unknown Source)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
    at java.lang.Thread.run(Thread.java:745)

[Tim Fox]

On 08/07/15 10:42, Oliver Rolle wrote:

Hi,

atm I am porting my vertx2 to vertx3 and discovered a strange issue. Vertx3 seems to block the event loop during host name resolution.

I think you'll find the same behaviour in V2. The blocking is actually occurring in the JDK InetAddress class which does the DNS lookup.

In your case this lookup happens to be extremely slow (6 seconds... seems a very long time).

We could do this lookup on a worker thread or use the Vert.x async DNS lookup functionality. Could you add a BZ for this?

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Oliver Rolle]

The lookup is slow because my wifi here is bad (package losses). The ping command resolves the dns name within 1-2 seconds and works with vertx2. I believe there is some other issue as well.

Bug report: https://bugs.eclipse.org/bugs/show_bug.cgi?id=472166

[Nat]

It would also be very nice if HttpClient/NetClient interface accepts InetAddress + host name so that if the name has been previously resolved, Vert.x does not have to perform that again.

[aleksande...@gmail.com]

I noticed the same... even though I reuse the same HttpClient object it keeps resolving IP address on each get request. This seems wasteful, regardless of the blocking/non blocking issue. It would be great to have HttpClient either remember the resolved address or allow to pass it explicitly in the api.

[Alexander Lehmann]

It may be required to resolve the address occasionally in some applications or it might be necessary to observe a cache ttl, so resolving it only once by default isn't the right behaviour either. If the system has a dns cache, resolving the cached entry is a local operation as long as you get a cache hit.

[Tim Fox]

We will fix the blocking lookup issue after Netty 4.1.0 comes out (not long) as it contains support for async name resolution when binding servers and clients.

Until then you can workaround it by using the Vert.x DNSClient to do the lookup and caching the IP address yourself.

In general we don't want to cache IP (as Alex mentioned too) as we want IP changes to be picked up - e.g. you don't want your client trying to connect to the old IP after a web-site has migrated to a new IP (but this could be an option I guess).

Visit this group at http://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/2ecd3c1e-7847-42ce-9a6e-dae2eba1a0b6%40googlegroups.com.

[Jeremy Truelove]

Do you have anyone to work on this feature, the Netty 4.1.0 integration for DNS lookups?

[Julien Viet]

Hi, this should be in 3.3 release.

Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/80051111-3044-47c8-a6f5-1eb978eff081%40googlegroups.com.

[Jeremy Truelove]

So is the work already done or do you need a volunteer to do it?

--
You received this message because you are subscribed to a topic in the Google Groups "vert.x" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vertx/LL8LNjwD1Mg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vertx+un...@googlegroups.com.

Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/C967AB6F-630E-41D9-A49C-63D2DB0C6B85%40julienviet.com.

[Julien Viet]

no we don’t have anyone working on it yet, so it can be contributed. (there is an http2 branch in Vert.x Core that upgrades Vert.x to Netty 4.1.0-CR1 that can be reused).

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/CABdWGQuSu7k9NaOV%2BcaXoATJBbzc%3DbuUX4WAbwipm_DVjO42Ag%40mail.gmail.com.

[Jeremy Truelove]

Cool I'll try and pull that branch down and take a stab at it.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/67CACB15-3CE5-467C-B990-F045E4D08ABC%40julienviet.com.