vertx http client ... url formed incorrectly

[Patrick Conway]

I keep (seemingly randomly) get a network unreachable using a vertx http client with ssl ... any one have any ideas?

abstract public class GoogleDriveRequest extends StorageApiRequest {

    public static final int PORT = 443;

    protected static final String BASE_PATH = "/drive/v3";

    protected static final String FOLDER_MIME_TYPE = "application/vnd.google-apps.folder";

    protected static final String HOST = "https://www.googleapis.com";

    protected JsonObject json = new JsonObject();

    GoogleDriveRequest(Vertx vertx) {

        client = vertx.createHttpClient(new HttpClientOptions().setSsl(true));

    }

}

request = client.get(PORT,HOST,path);

java.net.SocketException: Network is unreachable: www.googleapis.com/2a00:1450:400c:c02:0:0:0:5f:443
at sun.nio.ch.Net.connect0(Native Method) ~[?:1.8.0_101]
at sun.nio.ch.Net.connect(Net.java:454) ~[?:1.8.0_101]
at sun.nio.ch.Net.connect(Net.java:446) ~[?:1.8.0_101]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[?:1.8.0_101]
at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:315) ~[InhanceCloudAPI-runnable.jar:?]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:256) [InhanceCloudAPI-runnable.jar:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1246) [InhanceCloudAPI-runnable.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:531) [InhanceCloudAPI-runnable.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:516) [InhanceCloudAPI-runnable.jar:?]
at io.netty.handler.ssl.SslHandler.connect(SslHandler.java:442) [InhanceCloudAPI-runnable.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:531) [InhanceCloudAPI-runnable.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:516) [InhanceCloudAPI-runnable.jar:?]
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.connect(CombinedChannelDuplexHandler.java:487) [InhanceCloudAPI-runnable.jar:?]
at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:47) [InhanceCloudAPI-runnable.jar:?]
at io.netty.channel.CombinedChannelDuplexHandler.connect(CombinedChannelDuplexHandler.java:288) [InhanceCloudAPI-runnable.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:531) [InhanceCloudAPI-runnable.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:516) [InhanceCloudAPI-runnable.jar:?]
at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50) [InhanceCloudAPI-runnable.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:531) [InhanceCloudAPI-runnable.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:516) [InhanceCloudAPI-runnable.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:498) [InhanceCloudAPI-runnable.jar:?]
at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:964) [InhanceCloudAPI-runnable.jar:?]
at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:248) [InhanceCloudAPI-runnable.jar:?]
at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:251) [InhanceCloudAPI-runnable.jar:?]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:339) [InhanceCloudAPI-runnable.jar:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:393) [InhanceCloudAPI-runnable.jar:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:742) [InhanceCloudAPI-runnable.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]

[Jayamine A.]

Hi,

Are you sure that "https://www.googleapis.com" works? 

For me it's not found. 

Most of the time the reason is the URL does not or no permission.

Here is a link.

http://www.wildpackets.com/resources/compendium/tcp_ip/unreachable

:-)

[Inhance]

yes it works. you only see 404 because there is no resource on root (/).

the code works most of the time... it just seems to sporadicaly insert what looks like an ipv6 address into the url
so the client isnt trying to reach www.googleapis.com:443 its attempting to connect to www.googleapis.com/2aao:77:00(...):443

and www.googleapis.com/2aao:77:00(...)

is not a valid hostname

Sent from my iPhone

m/resources/compendium/tcp_ip/unreachable

:-)

--
You received this message because you are subscribed to a topic in the Google Groups "vert.x" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vertx/7xTo_a_lOWk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/a414383f-a295-4501-ac83-ffee309fccf3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alexander Lehmann]

Google supports ipv6/ipv4 for all their hosts and since vert.x and netty support queries that check for ipv4 and ipv6 together and then use the one reply that came first, maybe your nameserver supports ipv6 but your routing doesn't (so the network unreachable).

When I resolve www.googleapis.com I get the following result:

Addresses:  2a00:1450:4001:815::200a
          216.58.210.10


I think you can use -Djava.net.preferIPv4Stack=true in netty, maybe that will work with vert.x as well.

[Inhance]

ah ok thanks. but the error suggests its appending the hostname with the ipv6 address rather than trying to connect to the ipv6 address. Although I could be reading that wrong.

Im using google compute engine to deploy, Ill look into its support of ipv6.

Sent from my iPhone

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/02c93c5e-6dfe-472e-9b3e-bb770b9b9c5c%40googlegroups.com.

[Alexander Lehmann]

the output in the error message with

www.googleapis.com/2a00:1450:400c:c02:0:0:0:5f:443

comes from the toString() of InetAddress, this is not really in the address at any point. The InetAddress object is created with the hostname www.googleapis.com and then resolved so that the object contains the hostname as well as the address.

When creating the error message, it uses address.toString() which will write both into the String.

the correct error message is probably

Network is unreachable: 2a00:1450:400c:c02:0:0:0:5f

I wouldn't think that google compute engine doesn't support ipv6, but its work a try (another possibility would be that there is actually a routing issue for the ipv6 network inside google network.

[Inhance]

ok thanks that makes sense now. compute engine doesn't support ipv6

Sent from my iPhone

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/48a271b4-ca01-481c-9054-538d43d781db%40googlegroups.com.

[Patrick Conway]

I disabled ipv6 on the vm...and set the java prefer ipv4 flag

still getting the error but it has changed slightly

to 

Protocol family unavailable: www.googleapis.com/2a00:1450:400c:c07:0:0:0:5f:443

[Alexander Lehmann]

This means that the dns request still gets an ipv6 address, but the local machine doesn't have ipv6 support.

I am not completely sure how vert.x handles the ipv6/ipv4 difference, maybe that is a bug (or the prefer ipv4 option is not available).

I will take a look at the source code, and I could get back to you some time tonight.

[Patrick Conway]

I found this flag, and so far haven’t seen the issue repeat

-Djava.net.preferIPv6Addresses=false

--
You received this message because you are subscribed to a topic in the Google Groups "vert.x" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vertx/7xTo_a_lOWk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/22634b13-b0a9-480a-93ad-aff4f5b582ea%40googlegroups.com.

[Patrick Conway]

-Djava.net.preferIPv6Addresses=false

Did not resolve the issue either

On Thursday, August 4, 2016 at 12:35:57 PM UTC+1, Patrick Conway wrote:

I found this flag, and so far haven’t seen the issue repeat

-Djava.net.preferIPv6Addresses=false

On 4 Aug 2016, at 12:34, Alexander Lehmann <alex...@gmail.com> wrote:

This means that the dns request still gets an ipv6 address, but the local machine doesn't have ipv6 support.

I am not completely sure how vert.x handles the ipv6/ipv4 difference, maybe that is a bug (or the prefer ipv4 option is not available).

I will take a look at the source code, and I could get back to you some time tonight.

On Thursday, August 4, 2016 at 11:28:27 AM UTC+2, Patrick Conway wrote:

I disabled ipv6 on the vm...and set the java prefer ipv4 flag

still getting the error but it has changed slightly

to 

Protocol family unavailable: www.googleapis.com/2a00:1450:400c:c07:0:0:0:5f:443

--
You received this message because you are subscribed to a topic in the Google Groups "vert.x" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vertx/7xTo_a_lOWk/unsubscribe.

To unsubscribe from this group and all its topics, send an email to vertx+unsubscribe@googlegroups.com.

[Alexander Lehmann]

I would expect that this property switches the dns resolver in netty to ipv4 preferred,maybe that is not working as expected.

I wonder if I could simulate the issue somehow

On Thursday, August 4, 2016 at 5:36:23 PM UTC+2, Patrick Conway wrote:

-Djava.net.preferIPv6Addresses=false

Did not resolve the issue either

On Thursday, August 4, 2016 at 12:35:57 PM UTC+1, Patrick Conway wrote:

I found this flag, and so far haven’t seen the issue repeat

-Djava.net.preferIPv6Addresses=false

On 4 Aug 2016, at 12:34, Alexander Lehmann <alex...@gmail.com> wrote:

This means that the dns request still gets an ipv6 address, but the local machine doesn't have ipv6 support.

I am not completely sure how vert.x handles the ipv6/ipv4 difference, maybe that is a bug (or the prefer ipv4 option is not available).

I will take a look at the source code, and I could get back to you some time tonight.

On Thursday, August 4, 2016 at 11:28:27 AM UTC+2, Patrick Conway wrote:

I disabled ipv6 on the vm...and set the java prefer ipv4 flag

still getting the error but it has changed slightly

to 

Protocol family unavailable: www.googleapis.com/2a00:1450:400c:c07:0:0:0:5f:443

--
You received this message because you are subscribed to a topic in the Google Groups "vert.x" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vertx/7xTo_a_lOWk/unsubscribe.

To unsubscribe from this group and all its topics, send an email to vertx+un...@googlegroups.com.

[Alexander Lehmann]

I cannot really reproduce the issue, when I run with -Djava.net.preferIPv6Addresses=false it resolves ipv4 address, when I run with =true, it resolves ipv6 addresses of the same address, which fails with Network unreachable when I run it in a vm that has ipv6 support disabled.

Would it be possible for you to run a unit test in the google cloud environment? I have created a simple project that connects to https://www.googleapis.com/, with logging that should show what the netty resolver tries to do.

https://github.com/alexlehm/vertx-ipv6-test

when running that with mvn test -Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.Log4jLogDelegateFactory -Djava.net.preferIPv6Addresses=false | tee mvn.log

it should create a log file that shows the resolver result.

[Inhance]

Thanks Alexander,

Ill run that tomorrow and let you know the result

Sent from my iPhone

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/744edead-3832-4301-b41f-ce29e97c7033%40googlegroups.com.

[Patrick Conway]

mvn.log attached

To unsubscribe from this group and all its topics, send an email to vertx+unsubscribe@googlegroups.com.

[Alexander Lehmann]

Ok,it looks like this time it worked, the address was resolved to a ipv4 address. Could you please run the test a few times to see if it sometimes fails? It would be interesting to see if that is ipv6 preferred=false as well.

In this case, there is probably a timing issue where the ipv4 and ipv6 resolution, which is probably not handled in the resolver.

[Patrick Conway]

I've run the test 250 times now without failure....

One change I made , I noticed the loopback interface still had an ipv6 address

so I disabled on all interfaces

https://wiki.centos.org/FAQ/CentOS7#head-8984faf811faccca74c7bcdd74de7467f2fcd8ee

Also wondering if added the ipv4 address to /etc/hosts would bypass the lookup?

I could live with the risk of google changing the address....  its a lesser problem than the failures

(although havent seen a failure since disable ipv6 on loopback)

Looking at sun.nio.ch.Net  it uses a native isIPv6Available() check  ..... Maybe its possible to find out what that check does?

https://github.com/frohoff/jdk8u-dev-jdk/blob/master/src/share/classes/sun/nio/ch/Net.java

static int connect(ProtocolFamily var0, FileDescriptor var1, InetAddress var2, int var3) throws IOException {
    boolean var4 = isIPv6Available() && var0 != StandardProtocolFamily.INET;
    return connect0(var4, var1, var2, var3);
}

static boolean isIPv6Available() {
    if(!checkedIPv6) {
        isIPv6Available = isIPv6Available0();
        checkedIPv6 = true;
    }

    return isIPv6Available;
}

private static native boolean isIPv6Available0();

[Alexander Lehmann]

the netty resolver uses the /etc/hosts file so that it would use the ip v4 address if you put one in

The native methods should be in the c++ code of the jdk, I will take a look

[Alexander Lehmann]

The openjdk8 sourcecode sets the ipv6 support flag here http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/src/share/native/java/net/net_util.c#l69

the function IPv6_supported() is checking something in the network stack and preferIPv4Stack is evaluated from the property java.net.preferIPv4Stack

[Patrick Conway]

hmmm...

I am setting preferIPv4Stack, but it doesn't seem to be working

this is the command I use to run my jar

java -Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.Log4j2LogDelegateFactory -server -Xms2g -Xmx4g -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:+ScavengeBeforeFullGC -XX:+CMSScavengeBeforeRemark -XX:+PrintGCDateStamps -verbose:gc -XX:+PrintGCDetails -Xloggc:/var/log/cloud-backup/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=50M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/cloud-backup/heapdump.hprof -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv6Addresses=false -d64 -jar CloudAPI-runnable.jar -conf /opt/cloud-api/devconfig.json

[Alexander Lehmann]

I suspect that this is an issue in netty, but I cannot reproduce it. The netty resolver is created as ipv4/ipv6 when useIpv6address is false and as ipv6/ipv4 when it is true.
In the other java code as in sun.net, the property -Djava.net.preferIPv4Stack=true turns off ipv6 complete (or almost completely), but that is not done in a similar way in netty.

If the issue happens when the ipv6 resolver gets a reply fast than the ipv4 resolver even though the ipv6 support is turned off (so that the ipv6 AAAA query is answered first), netty might choose the ipv6 address and then is unable to connect to it.

If the issue is still occuring in your program, it might be possible to add a bit of logging to the netty code or to patch the netty resolver to evaluate the perferIPv4Stack property and turn off ipv6 resolving completely in this case.

Would it be possible for you build your problem with a local netty version?

[Alexander Lehmann]

It would be interesting to create a reproducer that is a small program that shows the issue in google compute engine, but I am not sure how

[Patrick Conway]

I'm using maven so that should be possible

To unsubscribe from this group and all its topics, send an email to vertx+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/4c8046e5-a818-4370-8268-5792cb5c0ddc%40googlegroups.com.

[Alexander Lehmann]

Ok, I think its possible that I branch the netty project and you use that version to build your project with the same vert.x version.

For that I need to know which vert.x version you are using?

[Patrick Conway]

That would be great , thanks Alexander

I’m using version 3.3.0

but I just noticed 3.3.2 is available now…so I think I’ll move up to that

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/93dc38a0-d32a-4a21-9d02-520f6fe9abee%40googlegroups.com.

[Alexander Lehmann]

From 3.3.0 to 3.3.2, the netty version was not changed (4.1.1), I think that does not affect the resolver code.

I will try to create a snapshot version that can be used instead of the netty 4.1.1 with a bit of logging added.

[Alexander Lehmann]

There is an issue on the netty github now that describes the issue which was created by one of the netty authors.

https://github.com/netty/netty/issues/5657

(thanks to Julien for pointing to it)

The easiest way forward would be to wait for the netty version containing the  fix when it is finished and vert.x to use this version.

[Alexander Lehmann]

Patrick, if you want to try the fix before it gets done in netty, you could try the following (which is kind of a hack, but it should work).

Vert.x 3.3.2 contains the DNS code from netty since there were some fixes implemented that were not in the netty code yet (from 3.3.0-3.3.2, the next version will use the netty code directly), which means that the ipv6 resolver is set in a class in vertx., which you could replace by a changed class by putting it into your project.

https://gist.github.com/alexlehm/7659db45433605eb60de52f6e2d150ce

the class evaluates the prefer ipv4 stack property (https://gist.github.com/alexlehm/7659db45433605eb60de52f6e2d150ce#file-dnsnameresolver-java-L77) so that when you run your program with the property set, it should only use ipv4 dns resolution.

[Patrick Conway]

Thats great.

Thanks very much for your help Alexander, its much appreciated

Sent from my iPhone

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/d7a1f7f4-79c6-4775-b8b7-5abf3d1c68cb%40googlegroups.com.

[akshay kumar]

Did the latest version of Vert.x incorporate netty having this issue resolved?

[Julien Viet]

Hi,

the latest Vert.x uses Netty 4.1.19.Final which contains lot of fixes since Vert.x 3.3.2 :-)

Julien

You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.

Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/3d66fe44-99de-43c0-b82e-d4ddd94fb0fc%40googlegroups.com.