"Failed to create a new resolver"

1,237 views
Skip to first unread message

Konstantyn Smirnov

unread,
Nov 28, 2016, 9:43:01 AM11/28/16
to vert.x
Hi all.

In my Grails app I have a couple of vertx endpoints like:

    vertx.createHttpServer().websocketHandler( this.&wsHandler ).listen 8822

    vertx.createNetServer().connectHandler( this.&netHandler ).listen 8823

and they worked like charm, untill I added a dependency on https://github.com/MatrixCrawler/grails-spring-security-oauth2 (with scribejava inside).

Since then I'm getting the errors:

15:27:59.484 [main] ERROR i.v.core.http.impl.HttpServerImpl - java.lang.IllegalStateException: failed to create a new resolver
15:27:59.537 [main] ERROR io.vertx.core.net.impl.NetServerImpl - java.lang.IllegalStateException: failed to create a new resolver

How can I fix the problem?

Jez P

unread,
Nov 28, 2016, 10:02:42 AM11/28/16
to vert.x
Looks like this is coming out of netty rather than vert.x. Maybe you've created some kind of dependency conflict hell via a transitive dependency? 

Konstantyn Smirnov

unread,
Nov 28, 2016, 10:45:24 AM11/28/16
to vert.x
the dependency report looks clean.

I switched the logging for io.netty to DEBUG, but it didn't show any exception, like the AddressResolverGroup.java would.

I can see also the same errors coming from event-loop:

16:40:08.376 [vert.x-eventloop-thread-2] ERROR i.v.core.http.impl.HttpServerImpl - java.lang.IllegalStateException: failed to create a new resolver
16:40:08.388 [vert.x-eventloop-thread-2] ERROR io.vertx.core.net.impl.NetServerImpl - java.lang.IllegalStateException: failed to create a new resolver

Jez P

unread,
Nov 28, 2016, 11:14:36 AM11/28/16
to vert.x
Well that error message doesn't feature in vert.x code as far as I can see, but it does feature in the netty code that vert.x uses. So if it's not the netty code, sorry but you'll have to try debugging it I guess and see if you can see where it comes from.

Julien Viet

unread,
Nov 28, 2016, 4:58:56 PM11/28/16
to ve...@googlegroups.com
Hi,

can you configure logging to print the stack traces ?

-- 
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/e5b906c8-228f-4b2f-ba32-333c9cf211ce%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Konstantyn Smirnov

unread,
Nov 29, 2016, 5:50:05 AM11/29/16
to vert.x
this is my console output:

16:40:04.459 [main] DEBUG i.n.u.i.l.InternalLoggerFactory - Using SLF4J as the default logging framework
16:40:04.464 [main] DEBUG io.netty.util.ResourceLeakDetector - -Dio.netty.leakDetection.level: simple
16:40:04.466 [main] DEBUG io.netty.util.ResourceLeakDetector - -Dio.netty.leakDetection.maxRecords: 4
16:40:04.499 [main] DEBUG i.n.c.MultithreadEventLoopGroup - -Dio.netty.eventLoopThreads: 16
16:40:04.530 [main] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.Buffer.address: available
16:40:04.531 [main] DEBUG i.n.util.internal.PlatformDependent0 - sun.misc.Unsafe.theUnsafe: available
16:40:04.531 [main] DEBUG i.n.util.internal.PlatformDependent0 - sun.misc.Unsafe.copyMemory: available
16:40:04.532 [main] DEBUG i.n.util.internal.PlatformDependent0 - direct buffer constructor: available
16:40:04.533 [main] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.Bits.unaligned: available, true
16:40:04.533 [main] DEBUG i.n.util.internal.PlatformDependent0 - java.nio.DirectByteBuffer.<init>(long, int): available
16:40:04.535 [main] DEBUG io.netty.util.internal.Cleaner0 - java.nio.ByteBuffer.cleaner(): available
16:40:04.535 [main] DEBUG i.n.util.internal.PlatformDependent - Platform: Windows
16:40:04.536 [main] DEBUG i.n.util.internal.PlatformDependent - Java version: 8
16:40:04.536 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noUnsafe: false
16:40:04.536 [main] DEBUG i.n.util.internal.PlatformDependent - sun.misc.Unsafe: available
16:40:04.537 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noJavassist: false
16:40:04.585 [main] DEBUG i.n.util.internal.PlatformDependent - Javassist: available
16:40:04.585 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.tmpdir: C:\Users\me\AppData\Local\Temp (java.io.tmpdir)
16:40:04.585 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.bitMode: 64 (sun.arch.data.model)
16:40:04.585 [main] DEBUG i.n.util.internal.PlatformDependent - -Dio.netty.noPreferDirect: false
16:40:04.585 [main] DEBUG i.n.util.internal.PlatformDependent - io.netty.maxDirectMemory: 771751936 bytes
16:40:04.624 [main] DEBUG io.netty.channel.nio.NioEventLoop - -Dio.netty.noKeySetOptimization: false
16:40:04.624 [main] DEBUG io.netty.channel.nio.NioEventLoop - -Dio.netty.selectorAutoRebuildThreshold: 512
16:40:04.628 [main] DEBUG i.n.util.internal.PlatformDependent - org.jctools-core.MpscChunkedArrayQueue: available
16:40:04.679 [main] DEBUG i.n.resolver.dns.DnsServerAddresses - Default DNS servers: [/192.168.0.1:53] (sun.net.dns.ResolverConfiguration)
....
16:40:06.083 [main] DEBUG io.netty.buffer.AbstractByteBuf - -Dio.netty.buffer.bytebuf.checkAccessible: true
16:40:06.090 [main] DEBUG i.n.util.ResourceLeakDetectorFactory - Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@1b25d695
16:40:06.450 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.numHeapArenas: 7
16:40:06.450 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.numDirectArenas: 7
16:40:06.450 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.pageSize: 8192
16:40:06.451 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxOrder: 11
16:40:06.451 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.chunkSize: 16777216
16:40:06.451 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.tinyCacheSize: 512
16:40:06.451 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.smallCacheSize: 256
16:40:06.451 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.normalCacheSize: 64
16:40:06.451 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.maxCachedBufferCapacity: 32768
16:40:06.451 [main] DEBUG i.n.buffer.PooledByteBufAllocator - -Dio.netty.allocator.cacheTrimInterval: 8192
16:40:06.502 [main] DEBUG io.netty.util.NetUtil - -Djava.net.preferIPv4Stack: false
16:40:06.502 [main] DEBUG io.netty.util.NetUtil - -Djava.net.preferIPv6Addresses: false
16:40:06.545 [main] DEBUG io.netty.util.NetUtil - Loopback interface: lo (Software Loopback Interface 1, 127.0.0.1)
16:40:06.547 [main] DEBUG io.netty.util.NetUtil - \proc\sys\net\core\somaxconn: 200 (non-existent)
16:40:06.567 [main] DEBUG i.n.u.i.JavassistTypeParameterMatcherGenerator - Generated: io.netty.util.internal.__matchers__.io.netty.channel.socket.DatagramPacketMatcher
16:40:06.588 [main] DEBUG i.n.u.i.JavassistTypeParameterMatcherGenerator - Generated: io.netty.util.internal.__matchers__.io.netty.channel.AddressedEnvelopeMatcher
16:40:06.637 [main] ERROR i.v.core.http.impl.HttpServerImpl - java.lang.IllegalStateException: failed to create a new resolver
16:40:06.677 [main] ERROR io.vertx.core.net.impl.NetServerImpl - java.lang.IllegalStateException: failed to create a new resolver

Jez P

unread,
Nov 29, 2016, 6:45:43 AM11/29/16
to vert.x
Have you tried attaching a debugger to see where the IllegalStateException is being thrown? This is also likely to let you see the underlying exception/problem which will probably give you more insight as to what is going on. I don't see where Netty would log just before it throws that exception, so I think your assertion that you would see logging on the netty side relating to the exception is incorrect, by the way. I still think it's being thrown by Netty and vert.x is just propagating it outwards. 


On Monday, November 28, 2016 at 2:43:01 PM UTC, Konstantyn Smirnov wrote:

Konstantyn Smirnov

unread,
Nov 29, 2016, 11:07:06 AM11/29/16
to vert.x
Debugging a Grails application (embedded Tomcat + groovy code + tons of dependencies) is not really a simple thing to accomplish...

About my "assertions" it's simple: the vert.x servers used to work fine BEFORE I added a scribejava dependency into the project. scribejava is  using netty heavily, and is initialized prior to vert.x during app bootstrap.

Jez P

unread,
Nov 29, 2016, 11:15:03 AM11/29/16
to vert.x
That says nothing about your claim that you would see some kind of error logging in netty (the assertion to which I refer). The netty code doesn't support that statement at all (see the links to it I posted earlier, there's no debug logging of that exception, so no reason to think that if netty threw it it would appear in the log).

As regards debugging, it's the same as debugging any other java application. You start up your java process with debug flags (I'm sure grails lets you do this) and attach a debugger. All you then have to do is set a breakpoint triggered by throwing of IllegalStateException anywhere in your code, and then trigger your error. The breakpoint will do the rest and show you exactly where the exception is thrown and you'll be able to identify the triggering condition.

As I said, vert.x doesn't throw that exception, with that message, but netty does. One way or another that exception is coming out of netty. 

Konstantyn Smirnov

unread,
Nov 29, 2016, 11:23:22 AM11/29/16
to vert.x
>All you then have to do is set a breakpoint triggered by throwing of IllegalStateException anywhere in your code, and then trigger your error.

The problem is, that the ISE is not thrown but rather logged with ERROR level. Also it's not my code, as the error is clearly logged by io.vertx.core.** classes.

Jez P

unread,
Nov 29, 2016, 11:26:09 AM11/29/16
to vert.x
It's LOGGED by vert.x
It's THROWN by netty

If you want to find out what's wrong, you need to understand why netty is THROWING it.

Jez P

unread,
Nov 29, 2016, 11:28:39 AM11/29/16
to vert.x
You can't blindly blame vert.x when vert.x behaved perfectly well before you added scribejava as a dependency. There's clearly a conflict with the way the two work with netty, which may be resolvable or may not, but until the reason netty throws the exception is diagnosed, there's no chance of finding out whether or not it's resolvable. 

Konstantyn Smirnov

unread,
Nov 30, 2016, 5:50:09 AM11/30/16
to vert.x
I'm not blaming anything, I want to get my problem fixed.

I added

    vertx.createHttpServer().websocketHandler( this.&wsHandler ).listen( 8822 ){ res ->
      res.cause().printStackTrace()
    }

and got the stacktrace:


java.lang.IllegalStateException: failed to create a new resolver
    at io.netty.resolver.AddressResolverGroup.getResolver(AddressResolverGroup.java:69)
    at io.vertx.core.impl.AddressResolver.resolveHostname(AddressResolver.java:223)
    at io.vertx.core.impl.VertxImpl.resolveAddress(VertxImpl.java:690)
    at io.vertx.core.net.impl.AsyncResolveConnectHelper.doBind(AsyncResolveConnectHelper.java:74)
    at io.vertx.core.http.impl.HttpServerImpl.listen(HttpServerImpl.java:256)
    at io.vertx.core.http.impl.HttpServerImpl.listen(HttpServerImpl.java:191)
    at io.vertx.groovy.core.http.HttpServer.listen(HttpServer.groovy:178)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springsource.loaded.ri.ReflectiveInterceptor.jlrMethodInvoke(ReflectiveInterceptor.java:1426)
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrap.invoke(PogoMetaMethodSite.java:190)
    at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:71)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:133)
    at com.mozaiq.net.GatewayVerticle.start(GatewayVerticle.groovy:31)
    at io.vertx.lang.groovy.GroovyVerticle.start(GroovyVerticle.groovy:64)
    at com.mozaiq.net.GatewayVerticle.start(GatewayVerticle.groovy)
    at io.vertx.lang.groovy.GroovyVerticle$1.start(GroovyVerticle.groovy:93)
    at io.vertx.core.impl.DeploymentManager.lambda$doDeploy$8(DeploymentManager.java:434)
    at io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:316)
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:418)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:440)
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
    at io.netty.resolver.dns.DnsNameResolver.<init>(DnsNameResolver.java:213)
    at io.netty.resolver.dns.DnsNameResolverBuilder.build(DnsNameResolverBuilder.java:348)
    at io.vertx.core.impl.AddressResolver$1.newResolver(AddressResolver.java:167)
    at io.netty.resolver.AddressResolverGroup.getResolver(AddressResolverGroup.java:67)
    ... 27 more

does this ring any bell?

Jez P

unread,
Nov 30, 2016, 10:06:04 AM11/30/16
to vert.x
Only that DnsNameResolver line 213 (on netty master) doesn't appear to have code which would throw an NPE. Which version of netty is in use in your project, and are you definitely sure that vert.x and scribejava are using the same netty dependency version? 

Jez P

unread,
Nov 30, 2016, 10:20:47 AM11/30/16
to vert.x
213 in netty-resolver-dns 4.1.5 (used by vert.x 3.3) is

List cachedEntries = resolveCache.get(hostname);

So for whatever reason, use of scribejava alongside vert.x leads to resolveCache being null (assuming you're using 4.1.5) at this point. As far as I can see this is injected at instantiation time, so no idea why this is occurring. 

Konstantyn Smirnov

unread,
Nov 30, 2016, 10:41:13 AM11/30/16
to vert.x
I excluded the netty from security plugin dependencies:

  compile( 'org.grails.plugins:spring-security-oauth2:1.1.0' ){
    exclude group:'io.netty'
  }
  compile 'io.vertx:vertx-core:3.3.3'
  compile( 'io.vertx:vertx-lang-groovy:3.3.3' ){ exclude module:'groovy-all' }
  compile 'io.vertx:vertx-web:3.3.3'
  compile 'io.vertx:vertx-tcp-eventbus-bridge:3.3.3'

so it gives me the following deps:

+--- org.grails.plugins:spring-security-oauth2:1.1.0
|    +--- org.codehaus.groovy:groovy-all:2.4.3 -> 2.4.7
|    +--- com.ning:async-http-client:1.9.38
|    |    \--- org.slf4j:slf4j-api:1.7.12 -> 1.7.21
|    +--- org.asynchttpclient:async-http-client:2.0.4
|    |    +--- org.asynchttpclient:netty-resolver-dns:2.0.4
|    |    |    +--- org.asynchttpclient:netty-resolver:2.0.4
|    |    |    |    \--- org.slf4j:slf4j-api:1.7.21
|    |    |    +--- org.asynchttpclient:netty-codec-dns:2.0.4
|    |    |    |    \--- org.slf4j:slf4j-api:1.7.21
|    |    |    \--- org.slf4j:slf4j-api:1.7.21
|    |    +--- org.reactivestreams:reactive-streams:1.0.0
|    |    +--- com.typesafe.netty:netty-reactive-streams:1.0.4
|    |    |    \--- org.reactivestreams:reactive-streams:1.0.0
|    |    +--- org.javassist:javassist:3.20.0-GA -> 3.18.1-GA
|    |    \--- org.slf4j:slf4j-api:1.7.21
|    \--- com.github.scribejava:scribejava-apis:2.7.3
|         \--- com.github.scribejava:scribejava-core:2.7.3
+--- io.vertx:vertx-core:3.3.3
|    +--- io.netty:netty-common:4.1.5.Final
|    +--- io.netty:netty-buffer:4.1.5.Final
|    |    \--- io.netty:netty-common:4.1.5.Final
|    +--- io.netty:netty-transport:4.1.5.Final
|    |    +--- io.netty:netty-buffer:4.1.5.Final (*)
|    |    \--- io.netty:netty-resolver:4.1.5.Final
|    |         \--- io.netty:netty-common:4.1.5.Final
|    +--- io.netty:netty-handler:4.1.5.Final
|    |    +--- io.netty:netty-buffer:4.1.5.Final (*)
|    |    +--- io.netty:netty-transport:4.1.5.Final (*)
|    |    \--- io.netty:netty-codec:4.1.5.Final
|    |         \--- io.netty:netty-transport:4.1.5.Final (*)
|    +--- io.netty:netty-handler-proxy:4.1.5.Final
|    |    +--- io.netty:netty-transport:4.1.5.Final (*)
|    |    +--- io.netty:netty-codec-socks:4.1.5.Final
|    |    |    \--- io.netty:netty-codec:4.1.5.Final (*)
|    |    \--- io.netty:netty-codec-http:4.1.5.Final
|    |         \--- io.netty:netty-codec:4.1.5.Final (*)
|    +--- io.netty:netty-codec-http:4.1.5.Final (*)
|    +--- io.netty:netty-codec-http2:4.1.5.Final
|    |    +--- io.netty:netty-codec-http:4.1.5.Final (*)
|    |    \--- io.netty:netty-handler:4.1.5.Final (*)
|    +--- io.netty:netty-resolver:4.1.5.Final (*)
|    +--- io.netty:netty-resolver-dns:4.1.5.Final
|    |    +--- io.netty:netty-resolver:4.1.5.Final (*)
|    |    +--- io.netty:netty-codec-dns:4.1.5.Final
|    |    |    \--- io.netty:netty-codec:4.1.5.Final (*)
|    |    \--- io.netty:netty-transport:4.1.5.Final (*)
|    +--- com.fasterxml.jackson.core:jackson-core:2.7.4 -> 2.6.7
|    \--- com.fasterxml.jackson.core:jackson-databind:2.7.4 -> 2.6.7 (*)

Jez P

unread,
Nov 30, 2016, 11:09:53 AM11/30/16
to vert.x
What if you didn't exclude the dependency? At the moment my best suggestion is that you may need to start up in debug mode with a breakpoint on instantiation of the DnsNameResolver class to see if you can spot why it's instantiated with a null cache. I wonder if some global setup is happening (doesn't look like it should from the netty code). 

Thomas SEGISMONT

unread,
Nov 30, 2016, 11:16:14 AM11/30/16
to ve...@googlegroups.com
Could you try again adding this on your command line -Dvertx.disableDnsResolver=true ?

Just a wild guess but given the exception it could be related to http://vertx.io/docs/vertx-core/java/#_host_name_resolution


--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+unsubscribe@googlegroups.com.

Konstantyn Smirnov

unread,
Nov 30, 2016, 11:55:25 AM11/30/16
to vert.x
If I don't exclude the dependencies, they will be overriden by younger versions. So, it doesn't change anything.

It indeed looks like, that some "static" (re-)initializtion happens in scribejava's netty setup. I'll try to debug it.

Julien Viet

unread,
Nov 30, 2016, 12:59:05 PM11/30/16
to ve...@googlegroups.com
great, looking forward to see the result of your investigations.

-- 
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.

Konstantyn Smirnov

unread,
Dec 2, 2016, 4:39:44 AM12/2/16
to vert.x
unfortunately that doesn't change anything...

Alexander Lehmann

unread,
Dec 3, 2016, 5:21:49 PM12/3/16
to vert.x
I looked around a bit in the Netty code for dns resolver, but I cannot quite locate where the issue should happen that matches the stacktrace.

Assuming that DnsCache is null, that is about the only argument in the DnsNameResolver constructor that is not validated so I would guess the programmers expect it to not be null in any case.

Would it be possible that you put the source of your project up (preferable stripped to the minimal parts that exhibit the issue) so we can check the issue directly?

If it is some kind of library conflict, I am not getting how it happens.

Konstantyn Smirnov

unread,
Jan 9, 2017, 8:04:26 AM1/9/17
to vert.x
At the end I decided to factor the http and net verticles from the grails app out, which I wanted to do at some moment anyway. The problem forced me to make a decision earlier.

Konstantyn Smirnov

unread,
Jul 14, 2017, 7:16:59 AM7/14/17
to vert.x
As a follow-up: in Vertx 3.4.2 no conflict arises!
Reply all
Reply to author
Forward
0 new messages