Not being able to resolve hostnames in Graphite Emitter with Druid 0.9.2

[s...@gameanalytics.com]

Hi

I'm trying to setup graphite emitter with Druid 0.9.2 and I've stumbled across a problem which I can't make sense of.

Configuration:

druid.monitoring.monitors=["com.metamx.metrics.JvmMonitor", "com.metamx.metrics.SysMonitor"]
druid.request.logging.type=emitter
druid.emitter=graphite
druid.emitter.logging.logLevel=debug
druid.emitter.graphite.hostname="carbon.hostedgraphite.com"
druid.emitter.graphite.port=2004
druid.emitter.graphite.eventConverter={"type":"all", "namespacePrefix": "druid.poc", "ignoreHostname":false, "ignoreServiceName":false}

The error is:

2017-02-10T12:23:10,251 INFO [GraphiteEmitter-0] io.druid.emitter.graphite.GraphiteEmitter - trying to connect to graphite server
2017-02-10T12:23:10,257 ERROR [GraphiteEmitter-0] io.druid.emitter.graphite.GraphiteEmitter - "carbon.hostedgraphite.com"
java.net.UnknownHostException: "carbon.hostedgraphite.com"
        at com.codahale.metrics.graphite.PickledGraphite.connect(PickledGraphite.java:182) ~[metrics-graphite-3.1.2.jar:3.1.2]
        at io.druid.emitter.graphite.GraphiteEmitter$ConsumerRunnable.run(GraphiteEmitter.java:145) [graphite-emitter-0.9.1.1.jar:0.9.1.1]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_121]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_121]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_121]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]

After a bit of digging in metrics-graphite-3.2 and java.net.inetAddress it seems the problem is due to name server lookups not working through Druid while dig as well as a stand alone Java program can do it[1].

I added a small patch to metrics-graphite-3.2 branch to see if this is the case, patch looks like:

diff --git a/metrics-graphite/src/main/java/com/codahale/metrics/graphite/PickledGraphite.java b/metrics-graphite/src/main/java/com/codahale/metrics/graphite/PickledGraphite.java
index 6917dd7..1d386ea 100644
--- a/metrics-graphite/src/main/java/com/codahale/metrics/graphite/PickledGraphite.java
+++ b/metrics-graphite/src/main/java/com/codahale/metrics/graphite/PickledGraphite.java
@@ -12,6 +12,7 @@ import java.io.OutputStream;
 import java.io.OutputStreamWriter;
 import java.io.Writer;
 import java.net.InetSocketAddress;
+import java.net.InetAddress;
 import java.net.Socket;
 import java.net.UnknownHostException;
 import java.nio.ByteBuffer;
@@ -176,7 +177,12 @@ public class PickledGraphite implements GraphiteSender {
         if (address == null) {
             address = new InetSocketAddress(hostname, port);
         }
+
         if (address.getAddress() == null) {
+            LOGGER.info("Is unresolved: {}", address.isUnresolved());
+            InetAddress i = InetAddress.getByName(hostname);
+            LOGGER.info("Trying to resolve {}", i);
+
             throw new UnknownHostException(address.getHostName());
         }

InetAddress.getByName is basically what's being called when InetAddress constructor when passing in a hostname and port.

Output is:

2017-02-12T16:23:36,233 INFO [GraphiteEmitter-0] io.druid.emitter.graphite.GraphiteEmitter - trying to connect to graphite server
2017-02-12T16:23:41,255 INFO [GraphiteEmitter-0] com.codahale.metrics.graphite.PickledGraphite - Is unresolved: true
2017-02-12T16:23:41,256 ERROR [GraphiteEmitter-0] io.druid.emitter.graphite.GraphiteEmitter - "carbon.hostedgraphite.com"
java.net.UnknownHostException: "carbon.hostedgraphite.com"
        at java.net.InetAddress.getAllByName0(InetAddress.java:1280) ~[?:1.8.0_121]
        at java.net.InetAddress.getAllByName(InetAddress.java:1192) ~[?:1.8.0_121]
        at java.net.InetAddress.getAllByName(InetAddress.java:1126) ~[?:1.8.0_121]
        at java.net.InetAddress.getByName(InetAddress.java:1076) ~[?:1.8.0_121]
        at com.codahale.metrics.graphite.PickledGraphite.connect(PickledGraphite.java:183) ~[metrics-graphite-3.1.2.jar:3.2.0-SNAPSHOT]
        at io.druid.emitter.graphite.GraphiteEmitter$ConsumerRunnable.run(GraphiteEmitter.java:150) [graphite-emitter-0.9.2.jar:0.9.2]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_121]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_121]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_121]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]

The exception being thrown on line 1280 with Oracle SE JDK 1.8.0_121 corresponds to this:

        if (addresses == unknown_array)
            throw new UnknownHostException(host);

addresses is set on line 1272

        InetAddress[] addresses = getCachedAddresses(host);

Its not clear to me how cacheInitIfNeeded works, it's been some time I've been tinkering with Java and before I dig further in perhaps someone has seen this issue or anything similar?

What's not making sense is why the name resolution doesn't work through Druid while it does with [1]? Does Druid change any system properties relating to name lookups?

Thanks,
Sam

[1]

import java.net.InetSocketAddress;
import java.net.InetAddress;
import java.net.UnknownHostException;

public class Resolve {

  public static void main(String[] args) {
    try {
      InetAddress i = InetAddress.getByName("carbon.hostedgraphite.com");
      System.out.println(i);
    } catch (UnknownHostException e) {
      System.out.println(e);
    }
  }
}

Gives result

carbon.hostedgraphite.com/136.243.95.165





[Slim Bouguerra]

Hi Sam Thanks for the report, i will try to look at this ASAP.

What is your OS ?

-- 

B-Slim
_______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______/\/\/\_______

--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/ebce28ff-c93e-44ed-98a5-cac0907e3f00%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[s...@gameanalytics.com]

Hi

I've tested this with Ubuntu 16.04.{1,2} LTS

Thanks,
Sam

[s...@gameanalytics.com]

Hi

After comparing DNS queries between the Resolve.java and plugin I pinpointed the problem. The configuration was wrong. It shouldn't be quoted hostname. I could see the quoted hostname being part of the lookup query which of course wouldn't work.