Tranquility - UnknownHostException when ingesting data from spark into druid

[ram.th...@gmail.com]

I am trying to ingest data from spark into druid through tranquility. I am following the exact sample code in the Tranquility repo. 

While ingesting, I am seeing the following when tranquility is trying to establish a ZK connection:

16/03/10 03:08:14 ERROR imps.CuratorFrameworkImpl: Background exception was not retry-able or retry gave up
java.net.UnknownHostException:  <HOST>.com
	at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
	at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:901)
	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1295)
	at java.net.InetAddress.getAllByName0(InetAddress.java:1248)
	at java.net.InetAddress.getAllByName(InetAddress.java:1164)
	at java.net.InetAddress.getAllByName(InetAddress.java:1098)
	at org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:61)
	at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:445)
	at org.apache.curator.utils.DefaultZookeeperFactory.newZooKeeper(DefaultZookeeperFactory.java:29)
	at org.apache.curator.framework.imps.CuratorFrameworkImpl$2.newZooKeeper(CuratorFrameworkImpl.java:154)
	at org.apache.curator.HandleHolder$1.getZooKeeper(HandleHolder.java:94)
	at org.apache.curator.HandleHolder.getZooKeeper(HandleHolder.java:55)
	at org.apache.curator.ConnectionState.reset(ConnectionState.java:219)
	at org.apache.curator.ConnectionState.start(ConnectionState.java:103)
	at org.apache.curator.CuratorZookeeperClient.start(CuratorZookeeperClient.java:190)
	at org.apache.curator.framework.imps.CuratorFrameworkImpl.start(CuratorFrameworkImpl.java:256)
	at com.paypal.fptistreaming.common.DruidEventRDDBeamFactory$.BeamInstance$lzycompute(DruidEventRDDBeamFactory.scala:29)
	at com.paypal.fptistreaming.common.DruidEventRDDBeamFactory$.BeamInstance(DruidEventRDDBeamFactory.scala:25)
	at com.paypal.fptistreaming.common.DruidEventRDDBeamFactory.makeBeam(DruidEventRDDBeamFactory.scala:17)
	at com.metamx.tranquility.spark.BeamFactory$class.tranquilizer(BeamFactory.scala:39)
	at com.paypal.fptistreaming.common.DruidEventRDDBeamFactory.tranquilizer$lzycompute(DruidEventRDDBeamFactory.scala:12)
	at com.paypal.fptistreaming.common.DruidEventRDDBeamFactory.tranquilizer(DruidEventRDDBeamFactory.scala:12)
	at com.metamx.tranquility.spark.BeamRDD$$anonfun$propagate$1.apply(BeamRDD.scala:36)
	at com.metamx.tranquility.spark.BeamRDD$$anonfun$propagate$1.apply(BeamRDD.scala:35)
	at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:902)

From the host that runs the spark task, I am able to ping and netcat the ZK url:port, so there is no firewall block here. 

I tried to fiddle around a bit, I get a similar error when I directly give the IP of my ZK cluster to the curator config:

java.net.UnknownHostException:  10.196.194.71: Temporary failure in name resolution
	at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
	at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:901)

Please note that the ZK host or the IP is not present in /etc/hosts file. Do we need it to be present in the hosts file for tranquility/curator towork? Can someone please give me some pointers here?

[Gian Merlino]

Hey Ram,

It looks like there might be an extra space at the start of your zk connect string, since there two spaces after "java.net.UnknownHostException:" but there should generally only be one. Could you double-check how you're providing that connect string?

Gian

--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/6c3533b7-0e91-44ce-a26d-ea73d8a0dcc4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ram.th...@gmail.com]

Thanks Gian for the help. Extra space in the comma-separated connect string was the issue :)

[ram.th...@gmail.com]

@Gian - Looking at this in zookeeper: https://github.com/apache/zookeeper/commit/d523f4e95069aafb1272efa45e788cc485bd3748

Looks like the issue was fixed 3 years back. Think we might have to bump the curator version in tranquility (it is on 2.6.0 now). I have submitted an issue to Tranquility.

Thanks,

Ram