org.hbase.async.HBaseClient - Failed to resolve the IP

593 views
Skip to first unread message

Mahesh Abnave

unread,
May 12, 2016, 9:39:44 AM5/12/16
to OpenTSDB
Hi 

I am trying out opentsdb Java API. TSD is running on the Ubuntu machine "01hw739015-ThinkCentre-M93p" with IP "10.17.68.109" which is not the machine on which I am running my code. (I am running code on Windows machine) But this machine is on the same LAN.

This is my small code:

import java.util.HashMap;
import java.util.Map;

import net.opentsdb.core.TSDB;
import net.opentsdb.core.WritableDataPoints;
import net.opentsdb.utils.Config;

public class OpenTSDBWriter {
public static void main(String[] args) throws Exception {
   final Config config = new Config("D:\\HadoopExp\\conf\\opentsdb.conf");
   final TSDB tsdb = new TSDB(config);
   Map<String, String> lTags = new HashMap<String, String>();
   lTags.put("room", "childroom");
   lTags.put("floor", "1");          
   WritableDataPoints lWritableDataPoints = tsdb.newDataPoints();
    lWritableDataPoints.setSeries("temperatures", lTags);
   lWritableDataPoints.addPoint(System.currentTimeMillis(), 22.5f);
   tsdb.flush();
            tsdb.shutdown();
}
}

I am getting following output with error message:

18:12:36.298 [main] INFO  net.opentsdb.utils.Config - Successfully loaded configuration file: D:\HadoopExp\conf\opentsdb.conf
18:12:36.308 [main] INFO  org.hbase.async.Config - Successfully loaded configuration file: D:\HadoopExp\conf\opentsdb.conf
18:12:36.462 [main] DEBUG net.opentsdb.core.TSDB - TSD Configuration:
File [D:\HadoopExp\conf\opentsdb.conf]
Key [tsd.storage.compaction.max_concurrent_flushes]  Value [10000]
Key [tsd.network.tcp_no_delay]  Value [true]
Key [tsd.storage.flush_interval]  Value [1000]
Key [tsd.core.preload_uid_cache.max_entries]  Value [300000]
Key [tsd.mode]  Value [rw]
Key [tsd.http.staticroot]  Value [/opt/opentsdb2/opentsdb-2.2.0/static]
Key [tsd.storage.hbase.zk_quorum]  Value [10.17.68.109]
Key [tsd.network.port]  Value [4242]
Key [tsd.storage.fix_duplicates]  Value [false]
Key [tsd.http.request.cors_domains]  Value []
Key [tsd.core.meta.enable_tsuid_tracking]  Value [false]
Key [tsd.http.request.cors_headers]  Value [Authorization, Content-Type, Accept, Origin, User-Agent, DNT, Cache-Control, X-Mx-ReqToken, Keep-Alive, X-Requested-With, If-Modified-Since]
Key [tsd.storage.hbase.meta_table]  Value [tsdb-meta]
Key [tsd.search.enable]  Value [false]
Key [tsd.core.tree.enable_processing]  Value [false]
Key [tsd.http.cachedir]  Value [/opt/opentsdb2/opentsdb-2.2.0/opentsdb_temp]
Key [tsd.http.show_stack_trace]  Value [true]
Key [tsd.http.request.max_chunk]  Value [4096]
Key [tsd.query.allow_simultaneous_duplicates]  Value [false]
Key [tsd.storage.compaction.flush_speed]  Value [2]
Key [tsd.core.uid.random_metrics]  Value [false]
Key [tsd.core.meta.enable_realtime_uid]  Value [false]
Key [tsd.core.preload_uid_cache]  Value [false]
Key [tsd.query.filter.expansion_limit]  Value [4096]
Key [tsd.storage.hbase.uid_table]  Value [tsdb-uid]
Key [tsd.storage.repair_appends]  Value [false]
Key [tsd.storage.hbase.zk_basedir]  Value [/hbase]
Key [tsd.storage.compaction.flush_interval]  Value [10]
Key [tsd.query.skip_unresolved_tagvs]  Value [false]
Key [tsd.no_diediedie]  Value [false]
Key [tsd.core.meta.enable_realtime_ts]  Value [false]
Key [tsd.http.query.allow_delete]  Value [false]
Key [tsd.network.bind]  Value [0.0.0.0]
Key [tsd.network.worker_threads]  Value []
Key [tsd.core.socket.timeout]  Value [0]
Key [tsd.core.meta.enable_tsuid_incrementing]  Value [false]
Key [tsd.core.auto_create_tagvs]  Value [true]
Key [tsd.network.async_io]  Value [true]
Key [tsd.rtpublisher.plugin]  Value []
Key [tsd.core.storage_exception_handler.enable]  Value [false]
Key [tsd.search.plugin]  Value []
Key [tsd.stats.canonical]  Value [false]
Key [tsd.storage.hbase.prefetch_meta]  Value [false]
Key [tsd.storage.enable_appends]  Value [false]
Key [tsd.core.auto_create_metrics]  Value [true ]
Key [tsd.storage.enable_compaction]  Value [true]
Key [tsd.core.auto_create_tagks]  Value [true]
Key [tsd.storage.compaction.min_flush_threshold]  Value [100]
Key [tsd.storage.hbase.data_table]  Value [tsdb]
Key [tsd.http.request.enable_chunked]  Value [false]
Key [tsd.core.plugin_path]  Value []
Key [tsd.query.timeout]  Value [0]
Key [tsd.rtpublisher.enable]  Value [false]
Key [tsd.storage.hbase.tree_table]  Value [tsdb-tree]
Key [tsd.network.reuse_address]  Value [true]
Key [tsd.network.keep_alive]  Value [true]
18:12:36.509 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
18:12:36.509 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:host.name=01HW738720.India.XYZ.com
18:12:36.509 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.7.0_02
18:12:36.509 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation
18:12:36.509 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.home=D:\sw\Java\jdk1.7.0_02\jre
18:12:36.509 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=<classpath_deleted>
18:12:36.509 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=C:\Users\user1\AppData\Local\Temp\
18:12:36.509 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
18:12:36.509 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.name=Windows 7
18:12:36.509 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.arch=x86
18:12:36.509 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.version=6.1
18:12:36.509 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:user.name=user1
18:12:36.509 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:user.home=C:\Users\user1
18:12:36.509 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:user.dir=D:\Development\ggts-3.3.0.M2\workspace_svn\HadoopExp
18:12:36.510 [main] INFO  org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=10.17.68.109 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@150cf54
18:12:36.513 [main] DEBUG org.apache.zookeeper.ClientCnxn - zookeeper.disableAutoWatchReset is false
18:12:36.594 [main] INFO  org.hbase.async.HBaseClient - Need to find the -ROOT- region
18:12:52.007 [main-SendThread(10.17.68.109:2181)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server 10.17.68.109/10.17.68.109:2181. Will not attempt to authenticate using SASL (unknown error)
18:12:52.008 [main-SendThread(10.17.68.109:2181)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established to 10.17.68.109/10.17.68.109:2181, initiating session
18:12:52.010 [main-SendThread(10.17.68.109:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Session establishment request sent on 10.17.68.109/10.17.68.109:2181
18:12:52.115 [main-SendThread(10.17.68.109:2181)] INFO  org.apache.zookeeper.ClientCnxn - Session establishment complete on server 10.17.68.109/10.17.68.109:2181, sessionid = 0x154a5be4e38000d, negotiated timeout = 5000
18:12:52.120 [main-EventThread] DEBUG org.hbase.async.HBaseClient - Got ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
18:12:52.121 [main-EventThread] DEBUG org.hbase.async.HBaseClient - Finding the -ROOT- or .META. region in ZooKeeper
18:12:52.145 [main-EventThread] DEBUG org.hbase.async.HBaseClient - Done handling ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
18:12:52.147 [main-SendThread(10.17.68.109:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x154a5be4e38000d, packet:: clientPath:/hbase/root-region-server serverPath:/hbase/root-region-server finished:false header:: 1,4  replyHeader:: 1,117,-101  request:: '/hbase/root-region-server,T  response::  
18:12:52.149 [main-SendThread(10.17.68.109:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x154a5be4e38000d, packet:: clientPath:/hbase/meta-region-server serverPath:/hbase/meta-region-server finished:false header:: 2,4  replyHeader:: 2,117,0  request:: '/hbase/meta-region-server,T  response:: #ffffffff0001a726567696f6e7365727665723a34313337337d5effffff8e2a16154a3050425546a28a1b303168773733393031352d5468696e6b43656e7472652d4d39337010ffffff9dffffffc3218ffffff94ffffffb9fffffff9ffffffadffffffca2a100183,s{49,49,1463069628916,1463069628916,0,0,0,0,81,0,49} 
18:12:52.545 [main-EventThread] ERROR org.hbase.async.HBaseClient - Failed to resolve the IP of `01hw739015-ThinkCentre-M93p' in 307048554ns
18:12:52.545 [main-EventThread] INFO  org.hbase.async.HBaseClient - Connecting to .META. region @ null:41373
18:12:52.625 [main-EventThread] DEBUG org.hbase.async.HBaseClient - Channel [id: 0xd3d69735]'s state changed: [id: 0xd3d69735] OPEN
18:12:52.628 [main-EventThread] DEBUG org.hbase.async.RegionClient - handleUpstream [id: 0xd3d69735] OPEN
18:12:52.660 [main-EventThread] ERROR org.apache.zookeeper.ClientCnxn - Caught unexpected throwable
java.lang.IllegalArgumentException: hostname can't be null
at java.net.InetSocketAddress.<init>(InetSocketAddress.java:139) ~[na:1.7.0_02]
at org.hbase.async.HBaseClient.newClient(HBaseClient.java:3042) ~[asynchbase-1.7.1.jar:na]
at org.hbase.async.HBaseClient.access$3200(HBaseClient.java:190) ~[asynchbase-1.7.1.jar:na]
at org.hbase.async.HBaseClient$ZKClient$ZKCallback.handleMetaZnode(HBaseClient.java:3765) ~[asynchbase-1.7.1.jar:na]
at org.hbase.async.HBaseClient$ZKClient$ZKCallback.processResult(HBaseClient.java:3621) ~[asynchbase-1.7.1.jar:na]
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:558) [zookeeper-3.4.5.jar:3.4.5-1392090]
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495) [zookeeper-3.4.5.jar:3.4.5-1392090]
18:12:53.813 [main-SendThread(10.17.68.109:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x154a5be4e38000d after 0ms
18:12:55.480 [main-SendThread(10.17.68.109:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x154a5be4e38000d after 0ms
18:12:57.146 [main-SendThread(10.17.68.109:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x154a5be4e38000d after 0ms
18:12:58.812 [main-SendThread(10.17.68.109:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x154a5be4e38000d after 0ms
18:13:00.478 [main-SendThread(10.17.68.109:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x154a5be4e38000d after 0ms

Above it says that it is unable to resolve IP of "01hw739015-ThinkCentre-M93p". However I did not specified this name "01hw739015-ThinkCentre-M93p" anywhere in conf file. So from where it is obtaining this name in order to resolve its IP? In fact I have specified only the IP of this machine in the opentsdb.conf file (tsd.storage.hbase.zk_quorum = 10.17.68.109). I dont get why it is still trying to resolve the IP of this machine?  On another Windows machine on which I have admin privileges I modified etc/hosts file to add this name and the corresponding IP. On that machine the code works just fine after adding the hosts entry (without hosts entry it gives same error). But on my current machine I don't have admin privileges so I cannot modify hosts file. So whats going on behind the scene. 

I have attached opentsdb.conf file.
opentsdb.conf
Message has been deleted

Mahesh Abnave

unread,
May 12, 2016, 3:39:34 PM5/12/16
to OpenTSDB
I guess I am missing some basic understanding here. I am new to opentsdb. So sorry for my possible stupidity.

Jonathan Creasy

unread,
May 12, 2016, 4:41:12 PM5/12/16
to Mahesh Abnave, OpenTSDB
It sounds like the TSD is trying to connect to HBase and somewhere it is providing its hostname, and HBase is trying to resolve that hostname back to an IP, possibly just to log it.

This doesn't sound like an OpenTSDB problem, but maybe a configuration issue with your machine?

Does your etc/hosts file on the "01hw....M93p" machine have it's own hostname defined and an IP? (either the IP of the machine or localhost IP)

On Thu, May 12, 2016 at 2:38 PM, Mahesh Abnave <abna...@gmail.com> wrote:
I guess I am missing some basic understanding here. Am new to opentsdb. So sorry for my possible stupidity.

Mahesh Abnave

unread,
May 13, 2016, 2:21:52 AM5/13/16
to Jonathan Creasy, OpenTSDB
Yes we have following entry in hosts file on machine 01hw739015-ThinkCentre-M93p which runs hadoop, hbase, tsdb:

10.17.68.109  01hw739015-ThinkCentre-M93p localhost

Is it correct? Should it be there or not?

Nicholas Whitehead

unread,
May 13, 2016, 12:42:59 PM5/13/16
to Mahesh Abnave, OpenTSDB, Jonathan Creasy

Hey Mahesh;

My guess is that HBase is binding the hmaster into zookeper using the host name, not the IP address. So when your client retrieves it, it attempts to connect back using that hostname.

There should be an HBase configuration option to use the IP address instead of the hostname.

Mahesh Abnave

unread,
May 17, 2016, 3:14:05 AM5/17/16
to Nicholas Whitehead, OpenTSDB, Jonathan Creasy

Thanks. I had a look at my current configs... I have never specified the hostname anywhere explicitly. Any more pointers?

Jonathan Creasy

unread,
May 17, 2016, 12:45:47 PM5/17/16
to Mahesh Abnave, Nicholas Whitehead, OpenTSDB
It's not in your configuration, Nicholas is probably right. Your Hbase setup is connecting to ZooKeeper and registering with it's hostname. When OpenTSDB connects to Zookeeper and says "Hey, where is that HBase anyway?" Zookeeper's data is providing the hostname so TSDB is trying to connect to it.

I *thought* HBase registered it's IP address, but that may not be true. You can query what Zookeeper has. 

bin/zkCli.sh -server 127.0.0.1:2181
get /hbase/master

It looks like being able to register the IP and not the hostname is not available in HBase yet: https://issues.apache.org/jira/browse/HBASE-11768.

The only way I can see to solve this is to set your hostname on your HBase servers to something that is resolvable, and/or make that resolvable by editing the hosts files on your client machines (the windows machine where the code runs).

Perhaps this guide will provide some useful information: http://sujee.net/2012/03/08/getting-dns-right-for-hadoop-hbase-clusters/

-Jonathan

Reply all
Reply to author
Forward
0 new messages