Duke sourcing Hbase Question

[Brandon Fletcher]

Larga, First of all, I must echo everyone else - this is an excellent product and thanks for your time investment!  We really appreciate all that you do - truly amazing.  I look forward to adding my company name to your list!

 

For the last month, I have been playing around with Duke and have been able to source CSV and SQL server tables for linking via the xml configuration, but have been unsuccessful with Hbase via the config. I cannot figure out why the xml parameters are not being used; I can read in the Hbase table from my main java program.  Can you use a JDBC connection for Hbase or would i have to create a new datasource connection? Hopefully the below information is helpful(main java code, log message , xml).  Could you provide any help or suggestions?    Please let me know if I can help further. 

How I connected in main java program:

            Configuration config = HBaseConfiguration.create();

            config.clear(); 

            

            String zookeeperQuorum = "xxx.xxx.xx.xxxx";

            String zookeeperPropertyClientPort = "xxxx";

            String master = "xxx.xxx.xx.xxxx";

            String masterPort = "xxxx";

            String tableName = "member";

            

            

            config.setInt("timeout", 12000);

            config.set("hbase.master", "*" + master + ":60000*");  

            config.set("hbase.zookeeper.quorum",zookeeperQuorum);

            config.set("hbase.zookeeper.property.clientPort",zookeeperPropertyClientPort);

            

            HBaseAdmin.checkHBaseAvailable(config);

            HTable table = new HTable(config, tableName);

XML: 

  <group>

 <jdbc>

   <param name="driver-class" value="org.apache.hadoop.hbase.jdbc.Driver"/>

   <param name="connection-string" value="jdbc:hbql;maxtablerefs=10;hbase.master=xxx.xxx.xx.xxxx:xxxx"/>

<param name="query" value="scan 'member'"/>  

   <column name="id"

property="id"/>

Log from Java main (successful log): 

Opening socket connection to server phpxxxxx.xxxx.local/xxx.xxx.xx.xxxx:xxx

zookeeper.ClientCnxn: Opening socket connection to server phpxxxxx.xxxx.local/xxx.xxx.xx.xxxx:xxx. Will not attempt to authenticate using SASL (unknown error)

zookeeper.ClientCnxn: Socket connection established to phpxxxxx.xxxx.local/xxx.xxx.xx.xxxx:xxx, initiating session

zookeeper.ClientCnxn: Session establishment complete

Log from xml config (unsuccessful log): 

zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x9f3d43e connecting to ZooKeeper ensemble=localhost:2181

zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0x9f3d43e, quorum=localhost:2181, baseZNode=/hbase

zookeeper.ClientCnxn: Opening socket connection to server 0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)

zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect

java.net.ConnectException: Connection refused: no further information

at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)

at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)

at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)

15/08/06 18:20:42 INFO zookeeper.ClientCnxn: Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)

15/08/06 18:20:42 WARN zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=localhost:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/hbaseid

[Lars Marius Garshol]

Hi Brandon,

* Brandon Fletcher

Larga, First of all, I must echo everyone else - this is an excellent product and thanks for your time investment!  We really appreciate all that you do - truly amazing.  I look forward to adding my company name to your list!

Thank you! If you ever start using Duke for real, please do let me know and you'll definitely go on the list.

 

For the last month, I have been playing around with Duke and have been able to source CSV and SQL server tables for linking via the xml configuration, but have been unsuccessful with Hbase via the config. I cannot figure out why the xml parameters are not being used; I can read in the Hbase table from my main java program.

As far as I can tell, the problem is here: "java.net.ConnectException: Connection refused"

So your HBase and Duke configurations don't match, because the HBase JDBC driver can't connect to the server. Clearly that's never going to work. You've  blanked out the details in your email, so I can't say what's wrong here.

 

Can you use a JDBC connection for Hbase or would i have to create a new datasource connection?

I honestly don't know. If JDBC for HBase works the same way as for SQL databases then there's no real reason why it shouldn't work.

 

Best,

--Lars Marius

[Monte Cillo Co]

Hi,

I guess you may use https://phoenix.apache.org/ :D