Use Titan with HBase (deployed with AWS EMR on EC2 hosts)

[SShah]

Hey,

Anybody has tried using Titan with the AWS EMR provided HBase installation on EC2 hosts. I am running into the following error (pasted below). I have enabled "ALL TCP" connections to my master node from my IP address and i also verified that my ZooKeeper is running (zk dump pasted below).

I am running the simple TitanGraph connect from my Java application (external to EC2 host) but i keep getting this error. I also tried to telnet from my host to the master node , zookeper port and i got a connection refused.

telnet ec2-54-80-231-233.compute-1.amazonaws.com 2181
Trying 54.80.231.233...
telnet: connect to address 54.80.231.233: Connection refused

== Code ==

Configuration conf = new BaseConfiguration();
conf.setProperty("storage.backend","hbase");
conf.setProperty("storage.hostname","<ec2 master node public dns name>");
conf.setProperty("storage.port", "2181");
TitanGraph g = TitanFactory.open(conf);

==== Zookeeper Dump from master's web interface ====
HBase is rooted at /hbase
Active master address: ip-XX.ec2.internal,60000,1409166344962
Backup master addresses:
Region server holding ROOT: ip-XX.ec2.internal,60020,1409166332654
Region servers:
 ip-XX.ec2.internal,60020,1409166344157
 ip-XX.ec2.internal,60020,1409166332654
Quorum Server Statistics:
 ip-XX.ec2.internal:2181
  Zookeeper version: 3.4.5-1392090, built on 09/30/2012 17:52 GMT
  Clients:
   /10.XX:52132[1](queued=0,recved=111,sent=120)
   /10.XX:41282[1](queued=0,recved=184,sent=203)
   /10.XX:41281[1](queued=0,recved=79,sent=79)
   /10.XX:41283[1](queued=0,recved=161,sent=162)
   /10.XX:41288[1](queued=0,recved=212,sent=214)
   /10.XX:41280[1](queued=0,recved=85,sent=88)
   /10.XX:41287[1](queued=0,recved=369,sent=386)
   /10.XX:55254[1](queued=0,recved=406,sent=424)
   /10.XX:52135[1](queued=0,recved=388,sent=405)
   /10.XX:41426[1](queued=0,recved=186,sent=191)
   /10.XX:41791[0](queued=0,recved=1,sent=0)
   /10.XX:41422[1](queued=0,recved=62,sent=62)
   /10.XX:52138[1](queued=0,recved=260,sent=263)
   /10.XX:52137[1](queued=0,recved=88,sent=88)
   .........
 
  Latency min/avg/max: 0/0/1126
  Received: 50363
  Sent: 52373
  Connections: 20
  Outstanding: 0
  Zxid: 0xe2d4
  Mode: standalone
  Node count: 35


== ERROR ==
992966 [main] ERROR org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper  - ZooKeeper exists failed after 3 retries
992966 [main] ERROR org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher  - hconnection Received unexpected KeeperException, re-throwing exception
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
    at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:172)
    at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:450)
    at org.apache.hadoop.hbase.zookeeper.ClusterId.readClusterIdZNode(ClusterId.java:61)
    at org.apache.hadoop.hbase.zookeeper.ClusterId.getId(ClusterId.java:50)
    at org.apache.hadoop.hbase.zookeeper.ClusterId.hasId(ClusterId.java:44)
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.ensureZookeeperTrackers(HConnectionManager.java:720)
    at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:789)
    at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:127)
    at com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.getAdminInterface(HBaseStoreManager.java:364)
    at com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.ensureColumnFamilyExists(HBaseStoreManager.java:261)
    at com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.openDatabase(HBaseStoreManager.java:215)
    at com.thinkaurelius.titan.diskstorage.Backend.getStore(Backend.java:231)
    at com.thinkaurelius.titan.diskstorage.Backend.initialize(Backend.java:243)
    at com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration.getBackend(GraphDatabaseConfiguration.java:1164)
    at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.<init>(StandardTitanGraph.java:75)
    at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:40)
    at amazon.actiontrace.graph.impl.TitanGraphFactory.create(TitanGraphFactory.java:27)
    at amazon.actiontrace.graph.impl.TitanGraphImpl.main(TitanGraphImpl.java:9)

[Dan LaRocque]

On 08/27/2014 04:30 PM, SShah wrote:
> Hey,
>
> Anybody has tried using Titan with the AWS EMR provided HBase
> installation on EC2 hosts. I am running into the following error
> (pasted below). I have enabled "ALL TCP" connections to my master node
> from my IP address and i also verified that my ZooKeeper is running
> (zk dump pasted below).
>
> I am running the simple TitanGraph connect from my Java application
> (external to EC2 host) but i keep getting this error. I also tried to
> telnet from my host to the master node , zookeper port and i got a
> connection refused.
>
> telnet ec2-54-80-231-233.compute-1.amazonaws.com 2181
> Trying 54.80.231.233...
> telnet: connect to address 54.80.231.233: Connection refused
>
> == Code ==
>
> Configuration conf = new BaseConfiguration();
> conf.setProperty("storage.backend","hbase");
> conf.setProperty("storage.hostname","<ec2 master node public dns name>");
> conf.setProperty("storage.port", "2181");
> TitanGraph g = TitanFactory.open(conf);

Hi SShah,

I'm not sure exactly what's going on here. Are you telnetting to and
getting connection refused from the public IP of the current ZK master?
If so, then why is storage.hostname set to presumably the same thing
(can't tell since the actual IP/host was edited)? I would try setting a
comma-separated quorum of internal IPs on storage.hostname if you
haven't already.

It would also be nice to know what version of Titan you're using.

thanks,
Dan