Constant HBase/Zookeeper reconnections?

430 views
Skip to first unread message

Adam Phelps

unread,
Jan 4, 2016, 5:40:37 PM1/4/16
to Aurelius
We're using Titan 0.5.4 on top of HBase (CDH 5.1) and connect to the
graph via Java. We've noticed that even when our application is idle it
appears to be frequently (roughly every 5 seconds) disconnecting and
reconnecting to Zookeeper. We haven't seen this sort of behavior in our
other Java applications which access non-Titan HBase tables.

Is this connection cycling normal for Titan, or is there some
configuration setting that will allow it to maintain a longer connection?

- Adam

2016-01-04 22:33:23.027258500 [pool-4-thread-1] INFO
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - Process
identifier=catalogtracker-on-hconnection-0xe4d2696 connecting to
ZooKeeper ensemble=AA.BB.CC.DD:2181
2016-01-04 22:33:23.027262500 [pool-4-thread-1] INFO
org.apache.zookeeper.ZooKeeper - Initiating client connection,
connectString=AA.BB.CC.DD:2181 sessionTimeout=90000
watcher=catalogtracker-on-hconnection-0xe4d2696,
quorum=AA.BB.CC.DD:2181, baseZNode=/hbase
2016-01-04 22:33:23.031587500
[pool-4-thread-1-SendThread(AA.BB.CC.DD:2181)] INFO
org.apache.zookeeper.ClientCnxn - Opening socket connection to server
AA.BB.CC.DD/AA.BB.CC.DD:2181. Will not attempt to authenticate using
SASL (unknown error)
2016-01-04 22:33:23.032295500
[pool-4-thread-1-SendThread(AA.BB.CC.DD:2181)] INFO
org.apache.zookeeper.ClientCnxn - Socket connection established to
AA.BB.CC.DD/AA.BB.CC.DD:2181, initiating session
2016-01-04 22:33:23.035536500
[pool-4-thread-1-SendThread(AA.BB.CC.DD:2181)] INFO
org.apache.zookeeper.ClientCnxn - Session establishment complete on
server AA.BB.CC.DD/AA.BB.CC.DD:2181, sessionid = 0x14f4788b0142ab3,
negotiated timeout = 60000
2016-01-04 22:33:23.043678500 [pool-4-thread-1] INFO
org.apache.zookeeper.ZooKeeper - Session: 0x14f4788b0142ab3 closed
2016-01-04 22:33:23.043742500 [pool-4-thread-1-EventThread] INFO
org.apache.zookeeper.ClientCnxn - EventThread shut down
2016-01-04 22:33:28.064266500 [pool-4-thread-1] INFO
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - Process
identifier=catalogtracker-on-hconnection-0xe4d2696 connecting to
ZooKeeper ensemble=AA.BB.CC.DD:2181
2016-01-04 22:33:28.064269500 [pool-4-thread-1] INFO
org.apache.zookeeper.ZooKeeper - Initiating client connection,
connectString=AA.BB.CC.DD:2181 sessionTimeout=90000
watcher=catalogtracker-on-hconnection-0xe4d2696,
quorum=AA.BB.CC.DD:2181, baseZNode=/hbase
2016-01-04 22:33:28.064971500
[pool-4-thread-1-SendThread(AA.BB.CC.DD:2181)] INFO
org.apache.zookeeper.ClientCnxn - Opening socket connection to server
AA.BB.CC.DD/AA.BB.CC.DD:2181. Will not attempt to authenticate using
SASL (unknown error)
2016-01-04 22:33:28.065737500
[pool-4-thread-1-SendThread(AA.BB.CC.DD:2181)] INFO
org.apache.zookeeper.ClientCnxn - Socket connection established to
AA.BB.CC.DD/AA.BB.CC.DD:2181, initiating session
2016-01-04 22:33:28.067627500
[pool-4-thread-1-SendThread(AA.BB.CC.DD:2181)] INFO
org.apache.zookeeper.ClientCnxn - Session establishment complete on
server AA.BB.CC.DD/AA.BB.CC.DD:2181, sessionid = 0x14f4788b0142ae5,
negotiated timeout = 60000
2016-01-04 22:33:28.074976500 [pool-4-thread-1] INFO
org.apache.zookeeper.ZooKeeper - Session: 0x14f4788b0142ae5 closed
2016-01-04 22:33:28.075288500 [pool-4-thread-1-EventThread] INFO
org.apache.zookeeper.ClientCnxn - EventThread shut down
^X2016-01-04 22:33:33.090543500 [pool-4-thread-1] INFO
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - Process
identifier=catalogtracker-on-hconnection-0xe4d2696 connecting to
ZooKeeper ensemble=AA.BB.CC.DD:2181
2016-01-04 22:33:33.090546500 [pool-4-thread-1] INFO
org.apache.zookeeper.ZooKeeper - Initiating client connection,
connectString=AA.BB.CC.DD:2181 sessionTimeout=90000
watcher=catalogtracker-on-hconnection-0xe4d2696,
quorum=AA.BB.CC.DD:2181, baseZNode=/hbase
2016-01-04 22:33:33.095093500
[pool-4-thread-1-SendThread(AA.BB.CC.DD:2181)] INFO
org.apache.zookeeper.ClientCnxn - Opening socket connection to server
AA.BB.CC.DD/AA.BB.CC.DD:2181. Will not attempt to authenticate using
SASL (unknown error)
2016-01-04 22:33:33.098406500
[pool-4-thread-1-SendThread(AA.BB.CC.DD:2181)] INFO
org.apache.zookeeper.ClientCnxn - Socket connection established to
AA.BB.CC.DD/AA.BB.CC.DD:2181, initiating session
2016-01-04 22:33:33.100650500
[pool-4-thread-1-SendThread(AA.BB.CC.DD:2181)] INFO
org.apache.zookeeper.ClientCnxn - Session establishment complete on
server AA.BB.CC.DD/AA.BB.CC.DD:2181, sessionid = 0x14f4788b0142b16,
negotiated timeout = 60000
2016-01-04 22:33:33.110387500 [pool-4-thread-1] INFO
org.apache.zookeeper.ZooKeeper - Session: 0x14f4788b0142b16 closed
2016-01-04 22:33:33.110572500 [pool-4-thread-1-EventThread] INFO
org.apache.zookeeper.ClientCnxn - EventThread shut down

Jianmin Liu

unread,
Jan 6, 2016, 10:36:29 AM1/6/16
to Aurelius
I have exactly the same issue for Titan-1.0.0-hadoop1 with HBase. After the Gremlin server is started, the console constantly displays those messages.

Arielle Bonnici

unread,
Jan 13, 2016, 8:36:49 AM1/13/16
to Aurelius
Same here...I'd be interested to know if it is 'normal' or if there's a way to address this.


Regards,

Arielle

Stephen Mallette

unread,
Jan 13, 2016, 9:37:57 AM1/13/16
to Aurelius
I'm not sure why hbase is doing that, but it doesn't seem right given that those requests are just milliseconds apart from each other.  Hbase is a mystery to me - David/Jason, you guys seem to know the hbase stuff reasonably well, do you ever see this in your usage?

--
You received this message because you are subscribed to the Google Groups "Aurelius" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aureliusgraph...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/aureliusgraphs/47ab95eb-76cc-4d8d-80df-befb15e21b5c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Jason Plurad

unread,
Jan 13, 2016, 10:02:12 AM1/13/16
to Aurelius
Titan-HBase has always done this. I think the initial start up of Titan-HBase is amazingly slow when compared to Titan-Cassandra because of this. I haven't been curious enough yet to dig into the code to see why it is doing that.

I usually just adjust the noisy logging with the appropriate log4j.properties file.

-- Jason

David

unread,
Jan 13, 2016, 4:49:40 PM1/13/16
to Aurelius
Perhaps it is worth opening an issue as a marker in case we should investigate this.

I will go ahead and do that, and refer back to this thread.  Then maybe we can
find time to take a deeper look if someone has not already figured it out by the
time we get there.

David

unread,
Jan 13, 2016, 4:53:19 PM1/13/16
to Aurelius

Sean MacDonald

unread,
Jan 13, 2016, 8:01:14 PM1/13/16
to Aurelius
I think this is related to the ensureTableExists function in the HBaseStoreManager class. It creates a new HBaseAdmin object which internally uses the CatalogTracker and establishes a new connection to ZooKeeper every time.

From HBase 0.98.16:

tableExists:


getCatalogTracker:


And the connection in CatalogTracker:

da zhang

unread,
Jun 14, 2016, 5:35:58 PM6/14/16
to Aurelius
I get the same issue too. I have tried multiple HBase and Titan version but this issue still remains. Is this an error? Is there any way to solve this?


On Monday, January 4, 2016 at 5:40:37 PM UTC-5, Adam Phelps wrote:
Reply all
Reply to author
Forward
0 new messages