Hbase Connection With Kerberos Authentication

Uğur Coşkun

unread,

Sep 7, 2018, 8:04:36 AM9/7/18

to JanusGraph users

Hey Everyone!

We are trying to connect HBase with Kerberos Auth. We can connect HBase and can get tables etc. with Kerberos Auth.

But we cant use JanusGraphFactory.build(). We get this error:

2862 [main-SendThread(172.16.250.10:24002)] INFO org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ClientCnxn - Socket connection established to 172.16.250.10/172.16.250.10:24002, initiating session
2867 [main-SendThread(172.16.250.10:24002)] INFO org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ClientCnxn - Session establishment complete on server 172.16.250.10/172.16.250.10:24002, sessionid = 0x1200000a81d89316, negotiated timeout = 90000
2880 [main-SendThread(172.16.250.10:24002)] ERROR org.apache.hadoop.hbase.shaded.org.apache.zookeeper.client.ZooKeeperSaslClient - An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when evaluating Zookeeper Quorum Member's received SASL token. Zookeeper Client will go to AUTH_FAILED state.
2880 [main-SendThread(172.16.250.10:24002)] ERROR org.apache.hadoop.hbase.shaded.org.apache.zookeeper.ClientCnxn - SASL authentication with Zookeeper Quorum member failed: javax.security.sasl.SaslException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when evaluating Zookeeper Quorum Member's received SASL token. Zookeeper Client will go to AUTH_FAILED state.

JanusGraphFactory.build()
  .set("storage.backend", "hbase")
  .set("gremlin.graph", "org.janusgraph.core.JanusGraphFactory")
  .set("storage.hostname", "172.16.250.10")
  .set("storage.hbase.ext.hbase.zookeeper.property.clientPort", "24002")
  .set("storage.hbase.table", "hbase_sample_table")
  .set("index.search.index-name","hbase_sample_table")
  .set("index.search.hostname", "172.16.250.10" + ":" + "21100")
  .set("index.search.backend", "solr")
  .set("index.search.solr.mode", "http")
  .set("index.search.solr.http-urls", "http://" + "172.16.250.10" + ":" + "21100" + "/solr")
  .set("istorage.hbase.ext.zookeeper.znode.parent", "/hbase-secure")
  .open();

  Thank you

HadoopMarc

unread,

Sep 8, 2018, 7:30:42 AM9/8/18

to JanusGraph users

Hi Uğur,

Two checks:

.set("istorage.hbase.ext.zookeeper.znode.parent", "/hbase-secure") is the istorage typo also on your system?
is the hbase clients's conf directory (with the hbase-client.jaas file among others) on the JanusGraph classpath?

Cheers, Marc

Op vrijdag 7 september 2018 14:04:36 UTC+2 schreef Uğur Coşkun:

jme...@gmail.com

unread,

Sep 9, 2018, 9:56:59 PM9/9/18

to JanusGraph users

Hi Uğur,

In my case, It's because of the hbase zookeeper String,

I replaced the zookeeper IP with hostname ,then error disappeared.

because what I stored in my kerberos database is zookeeper/nod...@COMPLANY.COM,

when use zookeeper with IP, the principle is zookeeper/127.1...@COMPLANY.COM,

thus result in the error.

Uğur Coşkun

unread,

Sep 10, 2018, 4:54:47 AM9/10/18

to JanusGraph users

Hi Marc and Mernio,

Thank you for response

I replaced storage.hostname as Mernio's response and it worked for me. Then i get a warning " Zookeeper is null

After that according to Marc

1) I checked istorage typo and changed .set("istorage.hbase.ext.zookeeper.znode.parent", "/hbase-secure") to .set("istorage.hbase.ext.zookeeper.znode.parent", "/hbase")

Then i get this error ::

41879 [hconnection-0x76c3e77a-shared--pool1-t1] INFO org.apache.hadoop.hbase.client.RpcRetryingCaller - Call exception, tries=10, retries=35, started=39101 ms ago, cancelled=false, msg=row 'test2,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=fusioninsight1,21302,1533903378708, seqNum=0

52014 [hconnection-0x76c3e77a-shared--pool1-t1] INFO org.apache.hadoop.hbase.client.RpcRetryingCaller - Call exception, tries=11, retries=35, started=49238 ms ago, cancelled=false, msg=row 'test2,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=fusioninsight1,21302,1533903378708, seqNum=0

52017 [main] WARN org.janusgraph.diskstorage.hbase.HBaseStoreManager - Unexpected exception during getDeployment()

java.lang.RuntimeException: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend

at org.janusgraph.diskstorage.hbase.HBaseStoreManager.getDeployment(HBaseStoreManager.java:377)

at org.janusgraph.diskstorage.hbase.HBaseStoreManager.getFeatures(HBaseStoreManager.java:416)

at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.<init>(GraphDatabaseConfiguration.java:1256)

at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:160)

at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:131)

at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:121)

at org.janusgraph.core.JanusGraphFactory$Builder.open(JanusGraphFactory.java:260)

at com.xxx.kg.examples.Main.main(Main.java:80)

Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend

at org.janusgraph.diskstorage.hbase.HBaseStoreManager.ensureTableExists(HBaseStoreManager.java:746)

at org.janusgraph.diskstorage.hbase.HBaseStoreManager.getLocalKeyPartition(HBaseStoreManager.java:535)

at org.janusgraph.diskstorage.hbase.HBaseStoreManager.getDeployment(HBaseStoreManager.java:374)

... 7 more

Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=36, exceptions:

Mon Sep 10 11:38:54 EET 2018, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=69418: row 'test2,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=fusioninsight1,21302,1533903378708, seqNum=0

at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:276)

at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:210)

at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)

at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)

at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327)

at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:302)

at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:167)

at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:162)

at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:797)

at org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:602)

at org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:366)

at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:406)

at org.janusgraph.diskstorage.hbase.HBaseAdmin1_0.tableExists(HBaseAdmin1_0.java:111)

at org.janusgraph.diskstorage.hbase.HBaseStoreManager.ensureTableExists(HBaseStoreManager.java:723)

... 9 more

Caused by: java.net.SocketTimeoutException: callTimeout=60000, callDuration=69418: row 'test2,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=fusioninsight1,21302,1533903378708, seqNum=0

at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:169)

at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Thank you

HadoopMarc

unread,

Sep 10, 2018, 9:10:51 AM9/10/18

to JanusGraph users

Hi Uğur,

My (too) implicit suggestion actually was (removing the "i" in "istorage"):

.set("storage.hbase.ext.zookeeper.znode.parent", "/hbase-secure")

Anyway, it should be the same as the zookeeper.znode.parent property in the hbase-site.xml of your cluster.

Cheers, Marc

Op maandag 10 september 2018 10:54:47 UTC+2 schreef Uğur Coşkun:

Uğur Coşkun

unread,

Sep 10, 2018, 9:28:16 AM9/10/18

to JanusGraph users

Hi Marc,

I fixed it but still get an exception like below.

41642 [hconnection-0x67c33749-shared--pool1-t1] INFO org.apache.hadoop.hbase.client.RpcRetryingCaller - Call exception, tries=10, retries=35, started=39008 ms ago, cancelled=false, msg=row 'test6,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=fusioninsight1,21302,1536580274600, seqNum=0

51786 [hconnection-0x67c33749-shared--pool1-t1] INFO org.apache.hadoop.hbase.client.RpcRetryingCaller - Call exception, tries=11, retries=35, started=49153 ms ago, cancelled=false, msg=row 'test6,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=fusioninsight1,21302,1536580274600, seqNum=0

51789 [main] WARN org.janusgraph.diskstorage.hbase.HBaseStoreManager - Unexpected exception during getDeployment()

java.lang.RuntimeException: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend

at org.janusgraph.diskstorage.hbase.HBaseStoreManager.getDeployment(HBaseStoreManager.java:377)

at org.janusgraph.diskstorage.hbase.HBaseStoreManager.getFeatures(HBaseStoreManager.java:416)

at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.<init>(GraphDatabaseConfiguration.java:1256)

at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:160)

at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:131)

at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:121)

at org.janusgraph.core.JanusGraphFactory$Builder.open(JanusGraphFactory.java:260)

at com.huawei.kg.examples.Main.main(Main.java:84)

Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend

at org.janusgraph.diskstorage.hbase.HBaseStoreManager.ensureTableExists(HBaseStoreManager.java:746)

at org.janusgraph.diskstorage.hbase.HBaseStoreManager.getLocalKeyPartition(HBaseStoreManager.java:535)

at org.janusgraph.diskstorage.hbase.HBaseStoreManager.getDeployment(HBaseStoreManager.java:374)

... 7 more

Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=36, exceptions:

Mon Sep 10 16:24:24 EET 2018, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=69283: row 'test6,,' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=fusioninsight1,21302,1536580274600, seqNum=0