Gremlin Server connection to Kerberised HBase

[Sarthak Ghosh]

Hi,

I have a Kerberised HDP 3.1.4 running on an environment. I am trying to connect to it using gremlin-server 3.4.1 which is running on a different environment (inside kubernetes).

This connection is not happening. I keep running into the following error: 

```

Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

    at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)

    at org.apache.hadoop.hbase.security.AbstractHBaseSaslRpcClient.getInitialResponse(AbstractHBaseSaslRpcClient.java:131)

    at org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler$1.run(NettyHBaseSaslRpcClientHandler.java:109)

    at org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler$1.run(NettyHBaseSaslRpcClientHandler.java:105)

    at java.security.AccessController.doPrivileged(Native Method)

    at javax.security.auth.Subject.doAs(Subject.java:422)

    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)

    at org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler.handlerAdded(NettyHBaseSaslRpcClientHandler.java:105)

    at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:606)

    at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:187)

    at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:380)

    at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:359)

    at org.apache.hadoop.hbase.ipc.NettyRpcConnection.saslNegotiate(NettyRpcConnection.java:200)

    ... 18 more

Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)

```

In Java options, I have provided the following configs:

JAVA_OPTIONS="-Xms512m -Xmx4096m -Djava.security.auth.login.config=/home/conf/jaas.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false -Dsun.security.krb5.debug=true"

Following is my jaas.conf file 

```

Client {

  com.sun.security.auth.module.Krb5LoginModule required

  useKeyTab=true

  keyTab="/home/conf/fcigraph.keytab"

  storeKey=true

  useTicketCache=false

  debug=true

  principal="gr...@FCI.SARTHAK";

};

```

HDP env's hostname and IP is present in /etc/hosts file for connectivity

Following is my janusgraph-hbase.properties file which is mentioned in gremlin-server.yaml file

```

storage.backend=hbasegremlin.graph=org.janusgraph.core.JanusGraphFactory

storage.hostname=hostname1,hostname2

storage.hbase.ext.zookeeper.znode.parent=/hbase-secure

storage.hbase.ext.hbase.zookeeper.property.clientPort=2181

cache.db-cache=true

storage.hbase.table=fci-graph

cache.db-cache-size=0.5

cache.db-cache-time=180000

cache.db-cache-clean-wait=20

index.search.backend=elasticsearch

index.search.hostname=fci-elasticsearch

index.search.port=9200

index.search.elasticsearch.ssl.enabled=true

index.search.elasticsearch.ssl.keystore.location=/fciexports/graph-writer/fci_universal_ks.jks

index.search.elasticsearch.ssl.keystore.keypassword=password

index.search.elasticsearch.ssl.keystore.storepassword=password

index.search.elasticsearch.ssl.allow-self-signed-certificates=true

index.search.elasticsearch.ssl.disable-hostname-verification=true

index.search.elasticsearch.http.auth.type=basic

index.search.elasticsearch.http.auth.basic.username=username

index.search.elasticsearch.http.auth.basic.password=password

java.security.krb5.conf=/etc/krb5.conf
```

JAVA_VERSION

```

openjdk version "1.8.0_242"

OpenJDK Runtime Environment (build 1.8.0_242-8u242-b08-0ubuntu3~18.04-b08)

OpenJDK 64-Bit Server VM (build 25.242-b08, mixed mode)
```

Let me know if you need any more info/logs. Thanks!

[Sarthak Ghosh]

I updated my janusgraph-hbase.properties and added few more properties

```

java.security.auth.login.config=/home/ibmadmin/conf/jaas.conf

storage.hbase.ext.hbase.security.authentication=kerberos

storage.hbase.ext.hbase.security.authorization=true

storage.hbase.ext.hadoop.security.authentication=kerberos

storage.hbase.ext.hadoop.security.authorization=true

storage.hbase.ext.hbase.rpc.protection=authentication

storage.hbase.ext.hbase.master.kerberos.principal=hbase/_HOST@FCI.SARTHAK

storage.hbase.ext.hbase.regionserver.kerberos.principal=hbase/_HOST@FCI.SARTHAK
```

The error remains same.. but I dig up more logs and found these

```

Java config name: /etc/krb5.conf

Loaded from Java config

>>>KinitOptions cache name is /tmp/krb5cc_1000

13:01:50.095 [main] DEBUG o.a.h.security.UserGroupInformation - hadoop login

13:01:50.095 [main] DEBUG o.a.h.security.UserGroupInformation - hadoop login commit

13:01:50.096 [main] DEBUG o.a.h.security.UserGroupInformation - using kerberos user:null

13:01:50.097 [main] DEBUG o.a.h.security.UserGroupInformation - using local user:UnixPrincipal: graph

13:01:50.098 [main] DEBUG o.a.h.security.UserGroupInformation - Using user: "UnixPrincipal: graph" with name graph

13:01:50.098 [main] DEBUG o.a.h.security.UserGroupInformation - User entry: "graph"

13:01:50.098 [main] DEBUG o.a.h.security.UserGroupInformation - Assuming keytab is managed externally since logged in from subject.

13:01:50.107 [main] DEBUG o.a.h.security.UserGroupInformation - UGI loginUser:graph (auth:KERBEROS)
```

And here I see
```

13:06:42.676 [Default-IPC-NioEventLoopGroup-1-2] DEBUG o.a.h.security.UserGroupInformation - PrivilegedAction as:graph (auth:KERBEROS) from:org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler.handlerAdded(NettyHBaseSaslRpcClientHandler.java:105)

13:06:42.677 [Default-IPC-NioEventLoopGroup-1-2] DEBUG o.a.h.security.UserGroupInformation - PrivilegedActionException as:graph (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
```

So basically, I want to override this value (graph) from UnixPrincipal with principal provided in jaas.conf file (gr...@FCI.SARTHAK) 

Thanks!!

[HadoopMarc]

Hi Sarthak,

Just a few comments:

• this is not really a gremlin-server problem. You might as well try to start the hbase shell as the graph user.
• instead of trying to configure a jaas conf file, you might also try to set KRB5-KTNAME environment variable
• be sure that you can issue a kinit as the graph user and that klist prints a valid tgt
• if you can succesfully run hbase shell as the sarthak user, you can also try to run gremlin-server as the sarthak user once as a check

HTH,    Marc

Op maandag 17 februari 2020 14:11:25 UTC+1 schreef Sarthak Ghosh:

[Vrunda Negandhi]

Hi,

I too have similar problem but what is not clear to me why jaas.conf can't work? Any suggestion.

[HadoopMarc]

Hi Vrunda,

My suggestion was to take small steps at a time because there too many ways a config can be wrong. If you need to use a keytab for gremlin-server running as a service, you either have to configure a jaas.conf with useKeyTab=true like the OP did or you have to issue a scheduled kinit using the keytab. But before that you want to be sure that:

 - you can run a list command in hbase shell as user vrunda

 - you can access janusgraph-hbase  as user vrunda (using gremlin console)

 - you can issue a "kinit -kt keytab principal" with the newly made keytab

 - etc.

HTH,     Marc

Op woensdag 19 februari 2020 14:35:48 UTC+1 schreef Vrunda Negandhi: