CDH 5.9 + OpenTSDB 2.3 + Kerberos

[sagonfor7]

Hi,

I'm trying to use OpenTSDB against a fully working Kerberized CDH 5.9 cluster. 

I've configured a jaas.conf and keytab for a principal I've created for the opentsdb user and granted this principal all privileges for HBase.

OpenTSDB fails to start with the stack trace below.

Has OpenTSDB been tested against a Kerberized Hadoop cluster?

Many thanks

11:38:37.268 INFO  [ZooKeeper.<init>] - Initiating client connection, connectString=manager-1:2181 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@59e5ddf

11:38:37.269 INFO  [HBaseClient.handleMetaZnode] - Connecting to .META. region @ 10.0.1.67:60020

11:38:37.270 INFO  [ZooKeeperSaslClient.run] - Client will use GSSAPI as SASL mechanism.

11:38:37.275 INFO  [ClientCnxn.logStartConnect] - Opening socket connection to server manager-1/10.0.1.53:2181. Will attempt to SASL-authenticate using Login Context section 'Client'

11:38:37.275 INFO  [KerberosClientAuthProvider.newSaslClient] - Connecting to hbase/datanode-0@REALM

11:38:37.275 INFO  [KerberosClientAuthProvider.run] - Client will use GSSAPI as SASL mechanism.

11:38:37.276 INFO  [ClientCnxn.primeConnection] - Socket connection established to manager-1/10.0.1.53:2181, initiating session

11:38:37.277 INFO  [ClientCnxn.onConnected] - Session establishment complete on server manager-1/10.0.1.53:2181, sessionid = 0x159dbe5876127d9, negotiated timeout = 5000

11:38:37.279 INFO  [RegionClient.channelConnected] - Initialized security helper: org.hbase.async.SecureRpcHelper96@3cb5f18b for region client: RegionClient@1058994674(chan=null, #pending_rpcs=1, #batched=0, #rpcs_inflight=0)

11:38:37.283 INFO  [ZooKeeper.close] - Session: 0x159dbe5876127d9 closed

11:38:37.283 INFO  [ClientCnxn.run] - EventThread shut down

11:38:37.283 INFO  [ClientCnxn.run] - EventThread shut down

11:38:37.285 INFO  [SecureRpcHelper96.handleResponse] - SASL client context established. Negotiated QoP: auth-conf on for: RegionClient@1058994674(chan=null, #pending_rpcs=1, #batched=0, #rpcs_inflight=0)

11:38:37.286 ERROR [RegionClient.exceptionCaught] - Unexpected exception from downstream on [id: 0xc949aed4, /10.0.1.53:39631 => /10.0.1.67:60020]

java.lang.IndexOutOfBoundsException: Not enough readable bytes - Need 191, maximum is 179

at org.jboss.netty.buffer.AbstractChannelBuffer.checkReadableBytes(AbstractChannelBuffer.java:668) ~[netty-3.9.4.Final.jar:na]

at org.jboss.netty.buffer.AbstractChannelBuffer.readBytes(AbstractChannelBuffer.java:338) ~[netty-3.9.4.Final.jar:na]

at org.jboss.netty.buffer.AbstractChannelBuffer.readBytes(AbstractChannelBuffer.java:344) ~[netty-3.9.4.Final.jar:na]

at org.hbase.async.SecureRpcHelper.wrap(SecureRpcHelper.java:235) ~[asynchbase-1.7.2.jar:na]

at org.hbase.async.RegionClient.encode(RegionClient.java:1385) ~[asynchbase-1.7.2.jar:na]

at org.hbase.async.RegionClient.sendRpc(RegionClient.java:998) ~[asynchbase-1.7.2.jar:na]

at org.hbase.async.RegionClient.sendQueuedRpcs(RegionClient.java:1141) ~[asynchbase-1.7.2.jar:na]

at org.hbase.async.RegionClient.becomeReady(RegionClient.java:664) ~[asynchbase-1.7.2.jar:na]

at org.hbase.async.SecureRpcHelper96.sendRPCHeader(SecureRpcHelper96.java:190) ~[asynchbase-1.7.2.jar:na]

at org.hbase.async.SecureRpcHelper96.handleResponse(SecureRpcHelper96.java:148) ~[asynchbase-1.7.2.jar:na]

at org.hbase.async.RegionClient.decode(RegionClient.java:1416) ~[asynchbase-1.7.2.jar:na]

at org.hbase.async.RegionClient.decode(RegionClient.java:88) ~[asynchbase-1.7.2.jar:na]

at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) ~[netty-3.9.4.Final.jar:na]

at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[netty-3.9.4.Final.jar:na]

[sagonfor7]

Update - backing off hbase.rpc.protection in HBase and OpenTSDB conf from "privacy" (using TLS) to "authentication" (not encrypting the link) makes this error go away. 

However, this means data is not secured between HBase and any clients. It's not possible negotiate authentication level protection for OpenTSDB alone as this fails the check that a 'common protection layer' has been established.

This is a showstopper for using OpenTSDB for us as we must use a secured Hadoop cluster.

[ManOLamancha]

On Saturday, January 28, 2017 at 7:54:12 AM UTC-8, sagonfor7 wrote:

Update - backing off hbase.rpc.protection in HBase and OpenTSDB conf from "privacy" (using TLS) to "authentication" (not encrypting the link) makes this error go away. 

However, this means data is not secured between HBase and any clients. It's not possible negotiate authentication level protection for OpenTSDB alone as this fails the check that a 'common protection layer' has been established.

This is a showstopper for using OpenTSDB for us as we must use a secured Hadoop cluster.

Would you mind filing a bug under https://github.com/OpenTSDB/asynchbase? We only have authentication enabled on our cluster to control user access but since telemetry is generally non-sensitive we haven't tested it with encryption. Likewise if you can create a VM image with kerberos and HBase setup, I'd be happy to work against it. Just had a hell of a time getting Kerberos working. Thanks.