Errors using opentsdb 2.2.0,2.3.0 pointed at hbase 1.3.0

[casey.d...@teamaol.com]

Trying to use Amazon managed EMR service running hbase 1.3.0 as a backend store for opentsdb. Tied both opentsdb version 2.2.0 and 2.3.0 and I am getting errors when trying to read from the cluster.

was wondering if anyone has a similar setup with opentsdb 2.2.0 or 2.3.0 pointed at hbase 1.3.0

Casey

[nwhitehead]

Tried this out. Not having any issues with single node HBase 1.0.3  using 2.2 or 2.3 (or 2.4, such as it is).

Got any more detail ?

[casey.d...@teamaol.com]

More info:

we are installing opentsdb 2.3.0 on the EMR master, that does the create tables and runs an instance we are using to "verify" the deployment is ok. On the master tsd instance I see the "stats" command running and inserting the tsd.* metrics with no errors, in fact I can see in the log where they are inserted for the first time and assigned their ids. However, when using the default http GUI running on the master tsd instance on port 4242, I cannot get results for any of the stats I know are in the system. The following is the most prevalent error I see.

org.hbase.async.InvalidResponseException: Scan RPC response was for scanner ID 0 but we expected2

at org.hbase.async.Scanner$CloseScannerRequest.deserialize(Scanner.java:1466) ~[asynchbase-1.7.2.jar:na] at org.hbase.async.RegionClient.decode(RegionClient.java:1497) ~[asynchbase-1.7.2.jar:na] at org.hbase.async.RegionClient.decode(RegionClient.java:88) ~[asynchbase-1.7.2.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[netty-3.9.4.Final.jar:na] at org.hbase.async.RegionClient.handleUpstream(RegionClient.java:1223) ~[asynchbase-1.7.2.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.handler.timeout.IdleStateAwareChannelHandler.handleUpstream(IdleStateAwareChannelHandler.java:36) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:294) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) ~[netty-3.9.4.Final.jar:na] at org.hbase.async.HBaseClient$RegionClientPipeline.sendUpstream(HBaseClient.java:3121) ~[asynchbase-1.7.2.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) [netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.9.4.Final.jar:na] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.9.4.Final.jar:na] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.9.4.Final.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_121] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_121] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121] on the Master i can run the tsdb uid MetricName utility and get back valid results for a sample of the default stats being inserted into the system. config items we are setting: tsd.core.auto_create_metrics = true tsd.core.meta.enable_realtime_ts = true tsd.core.meta.enable_realtime_uid = true tsd.storage.fix_duplicates = true tsd.storage.salt.width = 1 tsd.storage.salt.buckets = 256 Any help would be greatly appreciated to get this resolved, so we can stay current with latest EMR versions.

[ManOLamancha]

On Thursday, June 1, 2017 at 6:45:14 AM UTC-7, casey.d...@teamaol.com wrote:

More info:

we are installing opentsdb 2.3.0 on the EMR master, that does the create tables and runs an instance we are using to "verify" the deployment is ok. On the master tsd instance I see the "stats" command running and inserting the tsd.* metrics with no errors, in fact I can see in the log where they are inserted for the first time and assigned their ids. However, when using the default http GUI running on the master tsd instance on port 4242, I cannot get results for any of the stats I know are in the system. The following is the most prevalent error I see.

org.hbase.async.InvalidResponseException: Scan RPC response was for scanner ID 0 but we expected2

at org.hbase.async.Scanner$CloseScannerRequest.deserialize(Scanner.java:1466) ~[asynchbase-1.7.2.jar:na] at org.hbase.async.RegionClient.decode(RegionClient.java:1497) ~[asynchbase-1.7.2.jar:na] at org.hbase.async.RegionClient.decode(RegionClient.java:88) ~[asynchbase-1.7.2.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[netty-3.9.4.Final.jar:na] at org.hbase.async.RegionClient.handleUpstream(RegionClient.java:1223) ~[asynchbase-1.7.2.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.handler.timeout.IdleStateAwareChannelHandler.handleUpstream(IdleStateAwareChannelHandler.java:36) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:294) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) ~[netty-3.9.4.Final.jar:na] at org.hbase.async.HBaseClient$RegionClientPipeline.sendUpstream(HBaseClient.java:3121) ~[asynchbase-1.7.2.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) [netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.9.4.Final.jar:na] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.9.4.Final.jar:na] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.9.4.Final.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_121] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_121] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121] on the Master i can run the tsdb uid MetricName utility and get back valid results for a sample of the default stats being inserted into the system. config items we are setting: tsd.core.auto_create_metrics = true tsd.core.meta.enable_realtime_ts = true tsd.core.meta.enable_realtime_uid = true tsd.storage.fix_duplicates = true tsd.storage.salt.width = 1 tsd.storage.salt.buckets = 256 Any help would be greatly appreciated to get this resolved, so we can stay current with latest EMR versions.

Hmm, sounds like it could be something that changed with 1.3 if the server is now allowed to close the scanner. There's a PR https://github.com/OpenTSDB/asynchbase/pull/169 that I'll validate and pull in for AsyncHBase. 

[ManOLamancha]

On Monday, June 5, 2017 at 6:03:08 PM UTC-7, ManOLamancha wrote:

Hmm, sounds like it could be something that changed with 1.3 if the server is now allowed to close the scanner. There's a PR https://github.com/OpenTSDB/asynchbase/pull/169 that I'll validate and pull in for AsyncHBase. 

Try dropping in this jar https://oss.sonatype.org/content/repositories/snapshots/org/hbase/asynchbase/1.8.0-SNAPSHOT/asynchbase-1.8.0-20170608.052734-6.jar in place of the asynchbase jar and it should work with 1.3.0.