mapr opentsdb 2.0 configuration

[Sagar Shukla]

Hi All,

I have Mapr Hadoop cluster running properly. I am trying to setup opentsdb 2.0, but it cries for hbase client and region server ports. As per mapr documentation, hbase client and region server is not required, since entire communication is through Hbase APIs.

Have anyone done this configuration and can help me out with necessary steps?

2014-11-30 02:26:25,084 INFO  [Hashed wheel timer #2-EventThread] HBaseClient: Connecting to .META. region @ 127.0.0.1:60020
2014-11-30 02:26:25,086 WARN  [New I/O boss #130] HBaseClient: Couldn't connect to the RegionServer @ 127.0.0.1:60020
2014-11-30 02:26:25,086 INFO  [New I/O boss #130] HBaseClient: Lost connection with the .META. region
2014-11-30 02:26:25,087 INFO  [New I/O worker #98] HBaseClient: Invalidated cache for .META. as null still seems to be splitting or closing it.
2014-11-30 02:26:25,087 ERROR [New I/O boss #130] RegionClient: Unexpected exception from downstream on [id: 0xb031ecb8]
java.net.ConnectException: Connection refused: /127.0.0.1:60020
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.7.0_60]
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) ~[na:1.7.0_60]
        at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:150) ~[netty-3.9.1.Final.jar:na]
        at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105) ~[netty-3.9.1.Final.jar:na]
        at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79) ~[netty-3.9.1.Final.jar:na]
        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) ~[netty-3.9.1.Final.jar:na]
        at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42) ~[netty-3.9.1.Final.jar:na]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_60]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_60]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_60]

Thanks in advance.

Regards,
Sagar

[ManOLamancha]

Which version of HBase is your cluster running and which version of OpenTSDB are you trying to connect with? It may be that you're not using a version of AsyncHBase that's compatible with your cluster. Also, is your HBase master server and one of the regions running on the same host where you're trying to launch the TSD? It sounds like you connected to Zookeeper but it was trying to look at localhost for the META region.

[Sagar Shukla]

I am running Mapr hadoop cluster running:
- version of HBase 0.94.21
- OpenTSDB version 2.0.0
AsyncHBase version is 1.5.0

As recommended by Mapr, HBase master and region server services are not required on the cluster, since they operate through the HBase APIs, so there is no host dependency as such. Though I have an independent node for TSD service.

How can I configure Zookeeper to not look at localhost for META region?

[ManOLamancha]

On Monday, December 1, 2014 6:18:19 PM UTC-8, Sagar Shukla wrote:

I am running Mapr hadoop cluster running:
- version of HBase 0.94.21
- OpenTSDB version 2.0.0
AsyncHBase version is 1.5.0

1.5 should work fine with .94.

 

As recommended by Mapr, HBase master and region server services are not required on the cluster, since they operate through the HBase APIs, so there is no host dependency as such. Though I have an independent node for TSD service.

Do you mean you disabled the web service for the master(s) and region servers? That's fine, it's useful for debugging but isn't required for OpenTSDB. But it does require access to the RPC ports. 

How can I configure Zookeeper to not look at localhost for META region?

 HBase updates Zookeeper with the locations and ports of all of it's components. Is ZK running on a separate machine from your TSD? If you're using the config file for OpenTSDB, did you set "tsd.storage.hbase.zk_quorum" to point to your ZK host? (or if you're using the CLI it's "--zkquorum"

[Sagar Shukla]

As per MAPR guide, HBase master and Region services are not required to be running. The components use APIs for communication and hence it implied to me that RPC ports are also not required, if services need not be running.

I have configured zkquorum to point to my zookeeper nodes, so I do not feel that should be an issue.

[Ted Dunning]

Sagar,

Are you trying to run with HBase or with MapR-DB?  If you are trying to use HBase, then you need HBase master and region servers.  If you are using MapR DB, then you don't since that capability is built into the MapR file system.

If you are trying to use MapR DB, which version of MapR are you running?  There was a bug in MapR DB's filtering that used PERL compatible regexes previously.  This made TSDB 2.0 not work right.  This has been fixed, but you have to have the right version of MapR to get the fix.

[John A. Tamplin]

On Tue, Dec 2, 2014 at 6:10 PM, Sagar Shukla <sag...@gmail.com> wrote:

As per MAPR guide, HBase master and Region services are not required to be running. The components use APIs for communication and hence it implied to me that RPC ports are also not required, if services need not be running.

I have configured zkquorum to point to my zookeeper nodes, so I do not feel that should be an issue.

If HBase isn't running, where would the data be stored? 

--

John A. Tamplin

[Ted Dunning]

The MapR data platform implements multiple API's.  These include file-like API's like HDFS and NFS and also the HBase API.

Because the MapR DB capability implements the HBase API in the C++ filesystem layer and because the MapR filesystem has very different (generally looser) constraints than HDFS, the MapR filesystem implementation doesn't suffer from things like compaction delays, has much more uniform latency and much higher throughput.  The MapR implementation of the HBase API shares a bit of client-side code with HBase, but is otherwise a ground-up implementation with many architectural differences.

Overall, the performance combined with API compatibility makes MapR DB very nice for hosting Open TSDB, especially since tables are now available in the free version of MapR (the M3 edition).

Indeed, this is how we got the very high throughput that Jim Scott was describing a month ago or so.  The 110 million points per second he quoted have since been extended to >300 million points per second.

ANYWAY,

If supporting Sagar's questions gets too MapR specific, I will pull him off into a private email corner.  But until we establish what sort of persistence layer he is trying to use, the discussion should continue here.

[Sagar Shukla]

Hi Ted,

Thanks for the detailed information. I am running mapr version 4.0.1 .

As mentioned in the thread, only the above mentioned packages are running on the client node where I am trying to configure OpenTSDB.

Regards,
Sagar

[Ted Dunning]

Let me see if I can get Hakim in on this thread.  He did the latest work on this.

[Abdel Hakim Deneche]

Hi Sagar,

I generally get this error when opentsdb tries to access it's tables (tsdb, tsdb-uid, tsdb-meta) in hbase instead of mapr-db.

assuming your linux user is "sagar", the create_table.sh script should've created the following mapr-db tables:

/mapr/sagar/tsdb

/mapr/sagar/tsdb-uid

try adding the following arguments when launching the tsd:

--table=/mapr/sagar/tsdb --uidtable=/mapr/sagar/tsdb-uid

On Monday, December 1, 2014 11:14:54 AM UTC-8, Sagar Shukla wrote:

[Abdel Hakim Deneche]

you also need to replace asynchbase-1.0.5.jar with the one provided by mapr (asynchbase-1.5.0-mapr.jar).

I am working on a set of installation script to make it easier to run opentsdb on a mapr cluster, feel free to give them a try here:

https://github.com/adeneche/opentsdb-maprdb-install

They are a work in progress but they work fine for me.

[Sagar Shukla]

Hi Abdel,

Thanks. That did the trick. Setting tsd.storage.hbase.data_table and tsd.storage.hbase.uid_table variable values to my tables in HDFS got things working for me.

Appreciate your detailed information.

Regards,
Sagar

[Abdel Hakim Deneche]

I'm glad it worked :D

Best,

Hakim

[Michael Krysiak]

I ran the test in https://github.com/adeneche/opentsdb-maprdb-install, and although the scan works, querying seems to drop the tags.

[user@host opentsdb-maprdb-install]$ tsdb scan --import 1y-ago sum mymetric.stock
mymetric.stock 1407165399 196.3000030517578 symbol=VOD.L
mymetric.stock 1407165402 196.3000030517578 symbol=VOD.L
...

[user@hos opentsdb-maprdb-install]$ tsdb query 1y-ago sum mymetric.stock
mymetric.stock 1407165399000 680.500015 {}
mymetric.stock 1407165401000 904.625000 {}
...

[user@hos opentsdb-maprdb-install]$ tsdb query 1y-ago sum mymetric.stock symbol=VOD.L

[returns nothing]

[Sagar Shukla]

Hi Michael,

In my environment, when I try to scan or query, I am getting results in both the cases. Somehow, if I change the tag values in query request, results remain the same. Not sure if I am inserting data incorrectly.

I will investigate further at my end to see the results.

Regards,
Sagar

[Abdel Hakim Deneche]

Hi Michael,

This is strange indeed, I will investigate this and will let you know if I find anything.

Thanks

[Abdel Hakim Deneche]

Hi Michael,

we did some tests and identified the issue as a bug. We are actually working on a fix, I will let you know as soon as it is available.

Thanks,

Hakim

On Thursday, December 4, 2014 7:00:24 AM UTC-8, Michael Krysiak wrote:

[ManOLamancha]

On Tuesday, December 2, 2014 at 8:50:18 PM UTC-8, Abdel Hakim Deneche wrote:

you also need to replace asynchbase-1.0.5.jar with the one provided by mapr (asynchbase-1.5.0-mapr.jar).

I am working on a set of installation script to make it easier to run opentsdb on a mapr cluster, feel free to give them a try here:

https://github.com/adeneche/opentsdb-maprdb-install

They are a work in progress but they work fine for me.

Where can we see the code for that Mapr asynchbase? Is that in Github? (haven't looked yet) It'd be cool if we could get some of that into the main AsyncHBase repo. 

[Jim Scott]

This is the repo.. https://github.com/mapr/asynchbase

If I recall, the only real part of this library that is important from a modification standpoint is the functionality to bypass utilizing zookeeper for regionserver information. It bypasses that code and talks to the MapR client to interact with the MapR-FS.

--

Jim Scott

@kingmesal

[Jim Scott]

Was there anything in particular you were hoping to accomplish with the changes in the MapR version of that library? e.g. like including in the upstream so that one library might work with any distro?

--

Jim Scott

@kingmesal

[Michael Krysiak]

The new MapR asynchbase 1.5.0-mapr-1501 fixes the querying with tags issue.  https://github.com/mapr/asynchbase/tree/v1.5.0-mapr-1501

[Abdel Hakim Deneche]

we just updated the installation scripts to use the latest version of asynchbase. You can give it a try here:

https://github.com/mapr-demos/opentsdb-maprdb-install

queries with tag filters should run fine now.

Thanks!

[Michael Krysiak]

While the patch is an improvement for queries with tag filters, we're still occasionally seeing stack traces that we don't get with Apache HBase.  Here's a couple examples - this query works some of the time, but if I keep changing the time range, I eventually hit one of these errors.

10:22:59.936 ERROR [HttpQuery.logError] - [id: 0xb6680995, /192.168.6.24:50241 => /192.168.92.110:4242] Internal Server Error on /q?start=2015/01/26-07:45:00&end=2015/01/28-10:08:10&m=sum:statistics.client.timeavg%7Busername=*%7D&o=&yrange=%5B0:%5D&wxh=1505x649&json
net.opentsdb.core.IllegalDataException: HBase returned a row that doesn't match our scanner (Scanner(table="/opentsdb/tsdb", start_key=[0, 0, 70, 84, -56, -7, 112, 0, 0, 6, 0, 81, -102, 0, 0, 16, 0, 73, -105, 0, 0, 19, 0, 81, -90, 0, 0, 20, 0, 84, -119, 0, 0, 22, 0, 73, 71, 0, 0, 23, 0, 81, 109], stop_key="\x00\x00 FT\xC9\tl", columns={"t"}, populate_blockcache=true, max_num_rows=128, max_num_kvs=4096, region=null, filter=KeyRegexpFilter("(?s)^.{7}(?:.{6})*\Q^@^@^T\E.{3}(?:.{6})*$", ISO-8859-1), scanner_id=0x0000000000000000), mresultScanner=com.mapr.fs.MapRResultScanner@175a9e02)! [KeyValue(key=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0,  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], family="t", qualifier=[0, 0], value=[0, 0, 0, 0], timestamp=1422457601982)] does not start with [0, 0, 70]
        at net.opentsdb.core.TsdbQuery$1ScannerCB.call(TsdbQuery.java:398) ~[tsdb-2.0.1.jar:]
        at net.opentsdb.core.TsdbQuery$1ScannerCB.call(TsdbQuery.java:352) ~[tsdb-2.0.1.jar:]
        at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) ~[suasync-1.4.0.jar:fe17b98]
        at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) ~[suasync-1.4.0.jar:fe17b98]
        at com.stumbleupon.async.Deferred.callback(Deferred.java:1005) ~[suasync-1.4.0.jar:fe17b98]
        at org.hbase.async.HBaseRpc.callback(HBaseRpc.java:538) ~[asynchbase-1.5.0-mapr-1501.jar:na]
        at org.hbase.async.MapRThreadPool$ScanRpcRunnable.run(MapRThreadPool.java:436) ~[asynchbase-1.5.0-mapr-1501.jar:na]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_65]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_65]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]



10:29:45.472 ERROR [HttpQuery.logError] - [id: 0x8d71d509, /192.168.6.24:50426 => /192.168.92.109:4242] Internal Server Error on /q?start=2015/01/27-09:58:00&end=2015/01/28-10:29:19&m=statistics.client.timeavg%7Busername=*%7D&o=&yrange=%5B0:%5D&wxh=1505x649&json
net.opentsdb.core.IllegalDataException: HBase returned a row that doesn't match our scanner (Scanner(table="/opentsdb/tsdb", start_key=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], stop_key="\x00\x004T\xC9\x0Ea", columns={"t"}, populate_blockcache=true, max_num_rows=128, max_num_kvs=4096, region=null, filter=KeyRegexpFilter("(?s)^.{7}(?:.{6})*\Q^@^@^T\E.{3}(?:.{6})*$", ISO-8859-1), scanner_id=0x0EAD000CC000DEAD), mresultScanner=com.mapr.fs.MapRResultScanner@3c6a2757)! [KeyValue(key=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], family="t", qualifier=[0, 0], value=[0, 0, 0, 0], timestamp=1422372805257), KeyValue(key=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], family="t", qualifier=[0, 0], value=[0, 0, 0, 0], timestamp=1422374184071), KeyValue(key=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], family="t", qualifier=[0, 0], value=[0, 0, 0, 0], timestamp=1422376797761)] does not start with [0, 0, 52]
        at net.opentsdb.core.TsdbQuery$1ScannerCB.call(TsdbQuery.java:398) ~[tsdb-2.0.1.jar:]
        at net.opentsdb.core.TsdbQuery$1ScannerCB.call(TsdbQuery.java:352) ~[tsdb-2.0.1.jar:]
        at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) ~[suasync-1.4.0.jar:fe17b98]
        at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) ~[suasync-1.4.0.jar:fe17b98]
        at com.stumbleupon.async.Deferred.callback(Deferred.java:1005) ~[suasync-1.4.0.jar:fe17b98]
        at org.hbase.async.HBaseRpc.callback(HBaseRpc.java:538) ~[asynchbase-1.5.0-mapr-1501.jar:na]
        at org.hbase.async.MapRThreadPool$ScanRpcRunnable.run(MapRThreadPool.java:436) ~[asynchbase-1.5.0-mapr-1501.jar:na]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_65]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_65]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]

[Abdel Hakim Deneche]

What version of OpenTSDB are you using ? did you use opentsdb-mapr-install scripts ?

Could you give me more details about the data and the specific queries that generated those errors ?

Thanks!

[Michael Krysiak]

Hey Abdel,

Sorry for the slow response, I was away on vacation.   We're running 2.0.1, commit 76b928da99db7552a79e1ac04dbb3810908b8b5b.   I didn't use the script directly since we build an RPM for easier deployment, but my changes are fundamentally the same. 

This issue does seems to be triggered by a unique metric type of ours, but I'm not 100% confident in that statement yet. I can confirm that this same metric type works fine in Apache HBase though. Below is a snippet of the metric and tag structure.  There are about 1,000 unique remotename tagv's and also thousands of unique username tagv's.

statistics.client.timeavg 1423670082 511.5 class=namespace_read remotename=hosta username=usera node=3 host=servera proto=nfs3
statistics.client.timeavg 1423670360 81.5999984741211 class=namespace_read remotename=hostb username=userb node=3 host=servera proto=nfs3
statistics.client.timeavg 1423670010 1434.0 class=namespace_write remotename=hostc username=usera node=3 host=servera proto=nfs3