find metrics with specific tag values

[Jonathan Creasy]

I have this output from an HBASE Scan:

 x                                                                   column=id:tagv, timestamp=1365651075903, value=M\xED-

 x1bmqemfskrf2k0v6b81ae0acsdc0q7q                                    column=id:tagv, timestamp=1366111455718, value=N.~

 xjreilcdczfu8l4ap8f7ezhkkgqz8ef1                                    column=id:tagv, timestamp=1366067564742, value=N)/

 xss_regex_matched                                                   column=id:tagv, timestamp=1366237362249, value=NR?

 xweg0oyj7eadhu1il5nkacl2sag2sssp                                    column=id:tagv, timestamp=1366221567680, value=NF\xD1

 y                                                                   column=id:tagv, timestamp=1366044310590, value=N\x19\x03

 ygql7a9yceft78rck0kt85e07clcj9ua                                    column=id:tagv, timestamp=1366213233023, value=N@\xFE

 ymg9l862cqk3y8yxp3zjkg1x8goba8de                                    column=id:tagv, timestamp=1366063683555, value=N$\xA4

 zrjrc767mug1h0cv5jjntfsf3mj7u32h                                    column=id:tagv, timestamp=1366057305009, value=N!\

I'd like to know the best to identify the metrics which have data with the specific key values. 

For example "show me the metrics that have data with a tagv of zrjrc767mug1h0cv5jjntfsf3mj7u32h"

-Jonathan

--

Jonathan Creasy | Sr. Ops Engineer

e: j...@box.com | t: 314.580.8909

[ManOLamancha]

On Thursday, April 18, 2013 4:41:36 PM UTC-4, Jonathan Creasy wrote:

I'd like to know the best to identify the metrics which have data with the specific key values. 

For example "show me the metrics that have data with a tagv of zrjrc767mug1h0cv5jjntfsf3mj7u32h"

Are you looking for high-level or low-level? If the former, I'm almost done with the meta data and search code that would make it trivial to get that kind of information using a search engine (I have a working Elastic Search plugin). I just need a code review on some threading code and to finish up a CLI tool to generate TSMeta objects from old data then it'll be good to go. Then it would be possible to scan the uid table quickly to match TSUIDs.

If you're looking to go low-level and pull directly from HBase, you'd need to get the byte ID for your tagv, then setup a scanner with a byte value regex similar to what you'd find in TsdbQuery.createAndSetFilter() then run through the data table.

[Jonathan Creasy]

High level works. Is there a stable commit I should try?

[ManOLamancha]

On Thursday, April 18, 2013 8:41:41 PM UTC-4, Jonathan Creasy wrote:

High level works. Is there a stable commit I should try?

I'm just testing it out on our dev cluster but https://github.com/manolama/opentsdb/tree/next_meta has the code to generate TSUIDs for storage whenever a new UID or timeseries comes in. Just add "tsd.core.meta.enable_tracking = true" to a config file to get the ball rolling. The rough draft of the ES plugin is at https://github.com/manolama/OTSDBElasticSearch. I'll try to get some compilation script and ES mappings up this weekend as well as wrap up the indexing code. Meta data API docs are at http://opentsdb.net/docs/build/html/api_http/uid/index.html
 

[Chi Hoang]

Hi,
I've pulled in the latest commits on the next branch (HEAD is 405298a22d517daf20a774600a4f7469cb8a52ad), and am running with enable_tracking = true.  When running tsdb mkmetric, I get this:

tsdb mkmetric chi.test.metric1
Exception in thread "main" java.lang.NullPointerException
        at net.opentsdb.uid.UniqueId.getOrCreateId(UniqueId.java:389)
        at net.opentsdb.tools.UidManager.assign(UidManager.java:322)
        at net.opentsdb.tools.UidManager.runCommand(UidManager.java:194)
        at net.opentsdb.tools.UidManager.main(UidManager.java:159)

Looking at the code, UidManager.assign() doesn't call UniqueId.setTSDB(), so tsdb.getConfig() is probably throwing the NPE.

I think the metric still gets created:

tsdb uid grep chi
metrics chi.test.metric1: [0, 0, 11]

I also tried to run metasync, and got this:

tsdb uid metasync
2013-04-30 22:56:07,847 ERROR [Thread-17] UidManager: [16]Scanner Exception
java.lang.RuntimeException: [16] Should never be here
        at net.opentsdb.tools.UidManager$MetaSync.run(UidManager.java:1022) ~[tsdb-2.0.0.jar:85a1dfe]
org.hbase.async.UnknownRowLockException: org.apache.hadoop.hbase.UnknownRowLockException: Invalid row lock
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getLockFromId(HRegionServer.java:2664)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2021)
        at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1400)

Caused by RPC: GetRequest(table="tsdb-uid", key="\x00\x001", family="name", qualifiers=["metric_meta"], attempt=1, region=RegionInfo(table="tsdb-uid", region_name="tsdb-uid,,1367359058752.536783ba39c4b4bd6c8aadb9a3a3c9f2.", stop_key=""))
        at org.hbase.async.UnknownRowLockException.make(UnknownRowLockException.java:56) ~[asynchbase-1.4.1.jar:8c94a38]
        at org.hbase.async.UnknownRowLockException.make(UnknownRowLockException.java:32) ~[asynchbase-1.4.1.jar:8c94a38]
...

All this is from a completely new database.

Am I missing something?

Thanks.

Chi

[Chi Hoang]

I disabled tracking, and the problem still seems to be there.

I also noticed that trying to create multiple metrics doesn't work:

tsdb mkmetric chi.test.metric01 chi.test.metric02

Exception in thread "main" java.lang.NullPointerException
        at net.opentsdb.uid.UniqueId.getOrCreateId(UniqueId.java:389)
        at net.opentsdb.tools.UidManager.assign(UidManager.java:322)
        at net.opentsdb.tools.UidManager.runCommand(UidManager.java:194)
        at net.opentsdb.tools.UidManager.main(UidManager.java:159)

tsdb uid grep chi.test.metric0
metrics chi.test.metric01: [0, 0, 115]

Thanks.

Chi

[ManOLamancha]

On Tuesday, April 30, 2013 7:17:46 PM UTC-4, Chi Hoang wrote:

I disabled tracking, and the problem still seems to be there.

I also noticed that trying to create multiple metrics doesn't work:

Thanks for the report Chi. I'll try to figure out what's wrong tomorrow. Have to get some Tree code into testing first.

[ManOLamancha]

On Tuesday, April 30, 2013 7:09:40 PM UTC-4, Chi Hoang wrote:

Looking at the code, UidManager.assign() doesn't call UniqueId.setTSDB(), so tsdb.getConfig() is probably throwing the NPE.

I just pushed a patch to fix that bug so you shouldn't get the NPE any more. It also fixes the multiple metrics bug at the same time. Give it a shot please.

 

I also tried to run metasync, and got this:

tsdb uid metasync
2013-04-30 22:56:07,847 ERROR [Thread-17] UidManager: [16]Scanner Exception
java.lang.RuntimeException: [16] Should never be here
        at net.opentsdb.tools.UidManager$MetaSync.run(UidManager.java:1022) ~[tsdb-2.0.0.jar:85a1dfe]
org.hbase.async.UnknownRowLockException: org.apache.hadoop.hbase.UnknownRowLockException: Invalid row lock
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getLockFromId(HRegionServer.java:2664)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2021)
        at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1400)

Caused by RPC: GetRequest(table="tsdb-uid", key="\x00\x001", family="name", qualifiers=["metric_meta"], attempt=1, region=RegionInfo(table="tsdb-uid", region_name="tsdb-uid,,1367359058752.536783ba39c4b4bd6c8aadb9a3a3c9f2.", stop_key=""))
        at org.hbase.async.UnknownRowLockException.make(UnknownRowLockException.java:56) ~[asynchbase-1.4.1.jar:8c94a38]
        at org.hbase.async.UnknownRowLockException.make(UnknownRowLockException.java:32) ~[asynchbase-1.4.1.jar:8c94a38]
...

All this is from a completely new database.

This one will take a couple of more days for me to fix as I'm re-writing it all without row-locks as Tsuna showed me. But it is odd the lock was bad. What version of HBase are you running? Thanks!

[Chi Hoang]

hbase version
13/05/01 19:48:31 INFO util.VersionInfo: HBase 0.94.2-cdh4.2.0
13/05/01 19:48:31 INFO util.VersionInfo: Subversion git://ubuntu-slave07.jenkins.cloudera.com/var/lib/jenkins/workspace/CDH4.2.0-Packaging-HBase/build/cdh4/hbase/0.94.2-cdh4.2.0/source -r
13/05/01 19:48:31 INFO util.VersionInfo: Compiled by jenkins on Fri Feb 15 11:37:00 PST 2013

I'll rebuild with your fixes, and retest later today.  I think the locking may have caused some instability, so I'll pay a bit more attention to that compared to my experience at commit 7cb3fa2814f25248ffdff3072e1f568775eb7626.

Chi

[Paula Keezer]

Is this specific to the 2.0 branch?  Can it be used in the last version 1 release? If 2.0, can I run a separated 2.0 tsd to make use of this patch and keep all of my prod on 1.0?
Sorry for all the questions.  I've been away spinning up a 16K samples per seconds ingest on our openTSDB system.

Paula

[ManOLamancha]

On Thursday, May 2, 2013 3:58:55 PM UTC-4, Paula Keezer wrote:

Is this specific to the 2.0 branch?  Can it be used in the last version 1 release? If 2.0, can I run a separated 2.0 tsd to make use of this patch and keep all of my prod on 1.0?
Sorry for all the questions.  I've been away spinning up a 16K samples per seconds ingest on our openTSDB system.

Yeah it's all 2.0 specific but you can keep the 1.0 TSDs running and try building the meta with a 2.0 TSD, it won't affect 1.0 operation.

[Chi Hoang]

I ran with the fix, and it's working without a hitch.

Did you get a chance to finish up the search functionality?  I went through the api/uid docs and couldn't figure out how to effectively use the endpoints.

Thanks!

Chi

[ManOLamancha]

On Friday, May 3, 2013 12:45:20 PM UTC-4, Chi Hoang wrote:

I ran with the fix, and it's working without a hitch.

Did you get a chance to finish up the search functionality?  I went through the api/uid docs and couldn't figure out how to effectively use the endpoints.

I haven't written any query code yet that will let you hit the search engine from the OpenTSDB API but I should be able to get one cranked out in the next couple of weeks. For now though, you can use the rough Elastic Search plugin at https://github.com/manolama/OTSDBElasticSearch and the current "next" for of OpenTSDB will push new and edited meta data out to an ES cluster.

BTW, I could certainly use some help completing the plugin and figuring out the best mappings for Elastic Search if anyone wants to dig in. Thanks!