List of all Metric Names

[avaitla]

I realize there is an autocomplete query type, but is there any way to simply list all metric names being tracked in opentsdb?

[avaitla]

Sorry, this is surprisingly easy with: ./tsdb uid grep ''

[tsuna]

On Mon, Jul 16, 2012 at 10:32 AM, avaitla <avai...@gmail.com> wrote:
> Sorry, this is surprisingly easy with: ./tsdb uid grep ''

Yes if you just want metric names you can do:
$ tsdb uid grep metrics .

--
Benoit "tsuna" Sigoure
Software Engineer @ www.StumbleUpon.com

[Peter Speybrouck]

is there any way to do it using the HTTP API?

If I use this: /suggest?type=metrics&q=
I only get 29 metrics, while
./tsdb uid grep metrics '' shows me that there are 120.

A wild character like * doesn't help either.

An interesting observation perhaps, if I do the same for tagv, I also get 29 results (from 112 when using ./tsdb)
Any idea why 29?

I looked at the code and there is a special case in the getSuggestedScanner method in UniqueId.java but my knowlegde of the code is not sufficient enough to make sense of it...


Also, is there a way to scan for tags that are defined for a certain metric? The website seems to do something like that

[Kevin Ortman]

If you are interested you can check out tsquery (https://github.com/KevinOrtman/tsquery).  It's a lightweight JSON/P interface to OpenTSDB, a forked and heavily modified version of Facebook's tsdash serverlet component.  This is still under development but functional at the moment so comments/suggestions/pull requests are very welcome. 

You can kick the tires @ ttp://opentsdb01.cloudapp.net/tsquery/metrics

A few examples:

Retrieve List of Metrics ( ttp://opentsdb01.cloudapp.net/tsquery/metrics ) returns

     [

         "df.1kblocks.free",

         "df.1kblocks.total",

         "df.1kblocks.used",

         "df.inodes.free",

         "df.inodes.total",

         "df.inodes.used",

     etc...

     ]

Retrieve List of Tags/Values for a Metric ( ttp://opentsdb01.cloudapp.net/tsquery/header?params={"tsFrom":1325791058,"tsTo":1357413458,"metric":"net.sockstat.memory"} ) returns

    {

    "tags": {

        "host": ["opentsdb01"],

        "type": ["udp","tcp","ipfrag"]

    },

    "commontags": [

        "host",

        "type"

    ],

    "name": "net.sockstat.memory",

    "rows": 0

    }

Query TimeSeries Data ( ttp://opentsdb01.cloudapp.net/tsquery/data?params={"tsFrom":1357413358,"tsTo":1357413458,"metrics":[{"name":"net.sockstat.memory"},{"name":"iostat.disk.msec_total"}]} ) returns:

   {

    "series": [

        {

            "name": "host=opentsdb01",

            "data": [

[1357413378000,45056],

                [1357413393000,45056],

                [1357413408000,49152],

                [1357413423000,53248],

                [1357413438000,40960],

                [1357413453000,69632]

            ]

        },

        {

            "name": "host=opentsdb01",

            "data": [

                [1357413391000,11464111],

                [1357413451000,11464696]

            ]

        }

    ],

    "loadtime": 33,

    "serializationtime": 0

    }

---

Date: Sat, 5 Jan 2013 09:09:02 -0800
From: peter.sp...@gmail.com
To: open...@googlegroups.com
CC: avai...@gmail.com
Subject: Re: List of all Metric Names

[tsuna]

On Sat, Jan 5, 2013 at 9:09 AM, Peter Speybrouck
<peter.sp...@gmail.com> wrote:
> is there any way to do it using the HTTP API?

Not yet, although that could easily be added. Maybe we could add an
extra optional parameter to the /suggest endpoint so you can control
how many results you want back. Would that work for you?

> If I use this: /suggest?type=metrics&q=
> I only get 29 metrics, while
> ./tsdb uid grep metrics '' shows me that there are 120.

It's because the former is pre-configured to return only 25 results,
whereas the command line greps everything.

> A wild character like * doesn't help either.

The /suggest endpoint only does prefix matches.

> An interesting observation perhaps, if I do the same for tagv, I also get 29
> results (from 112 when using ./tsdb)
> Any idea why 29?

It should be 25. It returns 29 due to a bug that I fixed recently in
https://github.com/tsuna/opentsdb/commit/098ab0196be676285baea31e9eae435a14c38611

> Also, is there a way to scan for tags that are defined for a certain metric?
> The website seems to do something like that

This is not possible and would be hard to implement today, because
there is nothing that keeps track of associations between metric names
and tag names.

--
Benoit "tsuna" Sigoure

[Peter Speybrouck]

Thanks for the info.

having the possibility to get all metrics or with a custom limit would be fine.
A wildcard character for the q argument would be useful I think. Could be something I can look into perhaps.

[tsuna]

On Sun, Jan 13, 2013 at 11:45 AM, Peter Speybrouck
<peter.sp...@gmail.com> wrote:
> having the possibility to get all metrics or with a custom limit would be
> fine.

I filed a feature request @ https://github.com/OpenTSDB/opentsdb/issues/147

> A wildcard character for the q argument would be useful I think. Could be
> something I can look into perhaps.

This is more complicated to implement without doing a full table scan,
because HBase can only efficiently do prefix searches, not arbitrary
searches with arbitrary wildcards. Although since the number of
metrics & tags is expected to be small (generally in the tens of
thousands, and no more than 16M) we could in theory load everything in
memory and then do whatever search we want there.

--
Benoit "tsuna" Sigoure

[Peter Speybrouck]

My knowledge about HBase is still around zero at this moment. Thanks for explaining ;-)
I should look into the inner workings a little more...

[Kevin Ortman]

Check out the loadMap() function in https://github.com/facebook/tsdash/blob/master/server/src/com/facebook/tsdb/tsdash/server/data/hbase/IDMapSyncLoader.java for an example using the hbase java sdk.  It's called from doGet() in https://github.com/facebook/tsdash/blob/master/server/src/com/facebook/tsdb/tsdash/server/MetricsEndpoint.java if you want to follow the logic in a debug.

[anku...@gmail.com]

Is there any configuration for storing the number of metrics to be returned by an HTTP API for getting all metrics. Each time I query I get only 25 metrics , though there are more than that!

[Pradeep Chhetri]

Hello Ankur,

If you are running the latest revision of "next" branch, you can pass
"max" parameter like this,

$ curl -s
'http://opentsdb-1.nm.flipkart.com:4444/api/suggest?type=metrics&max=4'
| python -mjson.tool

If you want complete list of metrics, you can set the max value to some
very high number.

Thanks,

-Pradeep