Metric/Tag Metadata HBase Storage
ManOLamancha
I'm digging into the meta data storage for metrics and tags directly
in HBase and I want to see what ya'll think of this.
I'd add another column/qualifier to the Name column family only and it
would contain a JSON string with the meta data about the metric, tag
name (tagk) or metric hash (more on that later). It wouldn't need to
be in the ID column family as TSDB generally maps everything to an ID
anyway. I also don't need to have different names for the meta cell as
each row is a unique ID and can only have either a "metrics", "tagk"
or "tagv" cell. So the meta data will be associated with whatever cell
is named in the row. It's fully backwards compatible too, as it
doesn't change the contents of the qualifier cell. Take a look at the
table below and see if I have it right please.
+------+-------------+-------------+-------------+------+
| Key | metrics | tagk | tagv | meta |
+------+-------------+-------------+-------------+------+
| uid | string name | | | JSON |
| uid | | string name | | JSON |
| uid | | | string name | |
| hash | | | | JSON |
+------+-------------+-------------+-------------+------+
I think storing the metadata as a JSON string would be the easiest way
to work with the metadata, using Jackson to serialize and deserialize
the data quickly. I wanted to avoid having lots (5 to 10) cells for
individual meta data properties as it's tough to keep the names small
(as recommended by the HBase docs) and unique.
The metric hash is a bunch of rows that I would add to this table that
are the same as the "tsdb" table's row keys but missing the timestamp.
e.g. metric uid | tagk1 | tagv1 | tagk2 | tagv2. Two reasons I want
this here are:
1) It gives us a smaller, consecutive place to scan for different meta/
queries such as "give me all the tags associated with metric x" or
"what metrics are associated with tagk k = tagv v?". This would be a
lot faster than traversing the entire "tsdb" table.
2) Each unique metric will have metadata associated with it.
I'll write a small utility to walk existing user's "tsdb" table and
build the hashes, then all new metrics will automatically generate a
new hash entry too.
So let me know what ya'll think and thanks!
Dave Barr
never got around to implementing them. The two main use cases I had
was being able to have a description of the metric, and also a boolean
flag to indicate if this metric is a rate. This would be used as a
hint in the UI when you typed in this metric that it would
automatically check the 'rate' box for you.
The hash table is good too. Having a relatively easy way of saying
"gimme all the metrics for host=blah" would be good. We may want a
way though to reset/clear these, though, or perhaps bucketize them by
some time interval (week? month?). Metrics change and decay over
time, and roles of hosts change. Some box may have been a web server
one day and have a bunch of metrics one month, but later it's doing
something else and has a bunch of different metrics. At some point I
want to forget about associating that box with those old web metrics.
If we put like year + month in the key, those associations will be
automatically regenerated each month, but we will still be able to
search back and find old associations if we make queries against the
older data.
--Dave
ManOLamancha
On Feb 29, 12:58 pm, Dave Barr <dave.b...@gmail.com> wrote:
> I like this idea. From day one I wanted metadata for metrics but we
> never got around to implementing them. The two main use cases I had
> was being able to have a description of the metric, and also a boolean
> flag to indicate if this metric is a rate. This would be used as a
> hint in the UI when you typed in this metric that it would
> automatically check the 'rate' box for you.
things like:
max - a maximum value for calculating percentages (e.g. nic speed)
type - similar to RRD or maybe just the 'rate' flag
display_name - display name
interval - calculated value of how often this metric is comming in
notes - user notes
unit - units for this metric
first_received - when created
last_received - last time it was received (batch updated every hour or
so so we're not hammering HBase twice per metric)
retention - possibly how long to keep data
> The hash table is good too. Having a relatively easy way of saying
> "gimme all the metrics for host=blah" would be good. We may want a
> way though to reset/clear these, though, or perhaps bucketize them by
> some time interval (week? month?). Metrics change and decay over
> time, and roles of hosts change. Some box may have been a web server
> one day and have a bunch of metrics one month, but later it's doing
> something else and has a bunch of different metrics. At some point I
> want to forget about associating that box with those old web metrics.
> If we put like year + month in the key, those associations will be
> automatically regenerated each month, but we will still be able to
> search back and find old associations if we make queries against the
> older data.
given metric would be immutable. If we have a web server "web01" with
a metric like [cpu.user host=web01], then when someone changes the
role and renames the server, we'd have a new unique hash [cpu.user
host=db01] or something. Most of our metrics will have a "host" tag
but what cases do you think might goof that up?
Also, if we have the last_received field in the metadata, we could
have a scraper go through and clean out the old data if necessary and
that would take care of the old associations. Thanks!
Dave Barr
> I was thinking that with the cool tagging system ya'll have, that a
> given metric would be immutable. If we have a web server "web01" with
> a metric like [cpu.user host=web01], then when someone changes the
> role and renames the server, we'd have a new unique hash [cpu.user
> host=db01] or something. Most of our metrics will have a "host" tag
> but what cases do you think might goof that up?
In our case we are constantly recycling hosts in different roles. We
essentially never rename hosts (they are named based on their physical
location).
When we move a host to a different role, the combination of tags
though will generally be different. We use a hybrid scheme here,
where each service has a role name and a value (so you can have
overlap different services on a host, just not multiple instances of a
service on a given host). So for example, of a box is part of a
memcache pool "main" and also webserver pool "default", it would have
the tags "role_memcache=main role_web=default".
So yeah, since the list of tags, and thus the hash would change when
we moved a host's role, it would cover this case. Stale metrics would
only stick around if a service changed such that a metric was no
longer relevant or used. In this case it's arguable if you would want
to delete this association anyway as long as you wanted to retain the
historical data for that metric.
--Dave
ManOLamancha
On Feb 29, 5:04 pm, Dave Barr <dave.b...@gmail.com> wrote:
Igor Pomazanov
I was looking at the 'hash' portion of this problem as well. Today
after number of metrics/tags/tag values stored in OpenTSDB becomes
sufficiently large, 'suggest' functionality in OpenTSDB UI starts
being not very useful. Suggestions will always contain first 25
entries and if value you are looking for starts with the same prefix
but does not fall into the first 25, you will end up typing it.
One way to improve this would be for ‘suggest ‘ function to take into
account information already entered into other fields in the UI. So if
I already entered ‘cpu.utilization’ as a metric name, it will suggest
only tags and tag values which exist for that metric. If I entered
‘host’ as a tag name, it will suggest only tag values and metric names
which exist for ‘host’ tag. If I entered tag value ‘myhost’, it will
suggest only tag names having that value and corresponding metric
names. And if I entered metric ‘cpu.utilization’, first tag
datacenter=’datacenter1|datacenter2’, and second tag ‘host’, it will
suggest only tag values which exist for metric ‘cpu.utilization’ in
datacenters 1 and 2.
To support this, API of ‘suggest’ function should be something like:
To get a list of tag keys which exist for metric ‘cpu.utilization’:
http://localhost:4242/suggest?type=tagk&q=&metrics=cpu.utilization
To get a list of tag values which exist for metric ‘cpu.utilization’:
http://localhost:4242/suggest?type=tagv&q=&metrics=cpu.utilization
To get a list of tag values which exist for tag ‘host’:
http://localhost:4242/suggest?type=tagv&q=&tagk=host
To get a list of metrics which exist for tag ‘host’:
http://localhost:4242/suggest?type=metrics&q=&tagk=host
To get a list of tags which exist for tag value ‘myhost’:
http://localhost:4242/suggest?type=tagk&q=&tagv=myhost
To get a list of metrics which exist for tag value ‘myhost’:
http://localhost:4242/suggest?type=metrics&q=&tagv=myhost
To get a list of tag values which exist for metric ‘cpu.utilization’
and tag datacenter=’datacenter1|datacenter2’:
http://localhost:4242/suggest?type=tagv&q=&metrics=cpu.utilization&tagk=datacenter&tagv=datacenter1|datacenter2&tagk=host
Does this make sense?
Thanks!
ManOLamancha
I was thinking the same thing for the suggestions, that with this new
metadata it would be much easier to return just data related to the
metrics that you're searching for. Once the metadata is working (read/
writing) I'll extend the suggest endpoint. Thanks!
tsuna
always wanted these kinda features, just never got around to
implementing them.
Although I'm OK with JSON to store metadata in HBase, I'm also
thinking about ProtoBufs, because they have the advantage of having a
well-defined structure, as in once you write your .proto file, it's
pretty obvious what will be in HBase, whereas JSON is just a string
where you can make typos etc.
My other concern, regardless of whether or not JSON or PB is used as a
format, is that of concurrent updates. It sounds like the TSD will
need to lock the row in order to be able to update the metadata. Or
actually… maybe I can just implement support in asynchbase for the
atomic update RPC HBase has ("checkAndPut" or something like that).
Otherwise when you have multiple TSDs trying to update the same piece
of metadata the 2nd write might overwrite the first one.
Also the reason I originally decided not to keep track of associations
between metrics and tag combinations used in the wild is that they are
temporal as Dave pointed out, in the sense that they change as things
come and go. BTW just to correct the terminology a bit:
"proc.stat.cpu" is a metric, "host=foo" is a tag (where "host" is a
tag key, "foo" a tag value") and [proc.stat.cpu host=foo] is a time
series for the metric "proc.stat.cpu". In other words a metric + a
specific set of tags should be called a "time series".
The problem with keeping track of time series exist is that there are
many many many of them. If you want to be able to answer queries like
"what metrics use this tag foo=bar", then you need a simple inverted
index of tags. But if you want to be able to answer queries like
"what metrics use tags foo=bar *and* qux=baz" then you need to store
the combinatorial explosion of all tags and all metrics, which can
easily go into the billions of items. A year ago I looked at our TSD
traffic and found over 200000 unique time series. Virtually all our
datapoints have 3 tags, generally more. This adds up to 1333 billion
possible combinations. Sure they won't all exist and thus won't all
be stored, but the size of the data explodes exponentially, which is
not good. Adding the temporal aspect of the data makes things only
worse.
There is another strategy that can be used to auto-complete forms.
Once you have a metric, a start time and (optionally) an end time, you
can do a short scan to try to discover what kind of tags are used.
E.g. I entered the following in the TSD UI:
- Start: 2001/02/03-00:00:00
- End: 2001/03/04-00:00:00
- Metric: proc.stat.cpu
Then I can do a two short scans of proc.stat.cpu, one right after the
start time, one right before the end time. Assuming I scan just a few
hundred rows, or a few thousand rows at most, it should be possible to
"guess" most of the combinations of tags in that time range.
--
Benoit "tsuna" Sigoure
Software Engineer @ www.StumbleUpon.com
ManOLamancha
> Although I'm OK with JSON to store metadata in HBase, I'm also
> thinking about ProtoBufs, because they have the advantage of having a
> well-defined structure, as in once you write your .proto file, it's
> pretty obvious what will be in HBase, whereas JSON is just a string
> where you can make typos etc.
that a JSON format would be fine without adding extra dependencies on
ProtoBufs. The Jackson library does an awesome job of taking in a Java
class and serializing or deserializing the data so we wouldn't have to
worry too much about typos unless someone starts poking at the HBase
cells directly. And for the HTTP API, it's easy to just have Jackson
spit out the JSON directly to users for their GUIs or tools. Do you
think it would be worth implementing ProtoBufs for possible
interaction or just leave it as JSON?
> My other concern, regardless of whether or not JSON or PB is used as a
> format, is that of concurrent updates. It sounds like the TSD will
> need to lock the row in order to be able to update the metadata. Or
> actually… maybe I can just implement support in asynchbase for the
> atomic update RPC HBase has ("checkAndPut" or something like that).
> Otherwise when you have multiple TSDs trying to update the same piece
> of metadata the 2nd write might overwrite the first one.
metadata should be updated fairly infrequently.
> Also the reason I originally decided not to keep track of associations
> between metrics and tag combinations used in the wild is that they are
> temporal as Dave pointed out, in the sense that they change as things
> come and go. BTW just to correct the terminology a bit:
> "proc.stat.cpu" is a metric, "host=foo" is a tag (where "host" is a
> tag key, "foo" a tag value") and [proc.stat.cpu host=foo] is a time
> series for the metric "proc.stat.cpu". In other words a metric + a
> specific set of tags should be called a "time series".
>
think that a well designed time series definition should be immutable.
e.g. if you have "proc.stat.cpu host=foo", that should be created once
and always refer to that metric on host "foo". If host "foo" dies or
we rename it to "smoo", then we'll have a new time series but the old
one is still relevant for historical purposes. It's all in the tags
and combinations.
> The problem with keeping track of time series exist is that there are
> many many many of them. If you want to be able to answer queries like
> "what metrics use this tag foo=bar", then you need a simple inverted
> index of tags. But if you want to be able to answer queries like
> "what metrics use tags foo=bar *and* qux=baz" then you need to store
> the combinatorial explosion of all tags and all metrics, which can
> easily go into the billions of items. A year ago I looked at our TSD
> traffic and found over 200000 unique time series. Virtually all our
> datapoints have 3 tags, generally more. This adds up to 1333 billion
> possible combinations. Sure they won't all exist and thus won't all
> be stored, but the size of the data explodes exponentially, which is
> not good. Adding the temporal aspect of the data makes things only
> worse.
of them currently in use. Right now, all the meta data is just in
MySQL tables so lookups and searches are pretty quick.
For mapping, I was thinking of including arrays in each "metrics" and
"tagk" meta objects that list all of the relevant associates.
e.g. "metrics":[tagk_uid1, tagk_uid2, tagk_uid3...]
"tagk":[metric_uid1, metric_uid2, metric_uid3...]
This way it's easy to perform a quick lookup in HBase for the proper
tagk or metric and find out what is associated.
Hows that sound? thanks!
tsuna
> I was going to use the row-lock that you're using on the UID rows as
> metadata should be updated fairly infrequently.
Yeah this row-lock business is pretty ugly, I'd like to avoid using it
or spreading the disease as much as possible.
> Gotcha, I'll refer to the metric + tags as a time series now :) I
> think that a well designed time series definition should be immutable.
> e.g. if you have "proc.stat.cpu host=foo", that should be created once
> and always refer to that metric on host "foo". If host "foo" dies or
> we rename it to "smoo", then we'll have a new time series but the old
> one is still relevant for historical purposes. It's all in the tags
> and combinations.
That's exactly how it works. A time series is a unique combination of
a metric and set of tags. If you change a tag in any way, we're now
talking about a new time series.
> For mapping, I was thinking of including arrays in each "metrics" and
> "tagk" meta objects that list all of the relevant associates.
> e.g. "metrics":[tagk_uid1, tagk_uid2, tagk_uid3...]
> "tagk":[metric_uid1, metric_uid2, metric_uid3...]
> This way it's easy to perform a quick lookup in HBase for the proper
> tagk or metric and find out what is associated.
>
> Hows that sound? thanks!
That sounds reasonable.
ManOLamancha
The general meta entries are stored in the "name" CF in the same row as their parent UID. E.g. metric "if.bytes.in" with a UID of 1 will have a GeneralMeta entry in the cell "metrics_meta" in row 1. Data is serialized back and forth in JSON format. There are the default fields that I listed earlier in this thread as well as an array of customizable key/values as requested by another user. I haven't tackled storage of the TimeSeriesMeta yet but I think I'll just put it in the same "name" CF and use the time series hash.
For the mapping above, I was thinking of adding another cell in each row called "metrics_map" or "tagk_map".
I'll also add RPC calls for setting/getting/searching the meta data. Let me know what ya'll think, thanks!
ManOLamancha
Sri Kodali
ManOLamancha
Hello Igor,I was looking for the same queries like using suggest and getting tagv based on other tagv, but the url's which you listed are not working. The query is being done before q only. Can you please help me in order to make it work. Actually my data goes like this:But the drawback here is, I must not mention time because fetching user deviceId's are not based on time & metrics but they are only based on user name. That means I want get deviceId values based on user value. I tried it using "suggest endpoint" by following:and it's not working. Any suggestion will be very helpful for me.