different values shown for the same datapoint when selecting different time intervals
binbino
timestamp but with different combination of 18 tags ) to openTSDB.
When I query a sum-aggregated time series, I see different results for
the same timestamp when different time intervals are selected.
For example, say I want to get the sum-aggregated value for data
points between 8:50am-9:20am. I tried two queries below, one for
8:21-9:22am, the other one for 8:21-11:22am
1) q?start=2011/06/19-08:21:00&end=2011/06/19-09:22:00&m=sum:data.
4min{tag1=foo,tag2=bar}&ascii
data.4min 1308496320 215600.85399919376 tag1=foo tag2=bar
data.4min 1308496560 241825.8220620532 tag1=foo tag2=bar
data.4min 1308496800 261503.86639386433 tag1=foo tag2=bar
data.4min 1308497040 273473.3156679907 tag1=foo tag2=bar
data.4min 1308497280 283113.62028163625 tag1=foo tag2=bar
2) q?start=2011/06/19-08:21:00&end=2011/06/19-11:22:00&m=sum:data.
4min{tag1=foo,tag2=bar}&ascii
data.4min 1308496320 215600.85399919376 tag1=foo tag2=bar
// all data point values are different from the this point onwards
data.4min 1308496560 244544.01994963607 tag1=foo tag2=bar
data.4min 1308496800 266969.0098736382 tag1=foo tag2=bar
data.4min 1308497040 281889.3523545176 tag1=foo tag2=bar
data.4min 1308497280 294699.71699977806 tag1=foo tag2=bar
Any idea why ?!?
Thanks
-Eric
tsuna
> Every 4 min, I dump millions of data points (all with the same
> timestamp but with different combination of 18 tags ) to openTSDB.
> When I query a sum-aggregated time series, I see different results for
> the same timestamp when different time intervals are selected.
>
> For example, say I want to get the sum-aggregated value for data
> points between 8:50am-9:20am. I tried two queries below, one for
> 8:21-9:22am, the other one for 8:21-11:22am
Can you confirm nothing else is writing to TSDB while you make the query?
Can you post the few lines of log that surround each query?
Are you able to consistently reproduce the problem? Does the output
vary over time, or is it always the same based on the start/end dates?
--
Benoit "tsuna" Sigoure
Software Engineer @ www.StumbleUpon.com
binbino
datapoints every 4 min. I found that the problem is that not all time-
series has a data point every 4 min so as I select a larger time span,
more time-series fall within the span and are interpolated when their
datapoints go missing for a given 4-min granularity timestamp. The
behavior I want is that if a datapoint for a time-series at a given
time is missing, then ignore it when computing the aggregate.
For example, suppose I have the following 3 time series that match a
given tag between time=0~8 minutes
at time: [0, 4, 8], ts1=[0, 2, 3], ts2=[1, missing, 2], ts3=[5, 6,
missing]
What I am looking for is an aggregated time series=[6, 8, 5], but
openTSDB gives me [6, 9, 5] since ts2's value at time=4min is
interpolated as 1 despite its value is missing.
Is there any way to configure openTSDB to have the behavior I desire
when I am sending tightly-controlled equally-paced time-series? or do
I have to explicitly set ts2's value to be 0 at time=4min in this
case, in order to exclude its effect on the overall aggregate?
Thanks
On Jun 19, 6:29 pm, tsuna <tsuna...@gmail.com> wrote:
tsuna
> I have a single collector running that sends tens of thousands of
> datapoints every 4 min. I found that the problem is that not all time-
> series has a data point every 4 min so as I select a larger time span,
> more time-series fall within the span and are interpolated when their
> datapoints go missing for a given 4-min granularity timestamp. The
Is it "missing" because its value hasn't changed, or is because its
value should be 0?
> behavior I want is that if a datapoint for a time-series at a given
> time is missing, then ignore it when computing the aggregate.
>
> For example, suppose I have the following 3 time series that match a
> given tag between time=0~8 minutes
>
> at time: [0, 4, 8], ts1=[0, 2, 3], ts2=[1, missing, 2], ts3=[5, 6,
> missing]
>
> What I am looking for is an aggregated time series=[6, 8, 5], but
> openTSDB gives me [6, 9, 5] since ts2's value at time=4min is
> interpolated as 1 despite its value is missing.
There is no way for OpenTSDB to tell whether a value is missing or
not. So in the case of ts2, it assumes that the value is 1 until it
changes to become 2. If you want the missing value to be 0, then
store a 0 at that point.
> Is there any way to configure openTSDB to have the behavior I desire
> when I am sending tightly-controlled equally-paced time-series? or do
No, because OpenTSDB doesn't know about your fixed interval
convention, and there is no way to tell it about it, because there are
no such conventions in OpenTSDB.
> I have to explicitly set ts2's value to be 0 at time=4min in this
> case, in order to exclude its effect on the overall aggregate?
If your goal is to get the sum, then yes you have to store a 0 to mean
that something wasn't present at a given point in time.
It's simple: if you want the value to be 0, store a 0. Easy, heh? :)
Arvind Jayaprakash
>There is no way for OpenTSDB to tell whether a value is missing or
>not. So in the case of ts2, it assumes that the value is 1 until it
>changes to become 2. If you want the missing value to be 0, then
>store a 0 at that point.
>
>that something wasn't present at a given point in time.
>
>It's simple: if you want the value to be 0, store a 0. Easy, heh? :)
I see both sides of the argument and I have some mixed views. But before
that, let us start off with a simple data set
time t0, series m1 -> 12
time t0+ 3, series m2 -> 90
time t0+ 3, series m1 -> 4
time t0+ 8, series m2 -> 100
time t0+ 9, series m1 -> 10
If I query for [t0, t0+10],m1 I get the following:
time t0, series m1 -> 12
time t0+ 3, series m1 -> 4
time t0+ 9, series m1 -> 10
If I query for [t0, t0+10],m2 I get the following:
time t0+ 3, series m2 -> 90
time t0+ 8, series m2 -> 100
When this is passed on to a line series graph, the "interpolation" for
the missing points comes for free. At the same time, it also clear that
there were only 3 measured points.
If on the other hand anyone wishes to explicility mark the missing
points as zero, they are free to do as in intermediate step b/w
retreiving the data series and graphing. Doing it this way turns out to
be way more inexpensive that inserting zeroes right into the data store.
I say this because the original thread spoke of 18 tags and I'd assume
most series are sparse, so it is best to "back fill" the zeroes as a
post-processing step in querying.
And now onto what has completely driving me nuts...
If I query for [t0, t0+10], (m1+m2) I get the following:
time t0, -> 12 (m1)
time t0+ 3, -> 90 (m1) + 4 (m2)
time t0+ 8, -> 100 (m2) + x (some interpolated value of m1)
time t0+ 9, -> 10
The value for t0+8 is where my grief comes from. The current semantics
when combiling multiple series seems to be one of interpolating missing
points for overlapping ranges across series and then summing it up.
What I'd rather have is points from all series being "merged" first by
the selected aggregate function and then just return that series. i.e.
something that gives the following o/p
time t0, -> 12
time t0+ 3, -> 90+4
time t0+ 8, -> 100
time t0+ 9, -> 10
In short, the interpolation magic "hurts" in some cases only when
multiple series are merged. How do we make this configurable?
tsuna
> If on the other hand anyone wishes to explicility mark the missing
> points as zero, they are free to do as in intermediate step b/w
> retreiving the data series and graphing. Doing it this way turns out to
> be way more inexpensive that inserting zeroes right into the data store.
Storing data points isn't expensive. But my main concern at this
point isn't efficiency, it's correctness.
> And now onto what has completely driving me nuts...
>
> If I query for [t0, t0+10], (m1+m2) I get the following:
>
> time t0, -> 12 (m1)
> time t0+ 3, -> 90 (m1) + 4 (m2)
> time t0+ 8, -> 100 (m2) + x (some interpolated value of m1)
> time t0+ 9, -> 10
>
> The value for t0+8 is where my grief comes from. The current semantics
> when combiling multiple series seems to be one of interpolating missing
> points for overlapping ranges across series and then summing it up.
Just to be precise: it's not "interpolating _missing_ points", because
there is no such notion as a "missing" point. It's interpolating
because you're summing up two time series. Think of the time series
as discrete mathematical functions that are approximating continuous
functions. When you sum up two such discrete functions, if one of
them doesn't have a data point defined for the same input value as the
other, the only reasonable thing to do is to interpolate the other
function.
> What I'd rather have is points from all series being "merged" first by
> the selected aggregate function and then just return that series. i.e.
> something that gives the following o/p
>
> time t0, -> 12
> time t0+ 3, -> 90+4
> time t0+ 8, -> 100
> time t0+ 9, -> 10
>
>
> In short, the interpolation magic "hurts" in some cases only when
> multiple series are merged. How do we make this configurable?
I'm sorry but I really don't understand how the behavior your describe
can be correct. It doesn't seem mathematically sound to me. And it
seems it only works as expected when the aggregation function is
"sum". If you do an average instead, then all the zeroes that are
inserted will drive the average down.
In order to explain why lerp is needed, I created this page:
http://tsunanet.net/~tsuna/opentsdb/misc/lerp-or-not.html
It visually shows why lerp is needed in your example case above. It
uses different values for the 2nd time series in order to make the
problem more obvious.
Please help me understand why you think the behavior you describe is
correct if you're not convinced by my counter-example above.
Arvind Jayaprakash
>On Sat, Jul 9, 2011 at 8:17 AM, Arvind Jayaprakash <wo...@anomalizer.net> wrote:
>> If on the other hand anyone wishes to explicility mark the missing
>> points as zero, they are free to do as in intermediate step b/w
>> retreiving the data series and graphing. Doing it this way turns out to
>> be way more inexpensive that inserting zeroes right into the data store.
>
>Storing data points isn't expensive. But my main concern at this
>point isn't efficiency, it's correctness.
It is expensive to generate an exhausitive set of zeros. In my app,
the number of theoretical series possible is in millions but for the
most part, we get data points for atmost ~50K series. Even these 50K
series happen to be temporally sparse.
>I'm sorry but I really don't understand how the behavior your describe
>can be correct. It doesn't seem mathematically sound to me. And it
>seems it only works as expected when the aggregation function is
>"sum". If you do an average instead, then all the zeroes that are
>inserted will drive the average down.
Agreed.
>In order to explain why lerp is needed, I created this page:
>http://tsunanet.net/~tsuna/opentsdb/misc/lerp-or-not.html
>
>It visually shows why lerp is needed in your example case above. It
>uses different values for the 2nd time series in order to make the
>problem more obvious.
>
>Please help me understand why you think the behavior you describe is
>correct if you're not convinced by my counter-example above.
Well, there is a place for both interpretations.
Either we can consider the data points being recorded as sampling of a
continuous series, or complete set of discrete points. Much of tsdb has
been built on the former model and I believe that it is *correct* for
the apps that it was developed (system metrics like CPU/disk usage).
In visual terms, the former would be an area (polygonal) graph whereas
the latter would be a scatter plot. Clearly, the way to stack up an area
graph v/s a scatter plot is different.
All of this problem arises because folks like me are trying to use tsdb
to store discrete points.
In short, I do not think the current implementation is a bad thing; just
that it would help to have this aspect brought out clearly in the
official documentation. And for the nutcases like me who have apps that
need non-interpolating series merging/stacking, there is a
"zero-if-missing" sum available:
https://github.com/InMobi/opentsdb/commit/dafd5ee82c6104d7aaa41a9cca7e0c10d37bc84a
tsuna
> In visual terms, the former would be an area (polygonal) graph whereas
> the latter would be a scatter plot. Clearly, the way to stack up an area
> graph v/s a scatter plot is different.
>
> All of this problem arises because folks like me are trying to use tsdb
> to store discrete points.
At this point it would probably help if you could explain what problem
you're trying to solve with OpenTSDB. My understanding is that you're
trying to track impressions or clicks, so you can graph them, slice
and dice them, but I'd like to know as much as possible about the
problem you need to solve so we can work out if there's something more
reasonable that can be done in OpenTSDB.
I have a sneaking suspicion that you're storing independent
observations per time slice (e.g. "I've seen N clicks during this time
interval") instead of storing a monotonically increasing counter (e.g.
"up until now, I've seen M clicks so far" and have the TSD compute the
rate of change). It seems to me that if my guesswork about your
problem and approach is correct, then you wouldn't have this problem
with a monotonically increasing counter. Arguably, OpenTSDB doesn't
help you much with distributed counting problems, and it could make
counting things much much easier if it provided a nice counting API on
top of HBase's counting API. Am I right?
Arvind Jayaprakash
>you're trying to solve with OpenTSDB. My understanding is that you're
>trying to track impressions or clicks, so you can graph them, slice
>and dice them, but I'd like to know as much as possible about the
>problem you need to solve so we can work out if there's something more
>reasonable that can be done in OpenTSDB.
>
>I have a sneaking suspicion that you're storing independent
>observations per time slice (e.g. "I've seen N clicks during this time
>interval") instead of storing a monotonically increasing counter (e.g.
>"up until now, I've seen M clicks so far" and have the TSD compute the
>rate of change). It seems to me that if my guesswork about your
>problem and approach is correct, then you wouldn't have this problem
>with a monotonically increasing counter. Arguably, OpenTSDB doesn't
>help you much with distributed counting problems, and it could make
>counting things much much easier if it provided a nice counting API on
>top of HBase's counting API. Am I right?
Why I use tsdb and not hbase directly?
======================================
Here is my graphical story: http://anomalizer.net/tmp/tsdb-zimsum/
I like the slicing & dicing i.e. ability to query by tag/values.
Why do I prefer storing observations over counters?
===================================================
Robust data capturing. Log data colected from serves is micro-batched
(currently one batch per minute), aggregated and inserted into tsdb.
What no one ever gets to see is one of my secret tags contains the
batch-id agains every data point. So let us say there are 100K raw
points in each batch and post aggregation, there are 1K points. These 1K
points hit tsdb. If for some reason, the upload to tsdb stalls or there
any other partial failure in the pipeline, I get to re-insert the entire
batch rather than worry which specific points failed. This can be
further generalized to say that say if 23 out of 25 batches failed, I
can blindly reload all the 25 batches.
The whole notion of "when in doubt, reprocess/reinsert liberally" turns
out to be a very elegant operational solution and also the design is
simple enough that there isn't any magic needed to provide the
reslience i.e. the system runs on "at least once" semantics. This is
usually simpler than any design that relies on "exactly once" semantics.
Back to the original problem, there is no traffic/activity in every market
segment (modeled as a different tsdb series via tags) at all times. I
got into trouble when general higher order summaries since
traffic/clicks/revenues were getting interpolated where it was actually
absent.
tsuna
> Here is my graphical story: http://anomalizer.net/tmp/tsdb-zimsum/
OK, that makes sense.
> The whole notion of "when in doubt, reprocess/reinsert liberally" turns
> out to be a very elegant operational solution and also the design is
> simple enough that there isn't any magic needed to provide the
> reslience i.e. the system runs on "at least once" semantics. This is
> usually simpler than any design that relies on "exactly once" semantics.
Agreed.
> Back to the original problem, there is no traffic/activity in every market
> segment (modeled as a different tsdb series via tags) at all times. I
> got into trouble when general higher order summaries since
> traffic/clicks/revenues were getting interpolated where it was actually
> absent.
So the way I see it, there are at least 3 solutions to this problem:
1. Tell the system about fixed-interval time series, assume that a
missing interval has value 0.
2. Use counters and change the way you handle errors.
3. Keep doing what you're doing but insert extra 0s.
Personally I'm very much against option 1, because it's the one that
goes the most against OpenTSDB's design. Nothing in the code assumes
fixed intervals.
I find option 2 very reasonable. That's what Twitter does in Rainbird
and Facebook in Puma, their analytics systems. These systems are
solving the very same problem you seem to have (collecting impression
and click streams). It seems that what they both have in common is
that they use counters that roll-up at different time intervals, and
whenever there is a problem, they simply discard the counters in the
time period affected and re-play the logs to re-create the counters
correctly.
Option 3 is probably the easiest to implement given your current
system. It's not that hard, it only requires that the collector
computes the set difference post-aggregation of what time series have
been seen in the new batch but not in the old batch, to add the
necessary 0s. You can even implement this in a "dumb" way by doing
(Python code):
timestamp = time.time()
batch = []
while True:
timestamp = timestamp + 60 # if you have 1m batches
for timeseries in batch:
put(timeseries, timestamp, 0) # Store a 0 for each time series
seen in the previous batch
batch = get_new_batch()
for timeseries in batch:
put(timeseries, timestamp, timeseries.value)
If a time series has a value both in the previous batch and in the
current batch, the 2nd call to `put' will overwrite the 0 written by
the 1st one. Of course this doubles your write load on HBase, but
it's still an option.