Strange OpenTSDB Overwriting Behaviour

[Kenneth Sinder]

Hi,

For a particular use case of OpenTSDB I need to store a long (2+ year) timeseries record of some stats. I am relying on OpenTSDB's ability to overwrite values at the same timestamp to update the value, but I am encountering strange behaviour.
Here is a snippet of an OpenTSDB /api/query response:

"dps": {
  "1469015100": 901,
  "1469016000": 900,
  "1469016900": 900,
  "1469017800": 900,
  "1469018700": 111,
  "1469019600": 50,
  "1469020500": 900,
  "1469021400": 900,
  "1469022300": 50,
  "1469023200": 900,
  "1469024100": 900,
  "1469025000": 900,
  "1469025900": 50,
  "1469026800": 900,
  "1469027700": 900,
  "1469028600": 50
}

I have a script running which creates files that are then imported using tsdb import <filename>. These files are about 8MB in size and are deleted every time the script runs to avoid polluting the file system with junk.
One file might contain:
...
<metric_name> 1469022300 50 <rest of the tag keys and values>
<metric_name> 1469021400 900 <rest of the tag keys and values>
...

And, in 15 minutes, the corresponding lines in the next file would be:
<metric_name> 1469023200 52 <rest of the tag keys and values>
<metric_name> 1469022300 900 <rest of the tag keys and values>

The problem is, that point that ends in 2300 is clearly not updated, and you can see it's still at a value of 50 from the API response above. This usually happens around :45 of every hour, perhaps when OpenTSDB's cache is persisted? For reference, I have the following lines in my /etc/opentsdb.conf:
tsd.storage.enable_compaction = false
tsd.storage.fix_duplicates = true

There is no indication in the output of the TSDB import or any logs, or anywhere else, that a problem occurred when (trying to) overwriting the duplicate point.

Can anyone shed some light on why this is happening, and whether it might be possible to fix it so that the last write wins for different values saved to the same timestamp? I tried reading the documentation and didn't find anything useful there either.

Thanks!
-Kenneth

[Izak Marais]

Any progress on you problem? I was under the impression that with your config settings it will be last write wins...

[Kenneth Sinder]

I was able to circumvent it by importing files with timestamps in nondescending order as you go down the file line-by-line. It's weird, but OpenTSDB didn't seem to like the interlaced method. What I had before was:

<metric> <timestamp 300> <value1> <tags1>

<metric> <timestamp 600> <value1'> <tags1>

<metric> <timestamp 300> <value2> <tags2>

<metric> <timestamp 600> <value 2'> <tags2>

So by sorting it so we had the timestamps ending in 300 first in the file, and then the timestamps ending in 600, there were no longer any issues retrieving the last write from the TSDB using the HTTP api. Weird.

[ManOLamancha]

On Tuesday, August 16, 2016 at 3:55:54 AM UTC-7, Kenneth Sinder wrote:

I was able to circumvent it by importing files with timestamps in nondescending order as you go down the file line-by-line. It's weird, but OpenTSDB didn't seem to like the interlaced method. What I had before was:

Right, there was an issue with the bulk loader where-in it tracks the timestamp order and rejects timestamps lower or equal to the one before for the same series. Your previous setup *should* have worked though you had ts1 @t1, ts2 @t1, ts1 @t2, ts2 @t2, etc. In 2.4 we have the ability to not care about the timestamp order.