Strong validation on the timestamp when writing data into the TSDB
61 views
Skip to first unread message
spico...@gmail.com
Sep 23, 2015, 8:03:05 AM9/23/15
to OpenTSDB
Hello!
I have some use cases for writing the data into TSDB, that cannot be fulfilled with current version due to the strong validation on the timestamp in the class IncomingDataPoints.java. The condition for always writing new data prevents me to add data from the past.
// always maintain last_ts in milliseconds
if ((ms_timestamp ? timestamp : timestamp * 1000) <= last_ts) {
throw new IllegalArgumentException("New timestamp=" + timestamp
+ " is less than or equal to previous=" + last_ts
+ " when trying to add value=" + Arrays.toString(value)
+ " to " + this);
}
The use cases are the following:
1. I have many sensors data file that contains values for a sensor that they are not ordered by timestamp. This is normal in real life.
2. A client uploads file from with that sensor data from year 2009. After a while,
a new client uploads its file with the same sensors but from year 2008.
We would like to use OpenTSDB for these two scenarios, but the above limitations prevents us to use it.
Is this strong validation a core precondition for storing the data in TSDB? What implications involve if you disable it?
Is OpenTSDB suitable for my use cases?
I have some use cases for writing the data into TSDB, that cannot be fulfilled with current version due to the strong validation on the timestamp in the class IncomingDataPoints.java. The condition for always writing new data prevents me to add data from the past.
// always maintain last_ts in milliseconds
if ((ms_timestamp ? timestamp : timestamp * 1000) <= last_ts) {
throw new IllegalArgumentException("New timestamp=" + timestamp
+ " is less than or equal to previous=" + last_ts
+ " when trying to add value=" + Arrays.toString(value)
+ " to " + this);
}
The use cases are the following:
1. I have many sensors data file that contains values for a sensor that they are not ordered by timestamp. This is normal in real life.
2. A client uploads file from with that sensor data from year 2009. After a while,
a new client uploads its file with the same sensors but from year 2008.
We would like to use OpenTSDB for these two scenarios, but the above limitations prevents us to use it.
Is this strong validation a core precondition for storing the data in TSDB? What implications involve if you disable it?
Is OpenTSDB suitable for my use cases?
Izak Marais
Oct 5, 2015, 4:25:38 AM10/5/15
to OpenTSDB
Hi.
I was under the impression that OpenTSDB is suited to your usecase. Since you provide the timestamp with each PUT, I thought you could provide points in any order, as long as you give the timestamp for each one. I was unaware of this timestamp validation.
Please keep us posted on your results.
Regards
Izak
spico...@gmail.com
Oct 8, 2015, 12:28:40 PM10/8/15
to OpenTSDB
Hi!
The issue was related with the TextImporter class used by the importer tool provided by the OpenTSDB. With the Java API there is no such kind of issue.
Regards,
Florin
Izak Marais
Oct 9, 2015, 1:22:11 AM10/9/15
to OpenTSDB
Aah, I see. I suppose you could always write a small program that uses the Java or http api to git all your data in...
Regards
Izak
ManOLamancha
Nov 10, 2015, 6:34:20 PM11/10/15
to OpenTSDB
On Thursday, October 8, 2015 at 9:28:40 AM UTC-7, spico...@gmail.com wrote:
Hi!The issue was related with the TextImporter class used by the importer tool provided by the OpenTSDB. With the Java API there is no such kind of issue.Regards,Florin
A patch that allows a flag to disable the ordering would be great :)
Reply all
Reply to author
Forward
0 new messages