Millisecond timestamp.

[Conrad]

Requesting guidance with millisecond timestamps
in OpenTSDB.

What would be the best approach?

1. An extra tag storing millliseconds

2. customize the code to store/retrieve timestamp
as a 6 or 8 byte number (4 bytes normal timestamp + extra
2/4 bytes for the millisecond).

3. A hybrid approach: milliseconds treated as a tag until write/read
from database where it is stored as 6 or 8 byte timestamp

Any guidance would be greatly appreciated.

Thanks

Conrad

[tsuna]

The 2nd approach would be best. I'd keep the base time in the row key
on 4 bytes (so in seconds) and simply make the per-column offset be in
milliseconds. If you keep 1h rows, you need to support a delta of up
to 3600000ms, so you need at least 22 bits, so you need 3 bytes
instead of 2 for the column qualifier.

You'd also need to change a whole bunch of places in the code where
timestamps are stored in 32-bit integers (despite Java not supporting
unsigned integer arithmetic, which is really f***ing annoying – so
these are up-casted to 64-bit integers where needed).

--
Benoit "tsuna" Sigoure
Software Engineer @ www.StumbleUpon.com

[Mick]

I too want to store millisecond timestamps. Any chance that the base
code might be extended to support millisecond timestamps in the near
future. I would hate to fork the code and then be forced to merge in
changes from future releases.

Also, what is the disadvantage of 1 above? I like that I'm not
required to modify a lot of code and it uses an inherent, supported
OpenTSDB feature.

Thanks. Mick

On Apr 13, 12:05 am, tsuna <tsuna...@gmail.com> wrote:

[tsuna]

On Mon, Apr 16, 2012 at 10:51 AM, Mick <mick.hi...@gmail.com> wrote:
> I too want to store millisecond timestamps. Any chance that the base
> code might be extended to support millisecond timestamps in the near
> future. I would hate to fork the code and then be forced to merge in
> changes from future releases.
>
> Also, what is the disadvantage of 1 above? I like that I'm not
> required to modify a lot of code and it uses an inherent, supported
> OpenTSDB feature.

This is a frequent feature request, so it sounds like one way or
another it's gonna happen. Would probably have to be included in
OpenTSDB 2.0 if there are API changes needed. If someone wants to
start writing some code to make it work, please be my guest.

[Kristoffer Sjögren]

On Apr 16, 8:41 pm, tsuna <tsuna...@gmail.com> wrote:
> This is a frequent feature request, so it sounds like one way or
> another it's gonna happen.  Would probably have to be included in
> OpenTSDB 2.0 if there are API changes needed.  If someone wants to
> start writing some code to make it work, please be my guest.

Not sure if anyone started this yet, but I will do a fork and start
working on millisecond support now. I need it to monitor fine-grained
financial data.

After taking a quick look at the core, it seems like your advice
expanding the column qualifier wont break the API? Old data would
probably also remain backward compatible, as it seems.

Is there anything that would prevent you from pulling this change in
the near future?

[Mick]

I now have a working build of OpenTSDB that supports millisecond
timestamps. I have not tested it extensively so I'm sure there are
still bugs. I have verified that the import, query and scan cli
utilities work. Below is the output of my modified scan program
(small subset):

0, 0, 1, 79, -78, -103, -96, 0, 0, 1, 0, 0, 1] UNDERLYING_PRICE_ASK
1337104800000 (15 May 2012 13:00:00:000 -0500) {symbol=SP}
[0, 83, -55, 75] [68, -85, 72, 0] 343188 f 1370.25 1337105143188 (15
May 2012 13:05:43:188 -0500)
[0, 0, 1, 79, -78, -4, 16, 0, 0, 1, 0, 0, 1] UNDERLYING_PRICE_ASK
1337130000000 (15 May 2012 20:00:00:000 -0500) {symbol=SP}
[0, 77, 7, -37] [68, -85, 32, 0] 315517 f 1369.0 1337130315517 (15
May 2012 20:05:15:517 -0500)
[0, 79, 83, -21] [68, -85, 0, 0] 324926 f 1368.0 1337130324926 (15
May 2012 20:05:24:926 -0500)

As you can see, I am using it to store quotes with milliscond
timestamps.

The gnu plot http application also works but charts still show
DD:MM:SS on the x axis. I don't have a need to chart my quotes every
millisecond nor do I plan to really use the charting functionality in
my apps. I just need to be able to capture and query large volumes of
financial time series data.

I also ripped out the autotools build and replaced it with a maven
build :(. Sorry Benoit, but I didn't have the time to wrap my head
around all those autotools scripts! Maven is just so much easier... :)

I'd be happy to post the code to a branch or something if anyone is
interested.

Regards, Mick

[Eric Newton]

On Wed, May 16, 2012 at 3:18 PM, Mick <mick.hi...@gmail.com> wrote:

I now have a working build of OpenTSDB that supports millisecond
timestamps.

Neat.

 

I also ripped out the autotools build and replaced it with a maven
build :(. Sorry Benoit, but I didn't have the time to wrap my head
around all those autotools scripts! Maven is just so much easier... :)

Yes! 

 

I'd be happy to post the code to a branch or something if anyone is
interested.

I would love to get the changes for the maven build.  Did you end up moving the sources into the standard maven locations?

-Eric

[Mick Hittesdorf]

Yes. The source tree and package structure now conforms to maven conventions:

src/main/java/net/opentsdb
src/main/java/net/opentsdb/core
src/main/java/net/opentsdb/graph
src/main/java/net/opentsdb/stats
src/main/java/net/opentsdb/tools
src/main/java/net/opentsdb/tsd
src/main/java/net/opentsdb/tsd/client
src/main/resources
src/main/bin
src/main/assembly
pom.xml

I'm happy to share it with you but I'd like Benoit to own this going forward if possible. Let me see if he is interested in taking my changes before I start distributing a forked copy.

Mick

[Kristoffer Sjögren]

Hi Mick

Great news. I would be very interested to review the changes you did.

Do you mind posting the code anyways?

Cheers,
-Kristoffer

[Kristoffer Sjögren]

I just pushed a few changes on a millisecond fork on
https://github.com/deephacks/opentsdb/tree/millis.

These changes enable the TSDB gui to read millisecond data points.
Gnuplot does not support milliseconds out-of-the-box (as far as I can
tell), so the best way seems to keep plotting data points in the graph
as seconds. I have successfully injected around five million data
points (ms) and generated a few graphs using different timespans and
aggregators.

I still have a couple more iterations to go.

[Mick]

My forked repository can be found here:

https://mhitt...@github.com/mhittesdorf/opentsdb.git

Mick

[Derrick Burns]

Mick,

Are you using this fork in production now with millisecond values?  

Have you also changed the metric size to be larger than 3 bytes?  Will simply changing the value in TSDB.java to 4 be sufficient?

On Tuesday, May 22, 2012 4:04:13 PM UTC-7, Mick wrote:

My forked repository can be found here:

https://mhittesdorf@github.com/mhittesdorf/opentsdb.git

[Mick]

You can diff the code to see specifically what changes I've made. Conceptually, the delta from the base time stamp needs to be large enough to accommodate 3600000 milliseconds since the base time stamp is on a one hour boundary.  The devil is in the details though as the code relies heavily on bit level representation of data, uses lots of bit shifting and bit masks, etc.  I made no changes to any other aspects of the OpenTSDB schema.

I am using my code in a pilot production mode right now to capture trade and market data metrics, including raw ticks.

[David López]

Hi Mick,
thank you for the millisecond support!
I am using your forked repository to make some sensor data storing test, but I am afraid the http query command does not function properly or I do not understand well the results I am obtaining.
I will try to explain it with an example:

1 - I cloned the repository and build it with MAVEN (everything went ok)

2 - I imported into the TSD the following test data:

test.accx 1343900991056 345 node=testNode
test.accy 1343900991056 342 node=testNode
test.accz 1343900991056 341 node=testNode

test.accx 1343900991195 342 node=testNode
test.accy 1343900991195 339 node=testNode
test.accz 1343900991195 337 node=testNode

test.accx 1343900991334 339 node=testNode
test.accy 1343900991334 337 node=testNode
test.accz 1343900991334 335 node=testNode

test.accx 1343900991477 336 node=testNode
test.accy 1343900991477 334 node=testNode
test.accz 1343900991477 332 node=testNode

test.accx 1343900991617 334 node=testNode
test.accy 1343900991617 332 node=testNode
test.accz 1343900991617 330 node=testNode

test.accx 1343900991756 332 node=testNode
test.accy 1343900991756 330 node=testNode
test.accz 1343900991756 328 node=testNode

test.accx 1343900991895 330 node=testNode
test.accy 1343900991895 328 node=testNode
test.accz 1343900991895 326 node=testNode

3 - Then I tried to do some querys with different start times and get unexpected results:

If I query the database for the same timestamp the data was stored I get the following:

   1343900990 = 2012-08-02 09:49:50 

   http://localhost:4242/q?start=1343900991&m=sum:test.accx&json 

   {"plotted":0,"points":0,"etags":[[]],"timing":70}

No data points found... :/

If I query for the first second since epoch I see the datapoints were stored correctly:

   http://localhost:4242/q?start=0&m=sum:test.accx&json
   {"plotted":7,"points":7,"etags":[[]],"timing":62}

And now some more querys and their "unexpected" results:
 

   1343779200 = 2012-08-01 00:00:00
   http://localhost:4242/q?start=1343779200&m=sum:test.accx&json
   {"plotted":0,"points":0,"etags":[[]],"timing":58}

   890409328 = 1998-03-20 15:55:28
   http://localhost:4242/q?start=890409328&m=sum:test.accx&json
   {"plotted":7,"points":7,"etags":[[]],"timing":64}

   890409329 = 1998-03-20 15:55:29
   http://localhost:4242/q?start=890409329&m=sum:test.accx&json
   {"plotted":0,"points":0,"etags":[[]],"timing":59}

   700086870 = 1992-03-08 20:34:30
   http://localhost:4242/q?start=700086870&m=sum:test.accx&json
   {"plotted":7,"points":7,"etags":[[]],"timing":64}

   700086869 = 1992-03-08 20:34:29
   http://localhost:4242/q?start=700086869&m=sum:test.accx&json
   {"plotted":0,"points":0,"etags":[[]],"timing":70}

As you can see the start timestamp does not seem to work as expected. Am I doing something wrong?

Sorry for the long post and thank you very much in advance!

Regards,
David López

El miércoles, 23 de mayo de 2012 01:04:13 UTC+2, Mick escribió:

My forked repository can be found here:

https://mhittesdorf@github.com/mhittesdorf/opentsdb.git 

[Sidhant Gupta]

David,

Did you resolve this issue? 

I have also been looking at using OpenTSDB for collecting sensor data sampled at 500Hz, and thus the need for millisecond support. Indeed thanks to Mick for doing the heavy lifting here! Before I make up my mind on using OpenTSDB, I would love to get a sense of reliability of millisecond support.

Sidhant

[David López]

Hi Sidhant,

I have made many test and discovered that, when using much more data (not just data within the same second as I first tried), the querys for png and json return valid results. However, the query for data in "ascii" format for a specific timestamp seems to return all datapoints stored in HBase, not just those with a timestamp greater or equal to the specified... so, I'm still trying to findout if there is a problem with the querys or I am using them in the wrong way :/

That is what I can tell you so far, but I think you should give it a try anyway, because millisecond support seems to be working properly, the data is correctly stored in HBase at the right timestamp, where I am having trouble is at query level, and you may not encounter problems for your particular case.

Hope I helped a bit ;)

Regards,

David López

[Sidhant Gupta]

Thanks David!

I will give it a try. I was most interested in knowing if you have had success in using OpenTSDB in your application, since that seemed in line with my sensor data storage needs.

Sidhant

[Sambit Tripathy]

Great news David.
I hope down sampling works correctly the way it worked for seconds based timestamps. Did you get a chance to test that as well?

-Sambit Tripathy

[David López]

Hi Sambit,

I am afraid I have not tested the down sampling feature explicitly.

I am on holidays now, so I can not test it,  but I am pretty sure it should work :)

Regards,

David López

[Mick]

I implemented the &exact query parameter for ascii queries to address the fact that OpenTSDB by default returns data two hours before and after the query start and end date. If you only have a few minutes of data in your database your query may appear to return all of the data in the database. To force OpenTSDB to return data on the specific boundaries defined by the start and end parameters of your query, you can add &exact to the end of the http query string.   

[Sambit Tripathy]

Mick, can you please share the git repository with me. I tried building from https://github.com/mhittesdorf/opentsdb.git but whenever I tried to start a tsd I got this following error


-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy]
10:52:46,224 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
10:52:46,224 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [jar:file:/root/misc_tools/deploy/opentsdb-1.1.1-SNAPSHOT/lib/opentsdb-1.1.1-SNAPSHOT.jar!/logback.xml]
10:52:46,226 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs multiple times on the classpath.
10:52:46,226 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [jar:file:/root/misc_tools/deploy/opentsdb-1.1.1-SNAPSHOT/lib/opentsdb-1.1.1-SNAPSHOT.jar!/logback.xml]
10:52:46,226 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [file:/root/misc_tools/deploy/opentsdb-1.1.1-SNAPSHOT/logback.xml]
10:52:46,253 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList@596e1fb1 - URL [jar:file:/root/misc_tools/deploy/opentsdb-1.1.1-SNAPSHOT/lib/opentsdb-1.1.1-SNAPSHOT.jar!/logback.xml] is not of type file
10:52:46,291 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set
10:52:46,295 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
10:52:46,298 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [STDOUT]
10:52:46,317 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
10:52:46,361 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.read.CyclicBufferAppender]
10:52:46,361 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [CYCLIC]
10:52:46,366 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.apache.zookeeper] to INFO
10:52:46,366 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.hbase.async] to INFO
10:52:46,366 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [com.stumbleupon.async] to INFO
10:52:46,366 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO
10:52:46,366 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[ROOT]
10:52:46,366 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [CYCLIC] to Logger[ROOT]
10:52:46,368 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@4ce2cb55 - Registering current configuration as safe fallback point

2013-03-12 10:52:46,373 INFO  [main] TSDMain: Starting.
2013-03-12 10:52:46,376 INFO  [main] TSDMain: net.opentsdb built at revision 229c3c4 (MINT)
2013-03-12 10:52:46,376 INFO  [main] TSDMain: Built on 2013/03/12 05:17:47 +0000 by root@localhost:/root/misc_tools/opentsdb_milli/opentsdb
Exception in thread "main" java.lang.RuntimeException: Initialization failed
        at net.opentsdb.tools.TSDMain.main(TSDMain.java:148)
Caused by: java.lang.ExceptionInInitializerError
        at org.hbase.async.HBaseClient.getRegion(HBaseClient.java:1309)
        at org.hbase.async.HBaseClient.sendRpcToRegion(HBaseClient.java:1027)
        at org.hbase.async.HBaseClient.get(HBaseClient.java:755)
        at org.hbase.async.HBaseClient.ensureTableFamilyExists(HBaseClient.java:708)
        at org.hbase.async.HBaseClient.ensureTableExists(HBaseClient.java:728)
        at net.opentsdb.tools.TSDMain.main(TSDMain.java:124)
Caused by: java.lang.RuntimeException: static initializer failed
        at org.hbase.async.Bytes.<clinit>(Bytes.java:387)
        ... 6 more
Caused by: java.lang.NoSuchFieldException: buffer
        at java.lang.Class.getDeclaredField(Class.java:1899)
        at org.hbase.async.Bytes.<clinit>(Bytes.java:384)
        ... 6 more


Thanks
Sambit.

[tsuna]

On Mon, Mar 11, 2013 at 10:33 PM, Sambit Tripathy <samb...@gmail.com> wrote:
> Mick, can you please share the git repository with me. I tried building from
> https://github.com/mhittesdorf/opentsdb.git but whenever I tried to start a
> tsd I got this following error

[snip]

> Caused by: java.lang.NoSuchFieldException: buffer
> at java.lang.Class.getDeclaredField(Class.java:1899)
> at org.hbase.async.Bytes.<clinit>(Bytes.java:384)
> ... 6 more

This is caused by an incompatibility between asynchbase and Netty.
Looks like you're using some really old code. Try rebasing his
changes against the current HEAD of master.

--
Benoit "tsuna" Sigoure