Now this would reduce the number of queries by a factor of 100...down to 200 (from 20,000) but each query will return data for each host...so each query will return 100 times more data. The amount of data returned will still be the same ~80 million points. The real problem is that each query is returning way too much data.
The real way to improve performance in this scenario would be to return 1 point for each query...so 20,000 data points and not 80 million.
Does anyone have ideas on how I could add functionality to support a new query to OpenTSDB. I understand this may require adding/modifying code on OpenTSDB to handle another type of query to basically return the most recent data point. Could anyone point me in the right direction or has anyone made query code changes?
Does anyone have ideas on how I could add functionality to support a new query to OpenTSDB. I understand this may require adding/modifying code on OpenTSDB to handle another type of query to basically return the most recent data point. Could anyone point me in the right direction or has anyone made query code changes?
A member of my team actually has the same requirement, e.g. the ability to quickly get the last data point, so I've some code I'm cooking up that should be ready later this month for testing that will provide an optimization. It'll be based off the 2.0 branch though.
A member of my team actually has the same requirement, e.g. the ability to quickly get the last data point, so I've some code I'm cooking up that should be ready later this month for testing that will provide an optimization. It'll be based off the 2.0 branch though.
A member of my team actually has the same requirement, e.g. the ability to quickly get the last data point, so I've some code I'm cooking up that should be ready later this month for testing that will provide an optimization. It'll be based off the 2.0 branch though.
This would be really awesome and exactly what I'm looking for. Is there any way you could port your code over to version 1.1.0? This would really make a tremendous difference in performance for us, and make things a lot cleaner than some other workaround solutions!
A member of my team actually has the same requirement, e.g. the ability to quickly get the last data point, so I've some code I'm cooking up that should be ready later this month for testing that will provide an optimization. It'll be based off the 2.0 branch though.
Which packages will need to be updated besides OpenTSDB? Any new package dependencies that weren't there before? Any changes on sending data to OpenTSDB...will TCollector need to be updated? Are queries to get data the same?
Sorry for all the questions. I haven't had a chance to look into what's in 2.0 and what's required for the upgrade. Sorry for the questions.
A member of my team actually has the same requirement, e.g. the ability to quickly get the last data point, so I've some code I'm cooking up that should be ready later this month for testing that will provide an optimization. It'll be based off the 2.0 branch though.
Just to check, would this allow getting the entire last data point for a particular metric along with associated timestamp and tags? If so great! Can't wait.
Well I could make it do aggregation for all time series belonging to a metric if you like. I was just going to have a query like "/api/query/last?tsuids=010101,020202,030303"
Wouldn't the query itself define the aggregation? For example if you did a query for a metric with sum and no host tag...would mean to aggregate all the data from the different hosts.
I'm not sure I understand why the "last?tsuids=..." is needed?
Ok this would do what I need. I don't really need the aggregation, I was just asking to learn about the enhancement you are making. Why would be need the TSUIDS? A normal query doesn't require that.
I also have a question about the additional jars that will be needed:
1. In our environment, we don't have access to internet. We will need to package those jars and provide them as part of installation. Is there an offline installation method to do this?
2. The license of the jars is of concern. We can't redistribute jars if the license doesn't allow.
That's great news! Will you do a patch or release for this fix? If so when will that be available? In our environment, it is preferred to work with some form of stable release package.
I am at the point that I want to use this functionality. Did this function make it into 2.0 or is this a patch that I can apply to RC2 and build? Could you provide instructions.
Turns out it was actually easy to implement without specifying the TSUIDs (though that's still an option). Check out https://github.com/manolama/opentsdb/tree/next_lastdp for working code. I'll post documentation shortly and clean up the code a bit but you can get the last points by accessing /api/query/last?timeseries=<metric>[{tagname=value[,tagname2=value2...]}]
ManOLamancha,
Is there any way that you could post a diff or patch of the code that was changed to make this functionality work? I will go ahead and try to apply it to the latest RC2. We need this functionality ;) Thanks.
Hiya Bean, sorry I thought you were referring to the TSUID map. The last data point code is actually in the main repo now and it's already synced with all of the latest 2.0 patches. Just fetch the next_lastdp branch from the main repo.
Thanks ManOLamancha. I tried it using the next_lastdp branch and I can't seem to get it working. Here is my examples:
This query:
https://127.0.0.1:4242/api/query?start=1m-ago&m=sum:proc.loadavg.1min{host=pmserver}
returns json message:
[{"metric":"proc.loadavg.1min","tags":{"host":"pmserver"},"aggregateTags":[],"dps":{"1390514504":0.019999999552965164}}]
----------------
The last data point query looks like:
https://127.0.0.1:4242/api/query/last?timeseries=proc.loadavg.1min{host=pmserver}
and returns blank json message:
[]
What am I doing wrong? I also noticed the documentation and it states that metadata must be created. Is this not created by default? Also there are 4 options, which one needs to be used?
tsd.core.meta.enable_realtime_uid
tsd.core.meta.enable_tsuid_tracking
tsd.core.meta.enable_tsuid_incrementing
tsd.core.meta.enable_realtime_ts