Is there an upper limit for number of series retrieved in a single TSDB query using wildcard filter?
142 views
Skip to first unread message
hy...@zenoss.com
Feb 16, 2017, 12:29:39 PM2/16/17
to OpenTSDB
I am running TSDB 2.2 and trying to retrieve data for a metric with very high cardinality, ie, one of the tags can have tens of thousands of different values. The goal is to get the all the series for that particular metric, what is the best way to accomplish this?
Using wildcard in filter for the tag key worked well when tag values are in the thousands range. But when the tag values are in tens of thousands, the query never returns. Is there an upper limit for the number of series in a single query?
Here is query -
{
"start": "2h-ago",
"end": null,
"timezone": null,
"options": null,
"padding": false,
"queries": [
{
"aggregator": "avg",
"metric": "metricName",
"tsuids": null,
"downsample": "300s-avg",
"rate": false,
"filters": [
{
"tagk": "key1",
"filter": "abcdefg",
"group_by": true,
"type": "literal_or"
},
{
"tagk": "key2", <---- this is the tagk that can have tens of thousands different values
"filter": "*",
"group_by": true,
"type": "wildcard"
}
],
"rateOptions": {
"counter": true,
"counterMax": 1000,
"resetValue": 0,
"dropResets": false
}
}
],
"delete": false,
"noAnnotations": false,
"globalAnnotations": false,
"showTSUIDs": false,
"msResolution": false,
"showQuery": false,
"showStats": false,
"showSummary": false
}
hy...@zenoss.com
Feb 16, 2017, 1:02:08 PM2/16/17
to OpenTSDB
Just notices the following error in the tsdb log - CallbackOverFlowError. Simply way too many series to deal with at once.
So what is the best way to handle this, list the tag values in the filter instead of using wildcard and split into multiple queries?
com.stumbleupon.async.CallbackOverflowError: Too many callbacks in Deferred@32793120(state=PENDING, result=null, callback=net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@6060bdcc -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@52403171 -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@f3a9fd -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@22342ea1 -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@475f79f6 -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@14c713d5 -> .......
at com.stumbleupon.async.Deferred.addCallbacks(Deferred.java:669) ~[async-1.4.0.jar:na]at com.stumbleupon.async.Deferred.addCallback(Deferred.java:724) ~[async-1.4.0.jar:na]at net.opentsdb.tsd.HttpJsonSerializer.formatQueryAsyncV1(HttpJsonSerializer.java:845) ~[tsdb-2.2.0.jar:]at net.opentsdb.tsd.QueryRpc$1QueriesCB.call(QueryRpc.java:234) ~[tsdb-2.2.0.jar:]at net.opentsdb.tsd.QueryRpc$1QueriesCB.call(QueryRpc.java:217) ~[tsdb-2.2.0.jar:]at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) ~[async-1.4.0.jar:na]at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) ~[async-1.4.0.jar:na]at com.stumbleupon.async.Deferred.callback(Deferred.java:1005) ~[async-1.4.0.jar:na]at com.stumbleupon.async.DeferredGroup.done(DeferredGroup.java:173) ~[async-1.4.0.jar:na]at com.stumbleupon.async.DeferredGroup.recordCompletion(DeferredGroup.java:158) ~[async-1.4.0.jar:na]at com.stumbleupon.async.DeferredGroup.access$200(DeferredGroup.java:36) ~[async-1.4.0.jar:na]at com.stumbleupon.async.DeferredGroup$1NotifyOrdered.call(DeferredGroup.java:97) ~[async-1.4.0.jar:na]at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) ~[async-1.4.0.jar:na]at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) ~[async-1.4.0.jar:na]at com.stumbleupon.async.Deferred.callback(Deferred.java:1005) ~[async-1.4.0.jar:na]at net.opentsdb.core.TsdbQuery$1ScannerCB.close(TsdbQuery.java:798) ~[tsdb-2.2.0.jar:]at net.opentsdb.core.TsdbQuery$1ScannerCB.call(TsdbQuery.java:624) ~[tsdb-2.2.0.jar:]at net.opentsdb.core.TsdbQuery$1ScannerCB.call(TsdbQuery.java:570) ~[tsdb-2.2.0.jar:]at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) ~[async-1.4.0.jar:na]at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) ~[async-1.4.0.jar:na]at com.stumbleupon.async.Deferred.access$300(Deferred.java:430) ~[async-1.4.0.jar:na]at com.stumbleupon.async.Deferred$Continue.call(Deferred.java:1366) ~[async-1.4.0.jar:na]at com.stumbleupon.async.Deferred.doCall(Deferred.java:1278) ~[async-1.4.0.jar:na]at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1257) ~[async-1.4.0.jar:na]at com.stumbleupon.async.Deferred.callback(Deferred.java:1005) ~[async-1.4.0.jar:na]at org.hbase.async.HBaseRpc.callback(HBaseRpc.java:698) ~[asynchbase-1.7.1.jar:na]at org.hbase.async.RegionClient.decode(RegionClient.java:1516) ~[asynchbase-1.7.1.jar:na]at org.hbase.async.RegionClient.decode(RegionClient.java:88) ~[asynchbase-1.7.1.jar:na]at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) ~[netty-3.9.4.Final.jar:na]at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[netty-3.9.4.Final.jar:na]at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.9.4.Final.jar:na]at org.hbase.async.RegionClient.handleUpstream(RegionClient.java:1206) ~[asynchbase-1.7.1.jar:na]at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.4.Final.jar:na]at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.9.4.Final.jar:na]at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [netty-3.9.4.Final.jar:na]at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [netty-3.9.4.Final.jar:na]at org.jboss.netty.handler.timeout.IdleStateAwareChannelHandler.handleUpstream(IdleStateAwareChannelHandler.java:36) [netty-3.9.4.Final.jar:na]at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.4.Final.jar:na]at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [netty-3.9.4.Final.jar:na]at org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:294) [netty-3.9.4.Final.jar:na]at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.9.4.Final.jar:na]at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.4.Final.jar:na]at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.9.4.Final.jar:na]at org.hbase.async.HBaseClient$RegionClientPipeline.sendUpstream(HBaseClient.java:3108) [asynchbase-1.7.1.jar:na]at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.9.4.Final.jar:na]at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.9.4.Final.jar:na]at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [netty-3.9.4.Final.jar:na]at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [netty-3.9.4.Final.jar:na]at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) [netty-3.9.4.Final.jar:na]at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [netty-3.9.4.Final.jar:na]at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.9.4.Final.jar:na]at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.9.4.Final.jar:na]at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.9.4.Final.jar:na]at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_101]at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_101]at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
ManOLamancha
Apr 25, 2017, 4:05:29 PM4/25/17
to OpenTSDB
On Thursday, February 16, 2017 at 10:02:08 AM UTC-8, hy...@zenoss.com wrote:
Just notices the following error in the tsdb log - CallbackOverFlowError. Simply way too many series to deal with at once.So what is the best way to handle this, list the tag values in the filter instead of using wildcard and split into multiple queries
Right, unfortunately. It will work if you aggregate the metrics (meaning set group-by to false) otherwise what you'll have to do is split the query into multiple smaller queries. It's the conversion of the UIDs back to strings that is hitting you hard here. We'll be taking a stab at a fix for that in 3.0.
dreamersdw
May 3, 2017, 9:34:15 AM5/3/17
to OpenTSDB
I had meet a similar problem. it was fixed by change JVM parameters in /etc/init.d/opentsdb
from
: ${JVMXMX:=-Xmx6000m}
into
: ${JVMXMX:=-Xmx6000m -Xss16m}
Reply all
Reply to author
Forward
0 new messages