repeated "passthrough->passthrough..." when trying to access a metric using /api/query

[casey.d...@teamaol.com]

at some point recently we have started to get this error when using the api/query endpoint to retrieve metrics that we were previously able to retrieve.

"...passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough) (size=16383) when attempting to add cb=net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@596b326c@1500197484, eb=passthrough@97095169"

Some things to note:

1. tags for the metric were recently changed both number of tags(7 down to 3), and the tag keys themselves(one name was a tag key already in the system, two others are new)

2. We started adding a huge number of unique metrics with high cardinality to the system.

3. the older metric has a very low cardinality and is only distinguishable to a host for example by adding specific host identifiable tags

4. the new high cardinality metrics are able to be retrieved, and are retrieved much quicker in comparison to the time the original metrics were.

5. tsd compaction was turned off, due to its killing our ability to query for metrics at the hour, and we felt the splicer bridge in between could help to alleviate the need for compaction.

Any help in tracking this down would be greatly appreciated

[casey.d...@teamaol.com]

added info this is the start of the error:

13:17:21.429 ERROR [RegionClient.exceptionCaught] - Unexpected exception from downstream on [id:

 0x832de5f1, /172.17.0.5:43076 => /10.98.78.94:16020]

com.stumbleupon.async.CallbackOverflowError: Too many callbacks in Deferred@1358298081(state=PEN

DING, result=null, callback=net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@6fce7e91 -> net.ope

ntsdb.tsd.HttpJsonSerializer$1DPsResolver@57b83df1 -> net.opentsdb.tsd.HttpJsonSerializer$1DPsRe

solver@e1026c -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@7f34db62 -> net.opentsdb.tsd.H

ttpJsonSerializer$1DPsResolver@631208e3 -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@75d3

f6c5 -> net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@7d296871 -> ...

[casey.d...@teamaol.com]

Even more info: it appears to be related to a specific tag, there are 3 tags, so if my query contains no tags - ok, tag1 ok, tag2 ok, but tag3 - this error

On Friday, October 14, 2016 at 8:45:19 AM UTC-4, casey.d...@teamaol.com wrote:

[ManOLamancha]

On Friday, October 14, 2016 at 8:37:44 AM UTC-7, casey.d...@teamaol.com wrote:

Even more info: it appears to be related to a specific tag, there are 3 tags, so if my query contains no tags - ok, tag1 ok, tag2 ok, but tag3 - this error

Yeah, this is an ugly issue we don't have a great fix for. It happens when you have a REALLY high cardinality tag (e.g. 1,000s of values for a specific tag key). We have a couple of potential options that will help in 2.4 

[Java Enthu]

Hi ManOLama,

When can I expect a fix for this problem?  My project is really struggling with this problem.

Also, can you please let me know the next release dates (2.4, 3 etc)

Thanks,
Gopi

[ManOLamancha]

On Tuesday, May 9, 2017 at 10:17:06 AM UTC-7, Java Enthu wrote:

Hi ManOLama,

When can I expect a fix for this problem?  My project is really struggling with this problem.

Also, can you please let me know the next release dates (2.4, 3 etc)

We have some multi-get code in there, just have to fix it up for the UID resolution. I'll try and get a fix in this weekend. 2.4 is still a ways out, it'll be after HBase con and 3.0 is much further out still.