at some point recently we have started to get this error when using the api/query endpoint to retrieve metrics that we were previously able to retrieve.
"...passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough -> passthrough) (size=16383) when attempting to add cb=net.opentsdb.tsd.HttpJsonSerializer$1DPsResolver@596b326c@1500197484, eb=passthrough@97095169"
Some things to note:
1. tags for the metric were recently changed both number of tags(7 down to 3), and the tag keys themselves(one name was a tag key already in the system, two others are new)
2. We started adding a huge number of unique metrics with high cardinality to the system.
3. the older metric has a very low cardinality and is only distinguishable to a host for example by adding specific host identifiable tags
4. the new high cardinality metrics are able to be retrieved, and are retrieved much quicker in comparison to the time the original metrics were.
5. tsd compaction was turned off, due to its killing our ability to query for metrics at the hour, and we felt the splicer bridge in between could help to alleviate the need for compaction.
Any help in tracking this down would be greatly appreciated