Getting "Queue full" for metrics servlet

[Jordan Zimmerman]

I keep getting a queue full with the metrics servlet. What would cause this?

2014-09-22 11:45:52,965 com.netflix.hystrix.contrib.metrics.eventstream.HystrixMetricsPoller [] - Failed to output metrics as JSON

java.lang.IllegalStateException: Queue full

        at java.util.AbstractQueue.add(AbstractQueue.java:98)

        at com.netflix.hystrix.contrib.metrics.eventstream.HystrixMetricsStreamServlet$MetricJsonListener.handleJsonMetric(HystrixMetricsStreamServlet.java:191)

        at com.netflix.hystrix.contrib.metrics.eventstream.HystrixMetricsPoller$MetricsPoller.run(HystrixMetricsPoller.java:256)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:744)

[Ben Christensen]

Do you legitimately have more than 1000 commands?

If not, then this is a sign that your metrics polling rate is exceeding the rate your IO is draining the queue. 

An arbitrary limit (perhaps too low) of 1000 is set to prevent unbounded growth and signal a system problem if the polling/draining is out of balance. 

-- 
Ben Christensen - Netflix Edge Engineering
+1.310.782.5511  @benjchristensen

--
You received this message because you are subscribed to the Google Groups "HystrixOSS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hystrixoss+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jordan Zimmerman]

One of our servers has ~ 200. I don't know why we keep seeing the exception. BTW - we're running Turbine as well.

[Ben Christensen]

The only reason I can think of is that the interval on the producing side is far faster than the polling and IO on the consumer side.

-- 
Ben Christensen - Netflix Edge Engineering
+1.310.782.5511  @benjchristensen