Monitoring the size of the event loop queues

[Eran Harel]

We've implemented monitoring for the Netty event loop queues in order to understand issues with some of our Netty modules.
The monitor uses the io.netty.util.concurrent.SingleThreadEventExecutor#pendingTasks method, which works for most modules, but for a module that handle a few thousand HTTP requests per second it seem to be hung, or very slow.
I now realize that the docs strictly specify this can be an issue, and I feel pretty lame...

You can see our code here:
https://github.com/outbrain/ob1k/blob/6364187b30cab5b79d64835131d9168c754f3c09/ob1k-core/src/main/java/com/outbrain/ob1k/common/metrics/NettyQueuesGaugeBuilder.java

My question is, is there a better way to monitor the queue sizes?

This can be quite a useful metric, as it can be used to understand latency, and also to be used for back-pressure in some cases.

[Eran Harel]

Reposted the question in SO, in case someone wants to gain some rep points ;)
http://stackoverflow.com/questions/32933367/monitoring-the-size-of-the-netty-event-loop-queues

[Tim Boudreau]

The main challenge will be trading off accuracy for not affecting performance.

Probably subclass SingleThreadEventExecutor (or wrapper a SingleThreadEventExecutor in your own event executor interface, if you can intercept when to decrement your counter) and wrap all calls that enqueue a task to update a counter (for bonus points, write something like AtomicInteger but which has a lazyIncrement() method similar to AtomicInteger's lazySet() - but a volatile int will also do the job).

If you just want to know the max number that have ever been queued, you could subclass or wrap and use something like this:

https://github.com/timboudreau/util/blob/master/src/main/java/com/mastfrog/util/thread/AtomicMaximum.java

-Tim

[Eran Harel]

Thanks Tim.

It ain't trivial as you also need to know when a task ends, but I'll dig in to see if this is a reasonable direction.

[Tim Boudreau]

Thanks Tim.

It ain't trivial as you also need to know when a task ends, but I'll dig in to see if this is a reasonable direction.

Perhaps you can wrap each job in an implementation of the same interface that delegates to the real job, then decrements your counter.

-Tim