Storm 0.8 capacity metric

[Michael Rose]

Today we were working on launching our topologies into a new Storm 0.8.2 cluster and we noticed something rather strange about the capacity metric. In some cases we were seeing it >1 for certain CPU-heavy topologies.

From the 0.8.2 release notes, " The "capacity" metric is very useful and tells you what % of the time in the last 10 minutes the bolt spent executing tuples. If this value is close to 1, then the bolt is "at capacity" and is a bottleneck in your topology."

I think the algorithm is incorrect for multi-worker topologies, given we were seeing capacities well above 1. The topology had been running for several hours.

(defn compute-executor-capacity [^ExecutorSummary e]

  (let [stats (.get_stats e)

        stats (if stats

                (-> stats

                    (aggregate-bolt-stats true)

                    (aggregate-bolt-streams)

                    swap-map-order

                    (get "600")))

        uptime (nil-to-zero (.get_uptime_secs e))

        window (if (< uptime 600) uptime 600)

        executed (-> stats :executed nil-to-zero)

        latency (-> stats :execute-latencies nil-to-zero)

        ]

   (if (> window 0)

     (div (* executed latency) (* 1000 window))

     )))

From my interpretation, this is calculating the total execution time in ms over the window time in ms. With parallel execution, would total execution then many times be greater than real time? (num_executed * avg_execute_latency) / 600000

Should this not include a division over the number of executors?

-- 

Michael Rose (@Xorlev)

Senior Platform Engineer, FullContact
mic...@fullcontact.com

[anahap]

I have been seeing capacity values around 6-12 as well.

[Michael Rose]

Today we had a topology claiming 66 as a capacity.

Should this metric be scaled by the number of executors?