Irqbalance Strategies

[Kyle Kavanagh]

Looking for some information on how irqbalance assigns IRQs to cores.  Concrete information on the topic seems sparse - All I've been able to find is that it will use the estimates for IRQ workload to "fairly assign work across all cpus".  

Does the daemon take anything else into account (e.g current workload of each CPU on rebalance, isolcpu/taskset status)?  I would assume since the balancer runs a daemon, it can rebalance over time - assuming IRQ workload estimates stay the same, it must take something else into account, otherwise it would converge on a stable state after the first balance operation.

It would be optimal just to turn the balancer off and manually steer IRQs, but I'm not quite at that level of tuning just yet and want to see how the balancer works by default.

EDIT: Took a look at the source for irqbalancer, and can partially answer my question.  Looks like it determines load by reading irq and softirq loads from /proc/stat, ignoring user/system cpu time (https://github.com/Irqbalance/irqbalance/blob/master/procinterrupts.c#L457).  Additionally, the more recent versions will take isolcpus into account and treat them exactly like IRQBALANCE_BANNED_CPUS

Anything I might have missed in my interpretation of the code?

[Tom Lee]

On Sun, Apr 24, 2016 at 7:53 AM, Kyle Kavanagh <kdkav...@gmail.com> wrote:

Looking for some information on how irqbalance assigns IRQs to cores.  Concrete information on the topic seems sparse - All I've been able to find is that it will use the estimates for IRQ workload to "fairly assign work across all cpus".  

Does the daemon take anything else into account (e.g current workload of each CPU on rebalance)?  I would assume since the balancer runs a daemon, it can rebalance over time - assuming IRQ workload estimates stay the same, it must take something else into account, otherwise it would converge on a stable state after the first balance operation.

It sure seems that way:

https://github.com/Irqbalance/irqbalance/blob/2b40012c2709dcbbe33a3a443addffbf7b079b83/irqlist.c#L159-L174

Without trying to understand exactly what's going on, it looks like it might go to some effort to preserve balance across packages/cores/etc. and takes things like NUMA into account too. It seems like it's just keeping IRQ load away from busy CPUs while trying to minimize impact on caches etc. ... but again, I've not spent a huge amount of time trying to get my head around the exact details.

 

It would be optimal just to turn the balancer off and manually steer IRQs, but I'm not quite at that level of tuning just yet and want to see how the balancer works by default.

Perhaps you're already aware of it, but a half-way/lazy approach we've taken is "irqbalance --oneshot". Some distributions (Debian/Ubuntu, probably RHEL/CentOS) have a way to configure oneshot to run at system startup while simultaneously disabling the daemon via /etc/default config.

In our case, we just wanted to be sure all IRQs don't just sit on CPU0 -- don't necessarily care about the balance being perfect. I can imagine it might be possible to get a less-than-ideal configuration from --oneshot if you happen to be unlucky. We're taking that gamble in exchange for not having something running around messing with IRQ affinity all the time & not needing to worry about manual IRQ affinity jiggery in the typical case. Whether that's the right call is open to interpretation, but it's worked fine for us so far -- though keep in mind we're not as antsy about latency as other folks around these parts.

Cheers,

Tom

--

Tom Lee / http://tomlee.co / @tglee