High CPU usage / high number of interrupts

[0x1eef]

Hi, everyone!

When I use Chromium, I see a high rate of CPU usage across all four cores. The rate can be anywhere from 20% to 50%, even above that. I am not doing anything intensive, just browsing twitter, reddit, YouTube or GitHub. It has been like this since I installed FreeBSD, but since it's not a blocker I have been lazy about looking into it.

I don't know why it happens. I can see that there are a high number of interrupts on 'xhci0', and that seems to carry over to each CPU core as well:

# vmstat -i            

interrupt                          total       rate

irq1: atkbd0                          50          0

irq9: acpi0                          403          0

cpu0:timer                      30716618         98

cpu1:timer                      25457926         81

cpu2:timer                      34344531        109

cpu3:timer                      25542867         81

irq128: xhci0                  328107434       1044

irq130: nvme0:admin                   15          0

irq131: nvme0:io0                 701041          2

irq132: nvme0:io1                 692045          2

irq133: nvme0:io2                 792760          3

irq134: nvme0:io3                 693091          2

irq135: hdac0                    1718425          5

irq136: vgapci0                  6273295         20

Total                          455040501       1448

# dmesg | grep xhci0

xhci0: <Intel Ice Lake-LP USB 3.1 controller> mem 0x95110000-0x9511ffff at device 20.0 on pci0

xhci0: 32 bytes context size, 64-bit DMA

usbus0 on xhci0

It might also be helpful to know that I tried OpenBSD on the same computer but it was unusable for a similar reason: 95%+ interrupts on CPU. The impact that had made all tasks extremely slow. On FreeBSD it is not as bad, but I still think  think it is not normal.

Can anyone suggest what might be wrong, tips to debug, etc ? If more information is needed, please let me know. Thanks for your time.

Best,

0x1eef

[Paul Procacci]

Hey,

Not sure of the problem, but I don't see the correlation between Chrome and any usb driver.

Out of curiosity, have you pulled a usb device one by one until the interrupts disappear?

I'd be curious to know which device is slamming the system.

Thanks,

Paul

--

__________________

:(){ :|:& };:

[0x1eef]

Hi !

> Out of curiosity, have you pulled a usb device one by one until the interrupts disappear?

I have three USB devices connected: mouse, keyboard, and an ethernet adapter. 

I tried to remove each one by one, and I did not see the interrupt rate change.

I have also tried a cold boot without any USB devices connected, and the interrupt rate was about the same too.

I don't know if it could be related, but there's a trackpad connected to the laptop that does not work. Maybe it has no relation to the issue, but setting "hw.psm.synaptics_support" to "0" also did not help.

When Chromium loses focus, CPU usage usually drops to 0% and does not go above 10% - for as long as I am not using Chromium. I am using the i915 / drm kernel modules.. I saw another report of high CPU usage related to using those two kernel modules, but I wasn't able to identify that as the problem in my case.

Thanks for the help. 

Sent with Proton Mail secure email.

------- Original Message -------

[Paul Procacci]

Can you determine if irq 128 is being shared by any devices?

Usually this information can be found in `dmesg' or '/var/run/dmesg.boot'.

vmstat indeed shows a device but it sometimes doesn't show all the devices sharing that IRQ.  It's possible you're being misled by vmstat.

Just trying to get the complete picture here of devices.  ;)

Thanks,

Paul Procacci

--

__________________

:(){ :|:& };:

[0x1eef]

> Can you determine if irq 128 is being shared by any devices?

I couldn't determine that from dmesg.boot, but I think there could be some useful information in that file. I attached the file to this e-mail. Thank you!

Best,

0x1eef

Sent with Proton Mail secure email.

------- Original Message -------

[Paul Procacci]

usbconfig dump_stats

Can you provide the output of the above?  Run that a couple of times if you don't mind and do your very best to provide any rates of increases for any of the fields.  An estimation would be perfectly fine.

Thanks,

Paul

--

__________________

:(){ :|:& };:

[0x1eef]

The results are sort of interesting. At first, UE_INTERRUPT_OK was at 3k+ for the USB mouse. I unplugged the mouse, and then the keyboard jumped from 0 to 1k+ for UE_INTERRUPT_OK. 

I have since reattached the mouse, and now both the mouse and the keyboard have a rising interrupt count. I would guess they jump by 20, or 30 interrupts every 2-3 seconds, with the keyboard jumping with a higher frequency.

Paste

ugen0.1: <Intel XHCI root HUB> at usbus0, cfg=0 md=HOST spd=SUPER (5.0Gbps) pwr=SAVE (0mA)

{

    UE_CONTROL_OK       : 0

    UE_ISOCHRONOUS_OK   : 0

    UE_BULK_OK          : 0

    UE_INTERRUPT_OK     : 0

    UE_CONTROL_FAIL     : 0

    UE_ISOCHRONOUS_FAIL : 0

    UE_BULK_FAIL        : 0

    UE_INTERRUPT_FAIL   : 0

}

ugen0.2: <Realtek USB 10/100/1000 LAN> at usbus0, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=ON (350mA)

{

    UE_CONTROL_OK       : 7969

    UE_ISOCHRONOUS_OK   : 0

    UE_BULK_OK          : 11224

    UE_INTERRUPT_OK     : 0

    UE_CONTROL_FAIL     : 0

    UE_ISOCHRONOUS_FAIL : 0

    UE_BULK_FAIL        : 94

    UE_INTERRUPT_FAIL   : 0

}

ugen0.4: <Sonix Technology Co., Ltd. Integrated Camera> at usbus0, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=ON (500mA)

{

    UE_CONTROL_OK       : 10

    UE_ISOCHRONOUS_OK   : 0

    UE_BULK_OK          : 0

    UE_INTERRUPT_OK     : 0

    UE_CONTROL_FAIL     : 0

    UE_ISOCHRONOUS_FAIL : 0

    UE_BULK_FAIL        : 0

    UE_INTERRUPT_FAIL   : 0

}

ugen0.6: <vendor 0x0cf3 product 0xe500> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (100mA)

{

    UE_CONTROL_OK       : 16

    UE_ISOCHRONOUS_OK   : 53214

    UE_BULK_OK          : 0

    UE_INTERRUPT_OK     : 9

    UE_CONTROL_FAIL     : 0

    UE_ISOCHRONOUS_FAIL : 0

    UE_BULK_FAIL        : 0

    UE_INTERRUPT_FAIL   : 0

}

ugen0.5: <vendor 0x30fa USB OPTICAL MOUSE> at usbus0, cfg=0 md=HOST spd=LOW (1.5Mbps) pwr=ON (100mA)

{

    UE_CONTROL_OK       : 11

    UE_ISOCHRONOUS_OK   : 0

    UE_BULK_OK          : 0

    UE_INTERRUPT_OK     : 1707

    UE_CONTROL_FAIL     : 0

    UE_ISOCHRONOUS_FAIL : 0

    UE_BULK_FAIL        : 0

    UE_INTERRUPT_FAIL   : 0

}

ugen0.3: <SINO WEALTH Gaming KB> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (500mA)

{

    UE_CONTROL_OK       : 21

    UE_ISOCHRONOUS_OK   : 0

    UE_BULK_OK          : 0

    UE_INTERRUPT_OK     : 1790

    UE_CONTROL_FAIL     : 0

    UE_ISOCHRONOUS_FAIL : 0

    UE_BULK_FAIL        : 0

    UE_INTERRUPT_FAIL   : 0

}

Best,

0x1eef

------- Original Message -------

[Paul Procacci]

What's catching my eye here is the values for ugen0.6.  It's a usb bluetooth device.

With that in mind I searched the bug database for bluetooth usb and xhci.
I came across the following:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=238235

There does seem to be some sort of weird interaction with bluetooth and xhci as documented in the above link.

I believe the patch listed in that bug report made it to 13.1....so can you try:

sysctl -w net.bluetooth.usb_isoc_enable=0

Does that make any difference?

(This btw is a wild guess just based on the values from the output you provided me)

Thanks,

Paul

--

__________________

:(){ :|:& };:

[0x1eef]

Yes ! That makes a huge difference. I added "net.bluetooth.usb_isoc_enable=0" to /boot/loader.conf, and rebooted. The interrupts have gone down to between 0, and 47. 

# vmstat -i

interrupt                          total       rate

irq1: atkbd0                          10          0

irq9: acpi0                            9          0

cpu0:timer                         70196        342

cpu1:timer                         51387        250

cpu2:timer                         50952        248

cpu3:timer                         51181        249

irq128: xhci0                       9659         47

irq130: nvme0:admin                   15          0

irq131: nvme0:io0                   3253         16

irq132: nvme0:io1                   3243         16

irq133: nvme0:io2                   3341         16

irq134: nvme0:io3                   3883         19

irq135: hdac0                         13          0

irq136: vgapci0                    37917        185

Total                             285059       1389

However, the CPU interrupts seem quite high now. If you combine cpu*:timer i think it is over 1000, and maybe that relates to the original problem I reported ?

Thanks!

0x1eef

Sent with Proton Mail secure email.

------- Original Message -------

[Paul Procacci]

I've personally never had to fiddle with event timers so this may be out of my realm.
With that said the following man page describes that various timers that one could employ:

man eventtimers

When I add up all my timer values, their sum is slightly over 1300 so I'm not entirely convinced it's a problem.

I'm running around quite a bit today so I'll look a bit later to see if anything stands out and let you know.

Thanks,

Paul

--

__________________

:(){ :|:& };:

[0x1eef]

> man eventtimers

That's good to know!

> When I add up all my timer values, their sum is slightly over 1300 so I'm not entirely convinced it's a problem.

Also good to know.

> I'm running around quite a bit today so I'll look a bit later to see if anything stands out and let you know.

No worries. You have already helped a ton. God willing, I will see if the event timers man page leads anywhere. If not, it's not the end of the world. My system is not unusable because of this problem, but the CPU usage does seem a bit high for what I'd expect to be normal.

Thanks!

0x1eef

Sent with Proton Mail secure email.

------- Original Message -------