NMI watchdog soft lockup

[smbra...@gmail.com]

I have started to encounter a relatively new issue on my Beaglebone Black that I have been developing an app on for quite some time now. I am not quite sure when I first started seeing this and what may have changed, so I am looking for advice on how to troubleshoot this further:

Message from syslogd@beaglebone at Apr 19 19:55:10 ...

 kernel:[244638.300687] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [swapper/0:0]

Message from syslogd@beaglebone at Apr 19 19:55:10 ...

 kernel:[244666.198366] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [swapper/0:0]

Message from syslogd@beaglebone at Apr 19 19:55:10 ...

 kernel:[244694.095031] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [swapper/0:0]

Message from syslogd@beaglebone at Apr 19 19:55:10 ...

 kernel:[244721.991558] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [swapper/0:0]

[Robert Nelson]

This smells like an ancient kernel..

Run this command and share the output..

sudo /opt/scripts/tools/version.sh

Regards,

--
Robert Nelson
https://rcn-ee.com/

[smbra...@gmail.com]

git:/opt/scripts/:[9df8deb6b2ae7570c9102ab58f795d3bed592194]

eeprom:[A335BNLT000C1829BBBG0182]

model:[TI_AM335x_BeagleBone_Black]

dogtag:[BeagleBoard.org Debian Image 2017-03-19]

bootloader:[microSD-(push-button)]:[/dev/mmcblk0]:[U-Boot 2019.04-00002-gc8b5ad3a1f]:[location: dd MBR]

kernel:[4.9.147-ti-r120]

nodejs:[v4.8.0]

device-tree-override:[dtb=am335x-boneblack-emmc-overlay.dtb]

uboot_overlay_options:[enable_uboot_overlays=1]

uboot_overlay_options:[disable_uboot_overlay_video=1]

uboot_overlay_options:[disable_uboot_overlay_audio=1]

uboot_overlay_options:[disable_uboot_overlay_wireless=1]

pkg check: to individually upgrade run: [sudo apt install --only-upgrade <pkg>]

pkg:[bb-cape-overlays]:[4.1.20170309-0rcnee0~bpo80+20170309]

pkg:[bb-wl18xx-firmware]:[1.20170309-0rcnee1~bpo80+20170309]

pkg:[kmod]:[18-3]

pkg:[roboticscape]:[0.3.4-git20170307-0rcnee2~bpo80+20170307]:[GOT_REPLACED_BY_NEXT]

WARNING:pkg:[librobotcontrol]:[NOT_INSTALLED]

WARNING:pkg:[firmware-ti-connectivity]:[NOT_INSTALLED]

groups:[debian : debian adm kmem dialout cdrom floppy audio dip video plugdev users systemd-journal netdev i2c bluetooth admin spi tisdk weston-launch xenomai]

cmdline:[console=ttyO0,115200n8 root=/dev/mmcblk0p1 ro rootfstype=ext4 rootwait coherent_pool=1M net.ifnames=0 quiet cape_universal=enable]

dmesg | grep remote

dmesg | grep pru

[229286.908037] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss

[229314.805534] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss

[229342.703050] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss

[233094.641593] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss

[233122.539507] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss

[233150.437176] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss

[233178.334647] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss

[236945.719703] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss

[236973.617889] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss

[237001.515392] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss

[237029.412906] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss

[240790.547160] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss

[240818.444960] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss

[240846.343106] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss

[240874.240610] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss

[244638.308181] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss

[244666.205825] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss

[244694.102489] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss

[244721.999013] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss

[248482.167326] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss

[248510.065194] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss

[248537.963199] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss

[248565.860624] Modules linked in: ppp_deflate bsd_comp ppp_async ppp_generic slhc pvrsrvkm(O) qmi_wwan cdc_wdm option usbnet usb_wwan mii usbserial evdev uio_pdrv_genirq uio usb_f_mass_storage usb_f_acm u_serial usb_f_ecm usb_f_rndis u_ether libcomposite 8021q garp mrp stp llc spidev pru_rproc pruss_intc pruss

dmesg | grep pinctrl-single

dmesg | grep gpio-of-helper

lsusb

Bus 001 Device 018: ID 05c6:90b2 Qualcomm, Inc.

Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

END

[Robert Nelson]

On Fri, Apr 19, 2019 at 4:07 PM <smbra...@gmail.com> wrote:
>
> git:/opt/scripts/:[9df8deb6b2ae7570c9102ab58f795d3bed592194]
> eeprom:[A335BNLT000C1829BBBG0182]
> model:[TI_AM335x_BeagleBone_Black]
> dogtag:[BeagleBoard.org Debian Image 2017-03-19]
> bootloader:[microSD-(push-button)]:[/dev/mmcblk0]:[U-Boot 2019.04-00002-gc8b5ad3a1f]:[location: dd MBR]
> kernel:[4.9.147-ti-r120]

ah yes, 4.9.x upgrade to 4.14.x or downgrade to v4.4.x...

> nodejs:[v4.8.0]
> device-tree-override:[dtb=am335x-boneblack-emmc-overlay.dtb]

This can be removed as in v4.14.x, the next 4 line's enable the same thing..

> uboot_overlay_options:[enable_uboot_overlays=1]
> uboot_overlay_options:[disable_uboot_overlay_video=1]
> uboot_overlay_options:[disable_uboot_overlay_audio=1]
> uboot_overlay_options:[disable_uboot_overlay_wireless=1]

cd /opt/scripts/tools/
git pull
sudo ./update_kernel.sh --ti-channel --lts-4_14

> pkg check: to individually upgrade run: [sudo apt install --only-upgrade <pkg>]
> pkg:[bb-cape-overlays]:[4.1.20170309-0rcnee0~bpo80+20170309]

and also run, as ^ is going to break something..

sudo apt install --only-upgrade bb-cape-overlays

[smbra...@gmail.com]

Awesome, thank you Robert