[Iscsitarget-devel] BUG: soft lockup - CPU#0 stuck for 60s
Marko Kobal
Hi,
Today my ESXi 5 started reporting a warning:
---
ScsiDeviceIO: 3075: Failed write command to write-quiesced partition t10.94544500000000009637363796D20727F646D20353D25300:1
(2012-03-16T07:14:01.253Z cpu2:2773)
warning
---
And my iSCSI box is telling what you can see below. Any help would be appreciated! Is this actually a bug or is there something wrong in my configuration?
BUG: soft lockup - CPU#0 stuck for 60s! [istd1:2628]
CPU 0:
Modules linked in: iscsi_trgt(U) hidp l2cap bluetooth sunrpc ipv6 xfrm_nalgo crypto_api ext4 jbd2 crc16 dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec i2c_core dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport sr_mod cdrom sg e1000e(U) i3000_edac hpilo tg3 edac_mc serio_raw shpchp pcspkr tpm_tis tpm tpm_bios 8021q dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod usb_storage ata_piix libata cciss sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Pid: 2628, comm: istd1 Tainted: G ---- 2.6.18-274.7.1.el5 #1
RIP: 0010:[<ffffffff80064bcc>] [<ffffffff80064bcc>] .text.lock.spinlock+0x2/0x30
RSP: 0018:ffffffff804abb78 EFLAGS: 00000286
RAX: 00000000ffffffff RBX: ffff81006ffc9800 RCX: 0000000051359845
RDX: ffff81005c216ba8 RSI: ffff81005c2166c0 RDI: ffff810047d00058
RBP: ffffffff804abaf0 R08: ffff8100785fd7d8 R09: 0000000051359845
R10: ffffffff804abc10 R11: 0000000000000048 R12: ffffffff8005dc8e
R13: ffff810047d00000 R14: ffffffff80079330 R15: ffffffff804abaf0
FS: 0000000000000000(0000) GS:ffffffff8042c000(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00002aaaae525000 CR3: 0000000037e4e000 CR4: 00000000000006e0
Call Trace:
<IRQ> [<ffffffff88552e06>] :iscsi_trgt:conn_close+0x26/0x97
[<ffffffff88553161>] :iscsi_trgt:iet_state_change+0x24/0x3f
[<ffffffff800541fe>] tcp_fin+0x19c/0x1ec
[<ffffffff800360f2>] tcp_data_queue+0x2bd/0xab5
[<ffffffff8001c179>] tcp_rcv_established+0x7f5/0x8bd
[<ffffffff8003b446>] tcp_v4_do_rcv+0x2a/0x2fa
[<ffffffff8002752a>] tcp_v4_rcv+0xa1d/0xa71
[<ffffffff8024682f>] __qdisc_run+0x76/0x1f6
[<ffffffff8003471a>] ip_local_deliver+0x19f/0x265
[<ffffffff80035875>] ip_rcv+0x539/0x57c
[<ffffffff80020c21>] netif_receive_skb+0x470/0x49f
[<ffffffff8829b99a>] :e1000e:e1000_clean_rx_irq+0x2ba/0x390
[<ffffffff882990d9>] :e1000e:e1000_poll+0x79/0x2c0
[<ffffffff882ac672>] :e1000e:__kc_adapter_clean+0x32/0x60
[<ffffffff8000ca51>] net_rx_action+0xac/0x1b1
[<ffffffff882948f0>] :e1000e:e1000_intr_msi+0xf0/0x110
[<ffffffff80012562>] __do_softirq+0x89/0x133
[<ffffffff8005e2fc>] call_softirq+0x1c/0x28
[<ffffffff8006d636>] do_softirq+0x2c/0x7d
[<ffffffff8006d4c6>] do_IRQ+0xee/0xf7
[<ffffffff8005d615>] ret_from_intr+0x0/0xa
<EOI> [<ffffffff80064a58>] _spin_lock+0x3/0xa
[<ffffffff8855119b>] :iscsi_trgt:istd+0x64d/0x1073
[<ffffffff8025abf1>] tcp_sendpage+0x0/0x651
[<ffffffff88550b4e>] :iscsi_trgt:istd+0x0/0x1073
[<ffffffff800a2c39>] keventd_create_kthread+0x0/0xc4
[<ffffffff8003270f>] kthread+0xfe/0x132
[<ffffffff8005dfb1>] child_rip+0xa/0x11
[<ffffffff800a2c39>] keventd_create_kthread+0x0/0xc4
[<ffffffff80032611>] kthread+0x0/0x132
[<ffffffff8005dfa7>] child_rip+0x0/0x11
BUG: soft lockup - CPU#0 stuck for 60s! [istd1:2628]
CPU 0:
Modules linked in: iscsi_trgt(U) hidp l2cap bluetooth sunrpc ipv6 xfrm_nalgo crypto_api ext4 jbd2 crc16 dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec i2c_core dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport sr_mod cdrom sg e1000e(U) i3000_edac hpilo tg3 edac_mc serio_raw shpchp pcspkr tpm_tis tpm tpm_bios 8021q dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod usb_storage ata_piix libata cciss sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Pid: 2628, comm: istd1 Tainted: G ---- 2.6.18-274.7.1.el5 #1
RIP: 0010:[<ffffffff80064bcf>] [<ffffffff80064bcf>] .text.lock.spinlock+0x5/0x30
RSP: 0018:ffffffff804abb78 EFLAGS: 00000286
RAX: 00000000ffffffff RBX: ffff81006ffc9800 RCX: 0000000051359845
RDX: ffff81005c216ba8 RSI: ffff81005c2166c0 RDI: ffff810047d00058
RBP: ffffffff804abaf0 R08: ffff8100785fd7d8 R09: 0000000051359845
R10: ffffffff804abc10 R11: 0000000000000048 R12: ffffffff8005dc8e
R13: ffff810047d00000 R14: ffffffff80079330 R15: ffffffff804abaf0
FS: 0000000000000000(0000) GS:ffffffff8042c000(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00002aaaae525000 CR3: 0000000037e4e000 CR4: 00000000000006e0
Call Trace:
<IRQ> [<ffffffff88552e06>] :iscsi_trgt:conn_close+0x26/0x97
[<ffffffff88553161>] :iscsi_trgt:iet_state_change+0x24/0x3f
[<ffffffff800541fe>] tcp_fin+0x19c/0x1ec
[<ffffffff800360f2>] tcp_data_queue+0x2bd/0xab5
[<ffffffff8001c179>] tcp_rcv_established+0x7f5/0x8bd
[<ffffffff8003b446>] tcp_v4_do_rcv+0x2a/0x2fa
[<ffffffff8002752a>] tcp_v4_rcv+0xa1d/0xa71
[<ffffffff8024682f>] __qdisc_run+0x76/0x1f6
[<ffffffff8003471a>] ip_local_deliver+0x19f/0x265
[<ffffffff80035875>] ip_rcv+0x539/0x57c
[<ffffffff80020c21>] netif_receive_skb+0x470/0x49f
[<ffffffff8829b99a>] :e1000e:e1000_clean_rx_irq+0x2ba/0x390
[<ffffffff882990d9>] :e1000e:e1000_poll+0x79/0x2c0
[<ffffffff882ac672>] :e1000e:__kc_adapter_clean+0x32/0x60
[<ffffffff8000ca51>] net_rx_action+0xac/0x1b1
[<ffffffff882948f0>] :e1000e:e1000_intr_msi+0xf0/0x110
[<ffffffff80012562>] __do_softirq+0x89/0x133
[<ffffffff8005e2fc>] call_softirq+0x1c/0x28
[<ffffffff8006d636>] do_softirq+0x2c/0x7d
[<ffffffff8006d4c6>] do_IRQ+0xee/0xf7
[<ffffffff8005d615>] ret_from_intr+0x0/0xa
<EOI> [<ffffffff80064a58>] _spin_lock+0x3/0xa
[<ffffffff8855119b>] :iscsi_trgt:istd+0x64d/0x1073
[<ffffffff8025abf1>] tcp_sendpage+0x0/0x651
[<ffffffff88550b4e>] :iscsi_trgt:istd+0x0/0x1073
[<ffffffff800a2c39>] keventd_create_kthread+0x0/0xc4
[<ffffffff8003270f>] kthread+0xfe/0x132
[<ffffffff8005dfb1>] child_rip+0xa/0x11
[<ffffffff800a2c39>] keventd_create_kthread+0x0/0xc4
[<ffffffff80032611>] kthread+0x0/0x132
[<ffffffff8005dfa7>] child_rip+0x0/0x11
Kind regards, Marko Kobal
CTO, Arctur d.o.o.