quadstor stop frozen

[David Enjolras]

Hello,

I have quadstor-virt-3.2.19-rhel.x86_64 on a CentOS Linux release 7.9.2009 (Core)

after a reboot, the system load average was very high during days ( load is more than 100 )

I've try /etc/init.d/quadstor stop

But this never ends.

ps show me a  /quadstor/bin/scctl -u

process running for a long time ( started at 18h22 )

could you help ?

Thanks.

/var/log/messages shows :

Jun 15 18:22:29 dc01qquad01p systemd: Stopping QUADStor Storage Virtualization...
Jun 15 18:22:30 dc01qquad01p kernel: sd 5:0:0:0: [sdc] Synchronizing SCSI cache
Jun 15 18:22:32 dc01qquad01p kernel: sd 6:0:0:0: [sdd] Synchronizing SCSI cache
Jun 15 18:22:33 dc01qquad01p kernel: sd 9:0:0:0: [sdg] Synchronizing SCSI cache
Jun 15 18:22:33 dc01qquad01p kernel: sd 10:0:0:0: [sdh] Synchronizing SCSI cache
Jun 15 18:22:33 dc01qquad01p kernel: sd 11:0:0:0: [sdi] Synchronizing SCSI cache
Jun 15 18:22:33 dc01qquad01p kernel: sd 12:0:0:0: [sdj] Synchronizing SCSI cache
Jun 15 18:22:33 dc01qquad01p kernel: sd 13:0:0:0: [sdk] Synchronizing SCSI cache
Jun 15 18:22:34 dc01qquad01p kernel: sd 7:0:0:0: [sde] Synchronizing SCSI cache
Jun 15 18:22:34 dc01qquad01p kernel: sd 14:0:0:0: [sdl] Synchronizing SCSI cache
Jun 15 18:22:34 dc01qquad01p kernel: sd 15:0:0:0: [sdm] Synchronizing SCSI cache
Jun 15 18:22:34 dc01qquad01p kernel: sd 16:0:0:0: [sdn] Synchronizing SCSI cache
Jun 15 18:22:34 dc01qquad01p systemd: Stopping LVM2 PV scan on device 8:208...
Jun 15 18:22:34 dc01qquad01p lvm: Device 8:208 not found. Cleared from lvmetad cache.
Jun 15 18:22:34 dc01qquad01p systemd: Stopped LVM2 PV scan on device 8:208.
Jun 15 18:22:34 dc01qquad01p kernel: sd 8:0:0:0: [sdf] Synchronizing SCSI cache
Jun 15 18:22:34 dc01qquad01p kernel: sd 17:0:0:0: [sdo] Synchronizing SCSI cache
Jun 15 18:22:34 dc01qquad01p kernel: sd 18:0:0:0: [sdp] Synchronizing SCSI cache
Jun 15 18:22:36 dc01qquad01p kernel: __kern_exit:729 kern_inited; 1
Jun 15 18:22:36 dc01qquad01p kernel: __kern_exit:738 disable devices
Jun 15 18:22:36 dc01qquad01p kernel: __kern_exit:749 copy manager exit
Jun 15 18:22:36 dc01qquad01p kernel: __kern_exit:752 node cleanups wait
Jun 15 18:22:36 dc01qquad01p kernel: __kern_exit:755 node ha exit
Jun 15 18:22:36 dc01qquad01p kernel: __kern_exit:764 wait for FRLULQESX67_DS1 istate list
Jun 15 18:22:36 dc01qquad01p kernel: __kern_exit:766 done wait for FRLULQESX67_DS1 istate list
Jun 15 18:22:36 dc01qquad01p kernel: __kern_exit:764 wait for FRLULQESX70 istate list
Jun 15 18:25:09 dc01qquad01p systemd-logind: New session 4 of user root.
Jun 15 18:25:09 dc01qquad01p systemd: Started Session 4 of user root.
Jun 15 18:25:41 dc01qquad01p kernel: INFO: task gdevq_0:2992 blocked for more than 120 seconds.
Jun 15 18:25:41 dc01qquad01p kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 15 18:25:41 dc01qquad01p kernel: gdevq_0         D ffff90a559855860     0  2992      2 0x00000080
Jun 15 18:25:41 dc01qquad01p kernel: Call Trace:
Jun 15 18:25:41 dc01qquad01p kernel: [<ffffffff8658c3d9>] schedule+0x29/0x70
Jun 15 18:25:41 dc01qquad01p kernel: [<ffffffffc08ed838>] cv_wait+0x98/0x120 [coredev]
Jun 15 18:25:41 dc01qquad01p kernel: [<ffffffff85ec70a0>] ? wake_up_atomic_t+0x30/0x30
Jun 15 18:25:41 dc01qquad01p kernel: [<ffffffffc08ed8da>] __wait_on_chan+0x1a/0x20 [coredev]
Jun 15 18:25:41 dc01qquad01p kernel: [<ffffffffc0965cc8>] pgdata_post_write+0x368/0x510 [coredev]
Jun 15 18:25:41 dc01qquad01p kernel: [<ffffffff8608cf9d>] ? bio_put+0x7d/0xa0
Jun 15 18:25:41 dc01qquad01p kernel: [<ffffffffc096e1fc>] ? __tdisk_cmd_write+0x8ac/0x11f0 [coredev]
Jun 15 18:25:41 dc01qquad01p kernel: [<ffffffff85e2b59e>] ? __switch_to+0xce/0x580
Jun 15 18:25:41 dc01qquad01p kernel: [<ffffffffc096ebbc>] ? tdisk_cmd_write16+0x7c/0x290 [coredev]
Jun 15 18:25:41 dc01qquad01p kernel: [<ffffffff8658c050>] ? __schedule+0x320/0x680
Jun 15 18:25:41 dc01qquad01p kernel: [<ffffffffc09726ee>] ? tdisk_proc_cmd+0xa5e/0x16b0 [coredev]
Jun 15 18:25:41 dc01qquad01p kernel: [<ffffffffc08eda31>] ? __wait_on_chan_sig+0xc1/0x100 [coredev]
Jun 15 18:25:41 dc01qquad01p kernel: [<ffffffff85ec70a0>] ? wake_up_atomic_t+0x30/0x30
Jun 15 18:25:41 dc01qquad01p kernel: [<ffffffffc09189e0>] ? immedq_thread+0x160/0x160 [coredev]
Jun 15 18:25:41 dc01qquad01p kernel: [<ffffffffc0918af8>] ? devq_thread+0x118/0x1b0 [coredev]
Jun 15 18:25:41 dc01qquad01p kernel: [<ffffffffc09189e0>] ? immedq_thread+0x160/0x160 [coredev]
Jun 15 18:25:41 dc01qquad01p kernel: [<ffffffff85ec5fb1>] ? kthread+0xd1/0xe0
Jun 15 18:25:41 dc01qquad01p kernel: [<ffffffff85ec5ee0>] ? insert_kthread_work+0x40/0x40
Jun 15 18:25:41 dc01qquad01p kernel: [<ffffffff86599df7>] ? ret_from_fork_nospec_begin+0x21/0x21
Jun 15 18:25:41 dc01qquad01p kernel: [<ffffffff85ec5ee0>] ? insert_kthread_work+0x40/0x40
Jun 15 18:25:41 dc01qquad01p kernel: INFO: task gdevq_1:2993 blocked for more than 120 seconds.
Jun 15 18:25:41 dc01qquad01p kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 15 18:25:41 dc01qquad01p kernel: gdevq_1         D ffff90a5598568e0     0  2993      2 0x00000080
Jun 15 18:25:41 dc01qquad01p kernel: Call Trace:
Jun 15 18:25:41 dc01qquad01p kernel: [<ffffffff8658c3d9>] schedule+0x29/0x70
Jun 15 18:25:41 dc01qquad01p kernel: [<ffffffffc08ed838>] cv_wait+0x98/0x120 [coredev]
Jun 15 18:25:41 dc01qquad01p kernel: [<ffffffff85ec70a0>] ? wake_up_atomic_t+0x30/0x30
Jun 15 18:25:41 dc01qquad01p kernel: [<ffffffffc08ed931>] __wait_for_done+0x51/0x90 [coredev]
Jun 15 18:25:41 dc01qquad01p kernel: [<ffffffffc0968d3a>] __tdisk_cmd_read_int+0x61a/0xd30 [coredev]
Jun 15 18:25:41 dc01qquad01p kernel: [<ffffffffc096cb10>] ? __tdisk_cmd_read+0x60/0x270 [coredev]
Jun 15 18:25:41 dc01qquad01p kernel: [<ffffffffc0972608>] ? tdisk_proc_cmd+0x978/0x16b0 [coredev]

[QUADStor VTL Support]

Hi,

Please post a message to the virtualization group https://groups.google.com/g/quadstor-virt or you could send an email to sup...@quadstor.com

There is a new version for the virtualization software 3.2.20, you could try upgrading and see if the problem goes away

--
You received this message because you are subscribed to the Google Groups "QUADStor VTL" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quadstor-vtl...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quadstor-vtl/dec4db10-0ffb-44fb-99bd-c59451d4d340n%40googlegroups.com.