Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Bug#942579: qemu-utils: nbd disconnect/timout/system hang when writing to image mounted with qemu-nbd

131 views
Skip to first unread message

Marcel

unread,
Oct 18, 2019, 8:50:04 AM10/18/19
to
Package: qemu-utils
Version: 1:4.1-1+b3
Severity: normal

Dear Maintainer,

I experience the following syptoms since Debian buster. In Debian stretch it
works just fine:

Load Kernel Module: sudo modprobe nbd max_part=16
Create an image: qemeu-image create vdi /path/to/img-file.vdi 120G
Bind the image: qemeu-nbd -c /dev/nbd0 /path/to/img-file.vdi

Create a partion, e.g.: cfdisk /dev/nbd0
Format the partition: sudo mkfs.ext4 /dev/nbd0p1
Mount the partition: sudo mount /dev/nbd0p1 /mnt/nbd0

Now, copy over a large file, e.g. 20G. After about 10GB the procedure will stop
and there will be things like the following in /var/log/kernel.log


Oct 18 12:04:07 free kernel: [ 449.449298] block nbd1: Connection timed out
Oct 18 12:06:42 free kernel: [ 605.016877] INFO: task kworker/u2:4:142 blocked
for more than 120 seconds.
Oct 18 12:06:42 free kernel: [ 605.016882] Tainted: G OE
5.2.0-3-amd64 #1 Debian 5.2.17-1
Oct 18 12:06:42 free kernel: [ 605.016883] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 18 12:06:42 free kernel: [ 605.016885] kworker/u2:4 D 0 142 2
0x80004000
Oct 18 12:06:42 free kernel: [ 605.016904] Workqueue: writeback wb_workfn
(flush-254:3)
Oct 18 12:06:42 free kernel: [ 605.016906] Call Trace:
Oct 18 12:06:42 free kernel: [ 605.016925] ? __schedule+0x2bb/0x660
Oct 18 12:06:42 free kernel: [ 605.016931] ? blk_flush_plug_list+0xd7/0x100
Oct 18 12:06:42 free kernel: [ 605.016933] ? wbt_exit+0x30/0x30
Oct 18 12:06:42 free kernel: [ 605.016935] ? __wbt_done+0x30/0x30
Oct 18 12:06:42 free kernel: [ 605.016937] schedule+0x2f/0xa0
Oct 18 12:06:42 free kernel: [ 605.016939] io_schedule+0x12/0x40
Oct 18 12:06:42 free kernel: [ 605.016942] rq_qos_wait+0xfe/0x160
Oct 18 12:06:42 free kernel: [ 605.016955] ? karma_partition+0x210/0x210
Oct 18 12:06:42 free kernel: [ 605.016957] ? wbt_exit+0x30/0x30
Oct 18 12:06:42 free kernel: [ 605.016959] wbt_wait+0x99/0xe0
Oct 18 12:06:42 free kernel: [ 605.016962] __rq_qos_throttle+0x23/0x30
Oct 18 12:06:42 free kernel: [ 605.016964] blk_mq_make_request+0x11d/0x590
Oct 18 12:06:42 free kernel: [ 605.016973] ? dm_process_bio+0x15a/0x220
[dm_mod]
Oct 18 12:06:42 free kernel: [ 605.016975] generic_make_request+0xcf/0x310
Oct 18 12:06:42 free kernel: [ 605.016978] submit_bio+0x45/0x140
Oct 18 12:06:42 free kernel: [ 605.016998] ext4_io_submit+0x49/0x60 [ext4]
Oct 18 12:06:42 free kernel: [ 605.017014] ext4_writepages+0x707/0xeb0 [ext4]
Oct 18 12:06:42 free kernel: [ 605.017023] ? check_preempt_wakeup+0x18d/0x230
Oct 18 12:06:42 free kernel: [ 605.017028] ? do_writepages+0x41/0xd0
Oct 18 12:06:42 free kernel: [ 605.017042] ?
ext4_mark_inode_dirty+0x1d0/0x1d0 [ext4]
Oct 18 12:06:42 free kernel: [ 605.017045] do_writepages+0x41/0xd0
Oct 18 12:06:42 free kernel: [ 605.017055] ? __wake_up_common_lock+0x89/0xc0
Oct 18 12:06:42 free kernel: [ 605.017058]
__writeback_single_inode+0x3d/0x340
Oct 18 12:06:42 free kernel: [ 605.017067] writeback_sb_inodes+0x1e5/0x480
Oct 18 12:06:42 free kernel: [ 605.017070] __writeback_inodes_wb+0x5d/0xb0
Oct 18 12:06:42 free kernel: [ 605.017072] wb_writeback+0x25f/0x2f0
Oct 18 12:06:42 free kernel: [ 605.017075] wb_workfn+0x30c/0x400
Oct 18 12:06:42 free kernel: [ 605.017078] process_one_work+0x1a7/0x3b0
Oct 18 12:06:42 free kernel: [ 605.017086] worker_thread+0x1fa/0x390
Oct 18 12:06:42 free kernel: [ 605.017087] ? create_worker+0x1a0/0x1a0
Oct 18 12:06:42 free kernel: [ 605.017090] kthread+0x112/0x130
Oct 18 12:06:42 free kernel: [ 605.017092] ? __kthread_parkme+0x70/0x70
Oct 18 12:06:42 free kernel: [ 605.017094] ret_from_fork+0x35/0x40
Oct 18 12:06:42 free kernel: [ 605.017137] INFO: task kworker/0:2H:2962
blocked for more than 120 seconds.
Oct 18 12:06:42 free kernel: [ 605.017140] Tainted: G OE
5.2.0-3-amd64 #1 Debian 5.2.17-1
Oct 18 12:06:42 free kernel: [ 605.017141] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 18 12:06:42 free kernel: [ 605.017143] kworker/0:2H D 0 2962 2
0x80004000
Oct 18 12:06:42 free kernel: [ 605.017148] Workqueue: kblockd
blk_mq_timeout_work
Oct 18 12:06:42 free kernel: [ 605.017149] Call Trace:
Oct 18 12:06:42 free kernel: [ 605.017153] ? __schedule+0x2bb/0x660
Oct 18 12:06:42 free kernel: [ 605.017156] schedule+0x2f/0xa0
Oct 18 12:06:42 free kernel: [ 605.017159] schedule_preempt_disabled+0xa/0x10
Oct 18 12:06:42 free kernel: [ 605.017162] __mutex_lock.isra.8+0x2b5/0x4a0
Oct 18 12:06:42 free kernel: [ 605.017165] ?
blk_mq_dispatch_rq_list+0x48d/0x5c0
Oct 18 12:06:42 free kernel: [ 605.017171] sock_shutdown+0x58/0x80 [nbd]
Oct 18 12:06:42 free kernel: [ 605.017174] nbd_xmit_timeout+0xb7/0x150 [nbd]
Oct 18 12:06:42 free kernel: [ 605.017176] blk_mq_check_expired+0xe7/0x100
Oct 18 12:06:42 free kernel: [ 605.017179]
blk_mq_queue_tag_busy_iter+0x19a/0x2f0
Oct 18 12:06:42 free kernel: [ 605.017181] ? blk_poll+0x350/0x350
Oct 18 12:06:42 free kernel: [ 605.017182] ? __switch_to_asm+0x34/0x70
Oct 18 12:06:42 free kernel: [ 605.017184] ? blk_poll+0x350/0x350
Oct 18 12:06:42 free kernel: [ 605.017186] ? __switch_to+0x10/0x3e0
Oct 18 12:06:42 free kernel: [ 605.017188] blk_mq_timeout_work+0x54/0x120
Oct 18 12:06:42 free kernel: [ 605.017190] process_one_work+0x1a7/0x3b0
Oct 18 12:06:42 free kernel: [ 605.017191] worker_thread+0x30/0x390
Oct 18 12:06:42 free kernel: [ 605.017193] ? create_worker+0x1a0/0x1a0
Oct 18 12:06:42 free kernel: [ 605.017195] kthread+0x112/0x130
Oct 18 12:06:42 free kernel: [ 605.017197] ? __kthread_parkme+0x70/0x70
Oct 18 12:06:42 free kernel: [ 605.017199] ret_from_fork+0x35/0x40
Oct 18 12:06:42 free kernel: [ 605.017201] INFO: task jbd2/dm-3-8:3222 blocked
for more than 120 seconds.
Oct 18 12:06:42 free kernel: [ 605.017203] Tainted: G OE
5.2.0-3-amd64 #1 Debian 5.2.17-1
Oct 18 12:06:42 free kernel: [ 605.017204] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 18 12:06:42 free kernel: [ 605.017205] jbd2/dm-3-8 D 0 3222 2
0x80004000
Oct 18 12:06:42 free kernel: [ 605.017207] Call Trace:
Oct 18 12:06:42 free kernel: [ 605.017209] ? __schedule+0x2bb/0x660
Oct 18 12:06:42 free kernel: [ 605.017212] ? bit_wait_timeout+0x90/0x90
Oct 18 12:06:42 free kernel: [ 605.017213] schedule+0x2f/0xa0
Oct 18 12:06:42 free kernel: [ 605.017215] io_schedule+0x12/0x40
Oct 18 12:06:42 free kernel: [ 605.017217] bit_wait_io+0xd/0x50
Oct 18 12:06:42 free kernel: [ 605.017219] __wait_on_bit+0x73/0x90
Oct 18 12:06:42 free kernel: [ 605.017222] out_of_line_wait_on_bit+0x91/0xb0
Oct 18 12:06:42 free kernel: [ 605.017224] ? init_wait_var_entry+0x40/0x40
Oct 18 12:06:42 free kernel: [ 605.017230]
jbd2_journal_commit_transaction+0x14b5/0x1890 [jbd2]
Oct 18 12:06:42 free kernel: [ 605.017232] ? __switch_to_asm+0x40/0x70
Oct 18 12:06:42 free kernel: [ 605.017239] kjournald2+0xbd/0x270 [jbd2]
Oct 18 12:06:42 free kernel: [ 605.017242] ? finish_wait+0x80/0x80
Oct 18 12:06:42 free kernel: [ 605.017247] ? commit_timeout+0x10/0x10 [jbd2]
Oct 18 12:06:42 free kernel: [ 605.017249] kthread+0x112/0x130
Oct 18 12:06:42 free kernel: [ 605.017251] ? __kthread_parkme+0x70/0x70
Oct 18 12:06:42 free kernel: [ 605.017252] ret_from_fork+0x35/0x40
Oct 18 12:06:42 free kernel: [ 605.017254] INFO: task ext4lazyinit:3224
blocked for more than 120 seconds.
Oct 18 12:06:42 free kernel: [ 605.017256] Tainted: G OE
5.2.0-3-amd64 #1 Debian 5.2.17-1
Oct 18 12:06:42 free kernel: [ 605.017256] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 18 12:06:42 free kernel: [ 605.017258] ext4lazyinit D 0 3224 2
0x80004000
Oct 18 12:06:42 free kernel: [ 605.017259] Call Trace:
Oct 18 12:06:42 free kernel: [ 605.017261] ? __schedule+0x2bb/0x660
Oct 18 12:06:42 free kernel: [ 605.017263] ? blk_flush_plug_list+0xd7/0x100
Oct 18 12:06:42 free kernel: [ 605.017265] ? wbt_exit+0x30/0x30
Oct 18 12:06:42 free kernel: [ 605.017267] ? __wbt_done+0x30/0x30
Oct 18 12:06:42 free kernel: [ 605.017269] schedule+0x2f/0xa0
Oct 18 12:06:42 free kernel: [ 605.017271] io_schedule+0x12/0x40
Oct 18 12:06:42 free kernel: [ 605.017273] rq_qos_wait+0xfe/0x160
Oct 18 12:06:42 free kernel: [ 605.017275] ? karma_partition+0x210/0x210
Oct 18 12:06:42 free kernel: [ 605.017276] ? wbt_exit+0x30/0x30
Oct 18 12:06:42 free kernel: [ 605.017278] wbt_wait+0x99/0xe0
Oct 18 12:06:42 free kernel: [ 605.017281] __rq_qos_throttle+0x23/0x30
Oct 18 12:06:42 free kernel: [ 605.017282] blk_mq_make_request+0x11d/0x590
Oct 18 12:06:42 free kernel: [ 605.017288] ? dm_process_bio+0x15a/0x220
[dm_mod]
Oct 18 12:06:42 free kernel: [ 605.017290] generic_make_request+0xcf/0x310
Oct 18 12:06:42 free kernel: [ 605.017292] submit_bio+0x45/0x140
Oct 18 12:06:42 free kernel: [ 605.017294] blk_next_bio+0x33/0x40
Oct 18 12:06:42 free kernel: [ 605.017296]
__blkdev_issue_zero_pages+0x90/0x190
Oct 18 12:06:42 free kernel: [ 605.017297] blkdev_issue_zeroout+0xef/0x220
Oct 18 12:06:42 free kernel: [ 605.017312] ext4_init_inode_table+0x17b/0x390
[ext4]
Oct 18 12:06:42 free kernel: [ 605.017329] ext4_lazyinit_thread+0x2bd/0x3a0
[ext4]
Oct 18 12:06:42 free kernel: [ 605.017343] ?
ext4_unregister_li_request+0x60/0x60 [ext4]
Oct 18 12:06:42 free kernel: [ 605.017346] kthread+0x112/0x130
Oct 18 12:06:42 free kernel: [ 605.017348] ? __kthread_parkme+0x70/0x70
Oct 18 12:06:42 free kernel: [ 605.017349] ret_from_fork+0x35/0x40
Oct 18 12:08:43 free kernel: [ 725.788441] INFO: task kworker/u2:4:142 blocked
for more than 241 seconds.
Oct 18 12:08:43 free kernel: [ 725.788448] Tainted: G OE
5.2.0-3-amd64 #1 Debian 5.2.17-1
Oct 18 12:08:43 free kernel: [ 725.788450] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 18 12:08:43 free kernel: [ 725.788453] kworker/u2:4 D 0 142 2
0x80004000
Oct 18 12:08:43 free kernel: [ 725.788465] Workqueue: writeback wb_workfn
(flush-254:3)
Oct 18 12:08:43 free kernel: [ 725.788468] Call Trace:
Oct 18 12:08:43 free kernel: [ 725.788479] ? __schedule+0x2bb/0x660
Oct 18 12:08:43 free kernel: [ 725.788483] ? blk_flush_plug_list+0xd7/0x100
Oct 18 12:08:43 free kernel: [ 725.788487] ? wbt_exit+0x30/0x30
Oct 18 12:08:43 free kernel: [ 725.788490] ? __wbt_done+0x30/0x30
Oct 18 12:08:43 free kernel: [ 725.788493] schedule+0x2f/0xa0
Oct 18 12:08:43 free kernel: [ 725.788496] io_schedule+0x12/0x40
Oct 18 12:08:43 free kernel: [ 725.788500] rq_qos_wait+0xfe/0x160
Oct 18 12:08:43 free kernel: [ 725.788503] ? karma_partition+0x210/0x210
Oct 18 12:08:43 free kernel: [ 725.788506] ? wbt_exit+0x30/0x30
Oct 18 12:08:43 free kernel: [ 725.788509] wbt_wait+0x99/0xe0
Oct 18 12:08:43 free kernel: [ 725.788513] __rq_qos_throttle+0x23/0x30
Oct 18 12:08:43 free kernel: [ 725.788516] blk_mq_make_request+0x11d/0x590
Oct 18 12:08:43 free kernel: [ 725.788529] ? dm_process_bio+0x15a/0x220
[dm_mod]
Oct 18 12:08:43 free kernel: [ 725.788533] generic_make_request+0xcf/0x310
Oct 18 12:08:43 free kernel: [ 725.788536] submit_bio+0x45/0x140
Oct 18 12:08:43 free kernel: [ 725.788568] ext4_io_submit+0x49/0x60 [ext4]
Oct 18 12:08:43 free kernel: [ 725.788592] ext4_writepages+0x707/0xeb0 [ext4]
Oct 18 12:08:43 free kernel: [ 725.788600] ? check_preempt_wakeup+0x18d/0x230
Oct 18 12:08:43 free kernel: [ 725.788606] ? do_writepages+0x41/0xd0
Oct 18 12:08:43 free kernel: [ 725.788627] ?
ext4_mark_inode_dirty+0x1d0/0x1d0 [ext4]
Oct 18 12:08:43 free kernel: [ 725.788631] do_writepages+0x41/0xd0
Oct 18 12:08:43 free kernel: [ 725.788635] ? __wake_up_common_lock+0x89/0xc0
Oct 18 12:08:43 free kernel: [ 725.788640]
__writeback_single_inode+0x3d/0x340
Oct 18 12:08:43 free kernel: [ 725.788644] writeback_sb_inodes+0x1e5/0x480
Oct 18 12:08:43 free kernel: [ 725.788648] __writeback_inodes_wb+0x5d/0xb0
Oct 18 12:08:43 free kernel: [ 725.788652] wb_writeback+0x25f/0x2f0
Oct 18 12:08:43 free kernel: [ 725.788656] wb_workfn+0x30c/0x400
Oct 18 12:08:43 free kernel: [ 725.788660] process_one_work+0x1a7/0x3b0
Oct 18 12:08:43 free kernel: [ 725.788663] worker_thread+0x1fa/0x390
Oct 18 12:08:43 free kernel: [ 725.788665] ? create_worker+0x1a0/0x1a0
Oct 18 12:08:43 free kernel: [ 725.788668] kthread+0x112/0x130
Oct 18 12:08:43 free kernel: [ 725.788672] ? __kthread_parkme+0x70/0x70
Oct 18 12:08:43 free kernel: [ 725.788675] ret_from_fork+0x35/0x40
Oct 18 12:08:43 free kernel: [ 725.788726] INFO: task kworker/0:2H:2962
blocked for more than 241 seconds.
Oct 18 12:08:43 free kernel: [ 725.788729] Tainted: G OE
5.2.0-3-amd64 #1 Debian 5.2.17-1
Oct 18 12:08:43 free kernel: [ 725.788730] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 18 12:08:43 free kernel: [ 725.788732] kworker/0:2H D 0 2962 2
0x80004000
Oct 18 12:08:43 free kernel: [ 725.788738] Workqueue: kblockd
blk_mq_timeout_work
Oct 18 12:08:43 free kernel: [ 725.788740] Call Trace:
Oct 18 12:08:43 free kernel: [ 725.788744] ? __schedule+0x2bb/0x660
Oct 18 12:08:43 free kernel: [ 725.788748] schedule+0x2f/0xa0
Oct 18 12:08:43 free kernel: [ 725.788751] schedule_preempt_disabled+0xa/0x10
Oct 18 12:08:43 free kernel: [ 725.788754] __mutex_lock.isra.8+0x2b5/0x4a0
Oct 18 12:08:43 free kernel: [ 725.788758] ?
blk_mq_dispatch_rq_list+0x48d/0x5c0
Oct 18 12:08:43 free kernel: [ 725.788766] sock_shutdown+0x58/0x80 [nbd]
Oct 18 12:08:43 free kernel: [ 725.788771] nbd_xmit_timeout+0xb7/0x150 [nbd]
Oct 18 12:08:43 free kernel: [ 725.788774] blk_mq_check_expired+0xe7/0x100
Oct 18 12:08:43 free kernel: [ 725.788778]
blk_mq_queue_tag_busy_iter+0x19a/0x2f0
Oct 18 12:08:43 free kernel: [ 725.788781] ? blk_poll+0x350/0x350
Oct 18 12:08:43 free kernel: [ 725.788783] ? __switch_to_asm+0x34/0x70
Oct 18 12:08:43 free kernel: [ 725.788786] ? blk_poll+0x350/0x350
Oct 18 12:08:43 free kernel: [ 725.788790] ? __switch_to+0x10/0x3e0
Oct 18 12:08:43 free kernel: [ 725.788793] blk_mq_timeout_work+0x54/0x120
Oct 18 12:08:43 free kernel: [ 725.788796] process_one_work+0x1a7/0x3b0
Oct 18 12:08:43 free kernel: [ 725.788798] worker_thread+0x30/0x390
Oct 18 12:08:43 free kernel: [ 725.788801] ? create_worker+0x1a0/0x1a0
Oct 18 12:08:43 free kernel: [ 725.788803] kthread+0x112/0x130
Oct 18 12:08:43 free kernel: [ 725.788806] ? __kthread_parkme+0x70/0x70
Oct 18 12:08:43 free kernel: [ 725.788809] ret_from_fork+0x35/0x40
Oct 18 12:08:43 free kernel: [ 725.788812] INFO: task jbd2/dm-3-8:3222 blocked
for more than 241 seconds.
Oct 18 12:08:43 free kernel: [ 725.788815] Tainted: G OE
5.2.0-3-amd64 #1 Debian 5.2.17-1
Oct 18 12:08:43 free kernel: [ 725.788816] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 18 12:08:43 free kernel: [ 725.788818] jbd2/dm-3-8 D 0 3222 2
0x80004000
Oct 18 12:08:43 free kernel: [ 725.788820] Call Trace:
Oct 18 12:08:43 free kernel: [ 725.788824] ? __schedule+0x2bb/0x660
Oct 18 12:08:43 free kernel: [ 725.788828] ? bit_wait_timeout+0x90/0x90
Oct 18 12:08:43 free kernel: [ 725.788830] schedule+0x2f/0xa0
Oct 18 12:08:43 free kernel: [ 725.788833] io_schedule+0x12/0x40
Oct 18 12:08:43 free kernel: [ 725.788836] bit_wait_io+0xd/0x50
Oct 18 12:08:43 free kernel: [ 725.788840] __wait_on_bit+0x73/0x90
Oct 18 12:08:43 free kernel: [ 725.788843] out_of_line_wait_on_bit+0x91/0xb0
Oct 18 12:08:43 free kernel: [ 725.788847] ? init_wait_var_entry+0x40/0x40
Oct 18 12:08:43 free kernel: [ 725.788857]
jbd2_journal_commit_transaction+0x14b5/0x1890 [jbd2]
Oct 18 12:08:43 free kernel: [ 725.788860] ? __switch_to_asm+0x40/0x70
Oct 18 12:08:43 free kernel: [ 725.788870] kjournald2+0xbd/0x270 [jbd2]
Oct 18 12:08:43 free kernel: [ 725.788875] ? finish_wait+0x80/0x80
Oct 18 12:08:43 free kernel: [ 725.788882] ? commit_timeout+0x10/0x10 [jbd2]
Oct 18 12:08:43 free kernel: [ 725.788885] kthread+0x112/0x130
Oct 18 12:08:43 free kernel: [ 725.788888] ? __kthread_parkme+0x70/0x70
Oct 18 12:08:43 free kernel: [ 725.788891] ret_from_fork+0x35/0x40
Oct 18 12:08:43 free kernel: [ 725.788894] INFO: task ext4lazyinit:3224
blocked for more than 241 seconds.
Oct 18 12:08:43 free kernel: [ 725.788896] Tainted: G OE
5.2.0-3-amd64 #1 Debian 5.2.17-1
Oct 18 12:08:43 free kernel: [ 725.788897] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 18 12:08:43 free kernel: [ 725.788899] ext4lazyinit D 0 3224 2
0x80004000
Oct 18 12:08:43 free kernel: [ 725.788901] Call Trace:
Oct 18 12:08:43 free kernel: [ 725.788905] ? __schedule+0x2bb/0x660
Oct 18 12:08:43 free kernel: [ 725.788908] ? blk_flush_plug_list+0xd7/0x100
Oct 18 12:08:43 free kernel: [ 725.788911] ? wbt_exit+0x30/0x30
Oct 18 12:08:43 free kernel: [ 725.788914] ? __wbt_done+0x30/0x30
Oct 18 12:08:43 free kernel: [ 725.788917] schedule+0x2f/0xa0
Oct 18 12:08:43 free kernel: [ 725.788920] io_schedule+0x12/0x40
Oct 18 12:08:43 free kernel: [ 725.788923] rq_qos_wait+0xfe/0x160
Oct 18 12:08:43 free kernel: [ 725.788926] ? karma_partition+0x210/0x210
Oct 18 12:08:43 free kernel: [ 725.788929] ? wbt_exit+0x30/0x30
Oct 18 12:08:43 free kernel: [ 725.788931] wbt_wait+0x99/0xe0
Oct 18 12:08:43 free kernel: [ 725.788935] __rq_qos_throttle+0x23/0x30
Oct 18 12:08:43 free kernel: [ 725.788938] blk_mq_make_request+0x11d/0x590
Oct 18 12:08:43 free kernel: [ 725.788946] ? dm_process_bio+0x15a/0x220
[dm_mod]
Oct 18 12:08:43 free kernel: [ 725.788949] generic_make_request+0xcf/0x310
Oct 18 12:08:43 free kernel: [ 725.788953] submit_bio+0x45/0x140
Oct 18 12:08:43 free kernel: [ 725.788955] blk_next_bio+0x33/0x40
Oct 18 12:08:43 free kernel: [ 725.788958]
__blkdev_issue_zero_pages+0x90/0x190
Oct 18 12:08:43 free kernel: [ 725.788960] blkdev_issue_zeroout+0xef/0x220
Oct 18 12:08:43 free kernel: [ 725.788984] ext4_init_inode_table+0x17b/0x390
[ext4]
Oct 18 12:08:43 free kernel: [ 725.789010] ext4_lazyinit_thread+0x2bd/0x3a0
[ext4]
Oct 18 12:08:43 free kernel: [ 725.789035] ?
ext4_unregister_li_request+0x60/0x60 [ext4]
Oct 18 12:08:43 free kernel: [ 725.789038] kthread+0x112/0x130
Oct 18 12:08:43 free kernel: [ 725.789041] ? __kthread_parkme+0x70/0x70
Oct 18 12:08:43 free kernel: [ 725.789044] ret_from_fork+0x35/0x40
Oct 18 12:10:44 free kernel: [ 846.559910] INFO: task kworker/u2:4:142 blocked
for more than 362 seconds.
Oct 18 12:10:44 free kernel: [ 846.559917] Tainted: G OE
5.2.0-3-amd64 #1 Debian 5.2.17-1
Oct 18 12:10:44 free kernel: [ 846.559918] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 18 12:10:44 free kernel: [ 846.559921] kworker/u2:4 D 0 142 2
0x80004000
Oct 18 12:10:44 free kernel: [ 846.559931] Workqueue: writeback wb_workfn
(flush-254:3)
Oct 18 12:10:44 free kernel: [ 846.559933] Call Trace:
Oct 18 12:10:44 free kernel: [ 846.559943] ? __schedule+0x2bb/0x660
Oct 18 12:10:44 free kernel: [ 846.559946] ? blk_flush_plug_list+0xd7/0x100
Oct 18 12:10:44 free kernel: [ 846.559949] ? wbt_exit+0x30/0x30
Oct 18 12:10:44 free kernel: [ 846.559952] ? __wbt_done+0x30/0x30
Oct 18 12:10:44 free kernel: [ 846.559954] schedule+0x2f/0xa0
Oct 18 12:10:44 free kernel: [ 846.559957] io_schedule+0x12/0x40
Oct 18 12:10:44 free kernel: [ 846.559960] rq_qos_wait+0xfe/0x160
Oct 18 12:10:44 free kernel: [ 846.559963] ? karma_partition+0x210/0x210
Oct 18 12:10:44 free kernel: [ 846.559966] ? wbt_exit+0x30/0x30
Oct 18 12:10:44 free kernel: [ 846.559968] wbt_wait+0x99/0xe0
Oct 18 12:10:44 free kernel: [ 846.559971] __rq_qos_throttle+0x23/0x30
Oct 18 12:10:44 free kernel: [ 846.559974] blk_mq_make_request+0x11d/0x590
Oct 18 12:10:44 free kernel: [ 846.559984] ? dm_process_bio+0x15a/0x220
[dm_mod]
Oct 18 12:10:44 free kernel: [ 846.559987] generic_make_request+0xcf/0x310
Oct 18 12:10:44 free kernel: [ 846.559990] submit_bio+0x45/0x140
Oct 18 12:10:44 free kernel: [ 846.560016] ext4_io_submit+0x49/0x60 [ext4]
Oct 18 12:10:44 free kernel: [ 846.560035] ext4_writepages+0x707/0xeb0 [ext4]
Oct 18 12:10:44 free kernel: [ 846.560042] ? check_preempt_wakeup+0x18d/0x230
Oct 18 12:10:44 free kernel: [ 846.560047] ? do_writepages+0x41/0xd0
Oct 18 12:10:44 free kernel: [ 846.560064] ?
ext4_mark_inode_dirty+0x1d0/0x1d0 [ext4]
Oct 18 12:10:44 free kernel: [ 846.560067] do_writepages+0x41/0xd0
Oct 18 12:10:44 free kernel: [ 846.560071] ? __wake_up_common_lock+0x89/0xc0
Oct 18 12:10:44 free kernel: [ 846.560075]
__writeback_single_inode+0x3d/0x340
Oct 18 12:10:44 free kernel: [ 846.560078] writeback_sb_inodes+0x1e5/0x480
Oct 18 12:10:44 free kernel: [ 846.560082] __writeback_inodes_wb+0x5d/0xb0
Oct 18 12:10:44 free kernel: [ 846.560085] wb_writeback+0x25f/0x2f0
Oct 18 12:10:44 free kernel: [ 846.560089] wb_workfn+0x30c/0x400
Oct 18 12:10:44 free kernel: [ 846.560092] process_one_work+0x1a7/0x3b0
Oct 18 12:10:44 free kernel: [ 846.560095] worker_thread+0x1fa/0x390
Oct 18 12:10:44 free kernel: [ 846.560097] ? create_worker+0x1a0/0x1a0
Oct 18 12:10:44 free kernel: [ 846.560100] kthread+0x112/0x130
Oct 18 12:10:44 free kernel: [ 846.560103] ? __kthread_parkme+0x70/0x70
Oct 18 12:10:44 free kernel: [ 846.560106] ret_from_fork+0x35/0x40
Oct 18 12:10:44 free kernel: [ 846.560147] INFO: task kworker/0:2H:2962
blocked for more than 362 seconds.
Oct 18 12:10:44 free kernel: [ 846.560150] Tainted: G OE
5.2.0-3-amd64 #1 Debian 5.2.17-1
Oct 18 12:10:44 free kernel: [ 846.560151] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 18 12:10:44 free kernel: [ 846.560153] kworker/0:2H D 0 2962 2
0x80004000
Oct 18 12:10:44 free kernel: [ 846.560158] Workqueue: kblockd
blk_mq_timeout_work
Oct 18 12:10:44 free kernel: [ 846.560159] Call Trace:
Oct 18 12:10:44 free kernel: [ 846.560163] ? __schedule+0x2bb/0x660
Oct 18 12:10:44 free kernel: [ 846.560166] schedule+0x2f/0xa0
Oct 18 12:10:44 free kernel: [ 846.560169] schedule_preempt_disabled+0xa/0x10
Oct 18 12:10:44 free kernel: [ 846.560172] __mutex_lock.isra.8+0x2b5/0x4a0
Oct 18 12:10:44 free kernel: [ 846.560175] ?
blk_mq_dispatch_rq_list+0x48d/0x5c0
Oct 18 12:10:44 free kernel: [ 846.560182] sock_shutdown+0x58/0x80 [nbd]
Oct 18 12:10:44 free kernel: [ 846.560187] nbd_xmit_timeout+0xb7/0x150 [nbd]
Oct 18 12:10:44 free kernel: [ 846.560189] blk_mq_check_expired+0xe7/0x100
Oct 18 12:10:44 free kernel: [ 846.560192]
blk_mq_queue_tag_busy_iter+0x19a/0x2f0
Oct 18 12:10:44 free kernel: [ 846.560195] ? blk_poll+0x350/0x350
Oct 18 12:10:44 free kernel: [ 846.560197] ? __switch_to_asm+0x34/0x70
Oct 18 12:10:44 free kernel: [ 846.560199] ? blk_poll+0x350/0x350
Oct 18 12:10:44 free kernel: [ 846.560201] ? __switch_to+0x10/0x3e0
Oct 18 12:10:44 free kernel: [ 846.560204] blk_mq_timeout_work+0x54/0x120
Oct 18 12:10:44 free kernel: [ 846.560206] process_one_work+0x1a7/0x3b0
Oct 18 12:10:44 free kernel: [ 846.560208] worker_thread+0x30/0x390
Oct 18 12:10:44 free kernel: [ 846.560210] ? create_worker+0x1a0/0x1a0
Oct 18 12:10:44 free kernel: [ 846.560212] kthread+0x112/0x130
Oct 18 12:10:44 free kernel: [ 846.560226] ? __kthread_parkme+0x70/0x70
Oct 18 12:10:44 free kernel: [ 846.560228] ret_from_fork+0x35/0x40
Oct 18 12:35:42 free kernel: [ 0.000000] Linux version 5.2.0-3-amd64
(debian...@lists.debian.org) (gcc version 8.3.0 (Debian 8.3.0-23)) #1 SMP
Debian 5.2.17-1 (2019-10-06)
Oct 18 12:35:42 free kernel: [ 0.000000] Command line:
BOOT_IMAGE=/vmlinuz-5.2.0-3-amd64 root=/dev/mapper/test--lvm--vg-root ro quiet


And/or this:

Oct 13 11:19:03 free kernel: [ 213.420917] block nbd1: NBD_DISCONNECT
Oct 13 11:19:03 free kernel: [ 213.421006] block nbd1: Disconnected due to
user request.
Oct 13 11:19:03 free kernel: [ 213.421983] print_req_error: I/O error, dev
nbd1, sector 0 flags 0
Oct 13 11:19:03 free kernel: [ 213.421991] Buffer I/O error on dev nbd1,
logical block 0, async page read
Oct 13 11:19:03 free kernel: [ 213.422011] print_req_error: I/O error, dev
nbd1, sector 1 flags 0
Oct 13 11:19:03 free kernel: [ 213.422014] Buffer I/O error on dev nbd1,
logical block 1, async page read
Oct 13 11:19:03 free kernel: [ 213.422024] print_req_error: I/O error, dev
nbd1, sector 2 flags 0
Oct 13 11:19:03 free kernel: [ 213.422026] Buffer I/O error on dev nbd1,
logical block 2, async page read
Oct 13 11:19:03 free kernel: [ 213.422036] print_req_error: I/O error, dev
nbd1, sector 3 flags 0
Oct 13 11:19:03 free kernel: [ 213.422038] Buffer I/O error on dev nbd1,
logical block 3, async page read
Oct 13 11:19:03 free kernel: [ 213.422048] print_req_error: I/O error, dev
nbd1, sector 4 flags 0
Oct 13 11:19:03 free kernel: [ 213.422050] Buffer I/O error on dev nbd1,
logical block 4, async page read
Oct 13 11:19:03 free kernel: [ 213.422059] print_req_error: I/O error, dev
nbd1, sector 5 flags 0
Oct 13 11:19:03 free kernel: [ 213.422062] Buffer I/O error on dev nbd1,
logical block 5, async page read
Oct 13 11:19:03 free kernel: [ 213.422071] print_req_error: I/O error, dev
nbd1, sector 6 flags 0
Oct 13 11:19:03 free kernel: [ 213.422073] Buffer I/O error on dev nbd1,
logical block 6, async page read
Oct 13 11:19:03 free kernel: [ 213.422101] print_req_error: I/O error, dev
nbd1, sector 7 flags 0
Oct 13 11:19:03 free kernel: [ 213.422103] Buffer I/O error on dev nbd1,
logical block 7, async page read
Oct 13 11:19:03 free kernel: [ 213.422116] print_req_error: I/O error, dev
nbd1, sector 0 flags 0
Oct 13 11:19:03 free kernel: [ 213.422118] Buffer I/O error on dev nbd1,
logical block 0, async page read
Oct 13 11:19:03 free kernel: [ 213.422125] print_req_error: I/O error, dev
nbd1, sector 1 flags 0
Oct 13 11:19:03 free kernel: [ 213.422127] Buffer I/O error on dev nbd1,
logical block 1, async page read
Oct 13 11:19:03 free kernel: [ 213.422251] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:19:03 free kernel: [ 213.422421] Dev nbd1: unable to read RDB block

0
Oct 13 11:19:03 free kernel: [ 213.422599] nbd1: unable to read partition
table
Oct 13 11:19:03 free kernel: [ 213.422829] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:19:03 free kernel: [ 213.423002] Dev nbd1: unable to read RDB block

0
Oct 13 11:19:03 free kernel: [ 213.428372] nbd1: unable to read partition
table
Oct 13 11:19:03 free kernel: [ 213.436129] block nbd0: NBD_DISCONNECT
Oct 13 11:19:03 free kernel: [ 213.457035] block nbd1: NBD_DISCONNECT
Oct 13 11:19:03 free kernel: [ 213.461999] block nbd0: NBD_DISCONNECT
Oct 13 11:19:03 free kernel: [ 213.826134] block nbd1: NBD_DISCONNECT
Oct 13 11:19:03 free kernel: [ 213.832202] block nbd0: NBD_DISCONNECT
Oct 13 11:19:03 free kernel: [ 213.852389] block nbd1: NBD_DISCONNECT
Oct 13 11:19:03 free kernel: [ 213.857512] block nbd0: NBD_DISCONNECT
Oct 13 11:22:14 free kernel: [ 404.790015] block nbd1: NBD_DISCONNECT
Oct 13 11:22:14 free kernel: [ 404.797440] block nbd0: NBD_DISCONNECT
Oct 13 11:22:14 free kernel: [ 404.821874] block nbd1: NBD_DISCONNECT
Oct 13 11:22:14 free kernel: [ 404.828243] block nbd0: NBD_DISCONNECT
Oct 13 11:24:37 free kernel: [ 547.343900] block nbd1: NBD_DISCONNECT
Oct 13 11:24:37 free kernel: [ 547.343977] block nbd1: Disconnected due to
user request.
Oct 13 11:24:37 free kernel: [ 547.344914] print_req_error: 182 callbacks
suppressed
Oct 13 11:24:37 free kernel: [ 547.344917] print_req_error: I/O error, dev
nbd1, sector 0 flags 0
Oct 13 11:24:37 free kernel: [ 547.344923] buffer_io_error: 182 callbacks
suppressed
Oct 13 11:24:37 free kernel: [ 547.344925] Buffer I/O error on dev nbd1,
logical block 0, async page read
Oct 13 11:24:37 free kernel: [ 547.345114] print_req_error: I/O error, dev
nbd1, sector 1 flags 0
Oct 13 11:24:37 free kernel: [ 547.345119] Buffer I/O error on dev nbd1,
logical block 1, async page read
Oct 13 11:24:37 free kernel: [ 547.345129] print_req_error: I/O error, dev
nbd1, sector 2 flags 0
Oct 13 11:24:37 free kernel: [ 547.345131] Buffer I/O error on dev nbd1,
logical block 2, async page read
Oct 13 11:24:37 free kernel: [ 547.345139] print_req_error: I/O error, dev
nbd1, sector 3 flags 0
Oct 13 11:24:37 free kernel: [ 547.345141] Buffer I/O error on dev nbd1,
logical block 3, async page read
Oct 13 11:24:37 free kernel: [ 547.345151] print_req_error: I/O error, dev
nbd1, sector 4 flags 0
Oct 13 11:24:37 free kernel: [ 547.345153] Buffer I/O error on dev nbd1,
logical block 4, async page read
Oct 13 11:24:37 free kernel: [ 547.345161] print_req_error: I/O error, dev
nbd1, sector 5 flags 0
Oct 13 11:24:37 free kernel: [ 547.345163] Buffer I/O error on dev nbd1,
logical block 5, async page read
Oct 13 11:24:37 free kernel: [ 547.345171] print_req_error: I/O error, dev
nbd1, sector 6 flags 0
Oct 13 11:24:37 free kernel: [ 547.345173] Buffer I/O error on dev nbd1,
logical block 6, async page read
Oct 13 11:24:37 free kernel: [ 547.345185] print_req_error: I/O error, dev
nbd1, sector 7 flags 0
Oct 13 11:24:37 free kernel: [ 547.345187] Buffer I/O error on dev nbd1,
logical block 7, async page read
Oct 13 11:24:37 free kernel: [ 547.345203] print_req_error: I/O error, dev
nbd1, sector 0 flags 0
Oct 13 11:24:37 free kernel: [ 547.345205] Buffer I/O error on dev nbd1,
logical block 0, async page read
Oct 13 11:24:37 free kernel: [ 547.345213] print_req_error: I/O error, dev
nbd1, sector 1 flags 0
Oct 13 11:24:37 free kernel: [ 547.345215] Buffer I/O error on dev nbd1,
logical block 1, async page read
Oct 13 11:24:37 free kernel: [ 547.345357] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:24:37 free kernel: [ 547.345555] Dev nbd1: unable to read RDB block

0
Oct 13 11:24:37 free kernel: [ 547.345763] nbd1: unable to read partition
table
Oct 13 11:24:37 free kernel: [ 547.346034] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:24:37 free kernel: [ 547.346231] Dev nbd1: unable to read RDB block

0
Oct 13 11:24:37 free kernel: [ 547.346433] nbd1: unable to read partition
table
Oct 13 11:25:27 free kernel: [ 597.462406] block nbd1: NBD_DISCONNECT
Oct 13 11:25:27 free kernel: [ 597.462437] block nbd1: Disconnected due to
user request.
Oct 13 11:25:27 free kernel: [ 597.462721] print_req_error: 182 callbacks
suppressed
Oct 13 11:25:27 free kernel: [ 597.462722] print_req_error: I/O error, dev
nbd1, sector 0 flags 0
Oct 13 11:25:27 free kernel: [ 597.462723] buffer_io_error: 182 callbacks
suppressed
Oct 13 11:25:27 free kernel: [ 597.462724] Buffer I/O error on dev nbd1,
logical block 0, async page read
Oct 13 11:25:27 free kernel: [ 597.462730] print_req_error: I/O error, dev
nbd1, sector 1 flags 0
Oct 13 11:25:27 free kernel: [ 597.462731] Buffer I/O error on dev nbd1,
logical block 1, async page read
Oct 13 11:25:27 free kernel: [ 597.462734] print_req_error: I/O error, dev
nbd1, sector 2 flags 0
Oct 13 11:25:27 free kernel: [ 597.462734] Buffer I/O error on dev nbd1,
logical block 2, async page read
Oct 13 11:25:27 free kernel: [ 597.462737] print_req_error: I/O error, dev
nbd1, sector 3 flags 0
Oct 13 11:25:27 free kernel: [ 597.462737] Buffer I/O error on dev nbd1,
logical block 3, async page read
Oct 13 11:25:27 free kernel: [ 597.462740] print_req_error: I/O error, dev
nbd1, sector 4 flags 0
Oct 13 11:25:27 free kernel: [ 597.462741] Buffer I/O error on dev nbd1,
logical block 4, async page read
Oct 13 11:25:27 free kernel: [ 597.462743] print_req_error: I/O error, dev
nbd1, sector 5 flags 0
Oct 13 11:25:27 free kernel: [ 597.462744] Buffer I/O error on dev nbd1,
logical block 5, async page read
Oct 13 11:25:27 free kernel: [ 597.462746] print_req_error: I/O error, dev
nbd1, sector 6 flags 0
Oct 13 11:25:27 free kernel: [ 597.462747] Buffer I/O error on dev nbd1,
logical block 6, async page read
Oct 13 11:25:27 free kernel: [ 597.462751] print_req_error: I/O error, dev
nbd1, sector 7 flags 0
Oct 13 11:25:27 free kernel: [ 597.462751] Buffer I/O error on dev nbd1,
logical block 7, async page read
Oct 13 11:25:27 free kernel: [ 597.462757] print_req_error: I/O error, dev
nbd1, sector 0 flags 0
Oct 13 11:25:27 free kernel: [ 597.462757] Buffer I/O error on dev nbd1,
logical block 0, async page read
Oct 13 11:25:27 free kernel: [ 597.462760] print_req_error: I/O error, dev
nbd1, sector 1 flags 0
Oct 13 11:25:27 free kernel: [ 597.462760] Buffer I/O error on dev nbd1,
logical block 1, async page read
Oct 13 11:25:27 free kernel: [ 597.462805] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:25:27 free kernel: [ 597.462867] Dev nbd1: unable to read RDB block

0
Oct 13 11:25:27 free kernel: [ 597.462932] nbd1: unable to read partition
table
Oct 13 11:25:27 free kernel: [ 597.464997] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:25:27 free kernel: [ 597.465102] Dev nbd1: unable to read RDB block

0
Oct 13 11:25:27 free kernel: [ 597.465168] nbd1: unable to read partition
table
Oct 13 11:25:27 free kernel: [ 597.467891] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:25:27 free kernel: [ 597.467954] Dev nbd1: unable to read RDB block

0
Oct 13 11:25:27 free kernel: [ 597.468021] nbd1: unable to read partition
table
Oct 13 11:25:27 free kernel: [ 597.470461] block nbd0: NBD_DISCONNECT
Oct 13 11:25:27 free kernel: [ 597.471099] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:25:27 free kernel: [ 597.471162] Dev nbd1: unable to read RDB block

0
Oct 13 11:25:27 free kernel: [ 597.471227] nbd1: unable to read partition
table
Oct 13 11:25:27 free kernel: [ 597.471312] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:25:27 free kernel: [ 597.471374] Dev nbd1: unable to read RDB block

0
Oct 13 11:25:27 free kernel: [ 597.471438] nbd1: unable to read partition
table
Oct 13 11:25:27 free kernel: [ 597.474762] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:25:27 free kernel: [ 597.474824] Dev nbd1: unable to read RDB block

0
Oct 13 11:25:27 free kernel: [ 597.474889] nbd1: unable to read partition
table
Oct 13 11:25:27 free kernel: [ 597.484450] block nbd1: NBD_DISCONNECT
Oct 13 11:25:27 free kernel: [ 597.486235] block nbd0: NBD_DISCONNECT
Oct 13 11:26:20 free kernel: [ 650.829740] block nbd1: NBD_DISCONNECT
Oct 13 11:26:20 free kernel: [ 650.829838] block nbd1: Disconnected due to
user request.
Oct 13 11:26:20 free kernel: [ 650.830814] print_req_error: 575 callbacks
suppressed
Oct 13 11:26:20 free kernel: [ 650.830817] print_req_error: I/O error, dev
nbd1, sector 0 flags 0
Oct 13 11:26:20 free kernel: [ 650.830824] buffer_io_error: 574 callbacks
suppressed
Oct 13 11:26:20 free kernel: [ 650.830826] Buffer I/O error on dev nbd1,
logical block 0, async page read
Oct 13 11:26:20 free kernel: [ 650.830846] print_req_error: I/O error, dev
nbd1, sector 1 flags 0
Oct 13 11:26:20 free kernel: [ 650.830849] Buffer I/O error on dev nbd1,
logical block 1, async page read
Oct 13 11:26:20 free kernel: [ 650.830860] print_req_error: I/O error, dev
nbd1, sector 2 flags 0
Oct 13 11:26:20 free kernel: [ 650.830862] Buffer I/O error on dev nbd1,
logical block 2, async page read
Oct 13 11:26:20 free kernel: [ 650.830872] print_req_error: I/O error, dev
nbd1, sector 3 flags 0
Oct 13 11:26:20 free kernel: [ 650.830874] Buffer I/O error on dev nbd1,
logical block 3, async page read
Oct 13 11:26:20 free kernel: [ 650.830884] print_req_error: I/O error, dev
nbd1, sector 4 flags 0
Oct 13 11:26:20 free kernel: [ 650.830886] Buffer I/O error on dev nbd1,
logical block 4, async page read
Oct 13 11:26:20 free kernel: [ 650.830895] print_req_error: I/O error, dev
nbd1, sector 5 flags 0
Oct 13 11:26:20 free kernel: [ 650.830898] Buffer I/O error on dev nbd1,
logical block 5, async page read
Oct 13 11:26:20 free kernel: [ 650.830907] print_req_error: I/O error, dev
nbd1, sector 6 flags 0
Oct 13 11:26:20 free kernel: [ 650.830909] Buffer I/O error on dev nbd1,
logical block 6, async page read
Oct 13 11:26:20 free kernel: [ 650.830924] print_req_error: I/O error, dev
nbd1, sector 7 flags 0
Oct 13 11:26:20 free kernel: [ 650.830926] Buffer I/O error on dev nbd1,
logical block 7, async page read
Oct 13 11:26:20 free kernel: [ 650.830944] print_req_error: I/O error, dev
nbd1, sector 0 flags 0
Oct 13 11:26:20 free kernel: [ 650.830947] Buffer I/O error on dev nbd1,
logical block 0, async page read
Oct 13 11:26:20 free kernel: [ 650.830956] print_req_error: I/O error, dev
nbd1, sector 1 flags 0
Oct 13 11:26:20 free kernel: [ 650.830959] Buffer I/O error on dev nbd1,
logical block 1, async page read
Oct 13 11:26:20 free kernel: [ 650.831169] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:26:20 free kernel: [ 650.831402] Dev nbd1: unable to read RDB block

0
Oct 13 11:26:20 free kernel: [ 650.831644] nbd1: unable to read partition
table
Oct 13 11:26:20 free kernel: [ 650.831960] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:26:20 free kernel: [ 650.832193] Dev nbd1: unable to read RDB block

0
Oct 13 11:26:20 free kernel: [ 650.832434] nbd1: unable to read partition
table
Oct 13 11:26:20 free kernel: [ 650.850299] block nbd0: NBD_DISCONNECT
Oct 13 11:26:20 free kernel: [ 650.866829] block nbd1: NBD_DISCONNECT
Oct 13 11:26:20 free kernel: [ 650.872238] block nbd0: NBD_DISCONNECT
Oct 13 11:27:07 free kernel: [ 697.727569] block nbd1: NBD_DISCONNECT
Oct 13 11:27:07 free kernel: [ 697.727602] block nbd1: Disconnected due to
user request.
Oct 13 11:27:07 free kernel: [ 697.727916] print_req_error: 182 callbacks
suppressed
Oct 13 11:27:07 free kernel: [ 697.727917] print_req_error: I/O error, dev
nbd1, sector 0 flags 0
Oct 13 11:27:07 free kernel: [ 697.727919] buffer_io_error: 182 callbacks
suppressed
Oct 13 11:27:07 free kernel: [ 697.727920] Buffer I/O error on dev nbd1,
logical block 0, async page read
Oct 13 11:27:07 free kernel: [ 697.727927] print_req_error: I/O error, dev
nbd1, sector 1 flags 0
Oct 13 11:27:07 free kernel: [ 697.727928] Buffer I/O error on dev nbd1,
logical block 1, async page read
Oct 13 11:27:07 free kernel: [ 697.727930] print_req_error: I/O error, dev
nbd1, sector 2 flags 0
Oct 13 11:27:07 free kernel: [ 697.727931] Buffer I/O error on dev nbd1,
logical block 2, async page read
Oct 13 11:27:07 free kernel: [ 697.727934] print_req_error: I/O error, dev
nbd1, sector 3 flags 0
Oct 13 11:27:07 free kernel: [ 697.727935] Buffer I/O error on dev nbd1,
logical block 3, async page read
Oct 13 11:27:07 free kernel: [ 697.727937] print_req_error: I/O error, dev
nbd1, sector 4 flags 0
Oct 13 11:27:07 free kernel: [ 697.727938] Buffer I/O error on dev nbd1,
logical block 4, async page read
Oct 13 11:27:07 free kernel: [ 697.727941] print_req_error: I/O error, dev
nbd1, sector 5 flags 0
Oct 13 11:27:07 free kernel: [ 697.727941] Buffer I/O error on dev nbd1,
logical block 5, async page read
Oct 13 11:27:07 free kernel: [ 697.727944] print_req_error: I/O error, dev
nbd1, sector 6 flags 0
Oct 13 11:27:07 free kernel: [ 697.727945] Buffer I/O error on dev nbd1,
logical block 6, async page read
Oct 13 11:27:07 free kernel: [ 697.727949] print_req_error: I/O error, dev
nbd1, sector 7 flags 0
Oct 13 11:27:07 free kernel: [ 697.727950] Buffer I/O error on dev nbd1,
logical block 7, async page read
Oct 13 11:27:07 free kernel: [ 697.727955] print_req_error: I/O error, dev
nbd1, sector 0 flags 0
Oct 13 11:27:07 free kernel: [ 697.727956] Buffer I/O error on dev nbd1,
logical block 0, async page read
Oct 13 11:27:07 free kernel: [ 697.727959] print_req_error: I/O error, dev
nbd1, sector 1 flags 0
Oct 13 11:27:07 free kernel: [ 697.727959] Buffer I/O error on dev nbd1,
logical block 1, async page read
Oct 13 11:27:07 free kernel: [ 697.728008] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:27:07 free kernel: [ 697.728074] Dev nbd1: unable to read RDB block

0
Oct 13 11:27:07 free kernel: [ 697.728143] nbd1: unable to read partition
table
Oct 13 11:27:07 free kernel: [ 697.728234] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:27:07 free kernel: [ 697.728300] Dev nbd1: unable to read RDB block

0
Oct 13 11:27:07 free kernel: [ 697.728367] nbd1: unable to read partition
table
Oct 13 11:27:07 free kernel: [ 697.729640] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:27:07 free kernel: [ 697.729714] Dev nbd1: unable to read RDB block

0
Oct 13 11:27:07 free kernel: [ 697.729783] nbd1: unable to read partition
table
Oct 13 11:27:07 free kernel: [ 697.730297] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:27:07 free kernel: [ 697.730362] Dev nbd1: unable to read RDB block

0
Oct 13 11:27:07 free kernel: [ 697.730430] nbd1: unable to read partition
table
Oct 13 11:27:07 free kernel: [ 697.737367] block nbd0: NBD_DISCONNECT
Oct 13 11:27:07 free kernel: [ 697.751292] block nbd1: NBD_DISCONNECT
Oct 13 11:27:07 free kernel: [ 697.756432] block nbd0: NBD_DISCONNECT
Oct 13 11:27:23 free kernel: [ 713.452457] block nbd1: NBD_DISCONNECT
Oct 13 11:27:23 free kernel: [ 713.452489] block nbd1: Disconnected due to
user request.
Oct 13 11:27:23 free kernel: [ 713.452785] print_req_error: 383 callbacks
suppressed
Oct 13 11:27:23 free kernel: [ 713.452786] print_req_error: I/O error, dev
nbd1, sector 0 flags 0
Oct 13 11:27:23 free kernel: [ 713.452788] buffer_io_error: 382 callbacks
suppressed
Oct 13 11:27:23 free kernel: [ 713.452789] Buffer I/O error on dev nbd1,
logical block 0, async page read
Oct 13 11:27:23 free kernel: [ 713.452795] print_req_error: I/O error, dev
nbd1, sector 1 flags 0
Oct 13 11:27:23 free kernel: [ 713.452796] Buffer I/O error on dev nbd1,
logical block 1, async page read
Oct 13 11:27:23 free kernel: [ 713.452799] print_req_error: I/O error, dev
nbd1, sector 2 flags 0
Oct 13 11:27:23 free kernel: [ 713.452800] Buffer I/O error on dev nbd1,
logical block 2, async page read
Oct 13 11:27:23 free kernel: [ 713.452802] print_req_error: I/O error, dev
nbd1, sector 3 flags 0
Oct 13 11:27:23 free kernel: [ 713.452803] Buffer I/O error on dev nbd1,
logical block 3, async page read
Oct 13 11:27:23 free kernel: [ 713.452806] print_req_error: I/O error, dev
nbd1, sector 4 flags 0
Oct 13 11:27:23 free kernel: [ 713.452806] Buffer I/O error on dev nbd1,
logical block 4, async page read
Oct 13 11:27:23 free kernel: [ 713.452809] print_req_error: I/O error, dev
nbd1, sector 5 flags 0
Oct 13 11:27:23 free kernel: [ 713.452810] Buffer I/O error on dev nbd1,
logical block 5, async page read
Oct 13 11:27:23 free kernel: [ 713.452812] print_req_error: I/O error, dev
nbd1, sector 6 flags 0
Oct 13 11:27:23 free kernel: [ 713.452813] Buffer I/O error on dev nbd1,
logical block 6, async page read
Oct 13 11:27:23 free kernel: [ 713.452817] print_req_error: I/O error, dev
nbd1, sector 7 flags 0
Oct 13 11:27:23 free kernel: [ 713.452818] Buffer I/O error on dev nbd1,
logical block 7, async page read
Oct 13 11:27:23 free kernel: [ 713.452823] print_req_error: I/O error, dev
nbd1, sector 0 flags 0
Oct 13 11:27:23 free kernel: [ 713.452824] Buffer I/O error on dev nbd1,
logical block 0, async page read
Oct 13 11:27:23 free kernel: [ 713.452827] print_req_error: I/O error, dev
nbd1, sector 1 flags 0
Oct 13 11:27:23 free kernel: [ 713.452827] Buffer I/O error on dev nbd1,
logical block 1, async page read
Oct 13 11:27:23 free kernel: [ 713.452887] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:27:23 free kernel: [ 713.452950] Dev nbd1: unable to read RDB block

0
Oct 13 11:27:23 free kernel: [ 713.453015] nbd1: unable to read partition
table
Oct 13 11:27:23 free kernel: [ 713.453102] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:27:23 free kernel: [ 713.453165] Dev nbd1: unable to read RDB block

0
Oct 13 11:27:23 free kernel: [ 713.453230] nbd1: unable to read partition
table
Oct 13 11:27:23 free kernel: [ 713.454379] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:27:23 free kernel: [ 713.454442] Dev nbd1: unable to read RDB block

0
Oct 13 11:27:23 free kernel: [ 713.454507] nbd1: unable to read partition
table
Oct 13 11:27:23 free kernel: [ 713.454980] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:27:23 free kernel: [ 713.455102] Dev nbd1: unable to read RDB block

0
Oct 13 11:27:23 free kernel: [ 713.455166] nbd1: unable to read partition
table
Oct 13 11:27:23 free kernel: [ 713.465565] block nbd0: NBD_DISCONNECT
Oct 13 11:27:23 free kernel: [ 713.477996] block nbd1: NBD_DISCONNECT
Oct 13 11:27:23 free kernel: [ 713.482674] block nbd0: NBD_DISCONNECT
Oct 13 11:28:16 free kernel: [ 766.759620] block nbd1: NBD_DISCONNECT
Oct 13 11:28:16 free kernel: [ 766.765261] block nbd0: NBD_DISCONNECT
Oct 13 11:28:16 free kernel: [ 766.772551] block nbd1: NBD_DISCONNECT
Oct 13 11:28:16 free kernel: [ 766.777506] block nbd0: NBD_DISCONNECT
Oct 13 11:28:33 free kernel: [ 783.019791] block nbd1: NBD_DISCONNECT
Oct 13 11:28:33 free kernel: [ 783.019883] block nbd1: Disconnected due to
user request.
Oct 13 11:28:33 free kernel: [ 783.020851] print_req_error: 383 callbacks
suppressed
Oct 13 11:28:33 free kernel: [ 783.020855] print_req_error: I/O error, dev
nbd1, sector 0 flags 0
Oct 13 11:28:33 free kernel: [ 783.020861] buffer_io_error: 382 callbacks
suppressed
Oct 13 11:28:33 free kernel: [ 783.020864] Buffer I/O error on dev nbd1,
logical block 0, async page read
Oct 13 11:28:33 free kernel: [ 783.020884] print_req_error: I/O error, dev
nbd1, sector 1 flags 0
Oct 13 11:28:33 free kernel: [ 783.020887] Buffer I/O error on dev nbd1,
logical block 1, async page read
Oct 13 11:28:33 free kernel: [ 783.020898] print_req_error: I/O error, dev
nbd1, sector 2 flags 0
Oct 13 11:28:33 free kernel: [ 783.020900] Buffer I/O error on dev nbd1,
logical block 2, async page read
Oct 13 11:28:33 free kernel: [ 783.020910] print_req_error: I/O error, dev
nbd1, sector 3 flags 0
Oct 13 11:28:33 free kernel: [ 783.020912] Buffer I/O error on dev nbd1,
logical block 3, async page read
Oct 13 11:28:33 free kernel: [ 783.020922] print_req_error: I/O error, dev
nbd1, sector 4 flags 0
Oct 13 11:28:33 free kernel: [ 783.020924] Buffer I/O error on dev nbd1,
logical block 4, async page read
Oct 13 11:28:33 free kernel: [ 783.020933] print_req_error: I/O error, dev
nbd1, sector 5 flags 0
Oct 13 11:28:33 free kernel: [ 783.020936] Buffer I/O error on dev nbd1,
logical block 5, async page read
Oct 13 11:28:33 free kernel: [ 783.020945] print_req_error: I/O error, dev
nbd1, sector 6 flags 0
Oct 13 11:28:33 free kernel: [ 783.020947] Buffer I/O error on dev nbd1,
logical block 6, async page read
Oct 13 11:28:33 free kernel: [ 783.020961] print_req_error: I/O error, dev
nbd1, sector 7 flags 0
Oct 13 11:28:33 free kernel: [ 783.020964] Buffer I/O error on dev nbd1,
logical block 7, async page read
Oct 13 11:28:33 free kernel: [ 783.020982] print_req_error: I/O error, dev
nbd1, sector 0 flags 0
Oct 13 11:28:33 free kernel: [ 783.020985] Buffer I/O error on dev nbd1,
logical block 0, async page read
Oct 13 11:28:33 free kernel: [ 783.020996] print_req_error: I/O error, dev
nbd1, sector 1 flags 0
Oct 13 11:28:33 free kernel: [ 783.020998] Buffer I/O error on dev nbd1,
logical block 1, async page read
Oct 13 11:28:33 free kernel: [ 783.021167] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:28:33 free kernel: [ 783.021398] Dev nbd1: unable to read RDB block

0
Oct 13 11:28:33 free kernel: [ 783.021640] nbd1: unable to read partition
table
Oct 13 11:28:33 free kernel: [ 783.021953] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:28:33 free kernel: [ 783.022187] Dev nbd1: unable to read RDB block

0
Oct 13 11:28:33 free kernel: [ 783.022423] nbd1: unable to read partition
table
Oct 13 11:28:33 free kernel: [ 783.036979] block nbd0: NBD_DISCONNECT
Oct 13 11:28:33 free kernel: [ 783.051997] block nbd1: NBD_DISCONNECT
Oct 13 11:28:33 free kernel: [ 783.057931] block nbd0: NBD_DISCONNECT
Oct 13 11:28:44 free kernel: [ 794.232905] block nbd1: NBD_DISCONNECT
Oct 13 11:28:44 free kernel: [ 794.238458] block nbd0: NBD_DISCONNECT
Oct 13 11:28:44 free kernel: [ 794.247611] block nbd1: NBD_DISCONNECT
Oct 13 11:28:44 free kernel: [ 794.252883] block nbd0: NBD_DISCONNECT
Oct 13 11:29:06 free kernel: [ 816.240576] block nbd1: NBD_DISCONNECT
Oct 13 11:29:06 free kernel: [ 816.246196] block nbd0: NBD_DISCONNECT
Oct 13 11:29:06 free kernel: [ 816.254445] block nbd1: NBD_DISCONNECT
Oct 13 11:29:06 free kernel: [ 816.258645] block nbd0: NBD_DISCONNECT
Oct 13 11:29:21 free kernel: [ 831.380225] block nbd1: NBD_DISCONNECT
Oct 13 11:29:21 free kernel: [ 831.380318] block nbd1: Disconnected due to
user request.
Oct 13 11:29:21 free kernel: [ 831.381310] print_req_error: 182 callbacks
suppressed
Oct 13 11:29:21 free kernel: [ 831.381313] print_req_error: I/O error, dev
nbd1, sector 0 flags 0
Oct 13 11:29:21 free kernel: [ 831.381320] buffer_io_error: 182 callbacks
suppressed
Oct 13 11:29:21 free kernel: [ 831.381337] Buffer I/O error on dev nbd1,
logical block 0, async page read
Oct 13 11:29:21 free kernel: [ 831.381358] print_req_error: I/O error, dev
nbd1, sector 1 flags 0
Oct 13 11:29:21 free kernel: [ 831.381361] Buffer I/O error on dev nbd1,
logical block 1, async page read
Oct 13 11:29:21 free kernel: [ 831.381370] print_req_error: I/O error, dev
nbd1, sector 2 flags 0
Oct 13 11:29:21 free kernel: [ 831.381373] Buffer I/O error on dev nbd1,
logical block 2, async page read
Oct 13 11:29:21 free kernel: [ 831.381381] print_req_error: I/O error, dev
nbd1, sector 3 flags 0
Oct 13 11:29:21 free kernel: [ 831.381384] Buffer I/O error on dev nbd1,
logical block 3, async page read
Oct 13 11:29:21 free kernel: [ 831.381392] print_req_error: I/O error, dev
nbd1, sector 4 flags 0
Oct 13 11:29:21 free kernel: [ 831.381394] Buffer I/O error on dev nbd1,
logical block 4, async page read
Oct 13 11:29:21 free kernel: [ 831.381403] print_req_error: I/O error, dev
nbd1, sector 5 flags 0
Oct 13 11:29:21 free kernel: [ 831.381405] Buffer I/O error on dev nbd1,
logical block 5, async page read
Oct 13 11:29:21 free kernel: [ 831.381414] print_req_error: I/O error, dev
nbd1, sector 6 flags 0
Oct 13 11:29:21 free kernel: [ 831.381416] Buffer I/O error on dev nbd1,
logical block 6, async page read
Oct 13 11:29:21 free kernel: [ 831.381428] print_req_error: I/O error, dev
nbd1, sector 7 flags 0
Oct 13 11:29:21 free kernel: [ 831.381431] Buffer I/O error on dev nbd1,
logical block 7, async page read
Oct 13 11:29:21 free kernel: [ 831.381447] print_req_error: I/O error, dev
nbd1, sector 0 flags 0
Oct 13 11:29:21 free kernel: [ 831.381450] Buffer I/O error on dev nbd1,
logical block 0, async page read
Oct 13 11:29:21 free kernel: [ 831.381458] print_req_error: I/O error, dev
nbd1, sector 1 flags 0
Oct 13 11:29:21 free kernel: [ 831.381461] Buffer I/O error on dev nbd1,
logical block 1, async page read
Oct 13 11:29:21 free kernel: [ 831.381615] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:29:21 free kernel: [ 831.381830] Dev nbd1: unable to read RDB block

0
Oct 13 11:29:21 free kernel: [ 831.382052] nbd1: unable to read partition
table
Oct 13 11:29:21 free kernel: [ 831.382338] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:29:21 free kernel: [ 831.382551] Dev nbd1: unable to read RDB block

0
Oct 13 11:29:21 free kernel: [ 831.382770] nbd1: unable to read partition
table
Oct 13 11:29:21 free kernel: [ 831.395494] block nbd0: NBD_DISCONNECT
Oct 13 11:29:21 free kernel: [ 831.411843] block nbd1: NBD_DISCONNECT
Oct 13 11:29:21 free kernel: [ 831.419893] block nbd0: NBD_DISCONNECT
Oct 13 11:29:39 free kernel: [ 849.869377] block nbd1: NBD_DISCONNECT
Oct 13 11:29:39 free kernel: [ 849.869382] block nbd1: Disconnected due to
user request.
Oct 13 11:29:39 free kernel: [ 849.869383] block nbd1: shutting down sockets
Oct 13 11:29:39 free kernel: [ 849.869534] print_req_error: 182 callbacks
suppressed
Oct 13 11:29:39 free kernel: [ 849.869536] print_req_error: I/O error, dev
nbd1, sector 0 flags 0
Oct 13 11:29:39 free kernel: [ 849.869538] buffer_io_error: 182 callbacks
suppressed
Oct 13 11:29:39 free kernel: [ 849.869538] Buffer I/O error on dev nbd1,
logical block 0, async page read
Oct 13 11:29:39 free kernel: [ 849.869543] print_req_error: I/O error, dev
nbd1, sector 1 flags 0
Oct 13 11:29:39 free kernel: [ 849.869543] Buffer I/O error on dev nbd1,
logical block 1, async page read
Oct 13 11:29:39 free kernel: [ 849.869546] print_req_error: I/O error, dev
nbd1, sector 2 flags 0
Oct 13 11:29:39 free kernel: [ 849.869547] Buffer I/O error on dev nbd1,
logical block 2, async page read
Oct 13 11:29:39 free kernel: [ 849.869549] print_req_error: I/O error, dev
nbd1, sector 3 flags 0
Oct 13 11:29:39 free kernel: [ 849.869550] Buffer I/O error on dev nbd1,
logical block 3, async page read
Oct 13 11:29:39 free kernel: [ 849.869552] print_req_error: I/O error, dev
nbd1, sector 4 flags 0
Oct 13 11:29:39 free kernel: [ 849.869553] Buffer I/O error on dev nbd1,
logical block 4, async page read
Oct 13 11:29:39 free kernel: [ 849.869555] print_req_error: I/O error, dev
nbd1, sector 5 flags 0
Oct 13 11:29:39 free kernel: [ 849.869556] Buffer I/O error on dev nbd1,
logical block 5, async page read
Oct 13 11:29:39 free kernel: [ 849.869558] print_req_error: I/O error, dev
nbd1, sector 6 flags 0
Oct 13 11:29:39 free kernel: [ 849.869559] Buffer I/O error on dev nbd1,
logical block 6, async page read
Oct 13 11:29:39 free kernel: [ 849.869562] print_req_error: I/O error, dev
nbd1, sector 7 flags 0
Oct 13 11:29:39 free kernel: [ 849.869563] Buffer I/O error on dev nbd1,
logical block 7, async page read
Oct 13 11:29:39 free kernel: [ 849.869567] print_req_error: I/O error, dev
nbd1, sector 0 flags 0
Oct 13 11:29:39 free kernel: [ 849.869568] Buffer I/O error on dev nbd1,
logical block 0, async page read
Oct 13 11:29:39 free kernel: [ 849.869570] print_req_error: I/O error, dev
nbd1, sector 1 flags 0
Oct 13 11:29:39 free kernel: [ 849.869571] Buffer I/O error on dev nbd1,
logical block 1, async page read
Oct 13 11:29:39 free kernel: [ 849.869614] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:29:39 free kernel: [ 849.869673] Dev nbd1: unable to read RDB block

0
Oct 13 11:29:39 free kernel: [ 849.869734] nbd1: unable to read partition
table
Oct 13 11:29:39 free kernel: [ 849.871997] block nbd0: NBD_DISCONNECT
Oct 13 11:29:39 free kernel: [ 849.882161] ldm_validate_partition_table():
Disk read failed.
Oct 13 11:29:39 free kernel: [ 849.882222] Dev nbd1: unable to read RDB block

0
Oct 13 11:29:39 free kernel: [ 849.882303] nbd1: unable to read partition
table
Oct 13 11:29:39 free kernel: [ 849.882309] block nbd1: NBD_DISCONNECT
Oct 13 11:29:39 free kernel: [ 849.882311] block nbd1: Send disconnect failed
-32
Oct 13 11:29:39 free kernel: [ 849.885020] block nbd0: NBD_DISCONNECT
Oct 13 11:30:11 free kernel: [ 881.354868] block nbd1: Connection timed out
Oct 13 11:30:11 free kernel: [ 881.354879] block nbd1: Connection timed out
Oct 13 11:30:11 free kernel: [ 881.354904] print_req_error: 182 callbacks
suppressed
Oct 13 11:30:11 free kernel: [ 881.354907] print_req_error: I/O error, dev
nbd1, sector 251658120 flags 80700
Oct 13 11:30:11 free kernel: [ 881.354931] print_req_error: I/O error, dev
nbd1, sector 251658184 flags 80700
Oct 13 11:30:11 free kernel: [ 881.354974] print_req_error: I/O error, dev
nbd1, sector 251658120 flags 0
Oct 13 11:30:11 free kernel: [ 881.354977] buffer_io_error: 182 callbacks
suppressed
Oct 13 11:30:11 free kernel: [ 881.354979] Buffer I/O error on dev nbd1,
logical block 251658120, async page read
Oct 13 11:30:11 free kernel: [ 881.354990] print_req_error: I/O error, dev
nbd1, sector 251658121 flags 0
Oct 13 11:30:11 free kernel: [ 881.354993] Buffer I/O error on dev nbd1,
logical block 251658121, async page read
Oct 13 11:30:11 free kernel: [ 881.355001] print_req_error: I/O error, dev
nbd1, sector 251658122 flags 0
Oct 13 11:30:11 free kernel: [ 881.355003] Buffer I/O error on dev nbd1,
logical block 251658122, async page read
Oct 13 11:30:11 free kernel: [ 881.355011] print_req_error: I/O error, dev
nbd1, sector 251658123 flags 0
Oct 13 11:30:11 free kernel: [ 881.355013] Buffer I/O error on dev nbd1,
logical block 251658123, async page read
Oct 13 11:30:11 free kernel: [ 881.355021] print_req_error: I/O error, dev
nbd1, sector 251658124 flags 0
Oct 13 11:30:11 free kernel: [ 881.355023] Buffer I/O error on dev nbd1,
logical block 251658124, async page read
Oct 13 11:30:11 free kernel: [ 881.355031] print_req_error: I/O error, dev
nbd1, sector 251658125 flags 0
Oct 13 11:30:11 free kernel: [ 881.355033] Buffer I/O error on dev nbd1,
logical block 251658125, async page read
Oct 13 11:30:11 free kernel: [ 881.355041] print_req_error: I/O error, dev
nbd1, sector 251658126 flags 0
Oct 13 11:30:11 free kernel: [ 881.355043] Buffer I/O error on dev nbd1,
logical block 251658126, async page read
Oct 13 11:30:11 free kernel: [ 881.355052] print_req_error: I/O error, dev
nbd1, sector 251658127 flags 0
Oct 13 11:30:11 free kernel: [ 881.355054] Buffer I/O error on dev nbd1,
logical block 251658127, async page read
Oct 13 11:30:11 free kernel: [ 881.355247] Buffer I/O error on dev nbd1,
logical block 4096, async page read
Oct 13 11:30:11 free kernel: [ 881.355257] Buffer I/O error on dev nbd1,
logical block 4097, async page read
Oct 13 11:32:39 free kernel: [ 0.000000] Linux version 5.2.0-3-amd64
(debian...@lists.debian.org) (gcc version 8.3.0 (Debian 8.3.0-23)) #1 SMP
Debian 5.2.17-1 (2019-10-06)
Oct 13 11:32:39 free kernel: [ 0.000000] Command line:
BOOT_IMAGE=/vmlinuz-5.2.0-3-amd64 root=/dev/mapper/test--lvm--vg-root ro quiet



I've also tried with different settings of the --cache option to qemu-nbd, but
this only resulted in faster speeds, but the problem was the same.

Like I said, if I try the same on a current Stretch system, it works like a
charm. I am not sure if this is a pure qemu-nbd (NBD) problem or related to the
kernel (or both).








-- System Information:
Debian Release: bullseye/sid
APT prefers unstable
APT policy: (500, 'unstable')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 5.2.0-3-amd64 (SMP w/8 CPU cores)
Kernel taint flags: TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US:en (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages qemu-utils depends on:
ii libaio1 0.3.112-5
ii libc6 2.29-2
ii libgcc1 1:9.2.1-9
ii libglib2.0-0 2.62.1-1
ii libgnutls30 3.6.9-5
ii libnettle6 3.5.1+really3.4.1-1
ii zlib1g 1:1.2.11.dfsg-1+b1

qemu-utils recommends no packages.

Versions of packages qemu-utils suggests:
ii debootstrap 1.0.116
pn qemu-block-extra <none>

-- no debconf information

Marcel

unread,
Nov 20, 2019, 7:30:03 AM11/20/19
to
I did some more research and found this post, which pretty much is identical to what I have: https://lists.openwall.net/linux-ext4/2018/01/25/22

Here is some recent output from /var/log/syslog:


Nov 20 13:11:49 debian kernel: [ 160.820705] block nbd1: Connection timed out
Nov 20 13:15:12 debian kernel: [ 363.471204] INFO: task kworker/u2:3:128 blocked for more than 120 seconds.
Nov 20 13:15:12 debian kernel: [ 363.471210] Tainted: G OE 4.19.0-6-amd64 #1 Debian 4.19.67-2+deb10u2
Nov 20 13:15:12 debian kernel: [ 363.471212] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 20 13:15:12 debian kernel: [ 363.471214] kworker/u2:3 D 0 128 2 0x80000000
Nov 20 13:15:12 debian kernel: [ 363.471237] Workqueue: writeback wb_workfn (flush-254:11)
Nov 20 13:15:12 debian kernel: [ 363.471240] Call Trace:
Nov 20 13:15:12 debian kernel: [ 363.471262] ? __schedule+0x2a2/0x870
Nov 20 13:15:12 debian kernel: [ 363.471265] schedule+0x28/0x80
Nov 20 13:15:12 debian kernel: [ 363.471268] io_schedule+0x12/0x40
Nov 20 13:15:12 debian kernel: [ 363.471275] wbt_wait+0x205/0x300
Nov 20 13:15:12 debian kernel: [ 363.471291] ? trace_event_raw_event_wbt_step+0x130/0x130
Nov 20 13:15:12 debian kernel: [ 363.471295] rq_qos_throttle+0x31/0x40
Nov 20 13:15:12 debian kernel: [ 363.471298] blk_mq_make_request+0x111/0x530
Nov 20 13:15:12 debian kernel: [ 363.471302] generic_make_request+0x1a4/0x400
Nov 20 13:15:12 debian kernel: [ 363.471305] submit_bio+0x45/0x140
Nov 20 13:15:12 debian kernel: [ 363.471333] ext4_io_submit+0x49/0x60 [ext4]
Nov 20 13:15:12 debian kernel: [ 363.471353] ext4_bio_write_page+0x24a/0x4d0 [ext4]
Nov 20 13:15:12 debian kernel: [ 363.471371] mpage_submit_page+0x53/0x70 [ext4]
Nov 20 13:15:12 debian kernel: [ 363.471388] mpage_map_and_submit_buffers+0x10c/0x1d0 [ext4]
Nov 20 13:15:12 debian kernel: [ 363.471408] ext4_writepages+0xa12/0xf00 [ext4]
Nov 20 13:15:12 debian kernel: [ 363.471414] ? do_writepages+0x41/0xd0
Nov 20 13:15:12 debian kernel: [ 363.471430] ? ext4_mark_inode_dirty+0x1d0/0x1d0 [ext4]
Nov 20 13:15:12 debian kernel: [ 363.471432] do_writepages+0x41/0xd0
Nov 20 13:15:12 debian kernel: [ 363.471445] ? __wb_calc_thresh+0x3a/0x130
Nov 20 13:15:12 debian kernel: [ 363.471448] __writeback_single_inode+0x3d/0x350
Nov 20 13:15:12 debian kernel: [ 363.471459] writeback_sb_inodes+0x1e3/0x450
Nov 20 13:15:12 debian kernel: [ 363.471463] __writeback_inodes_wb+0x5d/0xb0
Nov 20 13:15:12 debian kernel: [ 363.471466] wb_writeback+0x25f/0x2f0
Nov 20 13:15:12 debian kernel: [ 363.471469] wb_workfn+0x30d/0x400
Nov 20 13:15:12 debian kernel: [ 363.471472] ? __switch_to+0x8c/0x440
Nov 20 13:15:12 debian kernel: [ 363.471476] process_one_work+0x1a7/0x3a0
Nov 20 13:15:12 debian kernel: [ 363.471487] worker_thread+0x30/0x390
Nov 20 13:15:12 debian kernel: [ 363.471491] ? create_worker+0x1a0/0x1a0
Nov 20 13:15:12 debian kernel: [ 363.471493] kthread+0x112/0x130
Nov 20 13:15:12 debian kernel: [ 363.471495] ? kthread_bind+0x30/0x30
Nov 20 13:15:12 debian kernel: [ 363.471499] ret_from_fork+0x35/0x40
Nov 20 13:15:12 debian kernel: [ 363.471504] INFO: task kworker/0:1H:165 blocked for more than 120 seconds.
Nov 20 13:15:12 debian kernel: [ 363.471506] Tainted: G OE 4.19.0-6-amd64 #1 Debian 4.19.67-2+deb10u2
Nov 20 13:15:12 debian kernel: [ 363.471508] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 20 13:15:12 debian kernel: [ 363.471510] kworker/0:1H D 0 165 2 0x80000000
Nov 20 13:15:12 debian kernel: [ 363.471515] Workqueue: kblockd blk_mq_timeout_work
Nov 20 13:15:12 debian kernel: [ 363.471516] Call Trace:
Nov 20 13:15:12 debian kernel: [ 363.471520] ? __schedule+0x2a2/0x870
Nov 20 13:15:12 debian kernel: [ 363.471523] schedule+0x28/0x80
Nov 20 13:15:12 debian kernel: [ 363.471525] schedule_preempt_disabled+0xa/0x10
Nov 20 13:15:12 debian kernel: [ 363.471528] __mutex_lock.isra.8+0x2b5/0x4a0
Nov 20 13:15:12 debian kernel: [ 363.471535] ? _dev_err+0x6c/0x90
Nov 20 13:15:12 debian kernel: [ 363.471541] sock_shutdown+0x4a/0x90 [nbd]
Nov 20 13:15:12 debian kernel: [ 363.471545] nbd_xmit_timeout+0xb0/0x180 [nbd]
Nov 20 13:15:12 debian kernel: [ 363.471548] blk_mq_check_expired+0xe9/0x100
Nov 20 13:15:12 debian kernel: [ 363.471551] bt_iter+0x4e/0x60
Nov 20 13:15:12 debian kernel: [ 363.471554] blk_mq_queue_tag_busy_iter+0x184/0x2e0
Nov 20 13:15:12 debian kernel: [ 363.471556] ? blk_mq_poll+0x360/0x360
Nov 20 13:15:12 debian kernel: [ 363.471559] ? __switch_to_asm+0x35/0x70
Nov 20 13:15:12 debian kernel: [ 363.471561] ? blk_mq_poll+0x360/0x360
Nov 20 13:15:12 debian kernel: [ 363.471564] ? syscall_return_via_sysret+0x13/0x83
Nov 20 13:15:12 debian kernel: [ 363.471567] blk_mq_timeout_work+0x54/0x120
Nov 20 13:15:12 debian kernel: [ 363.471570] process_one_work+0x1a7/0x3a0
Nov 20 13:15:12 debian kernel: [ 363.471573] worker_thread+0x30/0x390
Nov 20 13:15:12 debian kernel: [ 363.471576] ? create_worker+0x1a0/0x1a0
Nov 20 13:15:12 debian kernel: [ 363.471578] kthread+0x112/0x130
Nov 20 13:15:12 debian kernel: [ 363.471580] ? kthread_bind+0x30/0x30
Nov 20 13:15:12 debian kernel: [ 363.471583] ret_from_fork+0x35/0x40
Nov 20 13:15:12 debian kernel: [ 363.471620] INFO: task jbd2/dm-11-8:3239 blocked for more than 120 seconds.
Nov 20 13:15:12 debian kernel: [ 363.471622] Tainted: G OE 4.19.0-6-amd64 #1 Debian 4.19.67-2+deb10u2
Nov 20 13:15:12 debian kernel: [ 363.471623] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 20 13:15:12 debian kernel: [ 363.471625] jbd2/dm-11-8 D 0 3239 2 0x80000000
Nov 20 13:15:12 debian kernel: [ 363.471627] Call Trace:
Nov 20 13:15:12 debian kernel: [ 363.471631] ? __schedule+0x2a2/0x870
Nov 20 13:15:12 debian kernel: [ 363.471639] ? __wake_up_common_lock+0x89/0xc0
Nov 20 13:15:12 debian kernel: [ 363.471641] ? finish_wait+0x80/0x80
Nov 20 13:15:12 debian kernel: [ 363.471644] schedule+0x28/0x80
Nov 20 13:15:12 debian kernel: [ 363.471651] jbd2_journal_commit_transaction+0x257/0x1890 [jbd2]
Nov 20 13:15:12 debian kernel: [ 363.471656] ? __switch_to_asm+0x35/0x70
Nov 20 13:15:12 debian kernel: [ 363.471658] ? __switch_to_asm+0x41/0x70
Nov 20 13:15:12 debian kernel: [ 363.471661] ? __switch_to_asm+0x35/0x70
Nov 20 13:15:12 debian kernel: [ 363.471664] ? __switch_to_asm+0x35/0x70
Nov 20 13:15:12 debian kernel: [ 363.471666] ? __switch_to_asm+0x41/0x70
Nov 20 13:15:12 debian kernel: [ 363.471669] ? __switch_to_asm+0x35/0x70
Nov 20 13:15:12 debian kernel: [ 363.471671] ? __switch_to_asm+0x41/0x70
Nov 20 13:15:12 debian kernel: [ 363.471674] ? __switch_to_asm+0x35/0x70
Nov 20 13:15:12 debian kernel: [ 363.471676] ? __switch_to_asm+0x35/0x70
Nov 20 13:15:12 debian kernel: [ 363.471679] ? __switch_to_asm+0x41/0x70
Nov 20 13:15:12 debian kernel: [ 363.471682] ? __switch_to_asm+0x35/0x70
Nov 20 13:15:12 debian kernel: [ 363.471684] ? __switch_to_asm+0x35/0x70
Nov 20 13:15:12 debian kernel: [ 363.471687] ? __switch_to_asm+0x41/0x70
Nov 20 13:15:12 debian kernel: [ 363.471689] ? __switch_to_asm+0x41/0x70
Nov 20 13:15:12 debian kernel: [ 363.471692] ? finish_wait+0x80/0x80
Nov 20 13:15:12 debian kernel: [ 363.471695] ? lock_timer_base+0x67/0x80
Nov 20 13:15:12 debian kernel: [ 363.471701] kjournald2+0xbd/0x270 [jbd2]
Nov 20 13:15:12 debian kernel: [ 363.471704] ? finish_wait+0x80/0x80
Nov 20 13:15:12 debian kernel: [ 363.471710] ? commit_timeout+0x10/0x10 [jbd2]
Nov 20 13:15:12 debian kernel: [ 363.471712] kthread+0x112/0x130
Nov 20 13:15:12 debian kernel: [ 363.471714] ? kthread_bind+0x30/0x30
Nov 20 13:15:12 debian kernel: [ 363.471718] ret_from_fork+0x35/0x40
Nov 20 13:15:12 debian kernel: [ 363.471720] INFO: task umount:3921 blocked for more than 120 seconds.
Nov 20 13:15:12 debian kernel: [ 363.471722] Tainted: G OE 4.19.0-6-amd64 #1 Debian 4.19.67-2+deb10u2
Nov 20 13:15:12 debian kernel: [ 363.471724] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 20 13:15:12 debian kernel: [ 363.471725] umount D 0 3921 2085 0x00000000
Nov 20 13:15:12 debian kernel: [ 363.471727] Call Trace:
Nov 20 13:15:12 debian kernel: [ 363.471730] ? __schedule+0x2a2/0x870
Nov 20 13:15:12 debian kernel: [ 363.471733] ? __switch_to_asm+0x35/0x70
Nov 20 13:15:12 debian kernel: [ 363.471735] schedule+0x28/0x80
Nov 20 13:15:12 debian kernel: [ 363.471738] rwsem_down_write_failed+0x17c/0x3a0
Nov 20 13:15:12 debian kernel: [ 363.471741] ? mnt_get_count+0x39/0x50
Nov 20 13:15:12 debian kernel: [ 363.471746] call_rwsem_down_write_failed+0x13/0x20
Nov 20 13:15:12 debian kernel: [ 363.471749] down_write+0x29/0x40
Nov 20 13:15:12 debian kernel: [ 363.471752] deactivate_super+0x43/0x50
Nov 20 13:15:12 debian kernel: [ 363.471755] cleanup_mnt+0x3f/0x70
Nov 20 13:15:12 debian kernel: [ 363.471757] task_work_run+0x8a/0xb0
Nov 20 13:15:12 debian kernel: [ 363.471761] exit_to_usermode_loop+0xeb/0xf0
Nov 20 13:15:12 debian kernel: [ 363.471768] do_syscall_64+0x10d/0x110
Nov 20 13:15:12 debian kernel: [ 363.471772] entry_SYSCALL_64_after_hwframe+0x44/0xa9
Nov 20 13:15:12 debian kernel: [ 363.471775] RIP: 0033:0x7f08434ec507
Nov 20 13:15:12 debian kernel: [ 363.471780] Code: Bad RIP value.
Nov 20 13:15:12 debian kernel: [ 363.471782] RSP: 002b:00007ffed9c8ec38 EFLAGS: 00000202 ORIG_RAX: 00000000000000a6
Nov 20 13:15:12 debian kernel: [ 363.471784] RAX: 0000000000000000 RBX: 000055b66751ca40 RCX: 00007f08434ec507
Nov 20 13:15:12 debian kernel: [ 363.471786] RDX: 0000000000000003 RSI: 0000000000000002 RDI: 000055b66751d290
Nov 20 13:15:12 debian kernel: [ 363.471787] RBP: 0000000000000002 R08: 000055b667525fd0 R09: 00007f084356de80
Nov 20 13:15:12 debian kernel: [ 363.471788] R10: 0000000000000000 R11: 0000000000000202 R12: 000055b66751d290
Nov 20 13:15:12 debian kernel: [ 363.471789] R13: 00007f08436121c4 R14: 000055b66751cb38 R15: 000055b667525890
Nov 20 13:15:12 debian kernel: [ 363.471792] INFO: task dumpe2fs:3922 blocked for more than 120 seconds.
Nov 20 13:15:12 debian kernel: [ 363.471794] Tainted: G OE 4.19.0-6-amd64 #1 Debian 4.19.67-2+deb10u2
Nov 20 13:15:12 debian kernel: [ 363.471795] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 20 13:15:12 debian kernel: [ 363.471797] dumpe2fs D 0 3922 599 0x00000000
Nov 20 13:15:12 debian kernel: [ 363.471799] Call Trace:
Nov 20 13:15:12 debian kernel: [ 363.471802] ? __schedule+0x2a2/0x870
Nov 20 13:15:12 debian kernel: [ 363.471805] schedule+0x28/0x80
Nov 20 13:15:12 debian kernel: [ 363.471807] io_schedule+0x12/0x40
Nov 20 13:15:12 debian kernel: [ 363.471810] wbt_wait+0x205/0x300
Nov 20 13:15:12 debian kernel: [ 363.471812] ? trace_event_raw_event_wbt_step+0x130/0x130
Nov 20 13:15:12 debian kernel: [ 363.471816] rq_qos_throttle+0x31/0x40
Nov 20 13:15:12 debian kernel: [ 363.471818] blk_mq_make_request+0x111/0x530
Nov 20 13:15:12 debian kernel: [ 363.471822] generic_make_request+0x1a4/0x400
Nov 20 13:15:12 debian kernel: [ 363.471824] submit_bio+0x45/0x140
Nov 20 13:15:12 debian kernel: [ 363.471827] ? guard_bio_eod+0x32/0x100
Nov 20 13:15:12 debian kernel: [ 363.471830] submit_bh_wbc+0x163/0x190
Nov 20 13:15:12 debian kernel: [ 363.471833] __block_write_full_page+0x234/0x420
Nov 20 13:15:12 debian kernel: [ 363.471836] ? check_disk_change+0x60/0x60
Nov 20 13:15:12 debian kernel: [ 363.471838] ? touch_buffer+0x60/0x60
Nov 20 13:15:12 debian kernel: [ 363.471841] __writepage+0x19/0x50
Nov 20 13:15:12 debian kernel: [ 363.471843] write_cache_pages+0x1e1/0x470
Nov 20 13:15:12 debian kernel: [ 363.471845] ? __wb_calc_thresh+0x130/0x130
Nov 20 13:15:12 debian kernel: [ 363.471848] generic_writepages+0x56/0x90
Nov 20 13:15:12 debian kernel: [ 363.471851] do_writepages+0x41/0xd0
Nov 20 13:15:12 debian kernel: [ 363.471855] ? __handle_mm_fault+0x1090/0x1270
Nov 20 13:15:12 debian kernel: [ 363.471858] __filemap_fdatawrite_range+0xbe/0xf0
Nov 20 13:15:12 debian kernel: [ 363.471861] file_write_and_wait_range+0x4c/0xa0
Nov 20 13:15:12 debian kernel: [ 363.471864] blkdev_fsync+0x16/0x40
Nov 20 13:15:12 debian kernel: [ 363.471867] do_fsync+0x38/0x70
Nov 20 13:15:12 debian kernel: [ 363.471870] __x64_sys_fsync+0x10/0x20
Nov 20 13:15:12 debian kernel: [ 363.471873] do_syscall_64+0x53/0x110
Nov 20 13:15:12 debian kernel: [ 363.471877] entry_SYSCALL_64_after_hwframe+0x44/0xa9
Nov 20 13:15:12 debian kernel: [ 363.471878] RIP: 0033:0x7f0790255214
Nov 20 13:15:12 debian kernel: [ 363.471881] Code: Bad RIP value.
Nov 20 13:15:12 debian kernel: [ 363.471882] RSP: 002b:00007fff7f9474d8 EFLAGS: 00000246 ORIG_RAX: 000000000000004a
Nov 20 13:15:12 debian kernel: [ 363.471890] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f0790255214
Nov 20 13:15:12 debian kernel: [ 363.471891] RDX: 0000000000000008 RSI: 0000000000000000 RDI: 0000000000000003
Nov 20 13:15:12 debian kernel: [ 363.471892] RBP: 0000000000000003 R08: 0000000000000000 R09: 00007f07903ef680
Nov 20 13:15:12 debian kernel: [ 363.471893] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f07903ef680
Nov 20 13:15:12 debian kernel: [ 363.471894] R13: 000055cb7a993440 R14: 00007f07903ef680 R15: 0000000000000000
Nov 20 13:15:16 debian PackageKit: daemon quit
Nov 20 13:15:16 debian systemd[1]: packagekit.service: Main process exited, code=killed, status=15/TERM
Nov 20 13:15:16 debian systemd[1]: packagekit.service: Succeeded.
Nov 20 13:17:01 debian CRON[3941]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Nov 20 13:17:13 debian kernel: [ 484.242755] INFO: task kworker/u2:3:128 blocked for more than 120 seconds.
Nov 20 13:17:13 debian kernel: [ 484.242759] Tainted: G OE 4.19.0-6-amd64 #1 Debian 4.19.67-2+deb10u2
Nov 20 13:17:13 debian kernel: [ 484.242759] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 20 13:17:13 debian kernel: [ 484.242761] kworker/u2:3 D 0 128 2 0x80000000
Nov 20 13:17:13 debian kernel: [ 484.242766] Workqueue: writeback wb_workfn (flush-254:11)
Nov 20 13:17:13 debian kernel: [ 484.242767] Call Trace:
Nov 20 13:17:13 debian kernel: [ 484.242772] ? __schedule+0x2a2/0x870
Nov 20 13:17:13 debian kernel: [ 484.242773] schedule+0x28/0x80
Nov 20 13:17:13 debian kernel: [ 484.242774] io_schedule+0x12/0x40
Nov 20 13:17:13 debian kernel: [ 484.242776] wbt_wait+0x205/0x300
Nov 20 13:17:13 debian kernel: [ 484.242778] ? trace_event_raw_event_wbt_step+0x130/0x130
Nov 20 13:17:13 debian kernel: [ 484.242779] rq_qos_throttle+0x31/0x40
Nov 20 13:17:13 debian kernel: [ 484.242781] blk_mq_make_request+0x111/0x530
Nov 20 13:17:13 debian kernel: [ 484.242783] generic_make_request+0x1a4/0x400
Nov 20 13:17:13 debian kernel: [ 484.242784] submit_bio+0x45/0x140
Nov 20 13:17:13 debian kernel: [ 484.242797] ext4_io_submit+0x49/0x60 [ext4]
Nov 20 13:17:13 debian kernel: [ 484.242806] ext4_bio_write_page+0x24a/0x4d0 [ext4]
Nov 20 13:17:13 debian kernel: [ 484.242813] mpage_submit_page+0x53/0x70 [ext4]
Nov 20 13:17:13 debian kernel: [ 484.242820] mpage_map_and_submit_buffers+0x10c/0x1d0 [ext4]
Nov 20 13:17:13 debian kernel: [ 484.242829] ext4_writepages+0xa12/0xf00 [ext4]
Nov 20 13:17:13 debian kernel: [ 484.242832] ? do_writepages+0x41/0xd0
Nov 20 13:17:13 debian kernel: [ 484.242838] ? ext4_mark_inode_dirty+0x1d0/0x1d0 [ext4]
Nov 20 13:17:13 debian kernel: [ 484.242839] do_writepages+0x41/0xd0
Nov 20 13:17:13 debian kernel: [ 484.242841] ? __wb_calc_thresh+0x3a/0x130
Nov 20 13:17:13 debian kernel: [ 484.242842] __writeback_single_inode+0x3d/0x350
Nov 20 13:17:13 debian kernel: [ 484.242843] writeback_sb_inodes+0x1e3/0x450
Nov 20 13:17:13 debian kernel: [ 484.242852] __writeback_inodes_wb+0x5d/0xb0
Nov 20 13:17:13 debian kernel: [ 484.242854] wb_writeback+0x25f/0x2f0
Nov 20 13:17:13 debian kernel: [ 484.242856] wb_workfn+0x30d/0x400
Nov 20 13:17:13 debian kernel: [ 484.242857] ? __switch_to+0x8c/0x440
Nov 20 13:17:13 debian kernel: [ 484.242859] process_one_work+0x1a7/0x3a0
Nov 20 13:17:13 debian kernel: [ 484.242861] worker_thread+0x30/0x390
Nov 20 13:17:13 debian kernel: [ 484.242862] ? create_worker+0x1a0/0x1a0
Nov 20 13:17:13 debian kernel: [ 484.242863] kthread+0x112/0x130
Nov 20 13:17:13 debian kernel: [ 484.242864] ? kthread_bind+0x30/0x30
Nov 20 13:17:13 debian kernel: [ 484.242866] ret_from_fork+0x35/0x40
Nov 20 13:17:13 debian kernel: [ 484.242868] INFO: task kworker/0:1H:165 blocked for more than 120 seconds.
Nov 20 13:17:13 debian kernel: [ 484.242869] Tainted: G OE 4.19.0-6-amd64 #1 Debian 4.19.67-2+deb10u2
Nov 20 13:17:13 debian kernel: [ 484.242870] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 20 13:17:13 debian kernel: [ 484.242871] kworker/0:1H D 0 165 2 0x80000000
Nov 20 13:17:13 debian kernel: [ 484.242873] Workqueue: kblockd blk_mq_timeout_work
Nov 20 13:17:13 debian kernel: [ 484.242874] Call Trace:
Nov 20 13:17:13 debian kernel: [ 484.242875] ? __schedule+0x2a2/0x870
Nov 20 13:17:13 debian kernel: [ 484.242876] schedule+0x28/0x80
Nov 20 13:17:13 debian kernel: [ 484.242878] schedule_preempt_disabled+0xa/0x10
Nov 20 13:17:13 debian kernel: [ 484.242879] __mutex_lock.isra.8+0x2b5/0x4a0
Nov 20 13:17:13 debian kernel: [ 484.242881] ? _dev_err+0x6c/0x90
Nov 20 13:17:13 debian kernel: [ 484.242884] sock_shutdown+0x4a/0x90 [nbd]
Nov 20 13:17:13 debian kernel: [ 484.242885] nbd_xmit_timeout+0xb0/0x180 [nbd]
Nov 20 13:17:13 debian kernel: [ 484.242887] blk_mq_check_expired+0xe9/0x100
Nov 20 13:17:13 debian kernel: [ 484.242888] bt_iter+0x4e/0x60
Nov 20 13:17:13 debian kernel: [ 484.242889] blk_mq_queue_tag_busy_iter+0x184/0x2e0
Nov 20 13:17:13 debian kernel: [ 484.242890] ? blk_mq_poll+0x360/0x360
Nov 20 13:17:13 debian kernel: [ 484.242891] ? __switch_to_asm+0x35/0x70
Nov 20 13:17:13 debian kernel: [ 484.242892] ? blk_mq_poll+0x360/0x360
Nov 20 13:17:13 debian kernel: [ 484.242893] ? syscall_return_via_sysret+0x13/0x83
Nov 20 13:17:13 debian kernel: [ 484.242894] blk_mq_timeout_work+0x54/0x120
Nov 20 13:17:13 debian kernel: [ 484.242896] process_one_work+0x1a7/0x3a0
Nov 20 13:17:13 debian kernel: [ 484.242897] worker_thread+0x30/0x390
Nov 20 13:17:13 debian kernel: [ 484.242898] ? create_worker+0x1a0/0x1a0
Nov 20 13:17:13 debian kernel: [ 484.242899] kthread+0x112/0x130
Nov 20 13:17:13 debian kernel: [ 484.242900] ? kthread_bind+0x30/0x30
Nov 20 13:17:13 debian kernel: [ 484.242901] ret_from_fork+0x35/0x40
Nov 20 13:17:13 debian kernel: [ 484.242923] INFO: task jbd2/dm-11-8:3239 blocked for more than 120 seconds.
Nov 20 13:17:13 debian kernel: [ 484.242924] Tainted: G OE 4.19.0-6-amd64 #1 Debian 4.19.67-2+deb10u2
Nov 20 13:17:13 debian kernel: [ 484.242925] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 20 13:17:13 debian kernel: [ 484.242925] jbd2/dm-11-8 D 0 3239 2 0x80000000
Nov 20 13:17:13 debian kernel: [ 484.242926] Call Trace:
Nov 20 13:17:13 debian kernel: [ 484.242928] ? __schedule+0x2a2/0x870
Nov 20 13:17:13 debian kernel: [ 484.242929] ? __wake_up_common_lock+0x89/0xc0
Nov 20 13:17:13 debian kernel: [ 484.242930] ? finish_wait+0x80/0x80
Nov 20 13:17:13 debian kernel: [ 484.242931] schedule+0x28/0x80
Nov 20 13:17:13 debian kernel: [ 484.242935] jbd2_journal_commit_transaction+0x257/0x1890 [jbd2]
Nov 20 13:17:13 debian kernel: [ 484.242937] ? __switch_to_asm+0x35/0x70
Nov 20 13:17:13 debian kernel: [ 484.242938] ? __switch_to_asm+0x41/0x70
Nov 20 13:17:13 debian kernel: [ 484.242939] ? __switch_to_asm+0x35/0x70
Nov 20 13:17:13 debian kernel: [ 484.242940] ? __switch_to_asm+0x35/0x70
Nov 20 13:17:13 debian kernel: [ 484.242941] ? __switch_to_asm+0x41/0x70
Nov 20 13:17:13 debian kernel: [ 484.242942] ? __switch_to_asm+0x35/0x70
Nov 20 13:17:13 debian kernel: [ 484.242943] ? __switch_to_asm+0x41/0x70
Nov 20 13:17:13 debian kernel: [ 484.242944] ? __switch_to_asm+0x35/0x70
Nov 20 13:17:13 debian kernel: [ 484.242945] ? __switch_to_asm+0x35/0x70
Nov 20 13:17:13 debian kernel: [ 484.242946] ? __switch_to_asm+0x41/0x70
Nov 20 13:17:13 debian kernel: [ 484.242947] ? __switch_to_asm+0x35/0x70
Nov 20 13:17:13 debian kernel: [ 484.242948] ? __switch_to_asm+0x35/0x70
Nov 20 13:17:13 debian kernel: [ 484.242949] ? __switch_to_asm+0x41/0x70
Nov 20 13:17:13 debian kernel: [ 484.242950] ? __switch_to_asm+0x41/0x70
Nov 20 13:17:13 debian kernel: [ 484.242951] ? finish_wait+0x80/0x80
Nov 20 13:17:13 debian kernel: [ 484.242952] ? lock_timer_base+0x67/0x80
Nov 20 13:17:13 debian kernel: [ 484.242955] kjournald2+0xbd/0x270 [jbd2]
Nov 20 13:17:13 debian kernel: [ 484.242956] ? finish_wait+0x80/0x80
Nov 20 13:17:13 debian kernel: [ 484.242959] ? commit_timeout+0x10/0x10 [jbd2]
Nov 20 13:17:13 debian kernel: [ 484.242960] kthread+0x112/0x130
Nov 20 13:17:13 debian kernel: [ 484.242960] ? kthread_bind+0x30/0x30
Nov 20 13:17:13 debian kernel: [ 484.242962] ret_from_fork+0x35/0x40
Nov 20 13:17:13 debian kernel: [ 484.242963] INFO: task umount:3921 blocked for more than 120 seconds.
Nov 20 13:17:13 debian kernel: [ 484.242964] Tainted: G OE 4.19.0-6-amd64 #1 Debian 4.19.67-2+deb10u2
Nov 20 13:17:13 debian kernel: [ 484.242964] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 20 13:17:13 debian kernel: [ 484.242965] umount D 0 3921 2085 0x00000000
Nov 20 13:17:13 debian kernel: [ 484.242966] Call Trace:
Nov 20 13:17:13 debian kernel: [ 484.242967] ? __schedule+0x2a2/0x870
Nov 20 13:17:13 debian kernel: [ 484.242968] ? __switch_to_asm+0x35/0x70
Nov 20 13:17:13 debian kernel: [ 484.242969] schedule+0x28/0x80
Nov 20 13:17:13 debian kernel: [ 484.242970] rwsem_down_write_failed+0x17c/0x3a0
Nov 20 13:17:13 debian kernel: [ 484.242972] ? mnt_get_count+0x39/0x50
Nov 20 13:17:13 debian kernel: [ 484.242974] call_rwsem_down_write_failed+0x13/0x20
Nov 20 13:17:13 debian kernel: [ 484.242976] down_write+0x29/0x40
Nov 20 13:17:13 debian kernel: [ 484.242977] deactivate_super+0x43/0x50
Nov 20 13:17:13 debian kernel: [ 484.242979] cleanup_mnt+0x3f/0x70
Nov 20 13:17:13 debian kernel: [ 484.242980] task_work_run+0x8a/0xb0
Nov 20 13:17:13 debian kernel: [ 484.242981] exit_to_usermode_loop+0xeb/0xf0
Nov 20 13:17:13 debian kernel: [ 484.242983] do_syscall_64+0x10d/0x110
Nov 20 13:17:13 debian kernel: [ 484.242984] entry_SYSCALL_64_after_hwframe+0x44/0xa9
Nov 20 13:17:13 debian kernel: [ 484.242985] RIP: 0033:0x7f08434ec507
Nov 20 13:17:13 debian kernel: [ 484.242988] Code: Bad RIP value.
Nov 20 13:17:13 debian kernel: [ 484.242989] RSP: 002b:00007ffed9c8ec38 EFLAGS: 00000202 ORIG_RAX: 00000000000000a6
Nov 20 13:17:13 debian kernel: [ 484.242990] RAX: 0000000000000000 RBX: 000055b66751ca40 RCX: 00007f08434ec507
Nov 20 13:17:13 debian kernel: [ 484.242990] RDX: 0000000000000003 RSI: 0000000000000002 RDI: 000055b66751d290
Nov 20 13:17:13 debian kernel: [ 484.242991] RBP: 0000000000000002 R08: 000055b667525fd0 R09: 00007f084356de80
Nov 20 13:17:13 debian kernel: [ 484.242991] R10: 0000000000000000 R11: 0000000000000202 R12: 000055b66751d290
Nov 20 13:17:13 debian kernel: [ 484.242992] R13: 00007f08436121c4 R14: 000055b66751cb38 R15: 000055b667525890
Nov 20 13:17:13 debian kernel: [ 484.242993] INFO: task dumpe2fs:3922 blocked for more than 120 seconds.
Nov 20 13:17:13 debian kernel: [ 484.242994] Tainted: G OE 4.19.0-6-amd64 #1 Debian 4.19.67-2+deb10u2
Nov 20 13:17:13 debian kernel: [ 484.242994] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 20 13:17:13 debian kernel: [ 484.242995] dumpe2fs D 0 3922 599 0x00000000
Nov 20 13:17:13 debian kernel: [ 484.242996] Call Trace:
Nov 20 13:17:13 debian kernel: [ 484.242998] ? __schedule+0x2a2/0x870
Nov 20 13:17:13 debian kernel: [ 484.242999] schedule+0x28/0x80
Nov 20 13:17:13 debian kernel: [ 484.242999] io_schedule+0x12/0x40
Nov 20 13:17:13 debian kernel: [ 484.243001] wbt_wait+0x205/0x300
Nov 20 13:17:13 debian kernel: [ 484.243002] ? trace_event_raw_event_wbt_step+0x130/0x130
Nov 20 13:17:13 debian kernel: [ 484.243003] rq_qos_throttle+0x31/0x40
Nov 20 13:17:13 debian kernel: [ 484.243004] blk_mq_make_request+0x111/0x530
Nov 20 13:17:13 debian kernel: [ 484.243006] generic_make_request+0x1a4/0x400
Nov 20 13:17:13 debian kernel: [ 484.243007] submit_bio+0x45/0x140
Nov 20 13:17:13 debian kernel: [ 484.243008] ? guard_bio_eod+0x32/0x100
Nov 20 13:17:13 debian kernel: [ 484.243009] submit_bh_wbc+0x163/0x190
Nov 20 13:17:13 debian kernel: [ 484.243010] __block_write_full_page+0x234/0x420
Nov 20 13:17:13 debian kernel: [ 484.243012] ? check_disk_change+0x60/0x60
Nov 20 13:17:13 debian kernel: [ 484.243013] ? touch_buffer+0x60/0x60
Nov 20 13:17:13 debian kernel: [ 484.243014] __writepage+0x19/0x50
Nov 20 13:17:13 debian kernel: [ 484.243015] write_cache_pages+0x1e1/0x470
Nov 20 13:17:13 debian kernel: [ 484.243016] ? __wb_calc_thresh+0x130/0x130
Nov 20 13:17:13 debian kernel: [ 484.243017] generic_writepages+0x56/0x90
Nov 20 13:17:13 debian kernel: [ 484.243018] do_writepages+0x41/0xd0
Nov 20 13:17:13 debian kernel: [ 484.243020] ? __handle_mm_fault+0x1090/0x1270
Nov 20 13:17:13 debian kernel: [ 484.243021] __filemap_fdatawrite_range+0xbe/0xf0
Nov 20 13:17:13 debian kernel: [ 484.243023] file_write_and_wait_range+0x4c/0xa0
Nov 20 13:17:13 debian kernel: [ 484.243024] blkdev_fsync+0x16/0x40
Nov 20 13:17:13 debian kernel: [ 484.243025] do_fsync+0x38/0x70
Nov 20 13:17:13 debian kernel: [ 484.243027] __x64_sys_fsync+0x10/0x20
Nov 20 13:17:13 debian kernel: [ 484.243028] do_syscall_64+0x53/0x110
Nov 20 13:17:13 debian kernel: [ 484.243029] entry_SYSCALL_64_after_hwframe+0x44/0xa9
Nov 20 13:17:13 debian kernel: [ 484.243030] RIP: 0033:0x7f0790255214
Nov 20 13:17:13 debian kernel: [ 484.243031] Code: Bad RIP value.
Nov 20 13:17:13 debian kernel: [ 484.243032] RSP: 002b:00007fff7f9474d8 EFLAGS: 00000246 ORIG_RAX: 000000000000004a
Nov 20 13:17:13 debian kernel: [ 484.243032] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f0790255214
Nov 20 13:17:13 debian kernel: [ 484.243033] RDX: 0000000000000008 RSI: 0000000000000000 RDI: 0000000000000003
Nov 20 13:17:13 debian kernel: [ 484.243033] RBP: 0000000000000003 R08: 0000000000000000 R09: 00007f07903ef680
Nov 20 13:17:13 debian kernel: [ 484.243034] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f07903ef680
Nov 20 13:17:13 debian kernel: [ 484.243034] R13: 000055cb7a993440 R14: 00007f07903ef680 R15: 0000000000000000

Marcel

unread,
Nov 20, 2019, 8:50:03 AM11/20/19
to
So, I started to dig a bit deeper by simply downgrading the qmeu-utils package.

Running the recent version of buster but version qemu-utils_2.8+dfsg-6+deb9u8 (http://snapshot.debian.org/package/qemu/1%3A2.8%2Bdfsg-6%2Bdeb9u8/#qemu-utils_1:3a:2.8:2b:dfsg-6:2b:deb9u8) works.

All other versions greater than this one did not succeed. But with qemu-utils_2.8+dfsg-6+deb9u8 all works again, as expected.

I hope this helps.

Cheers,
Marcel

Rich

unread,
Jun 8, 2021, 12:10:03 AM6/8/21
to
Lest anyone think this is resolved, I just experienced it on buster,
running kernel 4.19.0-16-amd64 and 5.2+dfsg-9~bpo10+1 from
buster-backports.

In my case, though, I've used qemu-nbd without difficulty for raw
files on the order of 40G before without difficulty - but this time,
with a VDI of actual size ~19G and...let's call it "virtual" size
100GB, I wrote around a GB of data to it and then suddenly

[2005064.948700] block nbd0: Connection timed out
[2005064.951474] block nbd0: shutting down sockets
[2005064.951479] print_req_error: I/O error, dev nbd0, sector 39028592
[2005064.954230] block nbd0: Connection timed out
[2005064.956982] print_req_error: I/O error, dev nbd0, sector 39029104
[2005064.958271] block nbd0: Connection timed out
[2005064.959345] print_req_error: I/O error, dev nbd0, sector 39029360
[2005064.960527] block nbd0: Connection timed out
[2005064.961608] print_req_error: I/O error, dev nbd0, sector 39029616
[2005064.962677] block nbd0: Connection timed out
[2005064.963726] print_req_error: I/O error, dev nbd0, sector 39029872

(I presume if I hadn't noticed and had waited long enough I too would
see "task blocked for xyz seconds")

The fact that the original report on this particular bug was using a
VDI as well makes me suspect it might be a problem with handling VDIs
- I'm going to try converting it and report back...

- Rich

Rich

unread,
Jun 12, 2021, 8:10:03 AM6/12/21
to
I come bearing data!

After a kind of long bisect, the bad commit was:
commit 1e886639791762e89b51aa0507f523c6a1448831
Author: Paolo Bonzini <pbon...@redhat.com>
Date: Thu Jun 29 15:27:41 2017 +0200

vdi: make it thread-safe

And then, when I went to verify it on git master before reporting it
against qemu upstream, I couldn't reproduce it against
894fc4fd670aaf04a67dc7507739f914ff4bacf2.

Another round of bisecting points to
commit 050de36b13f7a841b7805391bca44f36370e86e4
Author: Paolo Bonzini <pbon...@redhat.com>
Date: Thu Mar 25 12:29:39 2021 +0100

coroutine-lock: Reimplement CoRwlock to fix downgrade bug

as the first commit where it works reliably. Which is unfortunate, as
cherrypicking that looks a bit more invasive than is probably
reasonable.

- Rich
0 new messages