Lockup on elevator switch to BFQ with upcoming 3.10.8 and related -stable releases

[Holger Hoffstätte]

Hi,

I just tested the upcoming 3.10.8-rc1 and was greeted with kernel fireworks & lockup on reboot. While deadline is the built-in default, I have a udev rule that switches elevators on a per-device basis (based on rotational flag).
Setting a device to BFQ resulted in:

Aug 19 12:39:32 ragnarok kernel: ------------[ cut here ]------------
Aug 19 12:39:32 ragnarok kernel: invalid opcode: 0000 [#1] SMP
Aug 19 12:39:32 ragnarok kernel: Modules linked in: usb_storage snd_hda_codec_hdmi usbhid snd_hda_codec_realtek radeon snd_hda_intel i2c_algo_bit snd_hda_codec drm_kms_helper snd_pcm ehci_pci ttm xhci_hcd coretemp snd_page_alloc ehci_hcd drm snd_timer usbcore i2c_i801 snd parport_pc usb_common parport i2c_core
Aug 19 12:39:32 ragnarok kernel: EIP: 0060:[<c02ab50d>] EFLAGS: 00010213 CPU: 0
Aug 19 12:39:32 ragnarok kernel: EIP is at deadline_exit_queue+0x1d/0x30
Aug 19 12:39:32 ragnarok kernel: EAX: f239f800 EBX: f20c9a00 ECX: 00000000 EDX: f239f808
Aug 19 12:39:32 ragnarok kernel: ESI: f20c9a2c EDI: 00000000 EBP: f27efecc ESP: f27efec0
Aug 19 12:39:32 ragnarok kernel:  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Aug 19 12:39:32 ragnarok kernel: CR0: 80050033 CR2: b775d000 CR3: 32b5b000 CR4: 000407f0
Aug 19 12:39:32 ragnarok kernel: DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Aug 19 12:39:32 ragnarok kernel: DR6: ffff0ff0 DR7: 00000400
Aug 19 12:39:32 ragnarok kernel:  c0298f25 f206c800 00000000 f27efefc c0298ff9 b775d000 80000002 f20c9a00
Aug 19 12:39:32 ragnarok kernel:  00716662 b775d000 f2b6edd8 b775d000 f20c9a00 f32c4000 00000003 f27eff18
Aug 19 12:39:32 ragnarok kernel:  c02999bf c066a4a4 f27eff18 f206ca2c f206cc30 c05e3c8c f27eff34 c029dbd6
Aug 19 12:39:32 ragnarok kernel:  [<c0298f25>] ? elevator_exit+0x25/0x40
Aug 19 12:39:32 ragnarok kernel:  [<c0298ff9>] elevator_change+0xb9/0x150
Aug 19 12:39:32 ragnarok kernel:  [<c02999bf>] elv_iosched_store+0x1f/0x50
Aug 19 12:39:32 ragnarok kernel:  [<c029dbd6>] queue_attr_store+0x46/0x80
Aug 19 12:39:32 ragnarok kernel:  [<c0219aa6>] sysfs_write_file+0x96/0xf0
Aug 19 12:39:32 ragnarok kernel:  [<c0219a10>] ? sysfs_open_file+0x200/0x200
Aug 19 12:39:32 ragnarok kernel:  [<c01c89cb>] vfs_write+0x9b/0x1c0
Aug 19 12:39:32 ragnarok kernel:  [<c0219a10>] ? sysfs_open_file+0x200/0x200
Aug 19 12:39:32 ragnarok kernel:  [<c01c8e72>] SyS_write+0x52/0xa0
Aug 19 12:39:32 ragnarok kernel:  [<c04b64be>] sysenter_do_call+0x12/0x26
Aug 19 12:39:32 ragnarok kernel: ---[ end trace bc4859ef143682ec ]---
Aug 19 12:39:32 ragnarok kernel: r8169 0000:04:00.0 eth0: unable to load firmware patch rtl_nic/rtl8168e-3.fw (-2)
Aug 19 12:39:32 ragnarok kernel: *pdpt = 00000000328ec001 *pde = 0000000000000000
Aug 19 12:39:32 ragnarok kernel: Oops: 0000 [#2] SMP
Aug 19 12:39:32 ragnarok kernel: Modules linked in: usb_storage snd_hda_codec_hdmi usbhid snd_hda_codec_realtek radeon snd_hda_intel i2c_algo_bit snd_hda_codec drm_kms_helper snd_pcm ehci_pci ttm xhci_hcd coretemp snd_page_alloc ehci_hcd drm snd_timer usbcore i2c_i801 snd parport_pc usb_common parport i2c_core
Aug 19 12:39:32 ragnarok kernel: r8169 0000:04:00.0 eth0: link down
Aug 19 12:39:32 ragnarok kernel: r8169 0000:04:00.0 eth0: link down
Aug 19 12:39:32 ragnarok kernel: IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Aug 19 12:39:32 ragnarok kernel: EIP: 0060:[<c02ab4a7>] EFLAGS: 00010046 CPU: 4
Aug 19 12:39:32 ragnarok kernel: EIP is at deadline_dispatch_requests+0xc7/0x110
Aug 19 12:39:32 ragnarok kernel: EAX: fffffb88 EBX: f239f800 ECX: 00000000 EDX: 00000000
Aug 19 12:39:32 ragnarok kernel: ESI: 00000000 EDI: f206c800 EBP: f2575b50 ESP: f2575b44
Aug 19 12:39:32 ragnarok kernel:  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Aug 19 12:39:32 ragnarok kernel: CR0: 80050033 CR2: 00000008 CR3: 328e9000 CR4: 000407f0
Aug 19 12:39:32 ragnarok kernel: DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
Aug 19 12:39:32 ragnarok kernel: DR6: ffff0ff0 DR7: 00000400
Aug 19 12:39:32 ragnarok kernel:  f24afc00 f206c800 f206c800 f2575b70 c029c9c9 00000000 f2575b70 c03674f5
Aug 19 12:39:32 ragnarok kernel:  f24afc00 f206c800 f1460000 f2575bb0 c036cac2 c01c029d 00000000 f29ecf68
Aug 19 12:39:32 ragnarok kernel:  00000077 f29eda48 00000000 f24afcbc c018eb2e f24afc28 f2708480 f29ecf68
Aug 19 12:39:32 ragnarok kernel:  [<c029c9c9>] blk_peek_request+0x129/0x1c0
Aug 19 12:39:32 ragnarok kernel:  [<c03674f5>] ? scsi_dispatch_cmd+0x95/0x150
Aug 19 12:39:32 ragnarok kernel:  [<c036cac2>] scsi_request_fn+0x32/0x410
Aug 19 12:39:32 ragnarok kernel:  [<c01c029d>] ? kmem_cache_alloc+0x6d/0xc0
Aug 19 12:39:32 ragnarok kernel:  [<c018eb2e>] ? mempool_alloc_slab+0xe/0x10
Aug 19 12:39:32 ragnarok kernel:  [<c0299e49>] __blk_run_queue+0x29/0x40
Aug 19 12:39:32 ragnarok kernel:  [<c029feb1>] blk_execute_rq_nowait+0x61/0xc0
Aug 19 12:39:32 ragnarok kernel:  [<c029ff6d>] blk_execute_rq+0x5d/0xb0
Aug 19 12:39:32 ragnarok kernel:  [<c029fe20>] ? blk_rq_map_user+0x240/0x240
Aug 19 12:39:32 ragnarok kernel:  [<c0290000>] ? crypto_shash_show+0x10/0x60
Aug 19 12:39:32 ragnarok kernel:  [<c014ba90>] ? abort_exclusive_wait+0x80/0x80
Aug 19 12:39:32 ragnarok kernel:  [<c029c264>] ? blk_get_request+0x54/0xc0
Aug 19 12:39:32 ragnarok kernel:  [<c036befb>] scsi_execute+0xab/0x140
Aug 19 12:39:32 ragnarok kernel:  [<c036c006>] scsi_execute_req_flags+0x76/0xe0
Aug 19 12:39:32 ragnarok kernel:  [<c036c105>] scsi_test_unit_ready+0x95/0x110
Aug 19 12:39:32 ragnarok kernel:  [<c0379d8d>] sd_check_events+0xed/0x130
Aug 19 12:39:32 ragnarok kernel:  [<c02a33fa>] disk_check_events+0x3a/0xe0
Aug 19 12:39:32 ragnarok kernel:  [<c02a4c86>] disk_clear_events+0x56/0xf0
Aug 19 12:39:32 ragnarok kernel:  [<c01f3e21>] check_disk_change+0x21/0x70
Aug 19 12:39:32 ragnarok kernel:  [<c0379ed7>] sd_open+0x67/0x130
Aug 19 12:39:32 ragnarok kernel:  [<c01f4768>] __blkdev_get+0x88/0x380
Aug 19 12:39:32 ragnarok kernel:  [<c01da4c3>] ? __d_free+0x33/0x50
Aug 19 12:39:32 ragnarok kernel:  [<c01f4bac>] blkdev_get+0x14c/0x270
Aug 19 12:39:32 ragnarok kernel:  [<c01f4d1d>] blkdev_open+0x4d/0x70
Aug 19 12:39:32 ragnarok kernel:  [<c01c6878>] do_dentry_open+0x1a8/0x220
Aug 19 12:39:32 ragnarok kernel:  [<c01d1e11>] ? __inode_permission+0x31/0x70
Aug 19 12:39:32 ragnarok kernel:  [<c01f4cd0>] ? blkdev_get+0x270/0x270
Aug 19 12:39:32 ragnarok kernel:  [<c01c6a9d>] finish_open+0x2d/0x40
Aug 19 12:39:32 ragnarok kernel:  [<c01d463b>] do_last.isra.50+0x26b/0xae0
Aug 19 12:39:32 ragnarok kernel:  [<c01d1e11>] ? __inode_permission+0x31/0x70
Aug 19 12:39:32 ragnarok kernel:  [<c01d1e61>] ? inode_permission+0x11/0x50
Aug 19 12:39:32 ragnarok kernel:  [<c01d1ef6>] ? link_path_walk+0x56/0x7a0
Aug 19 12:39:32 ragnarok kernel:  [<c01d4f46>] path_openat.isra.51+0x96/0x3e0
Aug 19 12:39:32 ragnarok kernel:  [<c02c2ebc>] ? copy_to_user+0x3c/0x50
Aug 19 12:39:32 ragnarok kernel:  [<c01d553c>] do_filp_open+0x2c/0x80
Aug 19 12:39:32 ragnarok kernel:  [<c01c7b8e>] do_sys_open+0xde/0x1b0
Aug 19 12:39:32 ragnarok kernel:  [<c01c7c7d>] SyS_open+0x1d/0x20
Aug 19 12:39:32 ragnarok kernel:  [<c04b64be>] sysenter_do_call+0x12/0x26
Aug 19 12:39:32 ragnarok kernel: CR2: 0000000000000008
Aug 19 12:39:32 ragnarok kernel: ---[ end trace bc4859ef143682ed ]---

This is reproducible: booting without the udev rule and then doing the usual "echo bfq > /sys/dev/sdX/queue/scheduler" also locks up pretty much immediately.

A look over the patches in 3.10.8 makes me think that this is very likely due to the following patch: http://lkml.org/lkml/2013/8/18/178 which fixes locking around elevator changes.

Hope you find this useful.

cheers
Holger

[Torsten Wegel]

same here and yes, the problem seems to be https://lkml.org/lkml/2013/8/18/178

thx

[Arianna Avanzini]

Il giorno lunedì 19 agosto 2013 20:56:33 UTC+2, Torsten Wegel ha scritto:

same here and yes, the problem seems to be https://lkml.org/lkml/2013/8/18/178

thx

Hello,

thank you for the detailed report; we have attempted to prepare a patch that should fix the issue. The attached patch should apply cleanly on top of BFQ-v6r2 for v3.10.0 and the resulting code should compile without warnings.

Please do not hesitate to report about any issue you might encounter.

[Torsten Wegel]

it works ;)

thx

[post-factum]

Confirm. Thanks for the fix, Arianna.

Вівторок, 20 серпня 2013 р. 15:48:43 UTC+3 користувач Torsten Wegel написав:

it works ;)

thx