cloud_controller cause the kernal CPU stuck

22 views

Skip to first unread message

Jianbo Sun

unread,

Oct 28, 2014, 11:07:19 PM10/28/14

to vcap...@cloudfoundry.org

Hi，

Recently, I deploy the cloud_controller component on virtual machine 。 “Linux raw1204 3.2.0-70-virtual #105-Ubuntu SMP Wed Sep 24 20:06:46 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux”

After push some apps and the apps may start failed and restart by the cloudcontroller again and again.

Then the VM which has CloudController will casue a kernal BUG.

And then I can never use any of command related with the process unless I restart the VM.

Hope someone can give some help.

Thanks in advance！

Below is the kernal log：

Oct 28 18:14:07 raw1204 kernel: [18848.216086] BUG: soft lockup - CPU#0 stuck for 22s! [ruby:1165]

Oct 28 18:14:07 raw1204 kernel: [18848.216447] Modules linked in: nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc isofs fb_sys_fops sysimgblt sysfillrect syscopyarea joydev i2c_piix4 mac_hid acpiphp usbhid hid floppy

Oct 28 18:14:07 raw1204 kernel: [18848.216447] CPU 0

Oct 28 18:14:07 raw1204 kernel: [18848.216447]

Oct 28 18:14:07 raw1204 kernel: [18848.216447] Pid: 1165, comm: ruby Not tainted 3.2.0-70-virtual #105-Ubuntu Xen HVM domU

Oct 28 18:14:07 raw1204 kernel: [18848.216447] RIP: 0010:[<ffffffff81046920>] [<ffffffff81046920>] flush_tlb_others_ipi+0x120/0x130

Oct 28 18:14:07 raw1204 kernel: [18848.216447] RSP: 0000:ffff88020224fc28 EFLAGS: 00000202

Oct 28 18:14:07 raw1204 kernel: [18848.216447] RAX: 000000000000000f RBX: ffff8802008ab87c RCX: 0000000000000003

Oct 28 18:14:07 raw1204 kernel: [18848.216447] RDX: 0000000000000040 RSI: 0000000000000040 RDI: 0000000000000292

Oct 28 18:14:07 raw1204 kernel: [18848.216447] RBP: ffff88020224fc58 R08: ffffffff81e0b0a0 R09: 0000000000000040

Oct 28 18:14:07 raw1204 kernel: [18848.216447] R10: 57ffcbc4bc8ed2c0 R11: 00000000000026d8 R12: ffff880202280a00

Oct 28 18:14:07 raw1204 kernel: [18848.216447] R13: 0000000000000208 R14: ffffffff811635e4 R15: ffff88020224fbb8

Oct 28 18:14:07 raw1204 kernel: [18848.216447] FS: 00007f51ef851700(0000) GS:ffff88020fc00000(0000) knlGS:0000000000000000

Oct 28 18:14:07 raw1204 kernel: [18848.216447] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033

Oct 28 18:14:07 raw1204 kernel: [18848.216447] CR2: 0000000005ad6db8 CR3: 0000000201dd4000 CR4: 00000000001406f0

Oct 28 18:14:07 raw1204 kernel: [18848.216447] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000

Oct 28 18:14:07 raw1204 kernel: [18848.216447] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400

Oct 28 18:14:07 raw1204 kernel: [18848.216447] Process ruby (pid: 1165, threadinfo ffff88020224e000, task ffff8802022e16f0)

Oct 28 18:14:07 raw1204 kernel: [18848.216447] Stack:

Oct 28 18:14:07 raw1204 kernel: [18848.216447] 00000000000026d8 57ffcbc4bc8ed2c0 ffff8801f025a300 ffff8801f025a5d0

Oct 28 18:14:07 raw1204 kernel: [18848.216447] 0000000005ad6db8 0000000005ad6db8 ffff88020224fc68 ffffffff81046aae

Oct 28 18:14:07 raw1204 kernel: [18848.216447] ffff88020224fc98 ffffffff81046c1e ffffffff81045914 ffff8801ee7c9790

Oct 28 18:14:07 raw1204 kernel: [18848.216447] Call Trace:

Oct 28 18:14:07 raw1204 kernel: [18848.216447] [<ffffffff81046aae>] native_flush_tlb_others+0xe/0x10

Oct 28 18:14:07 raw1204 kernel: [18848.216447] [<ffffffff81046c1e>] flush_tlb_page+0x5e/0xb0

Oct 28 18:14:07 raw1204 kernel: [18848.216447] [<ffffffff81045914>] ? ptep_set_access_flags+0x54/0x70

Oct 28 18:14:07 raw1204 kernel: [18848.216447] [<ffffffff8104592c>] ptep_set_access_flags+0x6c/0x70

Oct 28 18:14:07 raw1204 kernel: [18848.216447] [<ffffffff8113c8ea>] do_wp_page+0x37a/0x740

Oct 28 18:14:07 raw1204 kernel: [18848.216447] [<ffffffff813aabde>] ? info_for_irq+0xe/0x30

Oct 28 18:14:07 raw1204 kernel: [18848.216447] [<ffffffff813ac071>] ? notify_remote_via_irq+0x31/0x50

Oct 28 18:14:07 raw1204 kernel: [18848.216447] [<ffffffff813aca2b>] ? xen_send_IPI_one+0x2b/0x30

Oct 28 18:14:07 raw1204 kernel: [18848.216447] [<ffffffff8113e89b>] handle_pte_fault+0x1cb/0x200

Oct 28 18:14:07 raw1204 kernel: [18848.216447] [<ffffffff8113faa9>] handle_mm_fault+0x269/0x370

Oct 28 18:14:07 raw1204 kernel: [18848.216447] [<ffffffff81060ab0>] ? wake_up_state+0x10/0x20

Oct 28 18:14:07 raw1204 kernel: [18848.216447] [<ffffffff816619c4>] do_page_fault+0x184/0x550

Oct 28 18:14:07 raw1204 kernel: [18848.216447] [<ffffffff810a152e>] ? do_futex+0x12e/0x1b0

Oct 28 18:14:07 raw1204 kernel: [18848.216447] [<ffffffff810a16f2>] ? sys_futex+0x142/0x1a0

Oct 28 18:14:07 raw1204 kernel: [18848.216447] [<ffffffff811798ad>] ? vfs_read+0x10d/0x180

Oct 28 18:14:07 raw1204 kernel: [18848.216447] [<ffffffff8165e5f5>] page_fault+0x25/0x30

Oct 28 18:14:07 raw1204 kernel: [18848.216447] Code: c9 00 41 8d b6 cf 00 00 00 49 8d 7d 18 ff 90 d0 00 00 00 49 83 bc 24 98 b0 e0 81 00 0f 84 74 ff ff ff 66 0f 1f 84 00 00 00 00 00 <f3> 90 49 83 7d 18 00 75 f7 e9 5d ff ff ff 66 90 55 48 89 e5 0f