kernel panic - not syncing: out of memory and no killable processes
Eric Paris
Latest I tried was linux-next as of today, Sept 16. I spent the day
today bisecting between next-20090911 and next-20090914 which was the
last and first day linux-next broke.
My fedora 12 install gets to udev starting and then panics with the
message:
Kernel panic - not syncing: Out of memory and no killable processes...
Bisect comes down to: 774a694f8cd08115d130a290d73c6d8563f26b1b
commit 774a694f8cd08115d130a290d73c6d8563f26b1b
Merge: 4f0ac85... e1f8450...
Author: Linus Torvalds <torv...@linux-foundation.org>
Date: Fri Sep 11 13:23:18 2009 -0700
Merge branch 'sched-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
* 'sched-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip: (64 commits)
sched: Fix sched::sched_stat_wait tracepoint field
sched: Disable NEW_FAIR_SLEEPERS for now
sched: Keep kthreads at default priority
sched: Re-tune the scheduler latency defaults to decrease worst-case latencies
sched: Turn off child_runs_first
sched: Ensure that a child can't gain time over it's parent after fork()
sched: enable SD_WAKE_IDLE
sched: Deal with low-load in wake_affine()
sched: Remove short cut from select_task_rq_fair()
sched: Turn on SD_BALANCE_NEWIDLE
sched: Clean up topology.h
sched: Fix dynamic power-balancing crash
sched: Remove reciprocal for cpu_power
sched: Try to deal with low capacity, fix update_sd_power_savings_stats()
sched: Try to deal with low capacity
sched: Scale down cpu_power due to RT tasks
sched: Implement dynamic cpu_power
sched: Add smt_gain
sched: Update the cpu_power sum during load-balance
sched: Add SD_PREFER_SIBLING
...
Which doesn't even change code so I'm not sure what that's telling me.
Boot log and config are attached. Hard to test patches to fix ingo's
selinux disable problems when something from the scheduler appears to be
stopping me from booting at all *smile*
-Eric
Eric Paris
> Starting on Sept 14 linux-next became unbootable on my KVM guests.
> Latest I tried was linux-next as of today, Sept 16. I spent the day
> today bisecting between next-20090911 and next-20090914 which was the
> last and first day linux-next broke.
>
> My fedora 12 install gets to udev starting and then panics with the
> message:
>
> Kernel panic - not syncing: Out of memory and no killable processes...
the next-20090916 tree still panic's the same way. As does Linus'
latest git tree ab86e5765
I'll try bisecting between v2.6.31 and master if it would give better
data than trying to bisect between two -next trees, but I'm sorta stuck
with a machine that won't boot and doesn't tell me much about why.
-Eric
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majo...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Eric Paris
> On Thu, 17 Sep 2009, Eric Paris wrote:
>
> > the next-20090916 tree still panic's the same way. As does Linus'
> > latest git tree ab86e5765
> >
> > I'll try bisecting between v2.6.31 and master if it would give better
> > data than trying to bisect between two -next trees, but I'm sorta stuck
> > with a machine that won't boot and doesn't tell me much about why.
> >
>
> Try merging the following patch to get more information such as the
> allocation order and gfp_mask. It will also show the VM size of all
> kthreads and the state of your system memory to deterine whether there's a
> page allocator issue, lowmem_reserve_ratio is too high, etc.
I'll throw it in.
I did another bisect (mainly because the last one gave a commit that
obviously seemed bogus and Ingo ask me to)
3f2aa307c4d26b4ed6509d0a79e8254c9e07e921 is first bad commit
commit 3f2aa307c4d26b4ed6509d0a79e8254c9e07e921
Author: Ingo Molnar <mi...@elte.hu>
Date: Thu Sep 10 20:34:48 2009 +0200
sched: Disable NEW_FAIR_SLEEPERS for now
I'm about to try a linus latest kernel with just this one patch
reverted.
David Rientjes
> the next-20090916 tree still panic's the same way. As does Linus'
> latest git tree ab86e5765
>
> I'll try bisecting between v2.6.31 and master if it would give better
> data than trying to bisect between two -next trees, but I'm sorta stuck
> with a machine that won't boot and doesn't tell me much about why.
>
It's panicking because the oom killer is getting called during boot and
there are no user tasks to kill, i.e. there are only kthreads running,
which it can't kill. It sounds like something is trying to allocate
higher order pages that cannot be found on your 512M machine because of
fragmentation.
Try merging the following patch to get more information such as the
allocation order and gfp_mask. It will also show the VM size of all
kthreads and the state of your system memory to deterine whether there's a
page allocator issue, lowmem_reserve_ratio is too high, etc.
---
diff --git a/mm/oom_kill.c b/mm/oom_kill.c
--- a/mm/oom_kill.c
+++ b/mm/oom_kill.c
@@ -540,6 +540,15 @@ retry:
/* Found nothing?!?! Either we hang forever, or we panic. */
if (!p) {
read_unlock(&tasklist_lock);
+ printk(KERN_WARNING "%s invoked oom-killer: "
+ "gfp_mask=0x%x, order=%d, oomkilladj=%d\n",
+ current->comm, gfp_mask, order, current->oomkilladj);
+ task_lock(current);
+ cpuset_print_task_mems_allowed(current);
+ task_unlock(current);
+ dump_stack();
+ show_mem();
+ dump_tasks(NULL);
panic("Out of memory and no killable processes...\n");
Eric Paris
> On Thu, 2009-09-17 at 13:29 -0700, David Rientjes wrote:
> > On Thu, 17 Sep 2009, Eric Paris wrote:
> >
> > > the next-20090916 tree still panic's the same way. As does Linus'
> > > latest git tree ab86e5765
> > >
> > > I'll try bisecting between v2.6.31 and master if it would give better
> > > data than trying to bisect between two -next trees, but I'm sorta stuck
> > > with a machine that won't boot and doesn't tell me much about why.
> > >
> >
>
> > Try merging the following patch to get more information such as the
> > allocation order and gfp_mask. It will also show the VM size of all
> > kthreads and the state of your system memory to deterine whether there's a
> > page allocator issue, lowmem_reserve_ratio is too high, etc.
>
> I'll throw it in.
>
> I did another bisect (mainly because the last one gave a commit that
> obviously seemed bogus and Ingo ask me to)
>
> 3f2aa307c4d26b4ed6509d0a79e8254c9e07e921 is first bad commit
> commit 3f2aa307c4d26b4ed6509d0a79e8254c9e07e921
> Author: Ingo Molnar <mi...@elte.hu>
> Date: Thu Sep 10 20:34:48 2009 +0200
>
> sched: Disable NEW_FAIR_SLEEPERS for now
>
> I'm about to try a linus latest kernel with just this one patch
> reverted.
Odd, I tried to revert this patch but it still went boom....
[ 12.000281] restorecon used greatest stack depth: 3568 bytes left
[ 14.388728] udevd invoked oom-killer: gfp_mask=0x200da, order=0, oomkilladj=-17
[ 14.393200] udevd cpuset=/ mems_allowed=0
[ 14.394340] Pid: 894, comm: udevd Tainted: G W 2.6.31 #144
[ 14.396633] Call Trace:
[ 14.397875] [<ffffffff81112ed5>] __out_of_memory+0x1d5/0x1f0
[ 14.400036] [<ffffffff81113184>] ? out_of_memory+0x1e4/0x220
[ 14.401910] [<ffffffff81113132>] out_of_memory+0x192/0x220
[ 14.403313] [<ffffffff81117764>] __alloc_pages_nodemask+0x694/0x6b0
[ 14.406238] [<ffffffff81149a42>] alloc_page_vma+0x82/0x140
[ 14.408671] [<ffffffff811304fa>] do_wp_page+0x10a/0x960
[ 14.410578] [<ffffffff810ae666>] ? __lock_acquire+0x3c6/0x6d0
[ 14.412073] [<ffffffff81130fd1>] ? handle_mm_fault+0x281/0x8d0
[ 14.417878] [<ffffffff811313af>] handle_mm_fault+0x65f/0x8d0
[ 14.419853] [<ffffffff81536095>] ? do_page_fault+0x1f5/0x470
[ 14.421778] [<ffffffff81532aac>] ? _spin_unlock_irqrestore+0x5c/0xb0
[ 14.424084] [<ffffffff81096166>] ? down_read_trylock+0x76/0x80
[ 14.425358] [<ffffffff81536102>] do_page_fault+0x262/0x470
[ 14.427677] [<ffffffff8153199e>] ? trace_hardirqs_off_thunk+0x3a/0x3c
[ 14.429933] [<ffffffff81533215>] page_fault+0x25/0x30
[ 14.432015] Mem-Info:
[ 14.433872] Node 0 DMA per-cpu:
[ 14.436211] CPU 0: hi: 0, btch: 1 usd: 0
[ 14.438967] CPU 1: hi: 0, btch: 1 usd: 0
[ 14.440258] CPU 2: hi: 0, btch: 1 usd: 0
[ 14.440259] CPU 3: hi: 0, btch: 1 usd: 0
[ 14.440260] CPU 4: hi: 0, btch: 1 usd: 0
[ 14.440262] CPU 5: hi: 0, btch: 1 usd: 0
[ 14.440263] Node 0 DMA32 per-cpu:
[ 14.440265] CPU 0: hi: 186, btch: 31 usd: 60
[ 14.440266] CPU 1: hi: 186, btch: 31 usd: 25
[ 14.440268] CPU 2: hi: 186, btch: 31 usd: 70
[ 14.440269] CPU 3: hi: 186, btch: 31 usd: 0
[ 14.440270] CPU 4: hi: 186, btch: 31 usd: 45
[ 14.440271] CPU 5: hi: 186, btch: 31 usd: 9
[ 14.440274] Active_anon:29401 active_file:53 inactive_anon:242
[ 14.440275] inactive_file:366 unevictable:0 dirty:0 writeback:0 unstable:0
[ 14.440275] free:1173 slab:77594 mapped:66 pagetables:1206 bounce:0
[ 14.440276] Node 0 DMA free:1996kB min:76kB low:92kB high:112kB active_anon:10524kB inactive_anon:0kB active_file:0kB inactive_file:8kB unevictable:0kB present:14140kB pages_scanned:0 all_unreclaimable? no
[ 14.440280] lowmem_reserve[]: 0 483 483 483
[ 14.440283] Node 0 DMA32 free:2696kB min:2772kB low:3464kB high:4156kB active_anon:107080kB inactive_anon:968kB active_file:212kB inactive_file:1456kB unevictable:0kB present:494944kB pages_scanned:192 all_unreclaimable? no
[ 14.440287] lowmem_reserve[]: 0 0 0 0
[ 14.440289] Node 0 DMA: 22*4kB 0*8kB 0*16kB 1*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2040kB
[ 14.440296] Node 0 DMA32: 146*4kB 4*8kB 0*16kB 1*32kB 2*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2696kB
[ 14.440302] 517 total pagecache pages
[ 14.440304] 0 pages in swap cache
[ 14.440305] Swap cache stats: add 0, delete 0, find 0/0
[ 14.440306] Free swap = 0kB
[ 14.440307] Total swap = 0kB
Eric Paris
> On Thu, 2009-09-17 at 16:35 -0400, Eric Paris wrote:
> > On Thu, 2009-09-17 at 13:29 -0700, David Rientjes wrote:
> > > On Thu, 17 Sep 2009, Eric Paris wrote:
> > >
> > > > the next-20090916 tree still panic's the same way. As does Linus'
> > > > latest git tree ab86e5765
> > > >
> > > > I'll try bisecting between v2.6.31 and master if it would give better
> > > > data than trying to bisect between two -next trees, but I'm sorta stuck
> > > > with a machine that won't boot and doesn't tell me much about why.
> > > >
> > >
> >
> > > Try merging the following patch to get more information such as the
> > > allocation order and gfp_mask. It will also show the VM size of all
> > > kthreads and the state of your system memory to deterine whether there's a
> > > page allocator issue, lowmem_reserve_ratio is too high, etc.
> >
> > I'll throw it in.
> >
> > I did another bisect (mainly because the last one gave a commit that
> > obviously seemed bogus and Ingo ask me to)
> >
> > 3f2aa307c4d26b4ed6509d0a79e8254c9e07e921 is first bad commit
> > commit 3f2aa307c4d26b4ed6509d0a79e8254c9e07e921
> > Author: Ingo Molnar <mi...@elte.hu>
> > Date: Thu Sep 10 20:34:48 2009 +0200
> >
> > sched: Disable NEW_FAIR_SLEEPERS for now
> >
> > I'm about to try a linus latest kernel with just this one patch
> > reverted.
>
> Odd, I tried to revert this patch but it still went boom....
I reapplied 3f2aa307 and it exploded a little differently....
[ 14.625084] udevd invoked oom-killer: gfp_mask=0x200da, order=0, oomkilladj=-17
[ 14.627347] udevd cpuset=/ mems_allowed=0
[ 14.628659] Pid: 776, comm: udevd Tainted: G W 2.6.31 #145
[ 14.630408] Call Trace:
[ 14.631096] [<ffffffff81112ed5>] __out_of_memory+0x1d5/0x1f0
[ 14.633489] [<ffffffff81113184>] ? out_of_memory+0x1e4/0x220
[ 14.635323] [<ffffffff81113132>] out_of_memory+0x192/0x220
[ 14.636307] [<ffffffff81117764>] __alloc_pages_nodemask+0x694/0x6b0
[ 14.637785] [<ffffffff81149a42>] alloc_page_vma+0x82/0x140
[ 14.638866] [<ffffffff811304fa>] do_wp_page+0x10a/0x960
[ 14.639935] [<ffffffff810ae666>] ? __lock_acquire+0x3c6/0x6d0
[ 14.641074] [<ffffffff81130fd1>] ? handle_mm_fault+0x281/0x8d0
[ 14.642203] [<ffffffff811313af>] handle_mm_fault+0x65f/0x8d0
[ 14.643191] [<ffffffff81536095>] ? do_page_fault+0x1f5/0x470
[ 14.644484] [<ffffffff81532aac>] ? _spin_unlock_irqrestore+0x5c/0xb0
[ 14.645715] [<ffffffff81096166>] ? down_read_trylock+0x76/0x80
[ 14.646862] [<ffffffff81536102>] do_page_fault+0x262/0x470
[ 14.647941] [<ffffffff8153199e>] ? trace_hardirqs_off_thunk+0x3a/0x3c
[ 14.649149] [<ffffffff81533215>] page_fault+0x25/0x30
[ 14.650257] Mem-Info:
[ 14.650932] Node 0 DMA per-cpu:
[ 14.651718] CPU 0: hi: 0, btch: 1 usd: 0
[ 14.652670] CPU 1: hi: 0, btch: 1 usd: 0
[ 14.653680] CPU 2: hi: 0, btch: 1 usd: 0
[ 14.655669] CPU 3: hi: 0, btch: 1 usd: 0
[ 14.659353] CPU 4: hi: 0, btch: 1 usd: 0
[ 14.660280] CPU 5: hi: 0, btch: 1 usd: 0
[ 14.661642] Node 0 DMA32 per-cpu:
[ 14.663063] CPU 0: hi: 186, btch: 31 usd: 108
[ 14.664747] CPU 1: hi: 186, btch: 31 usd: 84
[ 14.666423] CPU 2: hi: 186, btch: 31 usd: 162
[ 14.668121] CPU 3: hi: 186, btch: 31 usd: 83
[ 14.669731] CPU 4: hi: 186, btch: 31 usd: 129
[ 14.671422] CPU 5: hi: 186, btch: 31 usd: 150
[ 14.673172] Active_anon:27531 active_file:17 inactive_anon:2293
[ 14.673174] inactive_file:18 unevictable:0 dirty:0 writeback:0 unstable:0
[ 14.673175] free:1172 slab:77543 mapped:32 pagetables:1046 bounce:0
[ 14.679168] Node 0 DMA free:2004kB min:76kB low:92kB high:112kB active_anon:12532kB inactive_anon:128kB active_file:0kB inactive_file:4kB unevictable:0kB present:14140kB pages_scanned:0 all_unreclaimable? no
[ 14.683444] lowmem_reserve[]: 0 483 483 483
[ 14.684728] Node 0 DMA32 free:2684kB min:2772kB low:3464kB high:4156kB active_anon:97592kB inactive_anon:9044kB active_file:68kB inactive_file:68kB unevictable:0kB present:494944kB pages_scanned:650 all_unreclaimable? no
[ 14.688339] lowmem_reserve[]: 0 0 0 0
[ 14.689330] Node 0 DMA: 8*4kB 2*8kB 0*16kB 1*32kB 0*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2000kB
[ 14.695062] Node 0 DMA32: 19*4kB 2*8kB 0*16kB 1*32kB 0*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2684kB
[ 14.698175] 98 total pagecache pages
[ 14.699025] 0 pages in swap cache
[ 14.699968] Swap cache stats: add 0, delete 0, find 0/0
[ 14.701026] Free swap = 0kB
[ 14.701786] Total swap = 0kB
[ 14.708232] 131056 pages RAM
[ 14.709022] 18283 pages reserved
[ 14.709803] 9774 pages shared
[ 14.710450] 109429 pages non-shared
[ 14.711298] [ pid ] uid tgid total_vm rss cpu oom_adj name
[ 14.712287] [ 1] 0 1 21723 43 5 0 init
[ 14.713833] [ 776] 0 776 3362 276 2 -17 udevd
[ 14.715017] [ 780] 0 780 3621 509 3 -17 udevd
[ 14.716186] [ 782] 0 782 3691 588 2 -17 udevd
[ 14.717174] [ 783] 0 783 3691 590 3 -17 udevd
[ 14.718516] [ 785] 0 785 3592 503 2 -17 udevd
[ 14.719962] [ 787] 0 787 3691 598 0 -17 udevd
[ 14.721166] [ 789] 0 789 3559 462 1 -17 udevd
[ 14.722152] [ 790] 0 790 3690 576 3 -17 udevd
[ 14.723558] [ 791] 0 791 3724 615 0 -17 udevd
[ 14.724900] [ 792] 0 792 3757 667 4 -17 udevd
[ 14.726116] [ 794] 0 794 3784 673 3 -17 udevd
[ 14.727101] [ 795] 0 795 3625 530 1 -17 udevd
[ 14.731047] [ 796] 0 796 3658 567 4 -17 udevd
[ 14.732241] [ 797] 0 797 3295 212 3 -17 udevd
[ 14.733226] [ 798] 0 798 3757 667 3 -17 udevd
[ 14.734582] [ 799] 0 799 3691 586 5 -17 udevd
[ 14.735852] [ 800] 0 800 3757 669 0 -17 udevd
[ 14.737066] [ 801] 0 801 3690 577 4 -17 udevd
[ 14.738243] [ 802] 0 802 3685 596 0 -17 udevd
[ 14.739231] [ 804] 0 804 3757 667 0 -17 udevd
[ 14.740665] [ 805] 0 805 3757 669 0 -17 udevd
[ 14.741946] [ 806] 0 806 3690 576 4 -17 udevd
[ 14.743151] [ 808] 0 808 3724 616 4 -17 udevd
[ 14.744138] [ 810] 0 810 3691 577 5 -17 udevd
[ 14.745583] [ 812] 0 812 3757 666 1 -17 udevd
[ 14.746832] [ 814] 0 814 3757 668 4 -17 udevd
[ 14.748017] [ 815] 0 815 3592 499 1 -17 udevd
[ 14.749172] [ 816] 0 816 3757 669 3 -17 udevd
[ 14.750161] [ 817] 0 817 3526 423 5 -17 udevd
[ 14.751534] [ 828] 0 828 3691 592 4 -17 udevd
[ 14.752859] [ 832] 0 832 3757 661 2 -17 udevd
[ 14.754063] [ 836] 0 836 3757 660 2 -17 udevd
[ 14.755267] [ 837] 0 837 3757 662 5 -17 udevd
[ 14.756255] [ 839] 0 839 3757 661 0 -17 udevd
[ 14.757634] [ 840] 0 840 3757 663 4 -17 udevd
[ 14.758921] [ 852] 0 852 3790 704 4 -17 udevd
[ 14.760091] [ 856] 0 856 3691 593 4 -17 udevd
[ 14.761079] [ 860] 0 860 3826 707 4 -17 udevd
[ 14.762403] [ 861] 0 861 3818 707 1 -17 udevd
[ 14.765543] [ 868] 0 868 3460 371 1 -17 udevd
[ 14.766766] [ 870] 0 870 3724 627 0 -17 udevd
[ 14.768009] [ 872] 0 872 3724 627 4 -17 udevd
[ 14.769186] [ 877] 0 877 3724 627 5 -17 udevd
[ 14.770175] [ 878] 0 878 3724 628 4 -17 udevd
[ 14.771487] [ 880] 0 880 3724 628 5 -17 udevd
[ 14.772752] [ 881] 0 881 3729 639 1 -17 udevd
[ 14.773932] [ 882] 0 882 3485 379 2 -17 udevd
[ 14.775106] [ 886] 0 886 3724 629 2 -17 udevd
[ 14.776093] [ 887] 0 887 3724 627 4 -17 udevd
[ 14.777412] [ 889] 0 889 3592 496 0 -17 udevd
[ 14.778400] [ 890] 0 890 3724 627 1 -17 udevd
[ 14.779826] [ 891] 0 891 3625 533 5 -17 udevd
[ 14.780980] [ 892] 0 892 3427 316 1 -17 udevd
[ 14.782156] [ 893] 0 893 3723 628 4 -17 udevd
[ 14.783144] [ 894] 0 894 3748 635 4 -17 udevd
[ 14.784467] [ 895] 0 895 3724 637 0 -17 udevd
[ 14.785669] [ 896] 0 896 3559 450 4 -17 udevd
[ 14.786911] [ 897] 0 897 3691 605 4 -17 udevd
[ 14.788076] [ 898] 0 898 3724 638 1 -17 udevd
[ 14.789064] [ 899] 0 899 3658 563 4 -17 udevd
[ 14.790377] [ 900] 0 900 2888 14 4 -17 devkit-disks-pa
[ 14.791775] [ 901] 0 901 3658 564 4 -17 udevd
[ 14.792949] [ 902] 0 902 3691 605 3 -17 udevd
[ 14.794120] [ 903] 0 903 1149 143 1 -17 modprobe
[ 14.795108] [ 904] 0 904 3493 388 0 -17 udevd
[ 14.796471] [ 905] 0 905 84 9 5 -17 console_check
[ 14.797862] [ 906] 0 906 3625 521 4 -17 udevd
[ 14.800984] [ 907] 0 907 827 9 4 -17 rename_device
[ 14.802262] [ 908] 0 908 3625 531 0 -17 udevd
[ 14.803250] [ 909] 0 909 3493 390 3 -17 udevd
[ 14.804579] [ 910] 0 910 3460 363 1 -17 udevd
[ 14.805851] [ 912] 0 912 3757 669 3 -17 udevd
[ 14.807011] [ 913] 0 913 3362 276 5 -17 udevd
[ 14.808183] Kernel panic - not syncing: Out of memory and no killable processes...
[ 14.808184]
[ 14.810288] Pid: 776, comm: udevd Tainted: G W 2.6.31 #145
[ 14.811275] Call Trace:
[ 14.812101] [<ffffffff8106fcac>] panic+0xbc/0x1a0
[ 14.813051] [<ffffffff81532c3f>] ? _spin_unlock+0x3f/0x80
[ 14.814127] [<ffffffff8111246f>] ? dump_tasks+0xff/0x130
[ 14.815110] [<ffffffff81018291>] ? dump_stack+0x81/0xa0
[ 14.816192] [<ffffffff81112eef>] __out_of_memory+0x1ef/0x1f0
[ 14.817180] [<ffffffff81113184>] ? out_of_memory+0x1e4/0x220
[ 14.818390] [<ffffffff81113132>] out_of_memory+0x192/0x220
[ 14.819379] [<ffffffff81117764>] __alloc_pages_nodemask+0x694/0x6b0
[ 14.820772] [<ffffffff81149a42>] alloc_page_vma+0x82/0x140
[ 14.821915] [<ffffffff811304fa>] do_wp_page+0x10a/0x960
[ 14.822955] [<ffffffff810ae666>] ? __lock_acquire+0x3c6/0x6d0
[ 14.824060] [<ffffffff81130fd1>] ? handle_mm_fault+0x281/0x8d0
[ 14.825175] [<ffffffff811313af>] handle_mm_fault+0x65f/0x8d0
[ 14.826163] [<ffffffff81536095>] ? do_page_fault+0x1f5/0x470
[ 14.827392] [<ffffffff81532aac>] ? _spin_unlock_irqrestore+0x5c/0xb0
[ 14.828380] [<ffffffff81096166>] ? down_read_trylock+0x76/0x80
[ 14.829807] [<ffffffff81536102>] do_page_fault+0x262/0x470
[ 14.830892] [<ffffffff8153199e>] ? trace_hardirqs_off_thunk+0x3a/0x3c
[ 14.832100] [<ffffffff81533215>] page_fault+0x25/0x30
Eric Paris
> On Thu, 2009-09-17 at 16:35 -0400, Eric Paris wrote:
> > On Thu, 2009-09-17 at 13:29 -0700, David Rientjes wrote:
> > > On Thu, 17 Sep 2009, Eric Paris wrote:
> > >
> > > > the next-20090916 tree still panic's the same way. As does Linus'
> > > > latest git tree ab86e5765
> > > >
> > > > I'll try bisecting between v2.6.31 and master if it would give better
> > > > data than trying to bisect between two -next trees, but I'm sorta stuck
> > > > with a machine that won't boot and doesn't tell me much about why.
> > > >
> > >
> >
> > > Try merging the following patch to get more information such as the
> > > allocation order and gfp_mask. It will also show the VM size of all
> > > kthreads and the state of your system memory to deterine whether there's a
> > > page allocator issue, lowmem_reserve_ratio is too high, etc.
> >
> > I'll throw it in.
> >
> > I did another bisect (mainly because the last one gave a commit that
> > obviously seemed bogus and Ingo ask me to)
> >
> > 3f2aa307c4d26b4ed6509d0a79e8254c9e07e921 is first bad commit
> > commit 3f2aa307c4d26b4ed6509d0a79e8254c9e07e921
> > Author: Ingo Molnar <mi...@elte.hu>
> > Date: Thu Sep 10 20:34:48 2009 +0200
> >
> > sched: Disable NEW_FAIR_SLEEPERS for now
> >
> > I'm about to try a linus latest kernel with just this one patch
> > reverted.
>
> Odd, I tried to revert this patch but it still went boom....
Guess I did something wrong lsat time I tried to revert. I'm now at 5
successful boots with the patch reverted. Don't ask me how changing
NEW_FAIR_SLEEPERS causes me to run out of Node 0 DMA32 memory. But it
does....
-Eric
David Rientjes
> [ 14.625084] udevd invoked oom-killer: gfp_mask=0x200da, order=0, oomkilladj=-17
Order-0 GFP_HIGHUSER_MOVABLE allocation from a task that is OOM_DISABLE.
> [ 14.627347] udevd cpuset=/ mems_allowed=0
> [ 14.628659] Pid: 776, comm: udevd Tainted: G W 2.6.31 #145
> [ 14.630408] Call Trace:
> [ 14.631096] [<ffffffff81112ed5>] __out_of_memory+0x1d5/0x1f0
> [ 14.633489] [<ffffffff81113184>] ? out_of_memory+0x1e4/0x220
> [ 14.635323] [<ffffffff81113132>] out_of_memory+0x192/0x220
> [ 14.636307] [<ffffffff81117764>] __alloc_pages_nodemask+0x694/0x6b0
> [ 14.637785] [<ffffffff81149a42>] alloc_page_vma+0x82/0x140
> [ 14.638866] [<ffffffff811304fa>] do_wp_page+0x10a/0x960
Writing to a shared page, the source of GFP_HIGHUSER_MOVABLE.
303MB of slab, which is well over half your system's total memory
capacity.
Looks udevd related, and you mentioned that this is the point where it
always panics in your initial report. It doesn't appear to be a VM issue.
Finding that change between next-20090911 and next-20090914 may be time
consuming.
Eric Paris
Absolutely 100% udevd related. It always pops right when udev starts.
Every time. I don't know how to get that to come out on the serial
port, you only get to see the kernel messages (although there is nothing
on the screen other than "Starting udev"
Reverting 3f2aa307c4d26b4ed6509d0a79e8254c9e07e921 causes the problem to
go away. I now am running a 79b520e87e kernel + your patch (adding
console_verbose()) + reverting 3f2aa307c4d2 and it seems to be working.
-Eric
David Rientjes
> Absolutely 100% udevd related. It always pops right when udev starts.
> Every time. I don't know how to get that to come out on the serial
> port, you only get to see the kernel messages (although there is nothing
> on the screen other than "Starting udev"
>
> Reverting 3f2aa307c4d26b4ed6509d0a79e8254c9e07e921 causes the problem to
> go away. I now am running a 79b520e87e kernel + your patch (adding
> console_verbose()) + reverting 3f2aa307c4d2 and it seems to be working.
>
Isolating udevd down to an interactivity scheduling change isn't _that_
bizarre. I think the setting of UDEVD_PRIORITY is already mostly
arbitrary anyway and it'll allow 192 children on your 512M machine by
default unless you changed UDEVD_MAX_CHILDS for uid 0.
The default timeout for idle workers is 3 seconds, which may just happen
to be long enough to panic your machine because of low memory. If that's
the case, I don't believe that it's a scheduler issue but rather a root
abuse of setting all udevd threads to be OOM_DISABLE.
What is your udevd --version? The latest is udev-146 released last month.
Maybe linux-hotplug (added to cc) already knows about this issue?
Eric Paris
> On Thu, 17 Sep 2009, Eric Paris wrote:
>
> > Absolutely 100% udevd related. It always pops right when udev starts.
> > Every time. I don't know how to get that to come out on the serial
> > port, you only get to see the kernel messages (although there is nothing
> > on the screen other than "Starting udev"
> >
> > Reverting 3f2aa307c4d26b4ed6509d0a79e8254c9e07e921 causes the problem to
> > go away. I now am running a 79b520e87e kernel + your patch (adding
> > console_verbose()) + reverting 3f2aa307c4d2 and it seems to be working.
> >
>
> Isolating udevd down to an interactivity scheduling change isn't _that_
> bizarre. I think the setting of UDEVD_PRIORITY is already mostly
> arbitrary anyway and it'll allow 192 children on your 512M machine by
> default unless you changed UDEVD_MAX_CHILDS for uid 0.
>
> The default timeout for idle workers is 3 seconds, which may just happen
> to be long enough to panic your machine because of low memory. If that's
> the case, I don't believe that it's a scheduler issue but rather a root
> abuse of setting all udevd threads to be OOM_DISABLE.
>
> What is your udevd --version? The latest is udev-146 released last month.
145
Let me try and clone the vm some I don't break my reproducer. I'll see
if adding more memory fixes it. Doesn't look like Fedora has built a
-146 yet, I'll see if I can get one of those as well.
udev bug, configuration issue, whatever, or not, it's a regression that
I used to be able to boot and updating my kernel leaves me unable to
boot. I think we all agree when 512M of memory isn't enough to boot to
runlevel 3 we've got a problem :)
-Eric
David Rientjes
> > Isolating udevd down to an interactivity scheduling change isn't _that_
> > bizarre. I think the setting of UDEVD_PRIORITY is already mostly
> > arbitrary anyway and it'll allow 192 children on your 512M machine by
> > default unless you changed UDEVD_MAX_CHILDS for uid 0.
> >
> > The default timeout for idle workers is 3 seconds, which may just happen
> > to be long enough to panic your machine because of low memory. If that's
> > the case, I don't believe that it's a scheduler issue but rather a root
> > abuse of setting all udevd threads to be OOM_DISABLE.
> >
> > What is your udevd --version? The latest is udev-146 released last month.
>
> 145
>
> Let me try and clone the vm some I don't break my reproducer. I'll see
> if adding more memory fixes it. Doesn't look like Fedora has built a
> -146 yet, I'll see if I can get one of those as well.
>
> udev bug, configuration issue, whatever, or not, it's a regression that
> I used to be able to boot and updating my kernel leaves me unable to
> boot. I think we all agree when 512M of memory isn't enough to boot to
> runlevel 3 we've got a problem :)
>
I totally agree, and my hypothesis is that the idle child workers are not
being killed in time that they quickly accumulate approaching
UDEVD_MAX_CHILDS and when the oom killer is called because of a write to
shared memory, it can't kill any of these threads either since udevd sets
them all to OOM_DISABLE and everything else is an unkillable kthread.
Bisecting that to a scheduler change would suggest that each udevd thread
isn't returning from its poll() timeout fast enough; there's essentially a
street race between udevd killing its own threads off because the poll
timeout was exceeded and all your memory being used up and the machine
panicking. The scheduling change seems to have affected the speed of the
former.
UDEVD_MAX_CHILDS defaults to 192 on your 512M machine unless overridden by
an environment variable of the same name, so you may find it helpful to
reduce this to a saner value. I'd suggest a value lower than the number
of udevd threads that were shown in your latest oom killer dump.
If that turns out to fix the issue for you, perhaps max_childs needs to be
calculated in a slightly more conservative way in the userspace package
since all threads come with the prerequisite of being OOM_DISABLE.