Filesystem kernel hangup, 2.6.3 (bad: scheduling while atomic!)
Mikael Wahlberg
On heavy FTP Load (About 1Gbit/s) running both reads and writes on two ServeRAID6m Raid5 controllers merged together to one filesystem with Raidtools we see the error below. The filesystem gets totally hanged up. Currently with XFS, but JFS gets the same problem (Actually even more often).
Anybody has got a good idea what can be wrong?
Distribution:
RedHat 9.0
Hardware:
IBM x345
2x2.4GHz Xeon
2xServeRAID6m
Error:
(/var/log/messages)
Feb 22 15:00:51 mserv1 kernel: Unable to handle kernel NULL pointer dereference at virtual address 000002a6
Feb 22 15:00:52 mserv1 kernel: printing eip:
Feb 22 15:00:52 mserv1 kernel: c011e5b5
Feb 22 15:00:52 mserv1 kernel: *pde = 00000000
Feb 22 15:00:52 mserv1 kernel: Oops: 0002 [#1]
Feb 22 15:00:52 mserv1 kernel: CPU: 1
Feb 22 15:00:52 mserv1 kernel: EIP: 0060:[<c011e5b5>] Not tainted
Feb 22 15:00:52 mserv1 kernel: EFLAGS: 00010003
Feb 22 15:00:52 mserv1 kernel: Process proftpd (pid: 7432, threadinfo=ee530000 task=ef59a6b0)
Feb 22 15:00:52 mserv1 kernel: Stack: c011e54a ee531a28 00000003 00000000 ddcdfeec ee530000 ddcdfee8 00000292
Feb 22 15:00:53 mserv1 kernel: ee531a50 c011e5af ddcdfee8 00000003 00000001 00000000 ddcdfee0 ddcdfe50
Feb 22 15:00:53 mserv1 kernel: 00000000 f5e36000 c0259e62 c0259b00 000000c8 c023038e ddcdfee0 000b2f80
Feb 22 15:00:53 mserv1 kernel: Call Trace:
Feb 22 15:00:53 mserv1 kernel: [<c011e54a>] __wake_up_common+0x3a/0x60
Feb 22 15:00:53 mserv1 kernel: [<c011e5af>] __wake_up+0x3f/0x70
Feb 22 15:00:53 mserv1 kernel: [<c0259e62>] mrunlock+0x82/0xb0
Feb 22 15:00:53 mserv1 kernel: [<c0259b00>] mraccessf+0xc0/0xe0
Feb 22 15:00:53 mserv1 kernel: [<c023038e>] xfs_iunlock+0x3e/0x80
Feb 22 15:00:53 mserv1 kernel: [<c023727b>] xfs_iomap+0x3bb/0x540
Feb 22 15:00:53 mserv1 kernel: [<c0163fc7>] bio_alloc+0xd7/0x1c0
Feb 22 15:00:53 mserv1 kernel: [<c025a17a>] map_blocks+0x7a/0x170
Feb 22 15:00:53 mserv1 kernel: [<c025b40b>] page_state_convert+0x52b/0x6d0
Feb 22 15:00:53 mserv1 kernel: [<c0236cb9>] xfs_imap_to_bmap+0x39/0x240
Feb 22 15:00:53 mserv1 kernel: [<c025be48>] linvfs_release_page+0xa8/0xb0
Feb 22 15:00:53 mserv1 kernel: [<c025bce0>] linvfs_writepage+0x60/0x120
Feb 22 15:00:53 mserv1 kernel: [<c014990c>] shrink_list+0x41c/0x710
Feb 22 15:00:53 mserv1 kernel: [<c0149df8>] shrink_cache+0x1f8/0x3d0
Feb 22 15:00:53 mserv1 kernel: [<c01b3a00>] journal_stop+0x220/0x330
Feb 22 15:00:53 mserv1 kernel: [<c014a6dc>] shrink_zone+0xbc/0xc0
Feb 22 15:00:53 mserv1 kernel: [<c014a7a5>] shrink_caches+0xc5/0xe0
Feb 22 15:00:54 mserv1 kernel: [<c014a87c>] try_to_free_pages+0xbc/0x190
Feb 22 15:00:54 mserv1 kernel: [<c0143043>] __alloc_pages+0x203/0x370
Feb 22 15:00:54 mserv1 kernel: [<c01431d5>] __get_free_pages+0x25/0x40
Feb 22 15:00:54 mserv1 kernel: [<c0173241>] __pollwait+0x41/0xd0
Feb 22 15:00:54 mserv1 kernel: [<c0358f93>] tcp_poll+0x33/0x190
Feb 22 15:00:54 mserv1 kernel: [<c032b0f9>] sock_poll+0x29/0x40
Feb 22 15:00:54 mserv1 kernel: [<c01736b7>] do_select+0x2f7/0x320
Feb 22 15:00:54 mserv1 kernel: [<c0173200>] __pollwait+0x0/0xd0
Feb 22 15:00:54 mserv1 kernel: [<c0173a12>] sys_select+0x302/0x540
Feb 22 15:00:54 mserv1 kernel: [<c010b08b>] syscall_call+0x7/0xb
Feb 22 15:00:54 mserv1 kernel:
Feb 22 15:00:54 mserv1 kernel: Code: ff 4b 14 8b 43 08 83 e0 08 75 0d 8b 5d f4 8b 75 f8 8b 7d fc
Feb 22 15:00:54 mserv1 kernel: [<c0173a12>] sys_select+0x302/0x540
Feb 22 15:00:54 mserv1 kernel: [<c010b08b>] syscall_call+0x7/0xb
Feb 22 15:00:54 mserv1 kernel:
Feb 22 15:00:54 mserv1 kernel: Code: ff 4b 14 8b 43 08 83 e0 08 75 0d 8b 5d f4 8b 75 f8 8b 7d fc
Feb 22 15:00:54 mserv1 kernel: <6>note: proftpd[7432] exited with preempt_count 2
Feb 22 15:00:54 mserv1 kernel: bad: scheduling while atomic!
Feb 22 15:00:54 mserv1 kernel: Call Trace:
Feb 22 15:00:54 mserv1 kernel: [<c011e48e>] schedule+0x6ee/0x700
Feb 22 15:00:54 mserv1 kernel: [<c014cdeb>] zap_pmd_range+0x4b/0x70
Feb 22 15:00:54 mserv1 kernel: [<c01591ba>] free_pages_and_swap_cache+0x6a/0xa0
Feb 22 15:00:54 mserv1 kernel: [<c014d0cc>] unmap_vmas+0x23c/0x2f0
Feb 22 15:00:54 mserv1 kernel: [<c0151774>] exit_mmap+0xf4/0x250
Feb 22 15:00:54 mserv1 kernel: [<c0120d2d>] mmput+0x6d/0xa0
Feb 22 15:00:54 mserv1 kernel: [<c0125a33>] do_exit+0x1a3/0x500
Feb 22 15:00:54 mserv1 kernel: [<c010c1ac>] die+0xfc/0x100
Feb 22 15:00:54 mserv1 kernel: [<c011b349>] do_page_fault+0x1f9/0x523
Feb 22 15:00:54 mserv1 kernel: [<c0141bab>] mempool_alloc+0x8b/0x190
Feb 22 15:00:54 mserv1 kernel: [<c011b150>] do_page_fault+0x0/0x523
Feb 22 15:00:54 mserv1 kernel: [<c010baf5>] error_code+0x2d/0x38
Feb 22 15:00:54 mserv1 kernel: [<c011e5b5>] __wake_up+0x45/0x70
Feb 22 15:00:54 mserv1 kernel: [<c011e54a>] __wake_up_common+0x3a/0x60
Feb 22 15:00:55 mserv1 kernel: [<c011e5af>] __wake_up+0x3f/0x70
Feb 22 15:00:55 mserv1 kernel: [<c0259e62>] mrunlock+0x82/0xb0
Feb 22 15:00:55 mserv1 kernel: [<c0259b00>] mraccessf+0xc0/0xe0
Feb 22 15:00:55 mserv1 kernel: [<c023038e>] xfs_iunlock+0x3e/0x80
Feb 22 15:00:55 mserv1 kernel: [<c023727b>] xfs_iomap+0x3bb/0x540
Feb 22 15:00:55 mserv1 kernel: [<c0163fc7>] bio_alloc+0xd7/0x1c0
Feb 22 15:00:55 mserv1 kernel: [<c025a17a>] map_blocks+0x7a/0x170
Feb 22 15:00:55 mserv1 kernel: [<c025b40b>] page_state_convert+0x52b/0x6d0
Feb 22 15:00:55 mserv1 kernel: [<c0236cb9>] xfs_imap_to_bmap+0x39/0x240
Feb 22 15:00:55 mserv1 kernel: [<c025be48>] linvfs_release_page+0xa8/0xb0
Feb 22 15:00:55 mserv1 kernel: [<c025bce0>] linvfs_writepage+0x60/0x120
Feb 22 15:00:55 mserv1 kernel: [<c014990c>] shrink_list+0x41c/0x710
Feb 22 15:00:55 mserv1 kernel: [<c0149df8>] shrink_cache+0x1f8/0x3d0
Feb 22 15:00:55 mserv1 kernel: [<c01b3a00>] journal_stop+0x220/0x330
Feb 22 15:00:55 mserv1 kernel: [<c014a6dc>] shrink_zone+0xbc/0xc0
Feb 22 15:00:55 mserv1 kernel: [<c014a7a5>] shrink_caches+0xc5/0xe0
Feb 22 15:00:55 mserv1 kernel: [<c014a87c>] try_to_free_pages+0xbc/0x190
Feb 22 15:00:55 mserv1 kernel: [<c0143043>] __alloc_pages+0x203/0x370
Feb 22 15:00:55 mserv1 kernel: [<c01431d5>] __get_free_pages+0x25/0x40
Feb 22 15:00:55 mserv1 kernel: [<c0173241>] __pollwait+0x41/0xd0
Feb 22 15:00:55 mserv1 kernel: [<c0358f93>] tcp_poll+0x33/0x190
Feb 22 15:00:55 mserv1 kernel: [<c032b0f9>] sock_poll+0x29/0x40
Feb 22 15:00:55 mserv1 kernel: [<c01736b7>] do_select+0x2f7/0x320
Feb 22 15:00:55 mserv1 kernel: [<c0173200>] __pollwait+0x0/0xd0
Feb 22 15:00:55 mserv1 kernel: [<c0173a12>] sys_select+0x302/0x540
Feb 22 15:00:55 mserv1 kernel: [<c010b08b>] syscall_call+0x7/0xb
Feb 22 15:00:55 mserv1 kernel:
Feb 22 15:00:55 mserv1 kernel: Unable to handle kernel NULL pointer dereference at virtual address 000002a6
Feb 22 15:00:55 mserv1 kernel: printing eip:
Feb 22 15:00:55 mserv1 kernel: c011e5b5
Feb 22 15:00:55 mserv1 kernel: *pde = 00000000
Feb 22 15:00:55 mserv1 kernel: Oops: 0002 [#2]
Feb 22 15:00:55 mserv1 kernel: CPU: 1
Feb 22 15:00:55 mserv1 kernel: EIP: 0060:[<c011e5b5>] Not tainted
Feb 22 15:00:55 mserv1 kernel: EFLAGS: 00010003
Feb 22 15:00:55 mserv1 kernel: EIP is at __wake_up+0x45/0x70
Feb 22 15:00:55 mserv1 kernel: eax: ee531a00 ebx: 00000292 ecx: 00000001 edx: 00000003
Feb 22 15:00:55 mserv1 kernel: esi: ddcdfee8 edi: 00000001 ebp: c2ae5d80 esp: c2ae5d60
Feb 22 15:00:55 mserv1 kernel: ds: 007b es: 007b ss: 0068
Feb 22 15:00:55 mserv1 kernel: Process pdflush (pid: 20, threadinfo=c2ae4000 task=c2aead20)
Feb 22 15:00:55 mserv1 kernel: Stack: c011e54a ee531a28 00000003 00000000 ddcdfeec c2ae4000 ddcdfee8 00000296
Feb 22 15:00:55 mserv1 kernel: c2ae5da4 c011e5af ddcdfee8 00000003 00000001 00000000 ddcdfee0 ddcdfe50
Feb 22 15:00:56 mserv1 kernel: e25319a0 f5e36000 c0259e62 00000008 00000008 c023038e ddcdfee0 ddcdfe50
Feb 22 15:00:56 mserv1 kernel: Call Trace:
Feb 22 15:00:56 mserv1 kernel: [<c011e54a>] __wake_up_common+0x3a/0x60
Feb 22 15:00:56 mserv1 kernel: [<c011e5af>] __wake_up+0x3f/0x70
Feb 22 15:00:56 mserv1 kernel: [<c0259e62>] mrunlock+0x82/0xb0
Feb 22 15:00:56 mserv1 kernel: [<c023038e>] xfs_iunlock+0x3e/0x80
Feb 22 15:00:56 mserv1 kernel: [<c023554a>] xfs_iflush+0x36a/0x550
Feb 22 15:00:56 mserv1 kernel: [<c02573b5>] xfs_inode_flush+0x255/0x2c0
Feb 22 15:00:56 mserv1 kernel: [<c011e4f0>] default_wake_function+0x0/0x20
Feb 22 15:00:56 mserv1 kernel: [<c025bc80>] linvfs_writepage+0x0/0x120
Feb 22 15:00:56 mserv1 kernel: [<c02645b2>] linvfs_write_inode+0x32/0x40
Feb 22 15:00:56 mserv1 kernel: [<c0182806>] write_inode+0x46/0x50
Feb 22 15:00:56 mserv1 kernel: [<c0182a60>] __sync_single_inode+0x250/0x2a0
Feb 22 15:00:56 mserv1 kernel: [<c0182d4c>] sync_sb_inodes+0x1ac/0x2a0
Feb 22 15:00:56 mserv1 kernel: [<c0182ec9>] writeback_inodes+0x89/0x130
Feb 22 15:00:56 mserv1 kernel: [<c01442c8>] background_writeout+0xb8/0x100
Feb 22 15:00:56 mserv1 kernel: [<c0144afa>] __pdflush+0x10a/0x220
Feb 22 15:00:56 mserv1 kernel: [<c0144c10>] pdflush+0x0/0x20
Feb 22 15:00:56 mserv1 kernel: [<c0144c1f>] pdflush+0xf/0x20
Feb 22 15:00:56 mserv1 kernel: [<c0144210>] background_writeout+0x0/0x100
Feb 22 15:00:56 mserv1 kernel: [<c0108d14>] kernel_thread_helper+0x0/0xc
Feb 22 15:00:56 mserv1 kernel: [<c0108d19>] kernel_thread_helper+0x5/0xc
Feb 22 15:00:56 mserv1 kernel:
Feb 22 15:00:56 mserv1 kernel: Code: ff 4b 14 8b 43 08 83 e0 08 75 0d 8b 5d f4 8b 75 f8 8b 7d fc
Feb 22 15:00:56 mserv1 kernel: <6>note: pdflush[20] exited with preempt_count 3
/Mikael
--
-----------------------------------------------------------------------
Mikael Wahlberg, M.Sc. Ardendo
Unit Manager Professional Services/ e-mail: mik...@ardendo.se
Technical Project Manager GSM: +46 733 279 274
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majo...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Christoph Hellwig
> Description:
>
> On heavy FTP Load (About 1Gbit/s) running both reads and writes on two ServeRAID6m Raid5 controllers merged together to one filesystem with Raidtools we see the error below. The filesystem gets totally hanged up. Currently with XFS, but JFS gets the same problem (Actually even more often).
What does the JFS oops look like?
> Feb 22 15:00:53 mserv1 kernel: [<c011e54a>] __wake_up_common+0x3a/0x60
> Feb 22 15:00:53 mserv1 kernel: [<c011e5af>] __wake_up+0x3f/0x70
This doesn't make a lot of sense, there's only two mrlocks in XFS, and
they're in the inodes that have well-defined and understood lifetime rules.
OTOH the previos oops might have messed quite a bit up in your system.
did you run memtest86 on the box? do you some strange patches applied or
external modules loaded? What's your .config?
> Feb 22 15:00:54 mserv1 kernel: [<c011b150>] do_page_fault+0x0/0x523
> Feb 22 15:00:54 mserv1 kernel: [<c010baf5>] error_code+0x2d/0x38
> Feb 22 15:00:54 mserv1 kernel: [<c011e5b5>] __wake_up+0x45/0x70
> Feb 22 15:00:54 mserv1 kernel: [<c011e54a>] __wake_up_common+0x3a/0x60
> Feb 22 15:00:55 mserv1 kernel: [<c011e5af>] __wake_up+0x3f/0x70
> Feb 22 15:00:55 mserv1 kernel: [<c0259e62>] mrunlock+0x82/0xb0
> Feb 22 15:00:55 mserv1 kernel: [<c0259b00>] mraccessf+0xc0/0xe0
> Feb 22 15:00:55 mserv1 kernel: [<c023038e>] xfs_iunlock+0x3e/0x80
> Feb 22 15:00:55 mserv1 kernel: [<c023727b>] xfs_iomap+0x3bb/0x540
> Feb 22 15:00:55 mserv1 kernel: [<c0163fc7>] bio_alloc+0xd7/0x1c0
> Feb 22 15:00:55 mserv1 kernel: [<c025a17a>] map_blocks+0x7a/0x170
> Feb 22 15:00:55 mserv1 kernel: [<c025b40b>] page_state_convert+0x52b/0x6d0
> Feb 22 15:00:55 mserv1 kernel: [<c0236cb9>] xfs_imap_to_bmap+0x39/0x240
> Feb 22 15:00:55 mserv1 kernel: [<c025be48>] linvfs_release_page+0xa8/0xb0
> Feb 22 15:00:55 mserv1 kernel: [<c025bce0>] linvfs_writepage+0x60/0x120
> Feb 22 15:00:55 mserv1 kernel: [<c014990c>] shrink_list+0x41c/0x710
> Feb 22 15:00:55 mserv1 kernel: [<c0149df8>] shrink_cache+0x1f8/0x3d0
> Feb 22 15:00:55 mserv1 kernel: [<c01b3a00>] journal_stop+0x220/0x330
> Feb 22 15:00:55 mserv1 kernel: [<c014a6dc>] shrink_zone+0xbc/0xc0
> Feb 22 15:00:55 mserv1 kernel: [<c014a7a5>] shrink_caches+0xc5/0xe0
> Feb 22 15:00:55 mserv1 kernel: [<c014a87c>] try_to_free_pages+0xbc/0x190
> Feb 22 15:00:55 mserv1 kernel: [<c0143043>] __alloc_pages+0x203/0x370
> Feb 22 15:00:55 mserv1 kernel: [<c01431d5>] __get_free_pages+0x25/0x40
Hmm, from the trace it looks like ->release_page was called from a context
where we can't sleep. XFS defintily doesn't handle that, so the question
is whether the kernel should do it.
Mikael Wahlberg
> On Sun, Feb 22, 2004 at 04:49:41PM +0100, Mikael Wahlberg wrote:
> > Description:
> >
> > On heavy FTP Load (About 1Gbit/s) running both reads and writes on two ServeRAID6m Raid5 controllers merged together to one filesystem with Raidtools we see the error below. The filesystem gets totally hanged up. Currently with XFS, but JFS gets the same problem (Actually even more often).
>
> What does the JFS oops look like?
Unfortunately I cannot find the JFS oops right now. We can try to
reproduce it on one of the machines (We have 4 identical setups). It
seems like the JFS oops never was written in the messages-file..
We are running the system on 2 CPUs (No HyperThreading) right now, and
it has not crashed for about 18 hours.. so it is a bit more stable with
less CPU:s it seems. We will try JFS without HT today.
> > Feb 22 15:00:53 mserv1 kernel: [<c011e54a>] __wake_up_common+0x3a/0x60
> > Feb 22 15:00:53 mserv1 kernel: [<c011e5af>] __wake_up+0x3f/0x70
>
> This doesn't make a lot of sense, there's only two mrlocks in XFS, and
> they're in the inodes that have well-defined and understood lifetime rules.
>
> OTOH the previos oops might have messed quite a bit up in your system.
Ok..
> did you run memtest86 on the box? do you some strange patches applied or
> external modules loaded? What's your .config?
No strange patches. Pure 2.6.3 dist kernel. We haven't run memtest86,
but as I mentioned above, we have 4 equal machines with error correcting
memory, so I find it unlikely to be a memory problem.
The .config is attached.
Ok..
If you need any more information please tell us, it is quite urgent for
us, since we really don't want to go back to 2.4, the performance
increase with 2.6 is really impressive (Except when it crashes :)
Christoph Hellwig
> If you need any more information please tell us, it is quite urgent for
> us, since we really don't want to go back to 2.4, the performance
> increase with 2.6 is really impressive (Except when it crashes :)
Can you check whether the small patch below gets rid of you problems?
It still wouldn't explain the JFS problems, though.
--- 1.59/fs/xfs/linux/xfs_aops.c Mon Feb 9 05:39:27 2004
+++ edited/fs/xfs/linux/xfs_aops.c Mon Feb 23 15:11:33 2004
@@ -1170,7 +1170,7 @@
if (!delalloc && !unwritten)
goto free_buffers;
- if (!(gfp_mask & __GFP_FS))
+ if ((gfp_mask & (__GFP_FS|__GFP_WAIT)) != (__GFP_FS|__GFP_WAIT))
return 0;
/* If we are already inside a transaction or the thread cannot
Seth Mos
>On Mon, 2004-02-23 at 13:19, Christoph Hellwig wrote:
> > On Sun, Feb 22, 2004 at 04:49:41PM +0100, Mikael Wahlberg wrote:
> > did you run memtest86 on the box? do you some strange patches applied or
> > external modules loaded? What's your .config?
>
>No strange patches. Pure 2.6.3 dist kernel. We haven't run memtest86,
>but as I mentioned above, we have 4 equal machines with error correcting
>memory, so I find it unlikely to be a memory problem.
The memory might be fine, but the mainboard might still be borked. I have
seen this once with Dell kit. Asuming can be dangerous. If you can spare
the down time it is always a good idea to make sure.
In my experience XFS shows faulty memory quite fast. That's from personal
experience.
Cheers
--
Seth
I don't make sense, I don't pretend to either. Questions?
Mikael Wahlberg
> > Description:
> >
> > On heavy FTP Load (About 1Gbit/s) running both reads and writes on two ServeRAID6m Raid5 controllers merged together to one filesystem with Raidtools we see the error below. The filesystem gets totally hanged up. Currently with XFS, but JFS gets the same problem (Actually even more often).
>
> What does the JFS oops look like?
Just tried it without HyperThreading, with JFS, the filesystem hanged
without any kernel oops after about 10 minutes of 1Gbit/s FTP input.
What I got was:
Feb 23 14:19:15 mserv4 kernel:
Feb 23 14:19:15 mserv4 kernel: swapper: page allocation failure.
order:0, mode:0x20
Feb 23 14:19:15 mserv4 kernel: Call Trace:
Feb 23 14:19:15 mserv4 kernel: [<c0143160>] __alloc_pages+0x320/0x370
Feb 23 14:19:15 mserv4 kernel: [<c01431d5>] __get_free_pages+0x25/0x40
Feb 23 14:19:15 mserv4 kernel: [<c01462ac>] cache_grow+0xcc/0x330
Feb 23 14:19:15 mserv4 kernel: [<c014660e>]
cache_alloc_refill+0xfe/0x2a0
Feb 23 14:19:15 mserv4 kernel: [<c0146b4e>] __kmalloc+0x7e/0x90
Feb 23 14:19:15 mserv4 kernel: [<c032e797>] alloc_skb+0x47/0xf0
Feb 23 14:19:15 mserv4 kernel: [<c02ad599>]
e1000_alloc_rx_buffers+0x59/0x100
Feb 23 14:19:15 mserv4 kernel: [<c02ad18e>]
e1000_clean_rx_irq+0xfe/0x4b0
Feb 23 14:19:15 mserv4 kernel: [<c02acfe6>]
e1000_clean_tx_irq+0x1d6/0x280
Feb 23 14:19:15 mserv4 kernel: [<c02acd9a>] e1000_intr+0x4a/0xc0
Feb 23 14:19:16 mserv4 kernel: [<c010d579>] handle_IRQ_event+0x49/0x80
Feb 23 14:19:16 mserv4 kernel: [<c010d952>] do_IRQ+0xd2/0x1d0
Feb 23 14:19:16 mserv4 kernel: [<c010b9f8>] common_interrupt+0x18/0x20
Feb 23 14:19:16 mserv4 kernel: [<c0146a51>] kmem_cache_alloc+0x31/0x50
Feb 23 14:19:16 mserv4 kernel: [<c032e775>] alloc_skb+0x25/0xf0
Feb 23 14:19:16 mserv4 kernel: [<c0368ee1>] tcp_send_ack+0x31/0xd0
Feb 23 14:19:16 mserv4 kernel: [<c0333ebd>]
netif_receive_skb+0x1cd/0x250
Feb 23 14:19:16 mserv4 kernel: [<c0369d9f>]
tcp_delack_timer+0x10f/0x220
Feb 23 14:19:16 mserv4 kernel: [<c02acfe6>]
e1000_clean_tx_irq+0x1d6/0x280
Feb 23 14:19:16 mserv4 kernel: [<c0369c90>] tcp_delack_timer+0x0/0x220
Feb 23 14:19:16 mserv4 kernel: [<c012c5d8>]
run_timer_softirq+0xf8/0x1e0
Feb 23 14:19:16 mserv4 kernel: [<c0278db1>]
add_interrupt_randomness+0x31/0x40
Feb 23 14:19:16 mserv4 kernel: [<c012793a>] do_softirq+0xca/0xd0
Feb 23 14:19:16 mserv4 kernel: [<c010d9dd>] do_IRQ+0x15d/0x1d0
Feb 23 14:19:16 mserv4 kernel: [<c0108a60>] default_idle+0x0/0x40
Feb 23 14:19:16 mserv4 kernel: [<c010b9f8>] common_interrupt+0x18/0x20
Feb 23 14:19:16 mserv4 kernel: [<c0108a60>] default_idle+0x0/0x40
Feb 23 14:19:17 mserv4 kernel: [<c0108a8d>] default_idle+0x2d/0x40
Feb 23 14:19:18 mserv4 kernel: [<c0108b26>] cpu_idle+0x46/0x50
Feb 23 14:19:18 mserv4 kernel: [<c0105000>] rest_init+0x0/0x70
Feb 23 14:19:18 mserv4 kernel: [<c0456930>] start_kernel+0x1a0/0x1e0
Feb 23 14:19:18 mserv4 kernel: [<c04564a0>]
unknown_bootoption+0x0/0x120
Feb 23 14:19:18 mserv4 kernel:
Feb 23 14:19:18 mserv4 kernel: swapper: page allocation failure.
order:0, mode:0x20
Feb 23 14:19:18 mserv4 kernel: Call Trace:
Feb 23 14:19:18 mserv4 kernel: [<c0143160>] __alloc_pages+0x320/0x370
Feb 23 14:19:18 mserv4 kernel: [<c03510c0>]
ip_local_deliver_finish+0x0/0x220
Feb 23 14:19:18 mserv4 kernel: [<c01431d5>] __get_free_pages+0x25/0x40
Feb 23 14:19:18 mserv4 kernel: [<c01462ac>] cache_grow+0xcc/0x330
Feb 23 14:19:18 mserv4 kernel: [<c014660e>]
cache_alloc_refill+0xfe/0x2a0
Feb 23 14:19:18 mserv4 kernel: [<c0146b4e>] __kmalloc+0x7e/0x90
Feb 23 14:19:19 mserv4 kernel: [<c032e797>] alloc_skb+0x47/0xf0
Feb 23 14:19:19 mserv4 kernel: [<c02ad599>]
e1000_alloc_rx_buffers+0x59/0x100
Feb 23 14:19:19 mserv4 kernel: [<c02ad18e>]
e1000_clean_rx_irq+0xfe/0x4b0
Feb 23 14:19:19 mserv4 kernel: [<c0333ebd>]
netif_receive_skb+0x1cd/0x250
Feb 23 14:19:19 mserv4 kernel: [<c02acd9a>] e1000_intr+0x4a/0xc0
Feb 23 14:19:19 mserv4 kernel: [<c010d579>] handle_IRQ_event+0x49/0x80
Feb 23 14:19:19 mserv4 kernel: [<c010d952>] do_IRQ+0xd2/0x1d0
Feb 23 14:19:19 mserv4 kernel: [<c0108a60>] default_idle+0x0/0x40
Feb 23 14:19:19 mserv4 kernel: [<c010b9f8>] common_interrupt+0x18/0x20
Feb 23 14:19:19 mserv4 kernel: [<c0108a60>] default_idle+0x0/0x40
Feb 23 14:19:19 mserv4 kernel: [<c0108a8d>] default_idle+0x2d/0x40
Feb 23 14:19:19 mserv4 kernel: [<c0108b26>] cpu_idle+0x46/0x50
Feb 23 14:19:19 mserv4 kernel: [<c0105000>] rest_init+0x0/0x70
Feb 23 14:19:19 mserv4 kernel: [<c0456930>] start_kernel+0x1a0/0x1e0
Feb 23 14:19:20 mserv4 kernel: [<c04564a0>]
unknown_bootoption+0x0/0x120
Feb 23 14:19:20 mserv4 kernel:
^[
Feb 23 14:31:57 mserv4 proftpd[22053]: mserv4 - ProFTPD killed (signal
15)
(I restardet proftpd when nothing happened)
Then if I try to read a file on the filesystem:
(Strace of less)
[root@mserv4 clips]# strace less f5.in.09
execve("/usr/bin/less", ["less", "f5.in.09"], [/* 26 vars */]) = 0
uname({sys="Linux", node="mserv4", ...}) = 0
brk(0) = 0x806b000
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS,
-1, 0) = 0x40013000
open("/etc/ld.so.preload", O_RDONLY) = -1 ENOENT (No such file or
directory)
open("/etc/ld.so.cache", O_RDONLY) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=62008, ...}) = 0
old_mmap(NULL, 62008, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40014000
close(3) = 0
open("/usr/lib/libncurses.so.5", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\300\341"...,
512) = 512fstat64(3, {st_mode=S_IFREG|0755, st_size=255300, ...}) = 0
old_mmap(NULL, 256076, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) =
0x40024000
old_mmap(0x4005a000, 36864, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED,
3, 0x36000) = 0x4005a000
close(3) = 0
open("/lib/libc.so.6", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\310V\1"...,
512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=1476244, ...}) = 0
old_mmap(NULL, 1207972, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) =
0x40063000
old_mmap(0x40184000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED,
3, 0x120000) = 0x40184000
old_mmap(0x40188000, 7844, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x40188000
close(3) = 0
munmap(0x40014000, 62008) = 0
ioctl(1, SNDCTL_TMR_TIMEBASE, {B38400 opost isig icanon echo ...}) = 0
brk(0) = 0x806b000
brk(0x806c000) = 0x806c000
brk(0) = 0x806c000
access("/root/.terminfo/x/xterm", R_OK) = -1 ENOENT (No such file or
directory)
access("/usr/share/terminfo/x/xterm", R_OK) = 0
open("/usr/share/terminfo/x/xterm", O_RDONLY) = 3
read(3, "\32\1\34\0\35\0\17\0i\1\273\3", 12) = 12
read(3, "xterm|X11 terminal emulator\0", 28) = 28
read(3, "\0\1\0\0\1\0\0\0\1\0\0\0\0\1\1\0\0\0\0\0\0\0\1\0\0\0\0"..., 29)
= 29
read(3, "\0", 1) = 1
read(3, "P\0\10\0\30\0\377\377\377\377\377\377\377\377\377\377\377"...,
30) = 30read(3,
"\0\0\4\0\6\0\10\0\31\0\36\0&\0*\0.\0\377\3779\0J\0L\0P"..., 722) = 722
read(3, "\33[Z\0\7\0\r\0\33[%i%p1%d;%p2%dr\0\33[3g\0\33["..., 955) = 955
read(3, "", 1) = 0
read(3, "", 10) = 0
close(3) = 0
ioctl(1, SNDCTL_TMR_TIMEBASE, {B38400 opost isig icanon echo ...}) = 0
ioctl(1, SNDCTL_TMR_TIMEBASE, {B38400 opost isig icanon echo ...}) = 0
ioctl(1, SNDCTL_TMR_TIMEBASE, {B38400 opost isig icanon echo ...}) = 0
ioctl(1, SNDCTL_TMR_TIMEBASE, {B38400 opost isig icanon echo ...}) = 0
ioctl(1, TIOCGWINSZ, {ws_row=61, ws_col=80, ws_xpixel=0, ws_ypixel=0}) =
0
ioctl(2, TIOCGWINSZ, {ws_row=61, ws_col=80, ws_xpixel=0, ws_ypixel=0}) =
0
open("/usr/bin/.sysless", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such
file or directory)
open("/etc/sysless", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or
directory)
open("/root/.less", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or
directory)
brk(0) = 0x806c000
brk(0x806d000) = 0x806d000
open("/usr/lib/locale/locale-archive", O_RDONLY|O_LARGEFILE) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=31202800, ...}) = 0
mmap2(NULL, 2097152, PROT_READ, MAP_PRIVATE, 3, 0) = 0x4018a000
close(3) = 0
brk(0) = 0x806d000
brk(0x806e000) = 0x806e000
open("/dev/tty", O_RDONLY|O_LARGEFILE) = 3
ioctl(3, SNDCTL_TMR_TIMEBASE, {B38400 opost isig icanon echo ...}) = 0
fsync(3) = -1 EINVAL (Invalid argument)
ioctl(3, SNDCTL_TMR_STOP, {B38400 opost isig -icanon -echo ...}) = 0
rt_sigaction(SIGINT, {0x8059e90, [INT], SA_RESTORER|SA_RESTART,
0x40089cf8}, {SIG_DFL}, 8) = 0
rt_sigaction(SIGTSTP, {0x8059ed0, [TSTP], SA_RESTORER|SA_RESTART,
0x40089cf8}, {SIG_DFL}, 8) = 0
rt_sigaction(SIGWINCH, {0x8059f10, [WINCH], SA_RESTORER|SA_RESTART,
0x40089cf8}, {SIG_DFL}, 8) = 0
pipe([4, 5]) = 0
vfork() = 22348
close(5) = 0
read(4, "", 1) = 0
--- SIGCHLD (Child exited) @ 0 (0) ---
close(4) = 0
wait4(22348, [WIFEXITED(s) && WEXITSTATUS(s) == 127], 0, NULL) = 22348
stat64("f5.in.09", {st_mode=S_IFREG|0644, st_size=67719168, ...}) = 0
stat64("f5.in.09", {st_mode=S_IFREG|0644, st_size=67719168, ...}) = 0
open("f5.in.09", O_RDONLY|O_LARGEFILE) = 4
_llseek(4, 1,
then hang..
Anybody got an idea?
/Mikael
--
-----------------------------------------------------------------------
Mikael Wahlberg, M.Sc. Ardendo
Unit Manager Professional Services/ e-mail: mik...@ardendo.se
Technical Project Manager GSM: +46 733 279 274
-
Mikael Wahlberg
> At 14:08 23-2-2004 +0100, Mikael Wahlberg wrote:
> >On Mon, 2004-02-23 at 13:19, Christoph Hellwig wrote:
> > > On Sun, Feb 22, 2004 at 04:49:41PM +0100, Mikael Wahlberg wrote:
> > > did you run memtest86 on the box? do you some strange patches applied or
> > > external modules loaded? What's your .config?
> >
> >No strange patches. Pure 2.6.3 dist kernel. We haven't run memtest86,
> >but as I mentioned above, we have 4 equal machines with error correcting
> >memory, so I find it unlikely to be a memory problem.
>
> The memory might be fine, but the mainboard might still be borked. I have
> seen this once with Dell kit. Asuming can be dangerous. If you can spare
> the down time it is always a good idea to make sure.
Yes, but four new identical boxes... seems very unlikely. But sure, we
can try it when I get to the computer room (It is not at our site, but
at a customers)
/Mikael
--
-----------------------------------------------------------------------
Mikael Wahlberg, M.Sc. Ardendo
Unit Manager Professional Services/ e-mail: mik...@ardendo.se
Technical Project Manager GSM: +46 733 279 274
-
Mikael Wahlberg
> On Mon, Feb 23, 2004 at 02:08:09PM +0100, Mikael Wahlberg wrote:
> > If you need any more information please tell us, it is quite urgent for
> > us, since we really don't want to go back to 2.4, the performance
> > increase with 2.6 is really impressive (Except when it crashes :)
>
> Can you check whether the small patch below gets rid of you problems?
> It still wouldn't explain the JFS problems, though.
>
> --- 1.59/fs/xfs/linux/xfs_aops.c Mon Feb 9 05:39:27 2004
> +++ edited/fs/xfs/linux/xfs_aops.c Mon Feb 23 15:11:33 2004
> @@ -1170,7 +1170,7 @@
> if (!delalloc && !unwritten)
> goto free_buffers;
>
> - if (!(gfp_mask & __GFP_FS))
> + if ((gfp_mask & (__GFP_FS|__GFP_WAIT)) != (__GFP_FS|__GFP_WAIT))
> return 0;
>
> /* If we are already inside a transaction or the thread cannot
Unfortunately yesterday evening during peak load we got another
filesystem hang, the oops looks like below:
We feel a bit sad to have do downgrade to a 2.4 kernel, with the
performance loss that would imply. So any ideas would be very helpful.
Mar 3 18:27:35 c-serv kernel: Unable to handle kernel NULL pointer
dereference at virtual address 00000000
Mar 3 18:27:35 c-serv kernel: printing eip:
Mar 3 18:27:36 c-serv kernel: c025c9a2
Mar 3 18:27:36 c-serv kernel: *pde = 00000000
Mar 3 18:27:36 c-serv kernel: Oops: 0000 [#1]
Mar 3 18:27:36 c-serv kernel: CPU: 0
Mar 3 18:27:36 c-serv kernel: EIP: 0060:[<c025c9a2>] Not tainted
Mar 3 18:27:36 c-serv kernel: EFLAGS: 00010206
Mar 3 18:27:36 c-serv kernel: EIP is at _pagebuf_find+0xd2/0x2c0
Mar 3 18:27:37 c-serv kernel: eax: 7930e000 ebx: 00000000 ecx:
7930e187 edx: 0003f000
Mar 3 18:27:37 c-serv kernel: esi: 00000000 edi: f63a5260 ebp:
c04cd068 esp: c6897b94
Mar 3 18:27:37 c-serv kernel: ds: 007b es: 007b ss: 0068
Mar 3 18:27:37 c-serv kernel: Process proftpd (pid: 15349,
threadinfo=c6896000 task=d479b900)
Mar 3 18:27:37 c-serv kernel: Stack: f7fe2740 0003f000 00000008
00000000 c02098c8 f638ae50 0000008e 0003f000
Mar 3 18:27:37 c-serv kernel: 00000008 c0e84980 0080003f
040001f8 0000a005 c025cc75 f7942fc0 040001f8
Mar 3 18:27:37 c-serv kernel: 00000000 00001000 0000a005
c0e84980 040001f8 00000000 0000a005 0080003f
Mar 3 18:27:37 c-serv kernel: Call Trace:
Mar 3 18:27:37 c-serv kernel: [<c02098c8>] xfs_bmapi_single+0xf8/0x1c0
Mar 3 18:27:37 c-serv kernel: [<c025cc75>] pagebuf_get+0xa5/0x180
Mar 3 18:27:37 c-serv kernel: [<c024d1df>]
xfs_trans_read_buf+0x32f/0x390
Mar 3 18:27:37 c-serv kernel: [<c021670f>] xfs_da_do_buf+0x7ef/0x9e0
Mar 3 18:27:37 c-serv kernel: [<c033f5a7>] nf_hook_slow+0xf7/0x160
Mar 3 18:27:37 c-serv kernel: [<c02169b7>] xfs_da_read_buf+0x57/0x60
Mar 3 18:27:37 c-serv kernel: [<c021451c>]
xfs_da_node_lookup_int+0x8c/0x390
Mar 3 18:27:37 c-serv kernel: [<c021451c>]
xfs_da_node_lookup_int+0x8c/0x390
Mar 3 18:27:37 c-serv kernel: [<c0221b2f>]
xfs_dir2_node_lookup+0x3f/0xc0
Mar 3 18:27:37 c-serv kernel: [<c0218ebf>] xfs_dir2_lookup+0x13f/0x150
Mar 3 18:27:37 c-serv kernel: [<c01a3d10>]
ext3_journal_dirty_data+0x0/0x70
Mar 3 18:27:37 c-serv kernel: [<c033041c>] memcpy_toiovec+0x5c/0x90
Mar 3 18:27:37 c-serv kernel: [<c024e44c>]
xfs_dir_lookup_int+0x4c/0x130
Mar 3 18:27:37 c-serv kernel: [<c02541d5>] xfs_lookup+0x65/0x90
Mar 3 18:27:37 c-serv kernel: [<c0261c27>] linvfs_lookup+0x67/0xa0
Mar 3 18:27:38 c-serv kernel: [<c016d65f>] real_lookup+0xcf/0x100
Mar 3 18:27:38 c-serv kernel: [<c016d956>] do_lookup+0xa6/0xc0
Mar 3 18:27:38 c-serv kernel: [<c016dee3>] link_path_walk+0x573/0xad0
Mar 3 18:27:38 c-serv kernel: [<c0174beb>] locks_delete_lock+0x8b/0xf0
Mar 3 18:27:38 c-serv kernel: [<c01756a5>]
__posix_lock_file+0x435/0x660
Mar 3 18:27:38 c-serv kernel: [<c016ea09>] __user_walk+0x49/0x60
Mar 3 18:27:38 c-serv kernel: [<c0168dbc>] vfs_lstat+0x1c/0x60
Mar 3 18:27:38 c-serv kernel: [<c0176d7d>] fcntl_setlk64+0x11d/0x2c0
Mar 3 18:27:38 c-serv kernel: [<c012be68>] del_timer_sync+0x38/0x140
Mar 3 18:27:38 c-serv kernel: [<c016953b>] sys_lstat64+0x1b/0x40
Mar 3 18:27:38 c-serv kernel: [<c01720e7>] sys_fcntl64+0x57/0xa0
Mar 3 18:27:38 c-serv kernel: [<c010b08b>] syscall_call+0x7/0xb
Mar 3 18:27:38 c-serv kernel:
Mar 3 18:27:38 c-serv kernel: Code: 8b 1b 0f 18 03 90 39 ee 75 e4 8b 5c
24 4c 85 db 0f 84 85 00
Mar 3 18:27:38 c-serv kernel: <6>note: proftpd[15349] exited with
preempt_count 1
Mar 3 18:27:38 c-serv kernel: bad: scheduling while atomic!
Mar 3 18:27:38 c-serv kernel: Call Trace:
Mar 3 18:27:38 c-serv kernel: [<c011e48e>] schedule+0x6ee/0x700
Mar 3 18:27:38 c-serv kernel: [<c014cdeb>] zap_pmd_range+0x4b/0x70
Mar 3 18:27:38 c-serv kernel: [<c01591ba>]
free_pages_and_swap_cache+0x6a/0xa0
Mar 3 18:27:38 c-serv kernel: [<c014d0cc>] unmap_vmas+0x23c/0x2f0
Mar 3 18:27:38 c-serv kernel: [<c0151774>] exit_mmap+0xf4/0x250
Mar 3 18:27:39 c-serv kernel: [<c0120d2d>] mmput+0x6d/0xa0
Mar 3 18:27:39 c-serv kernel: [<c0125a33>] do_exit+0x1a3/0x500
Mar 3 18:27:39 c-serv kernel: [<c010c1ac>] die+0xfc/0x100
Mar 3 18:27:39 c-serv kernel: [<c011b349>] do_page_fault+0x1f9/0x523
Mar 3 18:27:39 c-serv kernel: [<c0333fe1>] process_backlog+0x81/0x120
Mar 3 18:27:39 c-serv kernel: [<c0334164>] net_rx_action+0xe4/0x130
Mar 3 18:27:39 c-serv kernel: [<c010d9dd>] do_IRQ+0x15d/0x1d0
Mar 3 18:27:39 c-serv kernel: [<c011b150>] do_page_fault+0x0/0x523
Mar 3 18:27:39 c-serv kernel: [<c010baf5>] error_code+0x2d/0x38
Mar 3 18:27:39 c-serv kernel: [<c025c9a2>] _pagebuf_find+0xd2/0x2c0
Mar 3 18:27:39 c-serv kernel: [<c02098c8>] xfs_bmapi_single+0xf8/0x1c0
Mar 3 18:27:39 c-serv kernel: [<c025cc75>] pagebuf_get+0xa5/0x180
Mar 3 18:27:39 c-serv kernel: [<c024d1df>]
xfs_trans_read_buf+0x32f/0x390
Mar 3 18:27:39 c-serv kernel: [<c021670f>] xfs_da_do_buf+0x7ef/0x9e0
Mar 3 18:27:39 c-serv kernel: [<c033f5a7>] nf_hook_slow+0xf7/0x160
Mar 3 18:27:39 c-serv kernel: [<c02169b7>] xfs_da_read_buf+0x57/0x60
Mar 3 18:27:39 c-serv kernel: [<c021451c>]
xfs_da_node_lookup_int+0x8c/0x390
Mar 3 18:27:39 c-serv kernel: [<c021451c>]
xfs_da_node_lookup_int+0x8c/0x390
Mar 3 18:27:40 c-serv kernel: [<c0221b2f>]
xfs_dir2_node_lookup+0x3f/0xc0
Mar 3 18:27:40 c-serv kernel: [<c0218ebf>] xfs_dir2_lookup+0x13f/0x150
Mar 3 18:27:40 c-serv kernel: [<c01a3d10>]
ext3_journal_dirty_data+0x0/0x70
Mar 3 18:27:40 c-serv kernel: [<c033041c>] memcpy_toiovec+0x5c/0x90
Mar 3 18:27:40 c-serv kernel: [<c024e44c>]
xfs_dir_lookup_int+0x4c/0x130
Mar 3 18:27:40 c-serv kernel: [<c02541d5>] xfs_lookup+0x65/0x90
Mar 3 18:27:40 c-serv kernel: [<c0261c27>] linvfs_lookup+0x67/0xa0
Mar 3 18:27:40 c-serv kernel: [<c016d65f>] real_lookup+0xcf/0x100
Mar 3 18:27:40 c-serv kernel: [<c016d956>] do_lookup+0xa6/0xc0
Mar 3 18:27:40 c-serv kernel: [<c016dee3>] link_path_walk+0x573/0xad0
Mar 3 18:27:40 c-serv kernel: [<c0174beb>] locks_delete_lock+0x8b/0xf0
Mar 3 18:27:40 c-serv kernel: [<c01756a5>]
__posix_lock_file+0x435/0x660
Mar 3 18:27:40 c-serv kernel: [<c016ea09>] __user_walk+0x49/0x60
Mar 3 18:27:40 c-serv kernel: [<c0168dbc>] vfs_lstat+0x1c/0x60
Mar 3 18:27:40 c-serv kernel: [<c0176d7d>] fcntl_setlk64+0x11d/0x2c0
Mar 3 18:27:40 c-serv kernel: [<c012be68>] del_timer_sync+0x38/0x140
Mar 3 18:27:40 c-serv kernel: [<c016953b>] sys_lstat64+0x1b/0x40
Mar 3 18:27:40 c-serv kernel: [<c01720e7>] sys_fcntl64+0x57/0xa0
Mar 3 18:27:40 c-serv kernel: [<c010b08b>] syscall_call+0x7/0xb
Mar 3 18:27:40 c-serv kernel:
/Mikael
--
-----------------------------------------------------------------------
Mikael Wahlberg, M.Sc. Ardendo
Unit Manager Professional Services/ e-mail: mik...@ardendo.se
Technical Project Manager GSM: +46 733 279 274
-
Brad Allen
> hanged without any kernel oops after about 10 minutes of 1Gbit/s FTP
> input.
>
> mserv4 kernel: swapper: page allocation failure. order:0, mode:0x20 [...]
> mserv4 kernel: [<c02ad599>] e1000_alloc_rx_buffers+0x59/0x100
I just corresponded a problem I originally thought yours might be
related to, so now I think it's even less likely they're related.
After a due-diligence check for latest version before post, I found
and upgraded to kernel version 2.6.6-rc3, and during my tests of that
new version inadvertently finally found the test that revealed a
correspondence with these errors: running CBQ network bandwidth
limiting, queuing and prioritization, *and* high network use (using
that bandwidth limiting). The CBQ stuff is done at boot. As soon as
I started the high network use, the problem started (see below
"full_kernel_output" file for an idea of what kernel messages I was
seeing en masse), and then during this high network use, I turned off
CBQ and everything was normal again.
This is what I wrote before the above discovery, which gives
background:
I'm also seeing "page allocation failure" errors in a 2.6 series
kernel with e1000 driver with lots of filesystem access and lots of
outbound network, which I think are our common themes. I'm seeing the
call trace with e1000 driver routines in nearly every list.
Otherwise, much else is different:
* Linux kernel 2.6.6-rc2 used for this particular run
as documented, but same allocation failure errors
also seen in 2.6.6-rc3, 2.6.6-rc1, 2.6.5,
2.6.5+kraxel263-1, 2.6.4+kraxel263-1 (+kraxel is
drivers/media/video stuff ported in sooner than 2.6
series got it).
* I was doing relatively heavy client NFSv3, not XFS
or JFS like you: ~2MiB/s steady write, ~2MiB/s
steady read. (Test ran for ~2 hours. Crashed
with only 37s to go before filling up remote disk!
Coincidence, because it had no idea the remote disk
was getting full, I think ... but never did finish
filling it, which was my planned test end point.)
* I am using experimental IVTV driver, which is
itself having trouble and undergoing active
development, to record and play TV using Hauppauge
990 (WinTV PVR 350). This card does all the heavy
compute intensive stuff: MPEG2 encode & decode, as
well as TV stuff. The computer merely has to
bitshuffle & stuff, which is *not* that hard. The
bandwidth of the system ought to allow more like four
of these cards recording & at least one playing, if
things were tuned right, I think.
* My IRQ is shared with that above TV card, e1000
(Intel GbE) card, and video output card; the first
two get used extensively (the last one -- the local
console video -- almost never gets used, except for
printk).
* I had network output going almost full-bore, as
a stress test of the above, but I still get the
errors when this isn't the case. 61,840B/s inside
TCP/IP.
* I am using CBQ to limit network output speed to
Intenret, and priority of packets. 62,464B/s including
IP packets (headers, etc.). Link measured max at a
rough ~66,560B/s inside TCP/IP, so I stay away from it.
So, this is basically definitely hitting output
speed buffering on occasion.
* It's bridging between two ethernet cards (sender
of through-IP traffic (~61,840B/s) in from the
e1000 GbE card, being sent out an on-board 100MbE).
NFSv3 (from & to server) isn't bridged, but also
is going via that e1000 GbE card.
* The computer I'm seeing this on is relatively old
and moderately low resource for today's memory &
CPU use standards:
1999 Dell Optiplex GX1 PIII 450MHz 128MiB ram
I'm not sure if this is related to your problem. I searched
linux-kernel by using the two quoted phrases "page allocation failure"
"e1000_alloc_rx_buffers" in Google, and your above message is the only
one that came out.
I am worried that perhaps page allocation failures are very vague
diagnostic tools. I take it this simply means I ran out of (perhaps a
certain kind of, e.g. and what I suspect, kernel) memory.
Here is the kernel message output (one sample excerpt below):
ftp://ftp.sonic.net/pub/users/ulmo/ivtv/full_kernel_output.bz2
and .config options:
ftp://ftp.sonic.net/pub/users/ulmo/SW/OS/linux/2.6.6-rc2.config.bz2
Here's my CBQ script:
ftp://ftp.sonic.net/pub/users/ulmo/SW/OS/linux/tc.sh
Note that I *do* use LVM (everyone I talk to seems to -- although I
think it's called DM now; btw, I do use and like EVMS on more modern
hosts -- and I'm sure I'd like it on old ones, too, but just haven't
bothered to do it there), but since not much of that computer's access
is via local disk, I doubt that matters that much.
My MTU for GbE (e1000) is 9000, and NFS block size 8192 bytes.
That GbE is a consumer grade Intel model.
Here is a sample of the kernel output; note please that these errors
are not all exactly identical, so if you are reading them please check
the above full_kernel_output for any nuances and big differences (all
kernel messages for about two hours from boot to crash).
swapper: page allocation failure. order:3, mode:0x20
Call Trace:
[__alloc_pages+696/784] __alloc_pages+0x2b8/0x310
[__get_free_pages+34/80] __get_free_pages+0x22/0x50
[cache_grow+165/656] cache_grow+0xa5/0x290
[cache_alloc_refill+354/512] cache_alloc_refill+0x162/0x200
[do_timer+224/240] do_timer+0xe0/0xf0
[__kmalloc+140/176] __kmalloc+0x8c/0xb0
[alloc_skb+72/240] alloc_skb+0x48/0xf0
[e1000_alloc_rx_buffers+102/272] e1000_alloc_rx_buffers+0x66/0x110
[e1000_clean_rx_irq+225/1056] e1000_clean_rx_irq+0xe1/0x420
[scheduler_tick+31/1312] scheduler_tick+0x1f/0x520
[e1000_intr+58/144] e1000_intr+0x3a/0x90
[handle_IRQ_event+59/112] handle_IRQ_event+0x3b/0x70
[do_IRQ+150/336] do_IRQ+0x96/0x150
[common_interrupt+24/32] common_interrupt+0x18/0x20
[default_idle+38/64] default_idle+0x26/0x40
[cpu_idle+52/64] cpu_idle+0x34/0x40
[start_kernel+415/480] start_kernel+0x19f/0x1e0
[unknown_bootoption+0/288] unknown_bootoption+0x0/0x120
Andrew Morton
>
> My MTU for GbE (e1000) is 9000, and NFS block size 8192 bytes.
> That GbE is a consumer grade Intel model.
> swapper: page allocation failure. order:3, mode:0x20
The kernel simply doesn't have a hope of being able to find eight
physically-contiguous free pages at interrupt time.
You'll get some benefit from increasing /proc/sys/vm/min_free_kbytes
But it's beginning to look like we need separately-managed higher-order
page pools for this. At least.