Graceful failure instead of panicking in kmem_malloc

[Bharma Ji]

In FreeBSD 6_2, if kmem_malloc is unable to find space it panics. The
relevant code is in vm_kern.c
if ((flags & M_NOWAIT) == 0)
panic("kmem_malloc(%ld): kmem_map too small: %ld
total allocated",
(long)size, (long)map->size);

Is there any way to make the system log and then gracefully shut off instead
of panicking?
_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hacke...@freebsd.org"

[Kris Kennaway]

Bharma Ji wrote:
> In FreeBSD 6_2, if kmem_malloc is unable to find space it panics. The
> relevant code is in vm_kern.c
> if ((flags & M_NOWAIT) == 0)
> panic("kmem_malloc(%ld): kmem_map too small: %ld
> total allocated",
> (long)size, (long)map->size);
>
> Is there any way to make the system log and then gracefully shut off instead
> of panicking?

Not really, because those actions require memory allocation. The real
fix is to either

a) avoid running out of memory in the first place by tuning vm.kmem_size

b) perhaps trying harder to avoid panicking by first trying to more
aggressively reclaim memory.

You can try

http://www.freebsd.org/~pjd/patches/vm_kern.c.2.patch

which implements b) (patch against 7.0, but might apply to 6.2 unchanged).

Kris

[Bharma Ji]

Thanks for the response. I am hoping to keep some memory aside specifically
for handling out of memory allocation situations. Yes the real fix is to
avoid out of memory allocation. Thanks for the patch. Will try that. As a
first cut I am just trying to handle failure gracefully.

So asking again - if there is any way already discussed or standardized to
make the system handle failures gracefully

On Jan 8, 2008 4:30 PM, Kris Kennaway <kr...@freebsd.org> wrote:

> Bharma Ji wrote:
> > In FreeBSD 6_2, if kmem_malloc is unable to find space it panics. The
> > relevant code is in vm_kern.c
> > if ((flags & M_NOWAIT) == 0)
> > panic("kmem_malloc(%ld): kmem_map too small:
> %ld
> > total allocated",
> > (long)size, (long)map->size);
> >
> > Is there any way to make the system log and then gracefully shut off
> instead
> > of panicking?
>
> Not really, because those actions require memory allocation. The real
> fix is to either
>
> a) avoid running out of memory in the first place by tuning vm.kmem_size
>
> b) perhaps trying harder to avoid panicking by first trying to more
> aggressively reclaim memory.
>
> You can try
>

> http://www.freebsd.org/~pjd/patches/vm_kern.c.2.patch<http://www.freebsd.org/%7Epjd/patches/vm_kern.c.2.patch>

[Joshua Isom]

Why not try to take out some user processes? Going with a combination
of process priority and memory usage, it should at least be more
tolerable than a panic.

[Heiko Wundram (Beenic)]

Am Mittwoch, 9. Januar 2008 10:29:43 schrieb Joshua Isom:
> Why not try to take out some user processes? Going with a combination
> of process priority and memory usage, it should at least be more
> tolerable than a panic.

Ahemm. No. That's not tolerable in real world conditions. Have you ever had
the OOM-killer strike on Linux (which is known for this, and has been
criticized at other times for its braindead default behavior of overcommiting
virtual memory space almost two-fold)? That's a major, major PITA.

I'd rather have the system reboot and come back up to a clean and initialized
state than to "randomly" kill user processes and leave it crippled but
(somewhat) running (with sshd possibly killed off, which is especially bad on
remote boxes), as basically to recover cleanly from the OOM-killer striking,
you're going to have to reboot the box anyway.

--
Heiko Wundram
Product & Application Development

[Kris Kennaway]

This is kernel memory, not user memory. There is a fixed-size arena for
mallocs in the kernel, and the panic happens when it fills up and no
free space can be immediately reclaimed.

[Mike]

Bharma Ji wrote:
> Is there any way to make the system log and then gracefully shut off instead
> of panicking?

Is there any way to make the system log and then gracefully shut off while
guaranteeing that the logging/shutdown procedure won't also run out memory
somewhere?

[Bharma Ji]

>Is there any way to make the system log and then gracefully shut off while
>guaranteeing that the logging/shutdown procedure won't also run out memory
>somewhere?

For logging procedure, I am hoping that by keeping some memory aside, I
should be able to guarantee that the procedure will not run out of memory.
For shutdown, I don't know. I am not sure if doing the usual shutdown is
even required.
To me this appears to be a generic requirement of any system i.e.
a) Set aside some memory to handle out of memory conditions. Establish a low
threshold
b) When the system reaches the low threshold,
1) stop all processing (not sure right now if this translates to
shutdown)
2) record the error.

Kris
Thanks for sending the patch. The patch essentially will try to reclaim
memory 8 times before panicking. Is the understanding correct? If so, how
did you arrive at the number 8?