In article <3itc77$
...@ninurta.fer.uni-lj.si>,
Damjan Lango <lan
...@ana.fer.uni-lj.si> wrote:
>Absolutely agree!
>And I can't understand how this malloc bug came so far up to 1.1.x
>It *must* be fixed before 1.2!!!
Too late...
Anyway, it's not a simple matter of just checking how much free memory
there is: people seem to be completely unaware of how hard a problem
this actually is.
Please, read any book about resource dead-locks etc, and you'll find
that these dead-locks *can* be resolved, but at the cost of
- efficiency (to be sure you can handle any dead-lock, you'll have to
do a *lot* of work).
- usability (to make sure you never get any dead-lock, you have to say
no to somebody, and you'll have to say "no" a *lot* earlier than most
people seem to think).
In the case of the memory handling, actually counting pages isn't that
much of an overhead (we just have one reasource, and one page is as good
as any and they don't much depend on each other, so the setup is
reasonably simple), but the usability factor is major.
As it stands, you can add these lines to your /etc/profile:
ulimit -d 8192
ulimit -s 2048
and it will limit your processes to 8MB of data space, and 2MB of stack.
And no, it doesn't guarantee anything at all, but hey, your malloc()
will return NULL.
Personally, I consider the linux mm handling a feature, not a bug (hmm..
I wrote it, so that's not too surprising). People can whine all they
want, but please at least send out patches to fix it at the same time.
You'll find that some people definitely do *not* want to use your
patches.
Handling malloc() together with fork() makes for problems, adding the
problem of the stack space makes it worse, and adding the complexity of
private file mappings doesn't help. Before complaining, *please* think
about at least the following example scenarios (and you're allowed to
think up more of your own):
1) a database process maps in a database privately into memory. The
database is 32MB in size, but you only have 16MB free memory/swap.
Do you accept the mmap()?
- The database program probably doesn't re-write the database in memory:
it may change a few records in-core, but the number of pages it needs
might be less than 10 (the pages it doesn't modify don't count as
usage, as we can always throw them out when we want the memory back).
- on the other hand, how does the kernel *know*? It might be a program
that just mmap()'s something and then starts writing to all the
pages.
2) GNU emacs (ugh) wants to start up a shell script. In the meantime,
GNU emacs has (as it's wont to do) grown to 17 MB, and you obviously
don't have much memory left. Do you accept the fork?
- emacs will happily do an exec afterwards, and will actually use only
10 pages before that in the child process (stack, mainly). Sure, let
it fork().
- How is the kernel supposed to know that it will fork? No way can it
fork, as we don't have the potential 17MB of memory that now gets
doubled.
- vfork() isn't an option. Trust me on this one. vfork is *ugly*.
Besides, we might actually want to run the same process concurrently.
3) you have a nice quiescent little program that uses about 100kB of
memory, and has been a good little boy for the last 5 minutes. Now
it obviously wants to do something, so it forks 10 times. Do we
accept it?
- hell yes, we have 10MB free, and 10 forks of this program only uses
about a megabyte of that. Go ahead.
- hell, no: what if this nice little program just tried to make us feel
secure, and after the forks turns into the Program From Hell (tm)? It
might get into a recursive loop, and start eating up stack space.
Wheee.. Our 10MB of memory are gone in 5 seconds flat, and the OS is
left stranded wondering what the hell hit it.
4) You have a nice little 4MB machine, no swap, and you don't run X.
Most programs use shared libraries, and everybody is happy. You
don't use GNU emacs, you use "ed", and you have your own trusted
small-C compiler that works well. Does the system accept this?
- why, of course it does. It's a tight setup, but there's no panic.
- NO, DEFINITELY NOT. Each shared library in place actually takes up
600kB+ of virtual memory, and the system doesn't *know* that nothing
starts using these pages in all the processes alive. Now, with just
10 processes (a small make, and all the deamons), the kernel is
actually juggling more than 6MB of virtual memory in the shared
libraries alone, although only a fraction of that is actually in use
at that time.
It's easy to make malloc() return NULL under DOS: you just see if you
have any of the 640kB free, and if you have, it's ok.
It's easy to make malloc() return NULL under Windows: there is no fork()
system call, and nobody expects the machine to stay up anyway, so who
cares? When you say "I wrote a program that crashed Windows", people
just stare at you blankly and say "Hey, I got those with the system,
*for free*".
It's also easy to make malloc() return NULL under some trusted large
UNIX server: people running those are /expected/ to have an absolute
minimum of 256MB of RAM, and double that of swap, so we really can make
sure that any emacs that wants to fork() must have the memory available
(if you're so short of memory that 17MB is even close to tight, it's ok
to say that emacs can't fork).
It's *not* easy to say no to malloc() when you have 8-32MB of memory,
and about as much swap-space, and fork/mmap/etc works. You can do it,
sure, but you'd prefer a system that doesn't.
Linus
