malloc under linux
jacob navia
I was working under linux today when I discovered this:
man malloc
<quote>
[snip pages]
BUGS
By default, Linux follows an optimistic memory allocation strategy.
This means that when malloc() returns non-NULL there is no guarantee
that the memory really is available. This is a really bad bug.
In case it turns out that the system is out of memory, one or more
processes will be killed by the infamous OOM killer. In case Linux is
employed under circumstances where it would be less desirable to
suddenly lose some randomly picked processes, and moreover the kernel
version is sufficiently recent, one can switch off this
overcommitting behavior using a command like
# echo 2 > /proc/sys/vm/overcommit_memory
<end quote>
I just can't believe my eyes.
This makes any serious usage of malloc completely impossible!
It always return true!
And if the system discovers it promised more than it actually has,
then it will start killing processes at random.
This means that you are compiling a big piece of code, and then another
process or service gets killed because the compiler requested too
much memory!
Besides, the "work around" proposed supposes OF COURSE that you
are root!!!!
How is it possible to screw things up like this?
Has anyone here observed this behavior?
What is the work around? I mean at the C level.
How can you avoid using malloc() ???
jacob
--
jacob navia
jacob at jacob point remcomp point fr
logiciels/informatique
http://www.cs.virginia.edu/~lcc-win32
Eric Sosman
> Hi
>
> I was working under linux today when I discovered this:
>
> man malloc
>
> <quote>
> [snip pages]
>
> BUGS
> By default, Linux follows an optimistic memory allocation strategy.
> This means that when malloc() returns non-NULL there is no guarantee
> that the memory really is available. This is a really bad bug.
> In case it turns out that the system is out of memory, one or more
> processes will be killed by the infamous OOM killer. In case Linux is
> employed under circumstances where it would be less desirable to
> suddenly lose some randomly picked processes, and moreover the kernel
> version is sufficiently recent, one can switch off this
> overcommitting behavior using a command like
> # echo 2 > /proc/sys/vm/overcommit_memory
> <end quote>
>
> I just can't believe my eyes.
Old news. In fact, it's Question 7.14 in the FAQ.
See also <http://lwn.net/Articles/104179/>.
santosh
> Hi
>
> I was working under linux today when I discovered this:
Did you just now discover this feature? Haven't we had discussions of this
several times in this group, the last one, IIRC, a month or two ago?
> man malloc
>
> <quote>
> [snip pages]
>
> BUGS
> By default, Linux follows an optimistic memory allocation strategy.
> This means that when malloc() returns non-NULL there is no guarantee
> that the memory really is available. This is a really bad bug.
> In case it turns out that the system is out of memory, one or more
> processes will be killed by the infamous OOM killer. In case Linux is
> employed under circumstances where it would be less desirable to
> suddenly lose some randomly picked processes, and moreover the kernel
> version is sufficiently recent, one can switch off this
> overcommitting behavior using a command like
> # echo 2 > /proc/sys/vm/overcommit_memory
> <end quote>
>
> I just can't believe my eyes.
>
> This makes any serious usage of malloc completely impossible!
Theoretically, but practically this is not much of a problem for most
programs.
> It always return true!
No. It means what it says, i.e., even if malloc returns a non-null pointer,
memory might not actually be allocated. Linux attempts to do so, when the
memory is actually written to.
> And if the system discovers it promised more than it actually has,
> then it will start killing processes at random.
>
> This means that you are compiling a big piece of code, and then another
> process or service gets killed because the compiler requested too
> much memory!
>
> Besides, the "work around" proposed supposes OF COURSE that you
> are root!!!!
>
> How is it possible to screw things up like this?
> Has anyone here observed this behavior?
Yes, once, and that too because of an artificial program I created to mimic
a DOS attack. It ate up memory until it was killed. In my experience, the
offending process was always killed, not someother random one.
> What is the work around? I mean at the C level.
Nothing, except to try to allocate memory only when you need it. Off-loading
some objects to a disk file would be a possible mitigating strategy, but
it's a big chore for a bug in some other program.
> How can you avoid using malloc() ???
If you don't mind a slight loss of portability, Linux specific APIs like brk
and getrlimit/setrlimit can help. Also consider mmap. With plain vanilla
malloc, there is nothing much you can do, except be as sparing as possible
with your memory allocation.
Spiros Bousbouras
> jacob navia wrote:
> > Hi
>
> > I was working under linux today when I discovered this:
> > By default, Linux follows an optimistic memory allocation strategy.
> > This means that when malloc() returns non-NULL there is no guarantee
> > that the memory really is available. This is a really bad bug.
> > In case it turns out that the system is out of memory, one or more
> > processes will be killed by the infamous OOM killer. In case Linux is
> > employed under circumstances where it would be less desirable to
> > suddenly lose some randomly picked processes, and moreover the kernel
> > version is sufficiently recent, one can switch off this
> > overcommitting behavior using a command like
> > # echo 2 > /proc/sys/vm/overcommit_memory
> > <end quote>
>
> several times in this group, the last one, IIRC, a month or two ago?
Indeed. Plus I can't imagine how can anyone implement a
whole C compiler for Linux without having a look at the
malloc man page and several other man pages for common
library functions.
As much as I'm tempted I won't comment on the rest of
Santosh's reply because I believe that discussion of
how malloc behaves under Linux is out of topic here.
jacob navia
> Indeed. Plus I can't imagine how can anyone implement a
> whole C compiler for Linux without having a look at the
> malloc man page and several other man pages for common
> library functions.
>
No, sorry. I thought that malloc would work as specified in the
standard. NULL in failure, non-NULL means success.
How stupid, I see it now.
> As much as I'm tempted I won't comment on the rest of
> Santosh's reply because I believe that discussion of
> how malloc behaves under Linux is out of topic here.
>
Yes. It is the same thing as Microsoft with its 640K limit
but in the inverse direction. Now there are just NO LIMITS!
Unless the OOM killer destroys your system of course!
Walter Roberson
jacob navia <ja...@nospam.org> wrote:
>I was working under linux today when I discovered this:
>man malloc
>BUGS
>By default, Linux follows an optimistic memory allocation strategy.
>This means that when malloc() returns non-NULL there is no guarantee
>that the memory really is available. This is a really bad bug.
>In case it turns out that the system is out of memory, one or more
>processes will be killed by the infamous OOM killer.
>I just can't believe my eyes.
>This makes any serious usage of malloc completely impossible!
>It always return true!
Please re-read that. It does not always return a non-NULL pointer:
what it *does* do is sometimes return a non-NULL pointer when there
is insufficient real memory to back the demand.
>And if the system discovers it promised more than it actually has,
>then it will start killing processes at random.
>How is it possible to screw things up like this?
>Has anyone here observed this behavior?
This behaviour has been around for quite some time, predating
linux by a fair bit. I've been using a certified Unix with that
behaviour for at least 15 years.
>What is the work around? I mean at the C level.
>How can you avoid using malloc() ???
There isn't much you can do about it at the C level: it is an OS
behaviour that has to do with the handling of multiple processes.
The good news is that these days it isn't often a problem in practice.
Back in the days when 128 Mb of memory was a couple of thousand
dollars, the behaviour could sometimes be a serious problem. My main
unix system at home is 128 Mb, and I use it several hours every day,
but I can't remember the last time a "random process" got killed on it;
more than 5 years ago, I'm sure.
It is important to understand that malloc() is not just a C library
call, not just doing convenient things that you could otherwise code in
C: malloc() is an implementation interface that has to talk to the
operating system to ask the OS to allocate virtual memory to you. The
problem noted is not really in malloc(): the problem noted is really in
what the OS does on malloc()'s behalf, telling malloc that memory is
available when there might not really be enough.
There were good reasons behind the design of the memory allocator
to over-allocate memory in a Unix system. It turns out that often
Unix programs do not actually use more than a fraction of the memory
that they request. I know that might *sound* extremely wrong, but
I'm speaking about what happens in Unix, not about what happens in
standard C programs.
There is a fundamental Unix-ism that requests a lot of memory that will
seldom be used. That fundamental Unix-ism has to do with spawning other
processes, which in Unix involves fork() (either directly or on your
behalf by system() or popen() or the like). fork() is defined as
duplicating the entire virtual address space of the current Unix
process; if you are in a forked process and you read a memory location,
you must get the value of that location before the fork [until you
write to it], and if you write to a memory location, the value written
does -not- affect the other process. This is often implemented with
"copy on write" memory pages, in which a single copy of a memory page
is shared between the two processes until one of them asks to write to
it, at which point the page contents are duplicated and both processes
get now-independant copies. Thus, although in theory you need to
duplicate the entire process into real memory, in practice only the
pages written to need to be duplicated, and they don't need to be
duplicated until the time of the writing -- by which point more real
memory might be available... or real memory might not be available and
the memory has to come from -somewhere-...
This fundamental Unix behaviour of logically duplicating the entire
address space upon a fork() also affects system() and popen() and
how "shells" work and how daemons are created, because the Unix
mechanism for starting up a new program requires that an existing
program fork() and that one of the two fork()'d copies then exec*()'s
the new executable. You might have a process that is using
more than half of real memory and that process might want to
(for example) system("ls *.txt") . If you insist that there must
always be enough real memory to back up what *might* be written to,
then the OS would have to tell the program that the system() failed,
because the internal fork() / exec() pair needed to implement the
system() call would temporarily require duplicating the "more-than-half
of memory" before immediately replacing that duplicated memory with
the new "ls" process that only needed perhaps 40Kb. fork() / exec()
pairs happen a lot in Unix, and the system designers had to decide
whether to let half of memory go mostly unused, or to allow virtual
overdrafts that only occasionally lead to actual problems. So what
they did is create mechanisms that allow the virtual overdrafts, but
gave the system administrators mechanisms to control the amount
of virtual overcommitment so that the system administrators could
decide for themselves what level of risk they were willing to put
up with.
--
"Beware of bugs in the above code; I have only proved it correct,
not tried it." -- Donald Knuth
jacob navia
The question 7.14 is
--------------------------------
.14: I've heard that some operating systems don't actually allocate
malloc'ed memory until the program tries to use it. Is this
legal?
A: It's hard to say.
-----------------------------------
????????
Why not say "linux" instead of "some operating systems" or at least
"some operating systems like linux for instance"
And the answer is REALLY clear: "It's hard to say"...
I never thought you were speaking about linux.
It is clear that:
<quote>
The malloc function allocates space for an object whose size is
specified by "size" and whose value is indeterminate.
Returns
3 The malloc function returns either a null pointer or a pointer to the
allocated space.
<end quote>
NULL means failure, and non NULL means that an object space is
allocated.
Joachim Schmitz
news:fe0l7v$inj$1...@canopus.cc.umanitoba.ca...
> In article <4703c603$0$5079$ba4a...@news.orange.fr>,
> jacob navia <ja...@nospam.org> wrote:
>>How is it possible to screw things up like this?
>>Has anyone here observed this behavior?
>
> This behaviour has been around for quite some time, predating
> linux by a fair bit. I've been using a certified Unix with that
> behaviour for at least 15 years.
jacob navia
[snip very good explanation about why unices overcommit]
Thank you for your answer. It is easier to understand now.
I found a piece of commentary that may be intersting
in
http://lwn.net/Articles/104179/
Andries Brouwer came up with this analogy:
An aircraft company discovered that it was cheaper to fly its planes
with less fuel on board. The planes would be lighter and use less fuel
and money was saved. On rare occasions however the amount of fuel was
insufficient, and the plane would crash. This problem was solved by the
engineers of the company by the development of a special OOF
(out-of-fuel) mechanism. In emergency cases a passenger was selected and
thrown out of the plane. (When necessary, the procedure was repeated.) A
large body of theory was developed and many publications were devoted to
the problem of properly selecting the victim to be ejected. Should the
victim be chosen at random? Or should one choose the heaviest person? Or
the oldest? Should passengers pay in order not to be ejected, so that
the victim would be the poorest on board? And if for example the
heaviest person was chosen, should there be a special exception in case
that was the pilot? Should first class passengers be exempted? Now that
the OOF mechanism existed, it would be activated every now and then, and
eject passengers even when there was no fuel shortage. The engineers are
still studying precisely how this malfunction is caused.
:-)
Lew Pitcher
> Hi
>
> I was working under linux today when I discovered this:
>
> man malloc
>
> <quote>
> [snip pages]
>
> BUGS
> By default, Linux follows an optimistic memory allocation strategy.
> This means that when malloc() returns non-NULL there is no guarantee
> that the memory really is available.
> What is the work around? I mean at the C level.
> How can you avoid using malloc() ???
By using the ISO Standard calloc() function call, which initializes
the allocated memory to "all bits zero".
<offtopic>
So long as a userspace program actually /uses/ the memory it
allocates, Linux memory management is happy. The Linux OutOfMemory
Killer only affects those programs that allocate more memory than they
use. calloc() ensures that the memory is "in use", by writing to it,
and thus excludes the allocation from the OOM killer's view.
</offtopic>
jacob navia
Well thanks a lot! That is a real work around since you do not
have to be root to do it!
This should go into that FAQ question, that is quite bad (see my other post)
Keith Thompson
> jacob navia wrote:
[...]
>> This makes any serious usage of malloc completely impossible!
>
> Theoretically, but practically this is not much of a problem for most
> programs.
>
>> It always return true!
>
> No. It means what it says, i.e., even if malloc returns a non-null pointer,
> memory might not actually be allocated. Linux attempts to do so, when the
> memory is actually written to.
jacob's statement isn't an unreasonable inference, though. Given the
description, it's plausible that malloc() will always return a
non-null result (which can be interpreted as "true").
An experiment on my own Linux box shows that it doesn't *quite* work
that way; malloc() does return a null pointer for very large arguments
(above about 2.1 gigabytes on a system with 1 gigabyte of physical
memory).
--
Keith Thompson (The_Other_Keith) ks...@mib.org <http://www.ghoti.net/~kst>
San Diego Supercomputer Center <*> <http://users.sdsc.edu/~kst>
"We must do something. This is something. Therefore, we must do this."
-- Antony Jay and Jonathan Lynn, "Yes Minister"
Eric Sosman
You'll have to ask Steve Summit. Maybe he's aware of
other systems than Linux, or maybe he didn't want to tar
all Linuces simply because some can be configured this way,
or -- well, you'll have to ask Steve.
> And the answer is REALLY clear: "It's hard to say"...
Well, it *is* hard to say! Besides, you're trying
to pass off the one-line summary as "the answer," when
in actuality the answer is fairly lengthy (follow the
link it offers).
> I never thought you were speaking about linux.
> It is clear that:
> [...]
> NULL means failure, and non NULL means that an object space is
> allocated.
Right. It is also clear that the Standard forbids
memory errors: Memory retains its last-stored value for
as long as the program runs, even if subjected to a hail
of alpha particles and a thunderstorm of static discharges,
right?
It could be argued -- it *has* been argued -- that
a system becomes non-conforming when (and if) it fails to
honor a memory allocation that has already "succeeded."
But from the program's point of view I see no reason to
view such non-conformance differently than that which
occurs when the power fails or when the CPU goes crazy.
There's been a system failure, the program doesn't behave
as advertised, that's life.
(Personally, I dislike the overcommit strategy -- but
whether it's an unrelievedly bad thing is "hard to say.")
santosh
> santosh <santo...@gmail.com> writes:
>> jacob navia wrote:
> [...]
>>> This makes any serious usage of malloc completely impossible!
>>
>> Theoretically, but practically this is not much of a problem for most
>> programs.
>>
>>> It always return true!
>>
>> No. It means what it says, i.e., even if malloc returns a non-null
>> pointer, memory might not actually be allocated. Linux attempts to do so,
>> when the memory is actually written to.
> [...]
>
> jacob's statement isn't an unreasonable inference, though. Given the
> description, it's plausible that malloc() will always return a
> non-null result (which can be interpreted as "true").
>
> An experiment on my own Linux box shows that it doesn't *quite* work
> that way; malloc() does return a null pointer for very large arguments
> (above about 2.1 gigabytes on a system with 1 gigabyte of physical
> memory).
Yes, that's why I raised that point. I too have observed malloc failing for
amounts much larger than the system's virtual memory.
Incidentally I wonder how far Lew Pitcher's recommendation to use calloc is
a good workaround? To me it seems that it would merely trigger the "process
culling" a bit earlier, with potentially the process being killed even
before calloc returned.
Anyway we should probably stop. This is getting very OT.
A. Bolmarcich
> Lew Pitcher wrote:
>> By using the ISO Standard calloc() function call, which initializes
>> the allocated memory to "all bits zero".
[snip]
> Well thanks a lot! That is a real work around since you do not
> have to be root to do it!
Are you sure that it is a real work around? After all, calloc can
- not obtain memory for the allocated space and return (just like malloc)
- before returning tell the operating system that when the program first
references the allocated space to obtain memory and initialize it to all
bits zero (for space allocated by malloc the operating system would
obtain memory and not initialize it)
Jean-Marc Bourguet
> jacob navia wrote On 10/03/07 13:58,:
> > Why not say "linux" instead of "some operating systems" or at least
> > "some operating systems like linux for instance"
>
> You'll have to ask Steve Summit. Maybe he's aware of
> other systems than Linux, or maybe he didn't want to tar
> all Linuces simply because some can be configured this way,
> or -- well, you'll have to ask Steve.
AIX also had the overcommit feature. I got that it is still the case but
not activated by default.
IMHO, the control should not be a global one, but a pre process one.
Yours,
--
Jean-Marc
Keith Thompson
That doesn't necessarily help. Writing to the memory you just
allocated doesn't let you handle an allocation failure; if the system
can't allocate the memory for you as you write it, it's going to start
sending signals or killing processes rather than returning 0 from the
calloc() call. It could make the failure happen sooner, but not
necessarily in a way that can be handled.
Also, depending on how the virtual memory system works, zeroing a
chunk of memory might not even require physically writing to it; newly
allocated memory might already be full of zeros.
Keith Thompson
There are some very topical questions here, though. Does lazy
allocation satisfy the C standard's requirements for malloc()?
(That's in two parts: does it satisfy the letter of the requirements,
and does it satisfy the intent?) And if it does, what programmers
need to do if we want our code to be portable?
Either way, it's obviously a difficult question; should the standard
be updated to clarify this point one way or the other? (That's a
question for comp.std.c.)
If the standard doesn't permit lazy allocation, then there's not a
whole lot we can say about it here; it just means that a lot of
systems are non-conforming.
If a program can be arbitrarily terminated due to some condition in
the operating environment, that doesn't necessarily make the C
implementation non-conforming. If my program fails to complete
execution because I pulled the power plug, I'm not going to complain
to my C compiler vendor. Programs can die due to all sorts of
resource constraints. Having *my* program die because *your* program
allocated a lot of memory and the OS didn't complain at the time of
allocation is certainly unpleasant, but it's far from clear that it's
non-conforming behavior.
Richard Tobin
As others have said, it's surprising that you haven't noticed the
many previous threads on this subject.
In practice, it's not usually a problem. I remember back in the early
80s people would attempt to run programs on our Vax (2MB real memory,
8MB swap) that would run for days because they were paging instead of
working in real memory. No-one does that nowadays: swap is not a
way to run bigger programs, it's jst a way to avoid wasting memory
on unused data. Long before you really run out of memory, system
performance becomes so poor that people don't put up with it.
The only situation where this is really likely to happen is when a
program madly allocates huge amounts of memory, due to a bug or a
misunderstanding about how much memory is available. And that case
should be handled by per-user or per-process memory quotas.
Obviously there are situations where overcommit is unacceptable,
and I believe you can turn it off if necessary.
-- Richard
--
"Consideration shall be given to the need for as many as 32 characters
in some alphabets" - X3.4, 1963.
santosh
Seemingly it satisfies the Standard's requirements _as long as_ the actual
storage can be allocated and used in it's entirety. It would satisfy the
Standard even when some other process is killed and sufficient memory is
reclaimed to allow our process to continue, since Standard C has no concept
of multiple interacting processes anyway. Of course a silent forced
termination of the program when it tries to use it's allocated storage is
against the intent of the Standard. The Standard clearly says that a
successfull return from malloc implies that the requested storage was
allocated and is available for use in it's entirety.
Is it actually non-conforming behaviour. Possibly, depending on how exactly
you chose to look at malloc and the host environment, either as a part of
the C implementation, and hence answerable to the Standard, or outside the
implementation and hence outside the requirements of the Standard entirely.
Personally I consider such a setup broken, but I wouldn't blame the C
implementation though.
<snip>
> Either way, it's obviously a difficult question; should the standard
> be updated to clarify this point one way or the other? (That's a
> question for comp.std.c.)
Is the issue really important enough in actual practice to justify it's
mention and clarification in the Standard, since only a fraction of the
existing C implementations actually exhibit this problem?
> If the standard doesn't permit lazy allocation, then there's not a
> whole lot we can say about it here; it just means that a lot of
> systems are non-conforming.
I can't see how the Standard does not permit lazy allocation, since it says
nothing about the details of how *alloc's storage is actually allocated.
> If a program can be arbitrarily terminated due to some condition in
> the operating environment, that doesn't necessarily make the C
> implementation non-conforming. If my program fails to complete
> execution because I pulled the power plug, I'm not going to complain
> to my C compiler vendor. Programs can die due to all sorts of
> resource constraints. Having *my* program die because *your* program
> allocated a lot of memory and the OS didn't complain at the time of
> allocation is certainly unpleasant, but it's far from clear that it's
> non-conforming behavior.
Yes, the issue is certainly ambiguous. I would consider that it, (the lazy
allocation feature), doesn't violate the letter of the Standard, but
certainly goes against it's intent. In any case the Standard has very
little control over this and making a statement one way or the other is not
going to affect the actual practice or the potential problem.
Kenneth Brody
>
> Spiros Bousbouras wrote:
> > Indeed. Plus I can't imagine how can anyone implement a
> > whole C compiler for Linux without having a look at the
> > malloc man page and several other man pages for common
> > library functions.
> >
>
> No, sorry. I thought that malloc would work as specified in the
> standard. NULL in failure, non-NULL means success.
>
> How stupid, I see it now.
Well, is the "problem" really in the C compiler's implementation
of malloc and friends? Or is it that the kernel lies to the C
library, telling it that it gave the memory to the process, when
the memory might not be available?
Note that the statement says "Linux follows an optimistic memory
allocation strategy", and that the "fix" involves changing a
setting in the kernel's VM handler.
> > As much as I'm tempted I won't comment on the rest of
> > Santosh's reply because I believe that discussion of
> > how malloc behaves under Linux is out of topic here.
> >
>
> Yes. It is the same thing as Microsoft with its 640K limit
> but in the inverse direction. Now there are just NO LIMITS!
>
> Unless the OOM killer destroys your system of course!
I have a feeling that malloc() would still return NULL if the
"optimistically allocated memory" amount were to exceed the
current process size limit.
--
+-------------------------+--------------------+-----------------------+
| Kenneth J. Brody | www.hvcomputer.com | #include |
| kenbrody/at\spamcop.net | www.fptech.com | <std_disclaimer.h> |
+-------------------------+--------------------+-----------------------+
Don't e-mail me at: <mailto:ThisIsA...@gmail.com>
Kenneth Brody
[... Linux's "lazy allocation" ...]
>
> <offtopic>
> So long as a userspace program actually /uses/ the memory it
> allocates, Linux memory management is happy. The Linux OutOfMemory
> Killer only affects those programs that allocate more memory than they
> use. calloc() ensures that the memory is "in use", by writing to it,
> and thus excludes the allocation from the OOM killer's view.
> </offtopic>
Would that really solve the issue, or would it merely cause the
passenger to be ejected earlier in the flight?
Flash Gordon
> Eric Sosman <Eric....@sun.com> writes:
>
>> jacob navia wrote On 10/03/07 13:58,:
>
>>> Why not say "linux" instead of "some operating systems" or at least
>>> "some operating systems like linux for instance"
>> You'll have to ask Steve Summit. Maybe he's aware of
>> other systems than Linux, or maybe he didn't want to tar
>> all Linuces simply because some can be configured this way,
>> or -- well, you'll have to ask Steve.
>
> AIX also had the overcommit feature. I got that it is still the case but
> not activated by default.
Actually it is in AIX 5.3
http://publib.boulder.ibm.com/infocenter/systems/index.jsp?topic=/com.ibm.aix.genprogc/doc/genprogc/sys_mem_alloc.htm
This document also gives the reasons for this policy.
> IMHO, the control should not be a global one, but a pre process one.
Under AIX it is via the use of an environment variable. So whether C
implementations conform under AIX might depend on how the user invokes
the executable.
Another interesting point in terms of discussions of how malloc is
implemented is that under AIX how malloc is implemented depends on the
environment of the user running your application.
--
Flash Gordon
jacob navia
> Seemingly it satisfies the Standard's requirements _as long as_ the actual
> storage can be allocated and used in it's entirety. It would satisfy the
> Standard even when some other process is killed and sufficient memory is
> reclaimed to allow our process to continue, since Standard C has no concept
> of multiple interacting processes anyway. Of course a silent forced
> termination of the program when it tries to use it's allocated storage is
> against the intent of the Standard. The Standard clearly says that a
> successfull return from malloc implies that the requested storage was
> allocated and is available for use in it's entirety.
>
> Is it actually non-conforming behaviour. Possibly, depending on how exactly
> you chose to look at malloc and the host environment, either as a part of
> the C implementation, and hence answerable to the Standard, or outside the
> implementation and hence outside the requirements of the Standard entirely.
>
> Personally I consider such a setup broken, but I wouldn't blame the C
> implementation though.
>
I agree with this. I am not an Unix expert but I fail to see why
this overallocation couldn't be done only at the level of fork()
instead of propagating this to all levels like in malloc().
"Performance" at all costs. This is a sad development.
jacob navia
You can even replace malloc with a function of your own!
(Only in C of course, not in C++)
Setting an environment variable seems more accessible than
requiring root privileges to write into the running
kernel binary...
Erik Trulsson
> Hi
>
> I was working under linux today when I discovered this:
>
> man malloc
>
> <quote>
> [snip pages]
>
> BUGS
> By default, Linux follows an optimistic memory allocation strategy.
> This means that when malloc() returns non-NULL there is no guarantee
> that the memory really is available. This is a really bad bug.
> In case it turns out that the system is out of memory, one or more
> processes will be killed by the infamous OOM killer. In case Linux is
> employed under circumstances where it would be less desirable to
> suddenly lose some randomly picked processes, and moreover the kernel
> version is sufficiently recent, one can switch off this
> overcommitting behavior using a command like
> # echo 2 > /proc/sys/vm/overcommit_memory
> <end quote>
>
> I just can't believe my eyes.
>
> This makes any serious usage of malloc completely impossible!
> It always return true!
Not always. There are several situations when malloc() will return false
even with overcommit enabled, e.g. if the process would exceed the limits
it has on memory-usage.
>
> And if the system discovers it promised more than it actually has,
> then it will start killing processes at random.
Not completely random, but yes.
>
> This means that you are compiling a big piece of code, and then another
> process or service gets killed because the compiler requested too
> much memory!
Could be.
>
> Besides, the "work around" proposed supposes OF COURSE that you
> are root!!!!
>
> How is it possible to screw things up like this?
> Has anyone here observed this behavior?
That behaviour is actually fairly common in many Unix-variants.
The reason for it is that many programs has a tendency to allocate
much more memory than they actually use. The OS will not actually
allocate memory to the process until it uses the memory (this is done
on a page basis.)
In practice this actually lets you run *more* processes at the same time
without running out of memory - under normal circumstances anyway.
Of course when you do run out of virtual memory (physical RAM + swap) then
things do not work so well anymore.
Most Unix-variants do not handle running out swap at all well. That situation
is best avoided.
>
> What is the work around? I mean at the C level.
There is no work around at the C level.
The thing to remember when setting up a Unix-system is to make sure the
system has plenty of swap configured.
> How can you avoid using malloc() ???
You don't. You use it like you normally would and hope for the best.
--
<Insert your favourite quote here.>
Erik Trulsson
ertr...@student.uu.se
Tor Rustad
I don't see how this solves anything, malloc() and calloc() can both
trigger the OOM killer.
malloc() returns a *valid* address, a virtual memory page, even if there
is no physical page available, touching the page early... helps how?
The oom_badness() scoring function, is not affected by calloc(), so I
can't see calloc() affect which process-to-be-killed either.
--
Tor <torust [at] online [dot] no>
"Life is full of shit, if you look at it"
Keith Thompson
>> Either way, it's obviously a difficult question; should the standard
>> be updated to clarify this point one way or the other? (That's a
>> question for comp.std.c.)
>
> Is the issue really important enough in actual practice to justify it's
> mention and clarification in the Standard, since only a fraction of the
> existing C implementations actually exhibit this problem?
I think it's a pretty large fraction. It seems to affect almost all
Linux systems (systems that use the Linux kernel, if you want to be
picky), and a number of Unix system as well. That's a significant
chunk of hosted implementations. (Linux is used in some embedded
systems, but freestanding implementations aren't required to support
malloc() -- and no, "embedded" and "freestanding" aren't synomymous.)
[...]
Tor Rustad
> santosh <santo...@gmail.com> writes:
>> Keith Thompson wrote:
> [...]
>>> Either way, it's obviously a difficult question; should the standard
>>> be updated to clarify this point one way or the other? (That's a
>>> question for comp.std.c.)
>> Is the issue really important enough in actual practice to justify it's
>> mention and clarification in the Standard, since only a fraction of the
>> existing C implementations actually exhibit this problem?
>
> I think it's a pretty large fraction. It seems to affect almost all
> Linux systems (systems that use the Linux kernel, if you want to be
> picky), and a number of Unix system as well. That's a significant
> chunk of hosted implementations. (Linux is used in some embedded
> systems, but freestanding implementations aren't required to support
> malloc() -- and no, "embedded" and "freestanding" aren't synomymous.)
Embedded systems may use a more deterministic OOM killer. Furthermore,
OOM_DISABLE can be set for a particular process, also memory usage can
be controlled with POSIX setrlimit().
When doing fault-tolerant programming, one often use a monitor process,
which start/restart child processes.
If following MISRA-C, not using malloc is a required (IIRC) rule. You
want deterministic behavior on this.
Pierre Asselin
> Lew Pitcher <lpit...@teksavvy.com> writes:
> >
> > <offtopic>
> > So long as a userspace program actually /uses/ the memory it
> > allocates, Linux memory management is happy. The Linux OutOfMemory
> > Killer only affects those programs that allocate more memory than they
> > use. calloc() ensures that the memory is "in use", by writing to it,
> > and thus excludes the allocation from the OOM killer's view.
> > </offtopic>
> That doesn't necessarily help. Writing to the memory you just
> allocated doesn't let you handle an allocation failure; if the system
> can't allocate the memory for you as you write it, it's going to start
> sending signals or killing processes rather than returning 0 from the
> calloc() call. It could make the failure happen sooner, but not
> necessarily in a way that can be handled.
I agree that it doesn't help. AIX, one of the overcommitting OSes,
used to have a kludge involving a new signal, SIGDANGER, sent to
everybody when overcommitted memory got tight. IBM gave away the
source of an unsupported psmalloc() function that you could use as
a non-overcommitting malloc. It worked by touching each allocated
page (just one byte, not zeroing the whole thing), catching SIGDANGER
and lonjmp()-ing to free() the space and return NULL. If touching
the pages didn't trigger SIGDANGER, the normal path returned
memory that you could always use.
I didn't like it at the time --or rather, I didn't like the
overcommits-- but it was clever, and much better than nothing.
--
pa at panix dot com
Tor Rustad
[...]
> Obviously there are situations where overcommit is unacceptable,
> and I believe you can turn it off if necessary.
From the link Eric gave:
"Simply setting the sysctl parameter vm/overcommit_memory to 2 turns off
the overcommit behavior and keeps the OOM killer forever at bay."
Getting "kernel panic" instead, isn't a great alternative though.
Keith Thompson
> santosh <santo...@gmail.com> writes:
>> Keith Thompson wrote:
> [...]
>>> Either way, it's obviously a difficult question; should the standard
>>> be updated to clarify this point one way or the other? (That's a
>>> question for comp.std.c.)
>>
>> Is the issue really important enough in actual practice to justify it's
>> mention and clarification in the Standard, since only a fraction of the
>> existing C implementations actually exhibit this problem?
>
> I think it's a pretty large fraction. It seems to affect almost all
> Linux systems (systems that use the Linux kernel, if you want to be
> picky), and a number of Unix system as well. That's a significant
> chunk of hosted implementations. (Linux is used in some embedded
> systems, but freestanding implementations aren't required to support
> malloc() -- and no, "embedded" and "freestanding" aren't synomymous.)
It's been said in this thread (sorry, I don't recall by whom) that
studies had shown that many Unix programs allocate large amounts of
memory that they never use.
If this is true, it's an argument in favor of lazy allocation
<TOPICAL>which may or may not meet the C standard's requirements for
malloc()</TOPICAL>.
But I'm rather skeptical of the claim. It seems to me that a
well-written C program will malloc() only the memory that it actually
needs. Obviously the amount of memory to allocate can be determined
at run time; if you find later that you need more, there's always
realloc().
It's true that fork() can result in large chunks of virtual memory
space that are never used, since the child process typically replaces
itself with another executable via one of the exec*() functions. But
that's different from malloc().
Can anyone cite an actual study that says that Unix programs (or C
programs under any OS) often allocate much more memory than they
actually use? If so, does this really apply to malloc/calloc/realloc
calls, or just to other forms of allocation?
If, as I suspect, most C programs actually use most or all of the
memory that they explicitly allocate, lazy allocation doesn't really
buy you anything. And if future versions of various operating systems
were to drop it, or at least disable it by default, we could avoid
having this discussion yet again.
(Yes, I'm skating near the edge of topicality here, but I'd argue that
I'm using lazy allocation to illustrate questions about how malloc is
required to behave.)
Eric Sosman
> santosh wrote:
>>
>> Personally I consider such a setup broken, but I wouldn't blame the C
>> implementation though.
>>
>
> I agree with this. I am not an Unix expert but I fail to see why
> this overallocation couldn't be done only at the level of fork()
> instead of propagating this to all levels like in malloc().
malloc() and fork() compete for the same resource:
memory able to store and retain values.
Somewhere a ways upstream from your happy homestead,
AgriColossus has built a dam to irrigate their crops. They
promise to take only fifty percent of the total flow, so
you'll have plenty left for your own modest needs.
Unfortunately for you, GigaPower has also built a dam
diverting water to turn their turbines and cool their nuclear
power plants. They, too, take only fifty percent of the total
flow, leaving you plenty.
Alas, the two of them together have diverted all the water,
and you cannot even dampen your toothbrush in the now-dry creek.
Which one of them would you say is to blame for this sad outcome?
--
Eric Sosman
eso...@ieee-dot-org.invalid
Eric Sosman
> [...]
>
> It's true that fork() can result in large chunks of virtual memory
> space that are never used, since the child process typically replaces
> itself with another executable via one of the exec*() functions. But
> that's different from malloc().
In what way is it "different from malloc()?" Or, if you
prefer, from the system services that underlie malloc()? In
particular: If Process P uses malloc() to grab a bunch of
memory and then calls fork() to create Process Q, do you think
that Q's memory is somehow segregated into "stuff P obtained by
malloc()" and "stuff P obtained by other means?"
Both fork() and malloc() increase the demand for memory,
where "memory" is understood as some combination of RAM and swap
and hocus-pocus capable of preserving a stored value. Conservative
systems report failure on any operation that *might* require more
memory than they are able to supply; happy-go-lucky systems report
success every time and hope for the best. The actual systems I
know of fall along a spectrum between these extremes.
> Can anyone cite an actual study that says that Unix programs (or C
> programs under any OS) often allocate much more memory than they
> actually use? If so, does this really apply to malloc/calloc/realloc
> calls, or just to other forms of allocation?
My working life these days is spent with a (third-party)
application that typically occupies about 2-2.5 GB of memory.
Every time it starts a sub-process -- which happens routinely,
thank you -- it calls fork() and creates an instantaneous demand
to double its memory footprint. And, of course, the new process
very soon calls an exec() variant, throwing away all that inherited
memory and replacing the entire address space with a new program.
The potential memory demand is large; the actual demand is small.
Again, I ask whether you imagine that the original process'
mallocated memory is somehow tagged as "different" from all the
rest of its memory -- and if it were, what difference you think
that might make.
The O/S I work with lies toward the conservative end of the
spectrum (but not at the "100% guaranteed" point; I don't know of
any that go quite that far). One consequence is that we always
tell people to provide lots of swap space, but we're perfectly
happy to let them allocate it on big, slow disks. Why? Because
even though it must be there if fork() is to succeed, it will
almost never see actual use. Luckily, big slow disks are cheap.
--
Eric Sosman
eso...@ieee-dot-org.invalid
Walter Roberson
Keith Thompson <ks...@mib.org> wrote:
>It's been said in this thread (sorry, I don't recall by whom) that
>studies had shown that many Unix programs allocate large amounts of
>memory that they never use.
>But I'm rather skeptical of the claim. It seems to me that a
>well-written C program will malloc() only the memory that it actually
>needs.
>It's true that fork() can result in large chunks of virtual memory
>space that are never used, since the child process typically replaces
>itself with another executable via one of the exec*() functions. But
>that's different from malloc().
>Can anyone cite an actual study that says that Unix programs (or C
>programs under any OS) often allocate much more memory than they
>actually use?
Sorry that my posting was not clearer. I was referring to the
fork() behaviour; fork() is the -only- way to start a process in
POSIX.1 (vfork() is not standard POSIX). Every process starts off
as a clone of its parent, with pretty much just the process ID
differentiating the two. If malloc() stores its overhead within
the process memory, then all the information about what was malloc()'d
gets duplicated virtually (because all the memory pages must read out
the same); if malloc() stores its overhead outside the process memory,
then all the information about what was malloc()'d must be duplicated
by the OS (e.g., free() from the child must have the same result
as free() from the parent); either way, the malloc()'d memory must
get duplicated. At some meta level we can differentiate
between malloc() and fork() behaviour, but the effect at the process
level has to be the same as if all the malloc()'s had happened, so
the meta difference becomes moot for the purpose of this discussion.
Once the new process memory is allocated as a clone of its parent,
then the new process generally, as you noted, exec()'s off a different
binary, with the implicit freeing of the malloc()'d memory -- but
the exec()'d binary retains the same process container (and some
attributes of the old executable are inheritted, including some I/O
linkages). In-between the process essentially "gives back" all that
malloc()'d memory and starts afresh with a new malloc() arena and
likely with completely different memory page attributes (about which
pages are readable, writable, executable, etc.)
Once started afresh, processes are often fairly well behaved on memory
allocation... until, that is, they need to system() or popen() or the
like, at which point the OS becomes a virtual wastrel because of the OS
assumption that copies of what has gone before will be needed in both
the old and new processes.
Even so, it is (or was) not uncommon for processes to allocate
large blocks of memory, enough to hold the maximum problem size
they are configured to handle. realloc() does exist, yes, but has
the problem that the memory might move, with all the intendent
hastles of updating all the pointers into the memory block.
Sometimes programmers start out by allocating as much memory as
they might ever need, preferring to immediately catch the
low-memory condition and just not go very far, rather than
finding out 15 levels down 12 hours in that there isn't enough
memory to go further. The program might try to "checkpoint" then,
but checkpointing usually requires a block of working memory, so
it is usually better to checkpoint while you know you still have
enough memory, and then just die gracefully if memory runs out.
And then there's the hybrid strategy of allocating a large checkpoint
buffer at the beginning and leaving it untouched against the
eventuality of running out of memory and needing to checkpoint.
--
"Any sufficiently advanced bug is indistinguishable from a feature."
-- Rich Kulawiec
J. J. Farrell
> Eric Sosman wrote:
> > jacob navia wrote On 10/03/07 12:39,:
> >> Hi
>
> >> I was working under linux today when I discovered this:
>
> >> man malloc
>
> >> <quote>
> >> [snip pages]
>
> >> BUGS
> >> By default, Linux follows an optimistic memory allocation strategy.
> >> This means that when malloc() returns non-NULL there is no guarantee
> >> In case it turns out that the system is out of memory, one or more
> >> processes will be killed by the infamous OOM killer. In case Linux is
> >> employed under circumstances where it would be less desirable to
> >> suddenly lose some randomly picked processes, and moreover the kernel
> >> version is sufficiently recent, one can switch off this
> >> overcommitting behavior using a command like
> >> # echo 2 > /proc/sys/vm/overcommit_memory
> >> <end quote>
>
> >> I just can't believe my eyes.
>
> > See also <http://lwn.net/Articles/104179/>.
>
> The question 7.14 is
> --------------------------------
> .14: I've heard that some operating systems don't actually allocate
> malloc'ed memory until the program tries to use it. Is this
> legal?
>
> A: It's hard to say.
> -----------------------------------
> ????????
>
> "some operating systems like linux for instance"
Perhaps that text was written before Linux was created, or perhaps the
author wasn't aware that Linux worked this way. Why should it
expicitly mention one particular OS?
> And the answer is REALLY clear: "It's hard to say"...
>
> I never thought you were speaking about linux.
> It is clear that:
> <quote>
> The malloc function allocates space for an object whose size is
> specified by "size" and whose value is indeterminate.
> Returns
> 3 The malloc function returns either a null pointer or a pointer to the
> allocated space.
> <end quote>
>
> NULL means failure, and non NULL means that an object space is
> allocated.
Malloc operates in the process's virtual memory space; if there's
allocatable space available in the part of the process's virtual
memory which malloc controls, it will return that virtual range. If
the OS can't find any physical memory to implement part of the
process's virtual memory when the process tries to write to it, the OS
will kill the process. Malloc works as advertised, but the OS can
randomly kill the process at any time.
Keith Thompson
> Keith Thompson wrote:
>> [...]
>> It's true that fork() can result in large chunks of virtual memory
>> space that are never used, since the child process typically replaces
>> itself with another executable via one of the exec*() functions. But
>> that's different from malloc().
>
> In what way is it "different from malloc()?" Or, if you
> prefer, from the system services that underlie malloc()? In
> particular: If Process P uses malloc() to grab a bunch of
> memory and then calls fork() to create Process Q, do you think
> that Q's memory is somehow segregated into "stuff P obtained by
> malloc()" and "stuff P obtained by other means?"
>
> Both fork() and malloc() increase the demand for memory,
> where "memory" is understood as some combination of RAM and swap
> and hocus-pocus capable of preserving a stored value. Conservative
> systems report failure on any operation that *might* require more
> memory than they are able to supply; happy-go-lucky systems report
> success every time and hope for the best. The actual systems I
> know of fall along a spectrum between these extremes.
Hmm.
Here's what I *think* I'd like to suggest. Please poke holes in it.
(And yes, this is Unix-specific, but it's still related to whether
certain behaviors conform to standard C. Let me know if you'd rather
continue this in comp.unix.programmer.)
When a process invokes malloc(), the malloc() succeeds (returns a
non-null result) if and only if the memory is actually available. If
a later attempt to access that memory would have failed, let the
failure occur within malloc() itself, and cause it to return a null
pointer.
fork() allocates a (potentially) big new chunk of virtual memory,
including memory that had been successfully allocated by the parent
process. It's appropriate to do lazy allocation during fork(). A
child process can't be gauranteed that it will be able to access its
own memory (whether it came from malloc() or not). But even after the
fork(), the parent still owns the memory it allocated, and can expect
to be able to use it. (I'm assuming this distinction between parent
and child processes is sensible; maybe it isn't.)
This means that a portable ISO C program (which can't use fork()
because it's non-standard) can assume that malloc() will work as
intended; either it really gets the memory it requested, or malloc()
reports failure, by returning a null pointer. A program that uses
fork() has to cope with the system-specific vagaries that that
implies.
[...]
> Again, I ask whether you imagine that the original process'
> mallocated memory is somehow tagged as "different" from all the
> rest of its memory -- and if it were, what difference you think
> that might make.
No, immediately after the child is created by fork(), I imagine that
all its memory is treated the same way; accessing any of it risks
invoking the OOM killer. But if the child process calls malloc()
*after* fork(), then it owns that newly allocated memory (or else
the malloc() should have explicitly failed).
Other posters have mentioned ways to turn off lazy allocation on Linux
(by writing something under /proc) and AIX (by setting an environment
variable). In both cases, I presume that this doesn't cause fork() to
physically allocate all the duplicated memory, but only affects the
behavior of malloc/realloc/calloc. What I'm suggesting is that that
should be the default behavior.
[...]
Walter Roberson
J. J. Farrell <j...@bcs.org.uk> wrote:
>Malloc operates in the process's virtual memory space; if there's
>allocatable space available in the part of the process's virtual
>memory which malloc controls, it will return that virtual range. If
>the OS can't find any physical memory to implement part of the
>process's virtual memory when the process tries to write to it, the OS
>will kill the process. Malloc works as advertised, but the OS can
>randomly kill the process at any time.
Although the scheme you describe is not impossible, I have never
encountered a system in which malloc() worked as you describe; it would
not be the norm. The overcommitting operating systems we have been
discussing thus far in this thread do not work that way: they
generally keep track of physical memory (RAM) allocated and disk memory
(swap) allocated, and virtually allocated (but not-yet physically
backed) memory, and will refuse allocations that exceed the
system-configured limitations on total virtual (unbacked by RAM or
disk) memory. When such OS's are configured to permit 0 virtual
(unbacked) memory, the effect is the same as OS's that do not allow
overcommitting at all. I haven't encountered any OS which had
effectively unlimited unbacked memory.
The operating system I use most often, SGI IRIX, does not handle
overcommitting by way of a kernel writable parameter like Linux does;
instead, it handles it in the same way that it handles disk swap space.
Just as you can add or remove swap space while the system is running,
you can add or remove virtual swap while IRIX is running; indeed,
it uses the traditional 'swap' command for this purpose. The man
page has several paragraphs of descriptive warnings.
http://techpubs.sgi.com/library/tpl/cgi-bin/getdoc.cgi?coll=0650&db=man&fname=/usr/share/catman/a_man/cat1/swap.z
--
"No one has the right to destroy another person's belief by
demanding empirical evidence." -- Ann Landers
robert...@yahoo.com
> fork() allocates a (potentially) big new chunk of virtual memory,
> including memory that had been successfully allocated by the parent
> process. It's appropriate to do lazy allocation during fork(). A
> child process can't be gauranteed that it will be able to access its
> own memory (whether it came from malloc() or not). But even after the
> fork(), the parent still owns the memory it allocated, and can expect
> to be able to use it. (I'm assuming this distinction between parent
> and child processes is sensible; maybe it isn't.)
On the systems I'm sufficiently familiar with, there's no real
distinction between parent and child processes in that context - the
copy on write happens as soon as either party writes to the shared
page.
What your suggesting would require that when the parent process writes
to a shared page, the child process has to be suspended and the COW
has to happen there.
In fact, the implementation implications would strongly argue against
maintaining such a relationship. Think about how complex a case you
might have to deal with where dozens of processes in a complex tree
map a shared page (and with additional dozens of processes perhaps
already having COW'd, thus leaving the hierarchy sparse). The single
write might require changing the mappings in dozens of other address
spaces. Impossible? Hardly, but the usual approach just maps a new
page into the current address space and decrements the COW counter in
the old page.
Richard Heathfield
> On Oct 3, 6:58 pm, jacob navia <ja...@nospam.org> wrote:
<snip>
>> Why not say "linux" instead of "some operating systems" or at least
>> "some operating systems like linux for instance"
>
> Perhaps that text was written before Linux was created, or perhaps the
> author wasn't aware that Linux worked this way. Why should it
> expicitly mention one particular OS?
If you're looking for a conspiracy but find evidence against it, your best
bet is to find a way to twist such evidence until it appears to support
your case.
--
Richard Heathfield <http://www.cpax.org.uk>
Email: -http://www. +rjh@
Google users: <http://www.cpax.org.uk/prg/writings/googly.php>
"Usenet is a strange place" - dmr 29 July 1999
Chris Torek
we do not have "swap space", much less any issue of overcommit, so
the issue does not arise.)
>Eric Sosman <Eric....@sun.com> writes:
>> You'll have to ask Steve Summit [why the FAQ does not speak
only of Linux when describing VM systems with overcommit]. Maybe
>>he's aware of other systems than Linux ...
There are many.
>>or maybe he didn't want to tar all Linuces simply because some
>>can be configured this way,
Some are configured that way by default, some are not.
>>or -- well, you'll have to ask Steve.
Indeed.
In article <87y7ekv...@news.bourguet.org>
Jean-Marc Bourguet <j...@bourguet.org> wrote:
>AIX also had the overcommit feature. I got that it is still the case but
>not activated by default.
>
>IMHO, the control should not be a global one, but a pre process one.
Unfortunately, this idea -- while attractive -- does not actually
work, at least not in the obvious simple way. If you have a system
in which some processes have managed to overcommit swap space, and
the system encounters a deadlock situation, any number of "well
behaved" processes may wind up in the same trouble. One might
think that the system could simply terminate any or even all
"overcomit-able" processes, but sometimes this kind of termination
requires allocating new resources. (In short, one has to do the
equivalent of malloc() in order to complete the equivalent of
kill(), and it is the malloc() that needs to do the kill() -- hence
the deadlock.)
(It *is* possible to make it work if one sets limits on both the
over- and non-over-commitment levels. The accounting gets fairly
hairy. This also makes various assumptions about the size and
permanence of backing store, i.e., it may become impossible to
remove or shrink a backing-store area. If backing store is to be
shared with regular "disklike" storage, whose size varies depending
on the amount of live data, there are tradeoffs here as well.)
--
In-Real-Life: Chris Torek, Wind River Systems
Salt Lake City, UT, USA (40°39.22'N, 111°50.29'W) +1 801 277 2603
email: forget about it http://web.torek.net/torek/index.html
Reading email is like searching for food in the garbage, thanks to spammers.