Malloc vs. Calloc
Pat Helles
Hello,
1 2 3 4 5 6 7
1234567890123456789012345678901234567890123456789012345678901234567890
I need to pick the brains of some really serious C programmers.
This question has to do with the differences between 'malloc' & 'calloc'.
I had always been under the impression that the only real difference
was that 'calloc' initialized its memory block to all zeros before
returning the pointer & 'malloc' just returned the pointer to the
memory block. However, I was recently informed that 'calloc' was
significantly faster than 'malloc'; also, 'malloc' does not return
a block of contiguous memory & therefore, consumes more memory than
'calloc' which does return a contiguous block of memory.
The reason I am wondering about the real inner workings of these
two is because I'm having trouble finding anything anywhere else to
support these claims. The UNIX man page I have says that 'calloc'
calls 'malloc' to allocate the memory, initializes the block and
then returns a pointer to the iniitialized block. Seems to me that
if either routine were to take longer, it would be 'calloc' and
not 'malloc'.
I would very much appreciate any information either supporting
or disproving the claims made.
Doubting Thomas.
Kevin D. Quitt
>was that 'calloc' initialized its memory block to all zeros before
>returning the pointer & 'malloc' just returned the pointer to the
>memory block.
You were correct. There are some other, more subtle differences. On
some systems, malloc may not actually allocate the memory for you
(athough it does map the memory into your address space). On such a
system, you are assigned physical memory only when you try to access it
(this can help minimize the amount of memory in use, and therefore
reduce paging memory to disk). Calloc presumably actually allocates the
memory (as well as mapping it), since it writes on the memory.
>However, I was recently informed that 'calloc' was
>significantly faster than 'malloc'; also, 'malloc' does not return
>a block of contiguous memory & therefore, consumes more memory than
>'calloc' which does return a contiguous block of memory.
This person is seriously deranged. The memory returned by malloc is
(mapped as) a contiguous block of memory. Generally, calloc is slower
since in most implementations it first calls malloc for the memory,
then has to zero the allocated memory.
> The UNIX man page I have says that 'calloc'
>calls 'malloc' to allocate the memory, initializes the block and
>then returns a pointer to the iniitialized block.
Not guaranteed, but a good bet.
>Seems to me that
>if either routine were to take longer, it would be 'calloc' and
>not 'malloc'.
Bang on. I can conceive of a system where calloc and malloc were
equally fast (one where the OS zeroes all memory before giving to a
task), but not one where malloc is slower. I suppose on the above
system, malloc could calculate and fill memory with random numbers.
--
#include <standard.disclaimer>
_
Kevin D Quitt USA 91351-4454 96.37% of all statistics are made up
Miguel Carrasquer
Kevin D. Quitt <k...@prof.jpl.nasa.gov> wrote:
>
>Bang on. I can conceive of a system where calloc and malloc were
>equally fast (one where the OS zeroes all memory before giving to a
>task), but not one where malloc is slower. I suppose on the above
>system, malloc could calculate and fill memory with random numbers.
>
Or a malloc on such a zero-filling OS that tries to emulate an RS/6000,
and has to fill the memory with 'DEADBEEFDEADBEEF' :-)
--
Miguel Carrasquer ____________________ ~~~
Amsterdam [ ||]~
m...@inter.NL.net ce .sig n'est pas une .cig
Dave Mooney
> (athough it does map the memory into your address space). On such a
> system, you are assigned physical memory only when you try to access it
> (this can help minimize the amount of memory in use, and therefore
> reduce paging memory to disk). Calloc presumably actually allocates the
> memory (as well as mapping it), since it writes on the memory.
This isn't even always true. If the implementation of calloc() knows
that the OS zeroes allocate-on-demand memory, then it doesn't need to
touch any of the memory either. The only thing you can guarantee is that
calloc() returns zeroed memory and malloc() returns memory full of random
stuff.
dave
--
Dave Mooney d...@vnet.ibm.com
"We saw the buddha in a bar down south talking politics and nuclear fission"
Christopher R. Volpe
>
> Hello,
>
> 1 2 3 4 5 6 7
>1234567890123456789012345678901234567890123456789012345678901234567890
> I need to pick the brains of some really serious C programmers.
>This question has to do with the differences between 'malloc' & 'calloc'.
>I had always been under the impression that the only real difference
>was that 'calloc' initialized its memory block to all zeros before
>returning the pointer & 'malloc' just returned the pointer to the
>memory block. However, I was recently informed that 'calloc' was
>significantly faster than 'malloc';
The Standard does not constrain execution speed.
>also, 'malloc' does not return
>a block of contiguous memory & therefore, consumes more memory than
>'calloc' which does return a contiguous block of memory.
This is false.
>
> The reason I am wondering about the real inner workings of these
>two is because I'm having trouble finding anything anywhere else to
>support these claims. The UNIX man page I have says that 'calloc'
>calls 'malloc' to allocate the memory, initializes the block and
>then returns a pointer to the iniitialized block. Seems to me that
>if either routine were to take longer, it would be 'calloc' and
>not 'malloc'.
Probably true.
--
Chris Volpe Phone: (518) 387-7766 (Dial Comm 8*833
GE Corporate R&D Fax: (518) 387-6560
PO Box 8, Schenectady, NY 12301 Email: vol...@crd.ge.com
Larry Weiss
> I need to pick the brains of some really serious C programmers.
>This question has to do with the differences between 'malloc' & 'calloc'.
The standard is neutral about the execution time characteristics of
the standard functions calloc() and malloc(). Common sense would
indicate that since malloc() has less work to do (receive one less
argument, presumably make one less multiply, need not initialize
the allocated memory) that it would be faster than calloc(). Maybe
not much faster.
The relevant sections are in section 7.10.3 and its subsections.
--
Larry Weiss, l...@oc.com
214/888-0471
Dana How
>Thus wrote mor...@cybele.unm.edu (Pat Helles)
>>I had always been under the impression that the only real difference
>>was that 'calloc' initialized its memory block to all zeros before
>>returning the pointer & 'malloc' just returned the pointer to the
>>memory block.
>>
>>
>>However, I was recently informed that 'calloc' was
>>significantly faster than 'malloc'; also, 'malloc' does not return
>>a block of contiguous memory & therefore, consumes more memory than
>>'calloc' which does return a contiguous block of memory.
>
>This person is seriously deranged. The memory returned by malloc is
>(mapped as) a contiguous block of memory. Generally, calloc is slower
>since in most implementations it first calls malloc for the memory,
>then has to zero the allocated memory.
I think the answer here is quite simple. "This person" simply viewed
calloc() as a method for allocating an _array_, and was comparing this
to calling malloc() for each element of the array (and also having
to create an array of pointers to these objects). The man pages
for these two functions would certainly cause a novice to view them
this way.
>
>...
>
>Bang on. I can conceive of a system where calloc and malloc were
>equally fast (one where the OS zeroes all memory before giving to a
>task), but not one where malloc is slower.
Certainly the implementation of malloc is no slower, but the method above
would make its _use_ slower. I think that's all that's going on here.
Stephen Baynes
: Kevin D. Quitt <k...@prof.jpl.nasa.gov> wrote:
: This isn't even always true. If the implementation of calloc() knows
: that the OS zeroes allocate-on-demand memory, then it doesn't need to
: touch any of the memory either. The only thing you can guarantee is that
: calloc() returns zeroed memory and malloc() returns memory full of random
: stuff.
Persoanally I would not rely on malloc() returning a useful source of random
numbers. The standard does not define what distribution should be used
by the source of random numbers that malloc() uses.
--
Stephen Baynes bay...@mulsoc2.serigate.philips.nl
Philips Semicondutors Ltd
Southampton My views are my own.
United Kingdom
Doug McDonald
>Kevin D. Quitt <k...@prof.jpl.nasa.gov> wrote:
>> On some systems, malloc may not actually allocate the memory for you
>> (athough it does map the memory into your address space). On such a
>> system, you are assigned physical memory only when you try to access it
>> (this can help minimize the amount of memory in use, and therefore
>> reduce paging memory to disk).
This is true, BUT with an important caveat: the system MUST, if this is
a legal hosted C compiler, guarantee that sufficient memory exists
at the time of the call (either real memory and/or virtual memory on disk)
to fulfill the request, and must continue to guarantee that this memory
is assigned to the program. In other words, a system that simply assumes
that memory exists and later fails do to "insufficient memory" is
simply broken, and the C compiler/library MUST be clearly labelled as
NOT STANDARD CONFORMING and can't be advertised as "full ANSI C".
(It could be advertised as "unhosted ANSI C", as malloc is not part
of an unhosted compiler.)
Doug McDonald
Adrienne J. Schaffer
In a previous article, mcdo...@aries.scs.uiuc.edu (Doug McDonald) says:
>In article <31jo0p$12...@tornews.torolab.ibm.com> da...@neutron.torolab.ibm.com (Dave Mooney) writes:
>
>>Kevin D. Quitt <k...@prof.jpl.nasa.gov> wrote:
>>> On some systems, malloc may not actually allocate the memory for you
>>> (athough it does map the memory into your address space). On such a
>>> system, you are assigned physical memory only when you try to access it
>>> (this can help minimize the amount of memory in use, and therefore
>>> reduce paging memory to disk).
>
>
>This is true, BUT with an important caveat: the system MUST, if this is
>a legal hosted C compiler, guarantee that sufficient memory exists
>at the time of the call (either real memory and/or virtual memory on disk)
>to fulfill the request, and must continue to guarantee that this memory
>is assigned to the program. In other words, a system that simply assumes
I would like to see the citation on this.
While I agree that it would be horrible to have a program stop dead in the
middle of its execution due to insuffiecient memory, but I can some
situations where what you specify may not always be possible. First of all,
how does a compiler "continue to guarantee that this memory is assigned to
the program"?
Second, what if there is no way for a program running under a particular OS
to know exactly how much memory can actually be used. Say for example, all
available memory is used dynamically for disk cacheing, and when a memory
page that is used for the cache but was allocated to a user program gets
written to, it is zeroed and actually given to the program. In that case,
there may never be more than 8K free. Should all mallocs over 8K fail?
There may be no way to tell how much memory is available until a write
fails. Admittedly, there should be some way to avoid that.
Third, what if you *want* to allocate 100MB for a sparse array which may
only be written to 5 times. If it had to be available before being
allocated, malloc(100000000) would always fail (on most machines), but
otherwise you could have hundreds of these arays allocated if you needed
it.
--
aj...@po.cwru.edu (or in other words, NOT YOU!)
Norman Diamond
>In a previous article, mcdo...@aries.scs.uiuc.edu (Doug McDonald) says:
>>the system MUST, if this is a legal hosted C compiler, guarantee that
>>sufficient memory exists at the time of the call (either real memory
>>and/or virtual memory on disk) to fulfill the request, and must continue
>>to guarantee that this memory is assigned to the program.
>I would like to see the citation on this.
Look up "malloc" or "calloc" (your choice) in the index of the C standard.
>While I agree that it would be horrible to have a program stop dead in the
>middle of its execution due to insuffiecient memory, but I can some
>situations where what you specify may not always be possible. First of all,
>how does a compiler "continue to guarantee that this memory is assigned to
>the program"?
If the operating system doesn't supply an obvious way, then the library
must find a less obvious way. For example, malloc could store all zeroes
in the allocated memory just like calloc, even though it doesn't have to.
Or if the OS might steal those pages, then malloc and calloc can allocate
more than the required amount of memory, and stuff some non-zero value
into bytes before and after the chunk that they return to the C program.
Or if there really is no way, then malloc/calloc can always return null
pointers, saying that there is no available memory.
Or if even that is impossible, then there is no hosted C implementation
for that operating system, period.
>Second, what if there is no way for a program running under a particular OS
>to know exactly how much memory can actually be used.
Then you're running in the real world, I guess.
>Third, what if you *want* to allocate 100MB for a sparse array which may
>only be written to 5 times. If it had to be available before being
>allocated, malloc(100000000) would always fail (on most machines), but
>otherwise you could have hundreds of these arays allocated if you needed it.
This question seems to contradict itself. If you have a sparse array
containing less than 100MB of actual data, then you don't use a C array
and you don't allocate 100MB. If you have a sparse array that could
potentially occupy terabytes but averages around 100MB, then you need
100MB and it will fail on most machines.
--
<< If this were the company's opinion, I would not be allowed to post it. >>
segmentation fault (california dumped)
Christopher R. Volpe
>
>In a previous article, mcdo...@aries.scs.uiuc.edu (Doug McDonald) says:
>
>>In article <31jo0p$12...@tornews.torolab.ibm.com> da...@neutron.torolab.ibm.com (Dave Mooney) writes:
>>
>>>Kevin D. Quitt <k...@prof.jpl.nasa.gov> wrote:
>>>> On some systems, malloc may not actually allocate the memory for you
>>>> (athough it does map the memory into your address space). On such a
>>>> system, you are assigned physical memory only when you try to access it
>>>> (this can help minimize the amount of memory in use, and therefore
>>>> reduce paging memory to disk).
>>
>>
>>This is true, BUT with an important caveat: the system MUST, if this is
>>a legal hosted C compiler, guarantee that sufficient memory exists
>>at the time of the call (either real memory and/or virtual memory on disk)
>>to fulfill the request, and must continue to guarantee that this memory
>>is assigned to the program. In other words, a system that simply assumes
>I would like to see the citation on this.
I believe the Standard is clear enough about what the behavior of such
programs must be.
>While I agree that it would be horrible to have a program stop dead in the
>middle of its execution due to insuffiecient memory, but I can some
>situations where what you specify may not always be possible. First of all,
>how does a compiler "continue to guarantee that this memory is assigned to
>the program"?
Well, it can attempt to "lock on" to the memory by writing to all the
locations before malloc returns. If this cannot be done, then it may simply be
the case that C is not fully implementable on such a machine.
>
>Second, what if there is no way for a program running under a particular OS
>to know exactly how much memory can actually be used.
The program shouldn't have to know. It should make a malloc request, and if
the request succeeds, the storage should belong to the process. Period.
>Say for example, all
>available memory is used dynamically for disk cacheing, and when a memory
>page that is used for the cache but was allocated to a user program gets
>written to, it is zeroed and actually given to the program. In that case,
>there may never be more than 8K free. Should all mallocs over 8K fail?
I don't understand. Is there a total of 8K of virtual storage on the system?
If so, then yes, all mallocs over 8K should fail. There is no excuse for an OS
to "take back" storage that it has already given a process, just so that it
can do disk cacheing.
>There may be no way to tell how much memory is available until a write
>fails. Admittedly, there should be some way to avoid that.
Shouldn't be a tough problem, since various Unix implementations have been
doing the Right Thing for years now.
>
>Third, what if you *want* to allocate 100MB for a sparse array which may
>only be written to 5 times. If it had to be available before being
>allocated, malloc(100000000) would always fail (on most machines), but
>otherwise you could have hundreds of these arays allocated if you needed
>it.
Then provide an extension: a new library function, called spalloc(), that
allocates storage for sparse arrays and document it such that the user known
that if he uses it, he's taking his chances. Don't bastardize the behavior of
all strictly conforming programs with a broken malloc implementation.
Lawrence Kirby
aj...@po.CWRU.Edu "Adrienne J. Schaffer" writes:
>>This is true, BUT with an important caveat: the system MUST, if this is
>>a legal hosted C compiler, guarantee that sufficient memory exists
>>at the time of the call (either real memory and/or virtual memory on disk)
>>to fulfill the request, and must continue to guarantee that this memory
>>is assigned to the program. In other words, a system that simply assumes
>I would like to see the citation on this.
Consider this:
If I write a strictly conforming program which uses malloc (which seems
possible) and avoids things like rand, floating point precision problems
and anything that depends on variable input (*) then the action of my
program is precisely defined (I probably don't have to apply all of these
restrictions but they simplify analysis or at least clarify). If when my
program runs it fails to produce that action then the implementation is
not conforming. This includes having the program mysteriously terminate
in mid execution because the OS decided it ran out of memory.
* - library calls (including malloc) are of course permitted to return
failure codes (which I guess is a form of variable input). In such a case
my code is well defined to terminate immediately. What is not acceptable
is the code terminating without receiving an error code.
>While I agree that it would be horrible to have a program stop dead in the
>middle of its execution due to insuffiecient memory, but I can some
>situations where what you specify may not always be possible. First of all,
>how does a compiler "continue to guarantee that this memory is assigned to
>the program"?
It is the implementation which must be conforming, not just the compiler.
This includes the OS. Normally the compiler can place a veneer over the
OS to create a conforming implementation, but there is no guarantee that
this is possible for every OS.
>Second, what if there is no way for a program running under a particular OS
>to know exactly how much memory can actually be used. Say for example, all
>available memory is used dynamically for disk cacheing, and when a memory
>page that is used for the cache but was allocated to a user program gets
>written to, it is zeroed and actually given to the program. In that case,
>there may never be more than 8K free. Should all mallocs over 8K fail?
>There may be no way to tell how much memory is available until a write
>fails. Admittedly, there should be some way to avoid that.
The suggestion is that all memory used by a program must be reserved
for use by that program. Of course this need only be virtual memory but the
OS must guarantee that if a program legally accesses a memory location is has
legitimately reserved it must always succeed in doing so. Some VM systems
allow this, others do not and hence cannot support conforming C
implementations. This doesn't mean they aren't useful however.
>Third, what if you *want* to allocate 100MB for a sparse array which may
>only be written to 5 times. If it had to be available before being
>allocated, malloc(100000000) would always fail (on most machines), but
>otherwise you could have hundreds of these arays allocated if you needed
>it.
C provides no support for sparse arrays and no guarantee that you can
malloc such a large amount of memory. I see no problem if an implementation
provides such an extension - so long as it also always succeeds in running
my (or really any) strictly conforming program.
This is how I understand thing to be and if it is inaccurate I'll
certainly learn something in the process! :-) It begs questions like
what about when I interrupt my strictly conforming program from the console?
--
-----------------------------------------
Lawrence Kirby | fr...@genesis.demon.co.uk
Wilts, England | 7073...@compuserve.com
-----------------------------------------
Jonathan de Boyne Pollard
>certainly learn something in the process! :-) It begs questions like
>what about when I interrupt my strictly conforming program from the console?
Surely that is the appropriate time for you to put down your copy of
the ISO C standard, and pick up your copy of the POSIX.1 standard ?
¯ JdeBP ®
Gordon Burditt
>equally fast (one where the OS zeroes all memory before giving to a
>task), but not one where malloc is slower.
Remember that memory obtained by malloc() is not necessarily obtained
fresh from the OS. It might be from something recently free()d.
Systems that get memory with sbrk() can't release a good portion of
freed memory because there is still memory in use between it and the
current "break". Systems that can allocate and free memory on a
page-by-page basis probably have a reasonably large page size (e.g.
>= 1k), so short strings and small structures likely don't need a new
page from the OS. calloc() will need to clear re-used memory
(unless free() does it).
Gordon L. Burditt
sneaky.lonestar.org!gordon
Lawrence Kirby
How will POSIX tell me if a C implemenation conforms to the ANSI/ISO C
standard or not?
Stan Friesen
|> In article <32a598$r...@silver.jba.co.uk>
|> @jba.co.uk:JdeBP@osmium "Jonathan de Boyne Pollard" writes:
|>
|> >>what about when I interrupt my strictly conforming program from the console?
|> >
|> >Surely that is the appropriate time for you to put down your copy of
|> >the ISO C standard, and pick up your copy of the POSIX.1 standard ?
|>
|> How will POSIX tell me if a C implemenation conforms to the ANSI/ISO C
|> standard or not?
What he was trying to say was that your question falls outside of the
scope of the ISO C Standard. That is, no behavior at that point will
be such as to make your implementation non-conforming (anything is
allowed as far as ISO C is concerned).
He is suggesting that the correct standard to determine what is required
in that case is your OS standard.
--
s...@elsegundoca.ncr.com sar...@netcom.com
The peace of God be with you.
Adrienne J. Schaffer
In a previous article, vo...@bart.crd.ge.com (Christopher R. Volpe) says:
>>>>> On some systems, malloc may not actually allocate the memory for you
>>>>> (athough it does map the memory into your address space). On such a
>>>>> system, you are assigned physical memory only when you try to access it
>>>This is true, BUT with an important caveat: the system MUST, if this is
>>>a legal hosted C compiler, guarantee that sufficient memory exists
>>>at the time of the call (either real memory and/or virtual memory on disk)
>>>to fulfill the request, and must continue to guarantee that this memory
>>>is assigned to the program. In other words, a system that simply assumes
>>I would like to see the citation on this.
>I believe the Standard is clear enough about what the behavior of such
>programs must be.
*I* don't have the standard. People are always so anxious to cite the
Standard, why can't somebody do it for this situation.
>>situations where what you specify may not always be possible. First of all,
>>how does a compiler "continue to guarantee that this memory is assigned to
>>the program"?
>
>Well, it can attempt to "lock on" to the memory by writing to all the
>locations before malloc returns. If this cannot be done, then it may simply be
>the case that C is not fully implementable on such a machine.
Writing to all locations before malloc returns would not only make it slow,
but also defeat the purpose of not allocating memory pages until written to.
>>Say for example, all
>>available memory is used dynamically for disk cacheing, and when a memory
>>page that is used for the cache but was allocated to a user program gets
>>written to, it is zeroed and actually given to the program. In that case,
>>there may never be more than 8K free. Should all mallocs over 8K fail?
>
>I don't understand. Is there a total of 8K of virtual storage on the system?
>If so, then yes, all mallocs over 8K should fail. There is no excuse for an OS
>to "take back" storage that it has already given a process, just so that it
>can do disk cacheing.
Say there is no virtual storage (or none left). In that case, all available
memory is RAM (say 10MB of which are free). I find it completely reasonable
for an OS to want to use all available memory for disk/network cacheing. A
process could then allocate 10MB of memory, and still have whatever's not
used by the process used for disk cacheing. Once the process actually uses
part of that allocated memory, it gets it for good until freed. However, if
the process never uses part of it, the OS still gets to have part of it for
cache. By that same token, many processes may allocate large chunks of
memory which go mostly unused. If all those processes actually used all of
their memory, I have no idea what would happen.
Such a system is good for overall system performance because it allows all
unwritten blocks of memory to be used for caches of various sorts, and
prevents the overhead of assigning memory block which never get used.
The memory model I just presented is *my understanding* of how most decent
operating systems for PCs (e.g. WinNT, OS/2, Linux) operate. If I am wrong,
I would like to know, as I have been wondering about it for a long time.
Norman Diamond
>In a previous article, vo...@bart.crd.ge.com (Christopher R. Volpe) says:
>>>>>> On some systems, malloc may not actually allocate the memory for you
>>>>>> (athough it does map the memory into your address space). On such a
>>>>>> system, you are assigned physical memory only when you try to access it
>>>>This is true, BUT with an important caveat: the system MUST, if this is
>>>>a legal hosted C compiler, guarantee that sufficient memory exists
>>>>at the time of the call (either real memory and/or virtual memory on disk)
>>>>to fulfill the request, and must continue to guarantee that this memory
>>>>is assigned to the program. In other words, a system that simply assumes
>*I* don't have the standard. People are always so anxious to cite the
>Standard, why can't somebody do it for this situation.
ANSI Classic section 4.10.3, page 155 lines 17 to 20: "If the space cannot
be allocated, a null pointer is returned."
*You* should think about seeing if your university has a library.
>>>situations where what you specify may not always be possible. First of all,
>>>how does a compiler "continue to guarantee that this memory is assigned to
>>>the program"?
>>Well, it can attempt to "lock on" to the memory by writing to all the
>>locations before malloc returns. If this cannot be done, then it may simply be
>>the case that C is not fully implementable on such a machine.
>Writing to all locations before malloc returns would not only make it slow,
Yup. If you don't like it, don't use C. Or don't use that OS.
>but also defeat the purpose of not allocating memory pages until written to.
Yup. If you don't like it, don't use C. Or don't use that OS.
>Say there is no virtual storage (or none left).
Then malloc(), calloc(), and (if an increased size is requested) realloc()
must return a null pointer.
>In that case, all available memory is RAM (say 10MB of which are free).
Huh? If the program's addresses are virtual, and there's no virtual space
left, it doesn't matter whether the program's pages are swapped into RAM
or not. The program cannot address them. This has nothing to do with C
or with any operating system.
>I find it completely reasonable
>for an OS to want to use all available memory for disk/network cacheing.
And guess what, many do. But when a program gets a page fault, the OS
has to swap out some other page and swap in the requested one. Of course,
it doesn't have to give any priority or anything. If the program doesn't
get its next machine instruction executed for 3 years, too bad. If the
program gets killed, the pseudo-C implementation is not conforming C.
>A process could then allocate 10MB of memory,
If this is virtual space and the OS has 10MB of swap space for it, no
problem. If this is real space, then the process has those 10MB of RAM.
If this is real space and the process doesn't have those 10MB of RAM,
then you're talking about something that resembles C but is not standard C.
If you don't like standard C, don't use standard C. 'Bye.
But maybe you should start by learning what virtual memory means.
The C standard doesn't care whether program addresses are virtual
or not, but your attempted argument does, one way or the other.
Christopher R. Volpe
>
>In a previous article, vo...@bart.crd.ge.com (Christopher R. Volpe) says:
>
>>>>>> On some systems, malloc may not actually allocate the memory for you
>>>>>> (athough it does map the memory into your address space). On such a
>>>>>> system, you are assigned physical memory only when you try to access it
>>>>This is true, BUT with an important caveat: the system MUST, if this is
>>>>a legal hosted C compiler, guarantee that sufficient memory exists
>>>>at the time of the call (either real memory and/or virtual memory on disk)
>>>>to fulfill the request, and must continue to guarantee that this memory
>>>>is assigned to the program. In other words, a system that simply assumes
>>>I would like to see the citation on this.
>>I believe the Standard is clear enough about what the behavior of such
>>programs must be.
>*I* don't have the standard. People are always so anxious to cite the
>Standard, why can't somebody do it for this situation.
Ok, no problem: ANSI Classic section 4.10.3 (Memory management functions):
The pointer returned if the allocation succeeds is suitable
aligned so that itmay be assigned to a pointer to any type of
object and THEN USED TO ACCESS SUCH AN OBJECT OR AN ARRAY OF
OBJECTS IN THE SPACE ALLOCATED (until the space is explicitly
freed or reallocated). [...] If the space cannot be allocated,
a NULL pointer is returned.
[emphasis mine]
I think that's pretty unambiguous.
>
>>>situations where what you specify may not always be possible. First of all,
>>>how does a compiler "continue to guarantee that this memory is assigned to
>>>the program"?
>>
>>Well, it can attempt to "lock on" to the memory by writing to all the
>>locations before malloc returns. If this cannot be done, then it may simply be
>>the case that C is not fully implementable on such a machine.
>
>Writing to all locations before malloc returns would not only make it slow,
>but also defeat the purpose of not allocating memory pages until written to.
BUT, it would make the implementation conforming. Sometimes brain-dead
operating systems force implementations to be slow in order to be conforming.
>
>>>Say for example, all
>>>available memory is used dynamically for disk cacheing, and when a memory
>>>page that is used for the cache but was allocated to a user program gets
>>>written to, it is zeroed and actually given to the program. In that case,
>>>there may never be more than 8K free. Should all mallocs over 8K fail?
>>
>>I don't understand. Is there a total of 8K of virtual storage on the system?
>>If so, then yes, all mallocs over 8K should fail. There is no excuse for an OS
>>to "take back" storage that it has already given a process, just so that it
>>can do disk cacheing.
>Say there is no virtual storage (or none left). In that case, all available
>memory is RAM (say 10MB of which are free). I find it completely reasonable
>for an OS to want to use all available memory for disk/network cacheing. A
Then don't allocate it to the process.
>process could then allocate 10MB of memory, and still have whatever's not
>used by the process used for disk cacheing. Once the process actually uses
>part of that allocated memory, it gets it for good until freed. However, if
>the process never uses part of it, the OS still gets to have part of it for
>cache.
Fine. If the OS can determine that the process isn't using all that it has
allocated, the OS is free to use the unused portion so long as doing so does
not affect the process to whom the storage belongs.
> By that same token, many processes may allocate large chunks of
>memory which go mostly unused. If all those processes actually used all of
>their memory, I have no idea what would happen.
Well, if the operating system weren't braindead, all the processes would
behave the way they are supposed to.
>Such a system is good for overall system performance because it allows all
>unwritten blocks of memory to be used for caches of various sorts, and
>prevents the overhead of assigning memory block which never get used.
It is not the job of the OS to second-guess an application's intent to
eventually use resources which it has requested, and to grant such requests on
the assumption that they will go unused.
>
>The memory model I just presented is *my understanding* of how most decent
>operating systems for PCs (e.g. WinNT, OS/2, Linux) operate. If I am wrong,
>I would like to know, as I have been wondering about it for a long time.
I don't know how PC operating systems operate. I just know that if a C
implementation is built on top of such an OS, and does not take steps to
ensure the things that the OS doesn't ensure, then the implementation is
broken. Period.
Jonathan de Boyne Pollard
>In article <33ahl4$b...@usenet.INS.CWRU.Edu>, aj...@po.CWRU.Edu (Adrienne J. Schaffer) writes:
>>Writing to all locations before malloc returns would not only make it slow,
>>but also defeat the purpose of not allocating memory pages until written to.
>
>BUT, it would make the implementation conforming. Sometimes brain-dead
>operating systems force implementations to be slow in order to be conforming.
>
>>operating systems for PCs (e.g. WinNT, OS/2, Linux) operate. If I am wrong,
>>I would like to know, as I have been wondering about it for a long time.
>
>I don't know how PC operating systems operate. I just know that if a C
>implementation is built on top of such an OS, and does not take steps to
>ensure the things that the OS doesn't ensure, then the implementation is
>broken. Period.
Maybe I can help in this discussion.
Taking OS/2 as an example, memory allocation and use are two
different things. "Allocation" simply reserves a portion of the
address space in a process, creating appropriate entries in the page
tables and suchlike. "Commitment" is a second stage to the process,
and actually creates storage space for physical pages of memory
(in memory and the swapfile).
The idea being that this scheme allows huge sparse memory objects to
be created and used. Accessing a non allocated portion of memory
generates an exception, as does accessing an allocated portion of
memory that has not yet had storage space committed for it.
So an ISO compliant implementation of malloc() on an OS/2 system
will do two things. It will allocate a sparse memory object at
runtime startup, which will be the heap. Each call to malloc() will
then actually perform the commit of the required amount of memory
from the heap. If the commit fails (one possible reason for which
is that backing storage could not be allocated), then malloc()
returns NULL. free() can then decommit the memory, releasing the
backing storage for use by other processes.
¯ JdeBP ®
Jim Balter
I'm not familiar with OS/2, but this isn't the way malloc works in systems I'm
familiar with, and I suspect that it isn't really how OS/2 works. There is no
need to allocate a heap at startup; the "heap" is simply the address space
available to the process. malloc performs allocation; that is, it allocates a
section of the address space and returns the address of that section to the
caller; it need not examine or access the memory. Failure to allocate due to
exhaustion of available address space (possibly due to fragmentation) results
in a NULL return. *Access* to the allocated space by the calling program
causes "commitment"; it results in a trap, to the system, which then allocates
physical memory to correspond to the virtual memory (process address space).
In order to perform this allocation of physical memory to be committed to the
process, the system may need to suspend the process while paging or swapping
out memory belonging to other processes (or even other memory belonging to the
same process). A system that allocates virtual address space but is later
unable to commit physical memory to satisfy accesses, and therefore terminates
the process or sends it some "can't allocate memory" signal (apparently some
system does this) is nonconforming.
Perhaps OS/2 commits during malloc, as you suggest, rather than as a result of
an actual access of the memory by the caller, but if so it is foolishly
inefficient (that's the sort of inefficiency of touching evry location in the
malloced space, discussed earlier in this thread). Since commitment upon
access is basic textbook stuff in use since at least the 60's, I suspect that
instead you have simply misdescribed the OS/2 implementation.
Of course, one needn't really know anything about operating system
implementation in order to use and understand the ANSI C malloc abstraction.
If malloc returns an object (not NULL), you can access the object. An
implementation that doesn't honor the access, regardless of the best
"implementation-on-the-brain" excuses, is non-conforming.
--
<J Q B>
Stan Friesen
|> I'm not familiar with OS/2, but this isn't the way malloc works in systems I'm
|> familiar with, and I suspect that it isn't really how OS/2 works. There is no
|> need to allocate a heap at startup; the "heap" is simply the address space
|> *Access* to the allocated space by the calling program
|> causes "commitment"; it results in a trap, to the system, which then allocates
|> physical memory to correspond to the virtual memory (process address space).
|> In order to perform this allocation of physical memory to be committed to the
|> process, the system may need to suspend the process while paging or swapping
|> A system that allocates virtual address space but is later
|> unable to commit physical memory to satisfy accesses, and therefore terminates
|> the process or sends it some "can't allocate memory" signal (apparently some
|> system does this) is nonconforming.
Correct, so if the OS might do this if left to itself, it is up to the
C implementation to take steps to prevent this from happening.
|>
|> Perhaps OS/2 commits during malloc, as you suggest, rather than as a result of
|> an actual access of the memory by the caller, but if so it is foolishly
|> inefficient (that's the sort of inefficiency of touching evry location in the
|> malloced space, discussed earlier in this thread). Since commitment upon
|> access is basic textbook stuff in use since at least the 60's, I suspect that
|> instead you have simply misdescribed the OS/2 implementation.
Unless OS/2 has the property (mentioned above) that it might later, at
commitment time, decide it cannot honor the request. Then the C standard
*requires* that malloc() commit the memory, so that it can guarentee the
memory can be successfully accessed later.
In short, this may be required for the implementation to be conforming,
in which case it must be done regardless of the inefficiency.
Also note, the "commitment" here is only of virtual address space,
that is it is *not* necessarily the case that the actual main memory
is "allocated" by the commit process, only that sufficient backing
store be available so that the OS can guarentee the virtual space
can be supported. The actual logical page may be swapped in and out
subsequently, on demand, like any other memory page.
Jonathan de Boyne Pollard
>I'm not familiar with OS/2, but this isn't the way malloc works in systems I'm
>familiar with, and I suspect that it isn't really how OS/2 works.
Yes it is.
> There is no
>need to allocate a heap at startup; the "heap" is simply the address space
>available to the process. malloc performs allocation; that is, it allocates a
>section of the address space and returns the address of that section to the
>caller; it need not examine or access the memory.
<Sigh> I had hoped to clear up the confusion and misinformation (and
all the irrelevant discussion about disk caches and pages being
stolen from processes -- which doesn't happen, BTW) earlier in this
thread by describing how OS/2's memory management actually worked
and how malloc() has to be implemented on top of it.
Read what I wrote again. Allocating address ranges, and committing
memory for them are two separate things to OS/2, and a malloc()
implementation needs to do both. The address space of the heap as a
whole needs to be allocated on or before the first call to malloc(),
but the storage for pages in that address space doesn't need to be
committed until they are actually required by malloc().
OS/2 doesn't have UNIX-style monolithic memory management. OS/2
supports sparse memory objects. i.e. the whole of the addressable
range of the object need not be backed by virtual memory pages. The
C run-time heap used by malloc() is implemented as one such object.
The discussion up until now has been that malloc() requires that the
O/S provide memory that can be accessed without incurring an out of
memory or an addressing exception. In OS/2, memory needs to be both
allocated *and* committed for this to be true.
> *Access* to the allocated space by the calling program
>causes "commitment"; it results in a trap, to the system, which then allocates
>physical memory to correspond to the virtual memory (process address space).
This can be implemented under OS/2 by using guard pages, but the
result is a non-conforming implementation of malloc(), because the
system may run out of memory whilst trying to auto-commit pages for
that object at access time (bringing up the infamous "the swap file
is full" message).
If the system doesn't have enough memory, this *must* be indicated
in the return from malloc(), not when the object is accessed.
To make it trebly clear, we are not talking about the demand paging
mechanism here. That is invisible as far as the application
programmer is concerned. The storage for committed pages is virtual
storage.
>Perhaps OS/2 commits during malloc, as you suggest,
The OS/2 C compilers where I've had access to the runtime library
source do exactly this.
> but if so it is foolishly
>inefficient (that's the sort of inefficiency of touching evry location in the
>malloced space, discussed earlier in this thread).
Wrong. Don't make assumptions. Read the documentation instead.
OS/2 doesn't access every page as it commits it. It simply checks
that it has enough virtual memory for the operation and updates the
page table entries marking them as "demand zero fill".
¯ JdeBP ®
Andrew Phillips
: In article <33ahl4$b...@usenet.INS.CWRU.Edu>, aj...@po.CWRU.Edu (Adrienne J. Schaffer) writes:
: [Re how does a C program make sure that even if malloc() succeeds that the
: OS will not have no virtual memory left when it goes to use the memory ]
: >The memory model I just presented is *my understanding* of how most decent
: >operating systems for PCs (e.g. WinNT, OS/2, Linux) operate. If I am wrong,
: >I would like to know, as I have been wondering about it for a long time.
: I don't know how PC operating systems operate. I just know that if a C
: implementation is built on top of such an OS, and does not take steps to
: ensure the things that the OS doesn't ensure, then the implementation is
: broken. Period.
Such as C compilers on all(?) versions of UNIX?
kernel: Out of memory and swap: killing processes.
--
Andrew Phillips (News/Sys Admin) and...@teslab.lab.oz.au +61 2 287 6551
--------------------------------
Just a SPOKE, not a SPOKESPERSON
Jeff Dunlop
Jjcu> The OS/2 C compilers where I've had access to the runtime library
Jjcu> source do exactly this.
Which compilers? Under what conditions? DosAllocMem allows memory to be
reserved and committed, or just reserved.
Jjcu> Wrong. Don't make assumptions. Read the documentation instead.
Jjcu> OS/2 doesn't access every page as it commits it. It simply checks
Jjcu> that it has enough virtual memory for the operation and updates the
Jjcu> page table entries marking them as "demand zero fill".
It appears that a runtime library could do this, but OS/2 certainly does not do
it by default.
I refer you to the following test program:
#include <stdio.h>
int main(void)
{
return printf("%x\n", malloc(75000000L));
}
When I compiled this program with IBM C Set/2 and ran it with 43MB free on my
swap drive, the output was non-NULL. The reason is that MEMMAN= (found in
Config.sys) defaults to SWAP, but can also be set to NOSWAP and COMMIT. The
latter two values could cause a malloc to fail, but malloc doesn't fail with
MEMMAN=SWAP (up to 512MB) regardless of the state of the swap disk.
Jeff
--
: Fidonet: Jeff Dunlop 1:203/16 .. speaking for only myself.
: Internet: Jeff....@db-soft.wmeonlin.sacbbx.com
Lawrence Kirby
and...@teslab.lab.oz.au "Andrew Phillips" writes:
>Such as C compilers on all(?) versions of UNIX?
>
>kernel: Out of memory and swap: killing processes.
Some (arguably broken) versions of Unix do this. Others simply don't
allow a process to allocate if there is no more memory or swap space
available. Therefore once a process owns an address space it is always
guaranteed to be able to use it without dying mysteriously.
Christopher R. Volpe
>Christopher R. Volpe (vo...@bart.crd.ge.com) wrote:
>: In article <33ahl4$b...@usenet.INS.CWRU.Edu>, aj...@po.CWRU.Edu (Adrienne J. Schaffer) writes:
>: [Re how does a C program make sure that even if malloc() succeeds that the
>: OS will not have no virtual memory left when it goes to use the memory ]
>
>: >The memory model I just presented is *my understanding* of how most decent
>: >operating systems for PCs (e.g. WinNT, OS/2, Linux) operate. If I am wrong,
>: >I would like to know, as I have been wondering about it for a long time.
>
>: I don't know how PC operating systems operate. I just know that if a C
>: implementation is built on top of such an OS, and does not take steps to
>: ensure the things that the OS doesn't ensure, then the implementation is
>: broken. Period.
>
>Such as C compilers on all(?) versions of UNIX?
No, because not all versions of Unix are brain dead like this. But the answer
is "yes" for any versions of Unix that behave this way.
>
>kernel: Out of memory and swap: killing processes.
And you get this message on all versions of unix? Interesting. I've never
gottten it. But I've gotten plenty of "malloc returns NULL".
Jonathan de Boyne Pollard
> Jjcu> The OS/2 C compilers where I've had access to the runtime library
> Jjcu> source do exactly this.
>
>Which compilers? Under what conditions? DosAllocMem allows memory to be
>reserved and committed, or just reserved.
Yup, but a conforming malloc() must commit the memory (either using
DosAllocMem, which is possibly not the optimal way of implementing
the C run-time heap, or using DosSetMem).
>I refer you to the following test program:
>
>#include <stdio.h>
>int main(void)
>{
> return printf("%x\n", malloc(75000000L));
>}
>
>When I compiled this program with IBM C Set/2 and ran it with 43MB free on my
>swap drive, the output was non-NULL.
Then CSet/2 is (by previous argument) broken. Access to that memory
shouldn't cause the process to be terminated. Is anyone from IBM
lurking ?
BTW, as a generic test program, that's a trifle flawed. You didn't
include <stdlib.h> or a declaration of malloc() and you didn't print
your pointer with %p. (-:
¯ JdeBP ®
Jonathan de Boyne Pollard
> Jjcu> source do exactly this.
>
>Which compilers? Under what conditions? DosAllocMem allows memory to be
>reserved and committed, or just reserved.
Yup, but a conforming malloc() must commit the memory (either using
DosAllocMem, which is possibly not the optimal way of implementing
the C run-time heap, or using DosSetMem).
>I refer you to the following test program:
>
>#include <stdio.h>
>int main(void)
>{
> return printf("%x\n", malloc(75000000L));
>}
>
>When I compiled this program with IBM C Set/2 and ran it with 43MB free on my
>swap drive, the output was non-NULL.
Then CSet/2 is (by previous argument) broken. Access to that memory
Norman Diamond
>In <2f6_940...@wmeonlin.sacbbx.com>, Jeff....@db-soft.wmeonlin.sacbbx.com (Jeff Dunlop) writes:
>>#include <stdio.h>
>>int main(void) {
>> return printf("%x\n", malloc(75000000L));
>>}
>>When I compiled this program with IBM C Set/2 and ran it with 43MB free on my
>>swap drive, the output was non-NULL.
>Then CSet/2 is (by previous argument) broken. Access to that memory
>shouldn't cause the process to be terminated. Is anyone from IBM lurking ?
This time, that's a bit of an exaggeration. The manner in which Mr. Dunlop
invoked CSet/2 is a non-conforming implementation. If someone from IBM is
lurking, perhaps they can point out that a few days ago, someone told us some
OS/2 options that can defeat this feature. So an invocation of CSet/2 as a
conforming C implementation requires changing an OS/2 option.
The standard does not prohibit non-conforming implementations from being
very similar to conforming implementations. And in cases where such exist,
the standard does not require the conforming version to be easier to invoke
than the non-conforming version. If I were an implementor, I would try to
make a conforming version available as an option, but not as a default.