What's the substitution of brk system call in POSIX standard?
Terry
excluded from the POSIX.1 standard". Why is that and what's the
substitution?
Thanks!
Barry Margolin
malloc() and mmap(), I guess. brk() depends on a process memory layout
that's particular to traditional Unix implementations, but may not be
used in other environments and would be difficult to emulate. Programs
that need to call these directly are extremely rare, and it's hard to
imagine them being portable to OSes with different memory layouts.
--
Barry Margolin, bar...@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***
Terry
> In article <g102db$gqo$1...@aioe.org>, Terry <con...@gmail.com> wrote:
>
>> I read from the brk manual page that brk() and sbrk() "are deliberately
>> excluded from the POSIX.1 standard". Why is that and what's the
>> substitution?
>> Thanks!
>
> malloc() and mmap(), I guess. brk() depends on a process memory layout
> that's particular to traditional Unix implementations, but may not be
> used in other environments and would be difficult to emulate. Programs
> that need to call these directly are extremely rare, and it's hard to
> imagine them being portable to OSes with different memory layouts.
>
malloc() is a library routine and is originally implemented with brk()
system call. K&R 's book contains a sample implementation of it. Now
that brk() is removed, if its substitution is mmap(), then, all memory
allocations have to have something to do with virtual memory since this
is natural implementation of mmap(). This seems to have totally changed
the semantic of memory allocation by malloc() or brk() in that the
allocated memory is not instantly physically ready unless it's accessed.
Although mlock() can be used to reserve the space in RAM after mmap(),
it another system call. Then, brk()=mmap()+mlock()?
Ian Collins
How so? From the program's point of view, it is. The mechanism used to
make the memory available is an implementation detail.
All (user space) memory mappings on Unix systems use virtual memory.
--
Ian Collins.
Barry Margolin
> Barry Margolin wrote:
> > In article <g102db$gqo$1...@aioe.org>, Terry <con...@gmail.com> wrote:
> >
> >> I read from the brk manual page that brk() and sbrk() "are deliberately
> >> excluded from the POSIX.1 standard". Why is that and what's the
> >> substitution?
> >> Thanks!
> >
> > malloc() and mmap(), I guess. brk() depends on a process memory layout
> > that's particular to traditional Unix implementations, but may not be
> > used in other environments and would be difficult to emulate. Programs
> > that need to call these directly are extremely rare, and it's hard to
> > imagine them being portable to OSes with different memory layouts.
> >
>
> malloc() is a library routine and is originally implemented with brk()
> system call. K&R 's book contains a sample implementation of it. Now
But POSIX assumes you have a C library already, so malloc() is
available. I don't think there's any expectation that everything in the
C library can be written portably using only POSIX system calls.
> that brk() is removed, if its substitution is mmap(), then, all memory
> allocations have to have something to do with virtual memory since this
> is natural implementation of mmap(). This seems to have totally changed
> the semantic of memory allocation by malloc() or brk() in that the
> allocated memory is not instantly physically ready unless it's accessed.
> Although mlock() can be used to reserve the space in RAM after mmap(),
> it another system call. Then, brk()=mmap()+mlock()?
brk() uses virtual memory, too. All it does is change the size of the
data segment, it doesn't make any changes to physical memory layout.
phil-new...@ipal.net
| In article <g102db$gqo$1...@aioe.org>, Terry <con...@gmail.com> wrote:
|
|> I read from the brk manual page that brk() and sbrk() "are deliberately
|> excluded from the POSIX.1 standard". Why is that and what's the
|> substitution?
|> Thanks!
|
| malloc() and mmap(), I guess. brk() depends on a process memory layout
| that's particular to traditional Unix implementations, but may not be
| used in other environments and would be difficult to emulate. Programs
| that need to call these directly are extremely rare, and it's hard to
| imagine them being portable to OSes with different memory layouts.
IOW, if you want to call brk() or sbrk() you need to know some things about
the particular system that may be different enough to require different code
on some other system.
I've used mmap() for low level allocation of a few pages of memory in lieu
of using brk(). It seems to work fine.
--
|WARNING: Due to extreme spam, googlegroups.com is blocked. Due to ignorance |
| by the abuse department, bellsouth.net is blocked. If you post to |
| Usenet from these places, find another Usenet provider ASAP. |
| Phil Howard KA9WGN (email for humans: first name in lower case at ipal.net) |
Rainer Weikusat
> In article <g102db$gqo$1...@aioe.org>, Terry <con...@gmail.com> wrote:
>> I read from the brk manual page that brk() and sbrk() "are deliberately
>> excluded from the POSIX.1 standard". Why is that and what's the
>> substitution?
>> Thanks!
>
> malloc() and mmap(), I guess. brk() depends on a process memory layout
> that's particular to traditional Unix implementations, but may not be
> used in other environments and would be difficult to emulate. Programs
> that need to call these directly are extremely rare,
All malloc implementations for Linux-based systems I have used so far
are based on brk/ sbrk. Hence, programs not calling either of the two
are extremly rare, not the other way round. Wrapping some amount of
code intended to solve the 'lost' research problem of general purpose
memory allocation 'good enough' around this does not change it.
I have quite a bit of code without any calls to malloc, except insofar
standard library calls do so on their own. When I really need
'dynamic' memory allocation, ie the ability to repurpose an already
allocated chunk of memory, I usually use slab-style allocation based
on mmap'ing pages, otherwise, simple object caching based on a
shared, mmap'ed heap. What is only temporarily needed can usually be
allocated on the stack and if a program needed n different objects of
type x at a certain time, it will need all n of them again in future,
and there is no reason to 'free' them in between. Where applicable,
the code uses sbrk directly, eg if a certain amount of memory needs to
be allocated during initialization and persists during the entire
lifetime of the process, but is not know at compile time.
Barry Margolin
Rainer Weikusat <rwei...@mssgmbh.com> wrote:
> Barry Margolin <bar...@alum.mit.edu> writes:
> > In article <g102db$gqo$1...@aioe.org>, Terry <con...@gmail.com> wrote:
> >> I read from the brk manual page that brk() and sbrk() "are deliberately
> >> excluded from the POSIX.1 standard". Why is that and what's the
> >> substitution?
> >> Thanks!
> >
> > malloc() and mmap(), I guess. brk() depends on a process memory layout
> > that's particular to traditional Unix implementations, but may not be
> > used in other environments and would be difficult to emulate. Programs
> > that need to call these directly are extremely rare,
>
> All malloc implementations for Linux-based systems I have used so far
> are based on brk/ sbrk. Hence, programs not calling either of the two
> are extremly rare, not the other way round.
I said "call these DIRECTLY". That qualifier is crucial to the point I
was making.
The point is that malloc() usually has to be written in an
implementation-dependent way, and brk/sbrk are the
implementation-specific calls that it uses. But portable applications
can just use malloc() and not have to worry about the low-level system
calls that are used. POSIX is about portability.
David Schwartz
> malloc() is a library routine and is originally implemented with brk()
> system call.
Yes, but it can be implemented any way at all, POSIX doesn't care.
> K&R 's book contains a sample implementation of it. Now
> that brk() is removed, if its substitution is mmap(), then, all memory
> allocations have to have something to do with virtual memory since this
> is natural implementation of mmap().
Umm, no, because 'malloc' is still there, and 'malloc' can allocate
memory any way it wants to. It can use 'brk' because there is no law
that says that standard library implementation has to be strictly
POSIX compliant. (And typically it won't be, since the standard
library is more like the POSIX implementation than it is like
application code.)
> This seems to have totally changed
> the semantic of memory allocation by malloc() or brk() in that the
> allocated memory is not instantly physically ready unless it's accessed.
I have no idea what you think this means. The implementation is free
to allocate physical memory, allocate virtual memory, or manipulate
tea leaves. So long as a strictly-POSIX-compliant application finds
that the memory works when 'mmap' or 'malloc' returns, POSIX is happy.
> Although mlock() can be used to reserve the space in RAM after mmap(),
> it another system call. Then, brk()=mmap()+mlock()?
What makes you think 'brk' reserves space in RAM? That would defeat
the whole point of virtual memory with memory allocated using
'malloc', if 'malloc' were implemented using 'brk'. With the exception
of 'mlock', these are all virtual memory operations.
DS
Rainer Weikusat
> In article <87wsloo...@fever.mssgmbh.com>,
> Rainer Weikusat <rwei...@mssgmbh.com> wrote:
>> Barry Margolin <bar...@alum.mit.edu> writes:
>> > In article <g102db$gqo$1...@aioe.org>, Terry <con...@gmail.com> wrote:
>> >> I read from the brk manual page that brk() and sbrk() "are deliberately
>> >> excluded from the POSIX.1 standard". Why is that and what's the
>> >> substitution?
>> >> Thanks!
>> >
>> > malloc() and mmap(), I guess. brk() depends on a process memory layout
>> > that's particular to traditional Unix implementations, but may not be
>> > used in other environments and would be difficult to emulate. Programs
>> > that need to call these directly are extremely rare,
>>
>> All malloc implementations for Linux-based systems I have used so far
>> are based on brk/ sbrk. Hence, programs not calling either of the two
>> are extremly rare, not the other way round.
>
> I said "call these DIRECTLY". That qualifier is crucial to the point I
> was making.
And omitting it was crucial to highlighting the point you are trying
to hide: All applications which call malloc end up calling sbrk/ brk,
except for 'unusual' circumstances. This means that using sbrk/ brk is
actually pretty portable, because every mainstream UNIX(*) and
UNIX(*)-like system (I know of) has always supported it and will most likely
always support it.
> The point is that malloc() usually has to be written in an
> implementation-dependent way, and brk/sbrk are the
> implementation-specific calls that it uses. But portable applications
> can just use malloc() and not have to worry about the low-level system
> calls that are used.
General purpose dynamic heap allocation is rarely needed and
better methods than 'malloc' are available where it is. Even better
'portably standardized' methods, namely, mmap.
Scott Lurndal
>Barry Margolin wrote:
>> In article <g102db$gqo$1...@aioe.org>, Terry <con...@gmail.com> wrote:
>>
>>> I read from the brk manual page that brk() and sbrk() "are deliberately
>>> excluded from the POSIX.1 standard". Why is that and what's the
>>> substitution?
>>> Thanks!
>>
>> malloc() and mmap(), I guess. brk() depends on a process memory layout
>> that's particular to traditional Unix implementations, but may not be
>> used in other environments and would be difficult to emulate. Programs
>> that need to call these directly are extremely rare, and it's hard to
>> imagine them being portable to OSes with different memory layouts.
>>
>
>malloc() is a library routine and is originally implemented with brk()
The idea of posix is to describe the interfaces an application can use,
not the implementation. An implementation is perfectly free to provide
sbrk and use it in the malloc implementation. Portable applications are not
allowed to use sbrk/brk, but must use malloc instead.
scott
phil-new...@ipal.net
A malloc() implementation for some operating system FOO-OS could make use of
FOO-OS specific system calls which might not involve anything that even looks
like brk() or sbrk() or mmap() or shmat().
FYI, I have used shmat() before as a means to get low level memory access
with certain semantics that I needed. See lines 310 and 311 of this code:
http://vrb.slashusr.org/source/vrb/src/lib/vrb/vrb_init.c
phil-new...@ipal.net
No. All applications which call malloc end up calling whatever system calls
that malloc implementation happens to use. Apparently the default malloc in
Glibc for Linux does call sbrk/brk (I won't bother verifying this because I
don't care nor should I need to care). Alternate malloc implementations that
work on Linux may well use mmap(). I can't even rule out them using shmat().
And malloc implementations on other POSIX compliant systems can use anything
the underlying kernel provides. POSIX is about what interfaces are exposed
to the application program, not about how things like libraries are implemented.
The real point is that whatever ends up being called by the implementation of
malloc is not what is considered by the application itself. What the application
calls is only what it is coded to call. The whole VM process that ends up doing
some ABI trap into the kernel to do kernel system calls is more than just the
application; it is the application and linked in support library code as well.
It is not the application that ends up calling sbrk/brk in the referenced case
involving Linux; it is the whole of the process, application plus library.
This is all terminology semantics, of course.
Lew Pitcher
> On Wed, 21 May 2008 12:35:22 +0800 Terry <con...@gmail.com> wrote:
> | Barry Margolin wrote:
> |> In article <g102db$gqo$1...@aioe.org>, Terry <con...@gmail.com> wrote:
> |>
> |>> I read from the brk manual page that brk() and sbrk() "are
> |>> deliberately excluded from the POSIX.1 standard". Why is that and
> |>> what's the substitution?
[snip]
> A malloc() implementation for some operating system FOO-OS could make use
> of FOO-OS specific system calls which might not involve anything that even
> looks like brk() or sbrk() or mmap() or shmat().
Like on OS/390 Unix System Services (the IBM MVS OS subsystem that was
certified as Unix95 by The Open Systems Group), the system-level
memory-acquisition syscall is "GETMAIN" (and the release syscall
is "FREEMAIN"). There is no brk() or sbrk() there, but it is still a
branded Unix system supporting the POSIX/SUS APIs.
--
Lew Pitcher
Master Codewright & JOAT-in-training | Registered Linux User #112576
http://pitcher.digitalfreehold.ca/ | GPG public key available by request
---------- Slackware - Because I know what I'm doing. ------
Barry Margolin
Rainer Weikusat <rwei...@mssgmbh.com> wrote:
Right. But we're talking about POSIX, not Unix. POSIX is a Unix-like
API that can be provided on many different OSes. brk() would be pretty
difficult to shoehorn into an OS that doesn't use Unix-like memory
layout.
Rainer Weikusat
Not necessarily. Provided that it is supported, they could use mmap.
It would even be easy to emulate (s)brk based on mmap (or malloc, of
course) sufficiently complete to be useful (by allocating a large
area, leaving it to the VM subsystem to deal with actually providing
RAM pages as needed, and implement the mark/ release-style interface
on top of that).
An example why this is useful: One of the applications running
on the appliances I mostly work on is a simple job scheduler. On
startup, it reads a file with lines as shown below:
<interval in s> <job name>:<something> <to> <"pass to execlp">
and sets up the necessary data structures to start any jobs defined
this way every 'interval' seconds. There is no way to tell it to
reread this file, because that would just complicate the program
without any real gain. Changes to the job definition file are rare and
the time needed to parse it is negligient (probably misspelled), so,
just killing the process (will be restarted by init) after the file
has been changed is sufficient. Because the number of jobs isn't
known until after the file has been parsed and neither is the
total size of the execlp-strings, the memory used to store those
strings and the associated job management structures must be allocated
at run time. The C-library used is an old version of uClibc using a
brk-based power-of-two-freelist-allocator as malloc. When this
allocator gets a 'small' request, it allocates 4K and divides into
equally sized blocks with the size being the requested size rounded to
the next power of two. In order to reduce its external fragmentation,
I have already modified it to use a minimum block size of 32
bytes. But it would still tie up a complete page for the lifetime of
the process to store (presently) less than 128 bytes of string data
and three management structures. The unmodified allocator would have
used three pages to accomplish the same. There is no possibility
for paging anything on this system, which means that once a process
has accessed a particular page, it will permanently sit on
these particular 4K (total RAM is 64M). By allocating from a continous
area, there is no external fragmentation anymore and internal
fragementation is reduced to the overhead necessary for proper
alignment (4). 'brk' is the ideal primitive for this type of dynamic
memory usage.
BTW, the Linux brk is already an emulation: (Unfortunately) calling
it does not cause the process data segment to grow but allocates
(on 2.4) address space in an area just above the static data area.
Rainer Weikusat
That's not a contradiction to my text above, just a 'vaguefication'.
> Apparently the default malloc in
> Glibc for Linux does call sbrk/brk (I won't bother verifying this because I
> don't care nor should I need to care).
If you honestly believe that you should not care how memory allocation
works for an environment you develop for, you should
a) refrain from discussing it
b) presumably, use Perl/ Phyton/ Java/ Ruby etc instead
of any compiled language, because they enable you to avoid
more work you shouldn't be caring about either
NB: That's not something I would consider to be an inferior choice. I
am always happy when I can get away without bothering with such
details. But I often cannot.
> Alternate malloc implementations that work on Linux may well use
> mmap().
Typically, they use both. But I wasn't writing about hypothetical
malloc-implementations for Linux-based systems and not even about
actual malloc-implementations for such systems, but about the subset
of the latter which I have used so far.
[...]
> The real point is that whatever ends up being called by the implementation of
> malloc is not what is considered by the application itself.
> What the application calls is only what it is coded to call. The
> whole VM process that ends up doing some ABI trap into the kernel to
> do kernel system calls is more than just the application; it is the
> application and linked in support library code as well. It is not
> the application that ends up calling sbrk/brk in the referenced case
> involving Linux; it is the whole of the process, application plus library.
>
> This is all terminology semantics, of course.
I would rather call it 'this is all the result of operating a battery
of smoke mortars' and suggest a simpler distinction: The application
is written to use a specific set of interfaces and uses any code which
is executed on behalf of it at runtime.
phil-new...@ipal.net
| In comp.unix.programmer, phil-new...@ipal.net wrote:
|
|> On Wed, 21 May 2008 12:35:22 +0800 Terry <con...@gmail.com> wrote:
|> | Barry Margolin wrote:
|> |> In article <g102db$gqo$1...@aioe.org>, Terry <con...@gmail.com> wrote:
|> |>
|> |>> I read from the brk manual page that brk() and sbrk() "are
|> |>> deliberately excluded from the POSIX.1 standard". Why is that and
|> |>> what's the substitution?
| [snip]
|> A malloc() implementation for some operating system FOO-OS could make use
|> of FOO-OS specific system calls which might not involve anything that even
|> looks like brk() or sbrk() or mmap() or shmat().
|
| Like on OS/390 Unix System Services (the IBM MVS OS subsystem that was
| certified as Unix95 by The Open Systems Group), the system-level
| memory-acquisition syscall is "GETMAIN" (and the release syscall
| is "FREEMAIN"). There is no brk() or sbrk() there, but it is still a
| branded Unix system supporting the POSIX/SUS APIs.
:-) BTDT (in ASM/370 a lot, w/o the POSIX APIs).
phil-new...@ipal.net
| If you honestly believe that you should not care how memory allocation
| works for an environment you develop for, you should
|
| a) refrain from discussing it
|
| b) presumably, use Perl/ Phyton/ Java/ Ruby etc instead
| of any compiled language, because they enable you to avoid
| more work you shouldn't be caring about either
|
| NB: That's not something I would consider to be an inferior choice. I
| am always happy when I can get away without bothering with such
| details. But I often cannot.
I do care how it is done. But when considering the standard malloc()
interface, I don't need to care when what it does serves my needs.
My "do not care" assertion is in the context of making use of malloc()
which I do often in my programming ... along with alloc() in cases where
that non-POSIX interface is more useful (needing a variable chunk of
memory to last only for as long as a function is in control and the
performance of just grabbing it from the stack and not tracking its
release through many possible function return paths).
| I would rather call it 'this is all the result of operating a battery
| of smoke mortars' and suggest a simpler distinction: The application
| is written to use a specific set of interfaces and uses any code which
| is executed on behalf of it at runtime.
The way I look at it is the application that calls malloc() is only calling
malloc() and nothing more. If sbrk/brk/mmap/shmat/GETMAIN/DMSFREE/foo get
called by some implementation of malloc(), I do not think of it as that
application calling that system function. The process would (but that is
a mix of application and linked library).
Rainer Weikusat
> On Thu, 22 May 2008 17:03:54 +0200 Rainer Weikusat <rwei...@mssgmbh.com> wrote:
> | If you honestly believe that you should not care how memory allocation
> | works for an environment you develop for, you should
> |
> | a) refrain from discussing it
> |
> | b) presumably, use Perl/ Phyton/ Java/ Ruby etc instead
> | of any compiled language, because they enable you to avoid
> | more work you shouldn't be caring about either
> |
> | NB: That's not something I would consider to be an inferior choice. I
> | am always happy when I can get away without bothering with such
> | details. But I often cannot.
>
> I do care how it is done. But when considering the standard malloc()
> interface, I don't need to care when what it does serves my needs.
IOW, you do not care how it works, because of the a priori assumption
that however whatever malloc is implemented, it will 'serve your
needs'. But you already wrote that and my answer is still 'Fine. Be
happy. Consider using automatic memory managment.'
[...]
> | I would rather call it 'this is all the result of operating a battery
> | of smoke mortars' and suggest a simpler distinction: The application
> | is written to use a specific set of interfaces and uses any code which
> | is executed on behalf of it at runtime.
>
> The way I look at it is the application that calls malloc() is only calling
> malloc() and nothing more. If sbrk/brk/mmap/shmat/GETMAIN/DMSFREE/foo get
> called by some implementation of malloc(), I do not think of it as that
> application calling that system function.
And you already wrote that, too, and my answer would again be the same.