Re: Heavy malloc()/free() load and threads

Jonathan de Boyne Pollard

unread,

Mar 15, 2010, 10:21:09 PM3/15/10

to

>
>
> Much to our surprise, we discovered that most embedded malloc()/free()
> operations are horrifically slow [and more buggy].
>
That doesn't surprise me. It's a special case of the more general
principle being reiterated by several people elsewhere in this thread:
Rolling one's own allocator is tricky to get right. The C/POSIX
libraries for embedded systems sometimes haven't had as much work done
on them as the C/POSIX libraries for "mainstream" (for want of a better
word) systems. They are, in effect, roll-your-own efforts done by a
library developer for the target platform.

This is not to say that the "mainstream" implementatations are
invariably better and bug-free. However they are, by their very
natures, more extensively tested as general-purpose allocators.
Although implementors do write unit tests, the applications softwares
using the library are in general still the most effective test corpus.
I've encountered bugs in specialist implementations that have existed
for years simply because none of the applications softwares ever
exercised the library in the particular way necessary to exhibit the bug.

Jonathan de Boyne Pollard

unread,

Mar 20, 2010, 3:59:38 AM3/20/10

to

However, I wouldn't want all the dozens of apps on my desktop to all refuse to return memory back to the system just because they might want it again at some point in the future (thus unnecessarily forcing swapping to disk or a memory upgrade). Unless there is a very specific reason to think that performance is critical, my personal view is that it's polite for a userspace app to return memory back to the underlying system whenever it is reasonably possible.

malloc may or may not return memory to the system. Usually, it won't, except in fringe cases (eg 'large allocations' done via mmap). Memory allocations which originally happened by calling brk/sbrk cannot easily be returned to the system, anyway, only if freeing them happens to release a chunk of memory just below the current break.

On the contrary, usually it will. I'm revising my estimate of the quality of your "50 — 300 lines of code" implementation downwards as a result of this statement, because you are erroneously conflating allocating address space with committing pages. Most implementations that I am familiar with were written by people that didn't make this mistake.

My implementation (more correctly, one of my implementations (-:) calls the OS/2 DosSetMem() function to commit partially used pages and de-commit wholly unused pages within the heap arena as necessary. Several Win32 implementations that I'm aware of call VirtualAlloc() to commit and de-commit pages within arenas. (For a good explanation of this process, see Matt Pietrek's dissection of the DOS-Windows 9x HeapAlloc() function in his Windows 95 System Programming Secrets book.) The GNU C library version 2.11.1 calls madvise() with the MADV_DONTNEED flag for wholly unused pages.

The OS/2 and Win32 implementations are returning unused heap memory to the operating system as a matter of course. The GNU C library is intending to do the same, and is doing the best that it can with the more limited system API that it has to work with, and the operating system bugs that it has to cope with. (See, for example, Linux kernel bug #6282, reported by Samuel Thibault in 2007.)

Ian Collins

unread,

Mar 20, 2010, 4:27:33 AM3/20/10

to

On 03/20/10 08:59 PM, Jonathan de Boyne Pollard wrote:

Please stop cross-posting to so many groups and worse still, snipping
attributions.

--
Ian Collins

Richard Bos

unread,

Mar 20, 2010, 3:05:02 PM3/20/10

to

Ian Collins <ian-...@hotmail.com> wrote:

> On 03/20/10 08:59 PM, Jonathan de Boyne Pollard wrote:
>
> Please stop cross-posting to so many groups and worse still, snipping
> attributions.

Not to mention posting bloody HTML, in the raw. If you _must_ post HTML,
use the proper headers so that my server can dump it in the bit-bucket
in non-binary groups, as it should.

Richard

Rainer Weikusat

unread,

Mar 21, 2010, 3:10:03 PM3/21/10

to

Jonathan de Boyne Pollard <J.deBoynePoll...@NTLWorld.COM>
writes:
> <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">

In addition to the useless crosspost, the HTML is also unkind.
I was actually seriously trying to edit this into a readable text in
order to address your somewhat 'weird' statements about malloc
implementations not written by yourself but quickly grew tired of the
effort. Feel free to repost this in human-readable form and with a
selection of 'newsgroups' that lets you look less like a troll.

Scott Lurndal

unread,

Mar 21, 2010, 9:33:34 PM3/21/10

to

Ian Collins <ian-...@hotmail.com> writes:
>On 03/20/10 08:59 PM, Jonathan de Boyne Pollard wrote:
>
>Please stop cross-posting to so many groups and worse still, snipping
>attributions.
>

And even worse still, posting in HTML.