10MB Chunks of anonymous memory for each thread created from shared object

[google...@brentandamanda.net]

I'm having a problem when creating pthreads from a shared object
library. Every pthread that I spawn is accompanied by a 10MB chunk of
anonymous memory.

I manually set the stack size to 17k, and verified that it provides a
seg fault when I use more than I allocate. So I think I can rule-out
the stack.

But here's the perplexing part: If link the same code into an
executable, the 10MB chunks become 12KB.

When I link the offending code into a shared object, it doesn't matter
if I use dlopen or I dynamically link to it with g++, these chunks are
10MB.

Does anyone have any how I can find out what these anonymous memory
pages are? Once I know what they are I can figure out how to reduce
the size.

Thanks.

I'm running on:
Linux version 2.6.9-5.ELsmp (bhco...@decompose.build.redhat.com)
(gcc version 3.4.3 20041212 (Red Hat 3.4.3-9.EL4)) #1 SMP Wed Jan 5
19:30:39 EST 2005

This is currently running on a pentium4 box so I have tools like pmap
available. But our embedded system is an arm9.

The following is the snippet from pmap for 12 threads spawed from the
shared object:
08048000 8K r-x-- /home/brent/MAP/bin/FredInstantiatorDynamic
0804a000 4K rw--- /home/brent/MAP/bin/FredInstantiatorDynamic
091f8000 132K rw--- [ anon ]
b07d7000 4K ----- [ anon ]
b07d8000 10240K rw--- [ anon ]
b11d8000 4K ----- [ anon ]
b11d9000 10240K rw--- [ anon ]
b1bd9000 4K ----- [ anon ]
b1bda000 10240K rw--- [ anon ]
b25da000 4K ----- [ anon ]
b25db000 10240K rw--- [ anon ]
b2fdb000 4K ----- [ anon ]
b2fdc000 10240K rw--- [ anon ]
b39dc000 4K ----- [ anon ]
b39dd000 10240K rw--- [ anon ]
b43dd000 4K ----- [ anon ]
b43de000 10240K rw--- [ anon ]
b4dde000 4K ----- [ anon ]
b4ddf000 10240K rw--- [ anon ]
b57df000 4K ----- [ anon ]
b57e0000 10240K rw--- [ anon ]
b61e0000 4K ----- [ anon ]
b61e1000 10240K rw--- [ anon ]
b6be1000 4K ----- [ anon ]
b6be2000 10240K rw--- [ anon ]
b75e2000 4K ----- [ anon ]
b75e3000 10256K rw--- [ anon ]
b7ffe000 8K rw--- [ anon ]
bfe73000 1588K rw--- [ stack ]
ffffe000 4K ----- [ anon ]

When I run it from an executable, it becomes:
08048000 64K r-x-- /home/brent/MAP/bin/FredInstantiatorStatic
08058000 4K rw--- /home/brent/MAP/bin/FredInstantiatorStatic
084fd000 132K rw--- [ anon ]
b7fc7000 4K ----- [ anon ]
b7fc8000 12K rw--- [ anon ]
b7fcb000 4K ----- [ anon ]
b7fcc000 12K rw--- [ anon ]
b7fcf000 4K ----- [ anon ]
b7fd0000 12K rw--- [ anon ]
b7fd3000 4K ----- [ anon ]
b7fd4000 12K rw--- [ anon ]
b7fd7000 4K ----- [ anon ]
b7fd8000 12K rw--- [ anon ]
b7fdb000 4K ----- [ anon ]
b7fdc000 12K rw--- [ anon ]
b7fdf000 4K ----- [ anon ]
b7fe0000 32K rw--- [ anon ]
b7feb000 4K ----- [ anon ]
b7fec000 12K rw--- [ anon ]
b7fef000 4K ----- [ anon ]
b7ff0000 12K rw--- [ anon ]
b7ff3000 4K ----- [ anon ]
b7ff4000 12K rw--- [ anon ]
b7ff7000 4K ----- [ anon ]
b7ff8000 12K rw--- [ anon ]
b7ffb000 4K ----- [ anon ]
b7ffc000 16K rw--- [ anon ]
bff7f000 516K rw--- [ stack ]
ffffe000 4K ----- [ anon ]

[Paul Pluzhnikov]

google...@brentandamanda.net writes:

> Does anyone have any how I can find out what these anonymous memory
> pages are?

Run the program under gdb, set breakpoint on __mmap and __mmap64,
when hit type 'where' ?

I bet they are still coming from thread stack allocation,
probably due to a bug in libpthread ...

Cheers,
--
In order to understand recursion you must first understand recursion.
Remove /-nsp/ for email.

[google...@brentandamanda.net]

> I bet they are still coming from thread stack allocation,

You are correct. It was the stack.

> probably due to a bug in libpthread ...

It could definitely be in libpthread, but it could also be the g++
linker. The behavior was different whether our code that created the
threads was in a shared object or an executable. But both times we
linked to the same libpthread

Thanks for the help

[Paul Pluzhnikov]

google...@brentandamanda.net writes:

>> I bet they are still coming from thread stack allocation,
>
> You are correct. It was the stack.
>
>> probably due to a bug in libpthread ...
>
> It could definitely be in libpthread, but it could also be the g++
> linker.

There is no such thing as 'g++ linker'; it uses the system GNU linker.

I have reproduced the problem you observed, and it turns out to be
a bug in the way you build your shared library (or a rather obscure
interaction between your link line and symbol versioning).

First the result:

$ gcc -shared -fPIC -o create.so create.c # [1]
$ gcc -pthread main.c ./create.so
$ ./a.out
stack size: 163840
&p = 0xb7fee460
in region [0xb75ef000, 0xb7ff1000) (10493952)

Above, I asked for stack size of 160K (10*PTHREAD_STACK_MIN),
but got 10MB (just like you). I am guessing that's exactly what
you did. The problem is that command [1] is incorrect.

Correct command:

$ gcc -pthread -shared -fPIC -o create.so create.c
^^^^^^^^
$ ./a.out
stack size: 163840
&p = 0xb7fee460
in region [0xb7fc8000, 0xb7ff1000) (167936)

produces expected result.

The reason:

$ gcc -shared -fPIC -o create.so create.c &&
objdump -T ./create.so | grep pthread_create
00000000 D *UND* 00000000 pthread_create
$ gcc -pthread -shared -fPIC -o create.so create.c &&
objdump -T ./create.so | grep pthread_create
00000000 DF *UND* 0000018c GLIBC_2.1 pthread_create

When create.so is built correctly, it uses pthread_create@GLIBC_2.1.
When it isn't, it uses pthread_create@GLIBC_2.0:

$ gcc -shared -fPIC -o create.so create.c
$ LD_DEBUG=bindings ./a.out 2>&1 | grep pthread_create
18884: binding file ./create.so to /lib/tls/libpthread.so.0: normal symbol `pthread_create'

$ gcc -pthread -shared -fPIC -o create.so create.c
$ LD_DEBUG=bindings ./a.out 2>&1 | grep pthread_create
18981: binding file ./create.so to /lib/tls/libpthread.so.0: normal symbol `pthread_create' [GLIBC_2.1]

Finally, here is the code I used:

$ cat main.c
int main() { create(); return 0; }

$ cat create.c
#include <pthread.h>
#include <limits.h>
#include <stdio.h>

void *fn(void *p)
{
char cmd[1024];
FILE *fp;
fprintf(stderr, "&p = %p\n", &p);
sprintf(cmd, "cat /proc/%d/maps", getpid());
fp = popen(cmd, "r");
while (fgets(cmd, sizeof(cmd), fp)) {
void *low, *hi;
int n = sscanf(cmd, "%x-%x", &low, &hi);
if (n) {
if (low < (void*)&p && (void*)&p < hi) {
fprintf(stderr, "in region [%p, %p) (%d)\n", low, hi, (char*)hi - (char*)low);
fclose(fp);
break;
}
}
}
return p;
}

void create()
{
pthread_t tid;
size_t size = 10*PTHREAD_STACK_MIN;
pthread_attr_t attr;
fprintf(stderr, "stack size: %d\n", size);

pthread_attr_init(&attr);
pthread_attr_setstacksize(&attr, size);
pthread_create(&tid, &attr, fn, NULL);
pthread_join(tid, NULL);