Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

1GB RAM limit for perl5.8.0 -Dusemymalloc under linux?

138 views
Skip to first unread message

Slaven Rezic

unread,
Apr 11, 2003, 6:54:43 PM4/11/03
to aacco...@cfa.harvard.edu, perl5-...@perl.org
Alberto Accomazzi <aacco...@cfa.harvard.edu> writes:

> I have recently recompiled perl 5.8.0 under linux/intel (redhat 7.3 +
> latest patches) with the -Dusemymalloc flag in order to get around a bug
> in libc-2.2.5's malloc affecting allocations of large hashes as detailed
> in http://archive.develooper.com/perl5-...@perl.org/msg88107.html
>
> Unfortunately the resulting binary seems to be limited to accessing
> approximately 900MB of RAM, at least from the tests I ran. Has anybody
> seen this and is there a way around it? My system has 3GB RAM (plus
> swap) and the memory limit per process (in user space) is 2.5GB, which I
> am able to access using the perl binary compiled with libc's malloc.
>

Can you rerun the test with -Dm (needs a debugging perl) and post the
numbers for the last sbrk request before the "out of memory" message?

Regards,
Slaven

--
Slaven Rezic - sla...@rezic.de
BBBike - route planner for cyclists in Berlin
WWW version: http://www.bbbike.de
Perl/Tk version for Unix and Windows: http://bbbike.sourceforge.net

Andreas J. Koenig

unread,
Apr 12, 2003, 3:21:17 AM4/12/03
to Slaven Rezic, aacco...@cfa.harvard.edu, perl5-...@perl.org
>>>>> On 12 Apr 2003 00:54:43 +0200, Slaven Rezic <sla...@rezic.de> said:

> Can you rerun the test with -Dm (needs a debugging perl) and post the
> numbers for the last sbrk request before the "out of memory" message?

First without -Dm:

installed-perls/perl/pDSfSgS/perl-5.8.0@19189/bin/perl -le 'use Config; my $malloc = $Config{usemymalloc}=~/y/ ? "my malloc" : "system malloc"; while () {$a.="x"x(1<<26); printf ">>> %s %3d - %s", $malloc, ++$n, `grep ^VmSize /proc/$$/status` }'
>>> my malloc 1 - VmSize: 134636 kB
>>> my malloc 2 - VmSize: 265720 kB
>>> my malloc 3 - VmSize: 527868 kB
>>> my malloc 4 - VmSize: 527868 kB
Out of memory during "large" request for 536875008 bytes, total sbrk() is 537270272 bytes at -e line 1.

The relevant output of the same with -Dm:

>>> my malloc 3 - VmSize: 527868 kB
0x81dbe08: (06312) free
0x81dbe48: (06313) free
0x81da608: (06314) malloc 68 bytes
0x81dbe48: (06315) malloc 32 bytes
0x81dd088: (06314) malloc 16 bytes
0x81da608: (06315) malloc 80 bytes
0x81da608: (06316) realloc 21 bytes the hard way
0x81dbe48: (06317) malloc 21 bytes
0x81da608: (06318) free
0x81da608: (06319) malloc 80 bytes
0x81da608: (06320) free
0x81dd088: (06321) free
0x81c4298: (06322) malloc 1 bytes
0x81c4298: (06323) realloc 5 bytes the hard way
0x81dd088: (06324) malloc 5 bytes
0x81c4298: (06325) free
0x81dd088: (06326) realloc 21 bytes the hard way
0x81dbe08: (06327) malloc 21 bytes
0x81dd088: (06328) free
>>> my malloc 4 - VmSize: 527868 kB
0x81dbe08: (06329) free
0x81dbe48: (06330) free
0x181eb008: (06331) realloc 335544321 bytes the hard way
sbrk(536875008) for 536875008-byte-long arena
0x81c4298: (06332) malloc 1 bytes
0x81c4298: (06333) realloc 42 bytes the hard way
0x81dbe48: (06334) malloc 42 bytes
0x81c4298: (06335) free
0x81dbe48: (06336) realloc 75 bytes the hard way
0x81da608: (06337) malloc 75 bytes
0x81dbe48: (06338) free
Out of memory during "large" request for 536875008 bytes, total sbrk() is 537272320 bytes at -e line 1.
0x81da608: (06339) free
0x81dbec8: (06340) free
0x0: (06341) free
0x819a508: (06342) free
0x819be08: (06343) free
[...]
0x818ce48: (06516) free
0x818ae88: (06517) free
0x818ce08: (06518) free

--
andreas

Ton Hospel

unread,
Apr 12, 2003, 7:01:12 AM4/12/03
to perl5-...@perl.org
In article <3E8B0FE0.80901@c_a.harvard.edu>,

Alberto Accomazzi <aacco...@cfa.harvard.edu> writes:
> I have recently recompiled perl 5.8.0 under linux/intel (redhat 7.3 +
> latest patches) with the -Dusemymalloc flag in order to get around a bug
> in libc-2.2.5's malloc affecting allocations of large hashes as detailed
> in http://archive.develooper.com/perl5-...@perl.org/msg88107.html
>
> Unfortunately the resulting binary seems to be limited to accessing
> approximately 900MB of RAM, at least from the tests I ran. Has anybody
> seen this and is there a way around it? My system has 3GB RAM (plus
> swap) and the memory limit per process (in user space) is 2.5GB, which I
> am able to access using the perl binary compiled with libc's malloc.
>
> TIA,
>
> -- Alberto
>
This sounds normal in case mymalloc uses only sbrk based allocation.
In that case you indeed run into a wall on linux around 900M
(after that you get to the place shared libraries are mapped, so the space
can't be extended further).
To allocate more on linux, you use mmap to get big chunks of memory.

Brand Hunt

unread,
Apr 14, 2003, 11:56:30 AM4/14/03
to
Alberto--

Can you switch to a different allocator?

I tried the test-case in the link you provided with ATS (the custom,
general purpose, allocator from the company I work for) and the time
to reset a large hash dropped dramatically (from 2 minutes to 2
seconds.)

(fortress.cvo.roguewave.com) perl test.pl 2000000
Mon Apr 14 08:47:52 2003
init done: Mon Apr 14 08:48:06 2003
exiting: Mon Apr 14 08:50:06 2003

(fortress.cvo.roguewave.com) LD_PRELOAD=libatsfs32.so perl test.pl
2000000
Mon Apr 14 08:50:17 2003
init done: Mon Apr 14 08:50:31 2003
exiting: Mon Apr 14 08:50:33 2003

We use the allocator internally on Python/Perl executables with great
success. If you're interested, please check out:
http://www.roguewave.com/ATSevan01

Cheers,
Brand

perl5-...@ton.iguana.be (Ton Hospel) wrote in message news:<b78rlo$afb$1...@post.home.lunix>...

0 new messages