heap limits: mmap(2) vs. break(2) on i386
Maxim Sobolev
I am trying to figure out why java fails to start with 1024MB of heap on
i386 with 4GB of RAM and 4GB of swap. Both MAXDSIZ and DFLDSIZ are set
to 2GB. Here is my limits:
Resource limits (current):
cputime infinity secs
filesize infinity kB
datasize 2097152 kB
stacksize 65536 kB
coredumpsize infinity kB
memoryuse infinity kB
memorylocked infinity kB
maxprocesses 5547
openfiles 20000
sbsize infinity bytes
vmemoryuse infinity kB
Running ktrace I see:
9154 java CALL
mmap(0,0x44000000,PROT_READ|PROT_WRITE|PROT_EXEC,MAP_PRIVATE|MAP_NORESERVE|MAP_ANON,0xffffffff,0,0)
9154 java RET mmap -1 errno 12 Cannot allocate memory
9154 java CALL write(0x1,0xbf9fe378,0x2b)
9154 java GIO fd 1 wrote 43 bytes
"Error occurred during initialization of VM
I made a small program that uses malloc(3) to allocate the same amount
of memory, and that works nicely, ktrace reveals why:
10108 a.out CALL
mmap(0,0x44000000,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,0xffffffff,0,0)
10108 a.out RET mmap -1 errno 12 Cannot allocate memory
10108 a.out CALL break(0x4c100000)
10108 a.out RET break 0
So the question is: why does mmap() fails while essentially the same
sbrk() request succeeds? This is really bad since, while native FreeBSD
programs can work around this by using malloc(3), Linux programs and
software that knows nothing about intricate details of the FreeBSD VM
(i.e. Java) will fail miserably.
I tried increasing vm.max_proc_mmap to 2147483647 from default 49344,
but it did not do any good. mmap() still fails with the request of this
size.
I have seen several threads on the issue over the years, but still no
resolution. It seems that only plausible solution is to limit heap size
in java, which may not work for all cases.
Funny thing is that the first sentence of the sbrk(2) manual page says:
The brk() and sbrk() functions are legacy interfaces from before
the advent of modern virtual memory management.
Yet, "legacy interfaces" seems to do much better job than "modern
virtual memory management interfaces"!
-Maxim
_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hacke...@freebsd.org"
Jason Evans
> I am trying to figure out why java fails to start with 1024MB of heap on
> i386 with 4GB of RAM and 4GB of swap. Both MAXDSIZ and DFLDSIZ are set
> to 2GB.
Some memory (1GiB?) is reserved for kernel address space, and you
reserved 2GiB for DSS. That leaves less than 1GiB available after
shared libraries and whatnot are mapped. If there is more than 1GiB
available, mmap can still fail due to the memory being non-contiguous.
Jason
Maxim Sobolev
> Maxim Sobolev wrote:
>> I am trying to figure out why java fails to start with 1024MB of heap
>> on i386 with 4GB of RAM and 4GB of swap. Both MAXDSIZ and DFLDSIZ are
>> set to 2GB.
>
> Some memory (1GiB?) is reserved for kernel address space, and you
> reserved 2GiB for DSS. That leaves less than 1GiB available after
> shared libraries and whatnot are mapped. If there is more than 1GiB
> available, mmap can still fail due to the memory being non-contiguous.
Jason,
So, are you saying that by allocating 2GB to MAXDSIZ, I limit myself
less than 1GB left to be allocated via mmap()?
Perhaps the cause of the problem is my interpretation of MAXDSIZ as an
overall limit of VM that the process will be able to allocate regardless
of the memory management interface is wrong, and in fact the process can
allocate up to MAXDSIZ using sbrk(2) and then some extra using mmap(2)
up to 3GB?
I tried lowering DFLDSIZ to 1.5GB, and it helped with Java. What is the
best strategy if I want to maximize amount of memory available to
applications? Most of modern applications use mmap(), isn't it? Then
where MAXDSIZ can bite me if I set it to say 512MB?
-Maxim
Jason Evans
> Jason Evans wrote:
>> Maxim Sobolev wrote:
>>> I am trying to figure out why java fails to start with 1024MB of heap
>>> on i386 with 4GB of RAM and 4GB of swap. Both MAXDSIZ and DFLDSIZ are
>>> set to 2GB.
>>
>> Some memory (1GiB?) is reserved for kernel address space, and you
>> reserved 2GiB for DSS. That leaves less than 1GiB available after
>> shared libraries and whatnot are mapped. If there is more than 1GiB
>> available, mmap can still fail due to the memory being non-contiguous.
>
> less than 1GB left to be allocated via mmap()?
Yes, my recollection is that MAXDSIZ controls the amount of virtual
address space dedicated to DSS, and this address space will not be
mapped via anonymous mmap. I wanted to move completely away from using
sbrk in malloc, but we can't completely remove DSS for backward
compatibility reasons, which means less heap address space than would be
ideal.
> What is the
> best strategy if I want to maximize amount of memory available to
> applications? Most of modern applications use mmap(), isn't it? Then
> where MAXDSIZ can bite me if I set it to say 512MB?
I would set MAXDSIZ to 0, so that the maximum amount of memory is
available for mapping shared libraries and files, and allocating via
malloc. This may cause problems with a couple of ports that implement
their own memory allocators based on sbrk, but otherwise it should be
all good. You might also set /etc/malloc.conf to 'd' in order to
disable the sbrk calls.
Jason
Maxim Sobolev
I see, thank you for the explanation. One of the problem that we are
having is that we use a lot of interpreted languages in our environment
(python, php etc), and most of those implement their own memory
allocators, some of which rely on sbrk(2) unfortunately. I believe
that's where that 2GB limit of ours comes from - one of our Python
applications is very memory hungry and we had to bump that limit to
allow it sufficient room.
Crazy idea, perhaps, but has anyone considered wrapping up sbrk(2) into
mmap(2), so that there is only one memory pool to draw from? Switch to
64-bit certainly helps, however there are lot of 32-bit machines hanging
around and we will see them for a while in the embedded space. Certainly
current situation with two separate sources of heap memory is not normal.
-Maxim
Andrew MacIntyre
> Jason Evans wrote:
>> I would set MAXDSIZ to 0, so that the maximum amount of memory is
>> available for mapping shared libraries and files, and allocating via
>> malloc. This may cause problems with a couple of ports that implement
>> their own memory allocators based on sbrk, but otherwise it should be
>> all good. You might also set /etc/malloc.conf to 'd' in order to
>> disable the sbrk calls.
>
> I see, thank you for the explanation. One of the problem that we are
> having is that we use a lot of interpreted languages in our environment
> (python, php etc), and most of those implement their own memory
> allocators, some of which rely on sbrk(2) unfortunately. I believe
> that's where that 2GB limit of ours comes from - one of our Python
> applications is very memory hungry and we had to bump that limit to
> allow it sufficient room.
While Python has its own allocator, it relies on the platform malloc()
rather than sbrk(), and therefore Jason's suggestion to use '-d' in
/etc/malloc.conf should be effective for it.
--
-------------------------------------------------------------------------
Andrew I MacIntyre "These thoughts are mine alone..."
E-mail: and...@bullseye.apana.org.au (pref) | Snail: PO Box 370
and...@pcug.org.au (alt) | Belconnen ACT 2616
Web: http://www.andymac.org/ | Australia
Tijl Coosemans
> I am trying to figure out why java fails to start with 1024MB of heap
> on i386 with 4GB of RAM and 4GB of swap. Both MAXDSIZ and DFLDSIZ are
> set to 2GB. Here is my limits:
>
> Resource limits (current):
> cputime infinity secs
> filesize infinity kB
> datasize 2097152 kB
> stacksize 65536 kB
> coredumpsize infinity kB
> memoryuse infinity kB
> memorylocked infinity kB
> maxprocesses 5547
> openfiles 20000
> sbsize infinity bytes
> vmemoryuse infinity kB
>
> Running ktrace I see:
>
> 9154 java CALL mmap(0,0x44000000,PROT_READ|PROT_WRITE|PROT_EXEC,MAP_PRIVATE|MAP_NORESERVE|MAP_ANON,0xffffffff,0,0)
> 9154 java RET mmap -1 errno 12 Cannot allocate memory
> 9154 java CALL write(0x1,0xbf9fe378,0x2b)
> 9154 java GIO fd 1 wrote 43 bytes
> "Error occurred during initialization of VM
On i386 a process has only 3GiB of address space. If you reserve 2GiB
for datasize (sbrk), there's less than 1GiB available for mmap. Unless
you have a program that still uses sbrk and needs 2GiB you should make
maxdsiz much smaller. Since FreeBSD 7 malloc can use mmap besides sbrk
so you can set maxdsiz to a really small value if you want to.
Ulrich Spörlein
> Crazy idea, perhaps, but has anyone considered wrapping up sbrk(2) into
> mmap(2), so that there is only one memory pool to draw from? Switch to
> 64-bit certainly helps, however there are lot of 32-bit machines hanging
> around and we will see them for a while in the embedded space. Certainly
> current situation with two separate sources of heap memory is not normal.
Alternative and very low tech test:
- Remove sbrk() from libc and /usr/include
- Run port test build
- ???
- PROFIT!
It shall be interesting to see which ports blow up thanks to sbrk()
missing.
Regards,
Uli