1GB memory limit

[Bert Belder]

For some reason I am unable to use more than 1 GB of memory.
As the memory usage approaches 1GB my program becomes slower and
slower, then dies with an out of memory error.

A simple test case looks like:

var a = [], i = 0, s = '';
while (true) {
s = a[i++] = s + i;
}

The system I'm using runs 64-bit Ubuntu with around 3GB of physical
memory free and a ton of swap space available, so there's plenty left
when v8 runs into trouble.

I know about the --max_old_space_size flag; however setting it to
something over 1GB (like 2000000000) doesn't help. Is there anything
(else) I can do?

- Bert

[Erik Corry]

Are you running the 64 bit version of V8?

file `which v8`

You need to use --arch=x64 when building to get the 64 bit version.

> --
> v8-users mailing list
> v8-u...@googlegroups.com
> http://groups.google.com/group/v8-users
>

--
Erik Corry, Software Engineer
Google Denmark ApS - Frederiksborggade 20B, 1 sal,
1360 København K - Denmark - CVR nr. 28 86 69 84

[Bert Belder]

On Aug 25, 9:57 am, Erik Corry <erik.co...@gmail.com> wrote:
> Are you running the 64 bit version of V8?

I think so. To eliminate other causes I did a fresh git clone from
git://github.com/v8/v8.git, then:
scons arch=x64 sample=shell
./shell --max_old_space_size=2000000000 test.js

> file `which v8`
./shell: ELF 64-bit LSB executable, x86-64, version 1 (SYSV),
dynamically linked (uses shared libs), for GNU/Linux 2.6.15, not
stripped

- Bert

[Vyacheslav Egorov]

Hi Bert,

I did a quick look into this problem. It seems V8 currently can't handle such big heaps:

0) Memory allocator and MarkCompact garbage collector encode certain things in a special way so without major rewriting you can try to get at most ~1.8gb heap on x64.

1) Memory management code is full of integer overflows --- it uses int where bigger type is required to handle big heap sizes. [I am attaching small dirty patch to fix some of this overflows. with it you can grow heap almost up to 1.8 gb heap]

2) GC pauses will be huge for huge heaps: mark-sweep will take at > 1 s, mark-compact will take > 3s, scavenge throughput will also be degraded (it needs to scan oldspaces for intergenerational pointers). 

--
Vyacheslav Egorov

- Bert

[Bert Belder]

Thanks for looking into it, I'll try the patch in a minute.

In the short term it should help me because I do not need a lot more
memory than 1GB. The biggest problem was the slowness due to V8 doing
full GCs like crazy when it's almost out of memory, so this should
help a little.

However, do you think this could/will be fixed in the long term?
Eventually I'd like to be able to use 16G of memory :-)

- Bert

On Aug 25, 1:06 pm, Vyacheslav Egorov <vego...@chromium.org> wrote:
> Hi Bert,
>
> I did a quick look into this problem. It seems V8 currently can't handle
> such big heaps:
>
> 0) Memory allocator and MarkCompact garbage collector encode certain things
> in a special way so without major rewriting you can try to get at most
>  ~1.8gb heap on x64.
>
> 1) Memory management code is full of integer overflows --- it uses int where
> bigger type is required to handle big heap sizes. [I am attaching small
> dirty patch to fix some of this overflows. with it you can grow heap almost
> up to 1.8 gb heap]
>
> 2) GC pauses will be huge for huge heaps: mark-sweep will take at > 1 s,
> mark-compact will take > 3s, scavenge throughput will also be degraded (it
> needs to scan oldspaces for intergenerational pointers).
>
> --
> Vyacheslav Egorov
>

> On Wed, Aug 25, 2010 at 10:36 AM, Bert Belder <bertbel...@gmail.com> wrote:
> > On Aug 25, 9:57 am, Erik Corry <erik.co...@gmail.com> wrote:
> > > Are you running the 64 bit version of V8?
>
> > I think so. To eliminate other causes I did a fresh git clone from
> > git://github.com/v8/v8.git, then:
> >  scons arch=x64 sample=shell
> >  ./shell --max_old_space_size=2000000000 test.js
>
> > > file `which v8`
> > ./shell: ELF 64-bit LSB executable, x86-64, version 1 (SYSV),
> > dynamically linked (uses shared libs), for GNU/Linux 2.6.15, not
> > stripped
>
> > - Bert
>
> > --
> > v8-users mailing list
> > v8-u...@googlegroups.com
> >http://groups.google.com/group/v8-users
>
>
>

>  overflows.patch
> 9KViewDownload

[Vitali Lovich]

I'd recommend opening a bug in v8's issue tracker so they're aware of this upstream.