On Thu, Oct 11, 2012 at 6:34 PM, Jaan Oras <jaan.o...@gmail.com> wrote:
> Hi
> I'm a bit puzzled by the growing RSS size of node process, and while this
> has been discussed here a few times I've not seen a good explanation for
> this.
> The problem: while running load tests against our software the v8 heap
> stays relatively stable, but RSS usage of the process grows quite high
> compared to the heap, I've seen about 700MB for 32bit node ja 1G for 64bit
> node. Sometimes the RSS growth stabilises(does not grow after running load
> tests) at different points, but after a while it grows again, but I havent
> seen it decline.
> Also, while inspecting the objects in the JS heap, there are only about 10
> Buffer objects and none of them are big, so it does not seem to be a Buffer
> leak. At the same time mdb-s ::findjsobjects finds thousands of Buffer
> objects with many wiht size about 40kB, that are not referenced from
> anywhere, but I dont know enough about v8 and about the mdb v8 module to
> deduct anything from that. Maybe it is somekind of internal buffer pool or
> just already freed objects.
> mdb log is available here http://dl.dropbox.com/u/26529751/debug.out,
> RSS/heap/heap used was 193986560/28994560/14488064 bytes at the time.
> The reason why I'm interested in the RSS growth is that we have seen a
> crash with std::bad_alloc with node 0.8.9 32bit at about 500MB level,
> though I havent been able to reproduce it with 0.8.11.
> So the question is, is this something I should worry about, or is this
> growth an expected behaviour, and it will stop at some limit or is
> something wrong?
> To illustrate the issue I've ran some tests and plotted the
> process.memoryUsage():
On Thu, Oct 11, 2012 at 1:34 PM, Jaan Oras <jaan.o...@gmail.com> wrote:
> Hi
> I'm a bit puzzled by the growing RSS size of node process, and while this
> has been discussed here a few times I've not seen a good explanation for
> this.
> The problem: while running load tests against our software the v8 heap stays
> relatively stable, but RSS usage of the process grows quite high compared to
> the heap, I've seen about 700MB for 32bit node ja 1G for 64bit node.
> Sometimes the RSS growth stabilises(does not grow after running load tests)
> at different points, but after a while it grows again, but I havent seen it
> decline.
> Also, while inspecting the objects in the JS heap, there are only about 10
> Buffer objects and none of them are big, so it does not seem to be a Buffer
> leak. At the same time mdb-s ::findjsobjects finds thousands of Buffer
> objects with many wiht size about 40kB, that are not referenced from
> anywhere, but I dont know enough about v8 and about the mdb v8 module to
> deduct anything from that. Maybe it is somekind of internal buffer pool or
> just already freed objects.
> mdb log is available here http://dl.dropbox.com/u/26529751/debug.out,
> RSS/heap/heap used was 193986560/28994560/14488064 bytes at the time.
> The reason why I'm interested in the RSS growth is that we have seen a crash
> with std::bad_alloc with node 0.8.9 32bit at about 500MB level, though I
> havent been able to reproduce it with 0.8.11.
> So the question is, is this something I should worry about, or is this
> growth an expected behaviour, and it will stop at some limit or is something
> wrong?
> To illustrate the issue I've ran some tests and plotted the
> process.memoryUsage():
That std::bad_alloc you got may have been caused (or rather: fixed) by
[1] which is pending a back-port to v0.8.
Regarding buffers, until last night[2] their sizes were not reported
properly in heap snapshots. That may not apply to mdb though, I don't
know if it uses the heap profiler API.
On Thu, Oct 11, 2012 at 1:59 PM, Ben Noordhuis <i...@bnoordhuis.nl> wrote:
> That std::bad_alloc you got may have been caused (or rather: fixed) by
> [1] which is pending a back-port to v0.8.
Belay that, Isaac back-ported it in v0.8.10. That's probably why it
was failing in 0.8.9 and working with 0.8.11.
Some users experience high RSS usage which does not shrink. Maybe node fails to properly free RSS memory used by buffers, so the process will use all memory it can get over the time (some days), though it does not crash but have a rather constant memory consumption.
Nico
Am Donnerstag, 11. Oktober 2012 13:54:23 UTC+2 schrieb Fedor Indutny:
> RSS size depends on what you're actually doing, TLS server may eat much > more memory than HTTP one, and just TCP server may eat even less.
> Cheers, > Fedor.
> On Thu, Oct 11, 2012 at 6:34 PM, Jaan Oras <jaan...@gmail.com<javascript:> > > wrote:
>> Hi
>> I'm a bit puzzled by the growing RSS size of node process, and while this >> has been discussed here a few times I've not seen a good explanation for >> this.
>> The problem: while running load tests against our software the v8 heap >> stays relatively stable, but RSS usage of the process grows quite high >> compared to the heap, I've seen about 700MB for 32bit node ja 1G for 64bit >> node. Sometimes the RSS growth stabilises(does not grow after running load >> tests) at different points, but after a while it grows again, but I havent >> seen it decline.
>> Also, while inspecting the objects in the JS heap, there are only about >> 10 Buffer objects and none of them are big, so it does not seem to be a >> Buffer leak. At the same time mdb-s ::findjsobjects finds thousands of >> Buffer objects with many wiht size about 40kB, that are not referenced from >> anywhere, but I dont know enough about v8 and about the mdb v8 module to >> deduct anything from that. Maybe it is somekind of internal buffer pool or >> just already freed objects. >> mdb log is available here http://dl.dropbox.com/u/26529751/debug.out, >> RSS/heap/heap used was 193986560/28994560/14488064 bytes at the time.
>> The reason why I'm interested in the RSS growth is that we have seen a >> crash with std::bad_alloc with node 0.8.9 32bit at about 500MB level, >> though I havent been able to reproduce it with 0.8.11.
>> So the question is, is this something I should worry about, or is this >> growth an expected behaviour, and it will stop at some limit or is >> something wrong?
>> To illustrate the issue I've ran some tests and plotted the >> process.memoryUsage():
I've tried to 0.8.9 and I get consistent crashes, with 0.8.11 I dont get crashes, so I guess the crashing part is fixed. The RSS still grows but finally stops at some point(about 700MB for 32bit process), but it is still weird that sometimes after the same loadtest RSS is 700MB sometimes 250MB.
I guess it is not problem for me at the moment, but I figure that this kind of behaviour is quite limiting for people using cluster module.
0.8.12 claims to display proper retained size of buffer, I'll see what it shows.
> RSS size depends on what you're actually doing, TLS server may eat much > more memory than HTTP one, and just TCP server may eat even less.
> Cheers, > Fedor.
> On Thu, Oct 11, 2012 at 6:34 PM, Jaan Oras <jaan...@gmail.com<javascript:> > > wrote:
>> Hi
>> I'm a bit puzzled by the growing RSS size of node process, and while this >> has been discussed here a few times I've not seen a good explanation for >> this.
>> The problem: while running load tests against our software the v8 heap >> stays relatively stable, but RSS usage of the process grows quite high >> compared to the heap, I've seen about 700MB for 32bit node ja 1G for 64bit >> node. Sometimes the RSS growth stabilises(does not grow after running load >> tests) at different points, but after a while it grows again, but I havent >> seen it decline.
>> Also, while inspecting the objects in the JS heap, there are only about >> 10 Buffer objects and none of them are big, so it does not seem to be a >> Buffer leak. At the same time mdb-s ::findjsobjects finds thousands of >> Buffer objects with many wiht size about 40kB, that are not referenced from >> anywhere, but I dont know enough about v8 and about the mdb v8 module to >> deduct anything from that. Maybe it is somekind of internal buffer pool or >> just already freed objects. >> mdb log is available here http://dl.dropbox.com/u/26529751/debug.out, >> RSS/heap/heap used was 193986560/28994560/14488064 bytes at the time.
>> The reason why I'm interested in the RSS growth is that we have seen a >> crash with std::bad_alloc with node 0.8.9 32bit at about 500MB level, >> though I havent been able to reproduce it with 0.8.11.
>> So the question is, is this something I should worry about, or is this >> growth an expected behaviour, and it will stop at some limit or is >> something wrong?
>> To illustrate the issue I've ran some tests and plotted the >> process.memoryUsage():
