[nodejs] how to free() a buffer?

[Bryan Berry]

What is the best way to delete an allocated buffer?

node> var buf = new Buffer(1234);

node> delete buf

delete seems to reduce the size of the Heap Used but there is speculation that it also puts v8 into the "slow case"

tks

--
You received this message because you are subscribed to the Google Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com.
To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/nodejs?hl=en.

[Tim Caswell]

I've read in a few places that you should only use delete for properties of objects, not for variables.  I too am interested in the answer to this.  I'm hoping you don't have to free buffers.

[Tom Robinson]

"delete" in JS doesn't mean the same thing as delete/free() in C++/C. Objects (including Buffers) are tracked by the garbage collector and deallocated when there are no more references to it, so simply assigning null (or anything else) to buf should also work (assuming there aren't any other references).

And presumably the actual chunk of memory backing the Buffer object is automatically freed when it's Buffer is collected.

I'm not familiar with performance characteristics of "delete".

-tom

[bryanwb]

On May 4, 8:35 am, Tim Caswell <t...@creationix.com> wrote:
> I've read in a few places that you should only use delete for properties of objects, not for variables.  I too am interested in the answer to this.  I'm hoping you don't have to free buffers.
>
> On May 3, 2010, at 10:31 PM, Bryan Berry wrote:
>
> > What is the best way to delete an allocated buffer?
>
> > node> var buf = new Buffer(1234);
> > node> delete buf
>
> > delete seems to reduce the size of the Heap Used but there is speculation that it also puts v8 into the "slow case"
>
> > tks
>

Since Buffer gives you raw memory access, it should be consuming space
on the heap

Here is a quick look memory usage before and after allocating a buffer

node> process.memoryUsage()
{ rss: 6586368
, vsize: 46768128
, heapTotal: 2303488
, heapUsed: 1245968
}
node> buf = require('buffer').Buffer
node> var buf1 = new buf(12345)
node> process.memoryUsage()
{ rss: 6942720
, vsize: 47345664
, heapTotal: 2352640
, heapUsed: 1335588
}

as we can see here, heapTotal and heapUsed have both increased
substantially, though not in direct proportion to the # of blocks in
the buffer

node> delete buf1
node> process.memoryUsage()
{ rss: 6774784
, vsize: 47173632
, heapTotal: 2180608
, heapUsed: 1162864
}

here deleting the buffer, while perhaps an incorrect use of delete,
appears to reduce the size of the heap, but not in direct proportion
to the # of blocks that were in the buffer

[bryanwb]

On May 4, 8:50 am, Tom Robinson <tlrobin...@gmail.com> wrote:
> "delete" in JS doesn't mean the same thing as delete/free() in C++/C. Objects (including Buffers) are tracked by the garbage collector and deallocated when there are no more references to it, so simply assigning null (or anything else) to buf should also work (assuming there aren't any other references).
>
> And presumably the actual chunk of memory backing the Buffer object is automatically freed when it's Buffer is collected.
>
> I'm not familiar with performance characteristics of "delete".
>
> -tom

tom, tks for the suggestion

assigning null rather than delete seems to do the trick

node> process.memoryUsage()
{ rss: 6606848
, vsize: 46768128
, heapTotal: 2343168

, heapUsed: 1245968
}
node> buf = require('buffer').Buffer

node> var buf1 = new buf(2000)
node> process.memoryUsage()
{ rss: 6737920
, vsize: 47206400
, heapTotal: 2253056
, heapUsed: 1194996
}
node> buf1 = null
// wait 10 secs
node> process.memoryUsage()
{ rss: 6709248
, vsize: 47173632
, heapTotal: 2220288
, heapUsed: 1161192
}
// and the heapUsed goes down

[Ryan Dahl]

On Mon, May 3, 2010 at 8:31 PM, Bryan Berry <br...@olenepal.org> wrote:
> What is the best way to delete an allocated buffer?

Drop your reference to it. "delete buffer" does not force garbage
collection or deallocation.

[Marco Rogers]

I'm dealing with this right now in libxmljs. My lib will shortly be
updated to work the way Buffer does. Killing the reference to the js
object with delete does not force or even suggest garbage collection
in v8. v8 has it's own internal heuristics for when it decides to run
the GC. Unfortunately, external bindings like Buffer and libxmljs
allocate memory *outside* of the v8 heap space. So the context
doesn't care about that memory unless you tell it to.

The way you take part in this heuristic is by letting v8 know how much
memory you're allocating outside of v8. There is a function surfaced
in the API for this. Buffer does this so if you allocate lots of
buffer space, v8 will see this and the gc will run. If there are
buffers that can be collected, their destructors will be called and
the internal memory will be freed. But the delete operator has
nothing to do with that. All this does is reset the variable
reference. Hope that makes sense.

One idea I thought of if people are concerned about this is a
Buffer#destroy method. This would essentially free the internal
buffer. There are concerns with this though. If it's an instance
method, i.e. buf.destroy(), then you still have the js reference and
if you try to access it, you'll get a seg fault. If it's a class
method, Buffer.destroy(buf), that's a little better, but class methods
have their own issues in terms of testibility and reasoning about
external interactions. Could be useful in this narrow case though.

[Ryan Dahl]

On Tue, May 4, 2010 at 11:30 AM, Marco Rogers <marco....@gmail.com> wrote:
> I'm dealing with this right now in libxmljs.  My lib will shortly be
> updated to work the way Buffer does.  Killing the reference to the js
> object with delete does not force or even suggest garbage collection
> in v8.  v8 has it's own internal heuristics for when it decides to run
> the GC.  Unfortunately, external bindings like Buffer and libxmljs
> allocate memory *outside* of the v8 heap space.  So the context
> doesn't care about that memory unless you tell it to.
>
> The way you take part in this heuristic is by letting v8 know how much
> memory you're allocating outside of v8.  There is a function surfaced
> in the API for this.  Buffer does this so if you allocate lots of
> buffer space, v8 will see this and the gc will run.  If there are
> buffers that can be collected, their destructors will be called and
> the internal memory will be freed.  But the delete operator has
> nothing to do with that.  All this does is reset the variable
> reference.  Hope that makes sense.
>
> One idea I thought of if people are concerned about this is a
> Buffer#destroy method.  This would essentially free the internal
> buffer.  There are concerns with this though.  If it's an instance
> method, i.e. buf.destroy(), then you still have the js reference and
> if you try to access it, you'll get a seg fault.  If it's a class
> method, Buffer.destroy(buf), that's a little better, but class methods
> have their own issues in terms of testibility and reasoning about
> external interactions.  Could be useful in this narrow case though.

Making a destroy method is very dangerous. I suggest that if you're
concerned about this then you use require('freelist') and try to
reclaim memory after using it - which is what I do in net.js

[Marco Rogers]

I realized that I used lots of references to "internal" vs. "external"
memory that seems confusing. Here's a quick breakdown:

Buffer is a actually a js object created in C++ through the v8 API.
When you're in the v8 layer you've got objects that get created by v8
and those go in the v8 heap space. There is other memory that you
allocate yourself using malloc or that get's allocated by external C/C+
+ libraries. Like the bindings in libxmljs. This memory is NOT in
the v8 heap space and can't be seen and tracked by the js context.
This is what I'm calling "external" memory as in external to v8. You
have to tell v8 about this memory when it's allocated or freed because
it affects the garbage collection heuristics.

There is a parallel idea of memory to be aware of though. There is
the difference between javascript space memory, i.e. the actually buff
object in your script, vs the C array that is allocated to hold the
actual data. This C array is what makes the "internals" of the Buffer
object work. And this is what you're concerned about freeing. But
delete won't help you :) You can see this memory in
process.memoryUsage(), because I'm assuming that reads the full memory
for the process. But the v8 context doesn't see this memory and it
won't go away until the destructor for the Buffer object is called.

On further thought I think ry is right that destroy() is no good. We
really don't want to start being concerned about C memory in js
scripts. Bryan are you actually running into memory issues? If not,
you should probably be okay trusting that this memory will eventually
be collected.

:Marco


On May 4, 2:34 pm, Ryan Dahl <coldredle...@gmail.com> wrote:

[Bryan Berry]

I am not running into memory issues. I was just curious :) 

thanks for the great explanation.

[Camilo Aguilar]

FYI

http://groups.google.com/group/v8-users/browse_thread/thread/10598ec26a5b926c