Pack() to a file

[Phil Rhodes]

Is there no way of using Pack to write directly to a file, or am I missing something?

 

I'm writing data to a file and don't know the eventual size of the data, so I'd prefer to stream it directly to disk rather than put it in a Buffer().

 

Of course I can write to small buffers and zap those into files every so often, but there's a performance shortfall with that.

 

Happy holidays, etc,

 

Phil

[soubok]

On Thu, Dec 24, 2009 at 00:14, Phil Rhodes <phil_...@rocketmail.com> wrote:
> Is there no way of using Pack to write directly to a file, or am I missing
> something?

No, a Pack object can only be connected to a Buffer object.

>
> I'm writing data to a file and don't know the eventual size of the data, so
> I'd prefer to stream it directly to disk rather than put it in a Buffer().
>
> Of course I can write to small buffers and zap those into files every so
> often, but there's a performance shortfall with that.

I need to rethink my Buffer and Pack classes and perhaps replace them
by a StreamBuffer class.

>
> Happy holidays, etc,

Thanks.

>
> Phil
>
> --
> Group jslibs - http://groups.google.com/group/jslibs -
> jsl...@googlegroups.com
> Unsubscribe: jslibs-un...@googlegroups.com

[Phil Rhodes]

> I need to rethink my Buffer and Pack classes and perhaps replace them
> by a StreamBuffer class.

Please don't do that, I've just spent lots of time writing code that uses
them :) Also I wouldn't call the new idea "StreamBuffer". The current Buffer
works a bit like a FIFO to me, so it already feels like a "stream buffer".

Regardless of how you implement Buffer in native code, I think the current
JS API is fine if you can add peek(), poke() and insert() with good
performance (see my notes for Issue 108). The current problem is that my
peek() function on a 512KB Buffer can do about 700 peeks a second on an
Athlon X2 6000+, but this drops to about 700 peeks in THIRTY seconds on a
1MB buffer.

If there is a big performance benefit from it, you could introduce a
file-like pointer, and have all write()s automatically overwrite at that
pointer. I've written a JS constructo to create a faster buffer-like object
that does this, which I'll post here if you're interested. I think you
should be able to do this without breaking API compatibility with the
current Buffer.

The other time-saver would be to implement all the binary writing abilities
of Pack to all binary objects. I understand that there are presumably
native-code reasons for the way it currently works, but it seems to me that
binary things like Buffer, File and Socket should all have a symmetrical API
with the best of the binary methods (readInt, writeInt, readUntil, etc).
Could this be done in a reasonable amount of work?

Thanks also for your notes on event handling.

P