a simple pack

[Jose Silva]

hi, i want to pack an array in a binary string.

Something like

ruby-1.9.2-p136 :001 > [1,"10.0.0.1","foo"].pack 'l! a4 Z*'
=> "\x01\x00\x00\x00\x00\x00\x00\x0010.0foo\x00"

is that a working way to do that? I tried some npm modules like
msgpack, jspack and none of them, for that use case, works..

[Ben Noordhuis]

No built-ins (yet - might be a worthwhile addition). Buffer objects
have (undocumented) utility methods to write out integers and floats:

https://github.com/joyent/node/blob/02699a3/lib/buffer.js#L747

[Jose Silva]

hi Ben, thanks.

worthwhile, no, i would say mandatory to implement any kind of binary
protocol. no?


On Jun 22, 2:32 pm, Ben Noordhuis <i...@bnoordhuis.nl> wrote:

[Ben Noordhuis]

On Wed, Jun 22, 2011 at 14:38, Jose Silva <pela...@gmail.com> wrote:
> hi Ben, thanks.
>
> worthwhile, no, i would say mandatory to implement any kind of binary
> protocol. no?

I'm on the fence (but would support a well-written, well-tested
pack/unpack implementation). The bare essentials are already there and
most people that need a fast protocol parser drop to C++ anyway.

@nodejs-dev: binary pack / unpack in core - good idea, bad idea? I say good.

[Jann Horn]

Am Mittwoch, den 22.06.2011, 05:38 -0700 schrieb Jose Silva:
> hi Ben, thanks.
>
> worthwhile, no, i would say mandatory to implement any kind of binary
> protocol. no?

Well, JSON is in the core (actually, it's in v8), is supported nearly
everywhere and can be used for serializing stuff. If you need better
compression or so, you can always use a specialized module (part of
nodes philosphy is a small core library with many good modules around
it):
http://search.npmjs.org/
For example, there are things like msgpack and dht-bencode.

Jann

[Jose Silva]

both msgpack and dht-bencode don't support that you provide a template
as directive to the binary encoding. So if both are good for specific
usages, they are not generic enough to support many use cases.

>  signature.asc
> < 1KViewDownload

[bradley.meck]

I think it would be good to have an extensible pack implementation that allows templating. I don't think this is something that should be in core though (we generally want people to be able to avoid using binary data manipulation with this complexity I would think). That being said, I would avoid string based formatting as it is hard to extend (though fast to dev and somewhat readable) in whatever module provides this, having that on top of an extensible module is much easier than the other way around.

[Matt]

On Wed, Jun 22, 2011 at 10:35 AM, bradley.meck <bradle...@gmail.com> wrote:

I think it would be good to have an extensible pack implementation that allows templating. I don't think this is something that should be in core though (we generally want people to be able to avoid using binary data manipulation with this complexity I would think). That being said, I would avoid string based formatting as it is hard to extend (though fast to dev and somewhat readable) in whatever module provides this, having that on top of an extensible module is much easier than the other way around.

When you need it, you can't really avoid it.

Pack() is becoming fairly standard across languages now (in terms of using as string as the template language). I think it'd be worth having - I've had to "pack/unpack" my own stuff recently and it's a pain.

Though I kind of agree that it can't be in core, because unpack() would have to be a method on String, which obviously node isn't going to extend.

I'm thinking of an API something like:

  var packer = require('packer');

  var buf = packer.pack(template, list, of, items);

  var list_of_items = packer.unpack(template, buf);

Matt.

[billywhizz]

i put this binary packer together way back when (not even sure if it
still works) and it does something similar to what you need. it needs
some work to streamline it though as it was very much a first pass at
implementing something like this...

https://github.com/billywhizz/node-binary

you do this to pack:

bin.pack([
{"int": 1},
{"int16": 2},
{"int32": 3},
{"string": "goodbye"}
], buff, 0);

and this to unpack:
var tmp = bin.unpack("onNs7", 0, buff); // returns an array of [int,
int16, int32, string(7)]



On Jun 22, 4:18 pm, Matt <hel...@gmail.com> wrote:

[Isaac Schlueter]

This should absolutely be done in a userland module.

https://github.com/joyent/node/wiki/node-core-vs-userland

If someone wants to implement a good api for this, and support it, and
it rises to fairly universal acceptance within the community, and it
follows node's coding guidelines, and it is well tested and
documented, and Ryan et al feel that it's something that fits nicely
with the other network-programming utilities in node, then it can
always be pulled into core.

On the other hand, if it's awesome, and in npm, then you might not
actually need it to be in core anyhow. The Node Way is that it should
be *really hard* to justify putting something into node core, and
*really easy* to publish them in userland.

> --
> You received this message because you are subscribed to the Google Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com.
> To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/nodejs?hl=en.
>
>

[Matt]

I think the one argument that may eventually come up for putting something in core is if you want more protocol support (right now you only really natively support http) and if that protocol requires binary stuff. Then it'd make sense to have it in node to do the decoding.

For now however, I entirely agree.

[Matt]

Yeah I looked at it, but it doesn't support things like network strings ("N/a*" in perl terms) or lists of network strings ("(N/a*)*" in perl terms) which is what I wanted it for.

[Tim Smart]

What about BSON? Does that count?

Tim.

[Matt]

No. It doesn't do streaming.

With a network string in a file I can read the length and then create a readStream that only reads that part of the file. I can even just pipe() it directly over a network (which I do in haraka).

Matt.

[Marcus Westin]

I've got incomplete implementations of pack/unpack taken from php-js
and patched in https://github.com/marcuswestin/std.js (pack.js and
unpack.js). Had to fix a bug in unpack for my purposes.

They have not been thoroughly tested and I doubt they support all the
syntax you want. However, they did the job for https://github.com/marcuswestin/node-kafka.

Cheers!
Marcus

On Jun 23, 10:37 pm, Matt <hel...@gmail.com> wrote:
> No. It doesn't do streaming.
>
> With a network string in a file I can read the length and then create a readStream that only reads that part of the file. I can even just pipe() it directly over a network (which I do in haraka).
>
> Matt.
>
> On 23 Jun 2011, at 20:35, Tim Smart <t...@fostle.com> wrote:
>
>
>
> > What about BSON? Does that count?
>
> > Tim.
>
> > On 23 June 2011 06:55, Matt <hel...@gmail.com> wrote:
> > Yeah I looked at it, but it doesn't support things like network strings ("N/a*" in perl terms) or lists of network strings ("(N/a*)*" in perl terms) which is what I wanted it for.
>

> > For more options, visit this group athttp://groups.google.com/group/nodejs?hl=en.

>
> > --
> > You received this message because you are subscribed to the Google Groups "nodejs" group.
> > To post to this group, send email to nod...@googlegroups.com.
> > To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.

> > For more options, visit this group athttp://groups.google.com/group/nodejs?hl=en.

[Alan Gutierrez]

http://bigeasy.github.com/node-packet/

This should do what you want. Under development, but the packing and
unpacking noted above is implemented.

--
Alan Gutierrez - http://twitter.com/bigeasy - http://github.com/bigeasy