[ANN] go-msgpack: A rich encoding/decoding library for msgpack, with support for net/rpc codec.

[Ugorji Nwoke]

Announcing go-msgpack, a rich Go library for encoding/decoding to msgpack binary format, with support for net/rpc communication.

https://github.com/ugorji/go-msgpack

http://gopkgdoc.appspot.com/pkg/github.com/ugorji/go-msgpack/msgpack

It provides features similar to encoding packages in the standard library (ie json, xml, gob, etc).

Supports:

  - Standard Marshal/Unmarshal interface.

  - Standard field renaming via tags

  - Encoding from any value (struct, slice, map, primitives, pointers, interface{}, etc)

  - Decoding into a pointer to any non-nil value (struct, slice, map, int, float32, bool, string, etc)

  - Decoding into a nil interface{} (big)

  - Handles time.Time transparently (using RFC3339 format).

Usage

  // v can be interface{}, int32, bool, map[string]bool, etc

  dec = msgpack.NewDecoder(r)

  err = dec.Decode(&v) 

  

  enc = msgpack.NewEncoder(w)

  err = enc.Encode(v) 

  

  //methods below are convenience methods over functions above.

  data, err = msgpack.Marshal(v) 

  err = msgpack.Unmarshal(data, &v)

  //RPC Communication

  conn, err = net.Dial("tcp", "localhost:5555")

  rpcCodec := msgpack.NewRPCCodec(conn)

  client := rpc.NewClientWithCodec(rpcCodec)

  ...

Why?

I was initially looking at different binary serialization formats for an application I'm developing. I looked at Thrift, Protocol Buffers, BSON, Msgpack, etc. 

I finally decided on msgpack:

- compact

- supports basic types (just like JSON. Numbers, Bool, Null, Bytes/String, Map, List) from which other data structures else can be assembled

- raw bytes support (which can represent binary data or strings)

- no schema needed (just like JSON)

- cross-language support

- has pretty good mindshare

Unfortunately, the Go library on the msgpack.org site is old, does not build, and is not "smart" like the encoding/json package. Specifically, it doesn't allow me decode into a typed object (e.g. struct, bool, etc). 

I wrote go-msgpack to give the same conveniences we've gotten used to (spoiled by using the encoding/json package), while being really performant. 

Summary of my testing:

  - Decodes significantly faster (120% faster) and encodes only slightly slower (20% slower) than encoding/json

  - Uses less disk space (40% less)

  - May not require compression (compression only gave a 10% reduction in size).

    Since compression/decompression time may be significant, this may be an important win.

Hope folks use it and enjoy using it. I know I will. Please feel free to send me feedback.

[Remi Gillig]

Great work ! I'm really interested in compact binary formats, I'm sure I will use this in the future. Kudos for RPC support too, even the C# build of msgpack does not support it (according to their website).

Anybody working on Simple Declarative Language [1] support too? Seems interesting too.

Remi Gillig.

[1]  http://sdl.ikayzo.org/display/SDL/Home 

[Damian Gryski]

Le mercredi 11 avril 2012 19:31:01 UTC+2, Ugorji Nwoke a écrit :

Announcing go-msgpack, a rich Go library for encoding/decoding to msgpack binary format, with support for net/rpc communication.

https://github.com/ugorji/go-msgpack

http://gopkgdoc.appspot.com/pkg/github.com/ugorji/go-msgpack/msgpack

   This is great.  I did some work a while ago to bring the 'official' library version up-to-date, but my pull request sat unanswered for a while.  (I certainly wasn't the first to do this work, as a glance at the list of pull requests will show..)  I also have some work pending (i.e, another pull request) that fixes a number of bugs in the msgpack-rpc package, but that library is also incomplete and broken.  I had started a go-compatible msgpack un/marshal library at http://github.com/dgryski/dmsgpack , but wasn't really sure where to go with it since I don't use the library myself.

   Hopefully your library will become the more popular msgpack version and it will stay more supported and current :)

   Damian 

[Ugorji Nwoke]

Thanks.

Just to note that the RPC support basically builds a rpc.ClientCodec and rpc.ServerCodec implementation which can be used with the standard net/rpc package. Look at the code, it's basically one-line implementations of all the methods of those interfaces. 

This is opposed to the msgpack-rpc project which tries to build a whole RPC framework around msgpack. This takes the more prudent path IMO.

[Kyle Lemons]

I'd love to integrate this into go-rpcgen.  It already supports json, gob, and protobuf, and since you already wrote the net/rpc codec it's probably almost a trivial thing to add.  Thanks!

On Wed, Apr 11, 2012 at 10:31 AM, Ugorji Nwoke <ugo...@gmail.com> wrote:

Announcing go-msgpack, a rich Go library for encoding/decoding to msgpack binary format, with support for net/rpc communication.

https://github.com/ugorji/go-msgpack

http://gopkgdoc.appspot.com/pkg

 

Luke

[Jake Good]

I had started working on my own version internally because of theirs being broke.. I like your implementation a lot better :)

The one problem I ran into was using the msgpack RAW spec for strings as key values in maps... as Go does not allow an array/slice to be a key in a map, you have to have some form of standard conversion.

The spec allows for this but I had to detect such situations in code and convert to strings, but this broke down a nice decoupled implementation as you have to make a static decision to treat it as a string or tightly couple an implementation of what to do in those cases as a func call...

Either way... nice work!

[Ugorji Nwoke]

Hi Damian,

I saw your code while I was looking for existing implementations. Since it depended on the unsupported library from msgpack.org site, and basically does a 2-pass (allocating memory again), and I felt that the underlying msgpack.org library could be improved to support conveniences provided by encoding/json, etc, I decided to write one.

I'm hoping to use the library for a project I'm working on that uses leveldb, and I hope the benefits make compression unnecessary and gives really good performance.

BTW, I looked at BSON also. Kudos to Gustavo.

[Jake Good]

Also, you should probably include:

To install:

go get github.com/ugorji/go-msgpack/msgpack

in your README... I mean, yes, it's a common Go idiom to install this way, but your directory layout requires the extra /msgpack in the go get call...

Just a suggestion :)

[Ugorji Nwoke]

Thanks.

The way the code works actually accomodates this nicely.

If decoding into a map[interface{}]XXX, and we see []byte coming in for the key, we assume it must be a string (because you only use raw bytes for strings or raw bytes, and a map key cannot be a slice).

For other disambiguations, we use options i.e. when you make a decoder, you set options of how to treat somethings like

- should I use map[string]interface{} or map[interface{}]interface{}?

- should I treat raw bytes as a string, and where? e.g. in literals, slice elements, map values, etc

Taken together, this actually allows decode in a way that is feature-compatible with JSON and its restrictions.

P.S. I know some folks dislike bit masks, but I didn't want to use a struct, and we're programmers ;)

[Damian Gryski]

Le mercredi 11 avril 2012 20:19:58 UTC+2, Ugorji Nwoke a écrit :

This is opposed to the msgpack-rpc project which tries to build a whole RPC framework around msgpack. This takes the more prudent path IMO.

   However, having cross-language rpc support is nice.  That for me is one reason to support msgpack's rpc framework.  Not all the implementations are a s poor as the Go one :)

   Damian

[Ugorji Nwoke]

Dude, I had this in my checklist, and still forgot.

Doing it now.

Thanks.

[Ugorji Nwoke]

Yes, Please do and lemme know if you see any issues. I've banged on it for a few days and tried to throw different input at it, but it'd be nice for it to be used and stressed in bigger environments.

On Wednesday, April 11, 2012 2:20:29 PM UTC-4, Kyle Lemons wrote:

I'd love to integrate this into go-rpcgen.  It already supports json, gob, and protobuf, and since you already wrote the net/rpc codec it's probably almost a trivial thing to add.  Thanks!

[Damian Gryski]

Finally found some free time: a msgpack-rpc compatible net/rpc codec.

https://github.com/dgryski/go-msgpack/blob/master/msgrpc.go

[desaia...@gmail.com]

hi Ugorji

if I increase i loop 0 to 400.. or more, MsgPack or Gob takes too long to decode same message ( for i:=0 200 usec, if i=100 loop few in 200 usec few takes 6 millisec )

Please help

func main() {

    t1:=time.Now()

    wg := &sync.WaitGroup{}

    wg.Add(400)

    for i:=0;i<400;i++ {

        go invoke(wg, time.Now(), i, encodeddata)

    }

    wg.Wait()

    

    fmt.Printf("Total Time - %s\n", time.Since(t1))

}

func invoke(wg *sync.WaitGroup, t time.Time, i int, dat []byte){

    fmt.Printf("%s > %s > %s\n", i, t, time.Now())

    DecodeMsgPack(dat)

    wg.Done()

}

func DecodeMsgPack(b []byte) {

    var mh1 codec.MsgpackHandle 

    mh1.MapType = reflect.TypeOf(map[string]interface{}(nil))

    

    dec := codec.NewDecoderBytes(b, &mh1)

    c := new(Ux.MyStructl)

    dec.Decode(&c)

    //return c

[Dave Cheney]

Please see my reply to your duplicate post.