MsgPack vs. Marshal
Neil Conway
for serializing tuples stored in dbm tables and sent over channels.
MsgPack has the following pros/cons, in my view:
Pros:
(A) _slightly_ faster than Marshal [1]; I doubt the performance
difference would be significant compared to, say, system call overhead
or network latency
(B) serialized values can be exchanged between programs written in
different languages
(C) serialization format is more compact than Marshal's, sometimes
significantly so (typically for small integers, short strings, etc.)
Cons:
(D) cannot serialize arbitrary Ruby objects. Unhappily, it silently
produces an *incorrect* serialization for some values it cannot
serialize -- e.g., Symbols are converted to Strings
(E) additional gem dependency (not important)
(D) is the most significant point in my opinion, so I'd like to
propose we switch to using Marshal.
Note that for both MsgPack and Marshal, we can't currently dump a Bud
tuple directly, because Bud tuples are instances of an anonymous class
defined by Ruby's "Struct" machinery. For both MsgPack and Marshal
this can be solved by converting a tuple into an array of field values
and dumping that. This isn't ideal (e.g., column name accessors for
nested tuple values stored in dbm tables are lost); however, it
applies to both serialization methods.
Neil
Joe Hellerstein
This should be easy to isolate and benchmark so I say give it a whirl. A day's work at best. Yes?
J
Neil Conway
<helle...@cs.berkeley.edu> wrote:
> This should be easy to isolate and benchmark so I say give it a whirl. A day's work at best. Yes?
Sure, I'll take a look this week.
Neil
Neil Conway
Interestingly, there does seem to be a noticeable performance
difference between Marshal and MsgPack. Routing 16,000 messages around
a ring of 20 Bud agents using channels (see attached benchmark
program) requires ~7.16 seconds using Marshal and ~6.85 seconds using
MsgPack (mean of 20 runs, little variance between runs). The
performance difference remains for messages from 25 to 400 bytes in
length.
On that basis I'm inclined to stick with MsgPack for now. The Marshal
code can be found in the "marshal-channels" branch; I'll checkin the
ring benchmark shortly, since it might be worth seeing if we can
improve performance for this test case further.
Neil
P.S. The timing results above are for Ruby 1.9.3-p125. In case
anyone's curious, Ruby 1.8.7 + marshal takes ~11.4 seconds; Ruby 1.8.7
+ marshal + old Bud runtime takes ~30.17 seconds. So we're making
progress, at least :)
Joe Hellerstein
J
> <ring.rb>