Using Cap'n Proto structures directly as immutable data structures

[ezy...@mit.edu]

I understand the standard guidance for the C++ classes that Cap'n Proto generates is that you should mirror them with in-memory structs that you can mutate. So Cap'n Proto is first and foremost a serialization library, and only incidentally a data structure description language (at least, until https://groups.google.com/forum/#!searchin/capnproto/refcounted%7Csort:date/capnproto/Reg0wInHBdY/ftJUP3iTCQAJ happens). Fine. But what if I plan to treat my data immutably? The appeal of skipping out on the conversion boilerplate is quite great. Are there still other reasons why I might still want to schlep my Cap'n Protos into another format? Some I can think of:

* Reads will still be slightly slower, since all of the offset calculation and security validation. Intuitively, how would you tell if this is too much?

* In general, you'd have to assume that the struct is dynamically sized, and thus you cannot ever stack allocate it. (You would want some sort of refcounted version of MyStruct::Reader, to manage the lifetime of the memory?)

Are there other considerations?

[Kenton Varda]

I actually use Cap'n Proto for in-memory-only data structures all the time. It doesn't work for things that frequently change, but for any kind of write-once read-many data it works pretty well.

I recently added `capnp::clone()` to the library as a way to get a kj::Own<> of a capnp structure so that you don't have to mess around with the MessageReader yourself. You might find it handy. (It's not in a release version yet, though.)

capnp::clone() also takes advantage of the fact that pointer bounds are checked during the copy, so after the copy the bounds checks can be skipped. So accessing a cloned structure is somewhat faster than a normal MessageReader.

-Kenton

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/d181151b-4900-457e-bca9-efa496eb318d%40googlegroups.com.

[Ian Denhardt]

Quoting ezy...@mit.edu (2019-09-09 23:15:45)

> So Cap'n Proto is first and foremost a serialization library, and only
> incidentally a data structure description language (at least, until
> https://groups.google.com/forum/#!searchin/capnproto/refcounted%7Csort:
> date/capnproto/Reg0wInHBdY/ftJUP3iTCQAJ happens).

FWIW, the Haskell implementation (of which I am the author) already
implements a "plain-old-ADTs" API in the spirit of POCS, and I default
to using it, since it's much more ergonomic (and originally my personal
interest in capnproto was more about the rich semantics of the rpc
protocol than anything related to performance).

[Edward Yang]

On Tuesday, September 10, 2019 at 1:05:48 PM UTC-4, Kenton Varda wrote:

I recently added `capnp::clone()` to the library as a way to get a kj::Own<> of a capnp structure so that you don't have to mess around with the MessageReader yourself. You might find it handy. (It's not in a release version yet, though.)

Neat! Though, I guess that means I'll have to figure out how to vendor capnp master in my cmake :( Maybe I can plunder the code into a header file of my own, so I can keep using Ubuntu's packaged capnp...

> FWIW, the Haskell implementation (of which I am the author) already
> implements a "plain-old-ADTs" API in the spirit of POCS

Neat! I would have loved to have done this in Haskell, but I'm working on a program where memory usage is pretty important, so no Haskell for me :(