C++: passing around a struct, and Builder<->Reader

366 views
Skip to first unread message

Topher Cawlfield

unread,
Mar 29, 2021, 7:36:11 PM3/29/21
to Cap'n Proto
I'm new to Cap'nProto, but am trying to use it as a replacement for structs in an existing project. I'm needing to serialize/deserialize obviously, but also just access the data within various programs. I'm having a very hard time keeping a MessageBuilder, Builder, and Reader in a class.

It would be nice if I could deserialize a CapnProto struct in one method/function, modify it in another, read it in a third, and serialize in a fourth. But I'm stuck turning a PackedFdMessadeReader into a MessageBuilder (I don't mind any few memcpy's needed), and also creating a Builder instance as a member variable.

I'm attaching my closest attempt at this. But it fails to print out the original data value -- I get a null string -- and it fails to write the modified value out.

Any suggestions?

Topher

mystruct.capnp:
@0xed859a09d409be91;

struct MyStruct {
  foo @0 :Text;
}

main.cpp:
#include <string>
#include <iostream>
#include <capnp/message.h>
#include <capnp/serialize-packed.h>
#include "mystruct.capnp.h"

class MyClass {
private:
capnp::MallocMessageBuilder m_message; // Or capnp::MessageBuilder *m_message maybe

public:
MyStruct::Builder m_ms;

MyClass() :
m_message ()
, m_ms (m_message.initRoot<MyStruct>()) // sometimes segfaults here but absolutely required
{
m_ms = m_message.initRoot<MyStruct>(); // sometimes helps
}

void deserializeFrom(std::string filename) {
int fd = 0;
::capnp::PackedFdMessageReader msg(fd);
auto reader = msg.getRoot<MyStruct>();
m_message.setRoot(reader); // This leads to future problems with m_ms
}

void modify() {
m_ms.setFoo("oh joy");
}

void serializeTo(std::string filename) {
int fd = 1;
writePackedMessageToFd(fd, m_message);
}
};

int main() {
MyClass c;
c.deserializeFrom("some.capnp");
std::cerr << "foo is " << c.m_ms.getFoo().cStr() << std::endl; // read things occasionally
c.modify();
c.serializeTo("next.capnp");
}

Kenton Varda

unread,
Mar 29, 2021, 10:55:42 PM3/29/21
to Topher Cawlfield, Cap'n Proto
Hi Topher,

Unfortunately, Cap'n Proto is not well suited to representing mutable data structures in-memory. Generally, MessageReaders are read-only, and MessageBuilders are write-only. While technically you can modify builders over time, the memory allocation patterns this tends to lead to are not ideal. This is an unavoidable consequence of the memory allocation approach needed to support zero-copy serialization: we need to allocate memory in a contiguous space in order to be able to write it all out at once, but this means memory fragmentation is impossible to avoid when data changes shape over time.

This is covered a bit in the "Best practices" section of the docs here: https://capnproto.org/cxx.html#tips-and-best-practices

A feature I've long wanted to add to Cap'n Proto is support for generating appropriate "plain old C++ structs" (POCS) that mirror the Cap'n Proto structs, with the ability to copy from a MesasgeReader into a POCS, and from POCS to a MessageBuailder. Using POCS would mean you have to perform at least one copy (not zero-copy anymore), but may be more convenient in some use cases, especially when the structure will be modified in-memory many times.

-Kenton

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/6ce2d01c-43d4-44a6-9d26-0b3147742f55n%40googlegroups.com.

Topher Cawlfield

unread,
Mar 30, 2021, 2:07:23 PM3/30/21
to Cap'n Proto
Thanks for the reply. That perspective is very helpful. I've been wanting a system like Cap'n Proto for synchronizing a variety of configuration data between systems. I was on the fence about just copying the config data back and forth to plain old C++ structs, but when I first read the tips and best practices section, the first bullet point made me think it would be great to just pass around the Reader and avoid the extra copying and name-mapping code. But since I have an existing code base, not everything is going to be read-only. Thus the need for both getFoo and setFoo. My structs have about a hundred names each already, so going back and forth isn't trivial in terms of hand-written code.

But I'll start with that, and when I can I'll look into what it would take to implement generated "C++ structs" and the conversion logic (for where benefits of convenience outweigh those of speed).

For my project, I've also been considering FlatBuffers and ProtocolBuffers. The two main drawbacks to Cap'n Proto for me are this whole issue and also the enforcement of camelCase which forces name mismatches with existing code. I've also run into some rough weather with pycapnp. Still I believe the drawbacks of FlatBuffers and Protobufs are even more severe, so I'm still plowing ahead with Cap'n Proto. You have my heartfelt thanks for all this, by the way!!

Topher

Kenton Varda

unread,
Mar 30, 2021, 2:17:35 PM3/30/21
to Topher Cawlfield, Cap'n Proto
FWIW someone (not me) wrote this plugin to generate POCS:


I haven't used it nor carefully reviewed it yet, but based on the readme it looks pretty thorough. Note it uses std types rather than KJ types which may or may not be to your liking.

-Kenton

Reply all
Reply to author
Forward
0 new messages