High memory usage when constructing capnp message

[hdh...@york.ac.uk]

Hi,

I am new of capnp but trying to use it in C++ to send messages over a network.

I am trying to construct 100 capnp messages in a loop and store them in a std::vector before I send them.

My schema looks like this:

struct Message {

  field1 @0 : Text;

  field2 @1 : Int64;

}

The majority of my fields are Text, there are 35 fields overall.

I am then constructing the messages like this:

std::vector<kj::Array<capnp::word>> _msgs;

_msgs.resize(100);

MSG_1_CLASS _msg_1_class;

capnp::MallocMessageBuilder messagebuilder;

for (int i=0; i<100; i++) {

        Message::Builder message = messagebuilder.initRoot<Message>();

        message.field1(_msg_1_class.field1.c_str());

        message.field2(_msg_1_class.field2);

        _msgs[i] = capnp::messageToFlatArray(messagebuilder);

}

The problem that I am having with this is that my memory usage is extreamly high (260 KB per message) measured using /usr/bin/top.

Thanks,

Henry

[Kenton Varda]

Hi Henry,

The problem is that you're reusing a single MallocMessageBuilder to build all your messages. See the bullet point here that starts with "Messages are built in 'arena' or 'region' style":

https://capnproto.org/cxx.html#tips-and-best-practices

Each time you call initFoo() (or initRoot()) to init a new struct, you're allocating new space but not freeing the old space. So, each message ends up larger than the one before it.

Instead, create a new MallocMessageBuilder for each iteration (declare it inside the loop). If you want to reuse its backing memory and avoid any allocations, you can allocate a zero'd buffer once and pass it to MallocMessageBuilder's constructor for each iteration.

Maybe we should make initRoot() throw an exception if it is called a second time, to avoid this problem.

-Kenton

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.
Visit this group at https://groups.google.com/group/capnproto.

[Francisco Madueño Chulian]

Hi.

I have the same problem, can you explain what is a  "zero'd buffer"? 

Thanks.