Referencing or copying Builders

[stefan...@ai-solutions.com]

Hi Kenton,

I have a struct which has a field containing a list of structs of the same type, akin to that of a tree structure. Is it possible to copy or obtain references for the struct Builder instances associated with a specific struct so that I can subsequently initialize lists for each struct in a breadth-first style?

Initially, I attempted to store the message builder corresponding to the child nodes in a kj::Vector then std::move them to a parent nodes kj::Vector, hoping that I could initialize the children list in future iterations. However, I'm finding that when trying to reference a parent by its index, the message builder is instead references the current node's children (which just so happens to be the most recently instantiated instance of the message builder).

Here's some example code that outlines what I'm trying to do: https://gist.github.com/slnovak/2f8b14845019da98566c6fc2776ee186

Thanks so much!

[stefan...@ai-solutions.com]

Hey Kenton,

I came across the thread for Self-contained structures in C++ and it sounds like that what I'm needing to implement is a refcounted-builder. That thread hasn't been updated in a couple of years, so I was curious if that approach would still be valid.

Thanks for your support!

[Kenton Varda]

Hi Stefan,

Cap'n Proto's Readers and Builders both behave as pointers, not value types. They are copyable, but copying them doesn't copy the data, it just copies the pointer. Both actually point into the underlying message buffer, and calling methods on the Reader/Builder manipulates that underlying buffer.

To actually move data from one builder to another, you'd have to do something like:

    // Copy primitive fields like this.

    to.setFoo(from.getFoo());

    to.setBar(from.getBar());

    // Transfer ownership of pointer fields like this.

    to.adoptBaz(from.disownBaz());

    to.adoptQux(from.disownQux());

If you wanted to avoid manually specifying each field, you could write some code based on AnyStruct::Builder instead. Any struct builder type can be converted to AnyStruct::Builder, which then allows you to iterate over the raw data and pointers that make up the struct.

Note that mutating Cap'n Proto messages is awkward due to the memory allocation needs, namely that all objects need to be allocated adjacent in the message buffer, to allow zero-copy output. Because of this, Cap'n Proto is not an ideal format for an in-memory data structure that you need to modify over time. The "self-contained structures in C++" thread you linked to, also knows as "POCS" (plain-old-c-structs), is an idea I'd still like to implement at some point, providing a mutation-friendly (but non-zero-copy) in-memory representation corresponding to Cap'n Proto types. No work has been done on this yet, unfortunately.

-Kenton

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/capnproto.

[Wylder Keane]

Hey Kenton,

Any chance the mutation friendly in-memory representation feature is on the near-term roadmap? Seems like it would make using CapnProto even easier. I've already implemented a wrapper object akin to what you referred to (but using std::shared_ptr) and that works for immutable instances, but it's still impossible to keep an object around and mutate it without leaking memory every time a non-primitive field is set (which is not very obvious from the documentation btw).

Related, is there any way to adopt all of the allocated segments from another MallocMessageBuilder? Effectively, swap the two instances of MallocMessageBuilder?

Thanks,

Wylder

To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.

[Kenton Varda]

Hi Wylder,

On Mon, Aug 8, 2022 at 12:16 PM Wylder Keane <wylder...@gmail.com> wrote:

Hey Kenton,

Any chance the mutation friendly in-memory representation feature is on the near-term roadmap? Seems like it would make using CapnProto even easier. I've already implemented a wrapper object akin to what you referred to (but using std::shared_ptr) and that works for immutable instances, but it's still impossible to keep an object around and mutate it without leaking memory every time a non-primitive field is set (which is not very obvious from the documentation btw).

To be honest, there isn't really a roadmap for Cap'n Proto. I build the features I need when I need them, driven by my other projects. So far I haven't needed this feature.

In the absence of a use case of my own it's sort of hard for me to have intuition about how this should be built. Like, what *kinds* of mutations should be supported efficiently? E.g. Lists should be efficiently appendable probably, but is efficient insertion also needed? When I think about this I tend to land on the conclusion that I would usually want to define mutable structures in my actual programming language and not in Cap'n Proto.

Note that this hypothetical mutation-friendly representation would necessarily not be zero-copy to parse / serialize. Zero-copy and mutation-friendly just are not compatible design goals, unfortunately.

 

Related, is there any way to adopt all of the allocated segments from another MallocMessageBuilder? Effectively, swap the two instances of MallocMessageBuilder?

I'm not sure what you mean here -- adopting segments from one builder to another, vs. swapping them, seem like totally different things. Which is it?

If you want a swap, could you allocate both MessageBuilders on the heap and swap the pointers?

Or do you really want to be able to take one message and embed it within another, without copying? I have wanted that too, but it's difficult in part because far pointers refer to other segments by absolute index of the segment rather than relative, so you can't embed another message wholesale without fixing up the far pointers (unless your destination builder started out empty so that the segments could keep their same numbers). If I were to make one backwards-incompatible change to Cap'n Proto, this is what I'd change, incidentally.

-Kenton

 

To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/0eb01b62-ac7f-44ef-9bc1-f46e799c5be8n%40googlegroups.com.