Message data (blob) from AnyPointer

[John Demme]

Hi-

In the C++ API, is there an (easy) way to get the binary encoded message of an AnyPointer field? Or a generic field? (If I understand, they're encoded the same under the hood.) I don't think anyPointer->getAs<Data>() would work, would it?

The background here is that I'm trying to write a generic proxy for CapnProto messages using CapnProto RPC. Since I'm not re-compiling the proxy itself, it cannot know the possible types for AnyPointer. Since messages don't need any schema data to be traversed (though not interpreted) correctly, this should be possible.

To be more concrete:

interface ProxyChannel {

    send @0 (msg :AnyPointer);

    recv @1 () -> (respMsg :AnyPointer);

}

I'm using :Data instead of :AnyPointer now, but that puts a unnecessary burden on the clients.

~John

[John Demme]

I've been playing with this a bit and the closest I've been able to get is constructing a new message with the AnyPointer at the root. That, however, prepends a struct pointer to the whole thing. I can just chop it off, but I'm thinking there must be a better way.

    auto msg = context.getParams().getMsg(); // AnyPointer
    MallocMessageBuilder messageBuilder;
    messageBuilder.setRoot(msg);
    auto flatWordsArray = messageToFlatArray(messageBuilder);
    auto byteArray = flatWordsArray.asBytes();

~John

[Kenton Varda]

Hi John,

You have to copy into a new message, as you have done.

The reason is, the message tree pointed to by the AnyPointer is not necessarily contiguous in memory. Unlike most other serialization out there, Cap'n Proto objects point to their child objects using pointers that can cross over other object in between. (Most serializations use nested encoding instead, but this doesn't work well with zero-copy.) Generally, objects will be ordered in memory in the order in which they were allocated. Typically programs writing a message will work on one part of the tree at a time, in which case you'll get a depth-first ordering, but that's not guaranteed. If a program switches back and forth between building different parts of the message, the objects will end up interleaved in memory.

So the only way to get an object tree into contiguous memory is to copy it into a new arena -- i.e. a new MessageBuilder.

You could, however, avoid the second copy implied by `messageToFlatArray()`. Instead, before making the copy, use `getMsg().targetSize().wordCount` to see how much memory the message tree takes. Add 1 word for the root pointer. Then pass this to MallocMessageBuilder's constructor as the first segment size. Now copy the message. If you then call messageBuilder.getSegmentsForOutput(), you should find there is only one segment, containing the root pointer followed by the content. So now you can use that data without making an extra copy...

-Kenton

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/7f1b3509-0bac-4854-8256-d85a718031d7%40googlegroups.com.

[John Demme]

Hi All-

I've finally had time this afternoon to work on this and got it working in a way I find satisfactory.

    auto capnpMsgPointer = context.getParams().getMsg();
    KJ_REQUIRE(capnpMsgPointer.isStruct(), "Only messages can go in the 'msg' parameter");
    auto msgSize = capnpMsgPointer.targetSize();
    auto builder = make_unique<MallocMessageBuilder>(msgSize.wordCount + 1, AllocationStrategy::FIXED_SIZE);
    builder->setRoot(capnpMsgPointer);
    auto segments = builder->getSegmentsForOutput();
    KJ_ASSERT(segments.size() == 1);
    auto fstSegmentData = segments[0].asBytes();

and on the way out (this is bidirectional):

        auto segment = kj::ArrayPtr<word>((word*)blob->data(), blob->size() / 8).asConst();
        auto segments = kj::heapArray({segment});
        auto msgReader = make_unique<SegmentArrayMessageReader>(segments);
        context.getResults().getResp().set(msgReader->getRoot<AnyPointer>());

wherein 'blob' is a std::vector<uint8_t> which is semantically guarteed to be 0 mod 8 in length.

This works in a loopback test!

Thanks,

John

To unsubscribe from this group and stop receiving emails from it, send an email to capn...@googlegroups.com.