Unknown field retention

[Dmitry Shmorgun]

Hi!

According to https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-sbe.html there is a way to extract new unknown fields from a message and then construct the same or slightly modified message using an outdated scheme and place these fields back.

But how actually this can be done? I don't find any example.

I assume there should be something else than dynamic scheme loading.

Can I go further and just reinterpret a message type A as an message containing a header and a one struct that can be specified as a generic message B, modify and pack back as A? So in this way, I can even not know the scheme of A at all.

Many thanks

[Kenton Varda]

Hi Dmitry,

Unknown fields are automatically copied when you copy a struct -- that is, when you set a struct field to a Reader for the same type, like:

    someBuilder.setMyStructField(someReader.getMyStructField());

If you want to inspect or manipulate the unknown fields, you need to use the "any" API, found in `capnp/any.h`. Any struct Reader can be converted to an AnyStruct::Reader, which allows you to inspect the raw bytes and pointers of the struct.

-Kenton

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/CAMLcW_Ojh22Qv69Dt_bx078PsKr%2B4TYEvyNCzz%3D%2BZdYrbbHhFg%40mail.gmail.com.

[Dmitry Shmorgun]

I tried it, so any message can be reinterpret by a proper generic message that has AnyPointer fields in a scheme. Where AnyPointer can be any construction like List<List<AnyStruct>> for example, so i can serialize, store, manipulate, serialize back any message only knowing a generic structure. It works well.

I saw an example of a list truncation:

   auto orphan = myStruct.disownMyList();

    orphan.truncate(size);

    myStruct.adoptMyList(kj::mv(orphan));

though i haven't found how to do so for AnyList<AnyStruct> structure, so it was necessary to calculate m_filledEntries before the creation of the list:

        auto entriesBuilder = AnyPointer::Builder(r1.getEntries()).initAsListOfAnyStruct(m_entrySize, m_filledEntries, m_filledEntries);

        for (size_t i = 0, j = 0; i < m_entries.size(); ++i) {

            if (!m_entries[i].isBlank) {

                auto dataSection = entriesBuilder[j++].getDataSection();

                std::copy(m_entries[i].buff.begin(), m_entries[i].buff.begin() + m_entrySize, dataSection.begin());

            }

        }

Now i'm trying to  tell a server that uses generic messages that in a struct of a received message there is a key on the byte offset X with the size N. The server reads a special message that convey this information. So the server can keep a data snapshot in a hash key-value storage and manipulate entries by the the byte-key. The problem is how to find the offset and size of the specific key field to encode in the message with settings. It seems all these things are deeply hidden.

Any ideas maybe?

Thanks for your help.

[Kenton Varda]

You can get the offset and size of a field via the schema API. Check out schema.h and schema.capnp.

    auto fieldDef = KJ_ASSERT_NONNULL(
        capnp::Schema::from<MyStruct>().findFieldByName("myField")).getProto().getSlot();
    auto offset = fieldDef.getOffset();  // multiply by field size to get byte offset
    switch (fieldDef.getType()) { ... }

-Kenton

--

You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/4930b3a4-f153-458c-b445-d7b638d98366%40googlegroups.com.

[Dmitry Shmorgun]

Thanks!

Previously i tried to reinterpret messages knowing a generic scheme, can i somehow iterate over additional lists that aren't in my generic scheme?

For example i have a message with a few consecutive Lists of any structs, let's say 3 of them and in an incoming message there are 2 more lists of any structs, 5 total:

can i iterate over them and then later pack them in the same manner back? How it can be done?

To unsubscribe from this group and stop receiving emails from it, send an email to capn...@googlegroups.com.

[Kenton Varda]

Hi Dmitry,

Sorry but I do not understand the question.

-Kenton

To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/c0182786-760a-495a-b286-bc7e52d964d1%40googlegroups.com.