Using vectors of ubyte/uint8 for opaque data in flatcc.

[Marcus Johansson]

Hi!

I would like to use flatbuffers in C using flatcc to pack payload data into an array. In the schema example below, receiver would be used to decide how to interpret the data in receiver_data. Now in flatcc it is easy to create a vector for receiver_data, using the provided macros (eg. yadayada_vec_create and then type_vectormember_add), but when reading the buffer I cannot see a way using flatcc to access the raw array data. I don't want the higher level layer schema to be aware of the flatbuffers schema used for the underlying data passed in receiver_data.

table protostream_hdr {
  receiver:string;
  receiver_data:[ubyte];
}


root_type protostream_hdr;

The same kind of issue seems to be dealt with here:

https://groups.google.com/forum/#!topic/flatbuffers/aP74NvqwNpw

But in that case C++ is used instead and I cannot see the flatcc version of the ->Data() method.

[mikkelfj]

Hi Marcus,

There are some parts of the flatcc API where there could be better support for raw pointer access but it requires some careful thought.

That said, vector types, i.e. those ending in _vec_t are const pointers to the underlying type, so you can access it directly:

https://github.com/dvidelabs/flatcc#types

And you can see its use with a struct in the following piece of test code (mind you this refers so source code as of version 0.3.4, for future ref)

https://github.com/dvidelabs/flatcc/blob/master/test/monster_test/monster_test.c#L512-L522

I actually suspect the test will fail on big endian because it compares testvec_data[i].a unencoded with a decoded short value, but it's been a while.

For the generated reflection flatbuffer interface, you will see vec_t definition as

#define __flatbuffers_define_scalar_vector(N, T)\

typedef const T *N ## _vec_t;\

in https://github.com/dvidelabs/flatcc/blob/master/include/flatcc/reflection/flatbuffers_common_reader.h

Except for byte sized integers, it will only work for little endian platforms, but this is also what you ask for.

There is a also a _mutable_vec_t that can be cast for in-place modification, which is used by heap sort internally:

https://github.com/dvidelabs/flatcc/blob/master/doc/builder.md#sorting-vectors

As to need for abstraction in higher layers, this again is not entirely possible when considering all databytes and big endian systems, but one can add additional macro magic if needed.

Hope this helps.

Mikkel

[mikkelfj]

On Monday, August 29, 2016 at 10:15:11 PM UTC+2, Marcus Johansson wrote:

The same kind of issue seems to be dealt with here:

https://groups.google.com/forum/#!topic/flatbuffers/aP74NvqwNpw

 

BTW:

I only read briefly, but to deal with alignment, you might want to use ulong datatype to ensure any embedded flatbuffer is 8 byte aligned, or a vector an 8 byte struct to provide a meaningful typename.

Flatcc does a lot of work to manage alignment correctly for nested flatbuffers, more so than flatc for C++, I believe, but it does require knowledge of the embedded schema.

[Marcus Johansson]

Thanks for your help, much appreciated!

After some more digging into the documentation and examples, I realized I could simply use the accessor method of the uint vector to access the raw vector. For my use case I will put nested flatbuffers withing that uint vector, so proper alignment of the embedded buffer will be required (as you indicated) - but I also realize that flatcc provides some means to deal with this.

Little vs. big endian will probably not be an issue as such here.

Best Regards,

Marcus