Vector of tables, flatbuffers C

889 views
Skip to first unread message

Vadtec

unread,
Mar 17, 2021, 2:04:05 PM3/17/21
to FlatBuffers
Greetings,

I'm experimenting with flatbuffers, in C, just getting a feel for it. I'm having a weird issue. Well, a couple more than likely.

I *think* I'm building the flatbuffers and adding all the correct components in the correct manner. However, when I try to read a vector of tables, _vec_len(T) is returning size 0, even though when I hexdump the buffer I can clearly see the said tables being added to/removed from the buffer.

I've used more methods of getting the vector of tables to be readable than I can count. Most have ended with type mismatch compile errors. None of the ones that have actually compiled have resulted in _vec_len(T) returning anything other than 0.

I've also tried using _vec_at(0), but this causes an assertion to fail stating that the given index is out of range.

At this point, I'm either: a) missing something really simple, b) building the flatbuffer completely wrong, or c) making assumptions about how flatbuffers are built that are just plain incorrect.

I have attached my schema and source code, as well as debugging output from the program.

The line that seems to be failing is line 115 in the C code. Either that, or the entire block starting at line 37 is wrong.

By my understanding, using _start(B) and _end(B), everything that gets added to said component of the schema should be added to the buffer and placed into the v-tables for access automatically.

So, I'm not sure where to go. Can anyone point to what I'm doing wrong/misunderstanding? Any help is greatly appreciated.



(I tried attaching the files, but I kept getting an error.)

flatbuffers_test.fbs
------------------------------------------------------------



file_identifier "TEST";

table Ping {
    sent:uint;
}

table Pong {
    sent:uint;
}

union Type {
Ping,
Pong,
}

enum ServerTypes:ubyte {
mach = 0,
cnc,
mill,
press,
drill,
controller,
unknown,
}

table TransitHop {
serverID: uint64;
serverType: ServerTypes;
}

table TransitHeader {
from: TransitHop;
to: TransitHop;
hops: [TransitHop];
}

table TransitSequence {
    transitID:  uint64;
begin:      uint8;
current:    uint8;
total:      uint8;
}

table TestRoot {
type: Type;
sequence: TransitSequence;
routing: TransitHeader;
}

root_type TestRoot;



flatbuffers_test.c
--------------------------------------------------



#include <stdio.h>
#include "flatcc/support/hexdump.h"
#include "flatbuffers_test_builder.h"

// This allows us to verify result in optimized builds.
#define test_assert(x) do { if (!(x)) { assert(0); return -1; }} while(0)

int main(void) {

    flatcc_builder_t builder, *B;
    B = &builder;
    void *buff = NULL;
    size_t size;
    FILE *fh = NULL;

    flatcc_builder_init(B);

    printf("TestRoot - Routing\n\n");

    TestRoot_start_as_root(B);
    {
        // test_root type
        {
            Ping_ref_t testType = Ping_create(B, 0); // the 0 here would be the time the ping is sent

            TestRoot_type_add(B, Type_as_Ping(testType));
        }

        // test_root sequence
        {
            TransitSequence_ref_t sequence = TransitSequence_create(B, 1, 1, 1, 1);

            TestRoot_sequence_add(B, sequence);
        }

        // test_root routing
        {
            TransitHeader_start(B);
            {
                TransitHop_ref_t hopFrom = 0;
                TransitHop_ref_t hopTo = 0;
                TransitHeader_hops_start(B);
                {
                    hopFrom = TransitHop_create(B, 1, ServerTypes_mach);
                    TransitHop_create(B, 2, ServerTypes_controller);
                    TransitHop_create(B, 3, ServerTypes_controller);
                    hopTo = TransitHop_create(B, 4, ServerTypes_cnc);
                }
                TransitHop_ref_t hops = TransitHeader_hops_end(B);

                TransitHeader_from_add(B, hopFrom);
                TransitHeader_to_add(B, hopTo);
                TransitHeader_hops_add(B, hops);
            }
            TransitHeader_ref_t routing = TransitHeader_end(B);

            TestRoot_routing_add(B, routing);
        }
    }
    TestRoot_end_as_root(B);

    buff = flatcc_builder_finalize_aligned_buffer(&builder, &size);

    printf("flatbuffer size: %ld\n", size);

    TestRoot_table_t test_root = TestRoot_as_root(buff);

    test_assert(test_root != 0);

    int type_present = TestRoot_type_is_present(test_root);
    int sequence_present = TestRoot_sequence_is_present(test_root);
    int routing_present = TestRoot_routing_is_present(test_root);

    printf("    type present: %d\n", type_present);
    if (type_present == 1) {
        Type_union_type_t type = TestRoot_type_type(test_root);
        printf("\ttype: %s\n", Type_type_name(type));
    }

    printf("sequence present: %d\n", sequence_present);
    if (sequence_present == 1) {
        TransitSequence_table_t sequence = TestRoot_sequence(test_root);
        printf("\ttransitID: %ld\n", TransitSequence_transitID(sequence));
        printf("\t    begin: %d\n", TransitSequence_begin(sequence));
        printf("\t  current: %d\n", TransitSequence_current(sequence));
        printf("\t    total: %d\n", TransitSequence_total(sequence));
    }

    printf(" routing present: %d\n", routing_present);

    if (routing_present == 1) {
        TransitHeader_table_t routing = TestRoot_routing(test_root);

        TransitHop_table_t from = TransitHeader_from(routing);
        TransitHop_table_t to = TransitHeader_to(routing);
        TransitHop_vec_t hopsTable = TransitHeader_hops(routing);

        test_assert(hopsTable != 0);

        ServerTypes_enum_t serverTypeFrom = TransitHop_serverType(from);
        ServerTypes_enum_t serverTypeTo = TransitHop_serverType(to);

        printf("\tfrom: %ld\t%s\n", TransitHop_serverID(from), ServerTypes_name(serverTypeFrom));
        printf("\t  to: %ld\t%s\n", TransitHop_serverID(to), ServerTypes_name(serverTypeTo));

        int hops_present = TransitHeader_hops_is_present(routing);

        if (hops_present == 1) {
//            // this causes [Assertion `flatbuffers_vec_len(vec) > (i) && "index out of range"' failed.] ?????
//            TransitHop_table_t hopsTableZero = TransitHop_vec_at(hopsTable, 0);
//
//            test_assert(hopsTableZero == 0);


            size_t hops_vec_len = TransitHop_vec_len(hopsTable);
            printf("\thops: %lu\n", hops_vec_len);

            for (size_t i = 0; i < hops_vec_len; i++) {
                printf("\t\t%lu\n", i);
            }
        }
    }

    fh = fopen("/tmp/fbhd-test1", "w");

    if (fh == NULL) {
        printf("Cannot open /tmp/fbhd-test1 for writing\n\n");
        return -1;
    } else {
        hexdump(NULL, buff, size, fh);
    }

    fclose(fh);

    flatcc_builder_aligned_free(buff);

    return 0;
}




debug1.txt
--------------------------------------------------



Program output with the vector of tables being added to the flatbuffer:

TestRoot - Routing

flatbuffer size: 188
    type present: 1
        type: Ping
sequence present: 1
        transitID: 1
            begin: 1
          current: 1
            total: 1
 routing present: 1
        from: 1 mach
          to: 4 cnc
        hops: 0



00000000  0c 00 00 00 54 45 53 54  00 00 00 00 5c ff ff ff  |....TEST....\...|
00000010  01 00 00 00 70 00 00 00  5c 00 00 00 04 00 00 00  |....p...\.......|
00000020  7a ff ff ff 0c 00 00 00  3c 00 00 00 08 00 00 00  |z.......<.......|
00000030  00 00 00 00 96 ff ff ff  04 00 00 00 00 00 00 00  |................|
00000040  01 00 00 00 a6 ff ff ff  03 00 00 00 00 00 00 00  |................|
00000050  05 00 00 00 b6 ff ff ff  02 00 00 00 00 00 00 00  |................|
00000060  05 00 00 00 cc ff ff ff  01 00 00 00 00 00 00 00  |................|
00000070  00 00 00 00 e8 ff ff ff  01 00 00 00 00 00 00 00  |................|
00000080  01 01 01 00 fc ff ff ff  04 00 04 00 0c 00 0f 00  |................|
00000090  04 00 0c 00 0d 00 0e 00  06 00 0c 00 04 00 08 00  |................|
000000a0  0d 00 04 00 0c 00 0a 00  10 00 08 00 0c 00 04 00  |................|
000000b0  0c 00 14 00 04 00 08 00  0c 00 10 00              |............|





Program output without the vector of tables being added to the flatbuffer:

TestRoot - Routing

flatbuffer size: 74
    type present: 1
        type: Ping
sequence present: 1
        transitID: 1
            begin: 1
          current: 1
            total: 1
 routing present: 0



00000000  0c 00 00 00 54 45 53 54  00 00 00 00 cc ff ff ff  |....TEST........|
00000010  01 00 00 00 18 00 00 00  04 00 00 00 e8 ff ff ff  |................|
00000020  01 00 00 00 00 00 00 00  01 01 01 00 fc ff ff ff  |................|
00000030  04 00 04 00 0c 00 0f 00  04 00 0c 00 0d 00 0e 00  |................|
00000040  0a 00 10 00 04 00 08 00  0c 00                    |..........|





Program output when trying to use _vec_at(0):

TestRoot - Routing

flatbuffer size: 188
    type present: 1
        type: Ping
sequence present: 1
        transitID: 1
            begin: 1
          current: 1
            total: 1
 routing present: 1
        from: 1 mach
          to: 4 cnc
flatbuffers_test: /home/vadtec/schema/generated/flatbuffers_test_reader.h:192: TransitHop_vec_at: Assertion `flatbuffers_vec_len(vec) > (i) && "index out of range"' failed.

mikkelfj

unread,
Mar 17, 2021, 4:05:58 PM3/17/21
to FlatBuffers
Hi Vadtec,

I'm the author of flatcc.

I'm sorry you are having problems here. Flatcc should be a good match for the data you have.

Typically there is a steep learning curve until you get it, and then it becomes relatively easy.
It's difficult for me to just look at the code and say what is wrong, but I will work with you until we figure out a solution.

It seems to be that you may be forgetting to push the elements to the vector you have created around line 44.
You are staring and ending the vector, and creating child tables, but you do not appear to be pushing. As I recall you can create and push at the same time via dedicated call, or create, then push.

The monster_test.c file should have some examples to guide you.
Please let me know how it goes and if I can of further assistance.

Regards Mikkel

Vadtec

unread,
Mar 17, 2021, 4:21:01 PM3/17/21
to FlatBuffers
So, the act of surrounding the calls to _create() for the individual vector items starting on line 44 does not automatically add them to the vector. I was under the assumption that surrounding any _create() calls with _start(B) and _end(B) gave flatbuffers the information it needs to know where to add the created items. Now that you have clued me in to push+create as one step, I see a _vec_push_create() function. I will explore this further and get back to you.

Many thanks.

mikkelfj

unread,
Mar 17, 2021, 4:44:11 PM3/17/21
to FlatBuffers
Yes, there is push_create if need that, see

Note that the same table reference can be reused in multiple places so it is not desirable to always automatically add or push created tables.
However, there are many calls the depend on the current context being between start and end calls, including the push call you were apparently missing, or the push_create call you could also use. Likewise there are other helper functions in different situations that combines several operations in the current context, and this might have mislead you.

Vadtec

unread,
Mar 17, 2021, 4:59:25 PM3/17/21
to FlatBuffers
Mikkel,

Many thanks for the response. I have now gotten this working. I used a combination of _vec_push_start(B)/_vec_push_end(B) and _vec_push_create() to get everything working. As you said, sometimes you need a reference to a given table, so for those particular _create() calls, I surrounded them with _vec_push_start(B)/_vec_push_end(B), otherwise I used _vec_push_create().

I've been bashing on this for a couple days now, so to finally have it working is amazing. Thank you again for your response.

(posting the working code here for posterity)

                    TransitHop_vec_push_start(B);
                    {
                        hopFrom = TransitHop_create(B, 1, ServerTypes_mach);
                    }
                    TransitHop_vec_push_end(B);

                    TransitHop_vec_push_create(B, 2, ServerTypes_controller);
                    TransitHop_vec_push_create(B, 3, ServerTypes_controller);

                    TransitHop_vec_push_start(B);
                    {
                        hopTo = TransitHop_create(B, 4, ServerTypes_cnc);
                    }
                    TransitHop_vec_push_end(B);
            size_t hops_vec_len = TransitHop_vec_len(hopsTable);
            printf("\thops: %lu\n", hops_vec_len);

            TransitHop_table_t hop = 0;
            ServerTypes_enum_t hopServerType = 0;
            for (size_t i = 0; i < hops_vec_len; i++) {
                hop = TransitHop_vec_at(hopsTable, i);
                hopServerType = TransitHop_serverType(hop);

                test_assert(hop != 0);

                printf("\t\thop: %ld\t%s\n", TransitHop_serverID(hop), ServerTypes_name(hopServerType));
            }
        }
    }

    fh = fopen("/tmp/fbhd-test1", "w");

    if (fh == NULL) {
        printf("Cannot open /tmp/fbhd-test1 for writing\n\n");
        return -1;
    } else {
        hexdump(NULL, buff, size, fh);
    }

    fclose(fh);

    flatcc_builder_aligned_free(buff);

    return 0;
}





debug1.txt
--------------------------------------------------



TestRoot - Routing

flatbuffer size: 212
    type present: 1
        type: Ping
sequence present: 1
        transitID: 1
            begin: 1
          current: 1
            total: 1
 routing present: 1
        from: 1 mach
          to: 4 cnc
        hops: 4
                hop: 0  mach
                hop: 2  controller
                hop: 3  controller
                hop: 0  mach



00000000  08 00 00 00 54 45 53 54  40 ff ff ff 01 00 00 00  |....TEST@.......|
00000010  8c 00 00 00 78 00 00 00  04 00 00 00 5e ff ff ff  |....x.......^...|
00000020  0c 00 00 00 58 00 00 00  1c 00 00 00 04 00 00 00  |....X...........|
00000030  48 00 00 00 30 00 00 00  1c 00 00 00 04 00 00 00  |H...0...........|
00000040  a0 ff ff ff 8e ff ff ff  04 00 00 00 00 00 00 00  |................|
00000050  01 00 00 00 9e ff ff ff  03 00 00 00 00 00 00 00  |................|
00000060  05 00 00 00 ae ff ff ff  02 00 00 00 00 00 00 00  |................|
00000070  05 00 00 00 00 00 00 00  d8 ff ff ff cc ff ff ff  |................|
00000080  01 00 00 00 00 00 00 00  00 00 00 00 e8 ff ff ff  |................|
00000090  01 00 00 00 00 00 00 00  01 01 01 00 fc ff ff ff  |................|
000000a0  04 00 04 00 0c 00 0f 00  04 00 0c 00 0d 00 0e 00  |................|
000000b0  06 00 0c 00 04 00 08 00  0d 00 04 00 0c 00 0a 00  |................|
000000c0  10 00 08 00 0c 00 04 00  0c 00 14 00 04 00 08 00  |................|
000000d0  0c 00 10 00                                       |....|
Message has been deleted

mikkelfj

unread,
Mar 17, 2021, 8:24:53 PM3/17/21
to FlatBuffers

Thanks for letting me know. I know how frustrating these things can be.

Vadtec

unread,
Mar 17, 2021, 8:30:46 PM3/17/21
to FlatBuffers
Greetings again,

So, I noticed a weird bug. The output was showing "default" values for the first and last "hop".

I was originally using this code:

--------------------------------------------------

               TransitHeader_hops_start(B);
                {
                    TransitHop_vec_push_start(B);
                    {
                        hopFrom = TransitHop_create(B, 1, ServerTypes_mach);
                    }
                    TransitHop_vec_push_end(B);

                    TransitHop_vec_push_create(B, 2, ServerTypes_controller);
                    TransitHop_vec_push_create(B, 3, ServerTypes_controller);

                    TransitHop_vec_push_start(B);
                    {
                        hopTo = TransitHop_create(B, 4, ServerTypes_cnc);
                    }
                    TransitHop_vec_push_end(B);
                }
                TransitHop_ref_t hops = TransitHeader_hops_end(B);

--------------------------------------------------

Under the assumption I wasn't using _vec_push_start(B)/_vec_push_end(B) correctly, I changed to this code:

--------------------------------------------------

                TransitHeader_hops_start(B);
                {
                    hopFrom = TransitHop_create(B, 1, ServerTypes_mach);
                    TransitHop_vec_push(B, hopFrom);

                    TransitHop_vec_push_create(B, 2, ServerTypes_controller);
                    TransitHop_vec_push_create(B, 3, ServerTypes_controller);

                    hopTo = TransitHop_create(B, 4, ServerTypes_cnc);
                    TransitHop_vec_push(B, hopTo);
                }
                TransitHop_ref_t hops = TransitHeader_hops_end(B);

--------------------------------------------------

The new code produces the correct and expected output.

--------------------------------------------------

        hops: 4
                hop: 1  mach
                hop: 2  controller
                hop: 3  controller
                hop: 4  cnc

--------------------------------------------------

So....what the hell about _vec_push_start(B)/_vec_push_end(B) am I obviously missing? Or am I just completely missing something about _start(B)/_end(B) methods in general?

mikkelfj

unread,
Mar 17, 2021, 8:32:05 PM3/17/21
to FlatBuffers
I'm not sure this does what you intended:

                    TransitHop_vec_push_start(B);
                    {
                        hopTo = TransitHop_create(B, 4, ServerTypes_cnc);
                    }
                    TransitHop_vec_push_end(B);

The push_start starts a table, but you have to use add calls add member values to that table. The hopTo assignment goes nowhere.

I didn't look carefully, but baybe:

                    hopTo = TransitHop_create(B, 4, ServerTypes_cnc);
                    TransitHop_vec_push(B, hopTo);
or just
                   TransitHop_vec_push_create(B, 4, ServerTypes_cnc);

if you don't need the hopTo reference elsewhere.

Vadtec

unread,
Mar 17, 2021, 8:32:23 PM3/17/21
to FlatBuffers
Are the _start()/_end() family of functions only able to be used to surround calls to set individual values for a given table/thing? I feel like I'm misusing _start()/_end().

Vadtec

unread,
Mar 17, 2021, 8:37:22 PM3/17/21
to FlatBuffers
Ok. So between _start()/_end(), a table has already been created, so anything between the calls is only working on the current object. Which explains why the first and last nodes were using default values. Between the calls I wasn't doing anything to set their values. Which also explains why the second version of the code works as expected. I'm explicitly creating the objects *and* adding them to the vector.

So yeah. I was misunderstanding the entire _start()/_end() family of functions.

mikkelfj

unread,
Mar 17, 2021, 8:39:27 PM3/17/21
to FlatBuffers
It seems that you discovered the issue yourself at the same time I posted the above.

The start/end table calls sets up the context such that _add calls and related can work specifik to that table.
For vectors it sets up a context where you can push elements.
The push_start/end is special in that if starts a table and assumes a vector has already been started. At the end call, it ends the table, gets the reference and calls push with that reference to store it on the vector.

While these functions can make code simpler, they are not strongly typed, so you need to know what you are doing. The more basic functions have tend to have stronger typing because the functions are specific to their arguments instead of what happened before.

Most FlatBuffer language bindings will not let you nest operations. You have to first create a table, then create a vector, then add the table to the vector, then create a parent table, then add the vector to the parent table.

The C language binding (FlatCC) takes care of this for you by creating a number of mostly small stacks. The begin/end functions are used to manage this stack context.

mikkelfj

unread,
Mar 17, 2021, 8:49:07 PM3/17/21
to FlatBuffers
Something else to be aware of:

There are roughly to function patterns: <type_name>_operation, and <type_name>_<member_name>_operation.
The last pattern will implicitly assume and instance of that type (usally a table) is open, i.e. start has been called. The former pattern typically creates a new instance, or starts or ends such and instance. The member pattern will often end a sub table and add it to the containing table. The type pattern will just return a reference you have to add yourself. The fact that you can start a type while another type is open does not mean anything - this is generally forbidden in other languages but permitted in C for convenience - but the end call just completes the type and allow you to continue in the parent context as if nothing happened.

Vadtec

unread,
Mar 18, 2021, 11:11:53 PM3/18/21
to FlatBuffers
Posting the working code for posterity. (The previously posted code has a bug in it. Well...a bug by way of me misunderstanding how something worked.)

                    TransitHop_vec_push(B, hopFrom);

                    TransitHop_vec_push_create(B, 2, ServerTypes_controller);
                    TransitHop_vec_push_create(B, 3, ServerTypes_controller);

                    hopTo = TransitHop_create(B, 4, ServerTypes_cnc);
                    TransitHop_vec_push(B, hopTo);
                }
                TransitHop_ref_t hops = TransitHeader_hops_end(B);

Reply all
Reply to author
Forward
0 new messages