Bytes type limit of 4 GB

[David Yat Sin]

Hi, 

I am trying to store large buffers that can potentially grow to 32 GB, but when using the bytes type, I see the unpack fails when the size is just below 4GB.

Is that a known limitation? Would you have a recommendation on how to pack buffers > 32 GB.

proto file:

-------------------------------

syntax = "proto2";

message msg_test {

        required bytes buf = 1;

}

-------------------------------

C code:

-------------------------------

//#define PROTOBUF_BYTES_SIZE ((size_t)(0xFFFFFFFF)) //Fails

#define PROTOBUF_BYTES_SIZE ((size_t)(0xFFFFFFFF-5)) //Succeeds

int main(int argc, char** argv)

{

        int ret = 0;

        MsgTest *msg, *msg2;

        size_t len;

        uint8_t *buf;

        msg = malloc(sizeof(*msg));

        if (!msg) {

                printf("Failed to allocate proto structure\n");

                return -ENOMEM;

        }

        msg_test__init(msg);

        msg->buf.data = malloc(PROTOBUF_BYTES_SIZE);

        msg->buf.len = PROTOBUF_BYTES_SIZE;

        len = msg_test__get_packed_size(msg);

        buf = malloc(len);

        if (!buf) {

                printf("Failed to allocate memory (len:0x%lx)\n", len);

                return -ENOMEM;

        }

        msg_test__pack(msg, buf);

        //msg2 = msg_test__unpack(&my_allocator, len, buf);

        msg2 = msg_test__unpack(NULL, len, buf);

        if (msg2) {

                printf("Unpack successful\n");

                msg_test__free_unpacked(msg2, NULL);

        } else {

                printf("Unpack failed\n");

        }

        return ret;

}

-------------------------------

Thanks, 

David

[Adam Cozzette]

Actually protobuf messages must be strictly less than 2 GiB in size. It looks like you're using protobuf-c and perhaps that implementation will allow somewhat larger sizes, but the implementations maintained by Google enforce a 2 GiB limit. There is no real workaround except to break up the data into smaller pieces.

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/protobuf/1a5543f3-9dc7-4046-af60-954595859967n%40googlegroups.com.

[David Yat Sin]

Ok. Thanks Adam for clarifying.

Regards,

David