List/blob size limits

[Randall Nortman]

I wasn't able to find any of this explicitly discussed in the documentation, but from reading the encoding spec, it appears to me that data and text blobs are limited to 512MiB in size, due to them being encoded as a list, and lists would appear to be limited to 2^29 elements per list.  Is that right?  Are there any other (implicit or explicit) limits on message or object sizes?

[Kenton Varda]

Hi Randall,

Yes, that's correct. The 512MiB blob limit is arguably the only one anyone is ever likely to notice in practice.

- Non-blob lists are (as you saw) also limited to 512Mi elements, but of course you're much less likely to hit that limit with a regular list.

- Structs are limited to 2^16 fields -- but if you actually manage to hit that limit, you're doing something very wrong. :)

- A message is limited to 2^32 segments, each of which can be up to 2^32 bytes (for a theoretical max of 2^64 bytes).

Generally, you should aim to keep messages small. Since any blob (actually, any segment) will need to be loaded into contiguous memory, keeping segment sizes small helps avoid freaking out your memory allocator.

When using Cap'n Proto for RPC, as a rule of thumb you should never have a message larger than 1MB. Instead, split large data blobs into a stream of chunks, each of which is delivered as a separate RPC.

If you are writing a large file designed to be mmap()ed, then you may legitimately run up against the 512MiB limit. For example, Sandstorm.io's app package format (spk) currently has problems if an app wants to include a >512MiB file. Sandstorm will probably solve this by extending the format to allow representing a file as a list of blobs, not just a single blob. In retrospect, perhaps I should have made this limit a bit bigger, but there would clearly have to be a limit at 2^31 bytes for the sake of 32-bit address spaces. If you are hitting the 2^29 limit, you'd hit 2^31 soon enough. :)

-Kenton

On Fri, Dec 26, 2014 at 5:32 PM, Randall Nortman <wonde...@gmail.com> wrote:

I wasn't able to find any of this explicitly discussed in the documentation, but from reading the encoding spec, it appears to me that data and text blobs are limited to 512MiB in size, due to them being encoded as a list, and lists would appear to be limited to 2^29 elements per list.  Is that right?  Are there any other (implicit or explicit) limits on message or object sizes?

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.
Visit this group at http://groups.google.com/group/capnproto.