follow up on speeding up P4Runtimw

[Steffen Smolka]

Thanks Alan for the great talk on your experiments with speeding up P4Runtime.

I briefly scanned our internal resources for protobuf performance tips, here is a condensed summary of the most important bits:

1. Field numbers 1-15 are most efficient.
2. Type `bytes` is cheaper than `strings`, since the latter is validated to be valid UTF8.
3. Fixed ints (fixed32, fixed64) may use a bit more space than int32, int64 but are faster to decode roughly for integers >= 2^7
4. Message hierarchy is not cheap, since it can occur function calls, memory allocation, cache misses.
5. Use arenas to increase locality, amortize allocations, and make deallocations virtually free.
6. You can avoid making copies of strings/bytes when parsing them and use aliasing instead, but unfortunately this is not open-sourced yet.

My comments based on precursory glance over p4runtime.proto:

1. N/A, we already use small numbers.
2. N/A, we already use `bytes` in places where it would matter.
3. This may be worth giving a shot!
4. This is what you observed and exploited in your proposal.
5. This seems important. I believe you mentioned you already tried this?
6. N/A, unfortunately, since this is unfortunately not yet open source.

One more observation, note that not all hierarchy is equally bad. Having a tightly packed representation of all the data we care about many layers deep is basically just as good as having a tightly packed representation of that data at the top level, since you only need to traverse the indirection once (so it is amortized across the entire data).

Not much new here, but maybe 3. is worth giving a shot.

Cheers,

-- Steffen