How to speedup serialization and deserialization process?

[Ravi]

I have defined the Protocol Buffers message file as follows:

syntax = "proto3";
package Tutorial;
import "google/protobuf/timestamp.proto";
option csharp_namespace = "Tutorial";

message PointCloud {
  int32 width  = 1;
  int32 height = 2;

  message Point {
    float x     = 1;
    float y     = 2;
    float z     = 3;
    fixed32 rgb = 4;
  }
  repeated Point points = 3;
  google.protobuf.Timestamp timestamp = 4;
}

This is how I am preparing the data and serializing it in C#:

using Google.Protobuf;
using Tutorial;
using ZeroMQ;

PointCloud pointCloud = new PointCloud();
pointCloud.Height = Height
pointCloud.Width = Width;
pointCloud.Timestamp = Timestamp.FromDateTime(DateTime.UtcNow);

for (var index = 0; index < POINTS3D_COUNT; index++) {
  PointCloud.Types.Point point = new PointCloud.Types.Point {
    X = points3D[index].X,
    Y = points3D[index].Y,
    Z = points3D[index].Z,
    Rgb = (uint)((red << 16) | (green << 8) | blue)
  };

  pointCloud.Points.Add(point);
}

zmqPublisher.Send(new ZFrame(pointCloud.ToByteArray()));

This is how I deserialize the data in C++:

while (receive) {
  zmq::message_t msg;
  int rc = zmq_socket.recv(&msg);
  if (rc) {
    Tutorial::PointCloud point_cloud;
    point_cloud.ParseFromArray(msg.data(), msg.size());
  }point_cloud.ParseFromArray(msg.data(), msg.size())
}

I am able to get the data back properly. However, the serialization and deserialization process seems slow. 

• I used System.Diagnostics.Stopwatch in C# and noticed that pointCloud.ToByteArray() is taking 100ms approximately.
• Similarly I used std::chrono::steady_clock::now() in C++ and noticed that point_cloud.ParseFromArray(msg.data(), msg.size()) is taking 96ms approximately.
• Just for information, the length of the byte array is roughly 3,784,000.
  

I want to know how to speed up serialization and deserialization process?

-

Thanks

Ravi

[Ravi]

Any suggestions, please?

[Marc Gravell]

I did a test locally using protobuf-net (since that is what I'm most familiar with); to get an output of about 3,784,000 I used a count of 172000 items in the inner array - does that sound about right? Then I tested it in a loop as per this gist: https://gist.github.com/mgravell/10a21970531485008731d700b89ec732

The timings I get there are about 25ms to serialize, 40ms to deserialize - although my machine is quite fast, so this may mean you're pretty much getting "about right" numbers. What sort of numbers are you looking for here? Note that in .NET the first run will always be slightly slower due to JIT.

Additionally, I know nothing about the zmq overheads - are you including the zmq cost in your numbers?

If you want to squeeze the last few drops of performance, you usually can - for example, by looking at whether zmq allows you to pass a stream or span or an *oversized* array - again, I'm not as familiar with the Google C# version as I am with protobuf-net, but *if* (and it is a huge "if") the "ToByteArray()" is essentially writing to a MemoryStream then calling ToArray, you can probably avoid an extra blit and some allocs by providing your own re-used memory-stream and using GetBuffer to access the oversized array, remembering to limit to just the first .Length bytes. But again a lot of this depends on specifics of zmq and the Google C# version. It is almost certainly diminishing returns.

So: what numbers are you *looking to get*? what would be "acceptable"? And how complex is your data? is what you've shown the *only* data you need to transfer? if so, it might not be a bad candidate for fully manual explicit serialization not involving a library - just a payload of:

Height [int32 fixed 4 bytes]

Width [int32 fixed 4 bytes]

Time [int64 fixed 8 bytes]

ElementCount [int32 fixed 4 bytes]

then for each element: 16 bytes consisting of

    X [float fixed 4 bytes]

    Y [float fixed 4 bytes]

    Z [float fixed 4 bytes]

    RGB [int32 fixed 4 bytes]

this would require manual coding, but would typically outperform other options - but would be more brittle and would require you to be reasonably good at IO code.

Personally, I'd probably leave it alone...

Marc

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscribe@googlegroups.com.
To post to this group, send email to prot...@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

--

Regards,

Marc

[Ravi]

Marc,

Thank you very much for the detailed explanation. I got it.

Due to curiosity, I tried manual explicit serialization. I needed to code more but indeed it worked way faster as you suggested. 

Thanks again for the wonderful explanation.

-

Ravi

On Wednesday, December 20, 2017 at 10:00:31 PM UTC+9, Ravi wrote: