(objective-c) Techniques for marshal/unpacking structured data from .proto bytes field

[Rob Cecil]

I'm using the objective-c version Protobufs.

I have a .proto defined as

message RangeData { 

int32 rows = 1;

int32 columns = 2;

bytes data = 3;

}

The server-side has been developed in C#/.Net. I'm able to successfully send/receive and unpack the data from the 'data' field 3 above. In the c# port, 'bytes' are represented by a Google protobufs "ByteString" object.

In objective-c, they're represented as NSData instance.

I am currently taking a object array in C#  and packing that into the ByteString. I have a separate .Net WPF client that successfully retrieves the data from the ByteString instance.

The object array is essentially a two-dimensional (object[,]) array instance whose elements can be numeric, or strings of varying length.

    ary[0, 0] = ""

    ary[0, 1] = "Jan 2010"

    ary[0, 2] = "Feb 2010"

    ...

    ary[0, 13] = "- Year 2010"

    ary[1, 0] = 89544.994

    ary[1, 1] = 93202.257

    ...

    ary[1, 13] = 492331.908

    ary[2, 0] = "Report A"

    ...  

    ary[16, 13] = ...

The number of rows and columns for this two dimensional array are passed in an outer (Message) context.

What is the technique for dealing with the NSData and extracting & parsing back into an objective-c (or Swift) arrays?

Are there helper classes I should be aware of to help in these scenarios?

Thanks

[Jorge Luis Canizales Diaz]

[CC: Jon for the C# side and TVL for the ObjC side]

Hi Rob!

when you say:

> I am currently taking a object array in C#  and packing that into the ByteString.

How are you accomplishing that? I see from your example that the elements of your array aren't proto messages, so maybe .Net has its own serialization format that non-.Net languages aren't aware of? In that case, the only real solution I can think of would be to formalize that serialization by using protos. For example:

message RangeData { 

int32 num_rows = 1;

int32 num_columns = 2;

repeated Row row = 3;

}

message Row {

repeated Value value = 1;

}

message Value {

oneof value {

string text = 1;

double number = 2;

}

}

or, alternatively, if you're not going to mix the strings and the numbers:

message RangeData {

repeated string row_name = 1;

repeated string column_name = 2;

repeated Row row = 3;

}

message Row {

repeated double value = 1;

}

Hope that helps,

Jorge

[Rob Cecil]

Interesting solution. I will definitely try that approach as I'm getting no where to continue with my current tack. I'm not sure how familiar your are with the .Net side of things, but this how it is working for me so far. I have a server implementation (currently just brutally simple - using console projects) and a WPF test client that currently works with my defined RangeData protobuf, including the bytes field.

On the server side this what I do (narrating the CommonTypeExtensions.ToRangeData method in the debugger screenshot - attached)

       public static RangeData ToRangeData(this object[] source)
        {
            var result = new RangeData();
            result.Rows = (int)source[0];
            result.Columns = (int)source[1];
            var formatted = ObjectToByteArray(source[2]);
            result.Data = ByteString.CopyFrom(formatted);
            return result;
        }

- Get an object "blob", just an untyped object array (from db source, see attached image of VS debugger watch window showing "source" that is visible when the debugger stops in the ToRangeData() extension method).

- Call ObjectToByteArray extension method, which uses the .Net BinaryFormatter class (and MemoryStream) to convert the object to a byte array (byte[] in .Net)

- Call Protobuf's ByteString.CopyFrom() factory method to generate the ByteString data and stuff that into the RangeData object.

On the reverse side, in the WPF client (not completely shown in screenshots):

            var ba = rangeData.Data.ToByteArray();
            var array = (Array)ba.FromByteArrayToObject();
            ViewData = new ViewData((object[,])array);

- Get the RangeData instance from Protobufs, including ByteString instance in the data field.

- Call rangeData.Data.ToByteArray() to convert ByteString to byte[] array. ToByteArray() is Protobufs provided method.

- Call FromByteArrayToObject (in the CommonTypeExtensions.cs class, partially shown in screenshot), which uses the BinaryFormatter (and MemoryStream) in .Net to convert the byte[] array into a CLR "Object"

- Cast Object to Array, then cast to object[,] two-dimensional array ("ViewData", which makes the data more consumable to a WPF DataGrid).

That is how it is working in .Net/C#.

With the Objective-C side of things, I have a NSData, instead of the ByteString. I know the #rows and #columns. I just don't  know the techniques to do the analogous thing in objective-c (or Swift).

Thanks

[Rob Cecil]

[Jon Skeet]

This is the step that makes everything completely unportable:

Call ObjectToByteArray extension method, which uses the .Net BinaryFormatter class (and MemoryStream) to convert the object to a byte array (byte[] in .Net)

That's like using ObjectOutputStream in Java. I would definitely not expect to be able to deserialize that data usefully in Objective C.

It sounds like instead of a bytes field, you should probably use an Any field... or some other message type with a oneof { int32, int64, double, string} or whatever you need. Fundamentally, you'll need to stick to protos everywhere though.

(As an aside, if you have any usability feedback on the C# protobuf experience, now is the time to tell me so I can try to smooth out any points of friction before release!)

Jon

On 12 November 2015 at 03:18, Thomas Van Lenten <thom...@google.com> wrote:

I can't say I know C# to comment much on that side of things.  It does sorta look like you are using something more native to those objects for serializing into a blob, and simply using a proto to try and package the row & col counts along with the blob.  It probably would make more sense to do something like Jorge suggests and use protos for the full packaging rather than just the outer packaging.

TVL

[Rob Cecil]

Thanks Jon, for pointing out what should have been clear was an unportable approach. <facepalm>

I will try the fine-grained protos approach demonstrated by Jorge.

Thanks