[Protobuf] Can protobuf serialize datatable (C#)

[Ferryandi Chai]

Hi All,

Im just wondering can protobuf serialize datatable.
Im a c# developer, and when i tried to serialize datatable it prompt
this error:

"Only data-contract classes (and lists/arrays of such) can be
processed (error processing DataTable)"

Tried to google this problem but cannot find solutions.


Thx
F!

[Marc Gravell]

From the message, that *sounds* like protobuf-net...

There is no built-in handling of this, but it is possibly something that could be added, especially in "v2" which has a much more flexible model. For example, you *could* argue that there is an implicit schema that uses a repeated element per row, and the column ordinal for a field.

There are a couple of points here thought:

- protobuf /normally/ doesn't include schema information; so when deserializing *either* you would have to set up the schema (columns etc) *first* and then use Merge, /or/ I could include schema information as an exception

- it would need some thought re the difference between DataSet and DataTable - a data-set has relations etc, which don't necessarily fit very well here

I'm interested in your thoughts with this; are you just after something to load/save the data? Or interop with other platforms?

And the other important question: /an option/ is to move the data to/from a DTO during serialization; the DTO would serialize fine with protobuf-net.

Marc

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To post to this group, send email to prot...@googlegroups.com.
To unsubscribe from this group, send email to protobuf+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.

--
Regards,

Marc

[Desmond Davids]

Hey Marc,

I know this is an old article. but i need to ask you something regarding this.

We have a WCF service which we use to execute SqlCommands to our DB. We send Commands with DataContractSerialization. I came across protobuf-net and I have converted my WCF service to use protobuf. We now sit with a problem. we use table valued parameters in our sqlcommands. which means we need to serialize DataTables as part of an object. I there a way we can achieve this with protobuf.

Do you have any workaround/alternative or suggestion?

Regards

Desmond Davids

[Marc Gravell]

Ooh, the idea of allowing SQL over a service boundary sends all kinds of shivers up my back, and not the good kind of shivers. On your head be it, but: I wouldn't do that myself.

I once did some experimental datatable serialisation code - it isn't in the release binary, but it is still inside the solution; but: I really can't say how robust this would be, and you'd need to serialize it manually and store it in a byte[]. Of course, by the time you've done all that you might as well have simply asked the datatable to serialize itself using its own internal storage format (make sure you switch from XML to binary) - and again: shove it in a byte[] member. IIRC you can set the RemotingFormat and use BinaryFormatter to get the internal binary storage format.

Marc

To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com.

To post to this group, send email to prot...@googlegroups.com.

Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[Marc Gravell]

Yes, a `byte[]` can be declared as a `ProtoMember` - it maps to the `bytes` type in the protobuf spec. I would strongly advise *against* DynamicType here; that should be used as a last resort only. If you know the data you want to serialize is actually a finite set of types, I would use that if possible. For example, let's say you need to support `string`, `int` and `DataTable`; you could do something like (this is completely untested - writing in email client here, but the concept is sound):

    [ProtoContact]

    [ProtoInclude(10, typeof(ParameterValue<int>)]

    [ProtoInclude(11, typeof(ParameterValue<string>)]

    [ProtoInclude(12, typeof(ParameterValue<DataTableShim>)]

    public abstract class ParameterValue {

         public abstract Type Type {get;}

         public abstract object Value {get;set}

    }

    [ProtoContract]

    public sealed class ParameterValue<T> : ParameterValue {

        public override Type Type { get { return typeof(T); } }

 

        [ProtoMember(1)]

        public T TypedValue {get;set;}

        public override object Value {

            get { return TypedValue; }

            set { TypedValue = (T)value; }

        }

    }

    [ProtoContract]

    public sealed class DataTableShim {

        public DataTable Table {get;set;}

        [ProtoMember(1)]

        public byte[] TableSerialized {

            get { /* serialize Table to a MemoryStream, and return ms.ToArray(); */ }

            set { /* create a new MemoryStream(value), and deserialize to Table */ }

        }

    }

On 11 July 2014 07:54, Desmond Davids <dem...@gmail.com> wrote:

Hey Marc,

Thanks for the reply. We not sending the actual SQL commands over the line. we sending the info in such a way that way know which commands to execute on which parameters. Problem is our parameters are almost always Table Valued Parameters (TVP). and needs to sit in the request as DataTable properties. This is our problem protobuf-net cannot serialize DataTable type. It also has a problem with object type but i think setting DynamicType = true does the trick here as the types will always be primitive types. Am i correct? So we need some way to get these TVPs over the line using protobuf-net as serializer in WCF.

Can you tell if byte[] can be declared as a ProtoMember. I can therefore use protobuf-net-data to serialize the DataTable object and assign the this byte[] ProtoMember. When it comes to assigning my TVP I can just Deserialize with protobuf-net-data again.

I am looking for the fastest way I can do this as the sole reason we using this is that we love the performance of protobuf-net. If meaning we have to sacrifice the performance to serialize every single TVP we might as well revert back to DataContractSerializer which is definitely not what we want.

I think also you suggestion regarding converting DataTable to a DTO will be difficult as we dont really know what the schema of the DataTable will be. I will have to think about this more.

Looking forward to hear from you.

Regards

Desmond

--
Regards,

Marc