Protobuf order or serialization

[Pravesh Tanwar]

Hi All, 

I am currently using Protobuf serializer in my C# project. Currently I am serializing my Objects using "[ProtoContract(ImplicitFields = ImplicitFields.AllPublic)]". The serialized objects are stored in database and deserialized when needed. I have now landed in a position that requires me to add new DataMembers to existing object. I want to be able to do this without breaking backward compatibility (while deserializing) of my current serialized objects. 

Is there a way for me to check what order does the Protobuf serialize/deserialize objects? 

e.g of the Object being serialized: 

    [ProtoContract(ImplicitFields = ImplicitFields.AllPublic)]

    public class Product 
    {
        [DataMember]
        public int Id { get; set; }
 
        [DataMember]
        public string Name { get; set; }
 
        [DataMember]
        public double Quantity { get; set; }

        [DataMember]
        public bool? IsAvailable { get; set; }
    }
    
If I can know what order they are currently being serialized then I can use the "Order" data member attribute and make sure that the additional fields are ordered at the end. Hopefully this will retain backward compatibility while deserializing. e.g:

    public class Product 
    {
        [DataMember(Order = 1)]
        public int Id { get; set; }
  
        [DataMember(Order = 2)]
        public double Quantity { get; set; }

        [DataMember(Order = 3)]
        public bool? IsAvailable { get; set; }

        [DataMember(Order = 4)]
        public string Name { get; set; }
 
        [DataMember(Order = 5)]
        public string NewField1 { get; set; }
 
        [DataMember(Order = 6)]
        public string NewField2 { get; set; }

    }

Any ideas? 

[Marc Gravell]

This is specifically a protobuf-net question.

In short, yes - that is fine... ish. If you add the numbers manually ***and get them right***, then it will work. However, your example actually gets them wrong: the protobuf-net library specifically assumes an *alphabetical* order for the properties / fields when using ImplicitFields, because reflection does not guarantee any particular order - see the "Remarks" section on MSDN: http://msdn.microsoft.com/en-us/library/6ztex2dc.aspx and http://msdn.microsoft.com/en-us/library/kyaxdd3x.aspx

So the order is actually: Id = 1, IsAvailable = 2, Name = 3, Quantity = 4 - you would then add NewField1 and NewField2 as 5 and 6, presumably.

Just to quote from the documentation on ImplicitFields, too:

    /// <summary>
    /// Specifies the method used to infer field tags for members of the type
    /// under consideration. Tags are deduced using the invariant alphabetic
    /// sequence of the members' names; this makes implicit field tags very brittle,
    /// and susceptible to changes such as field names (normally an isolated
    /// change).
    /// </summary>

Marc

(protobuf-net)

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com.
To post to this group, send email to prot...@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Regards,

Marc

[Marc Gravell]

Oh, further: if you want to know what order it is using - the easiest is probably just to do:

    string proto = Serializer.GetProto<Product>();

and look at the .proto schema it generates. You can also interrogate the MetaType and ValueMember instances (if you were doing it programatically, this would be a good choice) but the above is much easier for ad-hoc scenarios.

Marc

(protobuf-net)

--
Regards,

Marc

[Ross McDermott]

Thanks for the help Marc, I ended up writing an app to generate a report of the ordering ProtoBuf was using, then going any applying that order to each property and removing the ProtoContract from everything.

 

Code may be useful to others in the future, so here it is as a starting point (need to tweak it to create the original first then do the compare and generate report).

 

Cheers.

Ross.

 

public class Program
    {
        static void Main(string[] args)
        {
            var original = Console.Out;
            Console.WriteLine("Outputting Report to file");

            using (var stream = new StreamWriter("Report.txt"))
            {
                Console.SetOut(stream);

                ISerializer serializer = new ProtobufSerializer(); // custom implementation class

                var toCheck = LoadOriginal();

                Console.WriteLine("Loaded OK");

                foreach (var item in toCheck.Types)
                {
                    Console.WriteLine("Processing Type [{0}]", item.Name);

                    var type = Type.GetType(item.Type);

                    if (type == null)
                    {

                        Console.WriteLine("ERROR LOADING TYPE!");
                        throw new Exception("Why?");
                    }

                    var ca = type.GetCustomAttributes(typeof(ProtoContractAttribute), false);

                    if (ca.Length > 0)
                    {
                        Console.WriteLine(
                            "\t*** WARNING: TYPE IS USING ProtoContractAttribute! This should be fixed. ***");
                    }

                    if(type.IsEnum)
                    {
                        Console.WriteLine("Enum - Ignoring order check.");
                    }
                    else
                    {
                        foreach (var field in item.Feilds)
                        {
                            PropertyInfo pi = type.GetProperty(field.Name,
                                                               BindingFlags.Public | BindingFlags.NonPublic |
                                                               BindingFlags.Instance);
                            Console.Write("\t[{0}]\t\tExpected [{1}]\t\t", field.Name, field.Order);

                            if (pi == null)
                            {
                                Console.Write("ERROR - PROPERTY MISSING!");
                            }
                            else
                            {
                                var attributes = pi.GetCustomAttributes(typeof(DataMemberAttribute), false);

                                if (attributes.Length == 0)
                                {
                                    Console.Write("ERROR! DataMember Attribute not found.");
                                }
                                else if (attributes.Length > 1)
                                {
                                    Console.Write("ERROR! Expecting one DataMember Attribute.");
                                }
                                else
                                {
                                    var attr = attributes[0] as DataMemberAttribute;

                                    if (attr.Order == field.Order)
                                    {
                                        Console.Write("OK");
                                    }
                                    else
                                    {
                                        Console.Write("ERROR! Incorrect order [{0}]", attr.Order);
                                    }
                                }
                            }

                            Console.WriteLine();

                        }
                    }
                

                    Console.WriteLine();
                }

                stream.Flush();
            }
            Console.SetOut(original);

            Console.WriteLine("Done.");

            Console.ReadKey();
        }

        private static CompatabilityOrder LoadOriginal()
        {
            if (File.Exists("Original.xml"))
            {
                XmlSerializer s = new XmlSerializer(typeof(CompatabilityOrder));
                return s.Deserialize(File.Open("Original.xml", FileMode.Open)) as CompatabilityOrder;
            }
            else
            {
                return new CompatabilityOrder();
            }
        }

        private static CompatabilityOrder GenerateOriginal()
        {
            var x = ProtoBuf.Meta.RuntimeTypeModel.Default;

            List<ProtoBuf.Meta.MetaType> mt = new List<MetaType>();
            mt.AddRange(x.GetTypes().OfType<ProtoBuf.Meta.MetaType>());

            // loop around and find any types that have the attribute ProtoContract on them
            // and add them to the collection

            foreach (Assembly a in AppDomain.CurrentDomain.GetAssemblies())
            {
                foreach (Type t in a.GetTypes())
                {
                    if (t.GetCustomAttributes(typeof(ProtoContractAttribute), false).Any())
                    {
                        mt.Add(x.Add(t, true));
                    }
                }
            }

            CompatabilityOrder co = LoadOriginal(); // build on what we have

            foreach (ProtoBuf.Meta.MetaType metaType in mt)
            {
                if (!co.Types.Any(t => t.Type == metaType.Type.AssemblyQualifiedName))
                {
                    var cot = new CompatabilityOrderType();
                    cot.Name = metaType.ToString();
                    cot.Type = metaType.Type.AssemblyQualifiedName;
                    cot.Feilds = new List<CompatabiltityOrderTypeField>();

                    foreach (var member in metaType.GetFields())
                    {
                        cot.Feilds.Add(new CompatabiltityOrderTypeField() { Name = member.Name, Order = member.FieldNumber });
                    }

                    co.Types.Add(cot);
                }
            }

            XmlSerializer s = new XmlSerializer(co.GetType());
            var fs = File.Create("Original.xml");
            s.Serialize(fs, co);

            return co;
        }

        public class CompatabilityOrder
        {
            public List<CompatabilityOrderType> Types = new List<CompatabilityOrderType>();
        }

        public class CompatabilityOrderType
        {

            public string Name { get; set; }

            public string Type { get; set; }
            public List<CompatabiltityOrderTypeField> Feilds = new List<CompatabiltityOrderTypeField>();

        }

        public class CompatabiltityOrderTypeField
        {

            public string Name { get; set; }

            public int Order { get; set; }
        }
    }

 

Sample output (once fixed)

Processing Type [XXX.Product]
 
 [Id]  Expected [1]  OK
 [Name]  Expected [2]  OK
 [Quantity]  Expected [3]  OK
 [IsAvailable]  Expected [4]  OK