[protobuf-net] Unknow wire-type 7 when serializing List data structure

[jeevankodali]

Hi I created a Protobuf-net class
[ProtoContract]
public class ProtoBlock
{
private string date;

[ProtoMember(1)]
public string Date
{
get { return date; }
set { date= value; }
}
}

I have a variable
List<ProtoBlock> data = new List<ProtoBlock>();

// serialize works fine
using (Stream outfile = File.Open(path,
FileMode.OpenOrCreate))
{
ProtoBuf.Serializer.Serialize(outfile, data);
}

but when I deserialize it using
using (Stream inFile = File.OpenRead(path))
{
data =
ProtoBuf.Serializer.Deserialize<List<ProtoBlock>>(inFile);
}

I am getting the error "Unknown wire-type 7".

Can some one help me on this?

Thanks
Jeevan

[Marc Gravell]

Hmmm... that should work fine; there *was* a related bug in an early build, but...

Can I check which version and framework you are using? I've checked on r275, and it works fine "as is", for both empty lists and some arbitrary data I made up.

Sorry for the delay, btw - seasonal break, etc...

Marc Gravell (protobuf-net)

[jeevankodali]

Thanks for the reply. I used r278 and Net20 folder in that zip file.

[Marc Gravell]

OK; I'll re-check with the Net20 variant...

2009/12/19 jeevankodali <jeevan...@gmail.com>

--

You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To post to this group, send email to prot...@googlegroups.com.
To unsubscribe from this group, send email to protobuf+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.

--
Regards,

Marc

[Marc Gravell]

Rechecked with Net20 from the r278 build, and it worked fine. Sorry to be a pain, but do you have a complete example that shows the issue?

Marc

2009/12/19 Marc Gravell <marc.g...@gmail.com>

--
Regards,

Marc

[jeevankodali]

Sorry for the late reply, what exactly do you want in the full
example, I can add the code that I am using, do you need the file as
well (and how can I send the file)?

Thanks

On Dec 19, 5:09 pm, Marc Gravell <marc.grav...@gmail.com> wrote:
> Rechecked with Net20 from the r278 build, and it worked fine. Sorry to be a
> pain, but do you have a complete example that shows the issue?
>
> Marc
>

> 2009/12/19 Marc Gravell <marc.grav...@gmail.com>

>
>
>
>
>
> > OK; I'll re-check with the Net20 variant...
>

> > 2009/12/19 jeevankodali <jeevankod...@gmail.com>

>
> > Thanks for the reply. I used r278 and Net20 folder in that zip file.
>
> >> On Dec 18, 2:36 pm, Marc Gravell <marc.grav...@gmail.com> wrote:
> >> > Hmmm... that should work fine; there *was* a related bug in an early
> >> build,
> >> > but...
>
> >> > Can I check which version and framework you are using? I've checked on
> >> r275,
> >> > and it works fine "as is", for both empty lists and some arbitrary data
> >> I
> >> > made up.
>
> >> > Sorry for the delay, btw - seasonal break, etc...
>
> >> > Marc Gravell (protobuf-net)
>
> >> --
>
> >> You received this message because you are subscribed to the Google Groups
> >> "Protocol Buffers" group.
> >> To post to this group, send email to prot...@googlegroups.com.
> >> To unsubscribe from this group, send email to

> >> protobuf+u...@googlegroups.com<protobuf%2Bunsubscribe@googlegroups.c­om>

> >> .
> >> For more options, visit this group at
> >>http://groups.google.com/group/protobuf?hl=en.
>
> > --
> > Regards,
>
> > Marc
>
> --
> Regards,
>

> Marc- Hide quoted text -
>
> - Show quoted text -

[Marc Gravell]

Really, it is whatever is enough to show the problem. For example, the following doesn't (for me) show any issues. You might also want to clarify your setup - could it be some x64/ia64 related bug, for example? (I'm testing on x86, Win7):

using System.Collections.Generic;

using System.Diagnostics;

using System.IO;

using ProtoBuf;

[ProtoContract]

public class ProtoBlock {

    private string date;

    [ProtoMember(1)]

    public string Date {

        get { return date; }

        set { date = value; }

    }

}

static class Program {

    static void Main() {

        List<ProtoBlock> data = new List<ProtoBlock>();

        RoundTrip(data); // 0

        data.Add(new ProtoBlock { Date = "abc" });

        RoundTrip(data); // 1

        data.Add(new ProtoBlock { Date = "def" });

        RoundTrip(data); // 2

        data.Add(new ProtoBlock { Date = "ghi" });

        RoundTrip(data); // 3

    }

    static void RoundTrip(List<ProtoBlock> list) {

        string path = "foo.bin";

        if (File.Exists(path)) File.Delete(path);

        using (Stream outfile = File.Open(path, FileMode.OpenOrCreate)) {

            ProtoBuf.Serializer.Serialize(outfile, list);

        }

        List<ProtoBlock> clone;

        using (Stream inFile = File.OpenRead(path)) {

            clone = ProtoBuf.Serializer.Deserialize<List<ProtoBlock>>(inFile)

                ?? new List<ProtoBlock>(); // blanks become null - limitation

        }

        Debug.Assert(!ReferenceEquals(clone, list)); // different lists

        Debug.Assert(clone.Count == list.Count); // same count

        for (int i = 0; i < list.Count; i++) {

            Debug.Assert(clone[i].Date == list[i].Date); // same data

        }

    }

}

2009/12/21 jeevankodali <jeevan...@gmail.com>

To unsubscribe from this group, send email to protobuf+u...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.

--
Regards,

Marc

[jeevankodali]

Finally I was able to reproduce the cause of this error. I am not
getting the same error in this sample code but a different one (but I
think reason is the same).

This is what I did which caused this:

1. First I stored a dictionary, key is int and value is a list into
the proto file and loaded this data - this step was fine.
2. Next I changed my code to store just list instead of dictionary in
the same file - store was successful
3. I tried to load this new file (which was the same name and location
as the original file with dictionary), I was getting the error.

But when I deleted the original file and reran the code (which was
storing the list), everything was fine.

Here is the code:

[ProtoContract]
public class ProtoTestClass
{
private string _id;

[ProtoMember(1)]
public string Id
{
get { return _id; }
set { _id = value; }
}
}

// below code throws error because I am using the same file
private static void ProtoTest()
{
List<ProtoTestClass> ownerList = new List<ProtoTestClass>
();
ProtoTestClass owner = new ProtoTestClass();
owner.Id = "799";
ownerList.Add(owner);

Dictionary<int, List<ProtoTestClass>> dictionaryList = new
Dictionary<int, List<ProtoTestClass>>();
dictionaryList.Add(1, ownerList);

using (Stream outfile = File.Open(@"d:\protobuftest.bin",
FileMode.OpenOrCreate))
{
ProtoBuf.Serializer.Serialize(outfile,
dictionaryList);
}

Dictionary<int, List<ProtoTestClass>> outDictionaryList =
new Dictionary<int, List<ProtoTestClass>>();
using (Stream inFile = File.Open(@"d:\protobuftest.bin",
FileMode.OpenOrCreate))
{
outDictionaryList =
ProtoBuf.Serializer.Deserialize<Dictionary<int, List<ProtoTestClass>>>
(inFile);
}

// now write just list again in the same file
using (Stream outfile = File.Open(@"d:\protobuftest.bin",
FileMode.OpenOrCreate))
{
ProtoBuf.Serializer.Serialize(outfile, ownerList);
}

List<ProtoTestClass> outList = new List<ProtoTestClass>();
using (Stream inFile = File.Open(@"d:\protobuftest.bin",
FileMode.OpenOrCreate))
{
outList =
ProtoBuf.Serializer.Deserialize<List<ProtoTestClass>>(inFile);
}

Console.WriteLine("Done with test");
}

> 2009/12/21 jeevankodali <jeevankod...@gmail.com>

> > <protobuf%2Bunsubscr...@googlegroups.c­om>

[Marc Gravell]

OK; now it makes sense. With the description and sample code above, this is nothing to do with protobuf-net (specifically), and everything to do with how you are doing your file access. If you use File.Open with OpenOrCreate on an **existing** file, it doesn't truncate the file. If you then write *fewer* bytes than were in the original file, you essentially have garbage at the end of the stream.

The "protocol buffers" wire format doesn't (by default) include a length prefix for the overall message, since it is designed to be possible to merge by appending. You have a few options:

1: delete the file before re-writing it (although this may impact any ACLs etc that you have defined)

2: specify FileMode.Truncate rather than FileMode.OpenOrCreate (truncates old data *before* writing)

3: call stream.SetLength(stream.Position) after writing (truncates old data *after* writing)

4: use SerializeWithLengthPrefix (but note that this will *retain* old garbage)

To reproduce just the stream aspects here, see below.

Marc

using System;

using System.IO;

static class Program {

    static Random rand = new Random();

    const string Path = @"scratch.bin";

    static void WriteGarbabe(Stream stream, int length) {

        byte[] data = new byte[length];

        rand.NextBytes(data);

        stream.Write(data, 0, length);

    }

    static void ShowLength(string path) {

        Console.WriteLine(new FileInfo(path).Length);

    }

    static void Main() {

        using (Stream dest = File.Open(Path, FileMode.OpenOrCreate)) {

            WriteGarbabe(dest, 50);

        }

        ShowLength(Path); // shows 50 like we expect

        using (Stream dest = File.Open(Path, FileMode.OpenOrCreate)) {

            WriteGarbabe(dest, 40);

        }

        ShowLength(Path); // shows 50, we might have expected 40

        using (Stream dest = File.Open(Path, FileMode.OpenOrCreate)) {

            WriteGarbabe(dest, 30);

            dest.SetLength(dest.Position);

        }

        ShowLength(Path); // shows 30; SetLength works

        using (Stream dest = File.Open(Path, FileMode.Truncate)) {

            WriteGarbabe(dest, 20);

            dest.SetLength(dest.Position);

        }

        ShowLength(Path); // shows 20; Truncate works

    }

}

2009/12/22 jeevankodali <jeevan...@gmail.com>

To unsubscribe from this group, send email to protobuf+u...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.

--
Regards,

Marc

[jeevankodali]

Thanks for your help, it makes sense.

> 2009/12/22 jeevankodali <jeevankod...@gmail.com>

> -- ...
>
> read more »- Hide quoted text -

[nirav patel]

Hello Marc,

Thanks for providing protobuf-net to community. 

I am facing same error message with Protobuf-net version 3.0.29. Error message is "Invalid wire-type (Varint); this usually means you have over-written a file without truncating or setting the length"

I thought main reason is Encoding type in my case. I passed the Hex string and having Encoding type = RawEncoding then its work good. But with other data with Encoding Type = HeatshrinkLzssEncoding, it gives me above mentioned error. Should it not support this encoding type?

Thanks in advance.

2009/12/22 jeevankodali <jeevan...@gmail.com>

> > > >> prot...@googlegroups.com<protobuf%2Bunsubscribe@googlegroups.c­om>

> > <protobuf%2Bunsubscr...@googlegroups.c­om>

> > > >> .
> > > >> For more options, visit this group at
> > > >>http://groups.google.com/group/protobuf?hl=en.
>
> > > > --
> > > > Regards,
>
> > > > Marc
>
> > > --
> > > Regards,
>
> > > Marc- Hide quoted text -
>
> > > - Show quoted text -
>
> > --
>
> > You received this message because you are subscribed to the Google Groups
> > "Protocol Buffers" group.
> > To post to this group, send email to prot...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > prot...@googlegroups.com<protobuf%2Bunsubscribe@googlegroups.c­om>

> > .
> > For more options, visit this group at
> >http://groups.google.com/group/protobuf?hl=en.
>
> --
> Regards,
>

> Marc- Hide quoted text -
>
> - Show quoted text -

--

You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To post to this group, send email to prot...@googlegroups.com.

To unsubscribe from this group, send email to prot...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.

--
Regards,

Marc

[Marc Gravell]

Hi; it is pretty hard to comment without code. Do you have a minimal repro that shows what you're seeing? The fact that you mention Encoding suggests that you're treating protobuf data as though it were text, which is never a good idea.

Also, since this is library specific, you may wish to move this discussion to https://github.com/protobuf-net/protobuf-net instead.