Problem : Serialized in protobuf-net, deserialize in C++ app

[Roey]

Hi All.

I am using ProtoBuf-Net in my .NET application to serialize the
following : (in .proto format)

message ProtoScreenBuffer {
optional int32 MediaId = 1;
optional bytes Data = 2;
optional bool LastBuffer = 3;
optional int64 StartTime = 4;
optional int64 StopTime = 5;
optional int32 Flags = 6;
optional int32 BufferSubType = 7;
optional int32 BufferType = 8;
repeated int32 EncryptedDataStart = 9;
repeated int32 EncryptedDataLength = 10;
}

I serialize this into a byte array, using protobuf-net,
and pass it to the C++ app.
What goes out of the .NET and what comes into the C++ app is 100%
identical.

It then should be deserialized in the C++ app.

I try to do :

m_screenBuffer.ParseFromArray(serBuffer, dwInputDataLen);
(where m_screenBuffer is ProtoScreenBuffer object, serBuffer is the
raw byte array I got from the .NET app, and dwInputDataLen is the
length of it.)

The problem is, I get a memory access violation when I try to
deserialize it :(
Unhandled exception at 0x02f166d8 in wmplayer.exe: 0xC0000005: Access
violation writing location 0x00000000.

Are any of the things I'm doing here wrong , for what I'm trying to do
(serialize in C# .NET and deserialize in C++?)

Thanks alot.

Roey

[Evan Jones]

On Mar 11, 2010, at 8:39 , Roey wrote:
> Unhandled exception at 0x02f166d8 in wmplayer.exe: 0xC0000005:
> Access violation writing location 0x00000000.

I believe this means you are accessing a NULL pointer somewhere. You
should attach a debugger to your process and look at the stack trace
to figure out where this is happening. My unhelpful guess is that
something is not initialized correctly. Good luck,

Evan

--
Evan Jones
http://evanjones.ca/

[Michael Poole]

Roey writes:

[snip]

> The problem is, I get a memory access violation when I try to
> deserialize it :(
> Unhandled exception at 0x02f166d8 in wmplayer.exe: 0xC0000005: Access
> violation writing location 0x00000000.
>
> Are any of the things I'm doing here wrong , for what I'm trying to do
> (serialize in C# .NET and deserialize in C++?)

Most likely yes, but it's hard to suggest what that is without a brief
code example that shows the crash. The crash is obviously trying to
write into a null pointer, but only you can find out what code is at
address 0x02f166d8.

Michael Poole

[Marc Gravell]

Hmm - the data field should be pretty trivial - in terms of the wire-format it is just a field-header, a length-prefix, and then the raw data.

How big is the message when serialized? (or: how big is a short message that reproduces this problem?) Could you possibly both the BLOB, either in hex or base-64, so that I can see what the bytes are?

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To post to this group, send email to prot...@googlegroups.com.
To unsubscribe from this group, send email to protobuf+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.

--
Regards,

Marc

[Roey]

The crash occurs in my auto-generated cc file for the google proto
structs , on this line :

// optional bytes Data = 2;
case 2: {
if
(::google::protobuf::internal::WireFormatLite::GetTagWireType(tag) ==
::google::protobuf::internal::WireFormatLite::WIRETYPE_LENGTH_DELIMITED)
{
parse_Data:

DO_(::google::protobuf::internal::WireFormatLite::ReadBytes( <-----------------------
crash occurs here
input, this->mutable_data()));

I do not know what this macro/code is trying to do (I guess it's
trying to deserialize my data field)

[Roey]

Just to clarify , the access violation occurs inside the
ParseFromArray function, right when it tries to parse the second field
of the struct ("data"),
this is the code that crashes inside the cc file :

// optional bytes Data = 2;
case 2: {
if
(::google::protobuf::internal::WireFormatLite::GetTagWireType(tag) ==
::google::protobuf::internal::WireFormatLite::WIRETYPE_LENGTH_DELIMITED)
{
parse_Data:

DO_(::google::protobuf::internal::WireFormatLite::ReadBytes( <--------------

This crashes
input, this->mutable_data()));

[Kenton Varda]

OK, so, which pointer is null?

We can't do this for you.  You are the only one who can run your code in the debugger.

If you want us to debug it, you need to provide a small, self-contained example program demonstrating the problem.  Just C++ code and input data is needed -- the C# part is not necessary.

[Kenton Varda]

Please keep the mailing list CC'd so that others can help.

The input_ field is normally null when parsing from a flat array, so that's not the problem.

ReadBytes() is an inline function found in google/protobuf/wire_format_lite_inl.h.  Your debugger should be able to descend into it.  You may have to compile with optimization disabled so that it doesn't actually inline the function.

On Thu, Mar 11, 2010 at 12:11 PM, Roey Lehman <roe...@gmail.com> wrote:

Hi Kenton, thanks for the reply.

this line causes the access violation :

DO_(::google::protobuf::internal::WireFormatLite::ReadBytes(input, this->mutable_data()));

 

I checked this in the debugger. This->  , is not null, and is the message object itself.

 

Input is a CodedInputStream type, and it is not null either.

I haven't the code for ReadBytes, I don't know what goes on inside it, but I can tell you that the "input_" field of the CodedInputStream, is null. (the "buffer_" field is not null ).

 

since I can't delve into ReadBytes, I have no idea what's causing the access violation....

[Roey]

The problem was that Protocol Buffers comes with a default VS2005
project, and my application is in VS2008.
Had to convert protobuf VS2005 to VS2008 and everything runs just
fine.
It seems that one of the STL Libraries had been changed between VS8
and VS9 and so was throwing all kinds of weird error/linkage errors.

Thanks for all the help.
Roey

> >>  On Thu, Mar 11, 2010 at 5:56 AM, Roey <roey...@gmail.com> wrote:
>
> >>> Just to clarify , the access violation occurs inside the
> >>> ParseFromArray function, right when it tries to parse the second field
> >>> of the struct ("data"),
> >>> this is the code that crashes inside the cc file :
>
> >>>  // optional bytes Data = 2;
> >>>      case 2: {
> >>>        if
> >>> (::google::protobuf::internal::WireFormatLite::GetTagWireType(tag) ==
>
> >>>  ::google::protobuf::internal::WireFormatLite::WIRETYPE_LENGTH_DELIMITED)
> >>> {
> >>>         parse_Data:
>
> >>> DO_(::google::protobuf::internal::WireFormatLite::ReadBytes(
> >>>  <--------------
> >>> This crashes
> >>>                input, this->mutable_data()));
>
> >>> --
> >>> You received this message because you are subscribed to the Google Groups
> >>> "Protocol Buffers" group.
> >>> To post to this group, send email to prot...@googlegroups.com.
> >>> To unsubscribe from this group, send email to

> >>> protobuf+u...@googlegroups.com<protobuf%2Bunsubscribe@googlegroups.c­om>

> >>> .
> >>> For more options, visit this group at

> >>>http://groups.google.com/group/protobuf?hl=en.- Hide quoted text -
>
> - Show quoted text -

[Roey Lehman]

I cannot find the ReadBytes function inside wire_format_lite_inl.h.

How come?

[Marc Gravell]

I'm really glad you found the cause of this; you had me worried I'd done something horrible with the .NET encoding ;-p

Probably one for Kenton, but I wonder if it might be prudent to include VS2008 (and presumably VS2010) project files for the core project?

Marc

--
Regards,

Marc

[Austin Ziegler]

On Sun, Mar 14, 2010 at 3:15 PM, Marc Gravell <marc.g...@gmail.com> wrote:
> I'm really glad you found the cause of this; you had me worried I'd done
> something horrible with the .NET encoding ;-p
> Probably one for Kenton, but I wonder if it might be prudent to include
> VS2008 (and presumably VS2010) project files for the core project?

We've got some VS2008 versions we can contribute; we deliberately
created them in parallel to to the existing vs/ directory.

-austin
--
Austin Ziegler • halos...@gmail.com • aus...@halostatue.ca
http://www.halostatue.ca/ • http://twitter.com/halostatue

[Kenton Varda]

I actually maintain the project files using VS2008, but I run a hack script that "downgrades" them to VS2005 (by simply replacing the version number) before release so that VS2005 users can use the package.

I'm confused about how you managed to compile the project using VS2008 without it automatically "upgrading" the files.  Doesn't VS immediately prompt you to upgrade when you open them?  I'm also confused why compilation would fail, considering that the only difference between the 2005 and 2008 versions of the project files is the version number.

I definitely do not want to try to maintain two separate copies of the VS project files.  Maintaining one set is already painful enough.  I really dislike the MSVC build system.

[Roey Lehman]

appearantly I didn't notice that when I clicked the .sln file, it opened with VS2005...

I only started looking into it when compilation failed with the 2005 version.

Linking with the 2005 version failed, the linker error I got was something to do with std::Base_container contstructor/destructor. I guess that's what changed between VS2005 and VS2008.

[Kenton Varda]

I do test each release with both VS2005 and VS2008, and haven't seen that problem before.  Odd.

[Austin Ziegler]

On Tue, Mar 16, 2010 at 3:24 PM, Kenton Varda <ken...@google.com> wrote:
> I do test each release with both VS2005 and VS2008, and haven't seen that
> problem before.  Odd.

Do you have 2005 and 2008 on a single system, or are they on
independent systems? If they're on the same system, VS does not always
offer to upgrade, but opens the VS projects in the same VS that they
were created in. You have to explicitly force them to upgrade.

I agree that if you can avoid it, you shouldn't maintain two different
sets of vsproj files, but if all you're doing is downconverting a
vs2008 vsproj to vs2005, then it couldn't hurt to ship both in the
tarball, even in separate directories.

[Kenton Varda]

On Tue, Mar 16, 2010 at 6:11 PM, Austin Ziegler <halos...@gmail.com> wrote:

On Tue, Mar 16, 2010 at 3:24 PM, Kenton Varda <ken...@google.com> wrote:
> I do test each release with both VS2005 and VS2008, and haven't seen that
> problem before.  Odd.

Do you have 2005 and 2008 on a single system, or are they on
independent systems? If they're on the same system, VS does not always
offer to upgrade, but opens the VS projects in the same VS that they
were created in. You have to explicitly force them to upgrade.

Different systems.

 

I agree that if you can avoid it, you shouldn't maintain two different
sets of vsproj files, but if all you're doing is downconverting a
vs2008 vsproj to vs2005, then it couldn't hurt to ship both in the
tarball, even in separate directories.

It also wouldn't help, because they are exactly the same except for the version number, and if a VS2008 user tries to open them, it auto-converts them to VS2008.

[Austin Ziegler]

Except in the case where the user has both VS2005 and VS2008 installed
on the same system. As many of the developers at my place of work do.
If you give me VS2005 projects and I have both VS2005 and VS2008
installed, I have to explicitly open the VS2005 projects with VS2008
to make them work. This also may not work in scenarios where someone
may need to provide compiled versions that work with VS2005 and VS2008
(separately built versions). It's able to be worked around (we copied
the VS2005 projects to a new directory and forced them to be opened
with VS2008), but it is a little annoying.

[Kenton Varda]

On Tue, Mar 16, 2010 at 7:44 PM, Austin Ziegler <halos...@gmail.com> wrote:

Except in the case where the user has both VS2005 and VS2008 installed

on the same system.

Oh, right.  Sorry.

Urgh.  I'll think about it.  This is the first I've heard of this problem, though.  Is it really that common?

[Austin Ziegler]

Common enough that I think it's worth doing something about it. We're
going to be facing the VS2010 problem by the end of the year, in
house.

[Kenton Varda]

OK, can you file an issue report, then?  I suppose it would not be too terrible to make this part of the release post-processing step.