valgrind invalid write and double free errors

[CB]

I've been using protobuf 2.3.0 for several weeks, using
SerializeToString. The
other day, I switched to using SerializeToOstream, and started seeing
the
following invalid write and double free errors reported by valgrind
when my
program is terminated;

==15778== Invalid write of size 4
==15778== at 0x6623464:
google::protobuf::FileDescriptorTables::~FileDescriptorTables()
(hashtable:532)
==15778== by 0x4D821BE: __run_exit_handlers (exit.c:78)
==15778== by 0x4D8222E: exit (exit.c:100)
==15778== by 0x4D69BDD: (below main) (libc-start.c:258)
==15778== Address 0x635dc80 is 0 bytes inside a block of size 48
free'd
==15778== at 0x4024851: operator delete(void*) (vg_replace_malloc.c:
387)
==15778== by 0x4D821BE: __run_exit_handlers (exit.c:78)
==15778== by 0x4D8222E: exit (exit.c:100)
==15778== by 0x4D69BDD: (below main) (libc-start.c:258)
==15778==
==15778== Invalid free() / delete / delete[]
==15778== at 0x4024851: operator delete(void*) (vg_replace_malloc.c:
387)
==15778== by 0x4D821BE: __run_exit_handlers (exit.c:78)
==15778== by 0x4D8222E: exit (exit.c:100)
==15778== by 0x4D69BDD: (below main) (libc-start.c:258)
==15778== Address 0x635dc80 is 0 bytes inside a block of size 48
free'd
==15778== at 0x4024851: operator delete(void*) (vg_replace_malloc.c:
387)
==15778== by 0x4D821BE: __run_exit_handlers (exit.c:78)
==15778== by 0x4D8222E: exit (exit.c:100)
==15778== by 0x4D69BDD: (below main) (libc-start.c:258)

The above are one example of several reports. it appears there is one
set of
these for each proto message I have defined.

I've tried to recreate this error in a smaller example program for
several days
without success. Am wondering if anyone on the list has seen this
problem, and
knows what might cause it?

I'm using SerializeToOstream in basically the following manner;


char buf[512];
std::stringbuf sbuf;
sbuf.pubsetbuf((char*)buf, 512);
std::ostream os(&sbuf);
message.SerializeToOstream(&os);

Any feedback on how to further debug this problem would be
appreciated.

--CB

[Evan Jones]

On Oct 13, 2010, at 16:53 , CB wrote:
> Any feedback on how to further debug this problem would be
> appreciated.

You aren't doing anything strange like using dlopen() to dynamically
load/unload libraries, are you? I can't think of anything obvious that
might cause this kind of error. The FileDescriptorTables are "static"
objects of sorts, I think. Are you calling ShutdownProtobufLibrary()
somewhere? Maybe more than once? Memory leaks *will* be reported by
valgrind if you don't call ShutdownProtobufLibrary(), but I don't know
what could cause a double free.

Evan

--
Evan Jones
http://evanjones.ca/

[CB]

>
> You aren't doing anything strange like using dlopen() to dynamically  
> load/unload libraries, are you?

Actually, yes, we have a shared library containing our protobuf code,
which we do load with dlopen. A command line option tells the app
which protocol it needs to use, and the app loads the appropriate
library. The open only happens once, very shortly after program
launch. We're not constantly loading and unloading.

This has worked very well, for several weeks, using the
SerializeToString functions. Only when we added use of
SerializeToOstream did we start to have this issue.

I can't think of anything obvious that  
> might cause this kind of error. The FileDescriptorTables are "static"  
> objects of sorts, I think.

As my example code showed, I'm creating an ostream that is wrapped
around a string buffer. The buffer and the ostream are transient
objects. What are the 'static' FileDescriptors protobuf is holding?
What's the relation between those and my transient ostreams?

Are you calling ShutdownProtobufLibrary()  
> somewhere? Maybe more than once? Memory leaks *will* be reported by  
> valgrind if you don't call ShutdownProtobufLibrary(), but I don't know  
> what could cause a double free.
>

No, there is no call to ShutdownProtobufLibrary(). I read the bits
about that in the c++ tutorial, and I will try to work that call into
the code as I try to weed out the leaks, but those are secondary
concern to the double free.

[Evan Jones]

On Oct 14, 2010, at 11:32 , CB wrote:
> Actually, yes, we have a shared library containing our protobuf code,
> which we do load with dlopen. A command line option tells the app
> which protocol it needs to use, and the app loads the appropriate
> library. The open only happens once, very shortly after program
> launch. We're not constantly loading and unloading.

Do you ever call dlclose() on this library? Protobuf has some
complicated initialization time and shutdown clean up code buried in
descriptor.cc that I don't really understand. At the very least, there
is a call to this:

internal::OnShutdown(&DeleteGeneratedPool);

I'm a little surprised that I don't see that function appear on your
stack trace, if that is in fact the problem, but it must be something
like that. Could you try adding a printf() to the
DeleteGeneratedPool() function in protobuf/descriptor.cc and see if
that is getting called multiple times?

This FileDescriptorTable object is used internally by the protobuf
library and I don't really understand it. I'm hoping someone who might
understand this code might be able to suggest where this double free
could be coming from.

[CB]

On Oct 14, 2:30 pm, Evan Jones <ev...@MIT.EDU> wrote:
> On Oct 14, 2010, at 11:32 , CB wrote:
>
> > Actually, yes, we have a shared library containing our protobuf code,
> > which we do load with dlopen.  A command line option tells the app
> > which protocol it needs to use, and the app loads the appropriate
> > library.  The open only happens once, very shortly after program
> > launch.  We're not constantly loading and unloading.
>
> Do you ever call dlclose() on this library? Protobuf has some  
> complicated initialization time and shutdown clean up code buried in  
> descriptor.cc that I don't really understand. At the very least, there  
> is a call to this:
>

No, we never dlclose() it. Once open it stays open until the program
exits.

[Kenton Varda]

These don't look like they have anything to do with the I/O change.  The "FileDescriptor" class describes the contents of a .proto file (it's not related to OS file descriptors).  The problems you're seeing appear to be happening while running global destructors -- i.e. at program shutdown.  Are you sure that this is the result of switching from strings to ostreams?  I strongly suspect that you changed something else at the same time which lead to the problem, or the problem actually existed before the change and you didn't notice for some reason.

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To post to this group, send email to prot...@googlegroups.com.
To unsubscribe from this group, send email to protobuf+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.