double free or corruption
Paulo R. Panhoto
From time to time, I get a "double free or corruption" error in my
app, always in the same point. This is an app with multiple data
sources, so it is a threaded app. To prevent synchronization issues,
each worker thread receives the message, preprocesses the packet and
passes it onto a thread local BufferReceiver.
After searching the forum, I've read there might be threading issues
related to SynchReceivers (such as BufferReceiver itself). Is that
plausible, even if the BufferReceivers are thread locals?
Regards, Paulo.
P.S: Stack trace below
#0 0x000000321ec32945 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/
linux/raise.c:64
#1 0x000000321ec34125 in abort () at abort.c:92
#2 0x000000321ec6f82b in __libc_message (do_abort=2,
fmt=0x321ed547f8 "*** glibc detected *** %s: %s: 0x%s ***\n")
at ../sysdeps/unix/sysv/linux/libc_fatal.c:186
#3 0x000000321ec75156 in malloc_printerr (action=3, str=0x321ed52945
"free(): invalid pointer",
ptr=<value optimized out>) at malloc.c:6283
#4 0x00007fcfec669320 in
QuickFAST::Messages::FieldIdentity::~FieldIdentity() ()
from /usr/local/lib/libQuickFAST.so
#5 0x00007fcfec6693be in
QuickFAST::Messages::FieldIdentity::freeFieldIdentity() const ()
from /usr/local/lib/libQuickFAST.so
#6 0x00007fcfec655345 in
QuickFAST::Codecs::FieldInstructionDecimal::decodeNop(QuickFAST::Codecs::DataSource&,
QuickFAST::Codecs::PresenceMap&, QuickFAST::Codecs::Decoder&,
QuickFAST::Messages::ValueMessageBuilder&) const () from /usr/local/
lib/libQuickFAST.so
#7 0x00007fcfec604f58 in
QuickFAST::Codecs::Decoder::decodeSegmentBody(QuickFAST::Codecs::DataSource&,
QuickFAST::Codecs::PresenceMap&,
boost::shared_ptr<QuickFAST::Codecs::SegmentBody const> const&,
QuickFAST::Messages::ValueMessageBuilder&) () from /usr/local/lib/
libQuickFAST.so
#8 0x00007fcfec6050b8 in
QuickFAST::Codecs::Decoder::decodeGroup(QuickFAST::Codecs::DataSource&,
boost::shared_ptr<QuickFAST::Codecs::SegmentBody const> const&,
QuickFAST::Messages::ValueMessageBuilder&) ()
from /usr/local/lib/libQuickFAST.so
#9 0x00007fcfec624831 in
QuickFAST::Codecs::FieldInstructionSequence::decodeNop(QuickFAST::Codecs::DataSource&,
QuickFAST::Codecs::PresenceMap&, QuickFAST::Codecs::Decoder&,
QuickFAST::Messages::ValueMessageBuilder&) const () from /usr/local/
lib/libQuickFAST.so
#10 0x00007fcfec604f58 in
QuickFAST::Codecs::Decoder::decodeSegmentBody(QuickFAST::Codecs::DataSource&,
QuickFAST::Codecs::PresenceMap&,
boost::shared_ptr<QuickFAST::Codecs::SegmentBody const> const&,
QuickFAST::Messages::ValueMessageBuilder&) () from /usr/local/lib/
libQuickFAST.so
#11 0x00007fcfec6062cd in
QuickFAST::Codecs::Decoder::decodeMessage(QuickFAST::Codecs::DataSource&,
QuickFAST::Messages::ValueMessageBuilder&) () from /usr/local/lib/
libQuickFAST.so
#12 0x00007fcfec5fc8ef in
QuickFAST::Codecs::StreamingAssembler::serviceQueue(QuickFAST::Communication::Receiver&)
() from /usr/local/lib/libQuickFAST.so
#13 0x0000000000442903 in
QuickFAST::Communication::SynchReceiver::tryServiceQueue() ()
#14 0x0000000000443038 in
QuickFAST::Communication::BufferReceiver::receiveBuffer(unsigned char
const*, unsigned long) ()
////////////============== App stack
#15 0x000000000045f65d in
Connection::receive(QuickFAST::Communication::Receiver&, unsigned char
const*, int) ()
#16 0x00000000004603f4 in Connection::messageLoop() ()
#17 0x00000000004245ba in Thread::threadRoutine(void*) ()
Dale Wilson
know when used that way it is quite safe -- and it has always been so.
Thread-local is actually a stronger guarantee than is actually needed.
Your code must be written so only one thread at a time can be touching
the buffer receiver, the decoder, etc.
The problem I see in the stack has to do with the smart pointer to a
FieldIdentity. (FieldIdentityCPtr) Unlike a normal boost smart
pointer, the pointer to a field identity is not threadsafe. It's is OK
to use it as a pointer in your application code to get to the underlying
field identity, but you should never copy it (always pass it by
reference, not by value, never save a local copy of the pointer, etc.)
In fact making this a pointer was a minor design mistake in QuickFAST.
This should not have been a pointer at all, but rather should have been
a reference to the FieldIdentity created at template-parse time.
Unfortunately fixing this would break a lot of existing code, so the
pointer is still passed as the argument.
Just remember not to copy it.
Dale
--
Dale Wilson
Principal Software Engineer
Object Computing, Inc.
Paulo R. Panhoto
the same point. One thread fails with the message "invalid pointer"
and the second one fails with "double free or corruption (fasttop)"
My first attempt to solve it will be place a lock to wrap
BufferReceiver::ReceiveBuffer(). Though, I am wondering about the why
was that FieldIdentity freed at that point.
This is how I see the concept (probably wrong): A FAST template is
just a descriptor of a finite state machine and the Field Identities
are part of this. It would make sense to allocate all FieldIdentities
when the XML is parsed and only free them on software termination.
Every instance of encoder and decoder would only iterate over this
finite state machine in a read-only manner and, at the same time, keep
the writable data (thread) local without any locks or synchronisation
problems.
Would it do any harm?
-- Paulo
Dale Wilson
> On a second look, I found a second thread calling the same free() in
> the same point. One thread fails with the message "invalid pointer"
> and the second one fails with "double free or corruption (fasttop)"
>
> My first attempt to solve it will be place a lock to wrap
> BufferReceiver::ReceiveBuffer(). Though, I am wondering about the why
> was that FieldIdentity freed at that point.
>
> This is how I see the concept (probably wrong): A FAST template is
> just a descriptor of a finite state machine and the Field Identities
> are part of this. It would make sense to allocate all FieldIdentities
> when the XML is parsed and only free them on software termination.
> Every instance of encoder and decoder would only iterate over this
> finite state machine in a read-only manner and, at the same time, keep
> the writable data (thread) local without any locks or synchronisation
> problems.
wrote it there were a few warts (fortunately I think very few) like
pointers rather than references to FieldIdentities.
All FieldIdentities can and should be allocated during the
initialization phase of the program.
Dale
Paulo Panhoto
I solved the problem surrounding the calls to receiveBuffer()/poll() with a lock.
Regards,
Paulo.
Hi Paulo!
I've faced with the same problem.
On Dec 7 2011, 1:31 am, "Paulo R. Panhoto" <paulo.panh...@gmail.com>
wrote:
> This is how I see the concept (probably wrong): A FAST template is
> just a descriptor of a finite state machine and the Field Identities
> are part of this. It would make sense to allocate all FieldIdentities
> when the XML is parsed and only free them on software termination.
> Every instance of encoder and decoder would only iterate over this
> finite state machine in a read-only manner and, at the same time, keep
> the writable data (thread) local without any locks or synchronisation
> problems.
>
> Would it do any harm?
You should create a separate instance of FAST template registry for
each Receiver if you use several instances of them in a multithreading
environment. This will fix the problem. I tried to send a description
of this problem by all my emails do not reach the quickfast_users
list :(
--
Max
Max Dmitrichenko
I've faced with the same problem.
wrote:
> just a descriptor of a finite state machine and the Field Identities
> are part of this. It would make sense to allocate all FieldIdentities
> when the XML is parsed and only free them on software termination.
> Every instance of encoder and decoder would only iterate over this
> finite state machine in a read-only manner and, at the same time, keep
> the writable data (thread) local without any locks or synchronisation
> problems.
>
> Would it do any harm?