BufferedStream memory leak?

[Matt Billenstein]

Hey guys, thanks for setting this list up ;)

One issue I've had is using BufferedStream -- seems no matter what I do,
I have a memory leak that I don't have just using a regular
SocketStream. I wrote up a simple testcase here:

http://vazor.com/mordor_bufferedstream_leak.tar.gz

I've also pastebin'd the server here: http://pastebin.com/S9xwEXVL

If you change the line #define LEAKY to true, you'll see the inner loop
receiving/sending messages seems to be leaking memory...

Anything obvious you see I'm doing wrong here? Also, any comments on
anything else here appreciated -- my c++ is a bit rusty and Mordor/boost
is new to me.

thx

Matt

--
Matt Billenstein
ma...@vazor.com
http://www.vazor.com/

[Cody Cutrer]

Matt,

Interesting.  I've got a couple of questions for you: what OS are you running on, and does the server get any exceptions?  I'm running on Windows, and what I see is that when the server doesn't get any exceptions, the BufferedStream vs. stack buffer end up with the same memory usage.  However, when it does get exceptions, memory usage is much higher at the end.  Which means that maybe there's some problem cleaning up the stacks of fibers that terminate in an exception?  Anyhow, I had to modify your server and client a bit to make sure there were no exceptions.  On the server, I just had to make sure to do the compare and write correctly for the stack buffer version (your buffer wasn't null terminated, but you were using strcmp and write(const char *) which expect null terminated versions).  On the client side, I needed to read the response to the quit, and make sure the socket closes gracefully.  My modified server is at http://pastebin.com/QJHBsT6N, and my client is at http://pastebin.com/eDVVwpE9.  I also added to the server to allow it to shut down cleanly (Ctrl-Break in a windows console Window, SIGTERM on other platforms).  That should allow it to be run under Valgrind (though it's been a while since I've run under valgrind).

Cody

[Matt Billenstein]

Hey Cody,

I'm on Ubuntu Linux -- my tests weren't throwing any exceptions -- I
thought I might have had a leak in there which you pointed out.

Basically, I spin up 500 connections, pass messages back and forth, shut
them down cleanly, then sleep for 30 seconds before starting this
process over again.

Using your server, I still see a leak, so perhaps this is specific to
Linux if you were testing on Windows -- if I watch the memory usage
every 10 seconds, it looks something like:

http://pastebin.com/2J4RDdz8 (sixth column here is RSS)

It goes up linearly while we're passing messages back and forth, then
drops back to almost what it started at once all the connections shut
down cleanly.

Here is the non-BufferedStream version which is flat:

http://pastebin.com/kn6TJ0Km

I did try valgrind last week, but I'll try it again -- when I was
killing the server before, it wasn't shutting down cleanly and the
output seemed impossible to analyze so your fixes will probably heap
this.

thx

m

On Tue, Mar 08, 2011 at 09:35:35AM -0700, Cody Cutrer wrote:
> Matt,
> Interesting. *I've got a couple of questions for you: what OS are you
> running on, and does the server get any exceptions? *I'm running on

> Windows, and what I see is that when the server doesn't get any
> exceptions, the BufferedStream vs. stack buffer end up with the same

> memory usage. *However, when it does get exceptions, memory usage is much
> higher at the end. *Which means that maybe there's some problem cleaning
> up the stacks of fibers that terminate in an exception? *Anyhow, I had to

> modify your server and client a bit to make sure there were no exceptions.

> *On the server, I just had to make sure to do the compare and write

> correctly for the stack buffer version (your buffer wasn't null
> terminated, but you were using strcmp and write(const char *) which expect

> null terminated versions). *On the client side, I needed to read the
> response to the quit, and make sure the socket closes gracefully. *My
> modified server is at*[1]http://pastebin.com/QJHBsT6N, and my client is
> at*[2]http://pastebin.com/eDVVwpE9. *I also added to the server to allow

> it to shut down cleanly (Ctrl-Break in a windows console Window, SIGTERM

> on other platforms). *That should allow it to be run under Valgrind

> (though it's been a while since I've run under valgrind).
> Cody
>

> On Mon, Mar 7, 2011 at 7:33 PM, Matt Billenstein <[3]ma...@vazor.com>

> wrote:
>
> Hey guys, thanks for setting this list up ;)
>
> One issue I've had is using BufferedStream -- seems no matter what I do,
> I have a memory leak that I don't have just using a regular

> SocketStream. *I wrote up a simple testcase here:
>
> [4]http://vazor.com/mordor_bufferedstream_leak.tar.gz
>
> I've also pastebin'd the server here: *[5]http://pastebin.com/S9xwEXVL

>
> If you change the line #define LEAKY to true, you'll see the inner loop
> receiving/sending messages seems to be leaking memory...
>

> Anything obvious you see I'm doing wrong here? *Also, any comments on

> anything else here appreciated -- my c++ is a bit rusty and Mordor/boost
> is new to me.
>
> thx
>
> Matt
> --
> Matt Billenstein

> [6]ma...@vazor.com
> [7]http://www.vazor.com/
>
> References
>
> Visible links
> 1. http://pastebin.com/QJHBsT6N
> 2. http://pastebin.com/eDVVwpE9
> 3. mailto:ma...@vazor.com
> 4. http://vazor.com/mordor_bufferedstream_leak.tar.gz
> 5. http://pastebin.com/S9xwEXVL
> 6. mailto:ma...@vazor.com
> 7. http://www.vazor.com/

[Matt Billenstein]

Another piece of info from valgrind:

==13262== 13,473 bytes in 499 blocks are possibly lost in loss record 1,088 of 1,102
==13262== at 0x4C2818A: operator new(unsigned long) (vg_replace_malloc.c:261)
==13262== by 0x57D1D98: std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&) (in /usr/lib/libstdc++.so.6.0.13)
==13262== by 0x57D28B4: ??? (in /usr/lib/libstdc++.so.6.0.13)
==13262== by 0x57D2A52: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, std::allocator<char> const&) (in /usr/lib/libstdc++.so.6.0.13)
==13262== by 0x410BFB: device_connection(boost::shared_ptr<Mordor::Socket>) (test_leak2.cpp:47)
==13262== by 0x4217C3: void boost::_bi::list1<boost::_bi::value<boost::shared_ptr<Mordor::Socket> > >::operator()<void (*)(boost::shared_ptr<Mordor::Socket>), >boost::_bi::list0>(boost::_bi::type<void>, void >(*&)(boost::shared_ptr<Mordor::Socket>), boost::_bi::list0&, int) >(bind.hpp:246)
==13262== by 0x41F62C: boost::_bi::bind_t<void, void (*)(boost::shared_ptr<Mordor::Socket>), boost::_bi::list1<boost::_bi::value<boost::shared_ptr<Mordor::Socket> > > >::operator()() (bind_template.hpp:20)
==13262== by 0x41D476: boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, void (*)(boost::shared_ptr<Mordor::Socket>), boost::_bi::list1<boost::_bi::value<boost::shared_ptr<Mordor::Socket> > > >, void>::invoke(boost::detail::function::function_buffer&) > >(function_template.hpp:153)
==13262== by 0x4F84527: Mordor::Fiber::entryPoint() (function_template.hpp:1013)
==13262== by 0x5F1CCAF: ??? (in /lib/libc-2.11.1.so)

Which seems to indicate I'm leaking the message from the call to
getDelimited() through the string line there somehow... The string
should just be copied from the function call, so I'm a little confused
by this.

m

On Tue, Mar 08, 2011 at 09:35:35AM -0700, Cody Cutrer wrote:
> Matt,

> Interesting. *I've got a couple of questions for you: what OS are you
> running on, and does the server get any exceptions? *I'm running on

> Windows, and what I see is that when the server doesn't get any
> exceptions, the BufferedStream vs. stack buffer end up with the same

> memory usage. *However, when it does get exceptions, memory usage is much
> higher at the end. *Which means that maybe there's some problem cleaning
> up the stacks of fibers that terminate in an exception? *Anyhow, I had to

> modify your server and client a bit to make sure there were no exceptions.

> *On the server, I just had to make sure to do the compare and write

> correctly for the stack buffer version (your buffer wasn't null
> terminated, but you were using strcmp and write(const char *) which expect

> null terminated versions). *On the client side, I needed to read the
> response to the quit, and make sure the socket closes gracefully. *My
> modified server is at*[1]http://pastebin.com/QJHBsT6N, and my client is

> at*[2]http://pastebin.com/eDVVwpE9. *I also added to the server to allow

> it to shut down cleanly (Ctrl-Break in a windows console Window, SIGTERM

> on other platforms). *That should allow it to be run under Valgrind

> (though it's been a while since I've run under valgrind).
> Cody
>

> On Mon, Mar 7, 2011 at 7:33 PM, Matt Billenstein <[3]ma...@vazor.com>

> wrote:
>
> Hey guys, thanks for setting this list up ;)
>
> One issue I've had is using BufferedStream -- seems no matter what I do,
> I have a memory leak that I don't have just using a regular

> SocketStream. *I wrote up a simple testcase here:
>
> [4]http://vazor.com/mordor_bufferedstream_leak.tar.gz
>
> I've also pastebin'd the server here: *[5]http://pastebin.com/S9xwEXVL

>
> If you change the line #define LEAKY to true, you'll see the inner loop
> receiving/sending messages seems to be leaking memory...
>

> Anything obvious you see I'm doing wrong here? *Also, any comments on

> anything else here appreciated -- my c++ is a bit rusty and Mordor/boost
> is new to me.
>
> thx
>
> Matt
> --
> Matt Billenstein

> [6]ma...@vazor.com
> [7]http://www.vazor.com/
>
> References
>
> Visible links
> 1. http://pastebin.com/QJHBsT6N
> 2. http://pastebin.com/eDVVwpE9
> 3. mailto:ma...@vazor.com
> 4. http://vazor.com/mordor_bufferedstream_leak.tar.gz
> 5. http://pastebin.com/S9xwEXVL
> 6. mailto:ma...@vazor.com
> 7. http://www.vazor.com/

--

[Cody Cutrer]

It looks like you hit Ctrl-C and killing the process, rather than sending SIGTERM and shutting down gracefully.  I'm now running on lucid amd64, and I get the exact same output regardless of if LEAKY is true or false, or if I actually handle any connections or not:

LEAK SUMMARY:

==30822==    definitely lost: 0 bytes in 0 blocks

==30822==    indirectly lost: 0 bytes in 0 blocks

==30822==      possibly lost: 288 bytes in 1 blocks

==30822==    still reachable: 1,616 bytes in 11 blocks

==30822==         suppressed: 0 bytes in 0 blocks

You mention that memory usage goes back down after your connections are all done.  That seems to me just misunderstood expectations, rather an a memory leak.  When using a BufferedStream, the default buffer size is 64KB, which is much larger than the 1KB buffer you're using with the stack based buffer.  It might also be using more than that if a single "line" crosses a 64KB boundary.  This just adds up to the BufferedStream version using a lot more memory than the stack based buffer, but once the connections leave, that memory is released.

Cody

[Matt Billenstein]

On Wed, Mar 09, 2011 at 02:51:57PM -0700, Cody Cutrer wrote:
> You mention that memory usage goes back down after your connections

> are all done. *That seems to me just misunderstood expectations,
> rather an a memory leak. *When using a BufferedStream, the default

> buffer size is 64KB, which is much larger than the 1KB buffer

> you're using with the stack based buffer. *It might also be using
> more than that if a single "line" crosses a 64KB boundary. *This

> just adds up to the BufferedStream version using a lot more memory
> than the stack based buffer, but once the connections leave, that
> memory is released.

Yeah, you're right, the memory isn't really leaked since it comes back
down after the fiber finishes. The absolute value doesn't bother me,
it's that it never levels off after all the connections are established.

I looked at the BufferedStream's bufferSize() and that seems to stay
constant at 64k, so I'm at a bit of a loss about this one -- I think
I'll put it aside for now and try to debug it more in the next week or
two. The char * buffer works for now.

thx

m