[Boost-users] Memory deallocation concerning boost::bind, boost::asio and shared_ptr

[Norman Kradepohl]

Hi there,

my code is allocating memory and never freeing it, even though it should
(at least in my opinion).

The header looks like this:

typedef boost::asio::ssl::stream<boost::asio::ip::tcp::socket>
sslSocket_t;

class Object {
boost::asio::io_service ioService_;
boost::asio::ip::tcp::acceptor acceptor_;
boost::asio::ssl::context context_;

void functionOne();
void functionTwo(shared_ptr<sslSocket_t>& sslSocket, const
boost::system::error_code& error)
}
And my source like this:

void Object::functionOne() {
for (int i = 0; i < 10; i++) {
shared_ptr<sslSocket_t> sslSocket(new
sslSocket_t(ioService_, context_));
acceptor_.async_accept(sslSocket->lowest_layer(),
boost::bind(&Object::functionTwo, this,
sslSocket, boost::asio::placeholders::error));
}
acceptor_.cancel();

boost::asio::io_service::work work(ioService_);
ioService_.run();
}

void functionTwo(shared_ptr<sslSocket_t>& sslSocket, const
boost::system::error_code& err) {
// Do nothing
}

So when i call *Object.functionOne()*, memory is getting allocated to
the *Object.ioService_* object, in order to be able to call the bound
asynchronous method. Then after the loop, all pending asynchronous
actions on the acceptor are getting canceled. The appropriate handler is
getting invoked as soon as *Object.ioService_.run()* is called (i've
been testing that). BUT for some reason, the allocated memory does not
get freed. I'm stuck on this problem for a while now, any hint or help
would be really appreciated.


Btw.: I'm working on debian and looking at "/proc/self/status" ->
"VmRSS" to whatch the used memory.


_______________________________________________
Boost-users mailing list
Boost...@lists.boost.org
http://lists.boost.org/mailman/listinfo.cgi/boost-users

[Lee Clagett]

On Thu, 28 Apr 2016 13:42:12 +0200
Norman Kradepohl <krad...@googlemail.com> wrote:

> Hi there,
>
> my code is allocating memory and never freeing it, even though it
> should (at least in my opinion).
>

[...]

>
> Btw.: I'm working on debian and looking at "/proc/self/status" ->
> "VmRSS" to whatch the used memory.
>

The C++ runtime is not required to have a 1-to-1 mapping into the OS
for new and delete calls. So in this case the shared_ptr delete calls
should be releasing the memory into the C++ runtime, which is holding
onto the memory and not releasing it the OS. Valgrind or
-fsanitize=address with newish versions of clang or gcc will provide
more accurate information on memory leaks.

If more precise control over memory management is needed, direct system
calls for acquiring and releasing memory will be necessary.

Lee

[Norman]

Yeah, was my first idea too, but I checked that.

My posted code is only a summary of the essential code passage, but i also
tried to allocate and deallocate the memory multiple times, hoping, that the
C++ runtime still "owned" the memory and hence would be able to manage the
second, third, ... and so on allocation, without acquiring more memory from
the OS, but every allocation took the same amount of memory from the OS.



--
View this message in context: http://boost.2283326.n4.nabble.com/Memory-deallocation-concerning-boost-bind-boost-asio-and-shared-ptr-tp4685746p4685749.html
Sent from the Boost - Users mailing list archive at Nabble.com.

[Norman]

Alright, i made an example, that shows my problem:

http://melpon.org/wandbox/permlink/S56tYueJySIHdsWO

Vinnie_win aka. Vinnie falco told me, that I might be looking at the wrong
indicator for my memory use. But I thought the Resident Set Size is the
value to look for, since I want to know how much memory my program is
currently allocating.

Any help would still be really appriciated :-)



--
View this message in context: http://boost.2283326.n4.nabble.com/Memory-deallocation-concerning-boost-bind-boost-asio-and-shared-ptr-tp4685746p4685760.html

Sent from the Boost - Users mailing list archive at Nabble.com.

[Lee Clagett]

On Thu, 28 Apr 2016 12:14:00 -0700 (PDT)
Norman <krad...@googlemail.com> wrote:
> Alright, i made an example, that shows my problem:
>
> http://melpon.org/wandbox/permlink/S56tYueJySIHdsWO

This does not show any problem, although I am uncertain on the intent
of the `async_accept` calls. The only non-completed handler in the
"during" stage should be the async_wait, however there is no guaranteed
sequencing between the acceptor handlers and the async_wait handler.
The counter indicates all of the other callbacks have completed in my
executions of the code.

The during and after printing only indicate the RSS is the same at
those two execution points. This does not necessarily indicate a
problem. In fact, the following `stop` call could result in a RSS
increase (additional code page mapping), and then the RSS could
decrease to the prior level after the `async_wait` and `run` complete
and free any used memory. I simply do not see any information learned
from this example.

> Vinnie_win aka. Vinnie falco told me, that I might be looking at the
> wrong indicator for my memory use. But I thought the Resident Set
> Size is the value to look for, since I want to know how much memory
> my program is currently allocating.
>
> Any help would still be really appriciated :-)
>

The RSS value indicates the number of virtual pages that have a
physical page mapping for the process. There is no guaranteed
correlation between the amount of memory requested via new/delete OR
the amount of memory being managed by the C++ runtime. "Allocated"
memory being managed by the C++ runtime can be swapped to disk which
lowers the RSS but not the amount of memory being managed by the
runtime. New code segments can be executed which requires additional
pages of the mmap'ed executable or shared library to be copied to
physical memory which will increase the RSS but not the amount of
memory being managed by the C++ runtime. A large allocation into the
C++ runtime can be fulfilled with a single anonymous mmap which delays
any RSS increase until the page(s) have been "touched". As an example
(stealing your getUsedMemory function):

#include <fstream>
#include <iostream>
#include <memory>
#include <string>

#include <boost/algorithm/string.hpp>

std::string getUsedMemory() {
try {
std::ifstream inFile("/proc/self/status");

if (!inFile.good()) {
std::cout << "Cannot open status file." << std::endl;
}

std::string line;
while (std::getline(inFile, line)) {
if (boost::starts_with(line, "VmRSS:")) {
inFile.close();
return std::string(&line[7], line.size() - 7);
}
}
inFile.close();
} catch (std::exception& e) {
std::cout << " Exception: " << e.what() << std::endl;
}
return "Unknown";
}

int main() {
std::unique_ptr<char[]> bytes(new char[50*1024*1024]); // 50 MiB
std::cout << "Used memory: " << getUsedMemory() << std::endl;
return 0;
}


That is unlikely to print 50+ MiB of "used memory". Wandbox (Gcc
5.3) prints:

Used memory: 5968 kB

And my local box prints even less. It is probably easier to use the
tools I mentioned earlier or something like massif if you are trying to
track general process memory usage or leaks.


Lee

[Niall Douglas]

On 28 Apr 2016 at 12:14, Norman wrote:

> Alright, i made an example, that shows my problem:
>
> http://melpon.org/wandbox/permlink/S56tYueJySIHdsWO
>
> Vinnie_win aka. Vinnie falco told me, that I might be looking at the wrong
> indicator for my memory use. But I thought the Resident Set Size is the
> value to look for, since I want to know how much memory my program is
> currently allocating.
>
> Any help would still be really appriciated :-)

You appear to not understand how virtual memory and the system memory
allocator works and interact with one another. Please read the papers
from Denning et al from the 1970s. All major OSs use his design.

RSS is the *cache* of memory held by the system in RAM of your
process' memory. It is entirely dependent on system load, how much
caching your kernel chooses, and what the libraries your processes do
internally.

It has little relevance to "how much memory my program is currently
allocating".

The closest proxy to such a thing - which doesn't really exist on any
major OS of the past thirty years - is probably the dirtied page
count. And even there, the system memory allocator will intentionally
cache a large quantity of memory it fetches from the OS, so just
because you free() everything you malloc() doesn't mean any memory is
released to the system.

If you really, really, really want to always free everything you
malloc, use mmap() to allocate and munmap() to free, or even use
sbrk(). Note that your program will run four orders of magntitude
slower.

tl;dr; If this really bothers you, use a custom STL allocator.
Otherwise so long as dirtied pages doesn't keep rising over time,
it's not an issue and don't worry about it.

Niall

--
ned Productions Limited Consulting
http://www.nedproductions.biz/
http://ie.linkedin.com/in/nialldouglas/

[Norman]

I understand, that the RSS is not a direct indicator for 'unused' memory.
Since I obviously have problems to clearify my problem, i made another
example:

Converging memory consumption example
<http://melpon.org/wandbox/permlink/i5k59o38K8azDX2o>

In this example, the amount of memory beeing allocated by the programm
converges to a limit, which shows exactly what you told me. So the C++
runtime holds on to some allocated memory, even though the objects have been
'freed', meaning that the application does not need to allocate more memory
for every function call.

My problem with this behaviour is, that my application should run for an
infinit amount of time. It's a server application, which allocates a greater
amount of memory in between. So if this memory is allocated for a little
while, even when the memory is 'not in use' anymore, that's fine with me.
But at some time, the memory needs to be released to the OS. And that's
exactly where my problem appears. I don't know, if the memory will ever be
reclaimed by the OS. Of course, RSS does not tell me how much memory my
application is currently using, but it tells me at least the minimum amount
of memory currently managed by the C++ runtime. And if this increases to a
huge amount, my OS does not have access to this memory, right?

So my remaining question is, if the ioService object will (or more precisly
the C++ runtime) will ever release the allocated memory to the OS or if i
have to do this manually.



--
View this message in context: http://boost.2283326.n4.nabble.com/Memory-deallocation-concerning-boost-bind-boost-asio-and-shared-ptr-tp4685746p4685966.html

Sent from the Boost - Users mailing list archive at Nabble.com.

[Oswin Krause]

Hi,

The exact behaviour depends on your run-time implementation.
It is most likely not very much memory that the run-time is claiming.
Have you seen any issue in practice? Or is it a theoretical problem like
"when do I get my 2 MB of my 64GB memory back?"

[Chris Cleeland]

On Tue, May 10, 2016 at 7:40 AM, Norman <krad...@googlemail.com> wrote:

In this example, the amount of memory beeing allocated by the programm
converges to a limit, which shows exactly what you told me. So the C++
runtime holds on to some allocated memory, even though the objects have been
'freed', meaning that the application does not need to allocate more memory
for every function call.

My problem with this behaviour is, that my application should run for an
infinit amount of time. It's a server application, which allocates a greater
amount of memory in between. So if this memory is allocated for a little
while, even when the memory is 'not in use' anymore, that's fine with me.
But at some time, the memory needs to be released to the OS. And that's
exactly where my problem appears. I don't know, if the memory will ever be
reclaimed by the OS. Of course, RSS does not tell me how much memory my
application is currently using, but it tells me at least the minimum amount
of memory currently managed by the C++ runtime. And if this increases to a
huge amount, my OS does not have access to this memory, right?

So my remaining question is, if the ioService object will (or more precisly
the C++ runtime) will ever release the allocated memory to the OS or if i
have to do this manually.

You need to ask that question of your C++ runtime and, more specifically, the heap allocator used by the runtime.

You also need to learn how your OS manages things and when it will reclaim.

RSS is not a good indicator of leaks or even of how much memory a process is "using".  All it tells you is how many OS level pages the OS currently has set aside in core for that process (or thread, if your OS gives you RSS for some more granular entity).  The value of RSS depends on many things outside your control such as other processes' demands for memory, the usage patterns for memory within your process, and the like.

If you're concerned about the memory footprint of your application, you're looking at symptoms rather than real problems.  Leaks are only one part of the problem.  Just as common is fragmentation and locality of reference.  Work to reduce those in your application, and eliminate leaks, and your app will likely exhibit better behavior with respect to memory usage.

--

Chris Cleeland

[Norman]

My first approach was to search for leaks with Valgrind and i didn't find
any. Also i never mentioned any leaks, so that's not my concern. The only
problem i had/have is, that the RSS is not decreasing during runtime, even
after a long time when my application is not using the allocated memory
anymore.
For instance i connected 30.000 dummy clients to my server, so the memory
consumption went up to ~ 50% of my available memory. Then i disconnected
them (leading to a lot of destructed objects), which should lead to a lot of
freed memory. But my application still claims 50% of my RAM, according to
RSS. If i now connect another 30.000 clients, my RSS changes only
marginally. But i cannot understand, why my application never releases any
of the claimed memory to the OS. It feels like there is some kind of a
capacity inside the application (a map-like container for instance), which
can only increase and will never decrease. Since its capacity does never
decrease, the claimed memory won't be released.

@Oswin Krause
It's more of a practical concern. Since i never observed my OS 'reclaiming'
the virtual pages, i don't know if this will ever happen. All i observed, is
that my application claimed some memory and never released it (or at least
only an alternating small portion of it).





--
View this message in context: http://boost.2283326.n4.nabble.com/Memory-deallocation-concerning-boost-bind-boost-asio-and-shared-ptr-tp4685746p4685972.html

[Chris Cleeland]

On Tue, May 10, 2016 at 9:29 AM, Norman <krad...@googlemail.com> wrote:

For instance i connected 30.000 dummy clients to my server, so the memory
consumption went up to ~ 50% of my available memory. Then i disconnected
them (leading to a lot of destructed objects), which should lead to a lot of
freed memory. But my application still claims 50% of my RAM, according to
RSS. If i now connect another 30.000 clients, my RSS changes only
marginally. But i cannot understand, why my application never releases any
of the claimed memory to the OS.

You're interpreting the data wrong.

Your app needed the memory; your app asked the heap allocator for memory; the heap allocator asked the OS; the OS gave it the memory.

Your app told the heap allocator it didn't need the memory.  What did the heap allocator do?  We don't know, but it probably held on to some of it in a free pool, or maybe all of it.  But, let's suppose it surrendered everything back (which would be silly and lead to thrash, but let's suppose)...it has no control over what the OS sets aside for your process.  At this point, it's up to the OS to decide what to do with the pages in the RSS.  If nobody else needs them, then why should it pull them back? If your app needed them at some point, and nobody else has needed them so far, then the OS is smart to just leave them in core on behalf of your process just in case it still needs them.

 

It feels like there is some kind of a
capacity inside the application (a map-like container for instance), which
can only increase and will never decrease. Since its capacity does never
decrease, the claimed memory won't be released.

Your expectation is wrong.

--

Chris Cleeland

[Norman]

Alright, let's asume you are right.

Can you explain the following behaviour: i create a vector<string> and fill
it with a huge amount of strings, so it takes up to ~50% of my RSS. After
the vector is filled, i observe the RSS and as expected, it's quite big.
Then i clear the vector and let it run out of scope and something special
happenes: The RSS is decreased by a significant amount (close to what it was
before i created the vector)! This should not happen in your scenario,
right, since the OS does not need the memory, but it still does get
reclaimed by the OS.

So i'm really sry, but this procedure, where memory is sometimes hold on to
and sometimes not does make no sense to me. I do understand that the OS
memory management tries to optimize the access onto the memory and hence not
all memory might be reclaimed right away, but not in this scale. I've never
seen programs holding on to memory forever until the OS reclaims it. And im
not talking about a few bytes here and there, i'm talking about hundreds of
megabytes.

By the way, the behaviour, described with a vector<string> also works with a
vector<shared_ptr&lt;sslSocket>>, as long as no assynchronous job, depending
on the sslSocket, has been given to the acceptor/ioService.



--
View this message in context: http://boost.2283326.n4.nabble.com/Memory-deallocation-concerning-boost-bind-boost-asio-and-shared-ptr-tp4685746p4685978.html

[Chris Cleeland]

On Tue, May 10, 2016 at 10:59 AM, Norman <krad...@googlemail.com> wrote:

Alright, let's asume you are right.

Can you explain the following behaviour: i create a vector<string> and fill
it with a huge amount of strings, so it takes up to ~50% of my RSS. After
the vector is filled, i observe the RSS and as expected, it's quite big.
Then i clear the vector and let it run out of scope and something special
happenes: The RSS is decreased by a significant amount (close to what it was
before i created the vector)! This should not happen in your scenario,
right, since the OS does not need the memory, but it still does get
reclaimed by the OS.

You're still ignoring the heap allocator.  Your application does not interact directly with the OS--it goes through several layers.

In the case you cite above--vector<string>--it's probably pretty easy for the heap allocator to identify that there is a whole bunch of memory that can be returned because it was all allocated together.  In your real world application, you more than likely suffer from fragmentation that prevents pages from being returned-- as long as just a single byte of memory from a page is in use by the heap allocator, that page cannot be surrendered to the OS.  Heap allocators cannot move stuff around in memory (unless they return pointers-to-pointers like the original Mac Toolbox did with its dynamic allocations), so it's just stuck until your app frees that up.  The best the heap allocator can do is coalesce free blocks for future surrender.

By the way, the behaviour, described with a vector<string> also works with a
vector<shared_ptr&lt;sslSocket>>, as long as no assynchronous job, depending
on the sslSocket, has been given to the acceptor/ioService.

Maybe that should give you a clue?

Asynchronous job?  So there is a thread holding on to the shared ptr?  does the thread need to be joined?

--

Chris Cleeland

[Norman]

It took me a while, but i finally managed to work it out by myself.

*So to clearify things, let's make sure, that the root of my problem is
understood:*
I'm developing a server application, which is meant to run for an infinit
amount of time. This application must be able to handle a lot of concurrent
incomming connections. At some point in time, there may be a peak in load,
leading to a lot of claimed memory to my application. Then after a while,
most incomming requests have been processed, causing a lot of objects to be
freed in runtime. Since the OS is in no need for memory (My application is
the only huge memory consumer on the server), all freed memory stays with my
application and can be reused at another point in time. This would be
absolutely fine with me, but some customers and administrators might
misinterpret the greater amount of constantly claimed memory as a memory
leaking application. To avoid this, i wanted to hand some of the unused
memory back to the OS manually. In my example, the bound handlers to the
*ioService* (e.g. accepting a new connection) would be freed in runtime, but
the appropriate memory won't be reclaimed by the OS. So to do this manually,
i found the following solution:

*Release unused heap memory under Linux in C/C++: int malloc_trim(size_t
pad)*

The documentation can be found here
<http://man7.org/linux/man-pages/man3/malloc_trim.3.html> . Simplified
explained, this method releases unused memory from the heap to the OS, which
is exactly what i've been searching for. I'm aware that under memory
optimization aspects, the manual use of this function maybe dangerous, but
since i only want to release the memory every few minutes, this performance
issue is acceptable to me.

Thank you all for your efforts and patience!



--
View this message in context: http://boost.2283326.n4.nabble.com/Memory-deallocation-concerning-boost-bind-boost-asio-and-shared-ptr-tp4685746p4686115.html

[Chris Cleeland]

On Tue, May 17, 2016 at 6:15 AM, Norman <krad...@googlemail.com> wrote:

In my example, the bound handlers to the
*ioService* (e.g. accepting a new connection) would be freed in runtime, but
the appropriate memory won't be reclaimed by the OS. So to do this manually,
i found the following solution:

*Release unused heap memory under Linux in C/C++: int malloc_trim(size_t
pad)*

Simplified explained, this method releases unused memory from the heap to the OS, which
is exactly what i've been searching for.

Note that in the documentation there is no guarantee that this will actually do anything. A good example of where this may not have much of an effect is when the heap gets fragmented, with a small remaining in-use allocation sitting near the top of heap and unused space beneath it.

While I understand the perception issue you face with your customers, in my experience you're far better in the long run to educate your customers or provide tools to show the ACTUAL memory use by your application rather than potentially thrashing the virtual memory system just to make things comport with their erroneous expectations.

--

Chris Cleeland