Large number of stacks with callee Autobahn-cpp example

[Chris Welch]

I'm looking at using Autobahn-cpp in an embedded arm system with bonefish as the WAMP router.

I built the callee example, but was surprised to see it has a very large memory foot print (a number of processes have been removed from this listing):

Mem: 61968K used, 177892K free, 0K shrd, 748600K buff, 235092K cached

CPU:   1% usr   9% sys   0% nic  88% idle   0% io   0% irq   0% sirq

Load average: 0.19 0.28 0.31 1/126 2641

  PID  PPID USER     STAT   VSZ %MEM %CPU COMMAND

 2622  1031 root     S    54500  23%   0% ./callee -r map300 -p 8888

 1025  1024 root     S     8104   3%   0% ./bonefish --realm map300 --websocket-port 8080 --rawsocket-port 8888

This so I dug in a bit, and pmap shows 6 "stack" signatures (we have 8K stacks on this box):

2622:   ./callee -r map300 -p 8888

00010000    336K r-x-- callee

00073000      4K r---- callee

00074000      4K rw--- callee

00075000    148K rw---   [ anon ]

b3a00000      4K -----   [ anon ]

b3a01000   8188K rw---   [ anon ]

b4200000      4K -----   [ anon ]

b4201000   8188K rw---   [ anon ]

b4a00000      4K -----   [ anon ]

b4a01000   8188K rw---   [ anon ]

b5200000      4K -----   [ anon ]

b5201000   8188K rw---   [ anon ]

b5a00000      4K -----   [ anon ]

b5a01000   8320K rw---   [ anon ]

b6221000    892K -----   [ anon ]

b63a8000      4K -----   [ anon ]

b63a9000   8188K rw---   [ anon ]

b6ba8000    412K r-x-- libm.so.6

b6c0f000     60K ----- libm.so.6

b6c1e000      4K r---- libm.so.6

b6c1f000      4K rw--- libm.so.6

but it only has one thread (from cat /proc/XXXX/status):

VmPMD:         0 kB

VmSwap:        0 kB

Threads:        1

SigQ:   0/1742

SigPnd: 0000000000000000

ShdPnd: 0000000000000000

Historically I've seen this when threads are not allowed to properly terminate, the stack memory gets orphaned and is not released until the program exits.

Any suggestions as this doesn't scale well enough for me to use Autobahn-cpp?

Thanks!

[Chris Welch]

Specifically, this looks like stack leaks where threads haven't been joined.

[Chris Welch]

This appears to be a consequence of the boost futures.  Is there anyway to avoid the high number of stacks required?

I'll have to look for another library if  I can't reduce the number of stacks as this will quickly exhaust our available memory and makes it unusable.

Thanks in advance for any suggestions!

[Tobias Oberstein]

Hi Chris,

> This so I dug in a bit, and pmap shows 6 "stack" signatures (we have 8K
> stacks on this box):

..

> but it only has one thread (from cat /proc/XXXX/status):

Right, this seems weird/wrong.

Any chance you could dig deeper by running the client under valgrind
memcheck to possibly find out more?

Another test: rip out everything from the client but the establishment
of the connection (up to onjoin) and let the client just sit still.

Yet another: do you see the same when the caller/publisher/subscriber
(rather the callee)?

Cheers,
/Tobias

[Chris Welch]

Any chance you could dig deeper by running the client under valgrind
memcheck to possibly find out more?

It is a bit tricky to do this on the target, but I'll spend a bit of time and see if I can get some results.  I had little success with valgrind with this type of problem (detecting leaked stacks) in the past, but will see if I can get additional details.

I'm able to reproduce the same problem running callee in Ubunto, so I'll try getting Valgrind results from there first.

 

Another test: rip out everything from the client but the establishment
of the connection (up to onjoin) and let the client just sit still.

I commented out the providers and the stacks shrunk by two.  There appears to be a one to one correspondence to the boost future and the stack.  Here is the pmap result:

~/bin/pmap 21682

21682:   ./callee -r myrealm -p 8888

00010000    316K r-x-- callee

0006e000      4K r---- callee

0006f000      4K rw--- callee

00070000    148K rw---   [ anon ]

b4a00000      4K -----   [ anon ]

b4a01000   8188K rw---   [ anon ]

b5200000      4K -----   [ anon ]

b5201000   8188K rw---   [ anon ]

b5a00000      4K -----   [ anon ]

b5a01000   8320K rw---   [ anon ]

b6221000    892K -----   [ anon ]

b6389000      4K -----   [ anon ]

b638a000   8188K rw---   [ anon ]

b6b89000    412K r-x-- libm.so.6

I also ran callee under Ubuntu (x86_64) and it shows the same stack pattern and some other very large memory requirement (this is the original callee source code):

0000000000400000    492K r-x-- callee

000000000067a000      4K r---- callee

000000000067b000      4K rw--- callee

000000000067c000      4K rw---   [ anon ]

0000000002506000    200K rw---   [ anon ]

00007fd216ffe000      4K -----   [ anon ]

00007fd216fff000   8192K rw---   [ anon ]

00007fd2177ff000      4K -----   [ anon ]

00007fd217800000   8192K rw---   [ anon ]

00007fd218000000    132K rw---   [ anon ]

00007fd218021000  65404K -----   [ anon ] 

00007fd21c186000      4K -----   [ anon ]

00007fd21c187000   8192K rw---   [ anon ]

00007fd21c987000      4K -----   [ anon ]

00007fd21c988000   8192K rw---   [ anon ]

00007fd21d188000      4K -----   [ anon ]

00007fd21d189000   8192K rw---   [ anon ]

00007fd21d989000   1056K r-x-- libm-2.23.so

Yet another: do you see the same when the caller/publisher/subscriber
(rather the callee)?

I originally encountered the problem using publisher.  I was using publisher.cpp to setup scalibility testing with our target when I noted the large memory foot print.  I haven't tried with caller/subscriber, but I suspect they will have the same problem.

 

Cheers,
/Tobias

I appreciate you looking into this and will try to get information from valgrind for you.

 

[Chris Welch]

I've attached the output of two Valgrind runs:

valgrind --tool=memcheck --leak-check=full --log-file=vlogcallee.txt --show-leak-kinds=all ./callee -h 10.14.20.87 -r myrealm -p 8888

valgrind --tool=memcheck --leak-check=full --log-file=vlogpublisher.txt --show-leak-kinds=all ./publisher -h 10.14.20.87 -r myrealm -p 8888

These were run in x86_64 Ubunto with 1.58.0 Boost to a bonefish WAMP router running on another machine.  I didn't setup any suppressions so the callee log is quite verbose.  callee was terminated using ^C on the command line.  publisher of course runs and exits so no manual termination was required.

callee and publisher are the original code I got from the Git repository commit 2bea540078979bfc60b5fda66fca9422aa2a1f99

[Chris Welch]

I've started working with cppWamp and it doesn't have stack problems, but it isn't using boost futures which I think is the source of the stack problem.  

 

I would prefer to use Autobahn-cpp so let me know if there is additional information I can provide to help with the investigation.

[Tobias Oberstein]

Hi Chris,

right now, I am at a conference (until monday), so unfort. wont be able to look. Maybe you could file an issue and ping/CC the Bonefish guys within the issue? They are also very familiar with the AutobahnCpp codebase (contributed lots of code) ..
Cheers,
/Tobias

Sent from Mobile (Google Nexus 5)

Am 18.11.2016 6:56 nachm. schrieb "Chris Welch" <ve3...@gmail.com>:

I've started working with cppWamp and it doesn't have stack problems, but it isn't using boost futures which I think is the source of the stack problem.  

 

I would prefer to use Autobahn-cpp so let me know if there is additional information I can provide to help with the investigation.

--
You received this message because you are subscribed to the Google Groups "Autobahn" group.
To unsubscribe from this group and stop receiving emails from it, send an email to autobahnws+unsubscribe@googlegroups.com.
To post to this group, send email to autob...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/autobahnws/d538488b-9b25-40d6-892a-133f208da2de%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Chris Welch]

- right now, I am at a conference (until monday), so unfort. wont be

able to look. Maybe you could file an issue and

- ping/CC the Bonefish guys within the issue? They are also very

familiar with the AutobahnCpp codebase (contributed

- lots of code) ..

Will do Tobias, like yourself though, I probably won't be able to get to
it until Monday.

[Chris Welch]

I've filed an issue report for it and sent an email to the folks at Topology/bonefish.

Let me know if I can be of any further assistance.