Over 200M memory consumption right after start

[Сергей Алексеев]

Hello, guys!

I have fresh crossbar installation and one node initialized with --template default.

ciroja@host1:~/cbnode$ ~/pypy-venv/bin/crossbar version
Automatically choosing optimal Twisted reactor
Running on Linux and optimal reactor (epoll) was installed.
     __  __  __  __  __  __      __     __
    /  `|__)/  \/__`/__`|__) /\ |__)  |/  \
    \__,|  \\__/.__/.__/|__)/~~\|  \. |\__/

 Crossbar.io        : 0.12.1
   Autobahn         : 0.12.1 (with JSON, MessagePack, CBOR)
   Twisted          : 15.5.0-EPollReactor
   LMDB             : 0.88/lmdb-0.9.17
   Python           : 2.7.10/PyPy-4.0.1
 OS                 : Linux-3.16.0-60-generic-x86_64-with-Ubuntu-14.04-trusty
 Machine            : x86_64

Before starting crossbar I check my memory consumption with top and see:

KiB Mem:   2042852 total,   346520 used,  1696332 free,    24144 buffers
KiB Swap:  2097148 total,        0 used,  2097148 free.   167800 cached Mem

Then I start crossbar...

ciroja@host1:~/cbnode$ ~/pypy-venv/bin/crossbar start
2016-02-10T22:35:07+0300 [Controller   1857] Automatically choosing optimal Twisted reactor
2016-02-10T22:35:07+0300 [Controller   1857] Running on Linux and optimal reactor (epoll) was installed.
2016-02-10T22:35:07+0300 [Controller   1857]      __  __  __  __  __  __      __     __
2016-02-10T22:35:07+0300 [Controller   1857]     /  `|__)/  \/__`/__`|__) /\ |__)  |/  \
2016-02-10T22:35:07+0300 [Controller   1857]     \__,|  \\__/.__/.__/|__)/~~\|  \. |\__/
2016-02-10T22:35:07+0300 [Controller   1857]
2016-02-10T22:35:07+0300 [Controller   1857]     Crossbar.io Version: 0.12.1
2016-02-10T22:35:07+0300 [Controller   1857]     Node Public Key: ...
2016-02-10T22:35:07+0300 [Controller   1857]
2016-02-10T22:35:07+0300 [Controller   1857] Running from node directory '/home/ciroja/cbnode/.crossbar'
2016-02-10T22:35:07+0300 [Controller   1857] Controller process starting (PyPy-EPollReactor) ..
2016-02-10T22:35:07+0300 [Controller   1857] Node configuration loaded from 'config.json'
2016-02-10T22:35:07+0300 [Controller   1857] Node ID 'host1' set from hostname
2016-02-10T22:35:07+0300 [Controller   1857] Using default node shutdown triggers [u'shutdown_on_worker_exit']
2016-02-10T22:35:07+0300 [Controller   1857] Joined realm 'crossbar' on node management router
2016-02-10T22:35:07+0300 [Controller   1857] Starting Router with ID 'worker1'...
2016-02-10T22:35:07+0300 [Router       1862] Worker process starting (PyPy-EPollReactor) ..
2016-02-10T22:35:08+0300 [Controller   1857] Router with ID 'worker1' and PID 1862 started
2016-02-10T22:35:08+0300 [Router       1862] Realm 'realm1' started
2016-02-10T22:35:08+0300 [Controller   1857] Router 'worker1': realm 'realm1' (named 'realm1') started
2016-02-10T22:35:08+0300 [Controller   1857] Router 'worker1': role 'role1' (named 'anonymous') started on realm 'realm1'
2016-02-10T22:35:08+0300 [Router       1862] Site starting on 8080
2016-02-10T22:35:08+0300 [Controller   1857] Router 'worker1': transport 'transport1' started

And right after start - and before any message was sent to crossbar - I look at the top again and see the following:

KiB Mem:   2042852 total,   490732 used,  1552120 free,    24120 buffers
KiB Swap:  2097148 total,        0 used,  2097148 free.   167804 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 1857 ciroja    20   0  171268 105896  35028 S   0.0  5.2   0:00.69 crossbar-contro
 1862 ciroja    20   0  173536 108060  34944 S   0.0  5.3   0:00.66 crossbar-worker

Also tried version 0.11 from official ubuntu distribution before - got the same result.

So I wonder - is it normal that two crossbar processes consume more that 200M of memory right after start?
Maybe I do something wrong or miss something?

Thanks in advance!

[Gareth Bult]

23025 xxxx    20   0  886.5m  74.1m   9.2m S   0.0  0.5   0:35.68 crossbar-worker                                                                                                
23013 xxxx    20   0  154.7m  65.4m   8.8m S   0.0  0.4   0:00.87 crossbar-contro        

So it looks a little on the high site, I would guess you're loading more than I am, or maybe have some debug

enabled (?) , is the size an issue?

[Сергей Алексеев]

четверг, 11 февраля 2016 г., 0:40:08 UTC+3 пользователь Gareth Bult написал:

Well, the thing is that it's not a high loaded site - it's just a development instance running inside VMWare on my desktop. I'm the only user here. I have not sent any message to router yet - just ran "crossbar init --template default" and then "crossbar start". Don't know what else I might have accidentally loaded in these processes.

Now in my Node.js project I use some hand-made simple messaging system based on Redis Pub/Sub, but it lacks some features (such as queuing, reconnecting, etc), also it's obvious that there's no much reason in reinventing the wheel. Sure I could use some other WAMP routers implemented for Node.js, but they don't support WAMP extended profile, also it seems that they are not as well supported as Crossbar. So I consider replacing all this with well-made and well-tested Crossbar.io

In this topic ( https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!searchin/autobahnws/memory/autobahnws/_r3ViipUgEw/5EzIHsBPbaUJ ) they say about " ~10MB for the controller and additional ~10MB RAM for the router. Cpp workers < 1MB RAM". And after some tests it increased to "~25MB RAM for controller and ~25MB for the router". It would be very good result for me because I will have ordinary web server, not some embedded system with tight memory limits. But the numbers I see are about ten times bigger... And while 200M of RAM is not a really big issue as is, I'm afraid that maybe I do something wrong, and after deploying it to production this 200M will become 2G+ and put everything down.

[Alexander Gödde]

Hi Sergey!

Starting the current release on Ubuntu, I get ~65 MB each for the controller and worker - so this does indeed seem to be normal. The post you are citing is from 2014, and a lot of functionality has been added since then, so I'm not particularly surprised that memory consumption has gone up.

Regards,

Alex

[Tobias Oberstein]

This is normal. Eg PyPy without anything else already consumes
VIRT=126M, RES=80M

https://gist.github.com/oberstet/0455c7a7f7ccc782b725

When running CB, I get

https://github.com/crossbario/crossbarexamples/tree/master/benchmark/rpc_roundtrip#system-load

eg the router worker: VIRT=274M, RES=220M

cpython will have a lower memory consumption though.

Cheers,
/Tobias

Am 10.02.2016 um 20:52 schrieb Сергей Алексеев:
> Hello, guys!
>
> I have fresh crossbar installation and one node initialized with
> --template default.
>
> |
> ciroja@host1:~/cbnode$ ~/pypy-venv/bin/crossbar version
> Automaticallychoosing optimal Twistedreactor

> Runningon Linuxandoptimal reactor (epoll)was installed.

> __ __ __ __ __ __ __ __
> /`|__)/ \/__`/__`|__) /\ |__) |/ \
> \__,| \\__/.__/.__/|__)/~~\| \. |\__/
>
> Crossbar.io : 0.12.1
> Autobahn : 0.12.1 (with JSON, MessagePack, CBOR)
> Twisted : 15.5.0-EPollReactor
> LMDB : 0.88/lmdb-0.9.17
> Python : 2.7.10/PyPy-4.0.1
> OS :
> Linux-3.16.0-60-generic-x86_64-with-Ubuntu-14.04-trusty
> Machine : x86_64
> |
>

> Before starting crossbar I check my memory consumption with *top* and see:
>
> |
> KiBMem:2042852total,346520used,1696332free,24144buffers
> KiBSwap:2097148total,0used,2097148free.167800cached Mem

> |
>
>
> Then I start crossbar...
> |
> ciroja@host1:~/cbnode$ ~/pypy-venv/bin/crossbar start
> 2016-02-10T22:35:07+0300[Controller1857]Automaticallychoosing optimal
> Twistedreactor
> 2016-02-10T22:35:07+0300[Controller1857]Runningon Linuxandoptimal
> reactor (epoll)was installed.

> 2016-02-10T22:35:07+0300[Controller1857] __ __ __ __ __ __
> __ __
> 2016-02-10T22:35:07+0300[Controller1857]/`|__)/ \/__`/__`|__) /\ |__)

> look at the *top* again and see the following:
> |
> KiBMem:2042852total,490732used,1552120free,24120buffers
> KiBSwap:2097148total,0used,2097148free.167804cached Mem

>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+COMMAND
> 1857ciroja 20017126810589635028S 0.05.20:00.69crossbar-contro
> 1862ciroja 20017353610806034944S 0.05.30:00.66crossbar-worker
> |
>
> Also tried version 0.11 from official ubuntu distribution before - got
> the same result.
>
> So I wonder - is it normal that two crossbar processes consume more that
> 200M of memory right after start?
> Maybe I do something wrong or miss something?
>
> Thanks in advance!
>

> --
> You received this message because you are subscribed to the Google
> Groups "Crossbar" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to crossbario+...@googlegroups.com
> <mailto:crossbario+...@googlegroups.com>.
> To post to this group, send email to cross...@googlegroups.com
> <mailto:cross...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/crossbario/cfb47a70-46a9-4765-accc-e33928c920ed%40googlegroups.com
> <https://groups.google.com/d/msgid/crossbario/cfb47a70-46a9-4765-accc-e33928c920ed%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[Tobias Oberstein]

> In this topic (
> https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!searchin/autobahnws/memory/autobahnws/_r3ViipUgEw/5EzIHsBPbaUJ
> ) they say about " ~10MB for the controller and additional ~10MB RAM for
> the router. Cpp workers < 1MB RAM". And after some tests it increased to
> "~25MB RAM for controller and ~25MB for the router". It would be very
> good result for me because I will have ordinary web server, not some
> embedded system with tight memory limits. But the numbers I see are
> about ten times bigger... And while 200M of RAM is not a really big
> issue as is, I'm afraid that maybe I do something wrong, and after
> deploying it to production this 200M will become 2G+ and put everything
> down.

There are 2 main reasons for the _apparent_ higher _base_ memory
consumption compared to above post:

- PyPy vs CPython
- a lot more stuff is now mandatory with Crossbar.io: eg TLS and other
crypto things.

A lot of above is _shared_ between multiple Crossbar.io worker processes.

Eg. look at

https://github.com/crossbario/crossbarexamples/tree/master/benchmark/rpc_roundtrip#system-load

You can't "add up" the 200M per worker to get the total memory load.

In fact, correctly measuring _real_ memory consumption is hard:

http://stackoverflow.com/questions/131303/how-to-measure-actual-memory-usage-of-an-application-or-process

I'd recommend either using valgrind, or simply real testing: run CB,
connect many clients, do massive messaging, and see at which point your
system collapses.

Cheers,
/Tobias

>
> --
> You received this message because you are subscribed to the Google
> Groups "Crossbar" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to crossbario+...@googlegroups.com
> <mailto:crossbario+...@googlegroups.com>.
> To post to this group, send email to cross...@googlegroups.com
> <mailto:cross...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/crossbario/64432364-f5fc-4911-9c22-459393b6a8f8%40googlegroups.com
> <https://groups.google.com/d/msgid/crossbario/64432364-f5fc-4911-9c22-459393b6a8f8%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Сергей Алексеев]

Alexander, Tobias - thank you so much for:
- your quick response
- all the great job you've made on Crossbar/Autobahn etc
- my increased confidence in that everything is ok, nothing extraordinary and I'm not doing something wrong :)