Kazoo 3.16 stress testing with sipp test tool

[Kazoo Tester]

Hello All,

    We are testing kazoo environment (3 Freeswitch + 2 Kazoo + 2 Couch DB ).

   All these instances are running in the cloud with 6GB Ram, 4 cpu and 30 GB HDD each.

   We are using SIPP test tool. Right now we are simply focusing on  simultaneous registrations cycle. We are surprised to see that kamailio is failing miserably when we increase per-second registration load to say 50.

   http://www.kamailio.org/docs/openser-performance-tests/ they claim around 10000 registrations per second. We did try to setup kamailio with high package and process memory around 3GB and 200 MB respectively but still no success. We have also ensured all these instances should be running on lowest possible log level. Surprisingly average cpu load on all these instances  are around 1.0 or less than that.

  I dont know now what to tune up to support good numbers also fail  to see any conscious effort has been made in this direction specially stress testing with the suggested environment despite of being modifying major component like kamailio and freeswitch.

    

  Thanks

[Darren Schreiber]

Hi there,

You haven’t really provided enough information for us to know what’s wrong, but I know Luis heavily tested Kazoo with sipp and registrations and far exceeded what you are indicating, so I am sure there is something wrong with your tests.

However, again, there’s no information really in your report, so nothing to comment on.

--
You received this message because you are subscribed to the Google Groups "2600hz-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to 2600hz-users...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Luis Azedo]

maybe your sipp templates are wrong ? can you share them ?

--

[Mahmoud]

I'm NOT Kazoo expert but SIPp allow you to create your own scenarios by generating custom XML files from Tcpdump or Wireshark to use them as a scenario files...If you want to test Kazoo simultaneous registrations you can register one of your SIP UAC to the Kazoo server and capture the traffic and generate XML scenario file to use with SIPp then increase the number of simultaneous registrations that way you should be sure that nothing is wornge with your SIPp files.

[Kazoo Tester]

Hello All,

   Ok.. I have attached my XML file  TestRegister.xml and here is the command that I am firing.

sipp -p 5062 -trace_err -r 500 -inf /tmp/number97_register.csv -sf sippp/TestRegister.xml  -l 800 -m 800 -mi 162.213.X.X 107.155.X.X

and here is an outcome File: OtcomeSippStressTesting.png attached here.

  I have tested with different per second registration load and found different results.

 Everything goes fine for all 800 users if I set per second registration to 10. But If I set the same with 500 then result is not at all acceptable.

 Parallelly, I have started debugging Kamailio and found kazoo_query and kazoo_publish  takes around 2 second each for an individual REGISTER request with increased load as I did in above command.

 Let me know if I need to tune up any configuration parameters either in Sipp or in Kamailio.

Thanks,

[Matthew Martin - VCCS Telecom]

Do you have anything to monitor network traffic at the same time? Try to run the test from 1 server to another so as not to loop the registrations.. Other things to look at would be your ulimits on linux 

[Niraj Roy]

Hi,
Thanks for the reply.
I am in the cloud and going on with their claim 10GB is their network connectivity, still I can't rule out what you are saying. While ulimit is set to 65535. So let me test that now.

--
You received this message because you are subscribed to a topic in the Google Groups "2600hz-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/2600hz-users/f33uK3gGRBM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to 2600hz-users...@googlegroups.com.

[Matthew Martin - VCCS Telecom]

Can you post all of your "ulimit -a" command please?

On Thursday, January 29, 2015 at 8:58:28 AM UTC-5, Kazoo Tester wrote:

[Kazoo Tester]

Hi,

   This I have set it for all the 7 instances.

 ulimit -a

core file size          (blocks, -c) 0

data seg size           (kbytes, -d) unlimited

scheduling priority             (-e) 0

file size               (blocks, -f) unlimited

pending signals                 (-i) 1159197

max locked memory       (kbytes, -l) 64

max memory size         (kbytes, -m) unlimited

open files                      (-n) 65535

pipe size            (512 bytes, -p) 8

POSIX message queues     (bytes, -q) 819200

real-time priority              (-r) 0

stack size              (kbytes, -s) 10240

cpu time               (seconds, -t) unlimited

max user processes              (-u) 65535

virtual memory          (kbytes, -v) unlimited

file locks                      (-x) unlimited

Thanks,

[Kazoo Tester]

Hi Darren,

   I think I have provided sufficient information. Do you have any suggestions?

  As you mentioned Luis is heavily testing kazoo with sipp can he share his view on this?

  I am skeptical right now, we are targetting around 1million customers base and evaluating hardware and other infrastructures. 

  I think Kazoo team should come up with what they have already tested with suggested platform.. Since this platform is not obviously for small enterprise or 50 to 100 users in call system.. It is meant to be something larger (as kazoo website mentioning to serve a small country) and should save time for every user to do this kind of stress and performance testing.

 

Thanks,

Niraj     

[Luis Azedo]

HI,

thank you for the attachments.

here's my results on @!# vm

[root@media-03 sipp]# sipp -p 5062 -trace_err -r 1300 -inf users.csv -sf reg.xml  -l 5000 -m 120000 -mi x.x.x.x y.y.y.y

Warning: open file limit > FD_SETSIZE; limiting max. # of open files to FD_SETSIZE = 1024

                                                                                         Resolving remote host 'y.y.y.y'... Done.

------------------------------ Scenario Screen -------- [1-9]: Change Screen --

  Call-rate(length)   Port   Total-time  Total-calls  Remote-host

1300.0(0 ms)/1.000s   5062      92.35 s       120000  y.y.y.y:5060(UDP)

  Call limit reached (-m 120000), 0.000 s period  0 ms scheduler resolution

  0 calls (limit 5000)                   Peak was 291 calls, after 39 s

  0 Running, 42916 Paused, 0 Woken up

  0 dead call msg (discarded)            0 out-of-call msg (discarded)

  1 open sockets

                                 Messages  Retrans   Timeout   Unexpected-Msg

    REGISTER ---------->         120000    0         0

         401 <----------         120000    0         0         0

    REGISTER ---------->         120000    0         0

         200 <----------         120000    0         0         0

------------------------------ Test Terminated --------------------------------

----------------------------- Statistics Screen ------- [1-9]: Change Screen --

  Start Time             | 2015-01-30   16:46:38:975    1422636398.975229

  Last Reset Time        | 2015-01-30   16:48:11:329    1422636491.329421

  Current Time           | 2015-01-30   16:48:11:329    1422636491.329482

-------------------------+---------------------------+--------------------------

  Counter Name           | Periodic value            | Cumulative value

-------------------------+---------------------------+--------------------------

  Elapsed Time           | 00:00:00:000              | 00:01:32:354

  Call Rate              |    0.000 cps              | 1299.348 cps

-------------------------+---------------------------+--------------------------

  Incoming call created  |        0                  |        0

  OutGoing call created  |        0                  |   120000

  Total Call created     |                           |   120000

  Current Call           |        0                  |

-------------------------+---------------------------+--------------------------

  Successful call        |        0                  |   120000

  Failed call            |        0                  |        0

-------------------------+---------------------------+--------------------------

  Call Length            | 00:00:00:000              | 00:00:00:036

------------------------------ Test Terminated --------------------------------

i sent 120.000 REG->401->REG cycles along 92 secs

for me, the relevant part is the final cumulative value ( 1299.348 cps ) being very very close of the target i set ( 1300 )

do we conclude that kamailio/kazoo has ~1300 limit call for registrations ? absolutely not.

the main reason is , kazoo is hardly used here.

for those more familiar on how stuff works, if all goes well the first call will cache the creds in kamailio , 

and all subsequent credential calls (kazoo_query) will be handled by kamailio cache, making only a cast to amqp that reaches ecallmgr

what about, if we comment the cache stuff in kamailio script, to do a full test of sending kazoo_query, having registrar handle the registration and responding back in each call?

will the results be the same ? actually, the results i posted were made with commented cache stuff :).

hey, OOB values ? again, absolutely not.

how did i scale ?

tcp childs = 150

amqp_max_channels = 300

amqp_internal_loop_count = amqp_consumer_loop_count = 1

5 consumers on registrar whapp

2 consumers on ecallmgr

my conclusion is something like, 

stress tests must have some adherence to reality orelse they serve no purpose.

10.000 registrations can be reached, with proper understanding of how things work, observation of behaviours, tuning server parameters(kazoo, kamailio, rabbitmq, haproxy, bigcouch), after that is just adding instances (even on the same server like the 5 consumers i added to registrar), oh, and don't forget, proper hardware, my @"! runs linux under old version of hyper-v which sucks at network virtualization.

just a comment to your comment

"I have started debugging Kamailio and found kazoo_query and kazoo_publish  takes around 2 second each for an individual REGISTER request with increased load as I did in above command."

kazoo_query "calls", ie, sends an expects response. it uses targeted exchange binded to a private queue. kazoo module will open ( amqp_max_channels , default 100)  channels for speed, but if you don't increase tcp childs they serve no purpose.

kazoo_publish is a cast so i find hard to believe on that 2 sec.

if you still find problems tunning kazoo/rabbitmq/kamailio/haproxy/bigcouch you can always ask sa...@2600hz.com for a quote to do custom evaluation/tunning of your clusters. 

Best and please keep digging.

--

[Darren Schreiber]

This is a great write-up of your work, Luis. Thanks for this.

Just to dove-tail in real-world experience, we have multiple clients now who are exceeding 10,000 registered handsets. Most are set at 60 or 120 second interval. This is, at minimum, 83 registrations per second. Throw in the NOTIFY and SUBSCRIBE and I’m sure we’re exceeding the 50 packets per second that “Kazoo Tester” described in the original post.

I’d say something isn’t setup right in the environment, as configured.

From: Luis Azedo <luis....@factorlusitano.com>
Reply-To: "2600hz...@googlegroups.com" <2600hz...@googlegroups.com>
Date: Monday, February 2, 2015 at 7:08 AM
To: "2600hz...@googlegroups.com" <2600hz...@googlegroups.com>

[Kazoo Tester]

Thanks Luis,

 Let me follow these lines and come back!

 Let me combine my experience with this and finally will publish on confluence.

Thanks again !

[Kazoo Tester]

Hi Luis,

    Can you please guide me, when you say

  5 consumers on registrar whapp

2 consumers on ecallmgr ?

   I mean what parameters I need to tune.

Thanks,

  

On Monday, February 2, 2015 at 8:38:21 PM UTC+5:30, Luis Azedo wrote:

[Kazoo Tester]

Hello All,

   I did follow the suggestions given by Luis and set transport as TCP, somehow  that helped me to overcome timeout and unexpected messages  on sipp and able to throw 1000 Registration per second, but getting all registrations done took around 120 seconds which effectively comes down to 8~9 Registrations/second. 

  As usual Kazoo_query is taking long time around 3 seconds. Then I started digging into the code base and can see it is rendering that to amqp bus which is rabbitmq...Now here I am not able to see any trace message through rabbitmq logs (I did try to set log level to max by chanigning kazoo/config.ini and rabbitmq.config files).

  I did watch James video on architectural presentations and understood well... Now need a light to see how can I penetrate myself in  to the next level of debugging since  there are very few configuration parameters available  to play with. There are suggestion in confluence for load testing  but doesn't seem to be relevant to current versions of kazoo. 

  I am still struggling to understand Luis suggestion:

  5 consumers on registrar whapp

 2 consumers on ecallmgr

  Any help from Gurus will be appreciated!

Thanks,

 

On Thursday, January 29, 2015 at 7:28:28 PM UTC+5:30, Kazoo Tester wrote:

[Yumminova Telecom]

If u are planning a userbase of a million users, then why not go with 2600hz to setup your clusters? I wish i was in your position and could deal with them!

Op woensdag 4 februari 2015 16:04:56 UTC+1 schreef Kazoo Tester:

[Gaurav Saini]

Hello Everyone,

Can anyone please explain where to find below two settings. I am currently facing problem with RabbitMQ when we have more than 200 calls. Also, didn't find any documentation regarding the tuning of servers for 1000+ calls. 

5 consumers on registrar whapp

2 consumers on ecallmgr

Thanks

Gaurav

[Darren Schreiber]

This is a very old version, this issue is largely obsolete at this point – can you please upgrade to latest?

 

From: <2600hz...@googlegroups.com> on behalf of Gaurav Saini <gaurav...@gmail.com>
Reply-To: "2600hz...@googlegroups.com" <2600hz...@googlegroups.com>
Date: Friday, June 30, 2017 at 11:10 AM
To: 2600hz-users <2600hz...@googlegroups.com>
Subject: Re: Kazoo 3.16 stress testing with sipp test tool

 

Hello Everyone,

--

You received this message because you are subscribed to the Google Groups "2600hz-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to 2600hz-users...@googlegroups.com.

Visit this group at https://groups.google.com/group/2600hz-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/2600hz-users/e28ee53a-03b6-4562-9daf-ed31a1f1eaf5%40googlegroups.com.