Kazoo Performance Metrics

[emnvn]

Hi all,

I have made a performance test for Kazoo4 (all in one).

- Server hardware: Amazon instance t2.medium
  + CPU: 2vCPU
  + RAM: 4GB
- I tested by create 500 ClickToCall requests to Kazoo server in 250 seconds. Each request will play 2 audio files at two enpoints, Duration time for each audio is 1 minute.

The result of this test, I saw at peak time, there are about 380 calls (a ClickToCall request create two calls)  in system, all calls are hangup with "NORMAL_CLEARING" status. CPU used is about 60 %, RAM is free more than 1 GB.

Then I increase to 550 request ins 250 seconds, the result I see that there are about 100 calls with "NO_ROUTE_RESPONSE" status, CPU is used about 60%.

Is it the limit for One Server ? Any one have a better performance ? any params that I have to tuning to improve more performance ?

Thanks.

[Darren Schreiber]

If you’re trying to do load testing you need to tweak:

* Token buckets

* Threads for API handlers

* Threads for AMQP queues for channels

--
You received this message because you are subscribed to the Google Groups "2600hz-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to 2600hz-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[emnvn]

I will try to tweak them.

Thanks very much.

[Vinay Saini]

Hello All,

We have 7 server cluster setup and everything is working fine with the normal load of calls. For the load testing we are using SIPP along with the scripts provided in the kazoo community scripts.

While doing load testing we noticed upto 350 concurrent calls(Approx.) everything seems to be normal but when we increase the CPS and concurrent calls to more then 350 or 400 hundred calls then rabbitmq starts giving error. In this situation we noticed that all the kazoo APIs were stuck in the pending state and also all the new calls were giving 403 error.

Here are the logs which I have found in the rabbitmq

=ERROR REPORT==== 27-Jun-2017::06:04:35 ===

Channel error on connection <0.8251.0> (209.212.63.82:56985 -> 209.212.63.81:5672, vhost: '/', user: 'guest'), channel 177:

{amqp_error,precondition_failed,

            "queue 'ecal...@kazoo2.xxxxxx.com-<0.1759.0>-11454260' in vhost '/' in use",

            'queue.delete'}

And following are the exception getting in the kazoo logs:

 processing dialplan fetch request 92a98664-5b17-11e7-a8b8-096409cd82fb (call OGM4ZWE0MmM3NTgzYmFlMGU3ZjIwMjMzZWQxZTcyYjE.) from frees...@kazoo.xxxxxx.com

kazoo.log:Jun 27 05:04:08 k2 2600hz[2268]: |OGM4ZWE0MmM3NTgzYmFlMGU3ZjIwMjMzZWQxZTcyYjE.|wh_amqp_worker:231 (<0.268.30>) poolboy exception: exit: {timeout,{gen_server,call,[wh_amqp_pool,{checkout,#Ref<0.0.9.101525>,false},2000]}}

kazoo.log:Jun 27 05:04:08 k2 2600hz[2268]: |OGM4ZWE0MmM3NTgzYmFlMGU3ZjIwMjMzZWQxZTcyYjE.|ecallmgr_fs_route:277 (<0.268.30>) did not receive route response for request 92a98664-5b17-11e7-a8b8-096409cd82fb: poolboy_fault

kazoo.log:Jun 27 05:04:16 k2 2600hz[2182]: |OGM4ZWE0MmM3NTgzYmFlMGU3ZjIwMjMzZWQxZTcyYjE.|omnip_subscriptions:187 (<0.25443.49>) received CHANNEL_CREATE but call is terminated already, dropping

kazoo.log:Jun 27 05:04:22 k2 2600hz[2182]: |OGM4ZWE0MmM3NTgzYmFlMGU3ZjIwMjMzZWQxZTcyYjE.|omnip_subscriptions:187 (<0.148.50>) received CHANNEL_CREATE but call is terminated already, dropping

 

Do I also need to tweak them as per the Darren's suggestion. Also let me know if there is any document or some link for this.

Thanks

Vinay

[Darren Schreiber]

You need to tune your system if you’re trying to get >300cps, you will exhaust the available queues by default among other things.

[Vinay Saini]

Thanks Darren for your resposne.

Is there any steps or documentation for it? What are all the things I need to consider for the tuning?

Thanks

Vinay

[Gaurav]

Hello Darren,

Can you please share what setting need to be done with RabbitMQ tuning as everything as far as resources of FreeSWITCH, Kazoo servers are concerned as perfectly fine. But after 200 concurrent calls facing similar errors. 

It will be really helpful Darren if you can help with these performance settings.

Thanks

Gaurav

[Darren Schreiber]

I do not know the settings offhand and frankly they tend to depend on type of traffic (short duration vs long vs media/no-media etc) so there’s multiple factors. If you really need this type of test (which is a very high CPS rate which means you must have a ton of traffic) you should reach out for some consulting support. Email me offline if you wish to engage in that route.

[Gaurav]

Hello Darren,

Thanks for your instant reply. I agree that there are multiple conditions and also depends upon the type of traffic. Currently, we have very simple setup without recording just basic 7 Server Kazoo setup and don't even have real traffic for that amount of CPS. We are just testing the performance for future so if that amount of calls will increase we know in advance. 

I just need some insights about the RabbitMq issue/errors as if some insights can be shared will be really helpful. I would be really thankful of you, If some direction you can give about this So I can dig more into the code and settings for debuging this.

Regards,

Gaurav