Restcomm performance testing

[Gennadiy Dubina]

Hi everybody

I'm trying to create restcomm cluster. and i need to use some performance test tool to check health of this cluster.
I have found a few scripts in official git repo - https://github.com/RestComm/RestComm-Core/tree/master/load_tests

Also i have some statistics - http://www.telestax.com/restcomm-scalability-and-high-availability/

now I am trying to reproduce these results.

so I have added PROD_MODE env variable to my fork of restcomm docker container. It reconfigures restcomm to use 8G RAM, setups WARN log level and turns off akka logs. https://github.com/hamsterksu/Restcomm-Docker/blob/7.5.0/scripts/restcomm_conf.sh#L350​

This docker images is available in docker hub - hamsterksu/restcomm:7.5.0.800 or hamsterksu/restcomm:7.6.0.824

Also I have created docker container with sipp test tool and my own ivr app server
Test container- https://github.com/dataart-telco/ivrtest-docker
Ivr app - https://github.com/dataart-telco/ivrtest
It’s pretty simple ivr app with 2 responses – makes gather and plays confirmation.
I have tested it with apache ab test. It can handle around 34k request per second(so it’s not a bottleneck)

Response for incoming call:
<Response>
<Gather action="http://192.168.176.186:7090/gather" method="POST" numDigits="1">
<Play>http://192.168.176.186:7080/pcap/demo-prompt.wav</Play>
</Gather>
<Hangup/>
</Response>

demo-prompt.wav is shared by nginx server. It’s located in docker container too
both docker containers(restcomm and ivrtest) are run in host mode(w/o docker proxy) on the same machine w/o any limitation (core i5 with 32G ram)

In the root directory of ivrtest-docker you can find 2 scripts to run test:
1. run_restcomm.sh – start restcomm
2. run.sh – just run ivrtest

My app calculates statistics of incoming and gathered calls. It will be printed in the end of test

But my result is not good.

It's dump from log file with java opt for jboss.

JAVA_OPTS: -server -XX:+UseCompressedOops -Xms2048m -Xmx8192m -Xmn512m -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -XX:+CMSIncrementalPacing -XX:CMSIncrementalDutyCycle=100 -XX:CMSIncrementalDutyCycleMin=100 -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:MaxPermSize=512m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.server.default.config=standalone-sip.xml

I made 2 test with different settings:

Test #1
I try to make 5000 calls with 80 cps. Rtp port range is 64000-65500
The result is – ~24.1 cps and my app receives only ~1400-2600 gathering answers(as i understand it means that restcomm doesn't handle incoming traffic correctly).

Test #2
Makes 5000 calls with 40 cps. Rtp port range is 64000-65500​
The result is – 23.959 cps and my app receives ~3000-3400 gathering answers.

----------------
I run both test for 2 versions of Restcomm: 7.5.0.800 and 7.6.0.824
1.1 version 7.5.0.800
i got a lot of errors in the log file

it's part of my log file: https://www.dropbox.com/s/x4zxidincowhk3g/restcomm_perf.log

also i have found a lot of messages in log file like:
12:16:56,481 ERROR [org.mobicents.protocols.mgcp.stack.JainMgcpStackProviderImpl] (Thread-596) The TransactionHandler not found for TransactionHandle 3786 May be the Tx timed out. Event = 400 0The transaction could not be executed due to a transient error

1.2 version 7.6.0.824
i juts have found errors similar to
13:03:39,090 ERROR [org.mobicents.protocols.mgcp.utils.PacketRepresentationFactory] (Thread-597) PRFactory underflow. Count = 31
13:03:39,090 ERROR [org.mobicents.protocols.mgcp.utils.PacketRepresentationFactory] (Thread-597) PRFactory underflow. Count = 32
13:03:39,090 ERROR [org.mobicents.protocols.mgcp.utils.PacketRepresentationFactory] (Thread-597) PRFactory underflow. Count = 33
13:03:39,090 ERROR [org.mobicents.protocols.mgcp.utils.PacketRepresentationFactory] (Thread-597) PRFactory underflow. Count = 34

but seems it doesn't print other errors types like timeout
------------​

Could somebody verify my test and settings. maybe something is wrong. maybe i use wrong strategy. any your feedback will help me.

script - https://github.com/dataart-telco/ivrtest-docker/blob/master/files/test-ivr.sh
sipp scenario - https://github.com/dataart-telco/ivrtest-docker/blob/master/files/tests/ivr/ivr-sipp.xml

Thanks,

[George Vagenas]

Hi Gennadly,

Later this week I am planning to work on some scripts that prepare Restcomm and run performance tests, I will check your work also and ping you here so we can coordinate and work together.

Regards
George

[Gennadiy Dubina]

Hi George, 

thanks for your reply.

i have made some changes in my ivrtest docker container - https://github.com/dataart-telco/ivrtest-docker

1. you can use env variable to change test arguments
1.  $TEST_CALLS_RATE - calls rate 
2. $TEST_CALLS_CONC - concurrent calls 
3. $TEST_CALLS_MAX - count of calls 
2. i have built new version of Restcomm with PROD_MODE param - hamsterksu/restcomm:7.6.0.831  i made pull request with these changes to restcomm docker repo too - https://github.com/RestComm/Restcomm-Docker/pull/43 
3. i have added new script to run series of tests - run_series.sh  It has array with test parameters
4. you can provide ip address of restocmm instance as command line parameters for run.sh and run_series.sh otherwise scripts will use ip of eth0 interface 
5. ivr test dumps results to csv file - ./result/result.csv  result folder should is attached to docker container. 

How to use ivrtest container:

to run ivr test you have to clone repo and run build.sh script. it creates docker container and runs it. after build you can restart test with run.sh or run_series.sh

sure you should start restcomm before test. 

test app registers callback number in restcomm. restcomm instance should have user with the following credentials:

login - admini...@company.com 

password - q1w2e3r4t5

in case if you use my restcomm docker image then password will by specified through env variable(provides md5hash of password). you can use run_restcomm.sh to start restcomm

if you would like to use your restcomm instance you have to create user with required credentials 

if you would like to build own restcomm container - you can reuse my code to add some useful features like PROD_MODE and INIT_PASSWORD flags:

https://github.com/RestComm/Restcomm-Docker/pull/43 

https://github.com/RestComm/Restcomm-Docker/pull/39 - be care in this case you have provide password in plain text format. script will convert it to md5has automatically 

Thanks,

вторник, 8 марта 2016 г., 16:09:46 UTC+2 пользователь George Vagenas написал:

[Gennadiy Dubina]

Hi George,

i have made a series of tests with my test environment - 2 different amazon m3.xlarge machines ​

you can find result here - 

https://docs.google.com/spreadsheets/d/1V03zlMWSq0qatXDRzbtPHPxZCTwSu1xh9CPXKLFfGiY/edit?usp=sharing​

вторник, 8 марта 2016 г., 16:09:46 UTC+2 пользователь George Vagenas написал:

Hi Gennadly,

[Henrique Rosa]

Hi Gennaddiy, 

The traces you see are not problematic, even though they're (wrongfully IMO) marked as ERROR.

12:13:18,366 ERROR [org.mobicents.protocols.mgcp.utils.PacketRepresentationFactory] (RestComm-akka.actor.default-dispatcher-14) PRFactory underflow. Count = 19

Can you share both RestComm AND Media Server logs to check what errors you may have that could hopefully explain the failed calls?

Sometimes it can also be necessary to extend the MGCP timeout interval in RestComm, especially when working with higher call rates (call per second).

Sometimes it can also be bad timing in your SIPp scenario that can generate false timeouts.

Can you please share a version of your RestComm application so I can test on my side as well?

It seems like a simple Gather application, but I need to know what your action url is doing behind scenes.

<Response> 
   <Gather action="http://192.168.176.186:7090/gather" method="POST" numDigits="1"> 
      <Play>http://192.168.176.186:7080/pcap/demo-prompt.wav</Play> 
   </Gather> 

</Response>

My plan is to start testing it locally, register results and then move to a docker environment to check for differences in performance.

Best Regards, 

- H

[Gennadiy Dubina]

you can reproduce my env very easy.

everything is packaged to docker container - https://github.com/dataart-telco/ivrtest-docker 

there are 2 branches "master" and "30sec"

master uses short calls - 8 sec.

"30sec" uses long calls - 30 sec

in repo you can find run_restcomm.sh it will start restocmm in docker. restcomm version is - 7.6.0.831

you should use STATIC_ADDRESS to specify restocmmm static address

also there are run.sh and run_series.sh they runs ivrtest(irv server + sipp test) docker container.

please specify ip address of your restcomm instance as command line argument.

also you can change test arguments in these scripts.

please use the following variables: TEST_CALLS_RATE, TEST_CALLS_CONC, TEST_CALLS_MAX

среда, 16 марта 2016 г., 15:06:51 UTC+2 пользователь Henrique Rosa написал:

[Henrique Rosa]

Hi Gennadiy, 

I performed load tests on a similar RestComm gather app. With 276 concurrent calls and a call rate of 50 calls per second, I only got 1 failed call (timeout) out of 100k.

This is much better results than the ones you reported. Now I'm moving my tests to RestComm Docker to check if there is any performance hit.

Would you please repeat the same tests using a binary (no docker involved) to check if you obtain better results? We need to validate if using Docker involves a performance hit and, in case that's true, measure what is the impact and how we can fix it.

Please, share any logs from RestComm and MS in case you have errors.

Best Regards, 

- H

[Gennadiy Dubina]

Hi Henrique, 

Very nice results. 

Have you got gather confirmation from all calls?

Could you share information about your env? if you configure restcomm inside docker you can make snapshot and share it. 

maybe my instance is not tuned enough well

четверг, 17 марта 2016 г., 13:38:11 UTC+2 пользователь Henrique Rosa написал:

[Henrique Rosa]

Hi Gennadiy, 

My machine specs are as follows:

MacBook Pro (mid-2014)

Processor: 2,5 GHz Intel Core i7

Memory: 16 GB 1600 MHz DDR3

I was running that test with the following JAVA_OPTS:

RestComm: JAVA_OPTS="-Xms6g -Xmx6g -Xmn512m -XX:+CMSIncrementalPacing -XX:CMSIncrementalDutyCycle=100 -XX:CMSIncrementalDutyCycleMin=100 -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:MaxPermSize=512m"

Media Server: JAVA_OPTS="-Xms3g -Xmx3g -Xmn512m -XX:+CMSIncrementalPacing -XX:CMSIncrementalDutyCycle=100 -XX:CMSIncrementalDutyCycleMin=100 -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:MaxPermSize=512m"

In the meantime, I installed Docker in my machine and started to run some tests.

The JAVA_OPTS I used in my docker container were:

RestComm: JAVA_OPTS="-Xms2g -Xmx2g -Xmn512m -XX:+CMSIncrementalPacing -XX:CMSIncrementalDutyCycle=100 -XX:CMSIncrementalDutyCycleMin=100 -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:MaxPermSize=512m"

Media Server: JAVA_OPTS="$JAVA_OPTS -Xms1024m -Xmx2048m -Xmn512m -XX:+CMSIncrementalPacing -XX:CMSIncrementalDutyCycle=100 -XX:CMSIncrementalDutyCycleMin=100 -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:MaxPermSize=512m"

I performed a test in Docker with 20 concurrent calls and 5 calls per second. Was able to run 20K calls successfully (attachement: sipp-docker-20k-calls.log)

This seems more consistent behaviour than you've got in your tests.

Have you got gather confirmation from all calls?

I enabled log traces for my RVD application and can confirm the DTMS digits were properly recognized. I had to send DTMF via SIP INFO (over SIPp) because the Media Server was not configured to recognize inbound DTMF tones (which is good because you can get false positives with inbound DTMF).

Got some timeouts but I believe it was because of having logs enabled on RVD application. 

You can cross check by running the following command and verify the output matches the number of successful calls (attachements: sipp-docker-rvd-log.log, rvd-log.txt)

cat rvd-log.txt | grep "Digit: 5" | wc -l 

I am now going to setup my docker image with production configuration and increase the load for the following tests.

I am sharing the application I'm using so you can try to run it on your environment, you can easily import it to your RVD environment via drag-and-drop. 

SIPp scenario is attached as well. (attachement: sipp-and-app.zip)

One last note before I go. I opened issue #45 which is surely a bottleneck for docker tests.

As you probably know, logging is #1 bottleneck when you're load testing. 

Best Regards, 

- H

[Gennadiy Dubina]

Hi

i use PROD_MODE flag for restcomm docker container to switch logger to WARN level and turn off logs for akka - https://github.com/RestComm/Restcomm-Docker/pull/43/commits/15f629881e241cd00ad940787c978be50bf817b1

also i use external app to provide rcml. i don't use rvd application because i haven't found easy way to import app automatically(by api or file system)

anyway i will review your app

Thanks,

пятница, 18 марта 2016 г., 19:05:19 UTC+2 пользователь Henrique Rosa написал:

[Henrique Rosa]

RVD application are quite valuable because they are portable.

To import an RVD application to your workspace you just need to access the RVD UI, select Voice Projects and drag and drop the application zip to the following area:

[Gennadiy Dubina]

yes i know about "import" button. but how to do it automatically? we need it to deploy instance automatically(from some script)


суббота, 19 марта 2016 г., 14:00:17 UTC+2 пользователь Henrique Rosa написал:

[Orestis Tsakiridis]

Hi Dubina,

In principle you can use the same method that is used by the RVD UI. However, there is still an authentication issue that will prevent the use of basic HTTP credentials and require RVD tickets (issued to the ui).

https://github.com/RestComm/Restcomm-Connect/issues/947

Stay tuned for patch and related documentation!

Regards

Orestis

[Oscar Carriles]

Hi guys,

Share with you my RestComm stress tests, first approach, next coming soon
Regards

[Oscar Carriles]

El martes, 1 de marzo de 2016, 14:26:34 (UTC-3), Gennadiy Dubina escribió:

Hi guys,
share with you my RestComm stress tests, first approach, more coming soon
Regards

[Henrique Rosa]

Hello everyone, 

I wrote a quick wiki based on my load tests with RestComm and RestComm-Docker on EC2 environment.

You can read all about it here.

Please feel free to share your comments and own results. Your comments to help expand the wiki are welcome as well.

Cheers,

- H