Load testing UniMRCP server

[Vipul Sharma]

Hi,

I want to load test UniMRCP server with the ASR plugin which I have built.

Right now I have implemented a new command in the client which looks like 

load recog 10. This command behaves like 10 simultaneous run recog

scenarios.

I don't know what metrics I should monitor right now. I just wanted to know if

anyone has tried load testing UniMRCP server? Would like to know how others

have approached with the load testing.

Thanks.

Vipul

[Dmitriy Borisov]

Hi!

CPU load? Memory use? Maybe i/o operations (if your plugin uses disk buffer, as for example)? Context switching per second rate? You can look at how CPU load distributes between CPU cores.

The main metric is "concurrent recognitions count with minimal error count", as I think.

пт, 27 дек. 2019 г. в 09:43, Vipul Sharma <vipul.s...@gmail.com>:

--
You received this message because you are subscribed to the Google Groups "UniMRCP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to unimrcp+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/unimrcp/563a7417-36ea-42ac-a31d-0b1f92132993%40googlegroups.com.

--

With best regards
   Dmitry Borisov

[Michael Levy]

>> The main metric is "concurrent recognitions count with minimal error count", as I think.

I would add:

The main metric is "concurrent recognitions count with minimal error count and acceptable latency", as I think.

[San'cheZ]

What specific parameters do you want to test? To create a large load, you can use Asterisk

пятница, 27 декабря 2019 г., 9:43:53 UTC+3 пользователь Vipul Sharma написал:

[Vipul]

Hi,

Thanks for your inputs everyone.

@Michael: Thanks. Yes, that is what I want to evaluate right now. Do you have any inputs on how can I create load using UniMRCP client?

@San'cheZ: Thanks. As Michael mentioned, The main metric is "concurrent recognitions count with minimal error count and acceptable latency"

Right now I can't think of a good way to simulate this scenario. Also, using Asterisk to create large load will need me to install

UniMRCP Asterisk module (right?). I was thinking if there is a programmatic way to do this at UniMRCP client/server level, that would be great.

Thanks.

Vipul

--

You received this message because you are subscribed to the Google Groups "UniMRCP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to unimrcp+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/unimrcp/e4a65c5c-ce9e-4211-9eda-3279a4e581d2%40googlegroups.com.

[San'cheZ]

Which speech recognizer do you use? (Google, Amazon, Yandex, etc.)

пятница, 27 декабря 2019 г., 9:43:53 UTC+3 пользователь Vipul Sharma написал:

Hi,

[Vipul]

Hi San'cheZ,

I've built my own plugin by taking demo recognition plugin as a reference.

It streams over gRPC to out speech server which does the transcriptions.

Thanks.

Vipul

--

You received this message because you are subscribed to the Google Groups "UniMRCP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to unimrcp+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/unimrcp/e96479d7-c60b-4008-af66-87fc4afee4b5%40googlegroups.com.

[David Villasmil]

You can use freeSWITCH and it’s originate command to do the load test.

You will need 2 servers:

1- where you will do the ASR, Let’s call it RECOGNIZER.

2- the originator of the call. let’s call it ORIGINATOR.

On 1, just configure your ASR.

On 2:

- Create a wav file with the audio you want recognized, or multiple audios with multiple content.

- install mod_unimrcp module in freeswitch (trivial)

- use the originate command like so:

bgapi originate sofia/profile/external/12345@YOUR-RECOGNIZER-IP &playback(YOUR-AUDIO-FILE)

That should do it.

Hope that helps!

David

To view this discussion on the web visit https://groups.google.com/d/msgid/unimrcp/CAMPtzKge0zSw3-ndwzg6iOzBst%2BB%3Dij%2BYbS_93QBeZ8gcTrX6w%40mail.gmail.com.

--

Regards,

David Villasmil

email: david.vill...@gmail.com

phone: +34669448337

[Vipul]

Hi David,

Thanks for the help.

I have an Asterisk instance with the UniMRCP module installed where I can do something similar.

I have verified the connectivity with the ASR, configured an extension and it works very well with

the MRCP server using the MRCPRecog dialplan application.

Do you know a way I can simulate this load in Asterisk using a pre recorded audio file?

I am familiar with Asterisk but unfortunately, I have never tried FreeSWITCH.

Thanks.

Vipul

To view this discussion on the web visit https://groups.google.com/d/msgid/unimrcp/CAFGRPVrcH-KLjVnE_OBycsYqpJ%3DiJj0uR%3DVPtahcWw%3D5ry1DcQ%40mail.gmail.com.

[David Villasmil]

Hello,

That asterisk you have already configured is the receiver. Now you need something to send call to your asterisk to the extension configured with unimrcp.

There are several ways of doing that:

- You can set up another asterisk box.

  You can probably do what i explained earlier with https://wiki.asterisk.org/wiki/display/AST/Asterisk+11+ManagerAction_Originate

- You can set up a freeSWITCH box. 

   Which is what i explained earlier. IMHO FS can take much more traffic than Asterisk.

- You can use sipp (http://sipp.sourceforge.net/), a very high traffic stress-test suite, with which you can also send bulk calls with an audio. Only downside for me is you have to get the audio file in a pcap file... kind of a pain. But it's really good.

- You can also use another test suite called https://github.com/jchavanton/voip_patrol I've never used this one, but i've heard good things.

In my opinion the simplest and most efficient way is with freeSWITCH, since it's really easy to setup. You just install it and basically without any config you can start sending calls. But i'm probably biased :)

I can help you out online if you want.

Hope this helps.

David

David Villasmil

email: david.vill...@gmail.com

phone: +34669448337

To view this discussion on the web visit https://groups.google.com/d/msgid/unimrcp/CAMPtzKghbGpua9pmHrP68ifHnoaqrYCm5ZWfVWcMw2wqctXFsg%40mail.gmail.com.

[San'cheZ]

I can help you. You need a script that generates call files. And the context that always responds and plays the audio file. Here it is difficult to describe every Shak. So add me to Skype sz493sz

[Vipul]

Hello,

Thanks David and San'cheZ for your help. I got some good insights :)

I tried using Local Channel of Asterisk to connect to my dialplan which
supports MRCP.  It works well and I can originate multiple calls using this in
a single Asterisk box. Maybe I should try using a different Asterisk box since
I think this setup will also take up some resources as both initiator and the
receiver is the same Asterisk. Need your suggestions here.

San'cheZ, I did some reading up and I understand the call files thing a little
now. I tried simulating calls using a Local Channel through call files which
seems to connect to the desired dialplan context and works as expected

Below is the sample dialplan which I am using:

[uni2]
exten => uni2,1,Answer()
same => n,MRCPRecog("builtin:speech/transcribe", spl=en-US&f=beep&p=uni2)
 
[caller]
exten => caller,1,Answer()
same => n,Playback(sample-8kHz)

I can originate using: asterisk -x "originate Local/caller@caller extension uni2@uni2"

Here, sample-8kHz is a PCM file which I also use in unimrcpclient to test.  I
am trying to play this same file when the call connects to the context uni2.
However, I am getting some distorted audio in the ASR service in our backend

systems which I am trying to debug. Just wanted to know if this dialplan makes

sense and is a decent implementation or not?

If this discussion seems off topic in this forum to anyone then we can take it

somewhere else :)

Thanks.

Vipul 

On Thu, Jan 23, 2020 at 9:49 PM San'cheZ <djsan...@ya.ru> wrote:

I can help you. You need a script that generates call files. And the context that always responds and plays the audio file. Here it is difficult to describe every Shak. So add me to Skype sz493sz

--
You received this message because you are subscribed to the Google Groups "UniMRCP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to unimrcp+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/unimrcp/224bf156-f82f-48c8-a70d-3d1d2a65cebb%40googlegroups.com.

[San'cheZ]

I think this dialplan makes sense. Because you can create the load necessary for testing. What kind of distortion do you get?

What version of Asterisk are you using?

пятница, 24 января 2020 г., 18:35:10 UTC+3 пользователь Vipul написал:

To unsubscribe from this group and stop receiving emails from it, send an email to uni...@googlegroups.com.

[Vipul]

Hi,

I am using Asterisk 14.7. We collect all the raw bytes from MRCP server and then convert it to FLAC in backend and it works fine with other clients but, with Asterisk the audio file generated sounds like static noise. Is it because I am using a PCM file in Asterisk?

Thanks.

Vipul

To unsubscribe from this group and stop receiving emails from it, send an email to unimrcp+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/unimrcp/e90cdb29-ffea-40e5-bd54-ac002ef232ba%40googlegroups.com.

[Arsen Chaloyan]

Hi Vipul,

I would first make a network capture and listen to what is actually sent/received on the wire.

To view this discussion on the web visit https://groups.google.com/d/msgid/unimrcp/CAMPtzKicmKhdV3kjXSGbEq4iJACo2R39G8Tc8fB4oQUQjAuqew%40mail.gmail.com.

--

Arsen Chaloyan
Author of UniMRCP
http://www.unimrcp.org

[Vipul]

Hi,

Thanks for your help everyone and pointing me to the right direction :)

I am now able to stream audios properly and simulate load using Asterisk through call files.

I resolved the audio issue by using a gsm file and it is working as desired.

Thanks again for the help. This is what I wanted to achieve.

Regards,

Vipul

To view this discussion on the web visit https://groups.google.com/d/msgid/unimrcp/CAC5Usc%2B3-9pMj%3DmBbBP-AJC-TwFyoDn6BHUL%3DJ6Y2RyVOrxtJQ%40mail.gmail.com.

[Michael Levy]

We took libasrclient and used SWIG - http://www.swig.org/ - to create script callable wrappers to the functions used by ASR Client. Then we were able to build out load tests.In our case, the tests are built in C#/.net and we run them like unit tests using NUnit - https://nunit.org/ - (you could do the same with SWIG, Java and JUnit). NUnit has support for parallel threads and the ability to keep track of success and fails, so I can kick of a batch of Nunit load tests, wait, and get the results.

Our use of libasrclient (rather than libunimrcpclient) grew out of us using ASRClient at the command line for most of our preliminary testing. Once we started to think about batch files running ASRClient tests, SWIG and libasrclient was a natural next step.

Our original plan was to use JMeter - http://jmeter.apache.org/. We had an simple implementation of JMeter load tests for MRCPv1, but never got very far with MRCPv2 load tests with JMeter. The engineers were too busy with other things and our simple SWIG/Nunit solution was good enough.