Looking for open source Webrtc load testing tool

[Antonis Tsakiridis]

Hello,

I'm looking for an open source tool for webrtc load testing mainly to test a webrtc media server where the tool will be the client creating multiple webrtc media connections towards the server. Although it seems like a very handy tool I haven't managed to find any out there.

Best regards,

Antonis Tsakiridis

[Alexandre GOUAILLARD]

AFAIK, there is no open source tool.

- "Puppet Master", which won an award at webrtc world and expo in 2013, was not made available.

- Jitsi was thinking about creating  a "hammer". Emil, do you have any update on that?

commercial

- some company advertised a "multiplier" tool recently (mixed webrtc sip and others)

- testRTC.com is your bet today

Note that your main "problem" will be the signaling. Since the signaling your app use is not know in advance, you will have to drive the browser or your native app to communicate with your server.

If you do not need modified versions of browsers, no extension (no screensharing), no specific profile on FF, etc, you could use browserstack, saucelabs, and all other could-based browser testing tool. Selenium 2.0 (ex selenium grid) + web drivers is, IMHO your best bet.

Alex.

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/CAM7er2OreUtxQN76WsGWFtOo-GraZ4Mfk0H_Gjmz82Lr1BuzAA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Alex. Gouaillard, PhD, PhD, MBA

------------------------------------------------------------------------------------

Principal Architect - Citrix, San Francisco

President - CoSMo Software Consulting, Singapore

------------------------------------------------------------------------------------

sg.linkedin.com/agouaillard

• 
  

[Antonis Tsakiridis]

Ok thanks. We 've been working on building such an open source tool that will aid in load testing webrtc clients/servers and be able to work in headless manner so that it is easily deployable to platform clouds such as EC2. And even though we have started getting some good results there still are a lot of challenges so far:

• Programmatically opening multiple browsers concurrently to make/receive webrtc calls by driving selenium from a python script didn't scale very well. Opening more than 20 browsers started causing issues.
• When taking selenium out of the picture and spawned the browsers directly from the python script we managed to get multiple browsers open and concurrently handling webrtc calls. The problem there is that after some point the browsers start being very slow and at some point this starts to mess with the scenarios. 
• To solve that we implemented a different strategy where each browser tab would only process 1 call and then be recycled (that is the browser tab tell the driving python script that it needs to spawn a new tab in it's stead and close). That got us better results and tabs stayed responsive to calls. But then we had another performance issue. After a point the browsers would start to recycle more and more slowly and that again started to mess with our scenarios

Do you guys have any experience with such issues? Are 50 browser tabs concurrently handling webrtc connections considered too much load? Are there any tutorials on webrtc testing out there that would help me understand the issues involved and how to tackle them?

Best regards,

Antonis

[Emil Ivov]

We do have an operational hammer that can send any number of streams:

https://github.com/jitsi/jitsi-hammer

However, it is highly specific to Jitsi Videobridge and I don't see it easily reused for other architectures.

Emil

--
sent from my mobile

[Antonis Tsakiridis]

Very interesting Emil, but does it use Webrtc for media or regular RTP?

On Friday, March 25, 2016 at 12:39:07 PM UTC+2, Emil Ivov wrote:

We do have an operational hammer that can send any number of streams:

https://github.com/jitsi/jitsi-hammer

However, it is highly specific to Jitsi Videobridge and I don't see it easily reused for other architectures.

Emil

On Thursday, 24 March 2016, Alexandre GOUAILLARD <agoua...@gmail.com> wrote:

AFAIK, there is no open source tool.

- "Puppet Master", which won an award at webrtc world and expo in 2013, was not made available.

- Jitsi was thinking about creating  a "hammer". Emil, do you have any update on that?

commercial

- some company advertised a "multiplier" tool recently (mixed webrtc sip and others)

- testRTC.com is your bet today

Note that your main "problem" will be the signaling. Since the signaling your app use is not know in advance, you will have to drive the browser or your native app to communicate with your server.

If you do not need modified versions of browsers, no extension (no screensharing), no specific profile on FF, etc, you could use browserstack, saucelabs, and all other could-based browser testing tool. Selenium 2.0 (ex selenium grid) + web drivers is, IMHO your best bet.

Alex.

On Thu, Mar 24, 2016 at 11:44 AM, Antonis Tsakiridis <antonis.tsakiridis@telestax.com> wrote:

Hello,

I'm looking for an open source tool for webrtc load testing mainly to test a webrtc media server where the tool will be the client creating multiple webrtc media connections towards the server. Although it seems like a very handy tool I haven't managed to find any out there.

Best regards,

Antonis Tsakiridis

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.

To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrtc+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/CAM7er2OreUtxQN76WsGWFtOo-GraZ4Mfk0H_Gjmz82Lr1BuzAA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Alex. Gouaillard, PhD, PhD, MBA

------------------------------------------------------------------------------------

Principal Architect - Citrix, San Francisco

President - CoSMo Software Consulting, Singapore

------------------------------------------------------------------------------------

sg.linkedin.com/agouaillard

• 
  

[Emil Ivov]

No, it does not use the webrtc implementation. It intentionally relies on pre-encoded streams that it only encrypts (it also updates time stamps and does tricks to handle FIRs and NACKs) the point being that in order to be able to generate a high load, it cannot be bothered with encoding.

Not sure what "regular" rtp means in your question. It certainly does not send irregular rtp but I am probably misunderstanding you

Emil

To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/CAM7er2OreUtxQN76WsGWFtOo-GraZ4Mfk0H_Gjmz82Lr1BuzAA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Alex. Gouaillard, PhD, PhD, MBA

------------------------------------------------------------------------------------

Principal Architect - Citrix, San Francisco

President - CoSMo Software Consulting, Singapore

------------------------------------------------------------------------------------

sg.linkedin.com/agouaillard

• 
  

--
sent from my mobile

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.

To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/5a71dbbc-bcf3-4e1a-a7f7-db9abdcf214a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
sent from my mobile

[Antonis Tsakiridis]

Ok thanks, I'm curious what loads are you able to accomplish with your tool? Concurrent calls, calls per second?

By the way, for 'regular RTP' I just wanted to differentiate it from Webrtc, but yeah I can understand how it can be confusing.

Bests,

Antonis

You received this message because you are subscribed to a topic in the Google Groups "discuss-webrtc" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/discuss-webrtc/bwpnB-43X6g/unsubscribe.
To unsubscribe from this group and all its topics, send an email to discuss-webrt...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/CAPvvaaKs35OP3vYY5Ab64cgUQSk044SX6g-Q4m5%2Beb3j1dbKoA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Antonis Tsakiridis

Lead Developer, Mobile SDKs at TeleStax

antonis.t...@telestax.com

http://www.telestax.com

[Emil Ivov]

On Fri, Mar 25, 2016 at 6:14 AM, Antonis Tsakiridis
<antonis.t...@telestax.com> wrote:
> Ok thanks, I'm curious what loads are you able to accomplish with your tool?
> Concurrent calls, calls per second?

To be clear: this is not a signalling load testing tool (like for
example most of the SIP-based frameworks are) but a traffic generating
one. In that sense the Jitsi hammer only starts N participants (i.e.,
1 call per participant where the total participants N are specified by
the user), then it never leaves this call until the test is over.

In terms of bitrate, the hammer sends as much as your input file
offers and that could easily fill up whatever the host can offer in
terms of bandwidth.

Emil

> https://groups.google.com/d/msgid/discuss-webrtc/CAM7er2MLJcmDaJZdi%3DxS80Wse_Eb2gjfGnYKJ2NGfAfCA36ZCQ%40mail.gmail.com.

>
> For more options, visit https://groups.google.com/d/optout.



--

https://jitsi.org

[Ivaylo Tsankov]

If you want to test hadware availability of the server you can install turn server and to create clients which create connections and to filter all candidates except turn candidates. Then you can roughly calculate how much resource it will take for one connection.

[mparis]

Hello everybody,
I am going to talk about our experience doing load-testing in Kurento.org, so I hope it can help you to design your own load tests ;)

We have automated tests based on Selenum Grid to test Kurento.org features and the functional WebRTC stuff in general that can be deployed in EC2, OpenStack, etc.
This was the start point for doing load testing, but quickly I realized that it was too expensive. The main reason is that the performance of the RTCPeerConnection is not so good and you need a lot of browsers and a lot of CPUs to try to overload the Media Server (I am talking of hundreds of CPUs).

From this observation, the design of the load testing is based on a mix real/mock clients with the next features:

• The media is provided and "injected" in the KMS (Kurento Media Server) by a real client (Chrome or Firefox).
• On the contrary of the idea of the jitsi-hammer, we prefer working with real media.
• We can add real clients like Chrome or Firefox to receive the media from the KMS
• We can do this using Selenium or in a manual way.
• We can gather stats info to monitor the QoS of the media and see if the KMS is working properly.
• To add overload in a cheap way, we have one or multiple mock KMSs that contains mock WebRTC clients.
  
• This clients are full WebRTC endpoints that support DTLS-SRTP, RTCP, NACKs, FIR, REMBs, etc.
• We also can gather stats info from these clients.
  

With these pieces we can develop tests like "how many legs we can have in a one2many", "how many participants we can have in a room", etc., and contrast results using 1 CPU, 2 CPUs...

Now we are working to finish the automation of some parts to deploy these tests in our Continuous Integration system.

[Antonis Tsakiridis]

Hello mparis, and thanks a lot for your insights, they are very helpful!

Actually, your scenario seems pretty close to ours meaning that our primary objective for this tool was testing our open source Restcomm Media Server and the way to do it was to start calls towards Restcomm from Sipp and have them forwarded to registered Webrtc clients at Chrome/Firefox, each using a separate tab. To stress our Media Server we added more registered Webrtc clients (i.e. more browser tabs). The problem is that when we reached around 50 Webrtc clients accepting calls we started observing that the browsers performance would start to degrade with no apparent reason (checking the load on the headless EC2 instance didn't seem too much). By degrade I mean taking too long to respond.

So what I'm trying to figure out is if 50 tabs talking audio only with Webrtc (i.e. Peer Connection) in parallel is considered too much for a single machin and also what are usually the bottlenecks on the browser side? Can you share your experience with that? Also, where you able to use Selenium to spawn tabs in parallel. Cause when we did that using Python & threads it started off pretty nice but it didn't scale very well, so we fell back to starting the browser tabs ourselves and adding logic in Javascript instead.

Best regards,

Antonis

[YAMANO Yuji]

On Fri, 25 Mar 2016 03:26:53 -0700 (PDT), Antonis Tsakiridis <antonis.t...@telestax.com> wrote:

> Do you guys have any experience with such issues? Are 50 browser tabs
> concurrently handling webrtc connections considered too much load? Are
> there any tutorials on webrtc testing out there that would help me
> understand the issues involved and how to tackle them?

We built a testing tool on EC2 spot instances. Each EC2 instance has
a headless browser and a sinatra API sever to manipulate the browser
from outside. The instances are clustered by serf.

--
YAMANO Yuji
OGIS-RI

[mparis]

Hello Antonis,
I do not have exact values, but I am sure that I needed a lot of machines to overload the media server, so I decided to create the testing infrastructure that I told you in the last post.

If I do not remember bad, in our automated test we create a browser instance per client.

[Antonis Tsakiridis]

Thanks mparis, any chance you recall how many clients per machine?

--

---
You received this message because you are subscribed to a topic in the Google Groups "discuss-webrtc" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/discuss-webrtc/bwpnB-43X6g/unsubscribe.
To unsubscribe from this group and all its topics, send an email to discuss-webrt...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/83b4c5c4-cc86-47fa-bbd6-1c61e889154b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[mparis]

Hello Antonis,
with KMS we did tests with audio and video, but not with only audio.
I do not remember the exact number of clients per machine, but we could not achieve more than 5 (let's say 10 as peak) Chrome clients per machine.
I hope this can help you ;)

[Antonis Tsakiridis]

Thank mparis, that helps a lot indeed :) I wonder what where the bottlenecks after going over 5-10 clients? Was the browser getting too slow and unresponsive? Did you also get any errors in the browser log?

In our case when we put too many clients in a machine we don't get any error in the browser log, it's just that the performance starts to degrade and media packets being lost. Which is weird.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/2679bf28-56ab-40ae-be3c-20ff96d4f76c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[mparis]

You are welcome!!
I saw the same, no error but bad performance.

[Antonis Tsakiridis]

Thank you all for your replies; it really helped us with the development of webrtc-test which although still young has helped us a lot with Restcomm Media Server load testing! As you can see the tool is hosted in GitHub, so if any of you are interested in browsing the code or even participating in its development please drop by and say hi ;)

Best regards

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/f2b990fb-1af4-4f11-b35e-f664977e4ef3%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[austin...@gmail.com]

Hi Antonis
You can use load multiplier (https://loadmultiplier.com) tool to do this testing.
Up to 20 endpoints it is free.

You can go for any number of calls/media stream, looks in a normal desktop you can have up to 50,000 endpoints running.
If load requirement is higher, you need to go for high end blade servers.

Thanks
Austin

[yq lv]

Is any update here？ 

Is  any  open source tool. with function ( mock  WebRTC endpoints that support DTLS-SRTP, RTCP, NACKs, FIR, REMBs, etc.) to stress test Media Server 

[Tsahi Levent-Levi]

The problem with such a requirement is that the tool you are looking for needs to be:

1. Highly distributed
2. Reliant on browsers (since they update so fast and change their WebRTC behavior between versions)
3. Able to scale out to 100's of machines in parallel to support high load
4. Stable (something that is hard to ask from something like Selenium)

From the experience I've had in the past two years at testRTC (this makes me rather subjective here, but with scars to show for it), there's usually too much effort involved in doing that and doing that in a meaningful way - especially now where the market needs are still too small for an open source business model to succeed.

There are just too many moving parts that needs to be taken into account here and the investment in in-house tools (open source or not) is quite expensive.

I'd seriously look at the commercial alternatives. They cost money, but a lot less than investing in completing the requirements missing in the available open source ones today.

[Akshaya Gupta]

Hi Antonis ,

We have Webrtc application and planning to check performance of application.

One meeting can have four guest users and one host  and I want test how many can happen at same time on that server ?

can you please help me with this approach ?

I have exp in automation testing 

[Tsahi Levent-Levi]

Not exactly open source, but you can check https://testrtc.com - we're a paid for subscription service.

Check us out :-)

[Antonis Tsakiridis]

Hi Akshaya,

As already discussed in this channel this is a challenging thing to do. What we did was to come up with a home grown solution, but the problem was that it didn't scale well, so to get clean results we ended up having few clients per headless instance, which made it pretty complex and expensive to run realistic scenarios. Unfortunately we didn't manage to resume this work and get it to a state where it can yield better results and be more reusable just yet.

If you still want to have a look and experiment you can check here https://github.com/RestComm/webrtc-test. Goes without saying that contributions are more than welcome ;)

Best regards,

Antonis

On Wednesday, January 24, 2018 at 6:03:09 PM UTC+2, Akshaya Gupta wrote:

[Selim Emre]

Hi, please review Ant Media Server WebRTC Load Testing Tool in this blog

24 Mart 2016 Perşembe 20:44:12 UTC+2 tarihinde Antonis Tsakiridis yazdı: