Pion Webrtc Janus Example

1,469 views
Skip to first unread message

Matt Nelson

unread,
May 29, 2020, 4:09:31 PM5/29/20
to meetecho-janus
Hello all!

Firstly, thanks for a great product. I've been building with it for a couple of months now and am really enjoying it.

Just found something that might be useful for the resources page (wish I had found it 2 months ago!): https://github.com/pion/example-webrtc-applications/tree/master/janus-gateway

Pion, is a pure Go implementation of the WebRTC API. The project linked above is an example of how to use it with the Janus Gateway. I've tested it and it works really well. 

I've been using Kite as a test framework and load balancer, but we are autoscaling to 800+ rooms so it takes 4 of the largest servers AWS can offer just to run the load balancing and bench marking tests (each participant spins up a headless chrome browser using selemium). The Pion example streams audio and video into a room with a tiny memory footprint. 

Just sharing in case others find it useful. I'm planning on building something for Load Testing a server so will share the repository once that's done. 

Thanks, Matt

Alexandre GOUAILLARD

unread,
May 29, 2020, 5:27:22 PM5/29/20
to Matt Nelson, meetecho-janus
Hi matt,

thanks for the link.

Very early on in the (open) design of KITE, the discussion of browser client vs native client, or server to generate load has happened. Jitsi, which had the "jitsi hammer", janus with Jattack, and others, were part of the discussion. It boils down to three things really, from important to not important:
- you want the result to be representative of what you would have in production, so you must be POSITIVE that what you send is exactly what your web app would send. Having a lower footprint for the test tool, at the price of accuracy of the result did not seem like a good tradeoff. Making sure your test tool produce the same thing on the wire that chrome does is a challenge in itself in the absence of any kind of RTCWEB compliance test suite at IETF.
- You do not want to maintain a separate tool with the same or higher complexity than your client to generate load, there is already enough to maintain.
- Self-testing is nice, but comparative testing is better. If you wanted a tool to generate load for different servers, or plugins in the case of janus, you would need at least to implement each signalling ... If you use the browser, you can directly use web apps provided by the different teams. You trust them to know what they are doing with their SFU. It reduce the barriers of entry for testing, and the surface for errors.

So eventually, it was decided that the extra load and cost triggered by using selenium and the browser was acceptable. It is a feature, not a bug.

Just checking a few things here, not assuming anything:

It looks like you have been trying to do load testing by yourself from the open source KITE repository (e.g. tickets 111 and 112), which only contains the interoperability test suite. Did you speak with the KITE team about using their commercial Grid Manager? Did you compare the footprint you witnessed in your own deployment against what they claim to achieve?

Did you speak with the janus team to try Jattack or other tools they might have?

HTH.

Regards,

--
You received this message because you are subscribed to the Google Groups "meetecho-janus" group.
To unsubscribe from this group and stop receiving emails from it, send an email to meetecho-janu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/meetecho-janus/12fe0646-c996-48d4-bb03-b09d92736f29%40googlegroups.com.


--
Alex. Gouaillard, PhD, PhD, MBA
------------------------------------------------------------------------------------
President - CoSMo Software Consulting, Singapore
------------------------------------------------------------------------------------

Matt Nelson

unread,
May 30, 2020, 8:18:40 AM5/30/20
to meetecho-janus
Kite is great, it let me quickly get an idea of how many rooms I could reasonable hold on various different sized servers in Amazon. It is the right design decision for what it was built for. I've been using the grid manager as (I presume) it's the only way to get tests across multiple machines. 

I've seen a couple of forum posts about Jattack, and they recommend contacting Meetecho directly. I've sent an email and filled out their contact form but haven't heard anything back yet about that.

This was more a post about finding something really useful for us (Pion with example for Janus) and I wanted to share it with the community. Sorry if it came across as a dig against Kite, that wasn't my intention at all! For our use case, as Go developers this really got us excited. We also have image classification needs (in call facial recognition, object detection) which we will be able to use Go for now so we can subscribe to the publishers and process the stream live. Currently we are processing images captured from a headless chrome instance using puppeteer! 

Thanks.


On Friday, May 29, 2020 at 10:27:22 PM UTC+1, Alexandre GOUAILLARD wrote:
Hi matt,

thanks for the link.

Very early on in the (open) design of KITE, the discussion of browser client vs native client, or server to generate load has happened. Jitsi, which had the "jitsi hammer", janus with Jattack, and others, were part of the discussion. It boils down to three things really, from important to not important:
- you want the result to be representative of what you would have in production, so you must be POSITIVE that what you send is exactly what your web app would send. Having a lower footprint for the test tool, at the price of accuracy of the result did not seem like a good tradeoff. Making sure your test tool produce the same thing on the wire that chrome does is a challenge in itself in the absence of any kind of RTCWEB compliance test suite at IETF.
- You do not want to maintain a separate tool with the same or higher complexity than your client to generate load, there is already enough to maintain.
- Self-testing is nice, but comparative testing is better. If you wanted a tool to generate load for different servers, or plugins in the case of janus, you would need at least to implement each signalling ... If you use the browser, you can directly use web apps provided by the different teams. You trust them to know what they are doing with their SFU. It reduce the barriers of entry for testing, and the surface for errors.

So eventually, it was decided that the extra load and cost triggered by using selenium and the browser was acceptable. It is a feature, not a bug.

Just checking a few things here, not assuming anything:

It looks like you have been trying to do load testing by yourself from the open source KITE repository (e.g. tickets 111 and 112), which only contains the interoperability test suite. Did you speak with the KITE team about using their commercial Grid Manager? Did you compare the footprint you witnessed in your own deployment against what they claim to achieve?

Did you speak with the janus team to try Jattack or other tools they might have?

HTH.

Regards,

On Fri, May 29, 2020 at 10:09 PM Matt Nelson <matt....@silverark.co.uk> wrote:
Hello all!

Firstly, thanks for a great product. I've been building with it for a couple of months now and am really enjoying it.

Just found something that might be useful for the resources page (wish I had found it 2 months ago!): https://github.com/pion/example-webrtc-applications/tree/master/janus-gateway

Pion, is a pure Go implementation of the WebRTC API. The project linked above is an example of how to use it with the Janus Gateway. I've tested it and it works really well. 

I've been using Kite as a test framework and load balancer, but we are autoscaling to 800+ rooms so it takes 4 of the largest servers AWS can offer just to run the load balancing and bench marking tests (each participant spins up a headless chrome browser using selemium). The Pion example streams audio and video into a room with a tiny memory footprint. 

Just sharing in case others find it useful. I'm planning on building something for Load Testing a server so will share the repository once that's done. 

Thanks, Matt

--
You received this message because you are subscribed to the Google Groups "meetecho-janus" group.
To unsubscribe from this group and stop receiving emails from it, send an email to meetech...@googlegroups.com.

uxmaster

unread,
May 31, 2020, 4:21:55 AM5/31/20
to meetecho-janus
In load testing one doesn't care about interoperability, that is a different domain of testing.

I find Kite too complex to set up, while Meetecho contact form apparently sends messages to /dev/null, so I've quickly written a plugin and a transport for Janus (in C) to do the load testing of not only a Janus instance (like Jattack) but of all the components (since it logs in like a normal user except not using a web browser).

Alexandre GOUAILLARD

unread,
May 31, 2020, 5:24:29 AM5/31/20
to uxmaster, meetecho-janus
hi jacek,

Thank you for sharing your opinion.

I do not think there is a perfect system when it comes to testing. That s why, instead of providing our opinion on how "complex", or "adequate" a given solution is, which would be subjective in nature and relate to one capacity and one use case, we try to document and explain the (arbitrary) choices made for a given tool so anybody make an informed choice given their own use case, their ressource, and their technical capacity. Matt decision, in his case, makes a lot of sense even if only from the programming language point of view.

Janus published a paper, we published multiple peer review papers, blog posts, and provided some code, so the information is publicly available to all. The kurento team also contributed a lot to the field. One can find below a list of the most recent papers on Webrtc testing, which do differentiate between testing types, some of them we authored.

I do not think anybody mentioned, or meant,  "interoperability" earlier. What was said is that you need to compare things that are comparable. Let's imagine you send 720p streams and in productions you get 1080p streams, of course it would change things. This is quite an obvious example, but some more subtle cases are equally impactfull. Everything that deals with packet loss or bandwidth adaptation can end up (through RED, FEC or RTX, or the re-send of full frames) impact the bandwidth usage and tip your server over the limit.

Given the multiple variations one can see in a "webrtc stream", how are you sure that your simulated stream during the test actually represent the real stream in production? To those who think the variations are small, and wonder how different streams could really be, I would like to point to the effort Cullen Jennings, Cisco CTO for communication did in 2018: counting all the specifications in the webrtc dependency tree. pop quizz; Anybody dare risking a number? 200+ specifications, that take 30 pages to list. For the curious, I'm adding the corresponding document to this email. Until the IETF decides to have a formal test suite, all those modules bring in variability surface.

The conclusion is: there is absolutely no way to be sure you are producing the same stream that you would get from your client software in production unless you actually use the same software client in your testing. That leads to using "instrumentation" of those clients instead of "simulation/emulation" of those clients. 

Now, that s the perfect scenario, and some people can say:
- I don't care about the approximation i m making, it's good enough,
- I want something quick and dirty
- I want something in a language i m proficient in
- ....

So there is more than one good answer, practically.

Most open source projects, when it comes to the audience, draw a line between competent people that have more time than money (a), and competent people that have more money than time or people not capable/willing to master the tech (b). a-type people use the open source and are super happy with it, b-type people reach out to the author and take a commercial package, with SLA, and possibly some professional services.

With respect to the complexity of open source software, it subjective and relative to  your own capacity. Mileage varies, especially depending on the language. I find at time KITE difficult, mainly because I don't know Java :-) As a sanity check, we had the CFO and the entire sales team install it and run it from the readme. After a few iterations on the readme, they succeeded. 

The open source KITE is only the interoperability testing engine. All the load testing modules and extensions are commercial. Trying to use the open source KITE for load testing is positioning oneself for challenges and complexity. CoSMo has a "grid manager" offer which prepackaged selenium nodes that gets automatically deployed and started for you, test get uploaded through a web GUI, and everything is run automatically. click and play. So easy a caveman could do it. Grid manager comes at under a couple of hundred bucks, and you use your own cloud account.

We hope this helped. Do not hesitate to ask for more info on the part where we would not have been clear enough, always happy to help.

2017: Real-time communication testing evolution with WebRTC 1.0
2017: WebRTC Testing: Challenges and Practical Solutions
https://ieeexplore.ieee.org/abstract/document/7992926
2017: Jattack: a WebRTC load testing tool
2018: Comparative Study of WebRTC Open Source SFUs for Video Conferencing





To unsubscribe from this group and stop receiving emails from it, send an email to meetecho-janu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/meetecho-janus/dac4d71c-015d-46fd-8eea-4567a9e25f28%40googlegroups.com.
draft-jennings-rtcweb-deps-22.txt

Alessandro Amirante

unread,
May 31, 2020, 5:46:23 AM5/31/20
to uxmaster, meetecho-janus
Meetecho contact form does not send messages to /dev/null! We try and reply to all inquiries we get, even though we're swamped with requests these days and may be a bit slower than usual. Sometimes our replies bounce back though, as people don't seem to pay attention to the email address they put in the contact form...

A.

To unsubscribe from this group and stop receiving emails from it, send an email to meetecho-janu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/meetecho-janus/dac4d71c-015d-46fd-8eea-4567a9e25f28%40googlegroups.com.

uxmaster

unread,
May 31, 2020, 11:42:52 AM5/31/20
to meetecho-janus
Glad to hear it Alessandro. I'm not complaining at all since it turned out very well for us.

uxmaster

unread,
May 31, 2020, 12:11:16 PM5/31/20
to meetecho-janus
I do not think anybody mentioned, or meant,  "interoperability" earlier.
It looks like you have been trying to do load testing by yourself from the open source KITE repository (e.g. tickets 111 and 112), which only contains the interoperability test suite.

--
Alex. Gouaillard, PhD, PhD, MBA
------------------------------------------------------------------------------------
President - CoSMo Software Consulting, Singapore
------------------------------------------------------------------------------------

Sapienti sat.
I'm out. I regret I posted a comment in this thread.

Lorenzo Miniero

unread,
May 31, 2020, 1:03:26 PM5/31/20
to meetecho-janus
Are you going to send it to /dev/null?

L.
Reply all
Reply to author
Forward
0 new messages