janus says slow_link, but it's not (need advice)

[Alex Y]

Hello.

Need advice. Janus is deployed on Digital Ocean. I'm using janus.videoroom plugin, it works well for me. But when my friends from other city tried to join same room, they get following message: {"videoroom":"slow_link","current-bitrate":128000} - and they can't see/hear each other.

But strange thing, they can see each other at https://janus.conf.meetecho.com/videoroomtest.html - with no errors or problems. Also, they easily can talk with any WebRTC service or skype. Problem seems to occure only our own server, but not for everyone. What should I debug/test next?

[Alex Y]

P.S. Another thins it's saying, at webrtcState callback: Janus says this WebRTC PeerConnection is down now

воскресенье, 25 сентября 2016 г., 19:56:16 UTC+8 пользователь Alex Y написал:

[Lorenzo Miniero]

You should check the Admin API for the affected handles. That will allow you to check if there are losses or NACKs that explain why the slow_link is being triggered. Besides, consider that if the problem is on the publisher side (publisher failing to send many packets) the same losses will be perceived by all viewers as well (they won't receive a packet Janus never received and so may try to NACK it several times), even though their bandwidth might be ok.

L.

[Chad Furman]

Do they see eachother ever on your own servers?  You might be getting blocked by some firewall settings.  Not sure if the videoroomtest on Janus server uses TURN / STUN.  I didn't experience this trouble on Linode, personally.  Never tried Digital Ocean.  Did have lots of trouble with AWS but our DevOps team eventually got it working on AWS (somehow :\ )

If you have firewalls or security settings at all, that's a big one.  Also, depending on where you and your friend are, the connection to Digital Ocean could be laggy.  Try getting a few people in the room and see (no pun intended) who you can and cannot see.  Have everyone watch JS console for slowlinks, and check the output of Janus in the terminal.  You might find that the terminal is showing you some useful information you're not seeing in your JS console (probably not, but worth checking out)

[Alex Y]

Thanks for answers Lorenzo and Chad. I can't deep into this since my friend is offline for a ~week, but I totally will.

Chad, they see "black screen" instead of webcam image. And no sound also. It worked sometime ago. Your words about TURN/STUN remind me I configured Xirsys' servers to use. Probably I should turn them off temporarily and check if anything changes.

четверг, 29 сентября 2016 г., 13:44:11 UTC+8 пользователь Chad Furman написал:

[Chad Furman]

Check for "ICE failed" messages, also.  If you're seeing black screens, it could be that someone's getting "ICE failed" messages or "hangup" messages sent down.  There could be a certificate error, also.  Janus requires a valid SSL certficiate, and getUserMedia only works on HTTPS domains.

[Lorenzo Miniero]

Logs and console can help, but don't rely on those too much. Check the Admin API instead. A black screen can mean different things, as explained here:

http://www.meetecho.com/blog/understanding-the-janus-admin-api/

L.

[Alex Y]

Hello!

Sorry for so long responding. Did other things and finally came back to test&debug WebRTC part of our app.

Situation: some users can't publish audio/video, but they can see other's video.

Here is log from Admin API for that handle: http://pastebin.com/sV8qrFTF

As you can see, state is "connecting". And it's never change. At browser side, Janus still says slow_link.

If it matters, at september we had 512mb RAM (~40mb free). Our swap usage was ~250 mb. Now we have 1gb RAM (~100 mb free). Our current swap usage is ~85 mb.

воскресенье, 25 сентября 2016 г., 19:56:16 UTC+8 пользователь Alex Y написал:

Hello.

[Alex Y]

Anyone, please help?

[Mirko Brankovic]

Hi,

I tested the slow_link and is triggered when you receive some number of NACKs, I think something like >200 bytes.

Maybe it it not noticable by eye but if nacks are sent then there is a problem on link.

Maybe you can open admin API from example same  and reproduce the problem. On Session/Handles you can see details like , packets received or sent, depending on the handle, and also number of NACKs

On Mon, Nov 21, 2016 at 7:31 AM, Alex Y <alexander...@gmail.com> wrote:

Anyone, please help?

--
You received this message because you are subscribed to the Google Groups "meetecho-janus" group.
To unsubscribe from this group and stop receiving emails from it, send an email to meetecho-janus+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Regards,

Mirko

[Lorenzo Miniero]

more than 8 nacks in a second trigger a slowlink at the moment. If you need this to be more/less sensible to that, change the code.

L.

[Alex Y]

Thanks Mirko and Lorenzo.

Where do you see number of NACKs? Is it "sdps.remote" field?

As said in first post, my friends has no problems using YOUR demo server.

As for me, I have no problems at ANY server: your or mine.

So.. by any chance, how do you think, may the problem be solved by (1) using other country server or (2) updating Janus or (3) upgrading server by adding more RAM and cpus?

I'm kinda stuck on this. All my friends has stable cable-based internet connection with good download/upload speed (50-100 mbit/sec both).

[Mirko Brankovic]

When you for example look at the admin page when call is in progress:

if you scroll down to bottom you will see the remote/ local sdp and also stats of the streams.

Similar thing you can get with chrome://webrtc-internals/ from Chrome browser, for specific ssrc, googNacksReceived,....

example:

--

You received this message because you are subscribed to the Google Groups "meetecho-janus" group.
To unsubscribe from this group and stop receiving emails from it, send an email to meetecho-janus+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Regards,

Mirko

[Alex Y]

1. Well.. I understand nothing. I try https://janus.conf.meetecho.com/echotest.html and there is only one slow_link message in 10 minutes. Then I try same echotest at our virtual servers = 9-10 slow_links in 10 minutes. Both servers - in Amsterdam and Moscow. Lorenzo, please show your server config - cpu, ram. Is it virtual server or bare metal? Which operating system?

2. And what minimal requirements to client connection? In other words.. Which tests in browser we can do if we want find out reasons of disconnects/slow_links/other problems? First one, speedtest.net of course. Second one is probably https://www.netscan.co. Another tests? The main reason why I question this: we have ~20 users who can and want to test our app. Problem is, HALF of them couldn't see/hear anything with Janus. And couldn't stream anything. But any other WebRTC chat is working for them. Skype is working for them. I wanna know why that's happening.

3. One of our users sent me following report: https://www.netscan.co/r/RtpJJ According to netscan, his browser doesn't support "WebRTC P2P". Is Janus using "WebRTC P2P"?

[Mirko Brankovic]

Well skype is not webrtc if you are using its client. Unless you are using web.skype.com whick might be beta webrtc.

Maybe you can ask them to test other websites and also other browsers, just in case.

https://test.webrtc.org

https://github.com/webrtc/samples

--

[Lorenzo Miniero]

Il giorno lunedì 28 novembre 2016 14:38:04 UTC+1, Alex Y ha scritto:

1. Well.. I understand nothing. I try https://janus.conf.meetecho.com/echotest.html and there is only one slow_link message in 10 minutes. Then I try same echotest at our virtual servers = 9-10 slow_links in 10 minutes. Both servers - in Amsterdam and Moscow. Lorenzo, please show your server config - cpu, ram. Is it virtual server or bare metal? Which operating system?

It's a minimal docker instance with Ubuntu 14.04 on it. You're most likely experiencing network issues. Slow links are caused by packet loss.

 

2. And what minimal requirements to client connection? In other words.. Which tests in browser we can do if we want find out reasons of disconnects/slow_links/other problems? First one, speedtest.net of course. Second one is probably https://www.netscan.co. Another tests? The main reason why I question this: we have ~20 users who can and want to test our app. Problem is, HALF of them couldn't see/hear anything with Janus. And couldn't stream anything. But any other WebRTC chat is working for them. Skype is working for them. I wanna know why that's happening.

IMHO speed tests are often misleading, and not at all representative when it comes to real-time media. They're very often based on TCP channels for throughput testing, which is not what you do with WebRTC. https://test.webrtc.org/ is probably a better option for checking.

 

3. One of our users sent me following report: https://www.netscan.co/r/RtpJJ According to netscan, his browser doesn't support "WebRTC P2P". Is Janus using "WebRTC P2P"?

WebRTC is *always* P2P, it's how it was conceived. Even when using a server like Janus, you're having P2P sessions with the server itself. So no support for WebRTC P2P either means he's using a non-WebRTC browser, or simply that WebRTC is filtered or constrained in those networks, and you can't really blame Janus for that.

L. 

[Alex Y]

It's a minimal docker instance with Ubuntu 14.04 on it. You're most likely experiencing network issues. Slow links are caused by packet loss.

Why I'm not experiencing network issues at http://janus.conf.meetecho.com/ ? One slow_link in 10 minutes seems fine to me. One slow_link every minute seems not fine. How can I confirm or refute that I'm having packet losses?
 

 

IMHO speed tests are often misleading, and not at all representative when it comes to real-time media. They're very often based on TCP channels for throughput testing, which is not what you do with WebRTC. https://test.webrtc.org/ is probably a better option for checking.

 I agree about speed tests. Thanks to Mirko for link.

 

WebRTC is *always* P2P, it's how it was conceived. Even when using a server like Janus, you're having P2P sessions with the server itself. So no support for WebRTC P2P either means he's using a non-WebRTC browser, or simply that WebRTC is filtered or constrained in those networks, and you can't really blame Janus for that.

 Yeah, thanks for confirmation. It just seems confusing to me that Netscan says browser support WebRTC and doesn't support P2P WebRTC. I really don't blame Janus for anything. IMHO Janus is great, but when real people from different locations experience same issues, it's strange.

BTW, yesterday we tested one location where WebRTC didn't worked. We used private VPN. Then webRTC started to work. TURN/STUN servers we use didn't help, but VPN at laptop helped. Right now we are deeping into this.

[Lorenzo Miniero]

Il giorno martedì 29 novembre 2016 10:58:06 UTC+1, Alex Y ha scritto:

It's a minimal docker instance with Ubuntu 14.04 on it. You're most likely experiencing network issues. Slow links are caused by packet loss.

Why I'm not experiencing network issues at http://janus.conf.meetecho.com/ ? One slow_link in 10 minutes seems fine to me. One slow_link every minute seems not fine. How can I confirm or refute that I'm having packet losses?

It's not at all unusual to have different experiences towards different servers/services. The network path to a destination is not always the same, and neither are the conditions along the way. Either your servers have some problems of their own, or the path to them do for some users.

 

 

 

IMHO speed tests are often misleading, and not at all representative when it comes to real-time media. They're very often based on TCP channels for throughput testing, which is not what you do with WebRTC. https://test.webrtc.org/ is probably a better option for checking.

 I agree about speed tests. Thanks to Mirko for link.

 

WebRTC is *always* P2P, it's how it was conceived. Even when using a server like Janus, you're having P2P sessions with the server itself. So no support for WebRTC P2P either means he's using a non-WebRTC browser, or simply that WebRTC is filtered or constrained in those networks, and you can't really blame Janus for that.

 Yeah, thanks for confirmation. It just seems confusing to me that Netscan says browser support WebRTC and doesn't support P2P WebRTC. I really don't blame Janus for anything. IMHO Janus is great, but when real people from different locations experience same issues, it's strange.

I know, it was a way to say "the issue is likely somewhere else" :-)

 

BTW, yesterday we tested one location where WebRTC didn't worked. We used private VPN. Then webRTC started to work. TURN/STUN servers we use didn't help, but VPN at laptop helped. Right now we are deeping into this.

This seems to suggest that location is heavily filtered. In those situations, often the only way to get through is via TURN/TCP over port 443, which is usually enough to "trick" the firewall into thinking it's HTTPS while it really isn't. Anyway, it's also going to be inefficient probably. If possible, try to get in touch with network admins to loosen up the constraints for your service/servers.

L.

[Alex Y]

вторник, 29 ноября 2016 г., 18:02:43 UTC+8 пользователь Lorenzo Miniero написал:

Il giorno martedì 29 novembre 2016 10:58:06 UTC+1, Alex Y ha scritto:

It's a minimal docker instance with Ubuntu 14.04 on it. You're most likely experiencing network issues. Slow links are caused by packet loss.

Why I'm not experiencing network issues at http://janus.conf.meetecho.com/ ? One slow_link in 10 minutes seems fine to me. One slow_link every minute seems not fine. How can I confirm or refute that I'm having packet losses?

It's not at all unusual to have different experiences towards different servers/services. The network path to a destination is not always the same, and neither are the conditions along the way. Either your servers have some problems of their own, or the path to them do for some users.

 

How do you think, traceroute and /or WinMTR tool could help to find out something useful?

[Mirko Brankovic]

Bare in mind that if you use turn server that he will relay traffic between your client and server, so janus server will see traffic coming from ip of turn server. Make sure that traffic is not blocked also.

--