Mac Firefox Lip-sync

[Dan Briggs]

Hello everyone,

I've noticed an audio/video sync issue that occurs on Mac in Firefox. The issue appears to be tied to CPU usage.

Steps to reproduce:
- Go the the janus video room (https://janus.conf.meetecho.com/videoroomtest.html) on a Mac using Firefox. I'm running v10.12.5 on a 2017 Mac Pro although I've also been able to reproduce it on an older Mac.
- Start streaming

- Open another tab and join the room. This can be on the same device or another device. This is the observer to watch for sync issues.
- Stress your CPU by pulling a 4k video on YouTube (https://www.youtube.com/watch?v=6pxRHBw-k8M). You could also just open up 4 or 5 tabs on the video room instead of pulling a YouTube video.

- Wait a minute or two

Expected result:
- Perfect audio-video synchronization.

Actual result:
- The stream my Mac is uploading is out of sync for viewers (audio before video). Sync issues begin fairly quickly (although the exact time seems to vary). The lip-sync gets progressively worse the longer the stream goes on for. I've attached a few graphs from webrtc internals from a viewer on Chrome.


Notes and observations
The reason I'm posting this here instead of filing a Mozilla bug report is that the issue doesn't seem to occur for p2p connections. I ran a p2p test where I was stressing my machine by pulling pulling two 4k YouTube videos simultaneously and didn't experience any lip-sync issues over a twenty minute conversation.

The issue doesn't seem to show up on Chrome unless I absolutely max out the CPU on my Mac. Even then, the delay appears to correct itself when the CPU stabilizes. Firefox doesn't seem to correct itself under any circumstances.

I've tested Firefox on Linux and Windows machines and haven't gotten any lip-sync issues, although that doesn't rule them out for sure. Based on the testing I have done, this appears to be Mac-Firefox specific. 


Any thoughts as to what might be causing this? For anyone using a Mac, are you able to reproduce this using the steps above?

Thanks in advance!

Dan

[Lorenzo Miniero]

Have you checked with the Admin API if any packet has been lost, either on the publisher or the viewers?

L.

[Dan Briggs]

Thanks for the quick response, Lorenzo. I've attached a snapshot of the data returned from the admin api about 20 minutes into a video conversation. There was about 2 seconds of A/V desync at this time. Both pub and viewer data are attached. Doesn't seem to be anything out of the ordinary but then again I'm not super familiar with the admin stats. Let me know if you want me to try anything else or grab some more data!

Cheers,

Dan

[Lorenzo Miniero]

I don't see any packet loss or NACK (well, 4 video nacks but over 20 minutes, so nothing), so it's not that. Have you tried recording the publisher and post-processing it to see if you see the desync there too? I seem to remember you mentioned experimenting with RTP forwarding too: is desync happening when getting the RTP-forwarded publisher too? By the way, when you say Mac OS only, do you mean when publishing from MacOS, i.e., all viewers (no matter what they're using) get the desync if the source is an overloaded Mac OS user?

L.

[Dan Briggs]

This is interesting. If I rtp-forward to a separate server that then transcodes the stream into rtmp I don't seem to have any lip-sync issue. Even when viewers of the same stream are seeing it as several seconds out of sync, the transcoded stream remains in sync.

From a viewers perspective, it doesn't seem to matter if the viewer is connected directly to the video room or if they are receiving the stream on a separate server that has been rtp-forwarded to. The lip-sync shows up in both scenarios in approximately equal magnitude.

That's correct with regards to Mac OS. It only appears to matter what device/browser the publisher is using. Viewers on all devices/browsers that I've tested with see the same stream get progressively further out of sync (tested on Windows, Mac, and Linux with Chrome and Firefox). From my observations, the viewer device/browser has no effect on the lip-sync.

I don't think this will come as a surprise but I'll mention it in case it is helpful: the lip-sync resets itself when the viewer refreshes the page. Example: I'm viewing a stream coming from Mac/Firefox. Over 10 minutes the stream is getting progressively further out of sync. After 10 minutes it is a second or two out of sync. If I refresh the page and connect to the stream it will now be perfectly in sync. Of course it will then begin slowly drifting out of sync again as it did originally.

Hope this helps!

Dan

[Lorenzo Miniero]

Wondering if this may be related to the SR RTCP packets we send to the viewers. One thing you can try to do is commenting these two lines out:

https://github.com/meetecho/janus-gateway/blob/master/ice.c#L3465

https://github.com/meetecho/janus-gateway/blob/master/ice.c#L3505

which will do exactly that. Can you check if you see any difference this way?

Thanks,

L.

[Dan Briggs]

Commented them both out and it seemed to resolve the issue! Can I safely comment these out permanently or are there repercussions that I should be worried about?

[Lorenzo Miniero]

Feel free to comment them out, for now. Not exactly sure if the experience can worsen without those (it shouldn't), but we debated whether they could be the culprit of this issue a few months ago already, and your test confirms this. Notice this doesn't mean SR are bad, they're not: it means something we put in there gives the recipient wrong information about delays and stuff, which results in what you experience. We should actually work on fixing those, as they provide statistics that are useful on the client side as well.

I'll work on this in the next few weeks: is it ok if, when something's available, I'll ask you to give these efforts  a run to see if things improve, considering you can replicate the problem easily?

Thanks,

Lorenzo

--
You received this message because you are subscribed to the Google Groups "meetecho-janus" group.
To unsubscribe from this group and stop receiving emails from it, send an email to meetecho-janus+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dan Briggs]

Excellent, good to hear :) I'll report back if I notice any weirdness caused by commenting these lines out.

Sounds good, I'd be more than happy to help out. Just let me know when you have some code to test!

Dan 

To unsubscribe from this group and stop receiving emails from it, send an email to meetecho-janu...@googlegroups.com.

[Dan Briggs]

Hi Lorenzo,

Just wondering if you've had a chance to work on this? Anything you need help testing? Let me know!

[Lorenzo Miniero]

No, looks like a Chrome thing with recvonly streams not liking SR packets. Works fine with Firefox. Did commenting the SR out indeed improve the experience for you as discussed?

L.

[Dan Briggs]

Apologize for the delay! It seems to have solved the reported issue with Firefox. Unfortunately I've now seen a few issues popping up on Chrome. Not sure if it is related to commenting out the SR or not though.

With some devices I've been able to consistently reproduce lip-sync issues by disabling video locally for awhile and then re-enabling by just toggling stream.getVideoTracks()[0].enabled = true/false. Have you ever seen issues with this before?

Dan

[Ju Ju]

on a mac I have big sync issue on both Chrome AND FF: 3 sec delay since seconde 1

it seems to happen only with the recording plugin and not when recording in videoroom

I can't comment out the 2 lines without having  compilation errors (for instance because of the for loop line #3010)

I'm on the last master and on a mac this plugin is just unusable.

Something weird: the sync issue didn't happen the "first" recording atempt of the day.

JJ-