iOS to Chromecast Lip Sync and FPS problems

201 views

Skip to first unread message

Jan Van Ottele

unread,

Feb 8, 2018, 3:17:38 AM2/8/18

to discuss-webrtc

We are using the WebRTC.org library on iOS. Streaming real-time 720p video from iPhone/iPad to Chromecast.

We provide raw video frames using the method

- (void)capturer:(RTCVideoCapturer *)capturer didCaptureVideoFrame:(RTCVideoFrame *)frame;

in the RTCVideoCapturerDelegate protocol.

For audio, we deliver 48kHz PCM data in 10ms chunks to a custom subclass of webrtc::AudioDeviceModule.

In general it works, but we face two problems that we hope to get some advice about:

1. Lip Sync Issue

Lip sync is about 0.5 sec off because the PCM audio data is only available to us in large chunks of 0.5 sec or even more. Thus, we deliver audio 0.5 or more seconds later than the corresponding video.

The only solution we could find is to delay the raw video frames by the appropriate amount of time, to match the corresponding audio. However, our code is running in a very constrained iOS app extension environment, with severe memory and other limitations (only 50MB available etc.). Buffering raw video frames requires a lot of memory and often crashes the software.

We are trying to solve this by adjusting the video and audio timestamps. However, had no luck with this yet. Any changes we try, both outside and inside the WebRTC library, don't seem to have any effect on the lip sync issue.

Could you please point us to the right place where to change PTS timestamps to make sure the audio and video plays in sync on Chromecast? Or another solution?

2. Slow-motion video on Chromecast with high FPS

When streaming to the Chrome browser on Mac running https://appr.tc , the 30 frames per second stream works well. When streaming the same to Chromecast with a custom app based on the https://appr.tc example, Chromecast plays a 30 FPS video stream in slow motion (so, effectively it plays only about 10 FPS), while audio plays at normal speed. The only solution we found is limiting the source video FPS to about 10, then Chromecast seems to play it at a normal speed. However, the video is not smooth because of the low framerate.

We can see that Chromecast is capable of playing higher FPS via WebRTC. For example, screen mirroring from the Chrome browser on Mac delivers a good performance with a high resolution stream.