Using WebRTC to send an iOS devices’ screen capture using ReplayKit

[Niro]

We would like to use WebRTC to send an iOS devices’ screen capture using ReplayKit. The ReplayKit has a processSampleBuffer callback which gives CMSampleBuffer.

But here is where we are stuck, we can’t seem to get the CMSampleBuffer to be sent to the connected peer. We have tried to create pixelBuffer from the sampleBuffer, and then create RTCVideoFrame.

we also extracted the RTCVideoSource from RTCPeerConnectionFactory and then used an RTCVideoCapturer and stream it to the localVideoSource.

Any idea what we are doing wrong?

var peerConnectionFactory: RTCPeerConnectionFactory?

override func processSampleBuffer(_ sampleBuffer: CMSampleBuffer, with sampleBufferType: RPSampleBufferType) {
 switch sampleBufferType {
           case RPSampleBufferType.video:

        // create the CVPixelBuffer
        let pixelBuffer:CVPixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)!;

        // create the RTCVideoFrame
        var videoFrame:RTCVideoFrame?;
        let timestamp = NSDate().timeIntervalSince1970 * 1000
        videoFrame = RTCVideoFrame(pixelBuffer: pixelBuffer, rotation: RTCVideoRotation._0, timeStampNs: Int64(timestamp))

        // connect the video frames to the WebRTC
        let localVideoSource = self.peerConnectionFactory!.videoSource()
        let videoCapturer = RTCVideoCapturer()
        localVideoSource.capturer(videoCapturer, didCapture: videoFrame!)

        let videoTrack : RTCVideoTrack =   self.peerConnectionFactory!.videoTrack(with: localVideoSource, trackId: "100”)

        let mediaStream: RTCMediaStream = (self.peerConnectionFactory?.mediaStream(withStreamId: “1"))!
        mediaStream.addVideoTrack(videoTrack)
        self.newPeerConnection!.add(mediaStream)

        break
    }
}

[BWSwift]

Hi, i have several ideas how to help you:

- generate time stamp using this: let timeStampNs: Int64 = Int64(CMTimeGetSeconds(CMSampleBufferGetPresentationTimeStamp(sampleBuffer)) * 1000000000)
- create VideoCapturer, VideoSource, MediaStream, VideoTrack only once in override func broadcastStarted(withSetupInfo setupInfo: [String : NSObject]?) method. In the processSampleBuffer only create VideoFrame and process it to the videoSource that you allocated before.

- in the latest version of WebRTC init with pixel buffer for RTCVideoFrame is deprecated so try to use RTCCVPixelBuffer. 

- make sure that you generate SDP-offer after setuping localStream in peerConneciton and it contains a=sendonly flag.

[VladimirTechMan]

Hi Niro,

In addition to BWSwift's recommendations, you may also check the sample code of broadcast extension in the AppRTCMobile demo app. You can check the details in my reply on related matter at the following link:

https://groups.google.com/forum/#!topic/discuss-webrtc/jAHCnB12khE

[utkarsh agarwal]

Hi BW Swift,

I am trying to implement screen sharing using replay kit but facing issues as no video is being delivered to media server.

I have few doubts

1) Do we need to add track for every buffer captured ?

2) Do we need to add stream in RTCPeerConnection for every track added ?
3) Is RTCRTPSender is an alternative to stream the video and we do need to implement RTCRtpReceiver as well for it ?

[Ashish Verma]

Hi Vladimir,

I have checked broadcast extension code of AppRTCMobile demo app, It's using some of WebRTC framework methods and classes which are not available in the latest version of WebRTC framework which is installed via Cocoa Pods.
For e.g:

<WebRTC/RTCRtpTransceiver.h> (not available)

[_peerConnection addTrack:track streamIds:@[ kARDMediaStreamId ]] (not available)

How can we get the latest sdk ? or What is the alternative solution?

[Neil Young]

Having exactly the same problem as the OP initially reported: No video is sent.

Would one of you mind to check this post?

https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/discuss-webrtc/sr9r6p2OXZY/C4d-xTBOCgAJ

TIA