how to handle (audio+video) stream and screen sharing stream in single peer connection.

4,991 views
Skip to first unread message

Adish komaravelli

unread,
Mar 20, 2019, 3:06:09 AM3/20/19
to discuss-webrtc
hello everyone,

  i am able add (audio + video) track to peer connection using pc.addTrack() and remote peer receiving tracks using pc.ontrack event after setting up remote description.

 now i want to add screen sharing option to my web application. so when user clicks on share screen option, i am getting screen sharing stream using navigator.mediaDevices.getDisplayMedia() WEB API. so i want to add this stream to peer connection.

 so later during the webrtc session, i tried adding screen sharing stream to peer connection using pc.addTrack() but pc.ontrack event not invoking at remote peer side. my question is: is there a way to handle multiple streams in single peer connection? if yes, am i missing anything here.

  please let me know if anyone have thoughts on this

Thank You

 


Neil Young

unread,
Mar 20, 2019, 3:31:31 AM3/20/19
to discuss-webrtc
That would be interesting for me too. In fact, a later pc.addTrack of the shared screen stream does not lead to a remote appearance of the new track. Instead a pc.onnegotiationneeded comes. Not sure what do do with this.

I could work around by first preparing the screen sharing stream and then - if it comes to navigator.getUserMedia - using this instead of the stream of the camera. If you add the sharing stream at the beginning, it works.

But I'm sure it would also work in the middle. Just also don't know how :)

Christoph Eggert

unread,
Mar 20, 2019, 4:23:41 AM3/20/19
to discuss-webrtc
Pardon me if it's a stupid question, but since it's not mentioned: Did you re-negotiate after adding the track? If not, there is no .ontrack fired because this only happens after you apply the remote stream, which still only has one track for the receiver. The receiver basically just doesn't know about the existence of a new track, it needs to receive the new SDP and negotiate how the new video is sent. 


Am Mittwoch, 20. März 2019 08:06:09 UTC+1 schrieb Adish komaravelli:

Neil Young

unread,
Mar 20, 2019, 4:49:58 AM3/20/19
to discuss...@googlegroups.com
Good question. In fact no, no renegotiation. What is exactly the expected behavior on pc.onnegotiationneeded? I mean what SDP has to be fetched and forwarded to remote and what is the expected behavior at remote side in that moment.

I suppose pc.localDescription has to be taken and put to where?

Thanks

Sent from my iPhone

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/9b4f352f-dd7e-411b-9675-5f23bed514c0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Philipp Hancke

unread,
Mar 20, 2019, 4:52:24 AM3/20/19
to discuss...@googlegroups.com
There is a sample for that:

Note that it doesn't rely on negotiationneeded which remains busted in Chrome:

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/D34D7E0F-79C6-4D83-B3A9-A7164FB0FD4A%40googlemail.com.

Christoph Eggert

unread,
Mar 20, 2019, 4:59:34 AM3/20/19
to discuss-webrtc
After you added the track, you can create a new offer to send to the other party and just negotiate it normally again as initially. Then create an answer on the other end etc., you know the drill. The connection will then extend itself without breaking anything in the meantime. You don't even need onnegotiationneeded for that since you already know when you are adding the track anyway.

Neil Young

unread,
Mar 20, 2019, 5:19:40 AM3/20/19
to discuss...@googlegroups.com
Thanks Philipp

Sent from my iPhone

Neil Young

unread,
Mar 20, 2019, 5:20:20 AM3/20/19
to discuss...@googlegroups.com
Very cool and logical. Thanks

Sent from my iPhone

Neil Young

unread,
Mar 20, 2019, 7:39:26 AM3/20/19
to discuss...@googlegroups.com
Works like a charm. Thanks again


Neil Young

unread,
Mar 20, 2019, 7:52:07 AM3/20/19
to discuss...@googlegroups.com
The only little drawback is: Audio disappears, once the screen capture stream is added...

Neil Young

unread,
Mar 20, 2019, 7:59:09 AM3/20/19
to discuss...@googlegroups.com
I think this is due to the fact, that the MediaStreamTrack from the screen recorder is "muted" and I'm placing it onto the same window, on which the remote video was placed before. That mutes the audio. I guess, if I would place it on a separate, new <video> element, it would be shown additionally (muted), but the original audio/video connection would not break

Let's try :)

Christoph Eggert

unread,
Mar 20, 2019, 8:21:46 AM3/20/19
to discuss-webrtc
The "muted" condition is actually an error case, the naming is just a little weird. So it usually means that you are not receiving the track anymore. Or is that just your wording? Could you take a look into the SDP during your renegotiation? Not quite sure how you are doing things, but I could imagine that you created a new audiotrack along with your screensharing that you added to the peer connection. When using Unified Plan, this would lead to a new ontrack firing containing the new audio track, the old transceiver with its track basically remains as a "corpse" and you'd have to switch to the new one. If it's just an organization issue on your frontend, keep in mind that you need to apply the old audio to the stream you are adding in that video element. 

Why are you replacing the content though? Is your actual target not to share an additional video, but replace the original with it? In that case, using replaceTrack would be much more straightforward and doesn't require you to negotiate etc.

Neil Young

unread,
Mar 20, 2019, 8:36:26 AM3/20/19
to discuss...@googlegroups.com

Roaming. Will answer later


Sent from my iPhone

Neil Young

unread,
Mar 20, 2019, 8:50:56 AM3/20/19
to discuss...@googlegroups.com
Yes. replacetrack is a great option. Will try that. The idea is this: I think if you share your screen you will concentrate on what you are doing. So you might not always show the most intelligent face. So why not hiding it? I was doing it at remote end by replacing the former a/v stream by the Video only stream from the capture. This explains the silence to me. 

The muted flag was property of the MediaTrack object received from the capturer

Sent from my iPhone

Neil Young

unread,
Mar 20, 2019, 9:00:16 AM3/20/19
to discuss...@googlegroups.com
Sure. Will send it later. Here

Sent from my iPhone

Neil Young

unread,
Mar 20, 2019, 12:03:32 PM3/20/19
to discuss...@googlegroups.com
OK, here is my code. I found the most of it here:



My current POC solution is establishing a video/audio call and after that I'm enabling a "Share" button, which acts like shown in function captureUserMedia.

this.pc is my active peer connection. this.signalingSendMessage is a function, which uses my signaling server in order to exchange SDP and ICE with remote

On remote I will receive an ontrack (here oPCTrack) event, which does simply just overwrite the currently displayed video in the one and only remote_video object.

This works but has the disadvantage, that audio gets lost after the remote screen capture is placed. I think I would rather need to have a second <video> element for displaying the screen capture from the source there or I would try to follow Christoph's suggestion to use replaceTrace instead of addTrack, which also would have the advantage of not opening too many streams.

Th

onPCTrack = (event) => {
console.log("Remote track(s) received")
event.streams[0].getTracks().forEach(track => {
console.log("Remote track", track)
})
if (this.remote_video.srcObject != event.streams[0]) {
this.remote_video.srcObject = event.streams[0]
console.log("Remote stream assigned to remote video display")
this.remote_video.onloadedmetadata = (e) => {
console.log("Remote video play")
this.remote_video.play().then(() => { console.log("Remote video playing") }).catch((e) => { console.log(e) })
}
}
}


showErrorMessage = (error, color) => {
console.log(error)
}

onGettingStream = async (stream) => {
stream.getTracks().forEach(track => {
console.log("Adding local shared screen track to peer", track)
this.pc.addTrack(track, stream)
})
await this.pc.setLocalDescription(await this.pc.createOffer({ offerToReceiveAudio: this.state.audio, offerToReceiveVideo: true }))
this.signalingSendMessage({ desc: this.pc.localDescription })
}

captureUserMedia = () => {
if (navigator.getDisplayMedia || navigator.mediaDevices.getDisplayMedia) {
if (navigator.mediaDevices.getDisplayMedia) {
navigator.mediaDevices.getDisplayMedia({ video: true }).then(stream => {
this.onGettingStream(stream, this.pc);
}, this.getDisplayMediaError).catch(this.getDisplayMediaError);
}
else if (navigator.getDisplayMedia) {
navigator.getDisplayMedia({ video: true }).then(stream => {
this.onGettingStream(stream, this.pc);
}, this.getDisplayMediaError).catch(this.getDisplayMediaError);
}
}
else {
if (DetectRTC.browser.name === 'Chrome') {
if (DetectRTC.browser.version == 71) {
this.showErrorMessage('Please enable "Experimental WebPlatform" flag via chrome://flags.');
} else if (DetectRTC.browser.version < 71) {
this.showErrorMessage('Please upgrade your Chrome browser.');
} else {
this.showErrorMessage('Please make sure that you are not using Chrome on iOS.');
}
}
if (DetectRTC.browser.name === 'Firefox') {
this.showErrorMessage('Please upgrade your Firefox browser.');
}
if (DetectRTC.browser.name === 'Edge') {
this.showErrorMessage('Please upgrade your Edge browser.');
}
if (DetectRTC.browser.name === 'Safari') {
this.showErrorMessage('Safari does NOT supports getDisplayMedia API yet.');
}
}
}


HTH. At least it should give you a good start

Neil Young

unread,
Mar 20, 2019, 12:45:15 PM3/20/19
to discuss-webrtc
OK, replacing works great.

onGettingStream = async (stream) => {
if (this.current_video_sender) {
this.current_video_sender.replaceTrack(stream.getVideoTracks()[0]).then(() => console.log("Current video track replaced")).catch((e) => console.log(e))
}
}

This is not production ready code (what happens, if this happens more than once), but at least it worked once and audio got not broken. No renegotiation necessary.

this.current_video_sender is setup on first add of the video track.

this.stream = await navigator.mediaDevices.getUserMedia({ audio: this.state.audio, video: true })
console.log("Got local stream", this.stream)
const videoTracks = this.stream.getVideoTracks()
if (videoTracks.length > 0) {
console.log(`Using video device: ${videoTracks[0].label}`)
}

const audioTracks = this.stream.getAudioTracks()
if (audioTracks.length > 0) {
console.log(`Using audio device: ${audioTracks[0].label}`)
}

this.stream.getTracks().forEach(track => {
console.log("Adding local track to peer", track)
let sender = this.pc.addTrack(track, this.stream)
if (videoTracks[0] === track) {
this.current_video_sender = sender
}
})

Nice. Maybe this can be programmed more elegantly, but this works for me.


To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrtc+unsubscribe@googlegroups.com.

Neil Young

unread,
Mar 20, 2019, 1:35:30 PM3/20/19
to discuss-webrtc
Finally, a fully working, but non-optimized sample: Making a call, hit "Share", select a thing to share, replace the outgoing video stream by the screen capture, and switch back to the original video stream on end of sharing


// Screen sharing
getDisplayMediaError = (error) => {
if (location.protocol === 'http:') {
this.showErrorMessage('Please test this WebRTC experiment on HTTPS.');
} else {
this.showErrorMessage(error.toString());
}
}

showErrorMessage = (error, color) => {
console.log(error)
}

onGettingStream = async (stream) => {
this.addStreamStopListener(stream, (e) => {console.log(`Mediastream ${e}`); this.current_video_sender.replaceTrack(this.stream.getVideoTracks()[0])})
if (this.current_video_sender) {
this.current_video_sender.replaceTrack(stream.getVideoTracks()[0]).then(() => console.log("Current video track replaced")).catch((e) => console.log(e))
}
}
addStreamStopListener = (stream, callback) => {
stream.addEventListener('ended', () => {
callback('ended')
callback = () => {}
}, false)
stream.addEventListener('inactive', function() {
callback('inactive')
callback = () => {}
}, false)
stream.getTracks().forEach((track) => {
track.addEventListener('ended', () => {
callback('ended')
callback = () => {}
}, false)
track.addEventListener('inactive', () => {
callback('inactive')
callback = () => {}
}, false)
test = () => {
this.captureUserMedia()
}

Christoph Eggert

unread,
Mar 20, 2019, 1:36:42 PM3/20/19
to discuss-webrtc
Good to hear! To optimize and probably solve your issue with the previous approach: It looks like you are just applying the stream of the new tracks you are getting -> this stream will not have the previous audio. So you would need to get your previous audioTrack (either by first getting the previous stream and attaching the audio to the new one, or by going through your peer connection's receivers and getting the track(s) from there). With the replaceTrack solution, also make sure that your further calls to getUserMedia are without audio, so that you can re-use your already established audio track rather than always creating a new one. That way your old audio track can just remain active and you won't have any hiccups due to the audio track switching along with the video one.

Good luck :)

OK, replacing works great.

Neil Young

unread,
Mar 20, 2019, 1:39:37 PM3/20/19
to discuss...@googlegroups.com
Not sure. As said: I'm starting the call with video (any case) and probably audio (per selection, but that doesn't matter). Then I share something and replace the outgoing video with the capture stream. If the sharing stops I revert this setting, so that the initial video appears again at remote side. This is exactly as intended and works great. 

Thanks


Reply all
Reply to author
Forward
0 new messages