Screen share mid call

[ost...@gmail.com]

I am trying to figure out how do send screen share and a video call at the same time.

The usecase is regular invite with video and audio, then mid call I want to send my screen share.

I am not quite sure how this works within the sipjs context, is it even possible?

You can send multiple streams, but how do I do it mid call, re-invite?

[Joseph Frazier]

Currently, SIP.js does not support re-INVITE, due to difficulties with media renegotiation. For the moment, I'd recommend sending both streams, but disabling the screen stream's video track (stream.getVideoTracks()[0].enabled = false) until you're ready to share it.

Alternatively, you can handle the logic at the application level by creating a new Session with the screen stream.

Let me know if I should clarify either of the above options,

Joseph

[ost...@gmail.com]

Hi and thanks for the reply.

If you could clarify the latter way it would be great.

[Joseph Frazier]

Sure, in Chrome (must be over https), you can obtain getUserMedia constraints for a screen stream using sip.desktopCapture.js in conjunction with this extension. Once you have the constraints, simply pass them into your UA's invite method as shown here.

Cheers,

Joseph

[Ph Wie]

If you want to have audio while sharing the screen you can do sth along those lines:

1. Create an audio-only stream

2. Get the screen stream (with extension see below)

3. Get the audioTrack from the audio-only stream and screen stream.addTrack it

4. Get ref to the peerConnection and removeStream the currently running stream

5. peerConnection.addStream the newly constructed stream

6. create offer and set local description.

I just took a quick look at some of my code, I hope this is not too confusing.

Also if you want to have an explanation for Screensharing Extension (new to chrome extension). I have an example app on:

https://github.com/wpp/ScreenStream

Cheers

[ost...@gmail.com]

So I am now finally gotten around to getting DTLS-SRTP <-> RTP bridging somewhat stable through rtpengine and I am ready to start working on this again.

I have started work on a javascript implementation of the bfcp protocol which I plan to contribute back into SIP.js eventually.

I have a much deeper understanding of the webrtc internals now and I have the following questions with how I should do things within SIP.js themselves (at least to get some pointers, so I dont waste time digging about blindly).

First of all, I am on node webkit, so I dont have to think about secure websockets and/or permissions etc so that part is OK, I can send a separate screen share just fine by just grabbing another stream from the outset and just not send anything over it (track disabled) when its not in use etc. And the fourth stream is a RTCDataChannel that I intend to send BFCP over for legacy interop.

Now, the use case I am after is for non-bfcp supported endpoints.

The normal way to do it is to try bfcp, and if you dont get a reply, just replace current video stream with either a PIP mux of your screenshare + your own camera or just replace the camera with the screen share.

Now, I am not quite sure if this is feasible. The way I have found so far is to just replace what streams are available on the peerConnection, but this will make a new SDP and force you to renegotiate (and technically re-invite I guess).

What I am trying to figure out, is if there is a way to in place replace the video stream.

Could I for example change the track of the webcam(video/audio) getUserMedia() to read from the track of the desktop share getUserMedia() call instead of reading from its local source?

Is it a better solution to from the outset make an intermediate stream that reads from local and writes to another local (which then would write to the stream that is added onto the peerConnection)?

Am I on the right track at all (no pun intended) by looking at swapping out tracks?

[Joseph Frazier]

> I have started work on a javascript implementation of the bfcp protocol which I plan to contribute back into SIP.js eventually. 

I'm not familiar with BFCP, but since it'll be running over a DataChannel, it sound like it'd fit quite well in a custom MediaHandler implementation (injected via the mediaHandlerFactory option)

> First of all, I am on node webkit, so I dont have to think about secure websockets and/or permissions etc so that part is OK

Ah, that makes things a lot easier :) FYI we recently merged some that allows SIP.js to be run that environment without first being browserified: https://github.com/onsip/SIP.js/commit/559fee5e1b7cbab6f85d21e2eaacce9ef91dcb41

> Is it a better solution to from the outset make an intermediate stream that reads from local and writes to another local (which then would write to the stream that is added onto the peerConnection)? 

I know this is possible for *audio*, using the Web Audio API, but I don't think there's currently a way to do it with video. However, I think Chromium's support for renegotiation may have improved to the point where you could use it instead, so I would recommend exploring that route first. Take a look at Github Issue 19, the hold/unhold/sendReinvite methods of the Session object, and the Session code that handles re-INVITEs for more information.

Cheers,

Joseph

[Oliver Severin Mulelid-Tynes]

I am already down that path, even with an identical SDP (one of our MCU system uses the INVITE-with-identical-SDP method to keep sessions refreshed) chrome freaks out (currently version 42 canary in the alpha of newest node-webkit) with the well known cryptical "cannot parse audio codecs" message if you try to set an SDP when there is already an active peerconnection going.

For whats it worth, I have already extended the current codebase so that if a mid-session re-INVITE comes in that has an identical SDP offer to what you already have set, it just calls getMediaDescription (which would normally happen in the setMediaDescription success closure) and sends that back in a 200OK.

So we are actually covered for most cases of re-INVITE that is not a new negotiation.

As for BFCP, the custom MediaHandler is the route I am already on, so that way is going smoothly :)

I am also aware of the browserify change, I am the one who reported the bug! 

I just tested it before I left work and it seems to work just fine, will comment in the issue about that tomorrow, appreciate the effort.

Anyhow, back to the kernel of my question.

What you basically say is that I have to generate a new SDP and ship that to go from video to screen share *if* I am not already streaming from both sources? (Or at least set them up as individual streams/tracks)

Sending a proper re-invite seems to me to require a new set of ice gathering (and completely different ports in general).

This will 'blip' the reciever end while it all gets set up again... or am I misunderstanding how it works in practice here?

Regards,

Oliver

--

Mvh,

Oliver Mulelid-Tynes

92089298

[Joseph Frazier]

> even with an identical SDP ... chrome freaks out

Hmm, I guess the WebRTC implementation is still very finicky when it comes to renegotiation :(

> I have already extended the current codebase so that if a mid-session re-INVITE comes in that has an identical SDP offer to what you already have set, it just calls getMediaDescription

Could you open a pull request with this change? It sounds like it would help address issue 18.

> I am also aware of the browserify change, I am the one who reported the bug!

> I just tested it before I left work and it seems to work just fine, will comment in the issue about that tomorrow, appreciate the effort.

Oh, I didn't realize you were @oteron! Glad to hear it's going well so far.

> What you basically say is that I have to generate a new SDP and ship that to go from video to screen share *if* I am not already streaming from both sources? (Or at least set them up as individual streams/tracks)

Yes, I'm not aware of a currently implemented way to programmatically modify/switch the video tracks sent over a PeerConnection without requiring renegotiation. However, I did find some w3c/webrtc/mozilla threads that indicate it may become possible in the future:

• replaceTrack proposal
  
• webrtc issue
• mozilla bug

In the mean time, one potential way to support renegotiation is to create a new PeerConnection for each offer/answer exchange. This solution would indeed be heavier (with respect to ICE/ports, as you point out) than proper renegotiation, but it should work (cross-browser, even). We at OnSIP have discussed (internally) implementing this in the WebRTC MediaHandler, but so far haven't had the time to do so. Personally, I'd like to see this implemented, so if you're interested, I'm willing to help out with the development.

Cheers,

Joseph

[ost...@gmail.com]

Hello and thank you for the reply.

I will make a pull request once we stabilize our code, its right now very ugly (very simple sdp check), will formalize with the sdp javascript npm package once we are sure its actually working as it should.

Could you elaborate more on the concept of multiple peer connections?
Wouldnt that mean that we would need multiple SDPs?

Or do you mean that if you get any sort of re-invite or anything that modifies the streams being sent, you would locally make a new peerconnection and then ship that off 'hidden' as a re-INVITE?

Then swap out the local streams to the new peerConnection when that is up and running?

On Thursday, February 26, 2015 at 9:37:20 PM UTC+1, Joseph Frazier wrote:
> > even with an identical SDP ... chrome freaks out
>
> Hmm, I guess the WebRTC implementation is still very finicky when it comes to renegotiation :(
>
>
> > I have already extended the current codebase so that if a mid-session re-INVITE comes in that has an identical SDP offer to what you already have set, it just calls getMediaDescription
>
>
> Could you open a pull request with this change? It sounds like it would help address issue 18.
>
>
> > I am also aware of the browserify change, I am the one who reported the bug!
> > I just tested it before I left work and it seems to work just fine, will comment in the issue about that tomorrow, appreciate the effort.
>
>
> Oh, I didn't realize you were @oteron! Glad to hear it's going well so far.
>
>
> > What you basically say is that I have to generate a new SDP and ship that to go from video to screen share *if* I am not already streaming from both sources? (Or at least set them up as individual streams/tracks)
>
>
> Yes, I'm not aware of a currently implemented way to programmatically modify/switch the video tracks sent over a PeerConnection without requiring renegotiation. However, I did find some w3c/webrtc/mozilla threads that indicate it may become possible in the future:
> replaceTrack proposal

> webrtc issuemozilla bug

[Joseph Frazier]

> I will make a pull request once we stabilize our code

Which SDP module are you referring to? There's currently no dependencies on SDP-related modules, so it may currently be preferable to simply check the string directly.

> Or do you mean that if you get any sort of re-invite or anything that modifies the streams being sent, you would locally make a new peerconnection and then ship that off 'hidden' as a re-INVITE? 

Yes, that's what I meant. The endpoints don't know (nor do they need to know) whether each other is using the same PeerConnection for a new offer/answer exchange, since all they see is the remote SDP.

> Then swap out the local streams to the new peerConnection when that is up and running? 

This is the part that might be tricky, trying to minimize any gap in media.

I briefly started on creating a PeerConnection abstraction that could seamlessly change the concrete PeerConnection behind the scenes, but I haven't gotten too far yet (in particular, I don't know if there's an easy way to make all of the stream/ICE-related events work correctly). If you're interested, it's on Github: https://github.com/joseph-onsip/peer-reconnection

Joseph

[Ph Wie]

If you have a SIP session (with audio and video)

and want to switch mid-call isn't something like:

https://gist.github.com/wpp/8227254237b3bc09ccaa

enough for achieving what you're after? There is no need for a sip-level re-invite if you

send the offer and answer SDP's via SIP messages and update the peerConnections

via setLocal/remoteDescription respectively.

Hope I'm not missing something here.

Cheers

[tcu...@icancerhelp.com]

How far are you with your BFCP translation? Out of curiosity are you using BFCP on mobile clients? Thanks in Advance.

[okai...@gmail.com]

Hey , i know this is an old post , but i really need to know if u solved this problem pls ?

and if yes , can u share some code of how u did it ?

so far i got the screen sharing (stream) , but i don't know how to send it to the other caller.

i tried to add the stream to (pc) , but nothing happenend on the other side the (onTrackAdded) never triggered neither the (onStreamAdded)

maybe i'm doing something wron 

please help , 

this is my code

var constraints = {

     media: {

         constraints: {

             audio: true,

             video: {

                 mandatory: {

                     chromeMediaSource: 'desktop',

                     // chromeMediaSourceId: event.data.sourceId,

                     maxWidth: window.screen.width > 1920 ? window.screen.width : 1920,

                     maxHeight: window.screen.height > 1080 ? window.screen.height : 1080

                 },

                 optional: []

             },

             render: {

                 remote: remoteVideo,

                 local: localVideo

             }

         }

     }

   }

    navigator.mediaDevices.getDisplayMedia(constraints)

   .then(function(stream) {

     //We've got media stream

     console.log("----------then-------------");

     console.log(stream);

     stream.getTracks().forEach(function(track) {

       pc.addTrack(track, stream);

     });    

      //creating offer

     pc.createOffer().then(function(offer) {

       console.log('-------------offer----------------');

       console.log(offer);

       return pc.setLocalDescription(offer);

     })

     .then(function() {

     })

     .catch(function(reason) {

       // An error occurred, so handle the failure to connect

       console.log('--------------createOffer Error-----------------');

       console.log(reason);

     });

    })

   .catch(function(error) {

     console.log("----------catch-------------");

     console.log(error);

   });

[clement pradeep]

Hey,

 

Did you get any solution for this screen sharing in mid-call??

I'm also looking for the same solution...

Please let me know if you found any solution.

Thanks much in advance

 

[okai...@gmail.com]

yes i found it , this worked for me , hope it will help you

to start screen share:

var option = {video: {mediaSource: 'screen'}, audio: true};

navigator.mediaDevices.getDisplayMedia(option)

.then(function(streams){

 var videoTrack = streams.getVideoTracks()[0];

 var sender = pc.getSenders().find(function(s) {   // pc refers to => session.sessionDescriptionHandler.peerConnection;

   return s.track.kind == videoTrack.kind;

 });

  console.log('found sender:', sender);

 sender.replaceTrack(videoTrack);

}, function(error){

 console.log("error ", error);

});

to stop screen share again coming back to old video:

var option = {video: true, audio: true};

navigator.mediaDevices.getUserMedia(option)

.then(function(streams){

 var videoTrack = streams.getVideoTracks()[0];

 var sender = pc.getSenders().find(function(s) {  // pc refers to => session.sessionDescriptionHandler.peerConnection;

   return s.track.kind == videoTrack.kind;

 });

  console.log('found sender:', sender);

 sender.replaceTrack(videoTrack);

}, function(error){

 console.log("error ", error);

});

[Shashi Kumar]

I am getting Uncaught (in promise) ReferenceError: pc is not defined