Call between audio-only and audio+video not working properly

[Y Du]

Hi there,

I am experimenting with WebRTC calls between two browsers where one side calls getUserMedia({"audio": true, "video": true}, ...), while the other side calls getUserMedia({"audio": true, "video": false}, ...). No matter which side is the caller, I always observe the behavior where the side sending both Audio and Video could not play the audio from the other side sending Audio only.

I feel it could be related to how I pass the constraints to createOffer and createAnswer. Here is what I have tested:

---------------

Case 1:

Caller:

Audio+Video getUserMedia({"audio": true, "video": true}, ...)

createOffer(..., {"mandatory":{"OfferToReceiveAudio":true,"OfferToReceiveVideo":true}})

Callee:

Audio-only getUserMedia({"audio": true, "video": false}, ...)

createAnswer(..., {"mandatory":{"OfferToReceiveAudio":true,"OfferToReceiveVideo":true}})

Results:

            Expected        Actual

Caller      A               nothing

Callee      A+V             A+V

---------------

Case 2:

The only difference from Case 1 is caller side createOffer setting OfferToReceiveVideo to false. I don't think it should be set this way, but just want to try if the caller knows the callee can provide audio only, whether the result would be the same if caller sets OfferToReceiveVideo to false.

Results: same as Case 1.

---------------

Case 3:

Caller:

Audio-only getUserMedia({"audio": true, "video": false}, ...)

createOffer(..., {"mandatory":{"OfferToReceiveAudio":true,"OfferToReceiveVideo":true}})

Callee:

Audio+Video getUserMedia({"audio": true, "video": true}, ...)

createAnswer(..., {"mandatory":{"OfferToReceiveAudio":true,"OfferToReceiveVideo":true}})

Results:

            Expected        Actual

Caller      A+V             A+V

Callee      A               nothing

---------------

Case 4:

The only difference from Case 3 is callee side createAnswer setting OfferToReceiveVideo to false. This is based on the same thinking: assume the callee knows the caller can provide audio only, whether the result would be the same if callee sets OfferToReceiveVideo to false.

Results: same as Case 3.

---------------

I searched the group and from this thread https://groups.google.com/forum/#!searchin/discuss-webrtc/video$20only/discuss-webrtc/Ox1fbPP2u2g/m4rngB6okiQJ, I know this should work.

I would also like to get some clarification regarding how the SDP direction attribute is determined by Chrome. I assume internally createOffer and createAnswer determine this attribute based on some union operation on getUserMedia constraints and OfferToReceiveAudio/OfferToReceiveVideo constraints? What confused me is, for example, in Case 2, the Caller  side produced SDP OFFER where a=sendrecv for m=video, even I specified createOffer(..., {"mandatory":{"OfferToReceiveAudio":true,"OfferToReceiveVideo":false}}); I was expecting a=sendonly in this case.

Could anyone please provide some advice?

Chrome version: Version 28.0.1500.72 m on Windows 7 Professional SP1. We have a media server in between the two browsers.

[Vikas]

Hi,

Thanks for the feedback, i tested with apprtc demo but could not recreate it on Version 29.0.1547.22 beta-m.  You can test audio only with apprtc demo by appending url argument 'media = audio' & if you don't append any media argument then it uses audio & video both. Regarding your question on offerToReceiveVideo & OfferToReceiveAudio constraint i think you need to set it for the side that does not have a mediaStream to send. In your case the side getting only audio from GUM does not have a video in the mediaStream but it would like to receive video, so it should set offerToReceiveVideo to true to tell the remote end that hey i am not sending any video but i would still like to receive video. 

/Vikas

[Vikas]

Hi,

Regarding your question of direction attribute. According to RFC 3264 section 6.1 : 

   " If a stream is offered as sendonly, the corresponding stream MUST be
   marked as recvonly or inactive in the answer.  If a media stream is
   listed as recvonly in the offer, the answer MUST be marked as
   sendonly or inactive in the answer.  If an offered media stream is
   listed as sendrecv (or if there is no direction attribute at the
   media or session level, in which case the stream is sendrecv by
   default), the corresponding stream in the answer MAY be marked as
   sendonly, recvonly, sendrecv, or inactive.  If an offered media
   stream is listed as inactive, it MUST be marked as inactive in the
   answer."

I think issue 1553 relates to what you observing.

/Vikas

On Thursday, July 18, 2013 8:09:38 AM UTC-7, Y Du wrote:

[Y Du]

Thanks for your comments Vikas.

I think issue 1553 relates to CASE 2 I listed above, and I also think that's something to be fixed in Chrome. But what I am not quite sure is whether its fix could resolve the problem I encountered.

Let's focus on CASE 1 in our discussion, and let me add SDP for your reference.

Caller:

getUserMedia(audio=true, video=true)

createOffer(OfferToReceiveAudio=true, OfferToReceiveVideo=true)

OFFER SDP produced by caller Chrome:

v=0^M

o=- 495832659917377818 2 IN IP4 127.0.0.1^M

s=-^M

t=0 0^M

a=group:BUNDLE audio video^M

a=msid-semantic: WMS 0w5lVdODRA5bVSLP6W1JQfBUmpngg8MBtyJB^M

m=audio 1 RTP/SAVPF 111 103 104 0 8 107 106 105 13 126^M

c=IN IP4 0.0.0.0^M

a=rtcp:1 IN IP4 0.0.0.0^M

a=ice-ufrag:C3O74GrNsgeoTmru^M

a=ice-pwd:UpoiV2Wg1CC8RABe838STKIT^M

a=ice-options:google-ice^M

a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level^M

a=sendrecv^M

a=mid:audio^M

a=rtcp-mux^M

a=crypto:1 AES_CM_128_HMAC_SHA1_80 inline:RamCEzFaBkcvYIKCN9zT8++V03L3OzUCpcmAxp/t^M

a=rtpmap:111 opus/48000/2^M

a=fmtp:111 minptime=10^M

a=rtpmap:103 ISAC/16000^M

a=rtpmap:104 ISAC/32000^M

a=rtpmap:0 PCMU/8000^M

a=rtpmap:8 PCMA/8000^M

a=rtpmap:107 CN/48000^M

a=rtpmap:106 CN/32000^M

a=rtpmap:105 CN/16000^M

a=rtpmap:13 CN/8000^M

a=rtpmap:126 telephone-event/8000^M

a=maxptime:60^M

a=ssrc:2746454444 cname:8X8HrG/y3RaBRoi+^M

a=ssrc:2746454444 msid:0w5lVdODRA5bVSLP6W1JQfBUmpngg8MBtyJB 0w5lVdODRA5bVSLP6W1JQfBUmpngg8MBtyJBa0^M

a=ssrc:2746454444 mslabel:0w5lVdODRA5bVSLP6W1JQfBUmpngg8MBtyJB^M

a=ssrc:2746454444 label:0w5lVdODRA5bVSLP6W1JQfBUmpngg8MBtyJBa0^M

m=video 1 RTP/SAVPF 100 116 117^M

c=IN IP4 0.0.0.0^M

a=rtcp:1 IN IP4 0.0.0.0^M

a=ice-ufrag:C3O74GrNsgeoTmru^M

a=ice-pwd:UpoiV2Wg1CC8RABe838STKIT^M

a=ice-options:google-ice^M

a=extmap:2 urn:ietf:params:rtp-hdrext:toffset^M

a=sendrecv^M

a=mid:video^M

a=rtcp-mux^M

a=crypto:1 AES_CM_128_HMAC_SHA1_80 inline:RamCEzFaBkcvYIKCN9zT8++V03L3OzUCpcmAxp/t^M

a=rtpmap:100 VP8/90000^M

a=rtcp-fb:100 ccm fir^M

a=rtcp-fb:100 nack ^M

a=rtcp-fb:100 goog-remb ^M

a=rtpmap:116 red/90000^M

a=rtpmap:117 ulpfec/90000^M

a=ssrc:107552631 cname:8X8HrG/y3RaBRoi+^M

a=ssrc:107552631 msid:0w5lVdODRA5bVSLP6W1JQfBUmpngg8MBtyJB 0w5lVdODRA5bVSLP6W1JQfBUmpngg8MBtyJBv0^M

a=ssrc:107552631 mslabel:0w5lVdODRA5bVSLP6W1JQfBUmpngg8MBtyJB^M

a=ssrc:107552631 label:0w5lVdODRA5bVSLP6W1JQfBUmpngg8MBtyJBv0^M

[Y Du]

The OFFER generated by the Caller Chrome looks fine, with a=sendrecv for both m=audio and m=video.

The OFFER SDP passed through a media server, and became this when it's received by the Callee side Chrome. Note the a=sendrecv lines were stripped off by the media server. This should be ok based on the standard, since sendrecv is the default. I think Chrome honours this default setting as well.

handlePeerMessage(8801, message: SDP { "messageType": "OFFER", "offererSessionId": "EuUwAHEm6BG91xmwEYeqd38A", "sdp": "v=0\r\no=Genesys 3 2 IN IP4 135.17.176.25\r\ns=WEBRTCGW-8.5.001.57\r\nc=IN IP4 135.17.176.25\r\nt=0 0\r\na=group:BUNDLE audio video\r\nm=audio 1 RTP\/SAVPF 0 8\r\na=rtcp-mux\r\na=mid:audio\r\na=ssrc:2137824184 cname:ECgxAHEm6BG91xmwEYeqd5DQ\r\na=ssrc:2137824184 label:0030E1DE-2671-11E8-BDD7-19B01187AA77a0\r\na=ssrc:2137824184 msid:0030E1DE-2671-11E8-BDD7-19B01187AA77 a0\r\na=ssrc:2137824184 mslabel:0030E1DE-2671-11E8-BDD7-19B01187AA77\r\na=ice-ufrag:WR7J\r\na=ice-pwd:KCW6lcVU54ssxWka+xywgf\r\na=rtpmap:0 pcmu\/8000\r\na=rtpmap:8 pcma\/8000\r\na=crypto:1 AES_CM_128_HMAC_SHA1_80 inline:ZLpn1wmD\/1hhQWPvquGyCL5x6CsrxGJuWSSLSlOP\r\nm=video 1 RTP\/SAVPF 100 106\r\na=rtcp-mux\r\na=mid:video\r\na=ssrc:135173254 cname:ECgxAHEm6BG91xmwEYeqd5DQ\r\na=ssrc:135173254 label:0030E1DE-2671-11E8-BDD7-19B01187AA77v0\r\na=ssrc:135173254 msid:0030E1DE-2671-11E8-BDD7-19B01187AA77 v0\r\na=ssrc:135173254 mslabel:0030E1DE-2671-11E8-BDD7-19B01187AA77\r\na=ice-ufrag:WR7J\r\na=ice-pwd:KCW6lcVU54ssxWka+xywgf\r\na=rtpmap:100 vp8\/90000\r\na=rtpmap:106 h264\/90000\r\na=fmtp:106 profile-level-id=42000B;packetization-mode=1\r\na=crypto:1 AES_CM_128_HMAC_SHA1_80 inline:ZLpn1wmD\/1hhQWPvquGyCL5x6CsrxGJuWSSLSlOP\r\n", "Candidates": [ { "sdpMLineIndex": 0, "sdpMid": "audio", "candidate": "a=candidate:4 1 udp 2013266431 192.168.122.1 36001 typ host" }, { "sdpMLineIndex": 1, "sdpMid": "video", "candidate": "a=candidate:4 1 udp 2013266431 192.168.122.1 36001 typ host" } ], "seq": 1, "tieBreaker": 92804743, "attacheddata": [ { "key": "CallUUID", "value": "0060LCGN1K8UH1PM04000VTAES000001" } ] } )

[Y Du]

Now the Callee side needs to generate ANSWER:

Callee:

getUserMedia(audio=true, video=false)

createOffer(OfferToReceiveAudio=true, OfferToReceiveVideo=true)

ANSWER SDP produced by caller Chrome: note a=recvonly for m=video, which is expected.

v=0^M

o=- 7375935203262638079 2 IN IP4 127.0.0.1^M

s=-^M

t=0 0^M

a=group:BUNDLE audio video^M

a=msid-semantic: WMS 3AEbXqa32cOgQqZ99cHDfhjtbg11ynstItwl^M

m=audio 1 RTP/SAVPF 0 8^M

c=IN IP4 0.0.0.0^M

a=rtcp:1 IN IP4 0.0.0.0^M

a=ice-ufrag:ZkRpHk4YLYWj54CF^M

a=ice-pwd:inW6yvD6Xu9NG2LXoh32uJWk^M

a=sendrecv^M

a=mid:audio^M

a=rtcp-mux^M

a=crypto:1 AES_CM_128_HMAC_SHA1_80 inline:CuBLWak/DgDu70AElXESfgPB1MWsGPQ6VIZpAK3R^M

a=rtpmap:0 PCMU/8000^M

a=rtpmap:8 PCMA/8000^M

a=ssrc:1579289828 cname:mCTjiLz0U2f+F9VY^M

a=ssrc:1579289828 msid:3AEbXqa32cOgQqZ99cHDfhjtbg11ynstItwl 3AEbXqa32cOgQqZ99cHDfhjtbg11ynstItwla0^M

a=ssrc:1579289828 mslabel:3AEbXqa32cOgQqZ99cHDfhjtbg11ynstItwl^M

a=ssrc:1579289828 label:3AEbXqa32cOgQqZ99cHDfhjtbg11ynstItwla0^M

m=video 1 RTP/SAVPF 100^M

c=IN IP4 0.0.0.0^M

a=rtcp:1 IN IP4 0.0.0.0^M

a=ice-ufrag:ZkRpHk4YLYWj54CF^M

a=ice-pwd:inW6yvD6Xu9NG2LXoh32uJWk^M

a=recvonly^M

a=mid:video^M

a=rtcp-mux^M

a=crypto:1 AES_CM_128_HMAC_SHA1_80 inline:CuBLWak/DgDu70AElXESfgPB1MWsGPQ6VIZpAK3R^M

a=rtpmap:100 VP8/90000^M

[Y Du]

Finally, the ANSWER SDP passed through the media server and arrived at the Caller Chrome: note the direction attribute is not stripped off in this case (because the value is not default for m=video).

handlePeerMessage(null, message: SDP { "messageType": "ANSWER", "offererSessionId": "102", "answererSessionId": "XAEvAHEm6BG91xmwEYeqd5VD", "sdp": "v=0\r\no=Genesys 7 2 IN IP4 135.17.176.25\r\ns=WEBRTCGW-8.5.001.57\r\nc=IN IP4 135.17.176.25\r\nt=0 0\r\na=group:BUNDLE audio video\r\nm=audio 1 RTP\/SAVPF 0 126\r\na=sendrecv\r\na=rtcp-mux\r\na=mid:audio\r\na=ssrc:1198323396 cname:sjtJAHUm6BG91xmwEYeqd27\/\r\na=ssrc:1198323396 label:002EF57C-2671-11E8-BDD7-19B01187AA77a0\r\na=ssrc:1198323396 msid:002EF57C-2671-11E8-BDD7-19B01187AA77 a0\r\na=ssrc:1198323396 mslabel:002EF57C-2671-11E8-BDD7-19B01187AA77\r\na=ice-ufrag:TGQt\r\na=ice-pwd:VcIqdZoXGb+V9O1uAYhngm\r\na=rtpmap:0 pcmu\/8000\r\na=rtpmap:126 telephone-event\/8000\r\na=fmtp:126 0-15\r\na=crypto:1 AES_CM_128_HMAC_SHA1_80 inline:Aw\/Q+fGzelg1WBQCexjoe3HpgLoVnaYDQXOm4U9c\r\nm=video 1 RTP\/SAVPF 100\r\na=recvonly\r\na=rtcp-mux\r\na=mid:video\r\na=ssrc:776151662 cname:sjtJAHUm6BG91xmwEYeqd27\/\r\na=ssrc:776151662 label:002EF57C-2671-11E8-BDD7-19B01187AA77v0\r\na=ssrc:776151662 msid:002EF57C-2671-11E8-BDD7-19B01187AA77 v0\r\na=ssrc:776151662 mslabel:002EF57C-2671-11E8-BDD7-19B01187AA77\r\na=ice-ufrag:TGQt\r\na=ice-pwd:VcIqdZoXGb+V9O1uAYhngm\r\na=rtpmap:100 vp8\/90000\r\na=crypto:1 AES_CM_128_HMAC_SHA1_80 inline:Aw\/Q+fGzelg1WBQCexjoe3HpgLoVnaYDQXOm4U9c\r\n", "Candidates": [ { "sdpMLineIndex": 0, "sdpMid": "audio", "candidate": "a=candidate:2 1 udp 2013266431 192.168.122.1 36000 typ host" }, { "sdpMLineIndex": 1, "sdpMid": "video", "candidate": "a=candidate:2 1 udp 2013266431 192.168.122.1 36000 typ host" } ], "seq": 1, "tieBreaker": 173160200 } )

[Y Du]

The direction attributes look fine in my eyes. One thing to clarify is: when the direction is missing from SDP OFFER, will Chrome honour a=sendrecv as default for sure?

[Y Du]

Another thinking: could it be possible that for CASE 1, once createOffer is called, the Caller side Chrome is determined to thinks it's going to receive both audio and video; so even later when it gets a=recvonly for m=video in ANSWER SDP, the Caller side Chrome still thinks it's going to receive both audio and video; so even the audio RTP has arrived, the Caller side Chrome is not playing it since it's waiting for the video stream to sync, which is never happening since the Callee is not sending video stream.

[Y Du]

In issue https://code.google.com/p/chromium/issues/detail?id=239067 (), Brave Yao pointed out that the FIR/NACK/REMB/RED/FEC lines are missing from the SDP, which might be causing the delayed video problem reported there. Just wonder if it might have anything to do with this issue or not?

[Vikas]

Hi,

AFAIK, the default direction attribute is sendrecv but if there is no mediastream it will be changed to recvonly. You can file an issue and add these details. Also if you can share a test application to recreate that would be useful. In your case you get no audio, so i don't think the FIR issue you pointed to relates to you.

/Vikas 

[Eric Green]

I have come across the same issue. I have determined that if I remove the video track manually this allows the audio to flow (this may cause the browser to crash ending the stream). Maybe parsing the SDP manually then removing the video track based on that information is a workaround for now (in js).

[Vikas]

Eric,

Can you share your test code that re-creates the crash?

/Vikas

[Eric Green]

I am using JsSip. in session.on('started', function() {

...

session.getRemoteStreams()[0].removeTrack(session.getRemoteStreams()[0].getVideoTracks()[0]);   //this causes chrome to crash

...

//attach to html video element

});

Adding this one line of code (currently hard coded before attaching to the html video element) allows a call that was sent out as audio/video and answered as audio-only 2-way audio. Without the line there is only 1-way audio. This does not appear to be an issue in Firefox or JsSip. Adding this line of code that removes the video track, causes chrome to crash when the call is ended. I have attached a picture of my crash. I am using Chrome Version 28.0.1500.71 on OS X 10.8.4. All latest updates. I have not tried to recreate this in Canary yet.

[Vikas]

Thanks for the feedback, if you open chrome://crashes do you see any crash id. Can you share that?

/Vikas

[Vikas]

Thanks i was able to reproduce the crash on M29 with your change. I have filed a bug here.

/Vikas

[Dmitry Sobinov]

Hi

I've also seen this issue. Remote audio is not played if video track is set to not send data (a=recvonly or a=inactive).

I managed to reproduce "no audio" issue with apprtc. See https://code.google.com/p/webrtc/issues/detail?id=2143

---
Dmitry Sobinov
AddLive.com
Live video and voice for your application

[Vikas]

Thanks, i will take a look at issue 2143.

/Vikas

[gugaiz]

Hi, is there any news about this issue?, I am having a similar problem as described below

Caller:

constraints = {video:false, audio:true};

sdpConstraints = {'mandatory': {'OfferToReceiveAudio':true,'OfferToReceiveVideo':true }};

Callee:

constraints = {video:true, audio:true};

sdpConstraints = {'mandatory': {'OfferToReceiveAudio':true,'OfferToReceiveVideo':true }};

Results:

              Expected     Actual

Caller      A+V            A

Callee     A                 A

The problem is that the caller side is not getting the video from the callee. Looking at the sdp information, when the caller send only audio information, the callee also respond with an only audio package. If I set the caller to send audio+video I get video and audio in both sides but I don't want to ask for video permission on the caller side. 

Chrome version: 31.0.1650.63 (Linux)

-- 

gugaiz

[Vikas]

Hi,

 Do you see the same problem with latest canary also? 

/Vikas

[gugaiz]

Hi,

Yes, it is the same problem with the latest chrome canary (on windows).

-- 

gugaiz

[Vikas]

Thanks, can you share a test page to recreate the problem?

/Vikas

[Gustavo Izquierdo]

While I try to do a web page to send you for testing, I would like to comment that I managed to make it work manipulating the sdp information, adding a fake video information (on the caller side), parsing the values (crypto,msid-semantic,ice-ufrag,ice-pwd,cname) and using them accordingly. The callee believes that will receive video and he reply with the video information also. Without adding the fake video information, when the caller send a only audio spd package, the callee always replay with an audio sdp package (without adding his video information, even he should do it). While this is working when the test is done on the same computer, it does not work when the communication is done between two different computer (on LAN). 

The problem I am seeing, (when working with two computers) is that, on the caller side, the latest ICE candidate event (the null one) never happen and it looks like the flow stop there, then, when I close the peer-connection manually, the onicecandidate event is trigger with the null candidate on it. On the callee side the work flow is normal with the remote and local stream added.

-- 

gugaiz

--
 
---
You received this message because you are subscribed to a topic in the Google Groups "discuss-webrtc" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/discuss-webrtc/PG92vqeY1qQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to discuss-webrt...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Gustavo]

Test pages have been sent privately....

On Fri, Dec 27, 2013 at 8:59 PM, Vikas <vikasm...@webrtc.org> wrote:

[Vikas]

Thanks i will give them a try.

/Vikas

[Uma Shunmugan]

Hi Vikas, is there any update on the original issue?

 

BTW, the issue gugaiz was having seems different, where video was missing on one side (I'm not sure, however, why the behavior was different in this case).  The original issue was that in a similar asymmetric video call, the "problem" side with both audio and video in local media stream, which received only audio, did not play the audio (as if it was waiting for the first video frame).  I tested this a few days ago with the stable Chrome version 33.0.1750.154 m, as well as the Canary version 36.0.1929.0, and the problem was still there, regardless of the fact which side made the offer.  The webrtc-internals stats graphs showed that audio was being received without any issues.  Note that the SDP received by the "problem" side (in both offer and answer scenarios) had "a=recvonly" set for video m-line correctly. 

Note:

The issue 2143 seems relevant, though not exactly the same.  #2143 has been merged with 2162, which is claimed to be fixed.

 Thanks,

  Uma 

[Uma Shunmugan]

FYI. I've found a workaround/solution for this issue by removing the a=ssrc attributes from an m-line of an SDP (in the gateway) when a=recvonly.  I still believe that Chrome shouldn't really care about the ssrc attributes, and should only look at media direction to decide how it should behave in this case.  You may also want to look at the webrtc issue #2143.

Thanks,

 Uma