Any reasons why remote-video track is not yielding any frames?
1,457 views
Skip to first unread message
sebastian pulido
Mar 28, 2019, 2:44:12 AM3/28/19
to discuss-webrtc
We are developing an Android Application with Scala, WebRTC and the Java SDK. Our current task consists of receiving the remote track that is sent in a second SDP negotiation when the Callee enables video. Thus the Caller is accessing that video track in the `onAddTrack` method of the peer-connection observer. In fact, the track is contained in the Array of MediaStreams, and then we are able to attach it to the VideoRenderer via a ProxySink - pretty much like in this example: https://webrtc.googlesource.com/src/+/refs/heads/master/examples/androidapp/src/org/appspot/apprtc/CallActivity.java#133.
This is the piece of code in charge of that:
override def onAddTrack(rtpReceiver: RtpReceiver, mediaStreams: Array[MediaStream]): Unit = {
info("FLOW: onAddTrack")
rtpReceiver.SetObserver(rtpObserver)
call.getCallType match {
case BothVideo(CallRoles.Caller) | RecvPeerVideo(CallRoles.Caller) => {
if (mediaStreams.size > 0) {
val mediaStream = mediaStreams(0)
val videoTracks: Buffer[VideoTrack] = mediaStream.videoTracks.asScala
this.remoteMediaStream = Some(mediaStream)
info(s"FLOW: onAddTrack, call-type: ${call.getCallType}")
if (call.callerShouldReceiveVideo(call.getCallType) && videoTracks.size > 0) {
info(s"FLOW: onAddTrack, enabling video-track")
val videoTrack: VideoTrack = videoTracks(0)
mediaStream.addPreservedTrack(videoTrack)
videoTrack.setEnabled(true)
videoTrack.addSink(flowManager.remoteProxySink)
...
}
}
}
The problem is that we are not able to get the frames from the Callee's remote track, and this is a bit surprising since we were already able to successfully set the Caller's remote track and render the frames when the Caller initiates the call with video and has to send the remote track to Callee.
So do you know about any general issues that might be preventing us from getting the frames from the Callee's remote track?
The version of webrtc we are using is the following: google-webrtc/1.0.26885
sebastian pulido
Mar 28, 2019, 1:23:24 PM3/28/19
to discuss-webrtc
I have more information about the issue. After some analysis about our application's thread, we found that for some reason there is a thread calling `dispose` on the remote videoTrack:
03-28 12:14:18.518 4506-4682/com.protime E/AndroidRuntime: FATAL EXCEPTION: protime-stats-debug-thread
Process: com.protime, PID: 4506
java.lang.IllegalStateException: MediaStreamTrack has been disposed.
at org.webrtc.MediaStreamTrack.checkMediaStreamTrackExists(MediaStreamTrack.java:120)
at org.webrtc.MediaStreamTrack.getNativeMediaStreamTrack(MediaStreamTrack.java:114)
at org.webrtc.PeerConnection.getStats(PeerConnection.java:1094)
at com.paz.service.call.webrtc.PeerConnObserver$$anon$1.run(PeerConnectionObserver.scala:130)
at java.lang.Thread.run(Thread.java:818)
We are not explicitly calling that method anywhere. How is it possible that some other thread is disposing of our track?sebastian pulido
Mar 28, 2019, 2:35:09 PM3/28/19
to discuss-webrtc
One more clue: We have detected that dispose is being called on the MediaStream, https://webrtc.googlesource.com/src/+/refs/heads/master/sdk/android/api/org/webrtc/MediaStream.java#75
The MediaStream, when disposed, iterates over the list of videoTracks calling dispose on them.
What might be calling this dipose? Any reasons why that method is called on the MediaStream?
The MediaStream, when disposed, iterates over the list of videoTracks calling dispose on them.
What might be calling this dipose? Any reasons why that method is called on the MediaStream?
Btw, we ourselves are not explicitly calling that method.
On Thursday, 28 March 2019 01:44:12 UTC-5, sebastian pulido wrote:
sebastian pulido
Mar 28, 2019, 6:45:52 PM3/28/19
to discuss-webrtc
One more clue: The onAddTrack event is genreated by the `setRemoteDescription` function. Right after `setRemoteDescription` we are Calling `createAnswer` on the Caller side, since this one received the offer with video stram from the Callee. createAnswer triggers and event of type `onSetSuccess` where we are calling `setLocalDescription`. We have discovered that this call to `setLocalDescription` is disposing of the MediaStream and the video track:
Do you know any reasons why setLocalDescription is disposing of the MediaStream and videoTrack?
On Thursday, 28 March 2019 01:44:12 UTC-5, sebastian pulido wrote:
sebastian pulido
Mar 28, 2019, 8:55:22 PM3/28/19
to discuss-webrtc
I have more information.
Analyzing the sdp that is triggered when the Caller creates the ANSWER I have found that the video-track's direction is "sendonly", which seems to be wrong since we are supposed to be receiving video from the Callee, and also send our local track to the Callee:
03-28 19:43:36.495 19693-19839/com.protime I/SdpObserver: FLOW: onCreateSuccess, CALLER ANSWER v=0
o=- 2191449863069826960 3 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1 2 3
a=msid-semantic: WMS stream0
m=audio 56825 UDP/TLS/RTP/SAVPF 111 103 9 102 0 8 105 13 110 113 126
c=IN IP4 35.167.127.194
a=rtcp:9 IN IP4 0.0.0.0
a=candidate:2294452033 1 udp 2122260223 192.168.2.159 60968 typ host generation 0 network-id 3 network-cost 10
a=candidate:559267639 1 udp 2122202367 ::1 39373 typ host generation 0 network-id 2
a=candidate:1510613869 1 udp 2122129151 127.0.0.1 56438 typ host generation 0 network-id 1
a=candidate:1876313031 1 tcp 1518222591 ::1 59473 typ host tcptype passive generation 0 network-id 2
a=candidate:344579997 1 tcp 1518149375 127.0.0.1 47922 typ host tcptype passive generation 0 network-id 1
a=candidate:842163049 1 udp 1686052607 201.233.49.70 60968 typ srflx raddr 192.168.2.159 rport 60968 generation 0 network-id 3 network-cost 10
a=candidate:1600295039 1 udp 41885439 35.167.127.194 56825 typ relay raddr 201.233.49.70 rport 60968 generation 0 network-id 3 network-cost 10
a=candidate:3811514541 1 udp 41885695 34.221.141.99 60923 typ relay raddr 201.233.49.70 rport 60968 generation 0 network-id 3 network-cost 10
a=ice-ufrag:GGkb
a=ice-pwd:owqpzSht95NQb92cgzKNkQmJ
a=ice-options:trickle renomination
a=fingerprint:sha-256 45:76:5A:81:C1:CD:6B:C8:17:C6:CB:7B:20:CD:C6:DD:59:A3:DC:11:47:78:D3:E1:E1:EE:55:1F:83:A5:32:E4
a=setup:passive
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:3 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
a=extmap:5 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id/
a=sendrecv
a=msid:stream0 audio0
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:103 ISAC/16000
a=rtpmap:9 G722/8000
a=rtpmap:102 ILBC/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:105 CN/16000
a=rtpmap:13 CN/8000
a=rtpmap:110 telephone-event/48000
a=rtpmap:113 telephone-event/16000
a=rtpmap:126 telephone-event/8000
a=ssrc:1249625177 cname:IgXNTJvPXp2HndWi
m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 127
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:GGkb
a=ice-pwd:owqpzSht95NQb92cgzKNkQmJ
a=ice-options:trickle renomination
a=fingerprint:sha-256 45:76:5A:81:C1:CD:6B:C8:17:C6:CB:7B:20:CD:C6:DD:59:A3:DC:11:47:78:D3:E1:E1:EE:55:1F:83:A5:32:E4
a=setup:passive
a=mid:1
a=extmap:14 urn:ietf:params:rtp-hdrext:toffset
a=extmap:12 urn:3gpp:video-orientation
a=extmap:3 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
a=extmap:5 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
a=sendonly
a=msid:stream0 video0
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:96 VP8/90000
a=rtcp-fb:96 goog-remb
a=rtcp-fb:96 transport-cc
a=rtcp-fb:96 ccm fir
a=rtcp-fb:96 nack
a=rtcp-fb:96 nack pli
a=rtpmap:97 rtx/90000
a=fmtp:97 apt=96
a=rtpmap:98 VP9/90000
a=rtcp-fb:98 goog-remb
a=rtcp-fb:98 transport-cc
a=rtcp-fb:98 ccm fir
a=rtcp-fb:98 nack
a=rtcp-fb:98 nack pli
a=rtpmap:99 rtx/90000
a=fmtp:99 apt=98
a=rtpmap:100 red/90000
a=rtpmap:101 rtx/90000
a=fmtp:101 apt=100
a=rtpmap:127 ulpfec/90000
a=ssrc-group:FID 1487952389 3127747783
a=ssrc:1487952389 cname:IgXNTJvPXp2HndWi
a=ssrc:3127747783 cname:IgXNTJvPXp2HndWi
m=application 9 DTLS/SCTP 5000
c=IN IP4 0.0.0.0
b=AS:30
a=ice-ufrag:GGkb
a=ice-pwd:owqpzSht95NQb92cgzKNkQmJ
a
Why in this renegotiation the SDP for the Caller's answer does not have direction "sendrecv"?
On Thursday, 28 March 2019 01:44:12 UTC-5, sebastian pulido wrote:
sebastian pulido
Mar 29, 2019, 1:04:39 AM3/29/19
to discuss-webrtc
We have found asolution for this problem, and it is a good thing to share it with the community:
In the renegotiation we need for enabling remote track on the Caller side, we found that we needed to change the direction of the video transceiver before setting the remote description and creating the answer:
In the renegotiation we need for enabling remote track on the Caller side, we found that we needed to change the direction of the video transceiver before setting the remote description and creating the answer:
....
val transceivers = pcConn.getTransceivers().asScala
transceivers.foreach { t =>
val isVideoTransc = t.getMediaType() == MediaStreamTrack.MediaType.MEDIA_TYPE_VIDEO
if (isVideoTransc && this.callerShouldReceiveVideo(this.getCallType)) {
t.setDirection(RtpTransceiver.RtpTransceiverDirection.SEND_RECV)
}
}
pcConn.setRemoteDescription(sdpObserver, makeSDP(SessionDescription.Type.OFFER, setupMsg.sdp))
pcConn.createAnswer(sdpObserver, WebRTCCall.sdpMediaConstraints(this.callerShouldReceiveVideo(this.getCallType)))
...
Something nice that we learned about video and audio tracks is that under the hood they are implemented with transceivers which allows us to set he direction before generating the answer. In this case we needed to set the direction of the video transceiver to "sendrecv".
When direction is "sendonly", the native code for setLocalDescription will just remove the remote video-track since it will assume that we do not need it.
On Thursday, 28 March 2019 01:44:12 UTC-5, sebastian pulido wrote:
Christoph Eggert
Mar 29, 2019, 1:51:54 AM3/29/19
to discuss-webrtc
Hi sebastian,
it honestly sounds to me like you are patchworking around the issue, rather than resolving the actual issue. Manipulating the transceivers afterwards should never be necessary in standard use cases. From what I can gather of your scenario, the SDP of the callee who activates the video should have sendonly, while the created answer of the caller who does not send a video should have a recvonly (sendrecv makes no sense as only one side sends the video). If things are applied the correct way, this should happen automatically when the callee adds the video to the peerconnection and sends it as an offer. With Unified Plan, you are correct that setting descriptions triggers tracks to be added and statuses of streams to be changed - because of that, my biggest assumption is that you have a slipup somewhere in your logic that creates offers/answers and sets the descriptions. If your caller who is supposed to only receives video has sendonly in the sdp all of a sudden, my biggest bet would be that you set the local description to the SDP that you received from the callee, rather than the other way around. With "luck" this can still generate valid SDPs, especially when working with sendrecv, so my biggest bet would be on that front.
David Arias
Mar 31, 2019, 3:08:43 AM3/31/19
to discuss-webrtc
Hi Christopher, I'm working with Sebastian on this project and what we want is both parties to send/receive video. The caller is already sending video, and the callee is going to do that as well. Think of it as a toggle feature where any party can send/stop sending video, that's what we trying to achiece. This toggling is not covered in the example and we've been having hard time finding documentation/references.
Vadim Eksler
Jul 6, 2019, 3:52:35 AM7/6/19
to discuss-webrtc
Hi Christopher. I think our issue related to this problem, can you take a look - https://bugs.chromium.org/p/webrtc/issues/detail?id=10788&q=getTransceivers()&colspec=ID%20Pri%20Stars%20M%20Component%20Status%20Owner%20Summary%20Modified
пятница, 29 марта 2019 г., 8:51:54 UTC+3 пользователь Christoph Eggert написал:
пятница, 29 марта 2019 г., 8:51:54 UTC+3 пользователь Christoph Eggert написал:
Reply all
Reply to author
Forward
0 new messages