Video Streaming not working over certain mobile networks (no codec selected)

[Carl Meyer]

Intro
Hello WebRTC Community! 

This is my first post :) I would be super grateful for any help with a particular issue that I am experiencing with WebRTC

The Problem

I have a web app that streams video from a CCTV camera. 

This always works over wifi and works most of the time on 4G connection except for certain mobile phones on certain network. 

Not working:
iPhone using EE sim card (UK)
iPhone using Three network (UK)

iPhone using AT&T sim card (US)

Google Pixel using EE sim card (UK)
Google Pixel using Three sim card (UK)

H

We have done lots of testing and we can verify that streaming always works over Wifi and it does not work over the above mentioned devices / networks. 

The investigation

We investigated the entire webRTC flow to try and figure out where the problem happens and what the problem is causing live video to stop working. 

We verified ice connectivity and that the camera and phone are able to communicate with each other and that the peer to peer connection is established successfully


We verified that everything on the camera is working as expected:

• The Hikvision CCTV camera uses an implementation of libnice to do ICE connection and video streaming
• We verified that the camera sends video over the peer to peer connection and arriving at the phone

Camera sending video data (captured on wireshark)

We can see the video data arrive at the phone

(we used wireshark on android and rvictl on iPhone)

We also looked at WebRTC internals in Chrome to verify this as well


So we know that the footage is arriving at the phone but still we can not play live video. 

We spent a lot of time looking at WebRTC internals to try and compare working cases vs non working cases and we found the following:

No codec selected in non working case

When working over Wifi we always get live video and we see the following on the RTCInboundRTPVideoStream object:

When we compare this with a non working case (iPhone using EE sim card) we can see that there is no codec selected

This is the only conclusion I can make as to why our live video stream isn't working. 

I have no idea why the codec is not being selected in this non working case but is always selected on wifi and some other networks. 

Is there anyone that can possibly help me make sense of this as I am not sure how the underlying webRTC engine decides how and when to add the codec to decode the video arriving at the phone. 

Sincerely
Carl

[Philipp Hancke]

the lack of a codecId simply means that no complete frame has reached the decoder. 27 lost packets out of 93 is pretty high, as is the PLI count. I'd guess the sender is not resending some of the missing packets that Chrome/libWebRTC asks for. You'll need to check wireshark and look at the RTP packets received - the unencrypted logging from https://bugs.chromium.org/p/webrtc/issues/detail?id=10675 may help more than a normal wireshark dump. https://webrtchacks.com/video_replay/ still has a lot of helpful hints.

--
This list falls under the WebRTC Code of Conduct - https://webrtc.org/support/code-of-conduct.
---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/aeff1685-2e55-4a3c-8c9a-52968584a0bfn%40googlegroups.com.

[Carl Meyer]

Thank for the reply. 

I am trying to understand why we might be running into packet loss when using direct 4G connection to the phone. 

What I find really interesting is that when we do USB tethering to a computer on the same 4G connection, live video works. (Same for hotspotting)

Any idea why we could be dropping packets even if the connection is stable? 

Also could this possibly be a timing issue in the way we call the webRTC library from our Javascript code? 

Here are some webRTC internal dumps if that could be useful

[Carl Meyer]

I might be grasping at straws, but there is something else I might want someone else to help me understand. 

The camera we are working with only supports H264 encoding. 

When I look at the offer we send the camera it contains the H264 codecs and we include the profile level id alongside them



but when we receive our answer from the camera, we also get similar entries for the H264 codec but this does not include any information about the profile level id:



Could this be a possible reason why the codec is not being attached even after we are connected?

(as you can see below, no profile level id selected, no codec, but we receive video but we can not play it)

[Carl Meyer]

Yesterday I tried a workaround on the Android Google Pixel using 4G EE network. 

I filtered the list of ice candidates that we send the camera so that we only send relay candidates, this way we would force the webRTC library to select a candidate pair that uses a relay candidate, meaning we then use our twillio stun and turn service for the ice connection. 

For some reason yesterday that seemed to have solved our issue and I could see live video. (I suspect since this had less hops we were able to receive a full video frame, codec got attached and we were able to play video). Sadly this workaround did not work for iOS devices (iPhone 13 Pro using 4G EE network). 

Update:
Today I tried testing this workaround again, but for some reason its not working anymore! 

I can see that we get our Turn messages via Stun and Turn and this is when I would expect  live video to start showing, but today I see the same problem even when forcing the use of stun and turn: 

No live video, inspecting RTPInbound we do not have a codec applied. 

On Wednesday, 4 October 2023 at 10:10:22 UTC+1 Carl Meyer wrote:

[Philipp Hancke]

the fmtp line in the SDP looks definetly broken but that doesn't explain why it would work on some networks but not on others.

You might want to ask on the gstreamer mailing list, the problem seems to be on that side of things.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/160571b4-e4b0-4229-9842-f536bbeb980fn%40googlegroups.com.

[Harald Alvestrand]

There's a known issue with some network operators and some Android versions and some WebRTC versions where the kernel will choose a source address for the IP packets of the RTCPeerConnection that causes the packets to be dropped. That's the only thing I can think of that would account for the difference between USB-tethered and direct-4G operation.

The source address in the image is cut off, so can't verify if it's a routable address.

However, if the transport state reaches "connected" at all, this shouldn't be the issue.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/CADxkKiLg4AWLHUpyvP_kmpMEkOw%3DOGkFkF0T%2BzW1%2BTrUcQg1bw%40mail.gmail.com.

[Carl Meyer]

Thanks for the replay Harald. 

Do you have a link to this issue as I wonder if this could be related as we see a lot of packet loss when the issue happens (and I wonder if that is why the video codec is not applied as we don't get a full frame render). 

I see a similar issue on iPhone also on the EE network operator, so it would be interesting to try and understand if I might be running into this issue

[Carl Meyer]

Thanks for the replies. 

I am going to go through and map out exactly how our Javascript library calls the webRTC interface (in order to determine if its an implementation issue and the sequence in which we call the webRTC library). 

Philipp you mentioned the SDP looks broken, could you maybe specify what parts might be broker or missing? 

We send our SDP offer to the camera via HTTP request and the camera responds via HTTP with an SDP answer 

We then transform this response into a SDP that we then use to set the remote description on the peer connection

But like you said, I don't know why it would work over Wifi but not over this specific 4G connection in iPhone and Pixel. 

[Philipp Hancke]

these answer bits you marked. ccmfir=true is not a fmtp, nor are goog-remb transport-cc.

On the other side nack is missing from the rtcp-fb (only nack pli is there) which seems like the culprit as NACKs are the #1 error resiliency mechanism.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/96f52a2c-b3d4-470a-b2c0-d424d8d324d7n%40googlegroups.com.

[Carl Meyer]

I have now tried to modify the SDP answer before calling setRemoteDescription() and now the SDP looks like this afterwards

[Carl Meyer]

Hi Philipp 😄

Thank you for this reply. That is really interesting as I always assumed we were getting a valid SDP answer from the camera. 

Unfortunately I am not able to make changes to the Camera FW at this time, but I do receive the SDP answer via HTTP beforew I setRemoteDescription

We already do some munging on the SDP Answer and I am wondering if further changes might be needed to change it into a more valid format. 

I will post the SDP Offer and SDP Answer that we send and receive. 

Could you or anyone possible help me understand what a valid SDP answer for the a=fmtp line could look like (and if we are missing properties in the offer as well). 

I am still building my knowledge of WebRTC so any help with these SDP payloads would be greatly appreciated 🙏


 

On Wednesday, 11 October 2023 at 05:04:46 UTC+1 Philipp Hancke wrote:

[Philipp Hancke]

that SDP looks ok! I'd strip the trailing ";" after the profile-level but the parser is more tolerable than me ;-)

If that doesn't help you'll need to go the full RTP dump route explained in the blog post.

Fingers crossed, good luck!

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/54123548-0742-4831-b92d-164ad5602d93n%40googlegroups.com.

[Carl Meyer]

Thanks again for your reply. 

Even after fixing the SDP, we still do not have video. 

We are going to try and create an Android App that tries to stream from the camera as our apps team has a previous WebRTC App that seems to be working fine (although working with a different piece of hardware as the sender). 

I have been looking at this issue for about 2 months now and have tried so many workarounds and changes and still have not found the root cause or a fix yet, so I am feeling a little bit frustrated. 

At least I learned a lot in the process thus far and I feel grateful to have big rockstars like you (Philipp) and Harald replying to try and help me. Thank you! 

[V I]

This is a very interesting issue, I have no idea what the culprit is, but let me speculate lol

The very first screenshot you sent indicates that there are either no key frames sent or the encoder is using intra-refresh mode when the things are broken. The reason for this could be a super weird case of symmetric NAT which somehow drops traffic in one direction (from the phone to the camera), so that PLIs fired on the receiver never reach the camera. I've never encountered exactly that, but I had an ISP which would allow UDP to send a few packets back and forth and then it would block it completely.

Another theory, a much more simple one: I've spotted that in your WiFi case screenshot profile_level_id is 640c1f, while in other places you set it to 42001f. Did you try to munge it to 640c1f in te offer?

I'd also try creating a recvonly offer (I can see a=sendrecv in your SDP) or maybe even start with the camera generating the offer if possible.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/93610f13-e3e7-4581-bdc2-ca6d6faf55b8n%40googlegroups.com.

[V I]

There's also this shady prediction scheme by Hikvision (they call it H.264+) which might be at play: https://www.securitymagazine.com/ext/resources/whitepapers/Hikvision-H264-Encoding-Technology.pdf

Could it be that the camera detects that you're using mobile connection and switches to this H.264+ mode to save bandwidth?