Weird H.264 communication problem between Raspberry PI app and Chrome

319 views
Skip to first unread message

Neil Young

unread,
Aug 15, 2021, 2:20:43 PM8/15/21
to discuss-webrtc
Hi,

It would be nice if one of the experts here could give me a pointer with this problem:

1) Raspberry PI running GST 18.4 and a Python app, utilizing "rpicamsrc"
2) JS web app in Chrome.

Summary:

I can establish VP8 connections. I can establish a H.264 connection, if the PI is sending the OFFER. If Chrome is sending the offer the connection is not successful.

"Connection" here means: SDP negotiation, the ICE connection is out of doubts successful all the time.

This is the GStreamer pipeline on the PI and it can be opened. I'm using the H.264 hardware encoding of the CSI cam:

# H.264 CSI CAMERA VIDEO (PI ONLY)
PIPELINE_PI_H264 = '''
webrtcbin name=webrtcbin bundle-policy=max-bundle stun-server=stun://stun.l.google.com:19302
rpicamsrc preview=false !
video/x-h264,width=1280,height=720,framerate=30/1 ! h264parse ! rtph264pay config-interval=1 !
application/x-rtp,media=video,encoding-name=H264  !
webrtcbin.
'''

If the PI offers,  the handshake looks like so:

The OFFER sent:

v=0
o=- 143230668495837560 0 IN IP4 0.0.0.0
s=-
t=0 0
a=ice-options:trickle
a=group:BUNDLE video0
m=video 9 UDP/TLS/RTP/SAVPF 96
c=IN IP4 0.0.0.0
a=setup:actpass
a=ice-ufrag:8OgtMvs3Zdn04lGPl61MEeQGtuq2XNS+
a=ice-pwd:GbtIZHC7CWPkgzNrnH7mWEtWNpaeWL5n
a=rtcp-mux
a=rtcp-rsize
a=sendonly
a=rtpmap:96 H264/90000
a=rtcp-fb:96 nack pli
a=framerate:30
a=fmtp:96 packetization-mode=1;profile-level-id=42c01f;sprop-parameter-sets=Z0LAH9kAUAW7AWoCAgKAAAADAIAAAB5HjBkk,aMuMsg==
a=ssrc:2817782937 msid:user2521903457@host-548ae76e webrtctransceiver0
a=ssrc:2817782937 cname:user2521903457@host-548ae76e
a=mid:video0
a=fingerprint:sha-256 1B:99:58:8C:F3:33:28:60:02:E6:11:66:74:B7:84:B4:D9:06:32:12:13:23:F0:E9:23:68:9D:C3:CC:79:85:C5

The Chrome ANSWER received (video flows), the webapp is not sending any media:

v=0
o=- 6562431427409268738 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE video0
a=msid-semantic: WMS
m=video 9 UDP/TLS/RTP/SAVPF 96
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:OoGw
a=ice-pwd:egHkzRQOlN6ADWLnaltQ6AkA
a=ice-options:trickle
a=fingerprint:sha-256 20:C2:D3:AA:37:07:31:87:FA:71:D4:20:CF:50:ED:07:EA:C8:B8:1D:EA:02:73:9C:7C:AC:24:5C:F3:C9:DA:60
a=setup:active
a=mid:video0
a=recvonly
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:96 H264/90000
a=rtcp-fb:96 nack pli
a=fmtp:96 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f

If Chrome is the offerer, the SDP negotiation is like so, video does not appear.

The OFFER generated by Chrome:

v=0
o=- 6799643414674800834 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0
a=extmap-allow-mixed
a=msid-semantic: WMS
m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 122 102 121 127 120 125 107 108 109 35 36 124 119 123 118 114 115 116 37
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:Tyef
a=ice-pwd:tct8Z2AfUb1yNAT/aWDfj2OI
a=ice-options:trickle
a=fingerprint:sha-256 F6:01:90:DF:E5:67:4E:B0:9F:DF:14:1A:3B:08:2F:E8:12:ED:CC:35:EB:4D:E2:22:AF:1F:18:7C:EB:37:D7:41
a=setup:actpass
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:toffset
a=extmap:3 urn:3gpp:video-orientation
a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:10 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
a=extmap:11 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
a=recvonly
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:96 VP8/90000
a=rtcp-fb:96 goog-remb
a=rtcp-fb:96 transport-cc
a=rtcp-fb:96 ccm fir
a=rtcp-fb:96 nack
a=rtcp-fb:96 nack pli
a=rtpmap:97 rtx/90000
a=fmtp:97 apt=96
a=rtpmap:98 VP9/90000
a=rtcp-fb:98 goog-remb
a=rtcp-fb:98 transport-cc
a=rtcp-fb:98 ccm fir
a=rtcp-fb:98 nack
a=rtcp-fb:98 nack pli
a=fmtp:98 profile-id=0
a=rtpmap:99 rtx/90000
a=fmtp:99 apt=98
a=rtpmap:100 VP9/90000
a=rtcp-fb:100 goog-remb
a=rtcp-fb:100 transport-cc
a=rtcp-fb:100 ccm fir
a=rtcp-fb:100 nack
a=rtcp-fb:100 nack pli
a=fmtp:100 profile-id=2
a=rtpmap:101 rtx/90000
a=fmtp:101 apt=100
a=rtpmap:122 VP9/90000
a=rtcp-fb:122 goog-remb
a=rtcp-fb:122 transport-cc
a=rtcp-fb:122 ccm fir
a=rtcp-fb:122 nack
a=rtcp-fb:122 nack pli
a=fmtp:122 profile-id=1
a=rtpmap:102 H264/90000
a=rtcp-fb:102 goog-remb
a=rtcp-fb:102 transport-cc
a=rtcp-fb:102 ccm fir
a=rtcp-fb:102 nack
a=rtcp-fb:102 nack pli
a=fmtp:102 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001f
a=rtpmap:121 rtx/90000
a=fmtp:121 apt=102
a=rtpmap:127 H264/90000
a=rtcp-fb:127 goog-remb
a=rtcp-fb:127 transport-cc
a=rtcp-fb:127 ccm fir
a=rtcp-fb:127 nack
a=rtcp-fb:127 nack pli
a=fmtp:127 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42001f
a=rtpmap:120 rtx/90000
a=fmtp:120 apt=127
a=rtpmap:125 H264/90000
a=rtcp-fb:125 goog-remb
a=rtcp-fb:125 transport-cc
a=rtcp-fb:125 ccm fir
a=rtcp-fb:125 nack
a=rtcp-fb:125 nack pli
a=fmtp:125 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f
a=rtpmap:107 rtx/90000
a=fmtp:107 apt=125
a=rtpmap:108 H264/90000
a=rtcp-fb:108 goog-remb
a=rtcp-fb:108 transport-cc
a=rtcp-fb:108 ccm fir
a=rtcp-fb:108 nack
a=rtcp-fb:108 nack pli
a=fmtp:108 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42e01f
a=rtpmap:109 rtx/90000
a=fmtp:109 apt=108
a=rtpmap:35 AV1X/90000
a=rtcp-fb:35 goog-remb
a=rtcp-fb:35 transport-cc
a=rtcp-fb:35 ccm fir
a=rtcp-fb:35 nack
a=rtcp-fb:35 nack pli
a=rtpmap:36 rtx/90000
a=fmtp:36 apt=35
a=rtpmap:124 H264/90000
a=rtcp-fb:124 goog-remb
a=rtcp-fb:124 transport-cc
a=rtcp-fb:124 ccm fir
a=rtcp-fb:124 nack
a=rtcp-fb:124 nack pli
a=fmtp:124 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=4d0032
a=rtpmap:119 rtx/90000
a=fmtp:119 apt=124
a=rtpmap:123 H264/90000
a=rtcp-fb:123 goog-remb
a=rtcp-fb:123 transport-cc
a=rtcp-fb:123 ccm fir
a=rtcp-fb:123 nack
a=rtcp-fb:123 nack pli
a=fmtp:123 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=640032
a=rtpmap:118 rtx/90000
a=fmtp:118 apt=123
a=rtpmap:114 red/90000
a=rtpmap:115 rtx/90000
a=fmtp:115 apt=114
a=rtpmap:116 ulpfec/90000
a=rtpmap:37 flexfec-03/90000
a=rtcp-fb:37 goog-remb
a=rtcp-fb:37 transport-cc
a=fmtp:37 repair-window=10000000


The ANSWER generated by webrtcbin:

v=0
o=- 6799643414674800834 2 IN IP4 0.0.0.0
s=-
t=0 0
a=group:BUNDLE 0
m=video 9 UDP/TLS/RTP/SAVPF 96
c=IN IP4 0.0.0.0
a=ice-ufrag:ej11/cl3FCkkhNmNliiql1R5MX5MFn/g
a=ice-pwd:SgutRYS/iWj9l8WvWvXGzrohbIiTLK+f
a=mid:0
a=rtcp-mux
a=setup:active
a=rtpmap:96 VP8/90000
a=rtcp-fb:96 nack pli
a=rtcp-fb:96 ccm fir
a=inactive
a=fingerprint:sha-256 BD:5C:65:0E:D3:75:67:8B:80:71:57:D5:08:14:70:1A:BB:4F:51:F8:54:53:80:17:8C:3E:44:DB:EA:E5:04:06

Later I opened the pipeline with payload=125 in order to match the seemingly same setup which was successful above:

# H.264 CSI CAMERA VIDEO (PI ONLY)
PIPELINE_PI_H264 = '''
webrtcbin name=webrtcbin bundle-policy=max-bundle stun-server=stun://stun.l.google.com:19302
rpicamsrc preview=false !
video/x-h264,width=1280,height=720,framerate=30/1 ! h264parse ! rtph264pay config-interval=1 !
application/x-rtp,media=video,encoding-name=H264,payload=125  !
webrtcbin.
'''

This time the answer of the PI looks like so (which is promising)...

v=0
o=- 612181932603522942 2 IN IP4 0.0.0.0
s=-
t=0 0
a=group:BUNDLE 0
m=video 9 UDP/TLS/RTP/SAVPF 125
c=IN IP4 0.0.0.0
a=ice-ufrag:ElQd9dpLf6mfqer/HDdOlmMHc9QJBGsp
a=ice-pwd:3BMFBF3syzu8RjoXrxxgKBSGRlQFYEMn
a=mid:0
a=rtcp-mux
a=setup:active
a=rtpmap:125 H264/90000
a=rtcp-fb:125 nack pli
a=rtcp-fb:125 ccm fir
a=fmtp:125 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f
a=sendonly
a=fingerprint:sha-256 64:EB:42:4D:4F:EA:83:FE:60:AB:A6:99:0B:DA:1B:66:C3:A0:8A:74:83:4E:A0:9C:FF:6F:25:A5:E2:E7:0F:88

...but in the end on the Chrome side just the spinning wheel is displayed.

Anybody having a pointer, how to make that run?



V I

unread,
Aug 16, 2021, 12:40:36 AM8/16/21
to discuss...@googlegroups.com
Looks like Pi offers only one option - H.264, Chrome agrees and it works as expected. When Chrome sends SDP, it includes VP8, VP9 and H.264. Pi picks VP8 as it's the first one in the list and you probably don't instruct it to stick to H.264, and then it probably pushes H.264 stream while it's expected to be VP8.
You need to force Chrome to offer only H.264 or configure Pi to accept H.264 exclusively, both can be done with this (it's also called SetCodecPreferences in the native API): https://developer.mozilla.org/en-US/docs/Web/API/RTCRtpTransceiver/setCodecPreferences

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/0a2c358e-4816-4b79-8f6f-253de0851ec4n%40googlegroups.com.

Neil Young

unread,
Aug 16, 2021, 2:22:40 AM8/16/21
to discuss-webrtc
Thanks, this sounds like a logical explanation. It doesn't explain why the connection cannot be established in case I force the PI to offer 125, but I will give it a try. A quick test for "setCodecPreferences" in the developer console looked promising.

Thanks for the great pointer.

Neil Young

unread,
Aug 16, 2021, 3:42:39 AM8/16/21
to discuss-webrtc
Unfortunately this hint is not going to work for me for several reasons:

1) My webapp (the offerer) is not sending video at all. Hence there is no videoTransceiver, whom's codecPreferences I can set. An attempt to add a dummy videoTransceiver with "inactive" or "recvonly" direction ends either in a "full" SDP offer again or an error on the webapp side
2) If I let my webapp send video, each attempt to limit the codecs to one H.264 only leads to a crash in GStreamers on_incoming_decodebin_stream (sigsev, see below) after adding a stream with completely obscure description( x-raw??)

2021-08-16 07:37:19,734 webrtc_client.py-INFO    : on_incoming_decodebin_stream video/x-raw, format=(string)I420, width=(int)600, height=(int)450, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)jpeg, colorimetry=(string)bt601, framerate=(fraction)0/1
2021-08-16 07:37:19,746 webrtc_client.py-ERROR   : caught SIGSEV <frame at 0xb504d030, file '/home/pi/.local/lib/python3.7/site-packages/gi/_ossighelper.py', line 106, code signal_notify>

At least the SDP handshake looks better that time: 

OFFER:

v=0
o=- 5555860852462237442 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0
a=extmap-allow-mixed
a=msid-semantic: WMS bOaGc7rWq8jdc3Ze1RyFyeT9SZGPwINBNOlD
m=video 9 UDP/TLS/RTP/SAVPF 106
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:PPSm
a=ice-pwd:l1dMxpJj+fxkPuc1QeEHuwu0
a=ice-options:trickle
a=fingerprint:sha-256 B9:57:21:4D:73:40:84:B1:A5:50:C0:16:D5:7E:74:3B:A3:36:B6:CD:03:44:BC:AB:14:ED:AB:D3:D8:90:84:9F
a=setup:actpass
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:toffset
a=extmap:3 urn:3gpp:video-orientation
a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:10 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
a=extmap:11 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
a=sendrecv
a=msid:bOaGc7rWq8jdc3Ze1RyFyeT9SZGPwINBNOlD 05e89a03-b4cd-4673-a031-0dc5b421dd2b
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:106 H264/90000
a=rtcp-fb:106 goog-remb
a=rtcp-fb:106 transport-cc
a=rtcp-fb:106 ccm fir
a=rtcp-fb:106 nack
a=rtcp-fb:106 nack pli
a=fmtp:106 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f
a=ssrc:1862760915 cname:o5p6u6Ix9Qa4ebB4
a=ssrc:1862760915 msid:bOaGc7rWq8jdc3Ze1RyFyeT9SZGPwINBNOlD 05e89a03-b4cd-4673-a031-0dc5b421dd2b
a=ssrc:1862760915 mslabel:bOaGc7rWq8jdc3Ze1RyFyeT9SZGPwINBNOlD
a=ssrc:1862760915 label:05e89a03-b4cd-4673-a031-0dc5b421dd2b


ANSWER:

v=0
o=- 5555860852462237442 2 IN IP4 0.0.0.0
s=-
t=0 0
a=group:BUNDLE 0
m=video 9 UDP/TLS/RTP/SAVPF 106
c=IN IP4 0.0.0.0
a=ice-ufrag:S2FpqaKWFv83hUUBhbkrGH+aKitpL7Dk
a=ice-pwd:yn/GZ3uu6PFgHPJpGJr8YeZ8ej1p2woy
a=mid:0
a=rtcp-mux
a=setup:active
a=rtpmap:106 H264/90000
a=rtcp-fb:106 nack pli
a=rtcp-fb:106 ccm fir
a=fmtp:106 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f
a=recvonly
a=fingerprint:sha-256 D0:7D:2C:17:65:18:31:72:B3:37:8A:86:83:30:64:EE:FB:FF:2F:77:C4:84:14:19:C6:53:B1:25:69:00:65:FB





Neil Young

unread,
Aug 16, 2021, 7:37:27 AM8/16/21
to discuss...@googlegroups.com
To give a more sophisticated answer on this:

With the limitation to

videoTransceiver.setCodecPreferences([
{ clockRate: 90000, mimeType: "video/H264", sdpFmtpLine: "level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f" },
])


...the answer of the Python app toggles between "recvonly" and "sendonly", but there is no video on the remote side. So setCodedPreferences did help at least to reduce the codec numbers, but not more ATM.





Reply all
Reply to author
Forward
0 new messages