h.264/rtp stream from raspberry - short freezes (packets dropped?)

[AlterSchwede]

If generate a stream from Raspberry Pi as follows:

raspivid --verbose --nopreview --width 640 --height 480 --framerate 15 --bitrate 1000000 --profile baseline --timeout 0 -o - | gst-launch-1.0 -v fdsrc !  h264parse ! rtph264pay config-interval=1 pt=96 ! udpsink host=192.168.178.22 port=8004

and view this using the Janus streaming demo and Firefox browser, the video seem to now and then freeze for a short while.

When I terminate the stream using gstreamer:

gst-launch-1.0 -v udpsrc port=9001 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! decodebin ! videoconvert ! autovideosink

I do not experience this problem.

I assume somehow and somewhere packets are lost or discarded - either related to Janus or to Firefox browser.

Is there a explanation of this behaviour - such as me using wrong gstreamer settings or Firefox limitation with respect to h.264 and webRTC...and has anyone else seen the same behaviour?

Generating VP8 from Raspberry would perhaps be a solution, but before I go into this domain, I would like to now if streaming using h.264 could be made to work properly.

[Tobias Wolf]

I noticed this too. You followed my short recipe on Raspberry Pi forum?

I recommend to enable overlay annotation with --annotate 512. It will show incrementing framecount overlay. You can see visually for how many frames it drops out.

When a dropout occurs it comes back on the next intra-frame. It must be somehow related to iframes. Two options relate to this --intra and --irefresh. I turned on --irefresh both and it seems to be better. I also set --intra 30 to give and iframe every 2 seconds.

--inline takes care of in-band signaling of video settings.

raspivid --verbose --nopreview --profile baseline --width 640 --height 480 --framerate 15 --bitrate 1000000 --intra 30 --irefresh both --inline --annotate '512' --timeout 0 -o - | gst-launch-1.0 -v fdsrc ! h264parse ! rtph264pay pt=96 ! udpsink host=127.0.0.1 port=8004

I think with this it is better.

[Mikael Boman]

Yes, I used your recipe on Raspberry Pi forum, with the difference that I am running the Janus GW on separate server.
However, the suggestions you made did not help.
I can only lower the freeze time by generating more I-frames by setting --intra to e.g. 5.

I very often seem to get a freeze in combination with:
Long poll time out for session 2175842461...
We have a message to serve...
    {"janus" : "keepalive"}
Request completed, freeing data
Got a HTTP GET request on /janus/2175842461...
 ... Just parsing headers for now...
Host: 192.168.178.22:8088
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:35.0) Gecko/20100101 Fir
efox/35.0
Accept: application/json, text/javascript, */*; q=0.01
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: http://192.168.178.22/janus/streamingtest.html
Origin: http://192.168.178.22
Connection: keep-alive
Got a HTTP GET request on /janus/2175842461...
 ... parsing request...
Session: 2175842461
Session 2175842461 found... returning up to 1 messages
... handling long poll...
[3401231580]  Got an RTCP packet (video stream)!

Can it be a problem with Firefox v35 in combination with h.264?

[Mikael Boman]

Update:

I have no packet loss in my network. Instead, the problem seem to depend on jitter or timing. 

If I connect the Raspberry Pi with cable instead of Wi-Fi to PC1 running Janus gw, I have no freezes in a Firefox browser

running on same PC.

If I connect via Wi-Fi, I have some freezes on same PC1 running Janus and even more freezes on PC2.

RPi/camera ---- Wi-Fi----PC1/Janus/Fx browser---- Wi-Fi-----PC2/Fx browser

I have very low latency - I can live with a little more - 

is there a way to increase buffers in Janus or Firefox to be able to cope with this problem?

I am using the streaming demo for testing.

[Lorenzo Miniero]

Have you tried the latest version of Janus? We experienced some similar issues and the cause there was the streaming plug-in was not negotiating nacks, and so browsers had no way to ask for a retransmission. Besides, also check the mtu being used for the streaming part: having it to close to the edge can cause issues swhen the stream is wrapped for webrtc, as srtp adds bytes of its own. Possibly both suggestions are unrelated but worth a try.

Lorenzo

[Mikael Boman]

I am using Janus v 0.0.8 compiled a few days ago and also tried with different mtu lengths, e.g:

raspivid --verbose --nopreview --width 640 --height 480 --framerate 15 \

--qp 20 --intra 40 --profile baseline --timeout 0 -o - | gst-launch-1.0 -v fdsrc \

!  h264parse ! rtph264pay config-interval=10 pt=96 mtu=1300 ! udpsink host=192.168.178.22 port=8004

but I see no significant difference. The latency is very small, ~200-300 ms.

To recap:

I see "video freezes" when sending a H.264/RTP stream from Raspberry Pi and using the Janus

streaming demo to view to video towards Mozilla/Firefox 35.0.1 (Linux).

Initially, I thought I had packetloss in my local Wi-Fi network, but this is not the case.

However, its RTT is varying quite a lot:

9714 packets transmitted, 9704 received, 0% packet loss, time 98265ms

rtt min/avg/max/mdev = 3.362/5.779/157.771/5.655 ms, pipe 9

...and a shorter sample...:

538 packets transmitted, 538 received, 0% packet loss, time 5372ms

rtt min/avg/max/mdev = 3.742/4.971/11.558/1.206 ms

The WebRTC log from Firefox says:

RTP statistics

inbound_rtp_video_1

Decoder: Avg. bitrate: 0.00 Mbps (0.00 SD) Avg. framerate: 0.00 fps (0.00 SD) Discarded packets: 0

Local: 22:07:29 GMT+0100 (CET) inboundrtp SSRC: 704087196 Received: 29806 packets (35501.08 Kb) Lost: 43 Jitter: 0

(why is birate=0?) SSRC reports 43 lost and 0 jitter.

 

Now I wanted to know if VP8/RTP from Gstreamer is behaving in a similar manner so I used the following pipe:

gst-launch-1.0 -v v4l2src ! 'video/x-raw, format=(string)I420, width=(int)320,\

height=(int)240, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, 

framerate=(fraction)30/1' ! vp8enc keyframe-max-dist=60 ! rtpvp8pay pt=100 mtu=1300 \

! udpsink host=192.168.178.22 port=9004

to generate a VP8 stream from a PC (VP8 is not supported by Raspberry Pi).

and the WebRTC log from Firefox says:

RTP statistics

inbound_rtp_video_1

Decoder: Avg. bitrate: 0.00 Mbps (0.00 SD) Avg. framerate: 0.00 fps (0.00 SD) Discarded packets: 0

Local: 22:33:13 GMT+0100 (CET) inboundrtp SSRC: 1625784456 Received: 8902 packets (9885.51 Kb) Lost: 17 Jitter: 4.591

(Again, bitrate=0?)

Also here I have some loss which is also visible but the difference is that for VP8 jitter is reported, for H.264 not.

There is also almost no 'freezes' and latency is larger - ~800-1000 ms.

I wanted to try a Gstreamer/H.264/RTP from a PC too but failed and instead only got 

"Still waiting for the DTLS stack for component 1 in stream 1..." - no idea what is missing since my VP8 pipeline worked.

So final test is to do a VP8 video call via Janus using Firefox and Chrome to check if use of their own VP8 enc/dec/WebRTC makes a

difference. And it does: although latency is very low, there are no freezes until this happens in Janus:

[ERR] [ice.c:janus_ice_send_thread:2238:] [2537241960] ... only sent -1 bytes? (was 1074)

[ERR] [ice.c:janus_ice_cb_nice_recv:1023:] [2537241960]     SRTP unprotect error: err_status_replay_old (len=1000-->1000, ts=4253741017, seq=48076)

 

Now to my own speculations: 

considering the very low video latency I see (~200-300 ms),  packet/frame arrival to the browser can be critical if Firefox reception buffer 

is too small - hence causing frames to be discarded since they arrive too 'late' - thus some sort of underrun.

   This can perhaps be regarded as confirmed since when connecting Raspberry Pi directly via LAN cable to Janus PC and viewing

the stream also on the Janus PC, I had much less 'freezes', also with the same low latency.

   It shall also be mentioned that viewing the RTP stream directly with Gstreamer on any PC in the Wi-Fi network

does not cause any freezes at all.

   When doing a video call using the browsers 'internals' the freezes are reduced to a minimum, still maintaining low latency (but using high bandwidth) and the browser manages to report a bandwidth which was not the case previously.

Is there some sort of bandwidth negotiation going wrong or missing in the initial case, e.g. due to bad Gstreamer pipeline?

Is WebRTC/Janus not handling the bandwidth/timing correctly when receiving a RTP stream?

Why do I get a jitter buffer for Gstreamer generated VP8 but not for h.264 - some setting in Firefox missing?

(I would like to setup a H.264 stream direct to Mozilla/Firefox to check but do not know how)

I would appreciate feedback to understand if I have something wrong in my setup or Gstreamer pipeline or if we really have an Raspberry, Janus or Firefox issue.

Can someone here confirm a 'working' H.264/RTP stream? If so, how does it look like?  

[Lorenzo Miniero]

It depends how many few days ago you got Janus: we update it regularly, and the fix I speak about dates back I think to Friday. Tags don't mean much as we just mark when we change something substantial, and so they don't necessarily indicate a stabler or more reliable version.

Since Firefox talks about 32 lost packets, a version before the one with the fix would not be able to compensate the loss on the client-Janus side, as the plugin didn't negotiate NACKs and so the browser isn't sending any, meaning lost packets are lost forever (and hence freezes if it happens for a key frame). The latest version does. No timing/bandwidth management is done in Janus (except for REMB feedback, but that's a different thing that doesn't affect the streaming plugin), frames are relayed as they come.

Apart from that, unfortunately I'm not familiar with Raspberrys and not a big fan of H.264, so I'll let others share their thoughts :-)

Lorenzo

[Mikael Boman]

Thanks for the update. 

I am using the latest version so NACK negotiation is in.

I would also prefer the license free VP8 but this is not available for RPi yet, to my knowledge.

/Mikael

[Hasan Tekin]

Hi,

Is there any update on this problem? I'm using Raspberry Pi 2 and it freezes when I use "Streaming Plugin". I'm using this command to start streming;

 raspivid --verbose --nopreview -hf -vf --width 640 --height 480 --framerate 15 --bitrate 1000000 --profile baseline --timeout 0 -o - | gst-launch-1.0 -v fdsrc ! h264parse ! rtph264pay config-interval=1 pt=96 ! udpsink host=127.0.0.1 port=8004

I'm using very recent version of Janus.

Thanks,

Hasan

[Hasan Tekin]

Hi, I'm having freeze problem and when video freezes I get these outputs from Janus;

When I see this particular lines, video works very fluently;

(process:2989): libnice-DEBUG: Agent 0x73612d38 : s1:1: sending 1170 bytes to [192.168.1.30]:50009

But sometimes this bunch of lines comes and video freezes until this group of lines gone;

(process:2989): libnice-DEBUG: Agent 0x73612d38 : Packet received on local socket 21 from  [192.168.1.30]:50009 (70 octets).

STUN error: RTP or other non-protocol packet!

[2906275641]  Got an RTCP packet (bundled stream)!

[2906275641] Incoming RTCP, bundling: this is video (no audio has been negotiated)

Got REMB bitrate 2163840

REMB for this PeerConnection: 2163840

Then I see this lines again;

(process:2989): libnice-DEBUG: Agent 0x73612d38 : s1:1: sending 1410 bytes to [192.168.1.30]:50009

Can you give me any clue about what may be happening?

Thanks,

Hasan

On Saturday, February 14, 2015 at 11:28:35 PM UTC+2, Mikael Boman wrote:

[Lorenzo Miniero]

None of those messages are errors, they're just very verbose libnice debug.

L.

[Hasan Tekin]

Yes it's normal when it happens sometimes. But apparently Firefox starts sending REMB consecutively for a while (1-2 seconds). And video freezes in that period.

I've uploaded a screenshot when it happend.

Lorenzo, I don't expect an answer. I'll be just updating this post as I get something new to show other people what is happenning if it's ok.

Thanks,

Hasan

[Lorenzo Miniero]

A REMB is just an indication from the browser on the estimated available bandwidth for receiving media, it cannot cause freezes. If it turns out the bitrate of the media is higher than the receiver can handle (because of network bandwidth, excessive packet loss, or other reasons) then issues can occur, e.g., video freezes. As such, a REMB is useful as it can tell you if the bandwidth for a user is lower than the bandwidth you're using for sending media, so that you can adapt and reduce the bitrate, for instance. It is not acted upon in the streaming plugin as it is in some other plugins, though: the external source in the streaming plugin case (ffmpeg, gstreamer or whatever else) cannot react on this information, and will keep on sending at the bitrate it was configured with. Freezes can also occur when the browser lost a keyframe: a request for a keyframe (PLI or FIR sent by the browser) cannot be honored either, because there's no way to tell the encoder to prepare one, the tool will keep on working on its own, meaning the video will only recover when the encoder eventually prepares a keyframe the browser can use.

L.

[Hasan Tekin]

Hi Lorenzo,

Recently I have figured out that the freezing problem stems not from Janus but RaspiVid and Gstreamer. Janus is working like a charm thanks to you and other contributors. Thanks a lot for that again.

I found a limited solution. The video freezes because raspivid cannot buffer gstreamer all the time. To heal this problem, I put a setting called "intra" on raspivid command. And now I get way shorter freezes every ten seconds. Here is the better command;

raspivid --nopreview -hf -vf --width 640 --height 480 --intra 5 --framerate 20 --bitrate 2000000 -ex fixedfps -awb off --ISO 800 --profile baseline --timeout 0 -o - | gst-launch-1.0 -v fdsrc ! h264parse ! rtph264pay config-interval=1 pt=96 ! udpsink host=127.0.0.1 port=8004

Here, only the intra level is responsible from the duration of the freeze.

Good luck.

[Mikael Boman]

I believe 'intra' indicates the interval for sending 'full frames'...

My problem actually was severe packet loss (in wlan - up to 15%). Due to this, I nowadays only test 

using 100mbit lan connection where I can simulate packet loss and latency for typical networks (LTE etc.).

I also belive transmitting RTP without support of RTCP (NACK etc.) is a little shaky, but as you indicated,

setting 'intra' low almost eliminates the problem in networks with 'normal' packet loss (...like 0.2%).

/M 

[Hasan Tekin]

Isn't that possible to give raspi camera stream to Janus in another way?

[Mikael Boman]

You can run Janus on RPi itself and connect from browser using webrtc directly to RPi. Then the rtp is only internal.

/M

[Hasan Tekin]

I think I wasn't clear. I'm running Janus on my RPi actually right now. But still I use that command to start camera stream. I used this blog to do that. You may have a look;

http://www.rs-online.com/designspark/electronics/eng/blog/building-a-raspberry-pi-2-webrtc-camera

So, can I access camera from C program of Janus?

[Mikael Boman]

Ok
That is pretty cool!
Not sure I can help you there.
I also feed janus with internal rtp stream and have no freezes. However, on a server and not on RPi.

/M

[Hasan Tekin]

I see. Thanks and good luck :)

[Mikael Boman]

One remark perhaps (just guessing here)...If you have problems with timing/sync between raspivid and Gstreamer (causing rtp packet/frame loss), could a 'queue' parameter in the Gstreamer pipe help?

So if you use RPi 2 would not raspvid and Gstreamer (after the queue) then run in two different threads (on two different cores) which perhaps would eliminate you problem?

Not sure how Janus does threading, but likely also benefiting from this since RPi 2 has 4 cores. Using 'queue' might actually even helps on RPi 1 since buffering is introduced. 

See also:

http://gstreamer.freedesktop.org/data/doc/gstreamer/head/manual/html/section-threads-uses.html

/M

[Jarogniew Muszyński]

Has anybody any success with raspivid + gstreamer streaming H246 to Chrome browsers without freezing? My command line:

raspivid --nopreview --width 640 --height 480 --intra 5 --framerate 10 --bitrate 2000000 -ex fixedfps --ISO 800 --profile baseline --timeout 0 -o - | gst-launch-1.0 -v fdsrc ! h264parse ! rtph264pay config-interval=1 pt=96 ! udpsink host=192.168.1.10 port=8004 alsasrc device=plughw:Set ! audioconvert ! audioresample ! opusenc ! rtpopuspay ! udpsink host=192.168.1.10 port=8005

This works fine with FireFox but Chrome displays 1 frame per ~4 seconds. Audio is always available without any delay.

At this moment, this thread is over 2 years old but I did not found any solution for Chrome + H264 problem. 

I'm using most recent software: raspivid 1.3.12, GStreamer 1.2.0 and Janus 0.2.3.

Maybe I should use VP8 instead of H264? But I cannot make VP8 stream work with Janus (stream seems to be unstable, Janus prints to log 'New stream detected' with new id every second). I'll appreciate any working example of raspivid+gstreamer with H264/VP8 (video with sound on FF and Chrome).

[Hasan Tekin]

You can try these;

gst-launch-1.0 rpicamsrc ! video/x-raw,width=640,height=480 ! x264enc speed-preset=ultrafast tune=zerolatency byte-stream=true bitrate=200 threads=1 ! h264parse config-interval=1 ! rtph264pay ! udpsink host=127.0.0.1 port=8004


gst-launch-1.0 rpicamsrc brightness=85 contrast=40 awb-mode=6 exposure-mode=0 ! video/x-raw,width=640,height=480 ! x264enc speed-preset=ultrafast tune=zerolatency byte-stream=true bitrate=200 threads=1 ! h264parse config-interval=1 ! rtph264pay ! udpsink host=127.0.0.1 port=8004 sync=false

Regards.

[Jarogniew Muszyński]

Works great (tested with RPI3). Thank you very much!