FFmpegFrameGrabber - Recording RTSP/RTP Stream and Packet Loss

[Greg Wilson]

I am recording a camera's RTP stream to an mp4 file on an unreliable wireless network. My goal is to detect missed frames due to packet loss and insert a "blank" frame (a frame that indicates video is missing) in order to complete a recording for the desired timeframe. For instance, If I record for 30 seconds at 25 fps and miss 50 frames then I need to insert blank frames so the recording is actually 30 seconds long, not 28 seconds due to missing frames.

I can successfully record the RTP stream and insert blank frames however the challenge is figuring out where in the recording the frames were dropped. I am using FFmpegFrameGrabber.grabImage() to capture the frame however this method appears to block while it is waiting on frame data with no indication that frame data is delayed or previous frames were missing. For completeness of the problem the connection to the camera is not lost its just that the wireless network is unreliable and there is considerable packet loss resulting in missed frames.

I used  ffprobe  to look at the recorded file's frame data and there does not appear to be gaps in the times reported by the frames. For example: pkt_pts_time, pkt_dts, pkt_pts, best_effort_timestamp_time, pkt_duration all appear to be pretty consistent between frames.

The other thing I did was try to record the system time when the frame was received and use that time to detect missing frames. The problem with that is I think ffmpeg might be caching frame data and delivering some frames in batches, resulting in them being received very quickly so I am not sure that is the best way to detect a dropped frame.

Looking for ideas/thoughts on how to detect the times of the missing frames in order to insert my "blank" frames so that my recorded files will have the correct length.

Thanks in advance!

[Samuel Audet]

To detect dropped frames, the camera needs to provide us with some
timestamps. I guess the first thing I would do if I had to work on that
is to figure out if/where the camera puts timestamps in those RTP
packets, and then see where they come out from FFmpeg's end

Samuel

On 10/16/24 04:53, Greg Wilson wrote:
> I am recording a camera's RTP stream to an mp4 file on an unreliable
> wireless network. My goal is to detect missed frames due to packet loss
> and insert a "blank" frame (a frame that indicates video is missing) in
> order to complete a recording for the desired timeframe. For instance,
> If I record for 30 seconds at 25 fps and miss 50 frames then I need to
> insert blank frames so the recording is actually 30 seconds long, not 28
> seconds due to missing frames.
>
> I can successfully record the RTP stream and insert blank frames however
> the challenge is figuring out where in the recording the frames were
> dropped. I am using FFmpegFrameGrabber.grabImage() to capture the frame
> however this method appears to block while it is waiting on frame data
> with no indication that frame data is delayed or previous frames were
> missing. For completeness of the problem the connection to the camera is
> not lost its just that the wireless network is unreliable and there is
> considerable packet loss resulting in missed frames.
>

> I used *ffprobe*  to look at the recorded file's frame data and there

> does not appear to be gaps in the times reported by the frames. For

> example: *pkt_pts_time*, *pkt_dts*, *pkt_pts*,
> *best_effort_timestamp_time*, *pkt_duration *all appear to be pretty