Saved Files: Evaluating Dropped Frames

[John Laurence Poole]

I understand that dropping frames in a stream might be exemplified in the following:

A stream has 2 seconds of video at 30 frames per second ("fps"):

Second 1: frames 1-20 come through, frame 1 is dropped, frames 22-30 come through

Second 2: frames 1-30 come though.

So in a perfect world there should be 60 frames in the 2 second video file, but because of a frame drop there are only 59.  This is my understanding of what frame drops mean and maybe it is incorrect.  Where I am heading with this is: I would like to be able to assess from a saved MP4 file how many frames were dropped in it.  So, is the nature of the file structure such that given the above 2 second stream, the MP4 file that Moonfire might deliver up though the web interface would have something in it that one could inspect and determine that there should be 60 frames and that frame 21 is missing?

I want to do some experiments on streaming servers and would like to save the streams for subsequent analysis as to how many frames were dropped.  Or is my 2 second MP4 file only going to be 59 frames without it being discernible between which frames the missing frame occurred?

--

John L. Poole

707-812-1323
jlpo...@gmail.com

[Scott Lamb]

On Sat, Feb 12, 2022 at 4:32 PM John Laurence Poole <jlpo...@gmail.com> wrote:

I understand that dropping frames in a stream might be exemplified in the following:

A stream has 2 seconds of video at 30 frames per second ("fps"):

Second 1: frames 1-20 come through, frame 1 is dropped, frames 22-30 come through

Second 2: frames 1-30 come though.

It's probably worse than that. Most video frames are stored as a delta from previous frames. There's typically only a self-contained frame ("I frame" = "intra-coded frame") every couple seconds. (Typically in a camera stream all I frames are "IDR" or "key" frames, which guarantees that not only does this frame no longer use previous frames as a reference, no following frame uses the previous frames as a reference either.) So basically dropping one frame not only means that frame is missing but also the next couple seconds of video may be incorrect (and often have weird artifacts / still regions).

 

So in a perfect world there should be 60 frames in the 2 second video file, but because of a frame drop there are only 59.  This is my understanding of what frame drops mean and maybe it is incorrect.  Where I am heading with this is: I would like to be able to assess from a saved MP4 file how many frames were dropped in it.  So, is the nature of the file structure such that given the above 2 second stream, the MP4 file that Moonfire might deliver up though the web interface would have something in it that one could inspect and determine that there should be 60 frames and that frame 21 is missing?

Moonfire doesn't take note of dropped frames today other than in the log files. ffmpeg didn't report them to us, so I didn't include any provision for this in the schema. Retina does report them, and it's certainly something I could add to the video index schema when I next rework it, which would let us expose an API for it, eventually have some visual indication in the UI, etc.

As for getting it into an exported .mp4 file, I don't think there's a commonly understood format for this. There's something called "RTP Reception Track Hints" that one might expect to do this, but as far as I can tell it just includes a bunch of fairly useless information. I can't figure out what it's good for and don't know of any software that interprets it. We could make up our own format, but of course it'd only be understood by our own tools. I'd be very interested to know if you find any .mp4 files that include this kind of information and/or existing tools that understand it. Or for that matter any other container format such as .mkv.

But if you know your camera sends frames at a fixed rate, the simplest thing might be to look for gaps, which must represent either dropped frames or time adjustments (camera resetting its clock via SNTP).

 

I want to do some experiments on streaming servers and would like to save the streams for subsequent analysis as to how many frames were dropped.  Or is my 2 second MP4 file only going to be 59 frames without it being discernible between which frames the missing frame occurred?

--

John L. Poole

707-812-1323
jlpo...@gmail.com

--
You received this message because you are subscribed to the Google Groups "moonfire-nvr-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moonfire-nvr-us...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/moonfire-nvr-users/CAB74VbLW7OffY8DSB%3Da13L8An6u%3D7WOScYLVXt%2ByV9XLpJ%3DQvQ%40mail.gmail.com.

--

Scott Lamb <https://www.slamb.org/>