Forced keyframes in 2 pass may not be coded as keyframes

[Christian Feldmann]

Hi,

We are experimenting with encoding longer video segments with libvpxvp9. But what we need are keyframes at certain positions to be able to split the output bitstream in a certain way. We use the flag VPX_FRAME_IS_KEY when pushing frames into the encoder and run 2 pass encoding. What we found is, that the frames, which are marked this way are not always coded as keyframes in the output. I experimented with the altref test-case and was able to reproduce it also with the latest version in the main branch (see attachment).

We investigated the rate control algorithm for 2 pass and it looks like the lookahead algorith that decides where to place the keyframes in the second pass does not consider the VPX_FRAME_IS_KEY flag at all. It just bases its decision on the stats from the first pass, where the flag was considered. While this results in keyframes in most of the cases, it is not guaranteed. We are currently testing how to fix it. Our idea is:

 - Put a flag into the first pass stats file that indicates if this is a forced keyframe. 

 - In the second pass, check that the same frames are marked as keyframes. Having this different between the passes makes no sense in our eyes.

 - In the function vp9_get_frames_to_next_key, consider the flag (get it from the stats) and thusly enforce a keyframe

Any input would be highly appreciated. Does this sound like a valid solution or are we not considering something? I will also be able to submit a PR.

Christian

[James Zern]

On Mon, Mar 27, 2023 at 4:43 AM 'Christian Feldmann' via Codec Developers <codec...@webmproject.org> wrote:

Hi,

We are experimenting with encoding longer video segments with libvpxvp9. But what we need are keyframes at certain positions to be able to split the output bitstream in a certain way. We use the flag VPX_FRAME_IS_KEY when pushing frames into the encoder and run 2 pass encoding. What we found is, that the frames, which are marked this way are not always coded as keyframes in the output. I experimented with the altref test-case and was able to reproduce it also with the latest version in the main branch (see attachment).

I think you mean VPX_EFLAG_FORCE_KF, but yes this is a known issue which hasn't been given any priority: https://crbug.com/webm/1151. Without lag/alt-ref this should work, though I see in keyframe_test.cc we're only covering VP8 currently.

 

We investigated the rate control algorithm for 2 pass and it looks like the lookahead algorith that decides where to place the keyframes in the second pass does not consider the VPX_FRAME_IS_KEY flag at all. It just bases its decision on the stats from the first pass, where the flag was considered. While this results in keyframes in most of the cases, it is not guaranteed. We are currently testing how to fix it. Our idea is:

 - Put a flag into the first pass stats file that indicates if this is a forced keyframe. 

 - In the second pass, check that the same frames are marked as keyframes. Having this different between the passes makes no sense in our eyes.

 - In the function vp9_get_frames_to_next_key, consider the flag (get it from the stats) and thusly enforce a keyframe

Any input would be highly appreciated. Does this sound like a valid solution or are we not considering something? I will also be able to submit a PR.

2-pass is able to work without alt-ref frames, so an additional flag in stats may not be needed if the frame is forced in pass 1. If you'd like to put together a PR, that would be great. Once that's in place it will be easier for others to comment.