PSA: An update to frame drop handling in VideoEncoder and EncodedImageCallback

[spr...@google.com]

tl;dr: We are making some changes to how encoders report dropped frames and the completion of an encoding operation. If you don't develop or maintain an implementation of the VideoEncoder interface you can stop reading now.

The WebRTC VideoEncoder interface uses an interface call EncodedImageCallback to deliver encoded frames as a result of an encoding operation. In addition, if the encoder decides to drop a frame (e.g. due to rate control), the EncodedImageCallback contains a method called OnDroppedFrame. However, this method had a significant drawback: it did not specify which frame was dropped, nor its spatial or simulcast index. Further, it was impossible for WebRTC to understand when all layers of the encoding had completed.

Because of this we've added a new callback method: 

virtual void OnFrameDropped(uint32_t rtp_timestamp,

int spatial_id,

bool is_end_of_temporal_unit) {}

In addition, we've added a new is_end_of_temporal_unit method to the EncodedImage class.

By the end of March 2026 we plan to deprecate and remove the old callback, including the obsolete DropReason enumeration, and will also mark the new callback method pure virtual.

What you need to do:

1) If you have a video encoder implementation that support frame dropping, you must call the new OnFrameDropped method for each layer frame that you drop - even if you previously didn't call OnDroppedFrame.

2) If you have created an implementation of the EncodedImageCallback interface, add the new method there and remove the old one if present.

3) For the last spatial or simulcast layer of an encoding operation, set the is_end_of_temporal_unit flag to true (either in EncodedImage or through the OnFrameDropped callback). Otherwise set that flag to false.

Note that this implies that even if your encoder implementation does not support either spatial/simulcast layers or frame dropping, you must still set the is_end_of_temporal_unit flag to true for each output.

Currently, failing to set this flag won't cause an immediate error, but it may cause undesirable side effects such as WebRTC needlessly holding onto frame buffer resources after the encoding is done. In the future, failing to set this flag or neglecting to call OnFrameDropped may become a strict error.

[spr...@google.com]

An update on this:

On April 14th we landed https://webrtc-review.git.corp.google.com/c/src/+/451940 which deprecates the old OnDroppedFrame() method and effectively makes it a noop.

In two weeks we aim to land https://webrtc-review.git.corp.google.com/c/src/+/466500 which removes the previous OnDroppedFrame() method and DropReason enum and makes the new OnFrameDropped() method pure virtual - making overriding it mandatory if you have subclassed EncodedImageCallback.