How much error resilient is vp9?

[Fuji Nafiul]

lets say I have vp9 encoded I-frame of 1500 bytes and I am sending it to network 10 times with 150 bytes of chunk each. Also have vp9 encoded subsequent 5 P-frame's of 300 bytes in each P-frame. And I am sending those P-frame's to network 2 times with 150 bytes each.

Now if a single chunk (150 bytes) from the I-frame lost in the network at first (assuming all other sent successfully), will the full I-frame and corresponding 5 P-frame's be un-decodable in the receiver end? 

Is this really true or I am missing something here as it shouldn't be and atleast there should be some sort of minimum resilience..! enlighten me please and thanks in advance. 

[James Zern]

Hi,

The error resilience in vp9 is mostly concerned with past frame dependence and entropy contexts, not loss of segments of a frame. You can see some of the usage in vp9_encoder.c [1].

[1] https://chromium.googlesource.com/webm/libvpx/+/refs/heads/main/vp9/encoder/vp9_encoder.c#814

[Fuji Nafiul]

Thanks for the reply. Yeah I understand a bit now but some times I get confused whenever I see - "Independently decodable column in vp9 encoding" kinda topics.. what about that actually? This is also for just faster decoding not some kind of error resilient feature the, right?  So, ultimately for a single layer it is kinda fully impossible to decode even if a single chunk of byte is missing from encoded data and cant even decode a partially broken image and to implement error resiliency I have to send some redundant data from sender upon packet loss indication on receiver end? 

[Fuji Nafiul]

like I found this flags in google libvpx example - encoder.cpp : 

• /*!\brief Error Resilient flags
•  *
•  * These flags define which error resilient features to enable in the
•  * encoder. The flags are specified through the
•  * vpx_codec_enc_cfg::g_error_resilient variable.
•  */
• typedef uint32_t vpx_codec_er_flags_t;
• /*!\brief Improve resiliency against losses of whole frames */
• #define VPX_ERROR_RESILIENT_DEFAULT 0x1
• /*!\brief The frame partitions are independently decodable by the bool decoder,
•  * meaning that partitions can be decoded even though earlier partitions have
•  * been lost. Note that intra prediction is still done over the partition
•  * boundary. */
• #define VPX_ERROR_RESILIENT_PARTITIONS 0x2

and these in google libvpx - libvpx_google\vpx\vp8cx.h

• /*!\brief Codec control function to set lossless encoding mode.
•    *
•    * VP9 can operate in lossless encoding mode, in which the bitstream
•    * produced will be able to decode and reconstruct a perfect copy of
•    * input source. This control function provides a mean to switch encoder
•    * into lossless coding mode(1) or normal coding mode(0) that may be lossy.
•    *                          0 = lossy coding mode
•    *                          1 = lossless coding mode
•    *
•    *  By default, encoder operates in normal coding mode (maybe lossy).
•    *
•    * Supported in codecs: VP9
•    */
•   VP9E_SET_LOSSLESS,
• 
  
•   /*!\brief Codec control function to set number of tile columns.
•    *
•    * In encoding and decoding, VP9 allows an input image frame be partitioned
•    * into separated vertical tile columns, which can be encoded or decoded
•    * independently. This enables easy implementation of parallel encoding and
•    * decoding. This control requests the encoder to use column tiles in
•    * encoding an input frame, with number of tile columns (in Log2 unit) as
•    * the parameter:
•    *             0 = 1 tile column
•    *             1 = 2 tile columns
•    *             2 = 4 tile columns
•    *             .....
•    *             n = 2**n tile columns
•    * The requested tile columns will be capped by the encoder based on image
•    * size limitations (The minimum width of a tile column is 256 pixels, the
•    * maximum is 4096).
•    *
•    * By default, the value is 6, i.e., the maximum number of tiles supported by
•    * the resolution.
•    *
•    * Supported in codecs: VP9
•    */
•   VP9E_SET_TILE_COLUMNS,
• 
  
•   /*!\brief Codec control function to set number of tile rows.
•    *
•    * In encoding and decoding, VP9 allows an input image frame be partitioned
•    * into separated horizontal tile rows. Tile rows are encoded or decoded
•    * sequentially. Even though encoding/decoding of later tile rows depends on
•    * earlier ones, this allows the encoder to output data packets for tile rows
•    * prior to completely processing all tile rows in a frame, thereby reducing
•    * the latency in processing between input and output. The parameter
•    * for this control describes the number of tile rows, which has a valid
•    * range [0, 2]:
•    *            0 = 1 tile row
•    *            1 = 2 tile rows
•    *            2 = 4 tile rows
•    *
•    * By default, the value is 0, i.e. one single row tile for entire image.
•    *
•    * Supported in codecs: VP9
•    */
•   VP9E_SET_TILE_ROWS,

which I thought the indication of reconstructing partially broken video frame from independently decodable tile's/column'ss/partition's over a lossy network. So, if there are 10 partitions in a frame and I send them to network in 20 packets and still it won't be decodable if a single chunk is missing, right?

[James Zern]

On Tue, Sep 13, 2022 at 11:52 PM Fuji Nafiul <nafiu...@gmail.com> wrote:

like I found this flags in google libvpx example - encoder.cpp : 

• /*!\brief Error Resilient flags
•  *
•  * These flags define which error resilient features to enable in the
•  * encoder. The flags are specified through the
•  * vpx_codec_enc_cfg::g_error_resilient variable.
•  */
• typedef uint32_t vpx_codec_er_flags_t;
• /*!\brief Improve resiliency against losses of whole frames */
• #define VPX_ERROR_RESILIENT_DEFAULT 0x1
• /*!\brief The frame partitions are independently decodable by the bool decoder,
•  * meaning that partitions can be decoded even though earlier partitions have
•  * been lost. Note that intra prediction is still done over the partition
•  * boundary. */
• #define VPX_ERROR_RESILIENT_PARTITIONS 0x2

I think I mentioned in an earlier reply or thread, this is VP8 only and deals with token partitions. I added a note:
https://chromium.googlesource.com/webm/libvpx/+/3cd417b6d2380b993f33a0d8f342e4d16717d16e/vpx/vpx_encoder.h#140

Tile columns in VP9 can be decoded independently, but libvpx will still report an error if decoding any of them or the surrounding bitstream fails.

[James Zern]

SVC can also provide some level of resilience if lower layers are available:

https://chromium.googlesource.com/webm/libvpx/+/417adb3aca7bbdc85426bf01b1240eb84a91d4d7/vpx/vpx_encoder.h#643

[Fuji Nafiul]

yeah we implemented SVC error resiliency already, so whenever any upper layer is missing but lower layers are available we shifted to low resolution. we are approaching to NACK for i-frame only condition in case of packet loss with a threshold along with SVC layer error resiliency.  Yeah you said some of these features for VP8 but I found VP9 mentions in some comments, thats why I asked with the comments. You said "Tile columns in VP9 can be decoded independently, but libvpx will still report an error if decoding any of them or the surrounding bitstream fails." - what does this mean? does this mean with some missing packets, maybe I will be able to decode 2/3 (independent) of 4 vertical columns but it will not be "viewable!" so it will give an error?

[James Zern]

On Tue, Sep 20, 2022 at 8:48 PM Fuji Nafiul <nafiu...@gmail.com> wrote:

yeah we implemented SVC error resiliency already, so whenever any upper layer is missing but lower layers are available we shifted to low resolution. we are approaching to NACK for i-frame only condition in case of packet loss with a threshold along with SVC layer error resiliency.  Yeah you said some of these features for VP8 but I found VP9 mentions in some comments, thats why I asked with the comments. You said "Tile columns in VP9 can be decoded independently, but libvpx will still report an error if decoding any of them or the surrounding bitstream fails." - what does this mean? does this mean with some missing packets, maybe I will be able to decode 2/3 (independent) of 4 vertical columns but it will not be "viewable!" so it will give an error?

The parsing of the bitstream will fail, the library will return an error and not emit a frame. Technically the decoded data up to that point could be displayed, but changes would be needed in the library to support this.