Comment #4 on issue 1729 by
chen...@google.com: Square glitches when encoding VP9
https://bugs.chromium.org/p/webm/issues/detail?id=1729#c4Update:
The square glitches appear in the rate distortion optimization process. In some speed setting, the glitches will appear. For example, --cpu-used = 2 and above.
The cause of the problem is complex. It involves prediction and transform. I put my analysis in the following.
The glitch blocks are on the boundary of the photo and the background.
In the rate distortion optimization process, the block picks the motion vector of its left neighbor to construct a prediction as it is of the smallest rate cost.
However, the left neighbor block finds a reference block from the previous frame that is far away, which means the motion vector is possibly a spurious local minimum.
Using this motion vector, the prediction of the current block is a block with complex texture (the grass) instead of the plain background.
Up to this point, the encoder produces a bad prediction, we expect the tranform and quantization process can mitigate it by coding the difference signal between source and prediction.
The encoder decides to skip the transform and quantization of the block, meaning the final reconstructed and displayed frame is the prediction block (bad and intrusive)!
The encoder's decision to skip the transform and quantization is based on a model based analysis of block signal in the process of rate distortion optimization.
The model based analysis aims to provide a better speed and quality tradeoff for the encoder.
However, it is difficult to adjust the model to suit for this special case.
If we force the encoder to ignore the model's decision to skip the transform and quantization, the glitches disappear.
Based on the above analysis, the issue triggers visual quality issues that is introduced by the model based analysis in the rate distortion optimization.
To solve this particular issue, I recommend to use a different speed setting (--cpu-used=1).
For the long term solution, we could re-evaluate the performance of the model based analysis, and its speed-quality tradeoff.
It might be an option to turn off the model based analysis, or using it conservatively, for example, do not skip the transform and quantization completely,
unless we have a very high confidence.