Issue 1339 in webm: libvpx master generates invalid vp9 bitstreams

[der… via monorail]

Status: Untriaged
Owner: ----
Labels: Type-Bug Pri-2
Components: libvpx

New issue 1339 by der...@vimeo.com: libvpx master generates invalid vp9 bitstreams
https://bugs.chromium.org/p/webm/issues/detail?id=1339

What is the expected behavior? What do you see instead?

I expect libvpx to output a valid and decodable VP9 bitstream. Neither vpxdec nor ffvp9 can decode it.

What version are you using? On what operating system?

OS: Debian 8.6
gcc: 4.9.2 (Debian 4.9.2-10)
libvpx: v1.6.0-699-ga987496

Can you reproduce using the vpxdec or vpxenc tools? What command line are
you using?

Encode:

vpxenc --codec=vp9 --cpu-used=1 -b 8 --frame-parallel=16 --kf-max-dist=72 --undershoot-pct=5 --overshoot-pct=5 --ivf -o p1.ivf --threads=16 --maxsection-pct=120 --best --passes=2 --pass=1 input.y4m --fpf=pass1-vpx.log --target-bitrate=4600
vpxenc --codec=vp9 --cpu-used=1 -b 8 --frame-parallel=16 --kf-max-dist=72 --undershoot-pct=5 --overshoot-pct=5 --ivf -o p1.ivf --threads=16 --maxsection-pct=120 --best --passes=2 --pass=2 input.y4m --fpf=pass1-vpx.log --target-bitrate=4600

Decode:

$ vpxdec p1.ivf >/dev/null
Warning: Failed to decode frame 2: Memory allocation error
Warning: Additional information: Failed to allocate bool decoder 0

or

$ ffmpeg -i p1.ivf -f null -
[...]
[vp9 @ 0x4306860] Invalid compressed header size
[vp9 @ 0x4321220] Marker bit was set
Error while decoding stream #0:0: Invalid data found when processing input
Last message repeated 1 times
[vp9 @ 0x4266de0] Invalid compressed header size
[vp9 @ 0x42817a0] Marker bit was set
Error while decoding stream #0:0: Invalid data found when processing input
Last message repeated 1 times
[vp9 @ 0x4370f60] Invalid compressed header size
[... continues for thousands of lines ...]

Please provide any additional information below.

The input is rather large, but can be provided upon request.
The output can be located here: https://www.dropbox.com/s/o5e7oze99avh3kv/broken.ivf?dl=0

--
You received this message because:
1. The project was configured to send all issue notifications to this address

You may adjust your notification preferences at:
https://bugs.chromium.org/hosting/settings

[ya… via monorail]

Comment #1 on issue 1339 by ya...@google.com: libvpx master generates invalid vp9 bitstreams
https://bugs.chromium.org/p/webm/issues/detail?id=1339#c1

Thanks for reporting the issue.

[ya… via monorail]

Comment #2 on issue 1339 by ya...@google.com: libvpx master generates invalid vp9 bitstreams
https://bugs.chromium.org/p/webm/issues/detail?id=1339#c2

We will take a look at the encoded stream first, and let you know if we need the source for testing the encoder for investigating the issue.

[ya… via monorail]

Comment #3 on issue 1339 by ya...@google.com: libvpx master generates invalid vp9 bitstreams
https://bugs.chromium.org/p/webm/issues/detail?id=1339#c3

Hi, there,

We were able to reproduce the error reported, but need to look into how the encoder set the marker bit that was not supposed to be set.

Would you share us the input video and information on configure/build/encoding parameters?

Thanks a lot
Yaowu

[der… via monorail]

Comment #4 on issue 1339 by der...@vimeo.com: libvpx master generates invalid vp9 bitstreams
https://bugs.chromium.org/p/webm/issues/detail?id=1339#c4

Hi. Here's a 1 second cut of the clip that I was able to reproduce it with:

https://www.dropbox.com/s/fms115bi0d413l4/finegrain-1second.y4m.xz?dl=0

The libvpx configure/build was trivial:

./configure --enable-vp9-highbitdepth && make -j32 && sudo make install

GCC version and OS are listed in my initial bug report, as are the exact encoding parameters used.

In case it is of further use, here's some relevant info on the CPU it was built/run on:

vendor_id : GenuineIntel
cpu family : 6
model : 79
model name : Intel(R) Xeon(R) CPU @ 2.20GHz
stepping : 0
microcode : 0x1
cpu MHz : 2200.000
cache size : 56320 KB
physical id : 0
siblings : 32
core id : 0
cpu cores : 16
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc eagerfpu pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch xsaveopt fsgsbase bmi1 hle avx2 smep bmi2 erms rtm adx
bogomips : 4400.00
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual

If there's anything else you need, let me know.

[ya… via monorail]

Comment #5 on issue 1339 by ya...@google.com: libvpx master generates invalid vp9 bitstreams
https://bugs.chromium.org/p/webm/issues/detail?id=1339#c5

Thanks for the information.

[ja… via monorail]

Updates:
Owner: ya...@google.com
Status: Assigned

Comment #6 on issue 1339 by ja...@google.com: libvpx master generates invalid vp9 bitstreams
https://bugs.chromium.org/p/webm/issues/detail?id=1339#c6

(No comment was entered for this change.)

[ya… via monorail]

Comment #7 on issue 1339 by ya...@google.com: libvpx master generates invalid vp9 bitstreams
https://bugs.chromium.org/p/webm/issues/detail?id=1339#c7

After investigating this, it turns out that vp9 encoder expects --frame-parallel to be a boolean value, i.e. either 0 (false) or 1(true). 16 has caused unexpected behavior in the encoder.

A fix of the issue is to add range check in vp9 encoder to make sure it is either 0 or 1.

[ya… via monorail]

Updates:
Status: Started

Comment #8 on issue 1339 by ya...@google.com: libvpx master generates invalid vp9 bitstreams
https://bugs.chromium.org/p/webm/issues/detail?id=1339#c8

(No comment was entered for this change.)

[bugdro… via monorail]

Comment #9 on issue 1339 by bugd...@chromium.org: libvpx master generates invalid vp9 bitstreams
https://bugs.chromium.org/p/webm/issues/detail?id=1339#c9

The following revision refers to this bug:
https://chromium.googlesource.com/webm/libvpx/+/0ffbb36ddc5a1be8ee38ab3bdc663fd5ea99da78

commit 0ffbb36ddc5a1be8ee38ab3bdc663fd5ea99da78
Author: Yaowu Xu
Date: Mon Nov 21 18:49:56 2016

Add validation of frame_parallel_decoding_mode

This is a boolean value that is written into bitstream, any value other
than 0 or 1 could have led to unexpected behavior. This commit fix the
issue by adding validation of the value to make sure it is boolean.

BUG=webm:1339

Change-Id: I2d3e69e8dbefcab9a0db9cb39a91a40ce531c5a1

[modify] https://crrev.com/0ffbb36ddc5a1be8ee38ab3bdc663fd5ea99da78/vp9/vp9_cx_iface.c

[ya… via monorail]

Updates:
Status: Fixed

Comment #10 on issue 1339 by ya...@google.com: libvpx master generates invalid vp9 bitstreams
https://bugs.chromium.org/p/webm/issues/detail?id=1339#c10

If you use previous version of source code, simply change your command line to

vpxenc --codec=vp9 --cpu-used=1 -b 8 --frame-parallel=1 --kf-max-dist=72 --undershoot-pct=5 --overshoot-pct=5 --ivf -o p1.ivf --threads=16 --maxsection-pct=120 --best --passes=2 --pass=2 input.y4m --fpf=pass1-vpx.log --target-bitrate=4600

The encoded stream should then work, noticing --frame-parallel=1 instead of 16.

With the fix in #9, the latest code will warn for any value other than 0 or 1 for --frame-parallel.