VP9 quality comparison part #2 - 3DMark03: Mother Nature calls

[Alexey Eromenko]

For those of you, who don't know: 3D Mark series by futuremark is a
beautiful real-time 3D renderings done by means of Direct3D and
powerful 3D accelerators and it makes for a great lossless source for
video encoder comparisons.
3D Mark 03 features "Mature Nature", one of the most beautiful scenes ever.
I captured the RAW video with Fraps (1280x720, HD quality with 8x
anti-aliasing on NVIDIA GTX 560).

Lossless screenshot: (after 11 seconds)
https://docs.google.com/file/d/0BycgkMZbeQOzdVhNVjdva2xoRmc/edit?usp=sharing
VP8 screenshot: (600 kbps after 11 seconds)
https://docs.google.com/file/d/0BycgkMZbeQOzSlVSZVlMZHhKcWs/edit?usp=sharing
VP9 screenshot: (600 kbps after 11 seconds)
https://docs.google.com/file/d/0BycgkMZbeQOzR3BQNHRzSGhpUTg/edit?usp=sharing
H.264 screenshot: (600 kbps after 11 seconds)
https://docs.google.com/file/d/0BycgkMZbeQOzbXpESjdyRGNGWVE/edit?usp=sharing
Quality comparison screenshot: (with H.264)
https://docs.google.com/file/d/0BycgkMZbeQOzLV9PMk0xOGY3T1k/edit?usp=sharing

Problems with VP9:
Turtle's armor is blurred. Stone is blurred. Grass is blurred (near
stone and far grass).
Water is more blurred too.
Open several screenshots in multiple tabs in FireFox and compare.
The VP8 image actually lies, and changed frame number from "3061" to
"3091" yay ! Why is this ?

VP8/VP9 Encoder settings:
./vpxenc --codec=vpX --passes=2 --good --cpu-used=0
--target-bitrate=600 --auto-alt-ref=1 --bias-pct=50 --minsection-pct=0
--maxsection-pct=2000 --lag-in-frames=25 --kf-min-dist=0
--kf-max-dist=9999 --static-thresh=0 --min-q=0 --max-q=63
--arnr-maxframes=7 --arnr-strength=5 --arnr-type=3 -v -o [OUTPUT]
[INPUT]
Version:
v1.2.0-1610-g1e8b3da

H.264 encoder settings:
x264 --pass 1 --preset veryslow --tune film --stats h264.stats
--bitrate 600 -o $OUTPUT $INPUT
x264 --pass 2 --preset veryslow --tune film --stats h264.stats
--bitrate 600 -o $OUTPUT $INPUT

VP8 video: (600 kbps, 2 MB)
https://docs.google.com/file/d/0BycgkMZbeQOzb2tpcEZiaDVCS1U/edit?usp=sharing
VP9 video: (600 kbps, 2 MB)
https://docs.google.com/file/d/0BycgkMZbeQOzR1NFalctX0drUTA/edit?usp=sharing
H.264 video: (600 kbps, 2 MB)
https://docs.google.com/file/d/0BycgkMZbeQOzNDM2em9WaHROeUE/edit?usp=sharing

Source lossless video: (1.15 GB)
https://docs.google.com/file/d/0BycgkMZbeQOzVVdaYllTZ1NhMlE/edit?usp=sharing

--
-Alexey Eromenko "Technologov"

[Pjotrs Ivanovs]

//For those of you, who don't know....  and it makes for a great lossless source for
video encoder comparisons.

This is doubtful i will cite from JCTVC-B084 (  proposal by Microsoft ):

"Unlike natural images, screen contents may not be very smooth. They usually have totally different statistics. For text or graphics contents, it is much sharper and with high contrast. Because of the high contrast, any little artifact may be perceived by users. Thus, coding for screen contents usually require a very high fidelity of the decoded video.

In this document, two new coding tools are presented to improve the coding performance for screen contents. It will be shown that by simply adding two modes into AVC (ITU-T Rec. H.264 | ISO/IEC 14496-10), the coding performance can be much better than the state-of-the-art image/video coding schemes."

you see after Motion Compensation residual from screen content will have a lot of sharp edges, not like smooth surfaces (  ideal for DCT based compression)  produced when encoding natural video images. 

Pjotrs

[Pjotrs Ivanovs]

ok

ssim results

3DMark03-600k.mp4avc.mp4,2254,0.822107,20.865399

3DMark03-600k.vp8.webm,2224,0.804603,17.565131

3DMark03-600k.vp9.webm,2139,0.827616,22.010484

2139/2254 = 95% -> vp9 file is smaller and with better quality

[Alexey Eromenko]

On Wed, Mar 20, 2013 at 3:47 AM, Pjotrs Ivanovs <notdo...@gmail.com> wrote:
> ok
>
> ssim results
>
> 3DMark03-600k.mp4avc.mp4,2254,0.822107,20.865399
> 3DMark03-600k.vp8.webm,2224,0.804603,17.565131
> 3DMark03-600k.vp9.webm,2139,0.827616,22.010484

The actual screenshots show quite a different picture. And how would
you evaluate the original screenshot & video ? As 100% ?

--
-Alexey Eromenko "Technologov"

[Pjotrs Ivanovs]

The actual screenshots show quite a different picture. And how would
you evaluate the original screenshot & video ? As 100% ?

--
-Alexey Eromenko "Technologov"

please see frame-by-frame.png

1) imho cherry picking frames is not  honest tactic

it's ratecontrol issue and as you using --bias-pct=50 you can't expect full vbr behavior

2) second you knowing that vp9 encoder now dont use segmentation, but againg in x264 settings you don't using --aq 0 ( a think default is --aq 1 )

3) --tune film  what?????  why??? is not supposed to use --tune animation

4) where is --psy-rd 0:0 .. ok you using psy-rd... it's ok 

so i compressed by night PeopleOnStreet_2560x1600_30_crop ( VCEG test secuence )

ssim

PeopleOnStreet-avc.mkv,2546,0.853089,28.051319

PeopleOnStreet-vp8.webm,2991,0.870394,32.940272

PeopleOnStreet-vp9.webm,2369,0.870321,32.918308

vp8 overshot bitrate so it's results dont relevant,  ok vp9 have smaller file size sssim value ~33  and  x264 ~ 28

so to cach up ssim value 33.9 i gues there should be  30-50% more bitrate for x264

[Pjotrs Ivanovs]

PeopleOnStreet test files

https://docs.google.com/file/d/0B1_37MKK47IQaHhpbFc5SlctMGs/edit?usp=sharing

AVC encode

https://docs.google.com/file/d/0B1_37MKK47IQNU4yb3dqMmFZczA/edit?usp=sharing

vp8 file

https://docs.google.com/file/d/0B1_37MKK47IQUWtvcGs1aDNzT0k/edit?usp=sharing

vp9

https://docs.google.com/file/d/0B1_37MKK47IQOWJVM0M3Z1RacFE/edit?usp=sharing

yuvssim frame by frame output

https://docs.google.com/file/d/0B1_37MKK47IQTktOSXMxRU1PTXc/edit?usp=sharing

cmd file to run test

[pjotrs]

P.S.

to be able to reproduce ssim results linh to yuvssim provided:

 

yuvssim is win32 console aplication based on avisynth  plugin SSM 0.24

( http://forum.doom9.org/showthread.php?p=370323 )

https://docs.google.com/file/d/0B1_37MKK47IQN3NudmhsSnJXdEU/edit?usp=sharing

there is source and compiled binary

it works identically as original plugin with lumimask = false option,

comand line reference

Usage:  <width> <height> <yuvfile1> <yuvfile2> <encodedfile>

it outputs results to console, and two log files ( all log files comma separated CSV files )

first log file Average SSIM: there writen encoded file name, then encoded file size in KB ( if file don't exist zero is writen ) , SSIM value, scalled SSIM value

( Scaled SSIM value calculated the same way as original plugin do it 100*pow(global,8);          )

second log file frame-by-frame.csv : for each frame writen  encoded file name, size of encoded file, SSIM value.

application  always append new data to log files, newer erasing old content

//And how would 

you evaluate the original screenshot & video ? As 100% ?

if you run aplication with two identical input yuv files SSIM value will be 1, and scaled value 100

when i writed % i was comparing file sizes, so 2139/2254 = 95%  you should understand as 3DMark03-600k.vp9.webm file size ( 2139 kb ) is ~95% of provided mp4 file size ( 2254 ) 

[Ronald S. Bultje]

Hi Alexey,

I've actually gone through this a little bit, and I think you're
raising some good points on specific artifacts that we should try to
get rid of.

A general comment on this particular encode: 600kbps is seriously very
low for this clip, so I doubt you'll ever get anything particularly
great at that bitrate. You'll probably have noticed that x264 produces
enormous high-frequency artifacts all throughout the clip (check the
grass, leaves, water at any point in the clip at 600kbps -
particularly in the moving picture rather than single frames) with
your settings. It also suffers a lot from blockiness artifacts (or
metal fence patterns), e.g. in the water-part in the first 320 frames
of the clip. It drops to well below 25dB at the end of the clip, which
is really awful to watch. Vp9 doesn't have these, but instead has a
lot of blurring across the scene (as you pointed out). I'm not saying
one is better or worse (we're essentially discussing the never-ending
story of which artifact is better), just that if you are expecting
better quality, you might want to increase the bitrate a little. If
you're doing this for test purposes, then it's fine, ignore this
comment.

However, you're right that there are particular sections in the clip
where we appear to be over-blurring or just never introduce any sort
of detail at all. An example is the sections where the butterfly flies
throughout the screen (frame 482-end); even at pretty high bitrates,
like 2mbps, the butterfly leaves a trail of blur, even if the exact
same point in the content had a lot of detail before the butterfly
flew over). I'm not quite sure why that happens (conceptually, if you
use the correct reference frame and correct motion vector, it should
be pretty cheap to get that detail back?), but it seems like a bug
that I hope we can fix.

Also, at the start of the clip, when new content slowly comes in from
the top, a lot of content that enters the screen takes a long while to
actually become clean (i.e. be not blurry), or just stays blurry all
throughout. Again, I'm not quite sure why that happens, but
introducing detail at the initial point where that content comes in
shouldn't be hard, and we have tools in our bitstream format to do
that, so it seems like a bug or shortcoming in the encoder that we
don't make full use of it.

In both cases, thanks for identifying these issues, this is very helpful.

Ronald

> --
> You received this message because you are subscribed to the Google Groups "Codec Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to codec-devel...@webmproject.org.
> To post to this group, send email to codec...@webmproject.org.
> Visit this group at http://groups.google.com/a/webmproject.org/group/codec-devel/?hl=en.
> For more options, visit https://groups.google.com/a/webmproject.org/groups/opt_out.
>
>

[Kieran Kunhya]

You are measuring SSIM but not tuning x264 for SSIM.

[Pjotrs Ivanovs]

Hi Kieran,

You right.  --tune ssim forces adaptive quantization and psy-rd 0

ie -tune ssim equal to --psy-rd 0:0  -aq > 0

--psy-rd 0:0 should be used of course

--aq > 0 should not because vp9 encoder now not using this feature ( but bitsream supports it ) 

running x264 with --aq 2 vs vp9  creates the same bias in test as running  x264 with --aq 0 vs vp8 with --tune=ssim

I will not re do test on PeopleOnStreet_2560x1600_30_crop.yuv because license says it only may used for testing and developing new MPEG standarts

I missed this when first running test, all other VCEG test clips licences doesn't restrictive so VP9 may be tested with them.

I you like i can do another test on Traffic_2560x1600_30_crop with you suggested x264 settings

Pjotrs Ivanovs

ps Traffic_2560x1600_30_crop license

Individuals and organizations extracting sequence from this archive agree that

the sequence and all intellectual property rights therein remain the property of the owner below.

This material may only be used for the purpose of developing, testing and promulgating technology standards.

This material cannot be distributed with charge.

The owner makes no warranties with respect to the material and expressly disclaims any warranties regarding its fitness for any purpose.

Mitsubishi 4-4-4 Video Content

Owner of the sequence

Traffic Owner: Plannet inc. 

Production:Plannet inc. and IMAGICA Corp.

[Paul Wilkins]

Just to clarify, the --tune=ssim option in VP8 and VP9 does not currently result in adaptive quantization.

There is clearly scope in the bit-streams for such (e.g. using segmentation) but we have not yet addressed this in the encoder. I suspect that this clip is one that would benefit quite a bit in terms of psycho-visual quality (and metrics like ssim) from adaptive quantization, as do clips like park-run and park-joy (which are particularly good test cases for adaptive quant). For most clips (in a large test corpus such as YouTube), we have found that the benefit is much less significant, but there are a number of categories of content and use cases where it does seem to make a big difference, so we will certainly be addressing this.

Thanks again for drawing our attention to what is clearly poor encoder behaviour in this clip. We will certainly use it as a test case in the coming weeks as we try and resolve various bit-stream and encoder issues.

Paul Wilkins

[Alexey Eromenko]

OK.
1. I will not change x.264 settings.. much (because those are given by
H.264 expert)
2. I am willing to change VP9 settings, if needed.
3. If you get a significant new milestone in VP9 development ping me,
and I will re-test.

Best wishes,
--
-Alexey Eromenko "Technologov"

[John Koleszar]

The current implementation adapts lambda, not Qp.

I think the general consensus is to turn it on for VP8. It doesn't do
anything for VP9 right now.

On Thu, Mar 28, 2013 at 6:13 PM, xyz <xo2y...@gmail.com> wrote:
>> Just to clarify, the --tune=ssim option in VP8 and VP9 does not currently
>> result in adaptive quantization.
>
>

> I was under the impression tune-ssim enables an activity mask, which I
> assumed works by 'adapting' qp to scene content. Does it currently do
> something different? Or is there a another distinction between its methods
> and what you hope to eventually add to libvpx?
>
> While we're on the subject, VP8 and VP9 defaults to not using the activity
> mask, and puts it under the label of tuning for a specific, artificial
> metric (ssim), which doesn't necessarily connote that the mask would benefit
> normal usage. Is tune=ssim not recommended for 'real' encodes as far as
> visual results are concerned?

[Pjotrs Ivanovs]

Hi Alexey

3) how do you define word significant? is 1% bitrate reduction for same quality significant? may be 5%? will it be possible 

visually detect such changes without using objective metrics use?

2) if you think that for that test clip only frame 330 is important, and given that this part of clip less complex than for example beginning,  you can use lower ( or lowest ) --pct-bias, also you can lower min-q  to force encoder not to overspend bits on keyframes ( because any way encoder on this bitrate for this clip running at low quantizers anyway )

1) for ssim --psy-rd 0:0 --aq 1 is optimal or --tune ssim is optimal,  if you want use other tuning ok, but using --tune film for clip that clearly is not film you give x264 wrong information  

[Pjotrs Ivanovs]

I done some test on VCEG Test clips ( mostly 416x240 and 832x480 resolution ) 

it' looks like for low resolution vp9 quality improvement is less than for higher resolution,

for example same clip content, different resolution:

BasketballDrill_832x480_50

https://docs.google.com/file/d/0B1_37MKK47IQSUpmZmgyRkhaR0U/edit?usp=sharing

BasketballPass_416x240_50

https://docs.google.com/file/d/0B1_37MKK47IQaHNKaWJEVXRwNWc/edit?usp=sharing

Full HD only tested one from ftp://hvc:US88...@ftp.tnt.uni-hannover.de/testsequences

Tennis_1920x1080_24 there for same quality than vp8, vp9 need ~50% less data rate ( on low bitrates )  and ~20% less data rate ( on high bitrates )

ssim graph

https://docs.google.com/file/d/0B1_37MKK47IQM3BCMkFSVlpYS2s/edit?usp=sharing

Tennis-vp9-1000.webm

https://docs.google.com/file/d/0B1_37MKK47IQNUwxVnZxS1l2X2s/edit?usp=sharing

Tennis-avc-aq1-1000.mkv

Tennis-vp9-1000.webm losslesly reencoded  ( for visual comparasion in players ) 

https://docs.google.com/file/d/0B1_37MKK47IQWHhJSU9uSTV2YUU/edit?usp=sharing

all test graphs

https://docs.google.com/file/d/0B1_37MKK47IQVFl1Ry1aWDdYZHc/edit?usp=sharing

used settings: ( ps bitrate showed here were used for 832x480 resolution )

for %%i in (384,512,768,1200,2000) do (

vpxencVP9.exe -w %width% -h %height% --fps=%framerate% --codec=vp8 --passes=2 --best --cpu-used=0 --target-bitrate=%%i --auto-alt-ref=1 --bias-pct=80 --lag-in-frames=25 --kf-max-dist=9999 --static-thresh=0 --min-q=0 --max-q=63 --q-hist=40 --rate-hist=16 --sharpness=1 -v -o %prefix%-vp8-Qoff-%%i.webm %source%.yuv

vpxencVP9.exe -w %width% -h %height% --fps=%framerate% --codec=vp8 --passes=2 --best --cpu-used=0 --target-bitrate=%%i --auto-alt-ref=1 --bias-pct=80 --lag-in-frames=25 --kf-max-dist=9999 --static-thresh=0 --min-q=0 --max-q=63 --q-hist=40 --rate-hist=16 --sharpness=1 --tune=ssim -v -o %prefix%-vp8-Qon-%%i.webm %source%.yuv

vpxencVP9.exe -w %width% -h %height% --fps=%framerate% --codec=vp9 --passes=2 --best --cpu-used=0 --target-bitrate=%%i --auto-alt-ref=1 --bias-pct=80 --lag-in-frames=25 --kf-max-dist=9999 --static-thresh=0 --min-q=0 --max-q=63 --q-hist=40 --rate-hist=16 --sharpness=1 -v -o %prefix%-vp9-%%i.webm %source%.yuv

)

for %%i in (384,512,768,1200,2000) do (

x264 --input-res %width%x%height% --keyint 9999 --fps %framerate% --pass 1 --preset veryslow --bitrate %%i --aq 2 --psy-rd 0:0 -o %prefix%-avc-aq2-%%i.mkv %source%.yuv

x264 --input-res %width%x%height% --keyint 9999 --fps %framerate% --pass 2 --preset veryslow --bitrate %%i --aq 2 --psy-rd 0:0 -o %prefix%-avc-aq2-%%i.mkv %source%.yuv

x264 --input-res %width%x%height% --keyint 9999 --fps %framerate% --pass 1 --preset veryslow --bitrate %%i --aq 0 --psy-rd 0:0 -o %prefix%-avc-aq0-%%i.mkv %source%.yuv

x264 --input-res %width%x%height% --keyint 9999 --fps %framerate% --pass 2 --preset veryslow --bitrate %%i --aq 0 --psy-rd 0:0 -o %prefix%-avc-aq0-%%i.mkv %source%.yuv

x264 --input-res %width%x%height% --keyint 9999 --fps %framerate% --pass 1 --preset veryslow --bitrate %%i --aq 1 --psy-rd 0:0 -o %prefix%-avc-aq1-%%i.mkv %source%.yuv

x264 --input-res %width%x%height% --keyint 9999 --fps %framerate% --pass 2 --preset veryslow --bitrate %%i --aq 1 --psy-rd 0:0 -o %prefix%-avc-aq1-%%i.mkv %source%.yuv

erase x264_2pass.*

Pjotrs Ivanovs

[Pjotrs Ivanovs]

PS forget avc clip

Tennis-avc-aq1-1000.mkv

https://docs.google.com/file/d/0B1_37MKK47IQVm5pUmY4N05KRVk/edit?usp=sharing

[Pjotrs Ivanovs]

PS sorry i made error 

same content different resolutions is  clips keiba, flowervase   ( basketballpass basketball dril have different content )

[Alexey Eromenko]

1. I prefer NOT to change x264 encoder settings. Only VP8/VP9 settings. Why ? Because those settings are provided by x264 expert. They are perfect.  There's an old cliche: If ain't broken - don't fix it.

(maybe I can remove "--tune film" at most...)

2. VP9 loses to x264 significantly. Maybe 30-40% loss.

3. Significant improvement means, that VP9 should give me similar level of detail to what I got with x264 on the clips I test. (like +30% better or so...) The current difference is _BIG_ and not in favor of VP9.

--

You received this message because you are subscribed to the Google Groups "Codec Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to codec-devel...@webmproject.org.
To post to this group, send email to codec...@webmproject.org.
Visit this group at http://groups.google.com/a/webmproject.org/group/codec-devel/?hl=en.
For more options, visit https://groups.google.com/a/webmproject.org/groups/opt_out.
 
 

--
-Alexey Eromenko "Technologov"

[Alexey Eromenko]

4. Your graphs are meaningless for me. Please give actual screenshots - *.png and actual videos.

--
-Alexey Eromenko "Technologov"

[Pjotrs Ivanovs]

Hi,Alexey

1)  ok

2) see messages at Mart 20

and second:

"please see frame-by-frame.png "

you don't like frame 330  but meaningless ( but objective numbers ) in frame  frame-by-frame.png  confirm that  vp9 frame 330 have worse quality than avc frame 330

but what you will say about opening scene? who is better there visually?

3) see 4

4) //Your graphs are meaningless for me. Please give actual screenshots - *.png and actual videos.

Tennis-avc-aq1-1000.mkv

https://docs.google.com/file/d/0B1_37MKK47IQVm5pUmY4N05KRVk/edit?usp=sharing

Tennis-vp9-1000.webm

https://docs.google.com/file/d/0B1_37MKK47IQNUwxVnZxS1l2X2s/edit?usp=sharing

Tennis-vp9-1000.webm losslesly reencoded  ( for visual comparasion in players ) 

https://docs.google.com/file/d/0B1_37MKK47IQWHhJSU9uSTV2YUU/edit?usp=sharing

source here ftp://hvc:US88...@ftp.tnt.uni-hannover.de/testsequences

you can compare visually, and for 3dMark file you can reencode with settings that will give more bitrate on simple scenes and less on complex

[Pjotrs Ivanovs]

//VP9 loses to x264 significantly. Maybe 30-40% loss.

my bitrate guess for Tennis file based on interpolation based on graphs i have,

you is subjective...

so to prove you right that vp9 3dmark have 40%  worse quality...

what does you mean by this? x264 for that clip reques 40% less bitrate for the same quality.... ( or what? )

ok you can check this: encode vp9 with 1000 kbit bitrate and x264 with bitrate that is 40% less ( 600 kbit )

if you right x264 should look visually the same as vp9 for such bitrates

[pjotrs]

Alexey actual video you requested

all tennis vp9 encodes

https://docs.google.com/file/d/0B1_37MKK47IQeFZBNWgtcGRReXM/edit?usp=sharing

Tennis-avc-aq1-1600.mkv

https://docs.google.com/file/d/0B1_37MKK47IQOXZ0bEFhdk1SUlU/edit?usp=sharing

I will try to upload 832x480 vp9 files later

personally i think Tennis-avc-aq1-1600.mkv  looks better than Tennis-vp9-1000.webm

but same bit rate file Tennis-avc-aq1-1000.mkv  have more blocking than Tennis-vp9-1000.webm...

probably Tennis-vp9-1000.webm  have clearly visible blocking only on start scene

[Pjotrs Ivanovs]

832x480 vp9 files (avc and vp8 included )

https://docs.google.com/file/d/0B1_37MKK47IQd1JoMU04aVJjLW8/edit?usp=sharing