[vp9] Where are test vectors for test_libvpx ?

[cch...@gmail.com]

Hi,

    test_libvpx requires a test vector suite.

By google, I found this: http://fate-suite.libav.org/vp9-test-vectors/

But it seems some files required by test_libvpx are missing (say, vp90-2-02-size-lf-1920x1080.webm).

My questions are:

1. Where can I get those missing vp9 files?

2. Those files in above site are the correct ones?

3. Is test_libvpx still the good way to guarantee the completeness of libvpx ?

[Ralph Giles]

On 2014-10-21 6:51 PM, cch...@gmail.com wrote:

> test_libvpx requires a test vector suite.
> By google, I found this: http://fate-suite.libav.org/vp9-test-vectors/
> But it seems some files required by test_libvpx are missing
> (say, vp90-2-02-size-lf-1920x1080.webm).

The 'make test' target downloads the test vectors, including the one you
mention, from http://downloads.webmproject.org/test_data/libvpx/

That url doesn't offer directly listings, so you need the list of test
file names, e.g from test-data.sha1. See the LIBVPX_TEST_DATA variable
in libs.mk

> 3. Is test_libvpx still the good way to guarantee the completeness of
> libvpx ?

I'd say 'make test' is a good place to start. I don't know what kind of
coverage is provides. I think google's continuous integration interfaces
are difficult to share publicly.

-r

[James Zern]

Hi,

On Wed, Oct 22, 2014 at 8:43 PM, Ralph Giles <gi...@thaumas.net> wrote:
> On 2014-10-21 6:51 PM, cch...@gmail.com wrote:
>
>> test_libvpx requires a test vector suite.
>> By google, I found this: http://fate-suite.libav.org/vp9-test-vectors/
>> But it seems some files required by test_libvpx are missing
>> (say, vp90-2-02-size-lf-1920x1080.webm).
>
> The 'make test' target downloads the test vectors, including the one you
> mention, from http://downloads.webmproject.org/test_data/libvpx/
>
> That url doesn't offer directly listings, so you need the list of test
> file names, e.g from test-data.sha1. See the LIBVPX_TEST_DATA variable
> in libs.mk
>

Correct, note the full file list is in test/test-data.mk. The
instructions are buried in the README [1].

>> 3. Is test_libvpx still the good way to guarantee the completeness of
>> libvpx ?
>
> I'd say 'make test' is a good place to start. I don't know what kind of
> coverage is provides. I think google's continuous integration interfaces
> are difficult to share publicly.
>

Tests for the public repo are visible on build.webmproject.org. The
coverage trend for the unit tests is updated nightly [2] and generated
per-commit [3], there is also one for the test vectors themselves [4].
Ideally this would give full coverage, but some system level testing
is needed, especially on the encode side. We do have other tests which
reuse the examples and unit tests in various configurations [5].

[1] https://gerrit.chromium.org/gerrit/gitweb?p=webm/libvpx.git;a=blob;f=README;h=6f864d8591ab652d9e7eab7feacaed4e729a5833;hb=HEAD#l138
[2] http://build.webmproject.org/jenkins/job/libvpx__unit_tests-coverage_trend/
[3] http://build.webmproject.org/jenkins/job/libvpx%20unit%20tests%20coverage/
[4] http://build.webmproject.org/jenkins/job/test%20vectors%20coverage/
[5] http://build.webmproject.org/jenkins/view/libvpx-nightly-tests/

[Ralph Giles]

On 2014-10-22 12:24 PM, 'James Zern' via WebM Discussion wrote:

> Tests for the public repo are visible on build.webmproject.org.

Neat! It would be helpful to add a link to this on the website. e.g. to
www.webmproject.org/code/ and to the 'Developer' menu in the site header.

-r

[James Zern]

Thanks for the suggestion, looks like it got added to /code, under
libvpx. It might due to be moved to a more general location given it's
shared between libvpx and libwebp.

[Ralph Giles]

On 2014-10-27 1:03 PM, 'James Zern' via WebM Discussion wrote:
> Thanks for the suggestion, looks like it got added to /code, under
> libvpx.

Thanks, I see it now.

> It might due to be moved to a more general location given it's
> shared between libvpx and libwebp.

Indeed.

-r