Cygwin libvpx tests no longer build

[Brian Inglis]

Hi folks,

Looking to adopt libvpx Cygwin (and Mingw cross) package to fix CVE, if I can
get tests to build and run, and pass distro acceptance criteria.
Previous maintainer built up to 1.7 ABI 5 in 2018:

https://cygwin.com/packages/summary/libvpx-src.html

but latest release tests will not build test_rc_interface.exe - please see
attached log, Fedora source for libvpx.ver, and patch to allow _FORTIFY_SOURCE=2.
I am still digging in the Makefiles, but feel I must be missing something
obvious about the test builds, like do they not support or include build tree
library search?

The shared library is built manually in the cygport recipe linked from the above
web page, but tests are usually built against static libs, and some test
programs are built properly: test_libvpx (135MB) and test_intra_pred_speed.

--
Take care. Thanks, Brian Inglis Calgary, Alberta, Canada

La perfection est atteinte Perfection is achieved
non pas lorsqu'il n'y a plus rien à ajouter not when there is no more to add
mais lorsqu'il n'y a plus rien à retirer but when there is no more to cut
-- Antoine de Saint-Exupéry

[James Zern]

Hi Brian,

On Tue, Oct 17, 2023 at 4:43 PM Brian Inglis <Brian....@shaw.ca> wrote:

Hi folks,

Looking to adopt libvpx Cygwin (and Mingw cross) package to fix CVE, if I can
get tests to build and run, and pass distro acceptance criteria.
Previous maintainer built up to 1.7 ABI 5 in 2018:

        https://cygwin.com/packages/summary/libvpx-src.html

but latest release tests will not build test_rc_interface.exe - please see
attached log, Fedora source for libvpx.ver, and patch to allow _FORTIFY_SOURCE=2.
I am still digging in the Makefiles, but feel I must be missing something
obvious about the test builds, like do they not support or include build tree
library search?

make V=1 will add a little more detail to the log.

 

The shared library is built manually in the cygport recipe linked from the above
web page, but tests are usually built against static libs, and some test
programs are built properly: test_libvpx (135MB) and test_intra_pred_speed.

Can you share the script being used to do the build? A static build works locally for me and cygwin will fail with --enable-shared, but I see libvpx.dll.a referenced in the log:

2023-10-17T17:06:29.6518816Z /usr/lib/gcc/x86_64-pc-cygwin/11/../../../../x86_64-pc-cygwin/bin/ld: ./libvpxrc.a(vp9_aq_cyclicrefresh.c.o): in function `vp9_cyclic_refresh_free':
2023-10-17T17:06:29.6525109Z /usr/src/debug/libvpx-1.13.1-1/vp9/encoder/vp9_aq_cyclicrefresh.c:55: multiple definition of `vp9_cyclic_refresh_free'; ./libvpx.dll.a(cygvpx_8_dll_d000554.o):(.text+0x0): first defined here

 

--
Take care. Thanks, Brian Inglis              Calgary, Alberta, Canada

La perfection est atteinte                   Perfection is achieved
non pas lorsqu'il n'y a plus rien à ajouter  not when there is no more to add
mais lorsqu'il n'y a plus rien à retirer     but when there is no more to cut
                                   -- Antoine de Saint-Exupéry

--
You received this message because you are subscribed to the Google Groups "WebM Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to webm-discuss...@webmproject.org.
To view this discussion on the web visit https://groups.google.com/a/webmproject.org/d/msgid/webm-discuss/d7ac02b4-e8f9-4245-9215-6c964bc234ac%40Shaw.ca.

[Brian Inglis]

Sorry - expected CCed reply - then checked here. 

Page linked above:

https://cygwin.com/packages/summary/libvpx-src.html

has link to packaging repo:

https://cygwin.com/cgit/cygwin-packages/libvpx/

where the playground branch has the updates to the package build control script definitions in 

https://cygwin.com/cgit/cygwin-packages/libvpx/tree/libvpx.cygport?h=playground

including replacements for src_compile() `make` function (which defaults to autotools),  

where `lndirs` links the sources into the corresponding build dirs, 

"manually" builds the shared library as a Cygwin dll, 

relinks the exes against that, 

plus src_install() `make install` function. 

The original CI build is at:

https://cygwin.com/cgi-bin2/jobs.cgi?id=7122&srcpkg=libvpx&user=Brian+Inglis

linking to the packaging repo updated and the original CI build log:

https://github.com/cygwin/scallywag/actions/runs/6550153924

and the latest verbose CI build at:

https://cygwin.com/cgi-bin2/jobs.cgi?id=7145&srcpkg=libvpx&user=Brian+Inglis

and log at:

https://github.com/cygwin/scallywag/actions/runs/6636190342

I will skip deleting test_rc_interface and see if the others build okay. 

Well that built okay but the test programs don't run with 32-bit relocations?, 

so I'll retry by not rebuilding all test programs:

https://cygwin.com/cgit/cygwin-packages/libvpx/commit/?h=playground

run:

https://cygwin.com/cgi-bin2/jobs.cgi?id=7147&srcpkg=libvpx&user=Brian+Inglis

log:

https://github.com/cygwin/scallywag/actions/runs/6645679387

--

Take care. Thanks, Brian Inglis              Calgary, Alberta, Canada

La perfection est atteinte                           Perfection is achieved
non pas lorsqu'il n'y a plus rien à ajouter   not when there is no more to add
mais lorsqu'il n'y a plus rien à retirer         but when there is no more to cut
                                   -- Antoine de Saint-Exupéry

On Tuesday, October 17, 2023 at 6:18:14 p.m. UTC-6 James Zern wrote:

Hi Brian,

On Tue, Oct 17, 2023 at 4:43 PM Brian Inglis <Brian....@shaw.ca> wrote:

Hi folks,

Looking to adopt libvpx Cygwin (and Mingw cross) package to fix CVE, if I can
get tests to build and run, and pass distro acceptance criteria.
Previous maintainer built up to 1.7 ABI 5 in 2018:

        https://cygwin.com/packages/summary/libvpx-src.html

but latest release tests will not build test_rc_interface.exe - please see
attached log, Fedora source for libvpx.ver, and patch to allow _FORTIFY_SOURCE=2.
I am still digging in the Makefiles, but feel I must be missing something
obvious about the test builds, like do they not support or include build tree
library search?

make V=1 will add a little more detail to the log.

 

The shared library is built manually in the cygport recipe linked from the above
web page, but tests are usually built against static libs, and some test
programs are built properly: test_libvpx (135MB) and test_intra_pred_speed.

Can you share the script being used to do the build? A static build works locally for me and cygwin will fail with --enable-shared, but I see libvpx.dll.a referenced in the log:

2023-10-17T17:06:29.6518816Z /usr/lib/gcc/x86_64-pc-cygwin/11/../../../../x86_64-pc-cygwin/bin/ld: ./libvpxrc.a(vp9_aq_cyclicrefresh.c.o): in function `vp9_cyclic_refresh_free':
2023-10-17T17:06:29.6525109Z /usr/src/debug/libvpx-1.13.1-1/vp9/encoder/vp9_aq_cyclicrefresh.c:55: multiple definition of `vp9_cyclic_refresh_free'; ./libvpx.dll.a(cygvpx_8_dll_d000554.o):(.text+0x0): first defined here

 

To unsubscribe from this group and stop receiving emails from it, send an email to webm-discuss...@webmproject.org.

[James Zern]

Hi Brian,

On Wed, Oct 25, 2023 at 10:24 PM Brian Inglis <Brian....@shaw.ca> wrote:

Duh!
Answering some of my own questions running commands from re-reading the README:

Thanks for the updates and working through this. Are there any other open questions or issues? Anything we should pull back into libvpx sources?

 

$ LIBVPX_TEST_DATA_PATH=../libvpx-test-data make -n testdata
for t in libs examples tools docs; do \
      make --no-print-directory target=$t testdata || exit $?;\
         done
echo "    [DOWNLOAD] ../libvpx-test-data/bus_352x288_420_f20_b8.yuv"
( \
   trap 'rm -f ../libvpx-test-data/bus_352x288_420_f20_b8.yuv' INT TERM; \
   curl="curl -S -s --retry 1 -L -o
../libvpx-test-data/bus_352x288_420_f20_b8.yuv
https://storage.googleapis.com/downloads.webmproject.org/test_data/libvpx/bus_352x288_420_f20_b8.yuv";
\
   $curl; ret=$?; \
   case "$ret" in \
     18) $curl -C - ;; \
     *) exit $ret ;; \
   esac \
)
...
[ -x "$(which sha1sum)" ] && sha1sum=sha1sum;\
           [ -x "$(which shasum)" ] && sha1sum=shasum;\
           [ -x "$(which sha1)" ] && sha1sum=sha1;\
           if [ -n "${sha1sum}" ]; then\
             set -e;\
             echo "Checking test data:";\

answers the where question:

https://storage.googleapis.com/downloads.webmproject.org/test_data/libvpx/

and the sha1 question:
note that under Cygwin (only?), if `which` returns nothing `test -x` is true:

$ for sh in bash dash sh; do
        echo $sh:; $sh -vxc '[ -x ]; echo $?'; $sh -vxc 'test -x; echo $?'; done
bash:
[ -x ]; echo $?
+ '[' -x ']'
+ echo 0
0
test -x; echo $?
+ test -x
+ echo 0
0
dash:
+ [ -x ]
+ echo 0
0
+ test -x
+ echo 0
0
sh:
+ [ -x ]
+ echo 0
0
+ test -x
+ echo 0
0

so test the expected path explicitly e.g.

$ test -x /usr/bin/sha1sum; echo $?
0
$ test -x /usr/bin/sha1; echo $?
1

or use which rc to test for executable existence e.g.

$ for sha in sha1sum shasum sha1; do sha1sum=$(which $sha) && break; done
$ echo $sha1sum
/usr/bin/sha1sum

--
Take care. Thanks, Brian Inglis              Calgary, Alberta, Canada

La perfection est atteinte                   Perfection is achieved
non pas lorsqu'il n'y a plus rien à ajouter  not when there is no more to add
mais lorsqu'il n'y a plus rien à retirer     but when there is no more to cut
                                 -- Antoine de Saint-Exupéry

On 2023-10-25 22:27, Brian Inglis wrote:
> Hi folks,
>
> That seems to have done the trick.
> The shared library (cygvpx-8.dll, to avoid collisions with Mingw cross-builds of
> libvpx-8.dll, or native Windows builds) is relinked with the vpxenc/dec
> utilities and the test_... programs remain static, which is normal in these builds.
> Both unstripped and stripped utilities display --help normally.
>
> $ du libvpx-1.13.1-1.x86_64/{build,inst/usr/bin}/*.{dll,exe}
> 23M     libvpx-1.13.1-1.x86_64/build/cygvpx-8.dll
> 26M     libvpx-1.13.1-1.x86_64/build/test_intra_pred_speed.exe
> 158M    libvpx-1.13.1-1.x86_64/build/test_libvpx.exe
> 33M     libvpx-1.13.1-1.x86_64/build/test_rc_interface.exe
> 3.3M    libvpx-1.13.1-1.x86_64/build/vpxdec.exe
> 3.5M    libvpx-1.13.1-1.x86_64/build/vpxenc.exe
> 3.4M    libvpx-1.13.1-1.x86_64/inst/usr/bin/cygvpx-8.dll
> 680K    libvpx-1.13.1-1.x86_64/inst/usr/bin/vpxdec.exe
> 700K    libvpx-1.13.1-1.x86_64/inst/usr/bin/vpxenc.exe
> $ ldd libvpx-1.13.1-1.x86_64/build/vpxenc
>          ntdll.dll => /mnt/c/WINDOWS/SYSTEM32/ntdll.dll (0x7fffc0890000)
>          KERNEL32.DLL => /mnt/c/WINDOWS/System32/KERNEL32.DLL (0x7fffbea00000)
>          KERNELBASE.dll => /mnt/c/WINDOWS/System32/KERNELBASE.dll (0x7fffbe350000)
>          cygvpx-8.dll =>
> /usr/src/libvpx/libvpx-1.13.1-1.x86_64/build/cygvpx-8.dll (0x583910000)
>          cygwin1.dll => /usr/bin/cygwin1.dll (0x7fffafd10000)
>          cygstdc++-6.dll => /usr/bin/cygstdc++-6.dll (0x57b190000)
>          cyggcc_s-seh-1.dll => /usr/bin/cyggcc_s-seh-1.dll (0x50caa0000)
> $ ldd libvpx-1.13.1-1.x86_64/inst/usr/bin/vpxdec
>          ntdll.dll => /mnt/c/WINDOWS/SYSTEM32/ntdll.dll (0x7fffc0890000)
>          KERNEL32.DLL => /mnt/c/WINDOWS/System32/KERNEL32.DLL (0x7fffbea00000)
>          KERNELBASE.dll => /mnt/c/WINDOWS/System32/KERNELBASE.dll (0x7fffbe350000)
>          cygvpx-8.dll =>
> /usr/src/libvpx/libvpx-1.13.1-1.x86_64/inst/usr/bin/cygvpx-8.dll (0x583910000)
>          cygwin1.dll => /usr/bin/cygwin1.dll (0x7fffafd10000)
>          cygstdc++-6.dll => /usr/bin/cygstdc++-6.dll (0x57b190000)
>          cyggcc_s-seh-1.dll => /usr/bin/cyggcc_s-seh-1.dll (0x50caa0000)
>
> It also appears that 1.4GB of tests were downloaded.
>
> Where is this done in the makefiles or scripts, and from where, as I need to
> note that change for this release, as this is non-standard, but understandable.
> It is not a problem for me, although some others may expect the test files to be
> included in the source package, to allow offline builds and test, which may be
> debatable in this case.
>
> I also noticed that there was a failure to find *sha1* using `which` rather than
> the usual sha...sum utilities, and it appears that .md5 may have been downloaded
> and used instead.
>
> The tests took 4-5 hours to run both locally and in CI, and generated nearly 6MB
> of logs with only 5 failed tests in CI and 2 locally shown below.
> Are any of those test failures particularly significant?
>
> 2023-10-25T22:02:32.5192787Z [  FAILED  ] C/AddNoiseTest.CheckNoiseAdded/1,
> where GetParam() = (4.4, 0x100700050) (0 ms)
> 2023-10-25T22:02:32.5937058Z [  FAILED  ] 1 test, listed below:
> 2023-10-25T22:02:32.5937965Z [  FAILED  ] C/AddNoiseTest.CheckNoiseAdded/1,
> where GetParam() = (4.4, 0x100700050)
> 2023-10-25T22:02:32.5940641Z  1 FAILED TEST
> 2023-10-25T22:50:15.5461070Z [  FAILED  ]
> VP9FrameSizeTestsLarge.TestInvalidSizes (4681 ms)
> 2023-10-25T23:08:42.1763744Z [  FAILED  ] 1 test, listed below:
> 2023-10-25T23:08:42.1764436Z [  FAILED  ] VP9FrameSizeTestsLarge.TestInvalidSizes
> 2023-10-25T23:08:42.1765185Z  1 FAILED TEST
> 2023-10-25T23:08:49.8589827Z [  FAILED  ] VP9FrameSizeTestsLarge.ValidSizes
> (4767 ms)
> 2023-10-25T23:27:35.7948274Z [  FAILED  ] SSE2/AddNoiseTest.CheckNoiseAdded/1,
> where GetParam() = (4.4, 0x1008fb000) (0 ms)
> 2023-10-25T23:27:35.8472764Z [  FAILED  ] 2 tests, listed below:
> 2023-10-25T23:27:35.8473611Z [  FAILED  ] VP9FrameSizeTestsLarge.ValidSizes
> 2023-10-25T23:27:35.8474641Z [  FAILED  ] SSE2/AddNoiseTest.CheckNoiseAdded/1,
> where GetParam() = (4.4, 0x1008fb000)
> 2023-10-25T23:27:35.8475565Z  2 FAILED TESTS
> 2023-10-26T00:05:24.2865417Z [  FAILED  ]
> VP9/RealtimeTest.IntegerOverflowLarge/0, where GetParam() = (0x100c7c9b8, 0)
> (4699 ms)
> 2023-10-26T00:09:14.0673899Z [  FAILED  ] 1 test, listed below:
> 2023-10-26T00:09:14.0675351Z [  FAILED  ]
> VP9/RealtimeTest.IntegerOverflowLarge/0, where GetParam() = (0x100c7c9b8, 0)
> 2023-10-26T00:09:14.0676513Z  1 FAILED TEST
>
> 15:59:20.817070 00:00:00.000085 [  FAILED  ] C/AddNoiseTest.CheckNoiseAdded/1,
> where GetParam() = (4.4, 0x100700050) (0 ms)
> 15:59:20.891232 00:00:00.000019 [  FAILED  ] 1 test, listed below:
> 15:59:20.891251 00:00:00.000019 [  FAILED  ] C/AddNoiseTest.CheckNoiseAdded/1,
> where GetParam() = (4.4, 0x100700050)
> 15:59:20.891294 00:00:00.000013  1 FAILED TEST
> 17:29:23.984830 00:00:00.000042 [  FAILED  ]
> SSE2/AddNoiseTest.CheckNoiseAdded/1, where GetParam() = (4.4, 0x1008fb000) (28 ms)
> 17:29:24.051839 00:00:00.000021 [  FAILED  ] 1 test, listed below:
> 17:29:24.051863 00:00:00.000024 [  FAILED  ]
> SSE2/AddNoiseTest.CheckNoiseAdded/1, where GetParam() = (4.4, 0x1008fb000)
> 17:29:24.051912 00:00:00.000014  1 FAILED TEST
>
> Thanks for any help you can provide in allowing us to get this updated package
> released.

[Brian Inglis]

On 2023-10-26 13:37, James Zern wrote:
> Note that under Cygwin, Msys2, and Mingw64 (only?), if `which` returns nothing `test -x` is true:

Thanks James,

Looks like test -x as above is an issue under Cygwin, Msys2, and Mingw64; may be
the same with all unary tests; trashed my Unix VM to rebuild larger to upgrade,
so can not compare right now.

[Brian Inglis]

--
Take care. Thanks, Brian Inglis Calgary, Alberta, Canada

La perfection est atteinte Perfection is achieved
non pas lorsqu'il n'y a plus rien à ajouter not when there is no more to add
mais lorsqu'il n'y a plus rien à retirer but when there is no more to cut
-- Antoine de Saint-Exupéry

> https://groups.google.com/a/webmproject.org/d/msgid/webm-discuss/d7ac02b4-e8f9-4245-9215-6c964bc234ac%40Shaw.ca <https://groups.google.com/a/webmproject.org/d/msgid/webm-discuss/d7ac02b4-e8f9-4245-9215-6c964bc234ac%40Shaw.ca>.

[Brian Inglis]

[Brian Inglis]

Duh!
Answering some of my own questions running commands from re-reading the README:

note that under Cygwin (only?), if `which` returns nothing `test -x` is true:

--
Take care. Thanks, Brian Inglis Calgary, Alberta, Canada

La perfection est atteinte Perfection is achieved
non pas lorsqu'il n'y a plus rien à ajouter not when there is no more to add
mais lorsqu'il n'y a plus rien à retirer but when there is no more to cut
-- Antoine de Saint-Exupéry

On 2023-10-25 22:27, Brian Inglis wrote:

[James Zern]

Thanks, I missed your point here initially. If we have an empty return from `which` it might be problematic with `test -x`. It's worth noting [ -x "" ] fails, which is what is included in libs.mk. It's possible there's an issue with quote expansion, but this works in my cygwin setup both before upgrading with "GNU bash, version 4.4.12(3)-release (x86_64-unknown-cygwin)" and after with "GNU bash, version 5.2.15(3)-release (x86_64-pc-cygwin)". SHELL should default to /bin/sh in make. I have sha1sum and shasum installed. Before making a change, I'd like to understand what is different in your environment.

[James Zern]

On Thu, Oct 26, 2023 at 7:30 PM Brian Inglis <Brian....@shaw.ca> wrote:

 > The tests took 4-5 hours to run both locally and in CI, and generated nearly 6MB
 > of logs with only 5 failed tests in CI and 2 locally shown below.
 > Are any of those test failures particularly significant?
 >

It would be helpful to see the error text of the failures. The size related ones should pass on all platforms, unless there's an allocation issue. I'm running a version now.

[James Zern]

On Fri, Oct 27, 2023 at 12:06 PM James Zern <jz...@google.com> wrote:

On Thu, Oct 26, 2023 at 7:30 PM Brian Inglis <Brian....@shaw.ca> wrote:

 > The tests took 4-5 hours to run both locally and in CI, and generated nearly 6MB
 > of logs with only 5 failed tests in CI and 2 locally shown below.
 > Are any of those test failures particularly significant?
 >

It would be helpful to see the error text of the failures. The size related ones should pass on all platforms, unless there's an allocation issue. I'm running a version now.

I forgot to mention, you can speed up the test run with `make -j test` which will use 10 shards by default. If you have more cores you can try `make -j test NUM_SHARDS=$N SHARDS=$(seq 1 $(( N - 1 )))`

[Brian Inglis]

On 2023-10-27 13:08, James Zern wrote:
> On Fri, Oct 27, 2023 at 12:06 PM James Zern wrote:

>> On Thu, Oct 26, 2023 at 7:30 PM Brian Inglis wrote:
>>> The tests took 4-5 hours to run both locally and in CI, and generated
>>> nearly 6MB of logs with only 5 failed tests in CI and 2 locally shown
>>> below.
>>> Are any of those test failures particularly significant?

>> It would be helpful to see the error text of the failures. The size related
>> ones should pass on all platforms, unless there's an allocation issue. I'm
>> running a version now.

The config size is 16K*16K; real 16GB; virtual 32GB across 2 drives.
I do not see any GNU dejagnu/expect like test output.
What is the best way to do that for just add_noise_test.cc?
Packages, autotools, and g++ I mainly deal with using boilerplate commands from
logs!
What test data is required? Just the args shown in the error message?

> I forgot to mention, you can speed up the test run with `make -j test` which
> will use 10 shards by default. If you have more cores you can try `make -j test
> NUM_SHARDS=$N SHARDS=$(seq 1 $(( N - 1 )))`

Hah I wish! Old retail Acer AMD quad 3.5GHz with hard drives until it dies ;^<

I will add the test definitions with $(nproc) for $N - that could help a little.

[Brian Inglis]

On 2023-10-27 23:01, Brian Inglis wrote:
> On 2023-10-27 13:08, James Zern wrote:
>> On Fri, Oct 27, 2023 at 12:06 PM James Zern wrote:
>>> On Thu, Oct 26, 2023 at 7:30 PM Brian Inglis wrote:
>>>> The tests took 4-5 hours to run both locally and in CI, and generated
>>>> nearly 6MB of logs with only 5 failed tests in CI and 2 locally shown
>>>> below.
>>>> Are any of those test failures particularly significant?
>
>>> It would be helpful to see the error text of the failures. The size related
>>> ones should pass on all platforms, unless there's an allocation issue. I'm
>>> running a version now.
>
> The config size is 16K*16K; real 16GB; virtual 32GB across 2 drives.
> I do not see any GNU dejagnu/expect like test output.
> What is the best way to do that for just add_noise_test.cc?
> Packages, autotools, and g++ I mainly deal with using boilerplate commands from
> logs!
> What test data is required? Just the args shown in the error message?
>
>> I forgot to mention, you can speed up the test run with `make -j test` which
>> will use 10 shards by default. If you have more cores you can try `make -j
>> test NUM_SHARDS=$N SHARDS=$(seq 1 $(( N - 1 )))`
>
> Hah I wish! Old retail Acer AMD quad 3.5GHz with hard drives until it dies ;^<
>
> I will add the test definitions with $(nproc) for $N - that could help a little.

Hi James,

Well that eliminated any test failures on my local system and got rid of the C
AddNoiseTest failures in CI - see attached log extracts.
Adding /proc/meminfo and /proc/cpuinfo output to tests.

[James Zern]

On Sat, Oct 28, 2023 at 7:38 AM Brian Inglis <Brian....@shaw.ca> wrote:
>
> On 2023-10-27 23:01, Brian Inglis wrote:
> > On 2023-10-27 13:08, James Zern wrote:
> >> On Fri, Oct 27, 2023 at 12:06 PM James Zern wrote:
> >>> On Thu, Oct 26, 2023 at 7:30 PM Brian Inglis wrote:
> >>>> The tests took 4-5 hours to run both locally and in CI, and generated
> >>>> nearly 6MB of logs with only 5 failed tests in CI and 2 locally shown
> >>>> below.
> >>>> Are any of those test failures particularly significant?
> >
> >>> It would be helpful to see the error text of the failures. The size related
> >>> ones should pass on all platforms, unless there's an allocation issue. I'm
> >>> running a version now.
> >
> > The config size is 16K*16K; real 16GB; virtual 32GB across 2 drives.
> > I do not see any GNU dejagnu/expect like test output.
> > What is the best way to do that for just add_noise_test.cc?

You can use the --gtest_filter option to limit the tests being run.
For example: --gtest_filter=*AddNoiseTest*

> > Packages, autotools, and g++ I mainly deal with using boilerplate commands from
> > logs!
> > What test data is required? Just the args shown in the error message?
> >
> >> I forgot to mention, you can speed up the test run with `make -j test` which
> >> will use 10 shards by default. If you have more cores you can try `make -j
> >> test NUM_SHARDS=$N SHARDS=$(seq 1 $(( N - 1 )))`
> >
> > Hah I wish! Old retail Acer AMD quad 3.5GHz with hard drives until it dies ;^<
> >
> > I will add the test definitions with $(nproc) for $N - that could help a little.
>
> Hi James,
>
> Well that eliminated any test failures on my local system and got rid of the C
> AddNoiseTest failures in CI - see attached log extracts.
> Adding /proc/meminfo and /proc/cpuinfo output to tests.
>

I see the AddNoiseTest failures locally as well. Changing the order of
the tests seems to affect the behavior. I'll file a bug for this.

As for TestInvalidSizes:
scallywag.log:2023-10-28T06:10:37.2906711Z [ RUN ]
VP9FrameSizeTestsLarge.TestInvalidSizes
scallywag.log-2023-10-28T06:10:39.5551590Z
test/encode_test_driver.cc:35: Failure
This only gets run when configured with --size-limit. How are you
configuring the library? I'll check the behavior with this option.

[James Zern]

https://bugs.chromium.org/p/webm/issues/detail?id=1826

> As for TestInvalidSizes:
> scallywag.log:2023-10-28T06:10:37.2906711Z [ RUN ]
> VP9FrameSizeTestsLarge.TestInvalidSizes
> scallywag.log-2023-10-28T06:10:39.5551590Z
> test/encode_test_driver.cc:35: Failure
> This only gets run when configured with --size-limit. How are you
> configuring the library? I'll check the behavior with this option.
>

I haven't been able to reproduce this failure. The log text was
slightly cut off, if you could include the full error and
configuration that might help.

[Brian Inglis]

On 2023-10-30 13:47, James Zern wrote:
> On Sat, Oct 28, 2023 at 7:38 AM Brian Inglis <Brian....@shaw.ca> wrote:
>>
>> On 2023-10-27 23:01, Brian Inglis wrote:
>>> On 2023-10-27 13:08, James Zern wrote:
>>>> On Fri, Oct 27, 2023 at 12:06 PM James Zern wrote:
>>>>> On Thu, Oct 26, 2023 at 7:30 PM Brian Inglis wrote:
>>>>>> The tests took 4-5 hours to run both locally and in CI, and generated
>>>>>> nearly 6MB of logs with only 5 failed tests in CI and 2 locally shown
>>>>>> below.
>>>>>> Are any of those test failures particularly significant?
>>>
>>>>> It would be helpful to see the error text of the failures. The size related
>>>>> ones should pass on all platforms, unless there's an allocation issue. I'm
>>>>> running a version now.
>>>
>>> The config size is 16K*16K; real 16GB; virtual 32GB across 2 drives.
>>> I do not see any GNU dejagnu/expect like test output.
>>> What is the best way to do that for just add_noise_test.cc?
>
> You can use the --gtest_filter option to limit the tests being run.
> For example: --gtest_filter=*AddNoiseTest*

Thanks James,

I will try running those standalone with shards limits.
I noticed that depending on load?, order, selection, all tests passed locally
running with only 1 shard due to my missing quotes!
The failures returned with -j5 NUM_SHARDS=4 SHARDS="0 1 2 3".
You had said seq 1... but hard coded values run 0..9?

>>> Packages, autotools, and g++ I mainly deal with using boilerplate commands from
>>> logs!
>>> What test data is required? Just the args shown in the error message?

>>>> I forgot to mention, you can speed up the test run with `make -j test` which
>>>> will use 10 shards by default. If you have more cores you can try `make -j
>>>> test NUM_SHARDS=$N SHARDS=$(seq 1 $(( N - 1 )))`

>>> Hah I wish! Old retail Acer AMD quad 3.5GHz with hard drives until it dies ;^<
>>> I will add the test definitions with $(nproc) for $N - that could help a little.

>> Well that eliminated any test failures on my local system and got rid of the C
>> AddNoiseTest failures in CI - see attached log extracts.
>> Adding /proc/meminfo and /proc/cpuinfo output to tests.

> I see the AddNoiseTest failures locally as well. Changing the order of
> the tests seems to affect the behavior. I'll file a bug for this.
>
> As for TestInvalidSizes:
> scallywag.log:2023-10-28T06:10:37.2906711Z [ RUN ]
> VP9FrameSizeTestsLarge.TestInvalidSizes
> scallywag.log-2023-10-28T06:10:39.5551590Z
> test/encode_test_driver.cc:35: Failure
> This only gets run when configured with --size-limit. How are you
> configuring the library? I'll check the behavior with this option.

Passing all tests locally, I released that build:

https://cygwin.com/packages/summary/libvpx-src.html

which links to the packaging repo where you can see the config and build
definitions in the cygport:

https://cygwin.com/cgit/cygwin-packages/libvpx/tree/libvpx.cygport?h=playground

where as I said above config size is 16K*16K (locally 4 cores, 16GB memory,
4-8GB free, 32GB swap across 2 drives) inherited from the original maintainer,
but looks likely based on Fedora:

https://src.fedoraproject.org/rpms/libvpx/blob/main/f/libvpx.spec

as both being RedHat derived and led libre projects, but Cygwin is all
volunteer, Fedora has more infrastructure, structure, funding, paid? staff or
employees, so we base some of our decisions and policies on theirs, as we have
nothing related documented or official.

If you compare the spec and cygport config args, or the history of each, I
dropped --enable-experimental, conflicting --dis/enable-install-srcs, and added
--enable-vp9-postproc, --enable-vp9-temporal-denoising, --enable-webm-io (needed
for test download I believe) as they seemed to add function.
I could not really judge about the others, as I am not a maths, codec, or media
guy (but am Cygwin libwebp maintainer, using it to save space on my old scans
and digital photos) noticed the libvxp CVE on a related project with no recent
Cygwin maintainer on my Open Source Security Atom/RSS feed, so wanted to get it
updated and released.
I would appreciate if you could point me to any project docs about the config
options and any dependencies from those.

Supporting an out of tree test data cache dir config option e.g.
/var/cache/libvpx/testdata/ for the 1.4GB would avoid the occasional download
failures I got as "connection reset by peer", whether from my ISP or your
storage site?
If we included the testdata in our package sources, we would download and cache
the non-local sources and patches, so that rebuilds do not redownload the sources.
When getting builds and tests working, we prefer to wipe the build tree
$PKG-$VER-$REL.$ARCH e.g. libvpx-1.13.1-1.x86_64/, but have no support for
automatically cached non-sources.

You could generate utility man pages using help2man e.g.

$ help2man -N -S WebM -n 'VP8 and VP9 Video Graphics Decoder'
--version-string=1.13.1 vpxdec > vpxdec.1
$ help2man -N -S WebM -n 'VP8 and VP9 Video Graphics Encoder'
--version-string=1.13.1 vpxenc > vpxenc.1

[Brian Inglis]

Sorry James,

Did not see your later emails come in while I was on the phone and while
replying to your earlier email.

That previous reply points to the config options, where they are, and came from
(Fedora Main spec).

Attached are a combined log of the failure contexts from all the recent builds
and tests locally and from Cygwin GH CI.

[Brian Inglis]

Hi James,

I noticed a note in the Wiki about libvpx requiring libvorbis but do not see any
other mentions or references: is it dynamically loaded if available?
Installing and specifying libvorbis-devel (which pulls in libogg-devel) in the
build-deps seems to speed up test runs locally and on CI by a noticeable factor.

I have tried caching the test data files out of the build tree and specifying
the dir in LIBVXP_TEST_DATA_PATH via the make command line
MAKEFLAGS/MAKEOVERRIDES mechanism but that does not seem to work, it
re-downloads everything into build/!

[Brian Inglis]

On 2023-10-31 01:20, Brian Inglis wrote:

Hi James,

I noticed a note in the Wiki about libvpx requiring libvorbis but do not see any
other mentions or references: is it dynamically loaded if available?
Installing and specifying libvorbis-devel (which pulls in libogg-devel) in the
build-deps seems to speed up test runs locally and on CI by a noticeable factor.

I have tried caching the test data files out of the build tree and specifying
the dir in LIBVXP_TEST_DATA_PATH via the make command line
MAKEFLAGS/MAKEOVERRIDES mechanism but that does not seem to work, it
re-downloads everything into build/!

[Brian Inglis]

On 2023-10-31 01:20, Brian Inglis wrote:

Hi James,

I noticed a note in the Wiki about libvpx requiring libvorbis but do not see any
other mentions or references: is it dynamically loaded if available?
Installing and specifying libvorbis-devel (which pulls in libogg-devel) in the
build-deps seems to speed up test runs locally and on CI by a noticeable factor.

I have tried caching the test data files out of the build tree and specifying

the dir in LIBVPX_TEST_DATA_PATH via the make command line
MAKEFLAGS/MAKEOVERRIDES mechanism but that does not seem to work, it still
re-downloads everything into build/!

[Brian Inglis]

Hi again,

Aha - export LIBVPX_TEST_DATA_PATH=... seems to do the trick there!

[James Zern]

Sorry, that was a typo. Shard indices go from 0..shard count - 1.

Just a few comments on the contents.

> DESCRIPTION="Video codec library libvpx from Google and AOMedia
> provides the reference implementation for VP8 and VP9 video codecs."
>

Technically the library is a part of the WebM Project, not AOMedia.
libaom is from the latter.

> HOMEPAGE=https://github.com/webmproject/$NAME

The canonical repo is https://chromium.googlesource.com/webm/libvpx
(github is a mirror), while the project homepage is
https://www.webmproject.org.

> DEPEND="libvorbis-devel"

There's no reference to vorbis in this project.

> ${CC} -shared -o cygvpx-$ABI.dll -Wl,--out-implib,libvpx.dll.a \
> -Wl,--whole-archive,libvpx.a,--no-whole-archive \
> || error "shared library build failed"

It would be some work since configure doesn't support shared libs on
this platform, but a linker script is generated from the exports_*
files in the tree. You could use that to reduce symbol visibility.

> where as I said above config size is 16K*16K (locally 4 cores, 16GB memory,
> 4-8GB free, 32GB swap across 2 drives) inherited from the original maintainer,
> but looks likely based on Fedora:
>
> https://src.fedoraproject.org/rpms/libvpx/blob/main/f/libvpx.spec
>
> as both being RedHat derived and led libre projects, but Cygwin is all
> volunteer, Fedora has more infrastructure, structure, funding, paid? staff or
> employees, so we base some of our decisions and policies on theirs, as we have
> nothing related documented or official.
>
> If you compare the spec and cygport config args, or the history of each, I
> dropped --enable-experimental, conflicting --dis/enable-install-srcs, and added
> --enable-vp9-postproc, --enable-vp9-temporal-denoising, --enable-webm-io (needed
> for test download I believe) as they seemed to add function.

That all makes sense. --enable-webm-io is enabled by default.

> I could not really judge about the others, as I am not a maths, codec, or media
> guy (but am Cygwin libwebp maintainer, using it to save space on my old scans
> and digital photos)

That's great to hear, thanks for using the library and keeping it up to date.

> noticed the libvxp CVE on a related project with no recent
> Cygwin maintainer on my Open Source Security Atom/RSS feed, so wanted to get it
> updated and released.
> I would appreciate if you could point me to any project docs about the config
> options and any dependencies from those.
>

The README mostly refers to configure --help:
https://chromium.googlesource.com/webm/libvpx/+/refs/tags/v1.13.1/README
The dependencies for tests and examples are all in the tree under
third_party/. The library doesn't reference them except x86inc which
is used when compiling assembly sources.

> Supporting an out of tree test data cache dir config option e.g.
> /var/cache/libvpx/testdata/ for the 1.4GB would avoid the occasional download
> failures I got as "connection reset by peer", whether from my ISP or your
> storage site?

The LIBVPX_TEST_DATA_PATH environment variable can be used to
configure the test data location. It looks like libvpx.cyport is
already using it.

[James Zern]

You should add --enable-pic since you're building a dll with the code.

[Brian Inglis]

Thanks James,

Fixed!

>> HOMEPAGE=https://github.com/webmproject/$NAME
>
> The canonical repo is https://chromium.googlesource.com/webm/libvpx
> (github is a mirror), while the project homepage is
> https://www.webmproject.org.

Like many projects, WebM says more about the project and container than the
video library, so I picked a page that gave a more user/developer friendly front
than googlesource, which is just a tree with refs, and none of the comforts
provided by cgit or gitweb. ;^>

>> DEPEND="libvorbis-devel"
>
> There's no reference to vorbis in this project.

Makes little sense, I agree, but:

http://wiki.webmproject.org/ffmpeg/building-with-libvpx

Dependencies
...
libvpx depends on libvorbis
libvorbis depends on libogg
...

and it did seem to speed up the tests?!

>> ${CC} -shared -o cygvpx-$ABI.dll -Wl,--out-implib,libvpx.dll.a \
>> -Wl,--whole-archive,libvpx.a,--no-whole-archive \
>> || error "shared library build failed"
>
> It would be some work since configure doesn't support shared libs on
> this platform, but a linker script is generated from the exports_*
> files in the tree. You could use that to reduce symbol visibility.

Not seeing anything obviously like ldscripts which I expect to match *86*.x*?

I tried adding it to MAKEOPTS but that didn't work so exported it and behold!

>> If we included the testdata in our package sources, we would download and cache
>> the non-local sources and patches, so that rebuilds do not redownload the sources.
>> When getting builds and tests working, we prefer to wipe the build tree
>> $PKG-$VER-$REL.$ARCH e.g. libvpx-1.13.1-1.x86_64/, but have no support for
>> automatically cached non-sources.
>>
>> You could generate utility man pages using help2man e.g.
>>
>> $ help2man -N -S WebM -n 'VP8 and VP9 Video Graphics Decoder'
>> --version-string=1.13.1 vpxdec > vpxdec.1
>> $ help2man -N -S WebM -n 'VP8 and VP9 Video Graphics Encoder'
>> --version-string=1.13.1 vpxenc > vpxenc.1

I added those to the Cygwin build but using -o vpx{dec,enc}.1 instead of >.

Thanks for all your help, and the --enable-pic hint in your other email.

[James Zern]

Time differences are probably just system noise. I think the intent of
the wiki was to install a set of open codecs, though opus might be a
better choice now. The dependency is incorrect; there was never a
direct dependency on that lib. Editing is disabled on that site, I'll
see if I can remove the bullet.

> >> ${CC} -shared -o cygvpx-$ABI.dll -Wl,--out-implib,libvpx.dll.a \
> >> -Wl,--whole-archive,libvpx.a,--no-whole-archive \
> >> || error "shared library build failed"
> >
> > It would be some work since configure doesn't support shared libs on
> > this platform, but a linker script is generated from the exports_*
> > files in the tree. You could use that to reduce symbol visibility.
>
> Not seeing anything obviously like ldscripts which I expect to match *86*.x*?
>

The raw files are:
vp8/exports_dec
vp8/exports_enc
vp9/exports_dec
vp9/exports_enc
vpx/exports_com
vpx/exports_dec
vpx/exports_enc
A linux build will create libvpx.ver from them.

[Brian Inglis]

On 2023-11-03 13:45, James Zern wrote:
> On Fri, Nov 3, 2023 at 10:01 AM Brian Inglis <Brian....@shaw.ca> wrote:
>>
>> On 2023-11-02 20:14, James Zern wrote:
>>> On Mon, Oct 30, 2023 at 2:55 PM Brian Inglis <Brian....@shaw.ca> wrote:
>>>>
>>>> On 2023-10-30 13:47, James Zern wrote:
>>>>> On Sat, Oct 28, 2023 at 7:38 AM Brian Inglis <Brian....@shaw.ca> wrote:
>>>>>>
>>>>>> On 2023-10-27 23:01, Brian Inglis wrote:
>>>>>>> On 2023-10-27 13:08, James Zern wrote:
>>>>>>>> On Fri, Oct 27, 2023 at 12:06 PM James Zern wrote:
>>>>>>>>> On Thu, Oct 26, 2023 at 7:30 PM Brian Inglis wrote:
>>>>>>>>>> The tests took 4-5 hours to run both locally and in CI, and generated
>>>>>>>>>> nearly 6MB of logs with only 5 failed tests in CI and 2 locally shown
>>>>>>>>>> below.
>>>>>>>>>> Are any of those test failures particularly significant?
>>>>>>>
>>>>>>>>> It would be helpful to see the error text of the failures. The size related
>>>>>>>>> ones should pass on all platforms, unless there's an allocation issue. I'm
>>>>>>>>> running a version now.
>>>>>>>
>>>>>>> The config size is 16K*16K; real 16GB; virtual 32GB across 2 drives.
>>>>>>> I do not see any GNU dejagnu/expect like test output.
>>>>>>> What is the best way to do that for just add_noise_test.cc?
>>>>>
>>>>> You can use the --gtest_filter option to limit the tests being run.
>>>>> For example: --gtest_filter=*AddNoiseTest*

Thanks James,

After trying more build tweaks which gave me *identical* exe and dll sizes,
rerunning all the failures using that filter option, monitoring the runs using
`top`, I /finally/ noticed that my "swap"/page file usage had jumped from 32GB
split across 2 drives to 40GB total.
I think that may have been the source of the slow runs and non-reproducible
failures.
I noticed at one point during the filtered tests, the single standalone job was
using 30GB virtual and a few GB real memory in `top`, so it appears likely that
some of the test runs may have been demanding more, or together were requesting
more than the paging system could provide.
That could also explain why the CI jobs run in CPU, memory, and swap/page
limited VMs experienced more failures, and more consistency in those failures.

If you size the tests by cpu time and memory footprint, you may find that some
of them should perhaps be tailored down to what can run reliably in a normal CI
VM configuration with a couple of server CPUs 2-3GHz and a few GB memory with
limited swap/page space, and the full scale tests demoted to "performance"
category, with notes on cp, memory, and swap/page expectations.

[I came from an application background where I also used to do (pre-SOx) system
management, administration, assembler customization, app, db, system and
hardware tuning, tweaking, and upgrading, finding and fixing whatever came up,
often in limited cofigurations, so have always made sure I have preallocated
2*memory swap/page space, across 2 drives if possible, to avoid OoM failures at
any cost, also the really weird glitches and failures that happen, especially on
Windows systems, when the memory load exceeds the availability, especially if
the cpu load is high at the same time.]

Please see the attached logs of GH Scallywag CI, local, and gtest_filter runs
for the latest random failures.

>>>>>> Well that eliminated any test failures on my local system and got rid of the C
>>>>>> AddNoiseTest failures in CI - see attached log extracts.
>>>>>> Adding /proc/meminfo and /proc/cpuinfo output to tests.
>>>>
>>>>> I see the AddNoiseTest failures locally as well. Changing the order of
>>>>> the tests seems to affect the behavior. I'll file a bug for this.
>>>>>
>>>>> As for TestInvalidSizes:
>>>>> scallywag.log:2023-10-28T06:10:37.2906711Z [ RUN ]
>>>>> VP9FrameSizeTestsLarge.TestInvalidSizes
>>>>> scallywag.log-2023-10-28T06:10:39.5551590Z
>>>>> test/encode_test_driver.cc:35: Failure
>>>>> This only gets run when configured with --size-limit. How are you
>>>>> configuring the library? I'll check the behavior with this option.
>>>>
>>>> Passing all tests locally, I released that build:
>>>>
>>>> https://cygwin.com/packages/summary/libvpx-src.html
>>>>
>>>> which links to the packaging repo where you can see the config and build
>>>> definitions in the cygport:
>>>>
>>>> https://cygwin.com/cgit/cygwin-packages/libvpx/tree/libvpx.cygport?h=playground

>>>> ${CC} -shared -pic -o cygvpx-$ABI.dll -Wl,--out-implib,libvpx.dll.a \

>>>> -Wl,--whole-archive,libvpx.a,--no-whole-archive \
>>>> || error "shared library build failed"
>>>
>>> It would be some work since configure doesn't support shared libs on
>>> this platform, but a linker script is generated from the exports_*
>>> files in the tree. You could use that to reduce symbol visibility.
>>
>> Not seeing anything obviously like ldscripts which I expect to match *86*.x*?
> The raw files are:
> vp8/exports_dec
> vp8/exports_enc
> vp9/exports_dec
> vp9/exports_enc
> vpx/exports_com
> vpx/exports_dec
> vpx/exports_enc
> A linux build will create libvpx.ver from them.

The build sources that from the Fedora libvpx repo main branch and presumably
uses it:

https://src.fedoraproject.org/rpms/libvpx/blob/main/f/libvpx.ver

[James Zern]

To balance this in our presubmits, tests with 'Large' in the name are
filtered out (--gtest_filter=*Large*). These tests often have a long
runtime and/or memory usage. With 8 executor slots and 32GiB of
memory, so ~4GiB/shard, this generally is enough. A few tests may
require 8-9GiB. We don't have anything more fine grained than the name
based filter currently.

I see it listed on line 17 of libvpx.cygport, but not mentioned when
creating the shared library on line 163.

> --
> Take care. Thanks, Brian Inglis Calgary, Alberta, Canada
>
> La perfection est atteinte Perfection is achieved
> non pas lorsqu'il n'y a plus rien à ajouter not when there is no more to add
> mais lorsqu'il n'y a plus rien à retirer but when there is no more to cut
> -- Antoine de Saint-Exupéry
>

> --
> You received this message because you are subscribed to the Google Groups "WebM Discussion" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to webm-discuss...@webmproject.org.

> To view this discussion on the web visit https://groups.google.com/a/webmproject.org/d/msgid/webm-discuss/86c2c559-747b-4f8b-8e2e-290472df3edd%40Shaw.ca.

[Brian Inglis]

Thanks James,

Added -Wl,--version-script,libvpx.ver as it did not like just libvpx.ver or
-Wl,libvpx.ver

Checking this against your exports, there are a number of additional functions
exported by the Fedora build - shown with count 1 in the attached log, created
by pasting libvpx.ver and the vp?/exports_??? with sed cleanup, sort, uniq -c to
see which appear in both sources, and which are only in libvpx.ver - any
comments on those additions?

I will also look to see what your Linux build does with your vp?/exports_???, as
that may be a better approach for the Cygwin build, given that it looks like all
symbols may have been exported from the previous ABI library, and the Fedora
restriction may reflect their package selection and build choices.

[James Zern]

The file looks like it has accumulated symbols across releases. For
example, the last references to vpx_codec_get_mem_map and
vpx_codec_set_mem_map were removed in:
https://chromium.googlesource.com/webm/libvpx/+/6d52fe2e715e93ae8aa1e6c9d7c14237c98535e6

> I will also look to see what your Linux build does with your vp?/exports_???, as
> that may be a better approach for the Cygwin build, given that it looks like all
> symbols may have been exported from the previous ABI library, and the Fedora
> restriction may reflect their package selection and build choices.
>

In a build configured for static only, you can generate libvpx.ver
with `make libvpx.ver`.

[Brian Inglis]

Funnily enough VP9FrameSizeTestsLarge and IntegerOverflowLarge are the CI
failures, and the former appears to require 20-30GB swap on my system, according
to top and procps.

>> Added -Wl,--version-script,libvpx.ver as it did not like just libvpx.ver or
>> -Wl,libvpx.ver
>>
>> Checking this against your exports, there are a number of additional
>> functions exported by the Fedora build - shown with count 1 in the attached
>> log, created by pasting libvpx.ver and the vp?/exports_??? with sed
>> cleanup, sort, uniq -c to see which appear in both sources, and which are
>> only in libvpx.ver - any comments on those additions?

> The file looks like it has accumulated symbols across releases. For
> example, the last references to vpx_codec_get_mem_map and
> vpx_codec_set_mem_map were removed in:
> https://chromium.googlesource.com/webm/libvpx/+/6d52fe2e715e93ae8aa1e6c9d7c14237c98535e6

Thanks James,

One of the Fedora spec comments says it included symbols used in Debian builds.
It appears that it may be to maintain a common ABI for legacy packages and
architectures that may not support all the same features.

>> I will also look to see what your Linux build does with your
>> vp?/exports_???, as that may be a better approach for the Cygwin build,
>> given that it looks like all symbols may have been exported from the
>> previous ABI library, and the Fedora restriction may reflect their package
>> selection and build choices.

I initially tweaked your gen_msvs_def.sh script, but then decided to tweak your
libs.mk awk generation to just include the echo of the first and last lines:

awk -e 'BEGIN {print "{ global:"}
{print $2";"}
END{print "local: *; };"}' vp?/exports_??? > libvpx.ver

links with your utility programs, and saves 80+KB ~2.5% over all exports:

$ du -ab {,libvpx-1.13.1-?.x86_64/inst}/usr/bin/*vpx* \
libvpx-1.13.1-?.x86_64/build/{vpx*.exe,*vpx*.dll}
2435091 /usr/bin/cygvpx-3.dll
4009371 /usr/bin/cygvpx-4.dll
4111379 /usr/bin/cygvpx-5.dll
3492883 /usr/bin/cygvpx-8.dll
695315 /usr/bin/vpxdec.exe
716307 /usr/bin/vpxenc.exe
23883706 libvpx-1.13.1-1.x86_64/build/cygvpx-8.dll
3393609 libvpx-1.13.1-1.x86_64/build/vpxdec.exe
3568485 libvpx-1.13.1-1.x86_64/build/vpxenc.exe
3492883 libvpx-1.13.1-1.x86_64/inst/usr/bin/cygvpx-8.dll
695315 libvpx-1.13.1-1.x86_64/inst/usr/bin/vpxdec.exe
716307 libvpx-1.13.1-1.x86_64/inst/usr/bin/vpxenc.exe
23799226 libvpx-1.13.1-2.x86_64/build/cygvpx-8.dll
3393609 libvpx-1.13.1-2.x86_64/build/vpxdec.exe
3568485 libvpx-1.13.1-2.x86_64/build/vpxenc.exe
3406867 libvpx-1.13.1-2.x86_64/inst/usr/bin/cygvpx-8.dll
695315 libvpx-1.13.1-2.x86_64/inst/usr/bin/vpxdec.exe
716307 libvpx-1.13.1-2.x86_64/inst/usr/bin/vpxenc.exe

> In a build configured for static only, you can generate libvpx.ver
> with `make libvpx.ver`.

That plays havoc with some test programs, which need all exports, and only works
with -j1, as two jobs run, and the final line echo "local: *; };" is duplicated,
failing ld!

I really appreciate all the help you have given me to understand the build, and
the time spent to educate me about your custom build process.
I hope not to have to bother you again /too much/ unless the package takes a
surprising new direction.

[Brian Inglis]

Funnily enough VP9FrameSizeTestsLarge and IntegerOverflowLarge are the CI
failures, and the former appears to require 20-30GB swap on my system, according
to top and procps.

>> Added -Wl,--version-script,libvpx.ver as it did not like just libvpx.ver or
>> -Wl,libvpx.ver
>>
>> Checking this against your exports, there are a number of additional
>> functions exported by the Fedora build - shown with count 1 in the attached
>> log, created by pasting libvpx.ver and the vp?/exports_??? with sed
>> cleanup, sort, uniq -c to see which appear in both sources, and which are
>> only in libvpx.ver - any comments on those additions?

> The file looks like it has accumulated symbols across releases. For
> example, the last references to vpx_codec_get_mem_map and
> vpx_codec_set_mem_map were removed in:
> https://chromium.googlesource.com/webm/libvpx/+/6d52fe2e715e93ae8aa1e6c9d7c14237c98535e6

Thanks James,

One of the Fedora spec comments says it included symbols used in Debian builds.
It appears that it may be to maintain a common ABI for legacy packages and
architectures that may not support all the same features.

>> I will also look to see what your Linux build does with your
>> vp?/exports_???, as that may be a better approach for the Cygwin build,
>> given that it looks like all symbols may have been exported from the
>> previous ABI library, and the Fedora restriction may reflect their package
>> selection and build choices.

> In a build configured for static only, you can generate libvpx.ver
> with `make libvpx.ver`.

That plays havoc with some test programs, which need all exports, and only works
with -j1, as two jobs run, and the final line echo "local: *; };" is duplicated,
failing ld!

I really appreciate all the help you have given me to understand the build, and
the time spent to educate me about your custom build process.
I hope not to have to bother you again /too much/ unless the package takes a
surprising new direction.

[James Zern]

That's great.

> $ du -ab {,libvpx-1.13.1-?.x86_64/inst}/usr/bin/*vpx* \
> libvpx-1.13.1-?.x86_64/build/{vpx*.exe,*vpx*.dll}
> 2435091 /usr/bin/cygvpx-3.dll
> 4009371 /usr/bin/cygvpx-4.dll
> 4111379 /usr/bin/cygvpx-5.dll
> 3492883 /usr/bin/cygvpx-8.dll
> 695315 /usr/bin/vpxdec.exe
> 716307 /usr/bin/vpxenc.exe
> 23883706 libvpx-1.13.1-1.x86_64/build/cygvpx-8.dll
> 3393609 libvpx-1.13.1-1.x86_64/build/vpxdec.exe
> 3568485 libvpx-1.13.1-1.x86_64/build/vpxenc.exe
> 3492883 libvpx-1.13.1-1.x86_64/inst/usr/bin/cygvpx-8.dll
> 695315 libvpx-1.13.1-1.x86_64/inst/usr/bin/vpxdec.exe
> 716307 libvpx-1.13.1-1.x86_64/inst/usr/bin/vpxenc.exe
> 23799226 libvpx-1.13.1-2.x86_64/build/cygvpx-8.dll
> 3393609 libvpx-1.13.1-2.x86_64/build/vpxdec.exe
> 3568485 libvpx-1.13.1-2.x86_64/build/vpxenc.exe
> 3406867 libvpx-1.13.1-2.x86_64/inst/usr/bin/cygvpx-8.dll
> 695315 libvpx-1.13.1-2.x86_64/inst/usr/bin/vpxdec.exe
> 716307 libvpx-1.13.1-2.x86_64/inst/usr/bin/vpxenc.exe
>
> > In a build configured for static only, you can generate libvpx.ver
> > with `make libvpx.ver`.
>
> That plays havoc with some test programs, which need all exports,

These are generally filtered out when --enable-shared is used, but
since you're using a static build they'll all be enabled.

> and only works
> with -j1, as two jobs run, and the final line echo "local: *; };" is duplicated,
> failing ld!
>

That's strange, I don't see the duplication on Linux or Cygwin, but
maybe there is a race.

> I really appreciate all the help you have given me to understand the build, and
> the time spent to educate me about your custom build process.
> I hope not to have to bother you again /too much/ unless the package takes a
> surprising new direction.
>

Glad to help. Thanks for working through all of this. The builds on
Cygwin are slower, so I know it can take some patience to try these
things out.