[request] libaom official GitHub mirror

116 views
Skip to first unread message

Jan Beich

unread,
Oct 10, 2021, 4:28:46 PM10/10/21
to av1-d...@aomedia.org, Jan Alexander Steffens, Brad Smith, Nia Alarie
Automatically generated tarballs from googlesource.com have unstable checksum,
making those unsuitable for downstream consumption. To make builds reproducible
Unix distributions often have to host a local copy that no longer matches upstream.
Some (e.g., Gentoo) have switched to signed tarballs (posted here, on av1-discuss)
but those are limited to to few releases (e.g., 1.0.0* and *-rc* are missing).
However, a downstream may need an arbitrary snapshot or cherry-pick fixes from
later releases.

https://github.com/archlinux/svntogit-packages/commit/7522d6aabb7b
https://github.com/netbsd/pkgsrc/commit/e89d1445e870

Can someone setup a mirror on GitHub in official capacity (marked as "public mirror")
under AOMediaCodec, webmproject, google, VideoLAN or maybe another account?

Steps to reproduce:

$ curl -O https://aomedia.googlesource.com/aom/+archive/v3.2.0-rc1.tar.gz
$ sha256 v3.2.0-rc1.tar.gz
SHA256 (v3.2.0-rc1.tar.gz) = 145ae6eb1d022079892eee1732c4138f1861134fa69308e009372e1b3a745ea0
$ curl -O https://aomedia.googlesource.com/aom/+archive/v3.2.0-rc1.tar.gz
$ sha256 v3.2.0-rc1.tar.gz
SHA256 (v3.2.0-rc1.tar.gz) = 32d869d8546ed8fb75cc9533effbce8ff911c7750fa496f6b9c6b6b61a2d0bb8

$ curl -o 86874609267d.patch 'https://aomedia.googlesource.com/aom/+/86874609267d%5E%21/?format=TEXT'
$ patch -p1 -i 86874609267d.patch -d /path/to/libaom-3.1.0
Hmm... I can't seem to find a patch in there anywhere.
$ b64decode 86874609267d.patch
b64decode: 86874609267d.patch: missing or bad "begin" line
$ file 86874609267d.patch
86874609267d.patch: ASCII text, with very long lines, with no line terminators

Existing non-official up-to-date mirrors:
* https://github.com/jbeich/aom set up via cronjob, used by FreeBSD and OpenBSD packages
* https://github.com/m-ab-s/aom set up via GitLab hook, doesn't prune obsolote tags

Note, I'm planning to kill my mirror once the official one is up.

James Zern

unread,
Oct 11, 2021, 7:35:02 PM10/11/21
to av1-d...@aomedia.org, Jan Alexander Steffens, Brad Smith, Nia Alarie
Hi,

On Sun, Oct 10, 2021 at 1:28 PM Jan Beich <jbe...@freebsd.org> wrote:
Automatically generated tarballs from googlesource.com have unstable checksum,
making those unsuitable for downstream consumption. To make builds reproducible
Unix distributions often have to host a local copy that no longer matches upstream.
Some (e.g., Gentoo) have switched to signed tarballs (posted here, on av1-discuss)
but those are limited to to few releases (e.g., 1.0.0* and *-rc* are missing).
However, a downstream may need an arbitrary snapshot or cherry-pick fixes from
later releases.

https://github.com/archlinux/svntogit-packages/commit/7522d6aabb7b
https://github.com/netbsd/pkgsrc/commit/e89d1445e870

Can someone setup a mirror on GitHub in official capacity (marked as "public mirror")
under AOMediaCodec, webmproject, google, VideoLAN or maybe another account?

Stable archives for the final release are available similar to what's done for libvpx for this purpose. Maybe that's lacking some visibility.
 

Steps to reproduce:

  $ curl -O https://aomedia.googlesource.com/aom/+archive/v3.2.0-rc1.tar.gz
  $ sha256 v3.2.0-rc1.tar.gz
  SHA256 (v3.2.0-rc1.tar.gz) = 145ae6eb1d022079892eee1732c4138f1861134fa69308e009372e1b3a745ea0
  $ curl -O https://aomedia.googlesource.com/aom/+archive/v3.2.0-rc1.tar.gz
  $ sha256 v3.2.0-rc1.tar.gz
  SHA256 (v3.2.0-rc1.tar.gz) = 32d869d8546ed8fb75cc9533effbce8ff911c7750fa496f6b9c6b6b61a2d0bb8

  $ curl -o 86874609267d.patch 'https://aomedia.googlesource.com/aom/+/86874609267d%5E%21/?format=TEXT'
  $ patch -p1 -i 86874609267d.patch -d /path/to/libaom-3.1.0
  Hmm...  I can't seem to find a patch in there anywhere.
  $ b64decode 86874609267d.patch
  b64decode: 86874609267d.patch: missing or bad "begin" line
  $ file 86874609267d.patch
  86874609267d.patch: ASCII text, with very long lines, with no line terminators

Existing non-official up-to-date mirrors:
* https://github.com/jbeich/aom set up via cronjob, used by FreeBSD and OpenBSD packages
* https://github.com/m-ab-s/aom set up via GitLab hook, doesn't prune obsolote tags

Note, I'm planning to kill my mirror once the official one is up.

--
You received this message because you are subscribed to the Google Groups "AV1 Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to av1-discuss...@aomedia.org.
To view this discussion on the web visit https://groups.google.com/a/aomedia.org/d/msgid/av1-discuss/sfx8-aj55-wny%40FreeBSD.org.

Brad Smith

unread,
Oct 12, 2021, 3:22:32 PM10/12/21
to Jan Beich, av1-d...@aomedia.org, Jan Alexander Steffens, Nia Alarie
Jan, thank you for the mirror.

Representing the OpenBSD port maintainer, I would very much welcome an
official github mirror
of the source repo.

I was using hand rolled tarballs for 1.0.0 snapshots, but switched to
GitHub for 2.0.0 and newer.

James Zern

unread,
Oct 12, 2021, 3:37:08 PM10/12/21
to av1-d...@aomedia.org, Jan Beich, Jan Alexander Steffens, Nia Alarie, Urvang Joshi
How about one on gitlab? https://gitlab.com/AOMediaCodec.
 

I was using hand rolled tarballs for 1.0.0 snapshots, but switched to
GitHub for 2.0.0 and newer.

--
You received this message because you are subscribed to the Google Groups "AV1 Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to av1-discuss...@aomedia.org.

Jan Beich

unread,
Oct 12, 2021, 5:23:55 PM10/12/21
to James Zern, av1-d...@aomedia.org, Jan Alexander Steffens, Nia Alarie, Urvang Joshi
I find GitLab UI slow and convoluted. Unless self-hosted (like VideoLAN)
there's no point. The mirror will likely be used for code search,
snapshot generation and social stuff like stars (bookmarking and
advertising to followers) and forks (collaborating on changes that may
not be intended for upstreaming). A mirror on a *popular* platform would
be advantageous but... there could be more than one mirror e.g.,
github.com, code.videolan.org (gitlab), codeberg.org (gitea).

From privacy POV gitlab.com forces Tor users to solve captcha even
for read-only access. Other GitLab instances usually do not.

James Zern

unread,
Oct 12, 2021, 8:02:16 PM10/12/21
to Jan Beich, av1-d...@aomedia.org, Jan Alexander Steffens, Nia Alarie, Urvang Joshi
Thanks for the background. I wasn't a part of the decision to move to
gitlab so I can't speak to that. I was just looking to reuse what the
project already had.

Brad Smith

unread,
Oct 15, 2021, 8:07:22 PM10/15/21
to James Zern, av1-d...@aomedia.org, Jan Alexander Steffens, Nia Alarie
On 10/11/2021 7:34 PM, James Zern wrote:
Hi,

On Sun, Oct 10, 2021 at 1:28 PM Jan Beich <jbe...@freebsd.org> wrote:
Automatically generated tarballs from googlesource.com have unstable checksum,
making those unsuitable for downstream consumption. To make builds reproducible
Unix distributions often have to host a local copy that no longer matches upstream.
Some (e.g., Gentoo) have switched to signed tarballs (posted here, on av1-discuss)
but those are limited to to few releases (e.g., 1.0.0* and *-rc* are missing).
However, a downstream may need an arbitrary snapshot or cherry-pick fixes from
later releases.

https://github.com/archlinux/svntogit-packages/commit/7522d6aabb7b
https://github.com/netbsd/pkgsrc/commit/e89d1445e870

Can someone setup a mirror on GitHub in official capacity (marked as "public mirror")
under AOMediaCodec, webmproject, google, VideoLAN or maybe another account?

Stable archives for the final release are available similar to what's done for libvpx for this purpose. Maybe that's lacking some visibility.

I used to use tarballs for libvpx pulled from  https://storage.googleapis.com/downloads.webmproject.org/releases/webm/
but there hasn't been any uploaded there since 1.6.1.

I looked at the tarballs in the URL above and they're not usable as is. The tarballs need to be
created with the source code rooted in a subdir, as in for 3.20.0 in a dir of the name aom-3.20.0.

James Zern

unread,
Oct 15, 2021, 10:08:40 PM10/15/21
to Brad Smith, av1-d...@aomedia.org, Jan Alexander Steffens, Nia Alarie, Wan-Teh Chang, Jerome Jiang
On Fri, Oct 15, 2021 at 5:07 PM Brad Smith <br...@comstyle.com> wrote:
On 10/11/2021 7:34 PM, James Zern wrote:
Hi,

On Sun, Oct 10, 2021 at 1:28 PM Jan Beich <jbe...@freebsd.org> wrote:
Automatically generated tarballs from googlesource.com have unstable checksum,
making those unsuitable for downstream consumption. To make builds reproducible
Unix distributions often have to host a local copy that no longer matches upstream.
Some (e.g., Gentoo) have switched to signed tarballs (posted here, on av1-discuss)
but those are limited to to few releases (e.g., 1.0.0* and *-rc* are missing).
However, a downstream may need an arbitrary snapshot or cherry-pick fixes from
later releases.

https://github.com/archlinux/svntogit-packages/commit/7522d6aabb7b
https://github.com/netbsd/pkgsrc/commit/e89d1445e870

Can someone setup a mirror on GitHub in official capacity (marked as "public mirror")
under AOMediaCodec, webmproject, google, VideoLAN or maybe another account?

Stable archives for the final release are available similar to what's done for libvpx for this purpose. Maybe that's lacking some visibility.

I used to use tarballs for libvpx pulled from  https://storage.googleapis.com/downloads.webmproject.org/releases/webm/
but there hasn't been any uploaded there since 1.6.1.


It looks like from v1.7.0 they were uploaded as .gz rather than bz2. The index is a little stale though:
 


I looked at the tarballs in the URL above and they're not usable as is. The tarballs need to be
created with the source code rooted in a subdir, as in for 3.20.0 in a dir of the name aom-3.20.0.


It seems like that could be addressed at extraction time, but I agree that it's not very convenient. We use this structure for libwebp.

Brad Smith

unread,
Oct 18, 2021, 8:04:23 PM10/18/21
to James Zern, av1-d...@aomedia.org, Jan Alexander Steffens, Nia Alarie, Wan-Teh Chang, Jerome Jiang

The only tarballs I see are the weird ones of the v1.7.0.tar.gz variety.

https://storage.googleapis.com/downloads.webmproject.org/releases/webm/libvpx-1.11.0.tar.gz shows 404.

I don't use the libwebp tarballs either. Same thing.

James Zern

unread,
Oct 18, 2021, 8:41:24 PM10/18/21
to Brad Smith, av1-d...@aomedia.org, Jan Alexander Steffens, Nia Alarie, Wan-Teh Chang, Jerome Jiang
Yes that's the current format, but they should be renamed to indicate the library.
 

I don't use the libwebp tarballs either. Same thing.


libwebp sounds like it follows what you were suggesting, unless I'm mistaken.
drwxr-xr-x 100/100           0 2021-08-13 13:42 libwebp-1.2.1/
drwxr-xr-x 100/100           0 2021-08-13 13:42 libwebp-1.2.1/extras/
-rw-r--r-- 100/100       33335 2021-08-13 13:42 libwebp-1.2.1/extras/Makefile.in
-rw-r--r-- 100/100        2443 2021-07-29 15:55 libwebp-1.2.1/extras/vwebp_sdl.c
-rw-r--r-- 100/100         890 2021-07-29 15:55 libwebp-1.2.1/extras/webp_to_sdl.h
-rw-r--r-- 100/100        1327 2021-07-29 15:55 libwebp-1.2.1/extras/Makefile.am
-rw-r--r-- 100/100        1582 2021-07-29 15:55 libwebp-1.2.1/extras/webp_quality.c
-rw-r--r-- 100/100        3839 2021-07-29 15:55 libwebp-1.2.1/extras/quality_estimate.c
-rw-r--r-- 100/100        4940 2021-07-29 15:55 libwebp-1.2.1/extras/extras.c
-rw-r--r-- 100/100       12256 2021-07-29 15:55 libwebp-1.2.1/extras/get_disto.c
Reply all
Reply to author
Forward
0 new messages