Bug#1019696: devscripts: uscan does not work with github anymore

Patrick Matthäi

unread,

Sep 13, 2022, 11:30:03 AM9/13/22

to

Package: devscripts
Version: 2.22.2
Severity: important

Hello,

github made some annoying changes to the release download pages (lazy loading?)
All debian/watch files using github are failing now:

version=4
https://github.com/liske/needrestart/releases \
/liske/needrestart/archive/refs/tags/v(.+)\.tar\.gz

$ uscan -v --debug
uscan info: uscan (version 2.22.2~bpo11+1) See uscan(1) for help
uscan info: Scan watch files in .
uscan info: Check debian/watch and debian/changelog in .
uscan info: package="needrestart" version="3.6-2" (as seen in debian/changelog)
uscan info: package="needrestart" version="3.6" (no epoch/revision)
uscan info: ./debian/changelog sets package="needrestart" version="3.6"
uscan info: Process watch file at: debian/watch
package = needrestart
version = 3.6
pkg_dir = .
uscan info: Last orig.tar.* tarball version (from debian/changelog): 3.6
uscan info: Last orig.tar.* tarball version (dversionmangled): 3.6
uscan info: Requesting URL:
https://github.com/liske/needrestart/releases
uscan info: Matching pattern:
(?:(?:https://github.com)?\/liske\/needrestart\/)?/liske/needrestart/archive/refs/tags/v(.+)\.tar\.gz
uscan warn: In debian/watch no matching files for watch line
https://github.com/liske/needrestart/releases /liske/needrestart/archive/refs/tags/v(.+)\.tar\.gz
uscan info: Scan finished

-- Package-specific info:

--- /etc/devscripts.conf ---
Empty.

--- ~/.devscripts ---
export DGET_VERIFY=no

-- System Information:
Debian Release: 11.5
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'stable-security'), (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 5.10.0-18-amd64 (SMP w/2 CPU threads)
Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages devscripts depends on:
ii dpkg-dev 1.20.12
ii fakeroot 1.25.3-1.1
ii file 1:5.39-3
ii gnupg 2.2.27-2+deb11u2
ii gnupg2 2.2.27-2+deb11u2
ii gpgv 2.2.27-2+deb11u2
ii libc6 2.31-13+deb11u4
ii libfile-dirlist-perl 0.05-2
ii libfile-homedir-perl 1.006-1
ii libfile-touch-perl 0.11-1
ii libfile-which-perl 1.23-1
ii libipc-run-perl 20200505.0-1
ii libmoo-perl 2.004004-1
ii libwww-perl 6.52-1
ii patchutils 0.4.2-1
ii perl 5.32.1-4+deb11u2
ii python3 3.9.2-3
ii sensible-utils 0.0.14
ii wdiff 1.2.2-2+b1

Versions of packages devscripts recommends:
ii apt 2.2.4
ii curl 7.74.0-1.3+deb11u3
ii dctrl-tools 2.24-3+b1
pn debian-keyring <none>
ii dput 1.1.0
ii dupload 2.9.7
pn equivs <none>
ii libdistro-info-perl 1.0
ii libdpkg-perl 1.20.12
ii libencode-locale-perl 1.05-1.1
pn libgit-wrapper-perl <none>
pn libgitlab-api-v4-perl <none>
ii liblist-compare-perl 0.55-1
ii liblwp-protocol-https-perl 6.10-1
pn libsoap-lite-perl <none>
ii libstring-shellquote-perl 1.04-1
ii libtry-tiny-perl 0.30-1
ii liburi-perl 5.08-1
ii licensecheck 3.1.1-2
ii lintian 2.115.3
ii man-db 2.9.4-2
ii patch 2.7.6-7
pn pristine-tar <none>
ii python3-apt 2.2.1
ii python3-debian 0.1.39
pn python3-magic <none>
ii python3-requests 2.25.1+dfsg-2
pn python3-unidiff <none>
pn python3-xdg <none>
ii strace 5.10-1
ii unzip 6.0-26+deb11u1
ii wget 1.21-1+deb11u1
ii xz-utils 5.2.5-2.1~deb11u1

Versions of packages devscripts suggests:
pn adequate <none>
ii at 3.1.23-1.1
pn autopkgtest <none>
pn bls-standalone <none>
ii bsd-mailx [mailx] 8.1.2-0.20180807cvs-2
ii build-essential 12.9
pn check-all-the-things <none>
pn cvs-buildpackage <none>
ii debhelper 13.3.4
pn diffoscope <none>
pn disorderfs <none>
pn dose-extra <none>
pn duck <none>
pn elpa-devscripts <none>
pn faketime <none>
pn gnuplot <none>
pn how-can-i-help <none>
pn libauthen-sasl-perl <none>
pn libdbd-pg-perl <none>
pn libfile-desktopentry-perl <none>
pn libnet-smtps-perl <none>
pn libterm-size-perl <none>
ii libtimedate-perl 2.3300-2
pn libyaml-syck-perl <none>
pn mmdebstrap <none>
pn mozilla-devscripts <none>
ii mutt 2.0.5-4.1+deb11u1
ii openssh-client [ssh-client] 1:8.4p1-5+deb11u1
pn piuparts <none>
pn postgresql-client <none>
pn pristine-lfs <none>
ii quilt 0.66-2.1
pn ratt <none>
pn reprotest <none>
pn svn-buildpackage <none>
pn w3m <none>

-- no debconf information

Peter Colberg

unread,

Sep 17, 2022, 2:10:04 PM9/17/22

to

On Tue, Sep 13, 2022 at 05:24:02PM +0200, Patrick Matthäi wrote:
> github made some annoying changes to the release download pages (lazy loading?)
> All debian/watch files using github are failing now:
>
> version=4
> https://github.com/liske/needrestart/releases \
> /liske/needrestart/archive/refs/tags/v(.+)\.tar\.gz

uscan still seems to work using the GitHub API, e.g.,

version=4
opts="searchmode=plain,pgpsigurlmangle=s/$/.asc/" \
https://api.github.com/repos/<user>/<project>/releases \
https://github.com/<user>/<project>/releases/download/v(?:\d[\d.]*)/<project>-v(\d[\d.]*)\.tar\.gz

Peter

Alban Browaeys

unread,

Sep 22, 2022, 10:30:03 AM9/22/22

to

On Tue, 13 Sep 2022 17:24:02 +0200 =?utf-8?q?Patrick_Matth=C3=A4i?=
<pmat...@debian.org> wrote:
> Package: devscripts
> Version: 2.22.2
> Severity: important
>
> Hello,
>
> github made some annoying changes to the release download pages (lazy
loading?)
> All debian/watch files using github are failing now:
>
> version=4
> https://github.com/liske/needrestart/releases \
> /liske/needrestart/archive/refs/tags/v(.+)\.tar\.gz
>

/releases pages listing is broken but not /tags: "
version=4
https://github.com/liske/needrestart/tags \
/liske/needrestart/archive/refs/tags/v(.+)\.tar\.gz
" should do.

Kind regards,

Alban

Sacha Hony

unread,

Sep 24, 2022, 9:10:04 AM9/24/22

to

Hello,

Unfortunately both proposed work-arounds have some issues:

1) Passing via api.github.com without authentication has a rate-limit of
60 requests per hour.
2) Passing via /tags does not give access to all tarballs listed on the
releases page. Notable problem are those projects where tags does not
include submodules. These are often listed on the releases page as
<PROJECTNAME>-<VERSION>.tar.(gz|bz2|xz).

One such example of tarballs not available on the tags page (of many)
is: https://github.com/cracklib/cracklib/releases/latest

In theory, it seems technically feasible to obtain the URL where the
release tarballs are found. On the /releases/latest there is an
include-fragment pointing to an URL /releases/expanded_assets. This URL
contains the tarballs previously listed directly on the /releases/latest
page.

I do not know if it is technically feasible for uscan to follow such a
link.

With kindest regards,

Sacha Hony

Judit Foglszinger

unread,

Sep 25, 2022, 10:30:04 PM9/25/22

to

Attached a suggestion from IRC for a workaround - the author is ok with sharing it here -
I mostly just want to throw it there, for the case someone feels inspired by it ;)
Have tested it with gallery-dl, but not any more packages.

0001-Fixing-handling-of-github-address.patch

David Steele

unread,

Sep 29, 2022, 4:40:04 PM9/29/22

to

Another workaround is to us the 'downloadurlmangle' option to map a
discovered 'tags' url to the expected 'releases' link.

From the sirikali package:

version=4

opts=downloadurlmangle=s/archive\/refs\/tags\/(.*)\.tar\.gz/releases\/download\/$1\/@PACKAGE@-$1\.tar\.xz/,\
pgpsigurlmangle=s/$/.asc/,\
dversionmangle=s/\+dfsg1//,repacksuffix=+dfsg1 \
https://github.com/mhogomchungu/@PACKAGE@/tags \
/mhogomchungu/@PACKAGE@/archive/refs/tags/(.*)\.tar\.gz

Note that this watch file is looking for a '.xz' release file and
.xz.asc signature file.

The pre-existing pgpsigurlmangle continued to work after the change.

--
AE0D BF5A 92A5 ADE4 9481 BA6F 8A31 71EF 3661 50CE