Q: uscan with GitHub

Hideki Yamane

unread,

Sep 19, 2022, 12:00:02 PM9/19/22

to

Hi,

Recent changes in GitHub releases pages, I cannot check upstream version
with uscan. How do you deal with it?

$ cat debian/watch
version=4
https://github.com/KittyKatt/screenFetch/releases /KittyKatt/screenFetch/archive/refs/tags/v*@ANY_VERSION@@ARCHIVE_EXT@

$ uscan --no-download
uscan warn: In debian/watch no matching files for watch line
https://github.com/KittyKatt/screenFetch/releases /KittyKatt/screenFetch/archive/refs/tags/v*(?:[-_]?(\d[\-+\.:\~\da-zA-Z]*))(?i)(?:\.(?:tar\.xz|tar\.bz2|tar\.gz|tar\.zstd?|zip|tgz|tbz|txz))

--
Regards,

Hideki Yamane henrich @ debian.org/iijmio-mail.jp

julien...@gmail.com

unread,

Sep 19, 2022, 12:10:03 PM9/19/22

to

Hi,

Le lundi 19 septembre 2022 à 20:50 +0900, Hideki Yamane a écrit :
>
> Recent changes in GitHub releases pages, I cannot check upstream
> version with uscan. How do you deal with it?

It's not that recent ; here is a working example:

version=4
opts="\
dversionmangle=s/\+dfsg.*$//,\
filenamemangle=s/.+\/[vV]?(\d\S+)\.tar\.gz/coq-$1\.tar\.gz/,\
repack,repacksuffix=+dfsg,\
" \
https://github.com/coq/coq/tags .*/[vV]?(\d[^\s+]+)\.tar\.gz

Cheers,

J.Puydt

Samuel Thibault

unread,

Sep 19, 2022, 12:20:03 PM9/19/22

to

julien...@gmail.com, le lun. 19 sept. 2022 18:00:37 +0200, a ecrit:

That works for the tags page, but not for the releases page... The
problem is that the tags page only contains snapshots of the repository,
and not an autoconf-ed tarball. For instance since recently uscan cannot
get the correct tarballs from

https://github.com/brailcom/speechd/releases

Samuel

Simon McVittie

unread,

Sep 19, 2022, 12:50:03 PM9/19/22

to

On Mon, 19 Sep 2022 at 18:12:15 +0200, Samuel Thibault wrote:
> julien...@gmail.com, le lun. 19 sept. 2022 18:00:37 +0200, a ecrit:
> > Le lundi 19 septembre 2022 à 20:50 +0900, Hideki Yamane a écrit :
> > > Recent changes in GitHub releases pages, I cannot check upstream
> > > version with uscan. How do you deal with it?
> >

> > version=4
> > opts="\
> > dversionmangle=s/\+dfsg.*$//,\
> > filenamemangle=s/.+\/[vV]?(\d\S+)\.tar\.gz/coq-$1\.tar\.gz/,\
> > repack,repacksuffix=+dfsg,\
> > " \
> > https://github.com/coq/coq/tags .*/[vV]?(\d[^\s+]+)\.tar\.gz
>
> That works for the tags page, but not for the releases page... The
> problem is that the tags page only contains snapshots of the repository,
> and not an autoconf-ed tarball.

It's quite noisy and unpleasant, but here's the best I could come up with:
https://salsa.debian.org/debian/bubblewrap/-/blob/debian/latest/debian/watch
(my upstream here uses tags like v1.2.3, adjust as necessary if yours
doesn't)

or if you need to repack for DFSG reasons:
https://salsa.debian.org/sdl-team/libsdl2/-/blob/master/debian/watch
(in this case my upstream uses tags like release-1.2.3)

smcv

Andrea Pappacoda

unread,

Sep 19, 2022, 1:00:03 PM9/19/22

to

Il giorno lun 19 set 2022 alle 18:12:15 +02:00:00, Samuel Thibault
<sthi...@debian.org> ha scritto:

> That works for the tags page, but not for the releases page... The
> problem is that the tags page only contains snapshots of the
> repository,
> and not an autoconf-ed tarball. For instance since recently uscan
> cannot
> get the correct tarballs from
>
> https://github.com/brailcom/speechd/releases

Hi, what I usually do with GitHub is to use its API, since it has the
advantage of not breaking uscan when they do changes to the web UI.

See
<https://salsa.debian.org/debian/yuzu/-/blob/debian/latest/debian/watch>
for example.

In your case, as you have to download an asset and not the automatic
tarball produced by GitHub, I'd do something like this:

version=4
opts="searchmode=plain, \

filenamemangle=s/.+\/speech-dispatcher-@ANY_VERSION@.tar.gz/@PACKAGE@-$1\.tar\.gz/"
\
https://api.github.com/repos/brailcom/speechd/releases \

https://github.com/brailcom/speechd/releases/download/\d[\.\d]*/speech-dispatcher-@ANY_VERSION@.tar.gz

It could probably be done in a cleaner way, but my knowledge of regular
expressions is pretty limited :/

--
OpenPGP key: 66DE F152 8299 0C21 99EF A801 A8A1 28A8 AB1C EE49

Paul Wise

unread,

Sep 19, 2022, 8:40:03 PM9/19/22

to

On Mon, 2022-09-19 at 18:54 +0200, Andrea Pappacoda wrote:

> Hi, what I usually do with GitHub is to use its API, since it has the
> advantage of not breaking uscan when they do changes to the web UI.

Since the API uses pagination, this has the minor downside of making it
harder to use the uscan --download-version option with older versions.

--
bye,
pabs

https://wiki.debian.org/PaulWise

signature.asc

Paul Wise

unread,

Sep 19, 2022, 8:40:03 PM9/19/22

to

On Mon, 2022-09-19 at 20:50 +0900, Hideki Yamane wrote:

> Recent changes in GitHub releases pages, I cannot check upstream
> version with uscan. How do you deal with it?

If you are using the automatically generated tarballs, then
just switching to the uscan git mode seems like the way to go.

For signed tarballs and other GitHub release artefacts, it sounds like
uscan will need to use the GitHub API or Debian QA fakeupstream.cgi
will need to add a redirector based on the GitHub API.

https://salsa.debian.org/qa/qa/-/blob/master/cgi-bin/fakeupstream.cgi
https://qa.debian.org/cgi-bin/fakeupstream.cgi

signature.asc

Paul Wise

unread,

Sep 19, 2022, 8:50:03 PM9/19/22

to

On Mon, 2022-09-19 at 18:12 +0200, Samuel Thibault wrote:

> The problem is that the tags page only contains snapshots of the
> repository, and not an autoconf-ed tarball.

I think that is an advantage, because then you can be sure that you can
rebuild the autotools generated files from source during the build.

The only reason I would use the autotools tarball is when it is signed.

signature.asc

Samuel Henrique

unread,

Sep 20, 2022, 4:40:03 AM9/20/22

to

On Tue 20 Sept 2022, 01:39 Paul Wise, <pa...@debian.org> wrote:

On Mon, 2022-09-19 at 18:54 +0200, Andrea Pappacoda wrote:

> Hi, what I usually do with GitHub is to use its API, since it has the
> advantage of not breaking uscan when they do changes to the web UI.

Since the API uses pagination, this has the minor downside of making it
harder to use the uscan --download-version option with older versions.

The GitHub pages are also paginated, so you get the same issue whether using the API or not.

I have been hit by this on cases where upstream does enough beta|rc|snapshots releases such that the latest GA release gets in the second page and uscan fails due to not finding it.

Stephan Lachnit

unread,

Sep 20, 2022, 6:10:03 AM9/20/22

to

On Tue, Sep 20, 2022 at 10:36 AM Samuel Henrique <samu...@debian.org> wrote:

On Tue 20 Sept 2022, 01:39 Paul Wise, <pa...@debian.org> wrote:
On Mon, 2022-09-19 at 18:54 +0200, Andrea Pappacoda wrote:

> Hi, what I usually do with GitHub is to use its API, since it has the
> advantage of not breaking uscan when they do changes to the web UI.

Since the API uses pagination, this has the minor downside of making it
harder to use the uscan --download-version option with older versions.

The GitHub pages are also paginated, so you get the same issue whether using the API or not.

There is a per_page parameter: https://docs.github.com/en/rest/releases/releases#list-releases

E.g. https://api.github.com/repos/torvalds/linux/tags?per_page=100

The maximum is 100 though.

Cheers,

Stephan

Bastian Blank

unread,

Sep 20, 2022, 7:00:03 AM9/20/22

to

On Tue, Sep 20, 2022 at 08:36:48AM +0800, Paul Wise wrote:
> On Mon, 2022-09-19 at 18:54 +0200, Andrea Pappacoda wrote:
> > Hi, what I usually do with GitHub is to use its API, since it has the
> > advantage of not breaking uscan when they do changes to the web UI.
> Since the API uses pagination, this has the minor downside of making it
> harder to use the uscan --download-version option with older versions.

Well, uscan can properly implement pagination. It is just not a simple
scraper anymore.

Bastian

--
It is more rational to sacrifice one life than six.
-- Spock, "The Galileo Seven", stardate 2822.3

Johannes Schauer Marin Rodrigues

unread,

Oct 2, 2022, 3:10:03 PM10/2/22

to

Hi,

Quoting Paul Wise (2022-09-20 02:38:30)

> On Mon, 2022-09-19 at 20:50 +0900, Hideki Yamane wrote:
> > Recent changes in GitHub releases pages, I cannot check upstream
> > version with uscan. How do you deal with it?
>
> If you are using the automatically generated tarballs, then
> just switching to the uscan git mode seems like the way to go.

the only reason I'm not using git mode for all my upstream projects that use
git is this paragraph from the uscan man page:

If the upstream publishes the released tarball via its web
interface, please use it instead of using this mode. This mode is
the last resort method.

This sounds like I should rather use other methods like rewriting my d/watch
files to use api.github.com before using this "last resort" method.

Is this paragraph still accurate?

Thanks!

cheers, josch

signature.asc

Volans

unread,

Oct 9, 2022, 1:10:04 PM10/9/22

to

On Mon, 19 Sep 2022 at 18:12:15 +0200, Samuel Thibault wrote:
> julien...@gmail.com, le lun. 19 sept. 2022 18:00:37 +0200, a ecrit:

> > Le lundi 19 septembre 2022 à 20:50 +0900, Hideki Yamane a écrit :
> > > Recent changes in GitHub releases pages, I cannot check upstream
> > > version with uscan. How do you deal with it?
> >

> > version=4
> > opts="\
> > dversionmangle=s/\+dfsg.*$//,\
> > filenamemangle=s/.+\/[vV]?(\d\S+)\.tar\.gz/coq-$1\.tar\.gz/,\
> > repack,repacksuffix=+dfsg,\
> > " \
> > https://github.com/coq/coq/tags .*/[vV]?(\d[^\s+]+)\.tar\.gz
>

> That works for the tags page, but not for the releases page... The

> problem is that the tags page only contains snapshots of the repository,
> and not an autoconf-ed tarball.

It's quite noisy and unpleasant, but here's the best I could come up with:
https://salsa.debian.org/debian/bubblewrap/-/blob/debian/latest/debian/watch
(my upstream here uses tags like v1.2.3, adjust as necessary if yours
doesn't)

or if you need to repack for DFSG reasons:
https://salsa.debian.org/sdl-team/libsdl2/-/blob/master/debian/watch
(in this case my upstream uses tags like release-1.2.3)

smcv

I've encountered the same problem and come out with a version of the watch file using the GitHub APIs. In my case I also needed to download the PGP signature. I've add some comments to the watch file for an easier understanding.

The approach should be generic enough to work for others [1].

The package name in the URL could also be replaced with @PACKAGE@ if that's the same of the GitHub project name.

In my case too, tags are prefixed with a 'v' (i.e. v1.2.3), but that's easily adaptable to different formats.

What is the process to find an agreement on which updated version should be published on the wiki [2] ?

Thanks,

Riccardo

[1] https://github.com/volans-/gjson-py/blob/debian/debian/watch

[2] https://wiki.debian.org/debian/watch#GitHub

Stephan Lachnit

unread,

Oct 11, 2022, 4:30:03 AM10/11/22

to

On Sun, Oct 9, 2022 at 7:06 PM Volans <volans...@gmail.com> wrote:
>
> I've encountered the same problem and come out with a version of the watch file using the GitHub APIs. In my case I also needed to download the PGP signature. I've add some comments to the watch file for an easier understanding.
>
> The approach should be generic enough to work for others [1].
>
> The package name in the URL could also be replaced with @PACKAGE@ if that's the same of the GitHub project name.
> In my case too, tags are prefixed with a 'v' (i.e. v1.2.3), but that's easily adaptable to different formats.
>
> What is the process to find an agreement on which updated version should be published on the wiki [2] ?

I mentioned it before, one should increase the limit of results per
page to the maximum (which is 100).
Also I don't think using `@PACKAGE@` is a good idea inside the URL as
it is maybe different from the Debian source package name.

For releases I use something like (w/o signatures):
```
version=4
opts="searchmode=plain,\
filenamemangle=s%v?@ANY_VERSION@%@PACKAGE@-$1.tar.gz%" \
https://api.github.com/repos/<user>/<project>/releases?per_page=100 \
https://api.github.com/repos/[^/]+/[^/]+/tarball/v?@ANY_VERSION@
```
For tags:
```
version=4
opts="searchmode=plain,\
filenamemangle=s%v?@ANY_VERSION@%@PACKAGE@-$1.tar.gz%" \
https://api.github.com/repos/<user>/<project>/tags?per_page=100 \
https://api.github.com/repos/[^/]+/[^/]+/tarball/refs/tags/v?@ANY_VERSION@
```

One could probably combine the tarball search link into a unified
regex expression, didn't try that yet.

Overall I agree that the Github API thing should be documented in the
Wiki and in the uscan man page.

Cheers,
Stephan

Volans

unread,

Oct 25, 2022, 9:00:03 AM10/25/22

to

Sure, I agree for the API page size, thanks for the suggestion.

To add some additional burden I noticed that the archive downloaded from the API [1] is different from the one downloaded from the browser [2] because they have a different top-level directory name, while the API one has:

{OWNER}-{REPO}-{REF_SHA1}/

where REF_SHA1 is the SHA1 of refs/tags/{TAG_NAME}, the browser one has:

{OWNER}-{VERSION}/

where, at least in my case, VERSION is the tag name without the leading 'v'. For a tag name of 'v0.2.0' I get 'gjson-py-0.2.0.tar.gz' that extracted creates 'gjson-py-0.2.0/' (note the lack of 'v' in the names).

Because of the two different directory structures, the tar.gz files are different so any md5 or gpg signature verification made on one would fail with the other, adding additional confusion or complexity.

For now I'll probably upload to the release page the signatures of both files, but this is definitely suboptimal.

Riccardo

[1] https://api.github.com/repos/{OWNER}/{REPO}/tarball/{RELEASE}

[2] https://codeload.github.com/{OWNER}/{REPO}/tar.gz/refs/tags/{RELEASE}

Jens Reyer

unread,

Feb 19, 2023, 12:40:04 PM2/19/23

to

[This is a followup to the thread "Q: uscan with GitHub" at
https://lists.debian.org/debian-devel/2022/10/msg00313.html. I manually
set in-reply-to, but am not sure if it'll work.]

> On Tue, Oct 11, 2022 at 10:23 AM Stephan Lachnit
<stephan...@debian.org> wrote:
>> On Sun, Oct 9, 2022 at 7:06 PM Volans <volans...@gmail.com> wrote:

[...]

> For releases I use something like (w/o signatures): ``` version=4
> opts="searchmode=plain,\
> filenamemangle=s%v?@ANY_VERSION@%@PACKAGE@-$1.tar.gz%" \
> https://api.github.com/repos/<user>/<project>/releases?per_page=100
> \ https://api.github.com/repos/[^/]+/[^/]+/tarball/v?@ANY_VERSION@
> ```

[...]

> To add some additional burden I noticed that the archive downloaded
> from the API [1] is different from the one downloaded from the
> browser [2] because they have a different top-level directory name,
> while the API one has: {OWNER}-{REPO}-{REF_SHA1}/ where REF_SHA1 is
> the SHA1 of refs/tags/{TAG_NAME}, the browser one has:
> {OWNER}-{VERSION}/ where, at least in my case, VERSION is the tag
> name without the leading 'v'. For a tag name of 'v0.2.0' I get
> 'gjson-py-0.2.0.tar.gz' that extracted creates 'gjson-py-0.2.0/'
> (note the lack of 'v' in the names).
>
> Because of the two different directory structures, the tar.gz files
> are different so any md5 or gpg signature verification made on one
> would fail with the other, adding additional confusion or complexity.
> For now I'll probably upload to the release page the signatures of
> both files, but this is definitely suboptimal.
>
> Riccardo
>
> [1] https://api.github.com/repos/{OWNER}/{REPO}/tarball/{RELEASE}
> [2]
> https://codeload.github.com/{OWNER}/{REPO}/tar.gz/refs/tags/{RELEASE}

I use the mentioned tarball from the browser release page, but it still
fails to verify with upstream's locally created signature:

My upstream creates a tarball with git-archive, creates a signature and
uploads it (as described in the wiki[3]). This used to work to verify
the github-created tarball, but fails now - while creating my own
tarball like upstream and verifying it with upstream's signature works.

The uncompressed .tar files are identical (same hashsum), just the
tar.gz differ. Does anyone know why, and how to fix it? I tried
non-default compression levels for gzip with git-archive, but that
didn't help to get an identical tar.gz like the one from github.

I'd like to avoid having my upstream downloading the github-created
tarball, verify&sign it and then upload this signature.

Greets
jre

[3]The "alternative local workflow" described at
https://wiki.debian.org/Creating%20signed%20GitHub%20releases

Guillem Jover

unread,

Feb 19, 2023, 3:00:04 PM2/19/23

to

Hi!

On Sun, 2023-02-19 at 18:34:56 +0100, Jens Reyer wrote:
> [This is a followup to the thread "Q: uscan with GitHub" at
> https://lists.debian.org/debian-devel/2022/10/msg00313.html. I manually set
> in-reply-to, but am not sure if it'll work.]

> My upstream creates a tarball with git-archive, creates a signature and
> uploads it (as described in the wiki[3]). This used to work to verify
> the github-created tarball, but fails now - while creating my own
> tarball like upstream and verifying it with upstream's signature works.
>
> The uncompressed .tar files are identical (same hashsum), just the tar.gz
> differ. Does anyone know why, and how to fix it? I tried non-default
> compression levels for gzip with git-archive, but that didn't help to get an
> identical tar.gz like the one from github.
>
> I'd like to avoid having my upstream downloading the github-created
> tarball, verify&sign it and then upload this signature.

I assume you (or whatever service or tool is failing the verification
while creating a local tarball) might be seeing issues with git having
switched implementation for gzip, and a mismatch with the implementation
being used in either side. Perhaps try to set git's
tar.tar.gz.command="gzip -c" (or/and «tgz» instead of «tar.gz») to use
the external command instead of the internal implementation? Or perhaps
you are using an old git that defaults to the external gzip but upstream
uses the internal one?

(There was a recent LWN article covering this, see
https://lwn.net/Articles/921787/.)

Thanks,
Guillem

FC Stegerman

unread,

Feb 19, 2023, 3:10:03 PM2/19/23

to

* Guillem Jover <gui...@debian.org> [2023-02-19 20:50]:

> > My upstream creates a tarball with git-archive, creates a signature and
> > uploads it (as described in the wiki[3]). This used to work to verify
> > the github-created tarball, but fails now - while creating my own
> > tarball like upstream and verifying it with upstream's signature works.
> >
> > The uncompressed .tar files are identical (same hashsum), just the tar.gz
> > differ. Does anyone know why, and how to fix it? I tried non-default
> > compression levels for gzip with git-archive, but that didn't help to get an
> > identical tar.gz like the one from github.
> >
> > I'd like to avoid having my upstream downloading the github-created
> > tarball, verify&sign it and then upload this signature.
>
> I assume you (or whatever service or tool is failing the verification
> while creating a local tarball) might be seeing issues with git having
> switched implementation for gzip, and a mismatch with the implementation
> being used in either side. Perhaps try to set git's
> tar.tar.gz.command="gzip -c" (or/and «tgz» instead of «tar.gz») to use
> the external command instead of the internal implementation? Or perhaps
> you are using an old git that defaults to the external gzip but upstream
> uses the internal one?

I was going to suggest that might be the issue, but you were faster :)
I do have some relevant links:

https://reproducible-builds.org/reports/2023-01/#news
https://lists.reproducible-builds.org/pipermail/rb-general/2022-October/002709.html
https://lists.reproducible-builds.org/pipermail/rb-general/2022-October/002710.html

> (There was a recent LWN article covering this, see
> https://lwn.net/Articles/921787/.)

That seems to be subscribers-only :(

- FC

Jens Reyer

unread,

Feb 19, 2023, 7:00:04 PM2/19/23

to

Thank you both, that solved it!

After further digging into the details I updated the wiki to suggest:

git -c tar.tar.gz.command='gzip -cn' \
archive --format=tar.gz --prefix="${tag}/" \
-o "../${tag}.tar.gz" "${tag}"

Jakub Wilk

unread,

Feb 20, 2023, 1:20:04 AM2/20/23

to

* FC Stegerman <f...@obfusk.net>, 2023-02-19 21:08:

>>(There was a recent LWN article covering this, see
>>https://lwn.net/Articles/921787/.)
>
>That seems to be subscribers-only :(

Here you go:
https://lwn.net/SubscriberLink/921787/ff1350f40f12fb8e/

--
Jakub Wilk

Stephan Verbücheln

unread,

Feb 20, 2023, 2:10:04 AM2/20/23

to

Note that kernel.org signs the raw tar file and not the compressed
file. This way, they avoid issues like that and also allow conversion
into different compression formats while the signature stays valid.

Downside is that you have to decompress it first and then hash quite a
big file for validation.

Regards

signature.asc