Bug#1057431: fakeupstream.cgi: add support for static.rust-lang.org

[Zixing Liu]

Package: qa.debian.org
Severity: wishlist
Tags: patch
X-Debbugs-Cc: zixin...@canonical.com

Hi Debian QA Team,

I would like to see a fakeupstream service implemented for
https://static.rust-lang.org/dist so that the debian/watch file inside the
rustc package could be simplified and actually useful for checking new
releases.

Attached below is a patch that I proposed to add the said functionality.
You can find the same patch at:
https://salsa.debian.org/qa/qa/-/merge_requests/48
if you prefer using Debian Salsa to merge the patch/merge request.

Thanks,
Zixing

[Bart Martens]

Hello Zixing Liu,

Can I view an example of a package in Debian that can be watched with such
extension of fakeupstream? The link https://static.rust-lang.org/dist displays
an error in xml format. A real example would help me understand the context.

Ciao,

Bart

[Zixing Liu]

Hi Bart,

> The link https://static.rust-lang.org/dist displays an error in xml format.

This is because this website is hosted on AWS S3 and is machine-readable only.
To list the files, you need to either use an S3-compatible tool or visit
https://static.rust-lang.org/?prefix=dist/ to see the file listing in
XML format.

> Can I view an example of a package in Debian that can be watched with such extension of fakeupstream?

This fakeupstream service is intended for https://tracker.debian.org/pkg/rustc.
Because it needs to download the signed source tarball from
https://static.rust-lang.org.

Thanks,
Zixing

On Tue, Dec 5, 2023 at 1:46 PM Bart Martens <ba...@debian.org> wrote:
>
> Hello Zixing Liu,
>
> Can I view an example of a package in Debian that can be watched with such
> extension of fakeupstream? The link https://static.rust-lang.org/dist displays
> an error in xml format. A real example would help me understand the context.
>
> Ciao,
>
> Bart
>
>
> On Mon, Dec 04, 2023 at 03:32:39PM -0700, Zixing Liu wrote:

[Zixing Liu]

Hi Bart,

> That should be suitable for a watch file. Right?

Yes, while that is true, the page is updated frequently with older
versions replaced with the new version.
In Debian, usually, we can't package the latest version as quickly as
possible due to bootstrapping requirements.
Hence, a fakeupstream service is needed to retrieve a non-latest
version from the upstream.

Also, another use-case might be, you want a specific nightly or beta
to test a certain regression. Since that page
does not list the previous releases, it would be difficult to rely on
it to retrieve a previous beta using uscan by
specifying `--download-version`.

Thanks,
Zixing

On Tue, Dec 12, 2023 at 2:58 PM Bart Martens <ba...@debian.org> wrote:
>
> Hi Zixing, the source code and signature are also advertised here:
> https://forge.rust-lang.org/infra/other-installation-methods.html#source-code
> That should be suitable for a watch file. Right?
>
>
> On Tue, Dec 05, 2023 at 02:44:06PM -0700, Zixing Liu wrote:
> > Hi Bart,

> >
> > > The link https://static.rust-lang.org/dist displays an error in xml format.
> >

> > This is because this website is hosted on AWS S3 and is machine-readable only.
> > To list the files, you need to either use an S3-compatible tool or visit
> > https://static.rust-lang.org/?prefix=dist/ to see the file listing in
> > XML format.
> >

> > > Can I view an example of a package in Debian that can be watched with such extension of fakeupstream?
> >

> > This fakeupstream service is intended for https://tracker.debian.org/pkg/rustc.
> > Because it needs to download the signed source tarball from
> > https://static.rust-lang.org.
> >
> > Thanks,
> > Zixing
> >
> > On Tue, Dec 5, 2023 at 1:46 PM Bart Martens <ba...@debian.org> wrote:
> > >

> > > Hello Zixing Liu,
> > >
> > > Can I view an example of a package in Debian that can be watched with such
> > > extension of fakeupstream? The link https://static.rust-lang.org/dist displays
> > > an error in xml format. A real example would help me understand the context.
> > >
> > > Ciao,
> > >
> > > Bart
> > >
> > >
> > > On Mon, Dec 04, 2023 at 03:32:39PM -0700, Zixing Liu wrote:

> > > > Package: qa.debian.org
> > > > Severity: wishlist
> > > > Tags: patch
> > > > X-Debbugs-Cc: zixin...@canonical.com
> > > >
> > > > Hi Debian QA Team,
> > > >
> > > > I would like to see a fakeupstream service implemented for
> > > > https://static.rust-lang.org/dist so that
> > >
>

> --

[Bart Martens]

Hi Zixing,

The newest stable upstream release is advertised in more than one way allowing
to write regular debian/watch files. Here below I give two working examples.

| version=4
| https://forge.rust-lang.org/infra/other-installation-methods.html#source-code \
| https://static.rust-lang.org/dist/rustc-(\d+(?:\.\d+)*)-src\.tar\.gz
|
| $ uscan --safe --package rustc --upstream-version 1.70.0+dfsg1 --watchfile watch1
| Newest version of rustc on remote site is 1.74.1, local version is 1.70.0+dfsg1
| => Newer package available from:
| => https://static.rust-lang.org/dist/rustc-1.74.1-src.tar.gz
|
| version=4
| https://github.com/rust-lang/rust/tags \
| .*/rust-lang/rust/archive/refs/tags/(\d+(?:\.\d+)*)\.tar\.gz
|
| Newest version of rustc on remote site is 1.74.1, local version is 1.70.0+dfsg1
| => Newer package available from:
| => https://github.com/rust-lang/rust/archive/refs/tags/1.74.1.tar.gz

And the watch file currently in Debian works fine. The version on the package
tracker is correct. https://tracker.debian.org/pkg/rustc

| A new upstream version is available: 1.74.1 high
| A new upstream version 1.74.1 is available, you should consider packaging it.

Fakeupstream is for when the above would be impossible.

In my understanding of the described special use cases, they can be covered
separately. For example, some scraping script can collect the info 4x/day and
produce a html page suitable for debian/watch. The loops in your patch fit
better in such script than in a cgi script.

Or maybe upstream is willing to put the info you need on their website. Have
you asked them?

Cheers,

Bart