Bug#1058579: apt: gives misleading error when not finding Packages.xz in Release (not InRelease)

[Yann Dirson]

Package: apt
Version: 2.6.1

# context

The repo is served as a "generic package repo" on gitlab. As a first step I'm
putting unsigned Release file there, because setting sigs there is another
adventure.

So I have Release not InRelease, and since it's 2 packages I chose to spare
space using just a Package.gz, hoping for maximum compatibility (apparently
mistakenly so).


# observations

When updating, apt acknowledges it got Release not InRelease, but its error message
seems to imply it checked *InRelease* to find a *Packages* file:

root@debian:~# apt update
Hit:1 http://deb.debian.org/debian bookworm InRelease
Hit:2 http://security.debian.org/debian-security bookworm-security InRelease
Ign:3 https://gitlab.com/api/v4/projects/xen-project%2Fxen-guest-agent/packages/generic/deb-amd64 ci/ InRelease
Hit:4 https://gitlab.com/api/v4/projects/xen-project%2Fxen-guest-agent/packages/generic/deb-amd64 ci/ Release
Ign:5 https://gitlab.com/api/v4/projects/xen-project%2Fxen-guest-agent/packages/generic/deb-amd64 ci/ Release.gpg
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
77 packages can be upgraded. Run 'apt list --upgradable' to see them.
W: Skipping acquire of configured file 'Packages' as repository 'https://gitlab.com/api/v4/projects/xen-project%2Fxen-guest-agent/packages/generic/deb-amd64 ci/ InRelease' does not seem to provide it (sources.list entry misspelt?)


Using "-oDebug::pkgAcquire::Worker=1 -o Debug::Acquire::https=1" indeed shows no attempt
at downloading anything after Release.gpg


# my interpretation

There are 3 misleading items in the same statement:
* it likely did not check *InRelease* contents but really *Release*
* OK it did not find *Packages* but only after looking for *Packages.xz*, and since
adding *Packages* back does work, it does not really push users to use the default
compression format.
* the "sources.list entry misspelt?" suggestion feels to throw the user completely off-track:
as it did find a Release file, the entry surely *does* point to a repo

[Yann Dirson]

I note after double-checking that, despite bookworm repos [1] providing
only Packages.xz and Packages.gz and not Packages, apt does really
seem to check only for Packages in my case:

root@debian:~# apt update
Hit:1 http://security.debian.org/debian-security bookworm-security InRelease
Hit:2 http://deb.debian.org/debian bookworm InRelease

Ign:3 https://gitlab.com/api/v4/projects/xen-project%2Fxen-guest-agent/packages/generic/deb-amd64 ci/ InRelease
Hit:4 https://gitlab.com/api/v4/projects/xen-project%2Fxen-guest-agent/packages/generic/deb-amd64 ci/ Release
Ign:5 https://gitlab.com/api/v4/projects/xen-project%2Fxen-guest-agent/packages/generic/deb-amd64 ci/ Release.gpg
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
77 packages can be upgraded. Run 'apt list --upgradable' to see them.
W: Skipping acquire of configured file 'Packages' as repository 'https://gitlab.com/api/v4/projects/xen-project%2Fxen-guest-agent/packages/generic/deb-amd64 ci/ InRelease' does not seem to provide it (sources.list entry misspelt?)

root@debian:~# curl https://gitlab.com/api/v4/projects/xen-project%2Fxen-guest-agent/packages/generic/deb-amd64/ci/Release
Date: Wed, 13 Dec 2023 09:55:54 +0000
Description: xen-guest-agent CI packages
Label: xen-guest-agent-ci
Suite: ci
MD5Sum:
4bbd32da77623285b8a54ef926a1f028 277 Contents-amd64.gz
9c8c7743a78d4c850fbdffac54c5e159 340 Contents-amd64.xz
557f3042ec38f51a838d518739ecf4c2 925 Packages.gz
a3b82abf0ab81c4d5f829ec631c9deb8 1008 Packages.xz
SHA1:


[1] http://ftp.debian.org/debian/dists/bookworm/main/binary-all/

[Yann Dirson]

> I note after double-checking that, despite bookworm repos [1]
> providing
> only Packages.xz and Packages.gz and not Packages, apt does really
> seem to check only for Packages in my case:

After more testing, it appears that the behavior is different depending on
the way the flat repository is used:

* sources.list with ".../deb-amd64/ ci/" (needs "Filename: ci/foo.deb" in Packages)
=> on "Packages" is checked, "Packages.xz" is ignored as previously described

* sources.list with ".../deb-amd64/ci/ ./" (needs "Filename: ./foo.deb" in Packages)
=> a long list of compressed formats is used

I'm a bit unsure whether this would be expected.

Anyway (assuming this is expected), the part of the error message mentioning "Packages"
name may in fact not be wrong per se, but in this (legacy?) use case it could possibly
be noted that it has to be uncompressed (since "Packages" is otherwise also AFACT used
to encompass compressed versions as well).

Best regards,
--
Yann

[David Kalnischkies]

On Wed, Dec 13, 2023 at 11:29:29AM +0100, Yann Dirson wrote:
> W: Skipping acquire of configured file 'Packages' as repository 'https://gitlab.com/api/v4/projects/xen-project%2Fxen-guest-agent/packages/generic/deb-amd64 ci/ InRelease' does not seem to provide it (sources.list entry misspelt?)

Sadly, this relates to your later observation:

> After more testing, it appears that the behavior is different
> depending on the way the flat repository is used:

[…]

> I'm a bit unsure whether this would be expected.

The "funny" thing about flat repositories is that nearly everything can
be one: http://deb.debian.org/debian/dists/sid/main/binary-amd64/ is
e.g. a flat repository as well… unsigned and without any sort of hashes
so apt will hate your guts by default, but with enough force it will
gobble it up anyhow (it contains a Release file, but that is optional as
well…).

So… if a repository you added doesn't contain a Packages file there is
a good chance that you misspelt something which happened to be still
a valid repository. Perhaps nowadays, that apt even dislikes unsigned
repos we could while not trim the support for these things at least
review and edit messages assuming that users will nowadays only interact
with proper repos…

Messages referring to a repository will usually speak about InRelease
as that is convenient to access from a code point of view. It doesn't
mean the file is used – or even exists. Its just that the code who
handles the file also handles if it doesn't exist, manages the fallbacks
and so on. In an ideal world we would have an easier and better way of
referring to a repository, but in this world we so far don't. There
probably is an "upgrade path", but I don't have the cycles to think
about it at the moment (nor while implementing part of that message
a few years ago). It is somehwhat similar with how messages tend to
talk nearly interchangeable about 'Packages' and 'Packages.xz' even
if neither exists and we end up using 'Packages.gz' (and Packages.lz4).

So, the gist is that yes, it can be misleading, but all individual parts
are correct or have at least a good explanation. I also don't see a lot
of ways to improve on it given that the message is already way too long
and we can not go full analyse mode on untrusted data either…

> root@debian:~# curl https://gitlab.com/api/v4/projects/xen-project%2Fxen-guest-agent/packages/generic/deb-amd64/ci/Release
> Date: Wed, 13 Dec 2023 09:55:54 +0000
> Description: xen-guest-agent CI packages
> Label: xen-guest-agent-ci
> Suite: ci
> MD5Sum:
> 4bbd32da77623285b8a54ef926a1f028 277 Contents-amd64.gz
> 9c8c7743a78d4c850fbdffac54c5e159 340 Contents-amd64.xz
> 557f3042ec38f51a838d518739ecf4c2 925 Packages.gz
> a3b82abf0ab81c4d5f829ec631c9deb8 1008 Packages.xz

APT wants checksums for the uncompressed Packages file – even if they
are not shipped in the repository – the info has to be there so that
apt can check that the data it got after uncompressing is really the
intended one. It will also be needed "later" for updates for the data
as files could be stored (re)compressed [different compressor versions
can produce different files having different checksums. Also, we tend to
use compressions for storage which aren't available online to begin
with], not to mention reverify, pdiffs, …

I said wants as there is still support for repos who don't have hashes
and so if you play your cards right, you could manage to make apt accept
your repo, but please don't. Just provide checksums for the uncompressed
files and be done.



So, in summary, I am not sure what to do about this… on some level
I agree, others I consider a user error, but if I look at the cost-
benefit ratio of it all I don't see the parts changing that would need
changing anytime soon so that this report has a very good chance of
staying open until the heat death of the universe (or apt being removed
from Debian, but that is only a theoretical possibility of course).


Best regards

David Kalnischkies