Bug#819852: apt-cacher-ng: support by-hash index files

Colin Watson

unread,

Apr 2, 2016, 10:40:03 PM4/2/16

to

Package: apt-cacher-ng
Version: 0.8.0-3
Severity: normal

I've very nearly finished the server-side implementation of by-hash
index files for Ubuntu. This feature was added to the client side (i.e.
apt) relatively recently, and there's a post-hoc specification here:

https://wiki.debian.org/RepositoryFormat#indices_acquisition_via_hashsums_.28by-hash.29

I just got to the point of being able to publish an unsigned test
archive with by-hash published (though not yet advertised with the
Acquire-By-Hash flag; for the moment I'm forcing it in sources.list).
My first client test failed to fetch the by-hash files and fell back to
the traditional by-name versions. For a moment I wondered why, then
realised that my usual schroot setup sends everything through
apt-cacher-ng. If I remove the proxy setup then all is well. I think
apt-cacher-ng should allow by-hash files by default.

This configuration file entry fixes the problem for me locally. I
suspect therefore that it would be enough to just add this to the
default pfilepat in source/acfg_defaults.cc (note: not vfilepat like the
by-name index files, because the point of this scheme is that the
content won't change without also changing the URL):

PfilePatternEx: /dists/.*/by-hash/.*

In case it helps, here's a debug dump from apt without this
configuration file entry:

# cat /etc/apt/sources.list
deb [trusted=yes by-hash=force] http://archive.dogfood.content.paddev.net/ubuntu xenial main universe
# apt-get -oDebug::Acquire::http=true update
0% [Working]GET http://archive.dogfood.content.paddev.net/ubuntu/dists/xenial/InRelease HTTP/1.1
Host: archive.dogfood.content.paddev.net
Cache-Control: max-age=0
Accept: text/*
User-Agent: Debian APT-HTTP/1.3 (1.2.9)

Answer for: http://archive.dogfood.content.paddev.net/ubuntu/dists/xenial/InRelease
HTTP/1.1 404 Not Found
Content-Length: 0
Date: Sun Apr 3 02:09:51 2016
Server: Debian Apt-Cacher NG/0.8.0
X-Original-Source: http://archive.dogfood.content.paddev.net/ubuntu/dists/xenial/InRelease
Connection: Keep-Alive

Ign:1 http://archive.dogfood.content.paddev.net/ubuntu xenial InRelease
0% [Working]GET http://archive.dogfood.content.paddev.net/ubuntu/dists/xenial/Release HTTP/1.1
Host: archive.dogfood.content.paddev.net
Cache-Control: max-age=0
Accept: text/*
User-Agent: Debian APT-HTTP/1.3 (1.2.9)

Answer for: http://archive.dogfood.content.paddev.net/ubuntu/dists/xenial/Release
HTTP/1.1 200 OK
Content-Length: 91416
Last-Modified: Sun, 03 Apr 2016 01:59:49 GMT
Content-Type: application/octet-stream
Date: Sun Apr 3 02:09:52 2016
Server: Debian Apt-Cacher NG/0.8.0
X-Original-Source: http://archive.dogfood.content.paddev.net/ubuntu/dists/xenial/Release
Connection: Keep-Alive

Get:2 http://archive.dogfood.content.paddev.net/ubuntu xenial Release [91.4 kB]
84% [Working]GET http://archive.dogfood.content.paddev.net/ubuntu/dists/xenial/Release.gpg HTTP/1.1
Host: archive.dogfood.content.paddev.net
User-Agent: Debian APT-HTTP/1.3 (1.2.9)

Answer for: http://archive.dogfood.content.paddev.net/ubuntu/dists/xenial/Release.gpg
HTTP/1.1 404 Not Found
Content-Length: 0
Date: Sun Apr 3 02:09:52 2016
Server: Debian Apt-Cacher NG/0.8.0
X-Original-Source: http://archive.dogfood.content.paddev.net/ubuntu/dists/xenial/Release.gpg
Connection: Keep-Alive

Ign:3 http://archive.dogfood.content.paddev.net/ubuntu xenial Release.gpg
89% [Working]GET http://archive.dogfood.content.paddev.net/ubuntu/dists/xenial/main/binary-amd64/by-hash/SHA256/9a3193137797622d5e78bd89c05fbbf726ac033fbaf7d392e6c510ef77ba36ae HTTP/1.1
Host: archive.dogfood.content.paddev.net
Cache-Control: max-age=0
Accept: text/*
User-Agent: Debian APT-HTTP/1.3 (1.2.9)

Answer for: http://archive.dogfood.content.paddev.net/ubuntu/dists/xenial/main/binary-amd64/by-hash/SHA256/9a3193137797622d5e78bd89c05fbbf726ac033fbaf7d392e6c510ef77ba36ae
HTTP/1.1 403 Forbidden file type or location: http://archive.dogfood.content.paddev.net/ubuntu/dists/xenial/main/binary-amd64/by-hash/SHA256/9a3193137797622d5e78bd89c05fbbf726ac033fbaf7d392e6c510ef77ba36ae
Content-Length: 186
Content-Type: text/html
Date: Sun Apr 3 02:09:52 2016
Server: Debian Apt-Cacher NG/0.8.0
Connection: Keep-Alive

Ign:4 http://archive.dogfood.content.paddev.net/ubuntu xenial/main amd64 Packages
50% [Working]GET http://archive.dogfood.content.paddev.net/ubuntu/dists/xenial/main/i18n/by-hash/SHA256/6c398405bfe4580bf368d174f8bf4cff7cc181dce947de6a009b43bb34c13f1a HTTP/1.1
Host: archive.dogfood.content.paddev.net
Cache-Control: max-age=0
Accept: text/*
User-Agent: Debian APT-HTTP/1.3 (1.2.9)

GET http://archive.dogfood.content.paddev.net/ubuntu/dists/xenial/universe/binary-amd64/by-hash/SHA256/0b32e4073addade93d2cac70bf9b9f26894ce13125b35485d080daaa87923afa HTTP/1.1
Host: archive.dogfood.content.paddev.net
Cache-Control: max-age=0
Accept: text/*
User-Agent: Debian APT-HTTP/1.3 (1.2.9)

GET http://archive.dogfood.content.paddev.net/ubuntu/dists/xenial/universe/i18n/by-hash/SHA256/76f8217c195476bf27a1cb2ed9de27b1b4565788f79a17703fc02475af997e84 HTTP/1.1
Host: archive.dogfood.content.paddev.net
Cache-Control: max-age=0
Accept: text/*
User-Agent: Debian APT-HTTP/1.3 (1.2.9)

Answer for: http://archive.dogfood.content.paddev.net/ubuntu/dists/xenial/main/i18n/by-hash/SHA256/6c398405bfe4580bf368d174f8bf4cff7cc181dce947de6a009b43bb34c13f1a
HTTP/1.1 403 Forbidden file type or location: http://archive.dogfood.content.paddev.net/ubuntu/dists/xenial/main/i18n/by-hash/SHA256/6c398405bfe4580bf368d174f8bf4cff7cc181dce947de6a009b43bb34c13f1a
Content-Length: 186
Content-Type: text/html
Date: Sun Apr 3 02:09:52 2016
Server: Debian Apt-Cacher NG/0.8.0
Connection: Keep-Alive

Ign:5 http://archive.dogfood.content.paddev.net/ubuntu xenial/main Translation-en
40% [Working]GET http://archive.dogfood.content.paddev.net/ubuntu/dists/xenial/main/binary-amd64/Packages.bz2 HTTP/1.1
Host: archive.dogfood.content.paddev.net
Cache-Control: max-age=0
User-Agent: Debian APT-HTTP/1.3 (1.2.9)

Answer for: http://archive.dogfood.content.paddev.net/ubuntu/dists/xenial/universe/binary-amd64/by-hash/SHA256/0b32e4073addade93d2cac70bf9b9f26894ce13125b35485d080daaa87923afa
HTTP/1.1 403 Forbidden file type or location: http://archive.dogfood.content.paddev.net/ubuntu/dists/xenial/universe/binary-amd64/by-hash/SHA256/0b32e4073addade93d2cac70bf9b9f26894ce13125b35485d080daaa87923afa
Content-Length: 186
Content-Type: text/html
Date: Sun Apr 3 02:09:52 2016
Server: Debian Apt-Cacher NG/0.8.0
Connection: Keep-Alive

Ign:6 http://archive.dogfood.content.paddev.net/ubuntu xenial/universe amd64 Packages
39% [Working]GET http://archive.dogfood.content.paddev.net/ubuntu/dists/xenial/main/i18n/Translation-en.bz2 HTTP/1.1
Host: archive.dogfood.content.paddev.net
Cache-Control: max-age=0
User-Agent: Debian APT-HTTP/1.3 (1.2.9)

Answer for: http://archive.dogfood.content.paddev.net/ubuntu/dists/xenial/universe/i18n/by-hash/SHA256/76f8217c195476bf27a1cb2ed9de27b1b4565788f79a17703fc02475af997e84
HTTP/1.1 403 Forbidden file type or location: http://archive.dogfood.content.paddev.net/ubuntu/dists/xenial/universe/i18n/by-hash/SHA256/76f8217c195476bf27a1cb2ed9de27b1b4565788f79a17703fc02475af997e84
Content-Length: 186
Content-Type: text/html
Date: Sun Apr 3 02:09:52 2016
Server: Debian Apt-Cacher NG/0.8.0
Connection: Keep-Alive

Ign:7 http://archive.dogfood.content.paddev.net/ubuntu xenial/universe Translation-en
39% [Working]GET http://archive.dogfood.content.paddev.net/ubuntu/dists/xenial/universe/binary-amd64/Packages.bz2 HTTP/1.1
Host: archive.dogfood.content.paddev.net
Cache-Control: max-age=0
User-Agent: Debian APT-HTTP/1.3 (1.2.9)

Answer for: http://archive.dogfood.content.paddev.net/ubuntu/dists/xenial/main/binary-amd64/Packages.bz2
HTTP/1.1 200 OK
Content-Length: 84737
Last-Modified: Sun, 03 Apr 2016 01:59:49 GMT
Content-Type: application/x-bzip2
Date: Sun Apr 3 02:09:52 2016
Server: Debian Apt-Cacher NG/0.8.0
X-Original-Source: http://archive.dogfood.content.paddev.net/ubuntu/dists/xenial/main/binary-amd64/Packages.bz2
Connection: Keep-Alive

Get:4 http://archive.dogfood.content.paddev.net/ubuntu xenial/main amd64 Packages [84.7 kB]
39% [4 Packages 0 B/84.7 kB 0%]Answer for: http://archive.dogfood.content.paddev.net/ubuntu/dists/xenial/main/i18n/Translation-en.bz2
HTTP/1.1 200 OK
Content-Length: 59831
Last-Modified: Sun, 03 Apr 2016 01:59:49 GMT
Content-Type: application/x-bzip2
Date: Sun Apr 3 02:09:52 2016
Server: Debian Apt-Cacher NG/0.8.0
X-Original-Source: http://archive.dogfood.content.paddev.net/ubuntu/dists/xenial/main/i18n/Translation-en.bz2
Connection: Keep-Alive

Get:5 http://archive.dogfood.content.paddev.net/ubuntu xenial/main Translation-en [59.8 kB]
87% [4 Packages store 0 B] [5 Translation-en 59.8 kB/59.8 kB 100%]GET http://archive.dogfood.content.paddev.net/ubuntu/dists/xenial/universe/i18n/Translation-en.bz2 HTTP/1.1
Host: archive.dogfood.content.paddev.net
Cache-Control: max-age=0
User-Agent: Debian APT-HTTP/1.3 (1.2.9)

Answer for: http://archive.dogfood.content.paddev.net/ubuntu/dists/xenial/universe/binary-amd64/Packages.bz2
HTTP/1.1 200 OK
Content-Length: 2058
Last-Modified: Sun, 03 Apr 2016 01:59:49 GMT
Content-Type: application/x-bzip2
Date: Sun Apr 3 02:09:52 2016
Server: Debian Apt-Cacher NG/0.8.0
X-Original-Source: http://archive.dogfood.content.paddev.net/ubuntu/dists/xenial/universe/binary-amd64/Packages.bz2
Connection: Keep-Alive

Answer for: http://archive.dogfood.content.paddev.net/ubuntu/dists/xenial/universe/i18n/Translation-en.bz2
HTTP/1.1 200 OK
Content-Length: 1341
Last-Modified: Sun, 03 Apr 2016 01:59:49 GMT
Content-Type: application/x-bzip2
Date: Sun Apr 3 02:09:52 2016
Server: Debian Apt-Cacher NG/0.8.0
X-Original-Source: http://archive.dogfood.content.paddev.net/ubuntu/dists/xenial/universe/i18n/Translation-en.bz2
Connection: Keep-Alive

Get:6 http://archive.dogfood.content.paddev.net/ubuntu xenial/universe amd64 Packages [2058 B]
Get:7 http://archive.dogfood.content.paddev.net/ubuntu xenial/universe Translation-en [1341 B]
Fetched 239 kB in 0s (518 kB/s)
Reading package lists... Done

-- Package-specific info:

-- System Information:
Debian Release: 8.4
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 3.16.0-4-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages apt-cacher-ng depends on:
ii adduser 3.113+nmu3
ii debconf [debconf-2.0] 1.5.56
ii dpkg 1.17.26
ii init-system-helpers 1.22
ii libbz2-1.0 1.0.6-7+b3
ii libc6 2.19-18+deb8u4
ii libgcc1 1:4.9.2-10
ii liblzma5 5.1.1alpha+20120614-2+b3
ii libssl1.0.0 1.0.1k-3+deb8u4
ii libstdc++6 4.9.2-10
ii libsystemd0 215-17+deb8u4
ii libwrap0 7.6.q-25
ii zlib1g 1:1.2.8.dfsg-2+b1

Versions of packages apt-cacher-ng recommends:
ii ed 1.10-2

Versions of packages apt-cacher-ng suggests:
ii avahi-daemon 0.6.31-5
ii curl 7.38.0-4+deb8u3
ii doc-base 0.10.6
ii libfuse2 2.9.3-15+deb8u2
ii wget 1.16-1

Thanks,

--
Colin Watson [cjwa...@debian.org]

Eduard Bloch

unread,

Apr 4, 2016, 1:10:05 PM4/4/16

to

Control: tags 819852 + pending

Hallo,
* Colin Watson [Sun, Apr 03 2016, 03:33:12AM]:

> This configuration file entry fixes the problem for me locally. I
> suspect therefore that it would be enough to just add this to the
> default pfilepat in source/acfg_defaults.cc (note: not vfilepat like the
> by-name index files, because the point of this scheme is that the
> content won't change without also changing the URL):
>
> PfilePatternEx: /dists/.*/by-hash/.*
>
> In case it helps, here's a debug dump from apt without this
> configuration file entry:
>
> # cat /etc/apt/sources.list
> deb [trusted=yes by-hash=force] http://archive.dogfood.content.paddev.net/ubuntu xenial main universe

Yeah, it should work that way, thanks!

And thanks for the repository link. I knew I forgot to put something on my
TODO list after the last DC...

Regards,
Eduard.

Colin Watson

unread,

Apr 12, 2016, 6:50:04 AM4/12/16

to

This mostly seems to be working, but there's a problem in some edge
cases. Let's say that a mirror is lagging a bit, so that
archive.ubuntu.com has a given by-hash file but de.archive.ubuntu.com
doesn't yet; /etc/apt-cacher-ng/backends_ubuntu has
de.archive.ubuntu.com followed by archive.ubuntu.com, and there are the
usual default Remap-* rules. In that case we see symptoms like this:

Answer for: http://archive.ubuntu.com/ubuntu/dists/xenial/main/source/by-hash/SHA256/8b81701065e3fa4957c5df3c6edc6a6f55301b7e20ac04d6a0aa19881b852320

HTTP/1.1 404 Not Found
Content-Length: 0

Date: Tue Apr 12 10:20:02 2016
Server: Debian Apt-Cacher NG/0.8.9
X-Original-Source: http://de.archive.ubuntu.com/ubuntu/dists/xenial/main/source/by-hash/SHA256/8b81701065e3fa4957c5df3c6edc6a6f55301b7e20ac04d6a0aa19881b852320
Connection: Keep-Alive

The unfortunate thing is that this 404 means that apt switches to its
fallback behaviour of fetching the old by-name version of the file
(http://archive.ubuntu.com/ubuntu/dists/xenial/main/source/Sources.xz in
this case), and since we're already in a situation of skewed mirrors
this tends to result in a hash sum mismatch as in the bad old days.

I think we want the default for by-hash files to be that it will try
other mirrors rather than responding with the 404 from the first one.
Do you agree, and if so how should that be achieved?

--
Colin Watson [cjwa...@debian.org]

Eduard Bloch

unread,

May 5, 2016, 4:40:03 PM5/5/16

to

Hallo,
* Colin Watson [Tue, Apr 12 2016, 11:40:54AM]:

> The unfortunate thing is that this 404 means that apt switches to its
> fallback behaviour of fetching the old by-name version of the file
> (http://archive.ubuntu.com/ubuntu/dists/xenial/main/source/Sources.xz in
> this case), and since we're already in a situation of skewed mirrors
> this tends to result in a hash sum mismatch as in the bad old days.
>
> I think we want the default for by-hash files to be that it will try
> other mirrors rather than responding with the 404 from the first one.
> Do you agree, and if so how should that be achieved?

In principle, I agree. There is already a similar mechanism which was
designed to detect a mirror which does not contain the wanted
architecture and therefore rotated the mirror selection to the next
possible one ("key file", see manual).

I will try to make it more generic and detect parts of the path as "key
factor".

Regards,
Eduard.

--
Die Kletterer haben den Mount Everest in die höchstgelegene Müllhalde
der Welt verwandelt.
-- Edmund Hillary