Bug#1004583: overwriting {,src}pkgcache.bin on near-every APT call strains flash

[Ivan Shmakov]

Package: apt
Version: 2.3.14
Severity: minor
Control: found -1 1.8.2.3

[Please do not Cc: me, for I’m “on the list,” so to say, and
I try to reserve my inbox for private communication only.
I’d have set up Mail-Followup-To:, but there doesn’t seem to
be a way to make it point to the bug being filed.]

As of this writing, pkgcache.bin and srcpkgcache.bin files for
an APT instance configured for testing / Sid are 61 MiB combined.

The way those caches are updated is by writing a new version
into a temporary file and then calling rename(2) to atomically
replace the old version with the new. As such, every time the
caches are updated, some 61 MiB gets written to the filesystem

While it could be argued that such an amount written makes
no measurable impact on HDDs, and rather marginally affects
the lifetime of contemporary SSDs, it’s not uncommon to start
Debian from a micro-SD card on single-board computers, where
write cycles may be considerably more limited.

Similarly, when filesystems-level snapshots are supported
(Btrfs, Nilfs, etc.), the issue described prevents space (and
bandwidth, when snapshots are transferred with btrfs-send(8) /
btrfs-receive(8)) savings by getting in the way of the
filesystem’s copy-on-write behavior.

(FWIW, apt is by no means the only package with such an issue:
dpkg’s status file is updated the same way, and so is the
templates.dat file debconf uses by default; yet dpkg/status is
smaller, and debconf can be configured to use Driver: PackageDir,
also lowering the impact. Outside of the Debian infrastructure,
tor caches are also ill-suited for write-limited media.)

Consider, for example, the following chroot environment with a
freshly installed Debian Bookworm (--arch=amd64 --variant=minbase
and a couple of packages on top of that):

chroot# apt-get update
Hit:1 http://cdn-fastly.deb.debian.org/debian bookworm InRelease
Hit:2 http://security.debian.org/debian-security bookworm-security/updates InRelease
Reading package lists...
chroot#

Now, let’s try to install some lightweight packages, checking
the ‘lifetime writes’ meter before and after the apt-get call:

base# mount -o remount,ro -- /dev/vgfoo/lvchild-z61f590 \
&& dumpe2fs -- /dev/vgfoo/lvchild-z61f590 | grep -E -- ^Lifetime
Lifetime writes: 5418 MB
base#

chroot# apt-get install -- jpeginfo
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
libjpeg62-turbo
The following NEW packages will be installed:
jpeginfo libjpeg62-turbo
0 upgraded, 2 newly installed, 0 to remove and 5 not upgraded.
Need to get 0 B/177 kB of archives.
After this operation, 725 kB of additional disk space will be used.
Do you want to continue? [Y/n]
debconf: delaying package configuration, since apt-utils is not installed
E: Can not write log (Is /dev/pts mounted?) - posix_openpt (19: No such device)
Selecting previously unselected package libjpeg62-turbo:amd64.
(Reading database ... 6737 files and directories currently installed.)
Preparing to unpack .../libjpeg62-turbo_1%3a2.1.2-1_amd64.deb ...
Unpacking libjpeg62-turbo:amd64 (1:2.1.2-1) ...
Selecting previously unselected package jpeginfo.
Preparing to unpack .../jpeginfo_1.6.1-1_amd64.deb ...
Unpacking jpeginfo (1.6.1-1) ...
Setting up libjpeg62-turbo:amd64 (1:2.1.2-1) ...
Setting up jpeginfo (1.6.1-1) ...
Processing triggers for libc-bin (2.33-3) ...
chroot#

base# mount -o remount,ro -- /dev/vgfoo/lvchild-z61f590 \
&& dumpe2fs -- /dev/vgfoo/lvchild-z61f590 | grep -E -- ^Lifetime
Lifetime writes: 5740 MB
base#

By the looks of it, installing 725 KiB worth of Debian packages
resulted in some 322 MiB getting written to the filesystem,
for which I’m inclined to think that APT cache update behavior
is in no small part responsible. (I suppose running apt-get
under strace would allow for a more accurate estimate.)

For instance, largest recently changed files on the FS right
after the # apt-get install call were (note that the .deb files
to be installed were obtained earlier, and also that
/var/cache/apt/archives was on a separate FS anyway):

chroot# find / -xdev -cmin -4 -printf %k\\t%p\\n | sort -srn
31568 /var/cache/apt/pkgcache.bin
31548 /var/cache/apt/srcpkgcache.bin
580 /usr/lib/x86_64-linux-gnu/libjpeg.so.62.3.0
88 /var/lib/dpkg/status-old
88 /var/lib/dpkg/status
52 /var/log/dpkg.log
40 /usr/share/doc/libjpeg62-turbo/copyright

A proper solution would be to move to some file format that
allows in-place updates, such as Berkeley DB or SQLite.

As a work-around, it’s possible to move the cache to tmpfs;
consider, e. g.:

$ cat < /etc/apt/apt.conf.d/99-run-pkgcache
Dir::Cache::pkgcache "/run/apt/pkgcache.bin";
Dir::Cache::srcpkgcache "/run/apt/srcpkgcache.bin";
$ grep -F -- /run/apt < /etc/fstab
run.apt /run/apt tmpfs rw,nodev,noexec,nosuid,mode=0755,size=256M,X-mount.mkdir 0 0
$

When RAM is also limited (which is to say, the filesystem size=
limit above isn’t negligible), it may make sense to remove the
files when not in use, such as by enabling the atime filesystem
option and using the following (untested) Cron job:

$ grep -F -- /run/apt < /etc/fstab
run.apt /run/apt tmpfs rw,nodev,noexec,nosuid,mode=0755,size=256M,X-mount.mkdir,atime 0 0
$ cat < /etc/cron.hourly/cleanup-run-apt
#!/bin/sh
set -e
set -C -u

test -d /run/apt \
|| exit

## .
exec find /run/apt/ -maxdepth 1 \
-type f -amin +113 -cmin +113 \
-size +63k -name \*pkgcache.bin\* \
-execdir rm -- {} +
$

--
FSF associate member #7257 http://am-1.org/~ivan/

[Julian Andres Klode]

Control: reopen -1

On Sun, Jan 30, 2022 at 07:10:34PM +0000, Ivan Shmakov wrote:
> A proper solution would be to move to some file format that
> allows in-place updates, such as Berkeley DB or SQLite.

No it would not be. The best we can do, and maybe should do
is move pkgcache.bin to /run.

In-place updates means _a lot_ of things can go wrong. Like
you forget to delete packages that are no longer in the
repository.

There's also no way to change this in any case, the database
format is an integral part of the public API, it would take
decade(s) to switch to anything else.

Unfortunately I only thought about moving pkgcache.bin to
/run directly after sending the close email, but I think
that might be a reasonable compromise to discuss, and if
we support $XDG_RUNTIME_DIR/apt/pkgcache.bin as well, then
that would even get us caching for non-root users.

How to move that file out of /var/cache/apt is going to be
a tricky question, as that location is to some extend part
of the API as well - people might set Dir::Cache to /tmp/apt
and expect pkgcache.bin to end up in /tmp/apt.

With regards to srcpkgcache.bin, which represents the available
packages, but no installed packages (pkgcache.bin is built from
it by adding the dpkg status file in-place, in-memory, and then
writing it back out), I think having that around after a reboot
is what we want such that you don't need to rebuild the entire
cache each boot.

--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer i speak de, en

[Ivan Shmakov]

>>>>> On 2022-02-01 12:27:50 +0100, Julian Andres Klode wrote:

> Control: reopen -1

I know there’re different approaches to this, but I believe that
a report shouldn’t be closed so long as it describes a behavior
that’s a. reproducible and b. known to affect at least some users.

At the very least, keeping the report open will help prevent
others from reporting the same issue over and over again.

For the cases where the developer finds the existing behavior
infeasible to alter, Debian BTS provides the ‘wontfix’ tag.

> On Sun, Jan 30, 2022 at 07:10:34PM +0000, Ivan Shmakov wrote:

>> A proper solution would be to move to some file format that
>> allows in-place updates, such as Berkeley DB or SQLite.

> No it would not be. The best we can do, and maybe should do
> is move pkgcache.bin to /run.

> In-place updates means _a lot_ of things can go wrong. Like you
> forget to delete packages that are no longer in the repository.

While I’m by no means familiar with the code base, this problem
does not sound insurmountable. Moreover, this is something
I personally would be interested in working on.

> There’s also no way to change this in any case, the database
> format is an integral part of the public API, it would take
> decade(s) to switch to anything else.

I wasn’t aware of this (though it does make sense, given the
availability of alternative interfaces to APT functions.)

Where this format, and its intended use, are documented?

> Unfortunately I only thought about moving pkgcache.bin to
> /run directly after sending the close email, but I think
> that might be a reasonable compromise to discuss, and if
> we support $XDG_RUNTIME_DIR/apt/pkgcache.bin as well, then
> that would even get us caching for non-root users.

I can’t say I readily see the utility of that, at least by itself.

Sure, I’ve used apt-get as a non-root user to download lists
and packages to copy to systems where the network connectivity
was poor (or non-existent), but I had to point Dir::Cache and
Dir::State::Lists to locations writable by that same user
to achieve my goals, not just pkgcache.bin.

> How to move that file out of /var/cache/apt is going to be
> a tricky question, as that location is to some extend part
> of the API as well - people might set Dir::Cache to /tmp/apt
> and expect pkgcache.bin to end up in /tmp/apt.

The APT configuration language doesn’t seem to provide
a straightforward solution for invalidating a default
for one option when another one is set by the user, indeed.

Given that the current behavior is, AFAIK, problematic only in
two cases (a. /var/cache/apt resides on an SD card; and
b. block-level backups are used for the filesystem containing
said directory; which is, incidentally, how I initially became
aware of this behavior), both of which seem relatively rare,
I think the best approach would be to either let the user
decide (such as via debconf), or at least inform them of the
potential problem.

Furthermore, as installing Debian on an SD card (or similar
flash-based device not rated for the amount of writing the
updating of pkgcache.bin might take) doesn’t seem common outside
of ARM-based SBCs, it may make sense to restrict the fix,
whichever that might be, to ARM (arm64, armhf, armel)
architectures.

More directly, the fix can be applied if it can be detected with
a degree of reliability at .postinst time that pkgcache.bin
might end up on an SD card. (The necessary info is mostly
there, see lsblk(8) output for instance, though I know of no
straightforward way to map a file to the stack of devices
it resides on. Presumably we can consider placing pkgcache.bin
under /run if we find something like mmcblk* while traversing
the stack upwards from the device that holds the filesystem.)

> With regards to srcpkgcache.bin, which represents the available
> packages, but no installed packages (pkgcache.bin is built from
> it by adding the dpkg status file in-place, in-memory, and then
> writing it back out), I think having that around after a reboot
> is what we want such that you don’t need to rebuild the entire
> cache each boot.

FWIW, I mainly use Debian on Socket AM2-class hardware (which
is to say, about 12 years old), and IME it takes something like
20 seconds to rebuild the cache. As such, I wouldn’t myself be
much concerned with rebuilding the cache from scratch.