Bug#1017996: dpkg: please provide `--force-really-unsafe-io` (or similar) option

[Ansgar]

Package: dpkg
Version: 1.21.9
Severity: wishlist
Tags: upstream d-i

Hi,

please reconsider to provide a `--force-really-unsafe-io` (or similar)
option that skips the calls to `fsync()` & friends in dpkg.

I tried installing a larger package set (in stable), including
texlive-full, KDE, GCC and other packages. It took:

Default dpkg: ~4h10m = 250m
--force-unsafe-io: ~2h20m = 140m
eatmydata: ~22m

So skipping all fsync() calls provides a speedup of 11 compared to
dpkg's defaults and still 6 compared to --force-unsafe-io. This is a
very noticable change.

The dpkg FAQ[1] asks users to use a different filesystem, non-default
mount options, `--force-unsafe-io` or `eatmydata`.

I don't think asking users to use a different filesystem is a good
option (and the installer doesn't ask in many cases); same for
non-default mount options. Doing so also has other downsides.

Options like `--force-unsafe-io` are okay (and d-i can and does use
it), but are suboptimal in many situations where one really does not
need any calls to `fsync`. `eatmydata` is a bit of a hack and harder
to use (make sure apt-get calls a wrapper dpkg script or everything
calls apt-get/aptitude/* via some wrapper) or not possible (d-i
doesn't use eatmydata). I think dpkg supporting to skip fsync on its
own is the better solution (compared to implementing eatmydata support
in d-i and other places).

There are various cases where this is useful:

- Initial installation (d-i or initial setup with ansible/puppet/scripts/*).
- Containers.
- CI systems.
- Anything else using throw-away systems (chroots, containers, VMs, ...).

Even dpkg itself uses eatmydata to avoid `fsync()`s (in .gitlab-ci.yml).

I know this was previously proposed and not implemented, but I think
the numer of cases where this is useful has increased over time and
will probably increase more in the future.

Ansgar

[1]: https://wiki.debian.org/Teams/Dpkg/FAQ#Q:_Why_is_dpkg_so_slow_when_using_new_filesystems_such_as_btrfs_or_ext4.3F

[Guillem Jover]

Control: forcemerge 985888 -1

On Tue, 2022-08-23 at 22:40:37 +0200, Ansgar wrote:
> Package: dpkg
> Version: 1.21.9
> Severity: wishlist
> Tags: upstream d-i

> please reconsider to provide a `--force-really-unsafe-io` (or similar)
> option that skips the calls to `fsync()` & friends in dpkg.

I think I've mentioned this elsewhere, but the main blocker has been
lack of database taint tracking support. I do have a draft branch for
a --force-reckless-io.

> I tried installing a larger package set (in stable), including
> texlive-full, KDE, GCC and other packages. It took:
>
> Default dpkg: ~4h10m = 250m
> --force-unsafe-io: ~2h20m = 140m
> eatmydata: ~22m
>
> So skipping all fsync() calls provides a speedup of 11 compared to
> dpkg's defaults and still 6 compared to --force-unsafe-io. This is a
> very noticable change.

All fsync()s from dpkg, which are not necessarily all fsync()s
performed indirectly by maintscripts/triggers.

Guillem