Disappearning shim-signed after failed dist-upgrade

[Gareth Evans]

A recent dist-upgrade on Buster (in a scripted cron job run at 01:00 daily) failed due to apt-listbugs complaining about the boot-breaking bug in shim-signed, and pinning v1.33 in the process.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=990082

The next (manual) dist-upgrade removed shim-signed v1.33

$ cat /var/log/apt/history.log
Start-Date: 2021-06-20 18:33:29
Commandline: apt-get -y dist-upgrade
Requested-By: xxxxxxxxx (1000)
Upgrade: shim-signed-common:amd64 (1.33+15+1533136590.3beb971-7, 1.36~1+deb10u1+15.4-5~deb10u1)
Remove: shim-signed:amd64 (1.33+15+1533136590.3beb971-7)
End-Date: 2021-06-20 18:33:30

unattended-upgrades (which I had forgotten was installed) upgraded some related packages earlier the same day, but not shim-signed itself:

$ cat /var/log/apt/history.log
Start-Date: 2021-06-20 06:26:31
Commandline: /usr/bin/unattended-upgrade
Upgrade: shim-helpers-amd64-signed:amd64 (1+15+1533136590.3beb971+7+deb10u1, 1+15.4+5~deb10u1), shim-unsigned:amd64 (15+1533136590.3beb971-7+deb10u1, 15.4-5~deb10u1)
End-Date: 2021-06-20 06:26:34

The only references to shim-signed in apt history logs were the initial Buster installation, and the recent removal:

/var/log/apt$ grep -n "shim-signed:" history.log*
history.log:209:Remove: shim-signed:amd64 (1.33+15+1533136590.3beb971-7)
history.log.6:33:Install: [...] shim-signed:amd64 (1.33+15+1533136590.3beb971-7) [...]


As I don't currently use secure boot, I ignored the bug warnings when I reinstalled it and dependencies (the buster-updates version per the email from debian-stable-announce yesterday
https://lists.debian.org/debian-stable-announce/2021/06/msg00001.html

...but still:

$ apt policy shim-signed
shim-signed:
Installed: 1.36~1+deb10u2+15.4-5~deb10u1
Candidate: 1.36~1+deb10u2+15.4-5~deb10u1

$ apt-listbugs list shim-signed
Retrieving bug reports... Done
Parsing Found/Fixed information... Done
grave bugs of shim-signed (→ ) <Outstanding>
b1 - #990082 - High chance of boot problems with buster's version of arm64 shim
grave bugs of shim-signed (→ ) <Resolved in some Version>
b2 - #987991 - shim-signed: Recent dbx update blacklists shimx64.efi (1.33+15+1533136590.3beb971-7) (Fixed: shim-signed/1.34)
Summary:
shim-signed(2 bugs)

$ apt-listbugs list shim-signed-common
critical bugs of shim-signed-common (→ ) <Outstanding>
b1 - #990158 - shim-signed-common: No UEFI boot with error "Could not create MokListXRT"
Summary:
shim-signed-common(1 bug)

Is this referring to the non buster-updates package?

Can anyone enlighten me as to:

Why might shim-signed v1.33 have been removed by dist-upgrade despite the previous upgrade attempt having been aborted by apt-listbugs?

What's the best way to reinstall an older package version and its old dependencies if affected by something like this, and it isn't to be found in /var/cache/apt/archives?

Thanks,
Gareth

[David Wright]

On Tue 22 Jun 2021 at 08:59:13 (+0100), Gareth Evans wrote:
> A recent dist-upgrade on Buster (in a scripted cron job run at 01:00 daily) failed due to apt-listbugs complaining about the boot-breaking bug in shim-signed, and pinning v1.33 in the process.
>
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=990082

AFAICT it looks as though you were just a victim of bad timing.
I happened to upgrade the point release at Start-Date:
2021-06-19 17:27:11, and my term.log shows:

Setting up fluidsynth (1.1.11-1+deb10u1) ...
Setting up shim-helpers-amd64-signed (1+15.4+5~deb10u1) ...
Installing for x86_64-efi platform.
Installation finished. No error reported.
Setting up python-libxml2 (2.9.4+dfsg1-7+deb10u2) ...
Setting up shim-signed:amd64 (1.36~1+deb10u1+15.4-5~deb10u1) ...
Installing for x86_64-efi platform.
Installation finished. No error reported.
Secure Boot not enabled on this system.
Processing triggers for mime-support (3.62) ...

By 01:00 next morning, the grave bug stopped your upgrade from
finishing. I guess that's a disadvantage of unattended upgrades:
you don't see the bug reports as they occur. (I download any
updates automatically, which serves as an announcement, but
always upgrade manually.)

> The next (manual) dist-upgrade removed shim-signed v1.33
>
> $ cat /var/log/apt/history.log
> Start-Date: 2021-06-20 18:33:29
> Commandline: apt-get -y dist-upgrade
> Requested-By: xxxxxxxxx (1000)
> Upgrade: shim-signed-common:amd64 (1.33+15+1533136590.3beb971-7, 1.36~1+deb10u1+15.4-5~deb10u1)
> Remove: shim-signed:amd64 (1.33+15+1533136590.3beb971-7)
> End-Date: 2021-06-20 18:33:30
>
> unattended-upgrades (which I had forgotten was installed) upgraded some related packages earlier the same day, but not shim-signed itself:
>
> $ cat /var/log/apt/history.log
> Start-Date: 2021-06-20 06:26:31
> Commandline: /usr/bin/unattended-upgrade
> Upgrade: shim-helpers-amd64-signed:amd64 (1+15+1533136590.3beb971+7+deb10u1, 1+15.4+5~deb10u1), shim-unsigned:amd64 (15+1533136590.3beb971-7+deb10u1, 15.4-5~deb10u1)
> End-Date: 2021-06-20 06:26:34
>
> The only references to shim-signed in apt history logs were the initial Buster installation, and the recent removal:
>
> /var/log/apt$ grep -n "shim-signed:" history.log*
> history.log:209:Remove: shim-signed:amd64 (1.33+15+1533136590.3beb971-7)
> history.log.6:33:Install: [...] shim-signed:amd64 (1.33+15+1533136590.3beb971-7) [...]
>
>
> As I don't currently use secure boot, I ignored the bug warnings when I reinstalled it and dependencies (the buster-updates version per the email from debian-stable-announce yesterday
> https://lists.debian.org/debian-stable-announce/2021/06/msg00001.html

AIUI that's the correct thing to do in our situation. (It's an upgrade
rather than a reinstall: my new shim-signed{,-common} debs arrived at
noon yesterday.)

> ...but still:
>
> $ apt policy shim-signed
> shim-signed:
> Installed: 1.36~1+deb10u2+15.4-5~deb10u1
> Candidate: 1.36~1+deb10u2+15.4-5~deb10u1
>
> $ apt-listbugs list shim-signed
> Retrieving bug reports... Done
> Parsing Found/Fixed information... Done
> grave bugs of shim-signed (→ ) <Outstanding>
> b1 - #990082 - High chance of boot problems with buster's version of arm64 shim
> grave bugs of shim-signed (→ ) <Resolved in some Version>
> b2 - #987991 - shim-signed: Recent dbx update blacklists shimx64.efi (1.33+15+1533136590.3beb971-7) (Fixed: shim-signed/1.34)
> Summary:
> shim-signed(2 bugs)
>
> $ apt-listbugs list shim-signed-common
> critical bugs of shim-signed-common (→ ) <Outstanding>
> b1 - #990158 - shim-signed-common: No UEFI boot with error "Could not create MokListXRT"
> Summary:
> shim-signed-common(1 bug)
>
> Is this referring to the non buster-updates package?

No. But I don't use secure boot, so I haven't been following along
with the shim's problem. (That is the same state of play shown by my system.)

> Can anyone enlighten me as to:
>
> Why might shim-signed v1.33 have been removed by dist-upgrade despite the previous upgrade attempt having been aborted by apt-listbugs?

$ aptitude why shim-signed
i grub-efi-amd64 Depends grub-efi-amd64-bin (= 2.02+dfsg1-20+deb10u4)
i A grub-efi-amd64-bin Recommends grub-efi-amd64-signed
i A grub-efi-amd64-signed Recommends shim-signed
$

so I have no dependency, only a Recommends. I don't know your option
settings, nor how they (or mine) react to a broken package, but
apt-get *dist-*upgrade certainly has the privilege of removing
packages in order to reach a satisfactory state.

> What's the best way to reinstall an older package version and its old dependencies if affected by something like this, and it isn't to be found in /var/cache/apt/archives?

In this case, you would probably follow Steve's instructions in the
bug report. And in general, you could use the pkg_version_number
options in, say, apt-get to downgrade. (As it says, with care.)

Nice to see a report where all the necessary info and logs are laid out.

Cheers,
David.

[Gareth Evans]

Thanks

> Cheers,
> David.

Is there any way to find out why a package was removed? I have googled but can't find any such command or log reference, nor indeed anything of relevance in my apt history or cron script logs.

Also, why does

apt-listbugs list shim-signed

still show the bug supposedly fixed by the buster-updates package... reversion? (even though with a higher version number)?

Thanks,
Gareth

[David Wright]

> Is there any way to find out why a package was removed? I have googled but can't find any such command or log reference, nor indeed anything of relevance in my apt history or cron script logs.

Do you mean at this step, quoted above:

> > > The next (manual) dist-upgrade removed shim-signed v1.33
> > >
> > > $ cat /var/log/apt/history.log
> > > Start-Date: 2021-06-20 18:33:29
> > > Commandline: apt-get -y dist-upgrade
> > > Requested-By: xxxxxxxxx (1000)
> > > Upgrade: shim-signed-common:amd64 (1.33+15+1533136590.3beb971-7, 1.36~1+deb10u1+15.4-5~deb10u1)
> > > Remove: shim-signed:amd64 (1.33+15+1533136590.3beb971-7)
> > > End-Date: 2021-06-20 18:33:30

>From the manpage, dist-upgrade "handles changing dependencies with new
versions of packages [ … ] may therefore remove some packages".

I presume whatever was causing shim-signed to be installed on your
system has itself become uninstalled. Did you try running
$ aptitude why shim-signed
on your own system, to see which that is?

>From man apt-listbugs, "When [apt-listbugs] is invoked by APT, any
non-zero exit status will cause the installation/upgrade to be aborted".
So the first run, in the middle of the night, aborted because of the
error. (I don't know if that was an upgrade or dist-upgrade, because
it's "hidden" behind /usr/bin/unattended-upgrade.)

But your evening run of apt-get -y dist-upgrade was unconstrained,
and so shim-signed could be removed because it was no longer being
held onto as a Depends or Recommends.

> Also, why does
>
> apt-listbugs list shim-signed
>
> still show the bug supposedly fixed by the buster-updates package... reversion? (even though with a higher version number)?

Although apt-listbugs can focus its check onto specific architectures
and versions of packages, that might rely on a perfect formatting of
the bug report. I've never followed up on this, because I perform
upgrades manually on my systems, so the final arbiter on apt-listbugs
complaints is my eyeball.

With unattended and automatic upgrades, there's still a role for the
sysadmin to monitor the logs. The good news is that breaking these
particular packages (that install your booting scheme) doesn't in
itself break booting (as Nicolas pointed out recently), but only your
ability to modify/upgrade it.

Cheers,
David.

[Andrei POPESCU]

On Lu, 28 iun 21, 09:46:17, David Wright wrote:
>
> But your evening run of apt-get -y dist-upgrade was unconstrained,
> and so shim-signed could be removed because it was no longer being
> held onto as a Depends or Recommends.

Except that `apt-get dist-upgrade` doesn't do that (`autoremove` does),
it only removes packages when it determines that it's needed to complete
the dist-upgrade, so a Conflicts or Breaks with an upgraded package.


Kind regards,
Andrei
--
http://wiki.debian.org/FAQsFromDebianUser

[David Wright]

On Tue 29 Jun 2021 at 08:29:22 (+0300), Andrei POPESCU wrote:
> On Lu, 28 iun 21, 09:46:17, David Wright wrote:
> >
> > But your evening run of apt-get -y dist-upgrade was unconstrained,
> > and so shim-signed could be removed because it was no longer being
> > held onto as a Depends or Recommends.
>
> Except that `apt-get dist-upgrade` doesn't do that (`autoremove` does),
> it only removes packages when it determines that it's needed to complete
> the dist-upgrade, so a Conflicts or Breaks with an upgraded package.

So we can presume, perhaps, that it's a case of ranking:
shim-signed being installed as a Recommends is ranked lower
than upgrading shim-signed-common, and so it gets removed.

All my systems were upgraded successfully because I ignored
any arm64 bug reports, so all my lists and caches have been
refreshed since, and I don't have access to the control data
for shim-signed/1.33+15+1533136590.3beb971-7.

Perhaps others can make better informed guesses.

Cheers,
David.

[Gareth Evans]

I suspect "aptitude why shim-signed" may not remain relevant, but if so:

$ aptitude why shim-signed
i grub-efi-amd64-signed Recommends shim-signed

apt history shows grub-efi-amd64-signed was last upgraded along with other grub* packages, apparently without issue, on 2021-03-03.

Even if I had made use of upgrade rather than dist-upgrade, presumably a dist-upgrade would have been indicated here (by the existence of packages kept back) and the same position would have resulted - ie. having to wait for a potentially essential package to be made available in upgraded form so it could be reinstalled so that it could be further upgraded in future... no?

$ aptitude why shim-signed-common
i shim-signed Depends shim-signed-common (>= 1.36~1+deb10u2+15.4-5~deb10u1)

It seems the upgrade of shim-signed-common may have removed shim-signed.

Shouldn't essential packages involved in dependencies only ever be available for upgrade together, unless perhaps the dependency >=version remains satisfied?

Before succeeding in installing the newer version of shim-signed, I did try installing the version originally installed with Buster, but apt said it wasn't available. In the end I didn't need to attempt installing from eg. DVD, but couldn't this be impossible too due to other conflicts?

In short, should a boot-related package ever be removed and not replaced in the same upgrade operation?

Is it to be expected to have to keep an eye on this sort of thing?

Thanks all.
Gareth

[Gareth Evans]

That and related comments might not make sense.

I didn't explain there was a period when I couldn't (re)install shim-signed as apt install reported not found or something like that. Thinking about it further after sending the above, I'm not entirely sure at what point I unpinned shim-signed, so will check logs and snapshots and rack brains and get back to you.

Thanks,
Gareth

[David Wright]

On Tue 29 Jun 2021 at 22:41:10 (+0100), Gareth Evans wrote:
> On Tue 29 Jun 2021, at 17:11, David Wright <deb...@lionunicorn.co.uk> wrote:
> > On Tue 29 Jun 2021 at 08:29:22 (+0300), Andrei POPESCU wrote:
> > > On Lu, 28 iun 21, 09:46:17, David Wright wrote:
> > > >
> > > > But your evening run of apt-get -y dist-upgrade was unconstrained,
> > > > and so shim-signed could be removed because it was no longer being
> > > > held onto as a Depends or Recommends.
> > >
> > > Except that `apt-get dist-upgrade` doesn't do that (`autoremove` does),
> > > it only removes packages when it determines that it's needed to complete
> > > the dist-upgrade, so a Conflicts or Breaks with an upgraded package.
> >
> > So we can presume, perhaps, that it's a case of ranking:
> > shim-signed being installed as a Recommends is ranked lower
> > than upgrading shim-signed-common, and so it gets removed.
> >
> > All my systems were upgraded successfully because I ignored
> > any arm64 bug reports, so all my lists and caches have been
> > refreshed since, and I don't have access to the control data
> > for shim-signed/1.33+15+1533136590.3beb971-7.
> >
> > Perhaps others can make better informed guesses.
>

> I suspect "aptitude why shim-signed" may not remain relevant, but if so:
>
> $ aptitude why shim-signed
> i grub-efi-amd64-signed Recommends shim-signed
>
> apt history shows grub-efi-amd64-signed was last upgraded along with other grub* packages, apparently without issue, on 2021-03-03.

The only difference from my system is that my grub-efi-amd64-signed
shows as automatic as well, and only installed through a Recommends
(by grub-efi-amd64-bin). I don't think it makes any difference here.

> Even if I had made use of upgrade rather than dist-upgrade, presumably a dist-upgrade would have been indicated here (by the existence of packages kept back) and the same position would have resulted - ie. having to wait for a potentially essential package to be made available in upgraded form so it could be reinstalled so that it could be further upgraded in future... no?

I think, rather, that a manual upgrade could have upgraded shim-signed,
just as long as you say OK to any complaint coming from apt-listbugs.

> $ aptitude why shim-signed-common
> i shim-signed Depends shim-signed-common (>= 1.36~1+deb10u2+15.4-5~deb10u1)
>
> It seems the upgrade of shim-signed-common may have removed shim-signed.
>
> Shouldn't essential packages involved in dependencies only ever be available for upgrade together, unless perhaps the dependency >=version remains satisfied?

That normally happens, and did in this case. AFAICT you got hit by the
caution understandably exercised by unattended-upgrades: it's easier
to come along after the event and push an upgrade forward, rather than
roll back one that caused an error or failure.

> Before succeeding in installing the newer version of shim-signed, I did try installing the version originally installed with Buster, but apt said it wasn't available. In the end I didn't need to attempt installing from eg. DVD, but couldn't this be impossible too due to other conflicts?

I think that installing an old version would be unwise, as well as
potentially difficult. Better to read up on the bug for more
information. AIUI in this case, only arm users were affected.

> In short, should a boot-related package ever be removed and not replaced in the same upgrade operation?

When it doesn't actually affect booting itself, then it's just another
package. That appears to be the case here. AFAICT linux-image-* debs
are about the only ones that install directly into /boot.

> Is it to be expected to have to keep an eye on this sort of thing?

Well, the caution practised by unattended-upgrades should make alerts
rare, but that's all the more reason to pay close attention when it happens.

Cheers,
David.

[Andrei POPESCU]

On Ma, 29 iun 21, 22:41:10, Gareth Evans wrote:
>
> Even if I had made use of upgrade rather than dist-upgrade, presumably
> a dist-upgrade would have been indicated here (by the existence of
> packages kept back) and the same position would have resulted - ie.
> having to wait for a potentially essential package to be made
> available in upgraded form so it could be reinstalled so that it could
> be further upgraded in future... no?

Maybe. This is exactly the reason why `dist-upgrade` should be ran
manually. Always.

> Shouldn't essential packages involved in dependencies only ever be
> available for upgrade together, unless perhaps the dependency
> >=version remains satisfied?

Debian has mechanisms to keep packages installed, unless the system
administrator forces it, so obviously 'shim-signed' is less important.

> Before succeeding in installing the newer version of shim-signed, I
> did try installing the version originally installed with Buster, but
> apt said it wasn't available. In the end I didn't need to attempt
> installing from eg. DVD, but couldn't this be impossible too due to
> other conflicts?

Difficult to tell without looking at the exact situation.

> In short, should a boot-related package ever be removed and not
> replaced in the same upgrade operation?

There are many installations that don't need 'shim' at all, e.g.
chroots, systems still using MBR, etc. Most of these packages don't even
need to be present on the system, as their payload isn't removed when
removing the package.

> Is it to be expected to have to keep an eye on this sort of thing?

In general you shouldn't run `dist-upgrade` unattended, ever, otherwise
you will eventually end up with removal of packages that you actually
need.

This is even more important on everything that is not pure Debian
'stable'. For keeping 'stable' updated `dist-upgrade` is overdoing it
anyway, `apt upgrade` (or `apt-get upgrade --with-new-pkgs` if you
really must have `apt-get`) is sufficient and safer.