Buster wget: The certificate of 'lists.debian.org' is not trusted

[Thomas Schmitt]

Hi,

new certification problem, this time on buster.

While looking why wget on Debian 8 does not work with lists.debian.org
i learn that it does not work on Debian 10 either:

$ wget -d https://lists.debian.org/debian-user/
DEBUG output created by Wget 1.20.1 on linux-gnu.

Reading HSTS entries from /home/thomas/.wget-hsts
URI encoding = ‘UTF-8’
Converted file name 'index.html' (UTF-8) -> 'index.html' (UTF-8)
--2021-10-01 18:51:06-- https://lists.debian.org/debian-user/
Certificates loaded: 126
Resolving lists.debian.org (lists.debian.org)... 2001:41b8:202:deb:216:36ff:fe40
Caching lists.debian.org => 2001:41b8:202:deb:216:36ff:fe40:4002 82.195.75.100
Connecting to lists.debian.org (lists.debian.org)|2001:41b8:202:deb:216:36ff:fec
Created socket 3.
Releasing 0x00005560a217b5d0 (new refcount 1).
ERROR: The certificate of ‘lists.debian.org’ is not trusted.
ERROR: The certificate of ‘lists.debian.org’ has expired.

Firefox works without problems.
The number of 126 certificates would match /etc/ssl/certs.

It's really on buster:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 10 (buster)
Release: 10
Codename: buster

The kernel is a modified 5.10.rc3 (i should upgrade to Debian 11 and
apply my patches to its kernel source).

Is this reproducible on other busters ?


Have a nice day :)

Thomas

[to...@tuxteam.de]

On Fri, Oct 01, 2021 at 07:09:45PM +0200, Thomas Schmitt wrote:
> Hi,
>
> new certification problem, this time on buster.
>
> While looking why wget on Debian 8 does not work with lists.debian.org
> i learn that it does not work on Debian 10 either:
>
> $ wget -d https://lists.debian.org/debian-user/
> DEBUG output created by Wget 1.20.1 on linux-gnu.

Hm. Works for me. Buster, too...

[...]

> Firefox works without problems.
> The number of 126 certificates would match /etc/ssl/certs.

...but mine loads... 127 certs :-/

Just for kicks: this is the top of my `ls -lt /etc/ssl/certs. Perhaps you
notice something funny, dunno.

total 568
-rw-r--r-- 1 root root 199974 Feb 7 2021 ca-certificates.crt
lrwxrwxrwx 1 root root 15 Jun 30 2020 0a775a30.0 -> GTS_Root_R3.pem
lrwxrwxrwx 1 root root 32 Jun 30 2020 0f5dc4f3.0 -> UCA_Extended_Validation_Root.pem
lrwxrwxrwx 1 root root 15 Jun 30 2020 1001acf7.0 -> GTS_Root_R1.pem
lrwxrwxrwx 1 root root 27 Jun 30 2020 14bc7599.0 -> emSign_ECC_Root_CA_-_G3.pem
...

But oh, those are just link farms to /usr/share/ca-certificates/mozilla:
There's newer stuff there:

total 548
-rw-r--r-- 1 root root 2772 Jan 28 2021 ACCVRAIZ1.crt
-rw-r--r-- 1 root root 1972 Jan 28 2021 AC_RAIZ_FNMT-RCM.crt
-rw-r--r-- 1 root root 2049 Jan 28 2021 Actalis_Authentication_Root_CA.crt
-rw-r--r-- 1 root root 1204 Jan 28 2021 AffirmTrust_Commercial.crt
...

The working Firefox suggests that perhaps some of the links got
lost -- or perhaps that you have to rebuild `ca-certificates.crt'

Cheers
- t

[Thomas Schmitt]

Hi,

i wrote:
> > The number of 126 certificates would match /etc/ssl/certs.

to...@tuxteam.de wrote:
> ...but mine loads... 127 certs :-/

> [...]

> -rw-r--r-- 1 root root 2772 Jan 28 2021 ACCVRAIZ1.crt
> -rw-r--r-- 1 root root 1972 Jan 28 2021 AC_RAIZ_FNMT-RCM.crt

Hm. The system is one year old. Time for
apt-get update
apt-get dist-upgrade

Afterwards i have success with
wget https://lists.debian.org/debian-user/
It reports
Certificates loaded: 137

Indeed
ls -t /etc/ssl/certs
lists a few new certificate files:

VeriSign_Universal_Root_Certification_Authority.pem
VeriSign_Class_3_Public_Primary_Certification_Authority_-_G4.pem
VeriSign_Class_3_Public_Primary_Certification_Authority_-_G5.pem
thawte_Primary_Root_CA_-_G3.pem
thawte_Primary_Root_CA_-_G2.pem
thawte_Primary_Root_CA.pem
GeoTrust_Primary_Certification_Authority_-_G3.pem
GeoTrust_Universal_CA.pem
GeoTrust_Primary_Certification_Authority_-_G2.pem
GeoTrust_Global_CA.pem
GeoTrust_Primary_Certification_Authority.pem

I will try whether they and their link targets repair wget on Debian 8 too.

(My current theory is that iceweasel and firefox already had the
decisive certificates but they were missing in /etc/ssl/certs or
in where wget looks them up.)

[Thomas Schmitt]

Hi,

i forgot to add a [SOLVED] tag to the subject of my previous mail.

So again, for the archives:

apt-get update
apt-get dist-upgrade

solved the wget certificate problem with the buster installation from
juli 2020.

[didier gaumet]

Hello,

- As it has already been said, Firefox does not use ca-certificates
- it seems that sometimes upgrades to a new version of ca-certificates
do not automatically enable all new certificates. I would suggest you
to do a dpkg-reconfigure of ca-certificates to verify what is proposed
and what you want to enable, perhaps this will solve your wget problem

[to...@tuxteam.de]

On Fri, Oct 01, 2021 at 08:25:44PM +0200, Thomas Schmitt wrote:
> Hi,
>
> i wrote:
> > > The number of 126 certificates would match /etc/ssl/certs.
>
> to...@tuxteam.de wrote:
> > ...but mine loads... 127 certs :-/
> > [...]
> > -rw-r--r-- 1 root root 2772 Jan 28 2021 ACCVRAIZ1.crt
> > -rw-r--r-- 1 root root 1972 Jan 28 2021 AC_RAIZ_FNMT-RCM.crt
>
> Hm. The system is one year old. Time for
> apt-get update
> apt-get dist-upgrade
>
> Afterwards i have success with
> wget https://lists.debian.org/debian-user/
> It reports
> Certificates loaded: 137

\o/

> (My current theory is that iceweasel and firefox already had the
> decisive certificates but they were missing in /etc/ssl/certs or
> in where wget looks them up.)

This makes sense.

Cheers
- t

[David Wright]

On Fri 01 Oct 2021 at 20:31:54 (+0200), Thomas Schmitt wrote:
>
> So again, for the archives:
>
> apt-get update
> apt-get dist-upgrade
>
> solved the wget certificate problem with the buster installation from
> juli 2020.

I have a buster system that was up-to-date from the last point-release
and kernel (2021-09-10 18:22:47).

The only certificate expiration problem I have observed (and still
observe, having taken no action) is with apt-listbugs:

The following packages will be upgraded:
libjavascriptcoregtk-4.0-18 libwebkit2gtk-4.0-37 tzdata
3 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/20.0 MB of archives.
After this operation, 76.8 kB of additional disk space will be used.
Do you want to continue? [Y/n]
Retrieving bug reports... 0% Fail
Error retrieving bug reports from the server with the following error message:
E: SSL_connect returned=1 errno=0 state=error: certificate verify failed (certificate has expired)
It could be because your network is down, or because of broken proxy servers, or the BTS server itself is down. Check network configuration and try again
Retry downloading bug information? [Y/n]
One package at a time? [Y/n]
One bug report at a time? [Y/n]
Fail
Error retrieving bug reports from the server with the following error message:
E: SSL_connect returned=1 errno=0 state=error: certificate verify failed (certificate has expired)
It could be because your network is down, or because of broken proxy servers, or the BTS server itself is down. Check network configuration and try again
Retry downloading bug information? [Y/n] n
Continue the installation anyway? [y/N] y
Retrieving bug reports... Done ← AIUI that's not truthful
Parsing Found/Fixed information... Done
Reading changelogs... Done
Preconfiguring packages ...

However, my next thought was to temporarily move my
/etc/apt/apt.conf file, which contains just the one proxy
line pointing at apt-cacher-ng, with the result that
apt-listbugs was able to run without any problem.

I have no problem visiting bugs.debian.org with firefox
and lynx, so there's no urgency. I'm just happy that
two days ago I did a full install of bullseye on a
machine through my proxy and with listbugs turned on.

But it is odd that telling APT not to retrieve any bugs
should still say "Retrieving bug reports... Done".

Cheers,
David.

[to...@tuxteam.de]

On Fri, Oct 01, 2021 at 03:18:22PM -0500, David Wright wrote:

[...]

> I have a buster system that was up-to-date from the last point-release
> and kernel (2021-09-10 18:22:47).
>
> The only certificate expiration problem I have observed (and still
> observe, having taken no action) is with apt-listbugs:

[...]

> However, my next thought was to temporarily move my
> /etc/apt/apt.conf file, which contains just the one proxy
> line pointing at apt-cacher-ng, with the result that
> apt-listbugs was able to run without any problem.

I understand correctly: using apt-cacher-ng somehow breaks
certificate validation for apt-listbugs?

> I have no problem visiting bugs.debian.org with firefox
> and lynx, so there's no urgency. I'm just happy that
> two days ago I did a full install of bullseye on a
> machine through my proxy and with listbugs turned on.

That would be no surprise (provided I understood correctly
above, that is): firefox and lynx don't proxy through
apt-cacher-ng.

> But it is odd that telling APT not to retrieve any bugs
> should still say "Retrieving bug reports... Done".

Proper messages are hard :)

Cheers
- t

[David Wright]

On Sat 02 Oct 2021 at 08:15:59 (+0200), to...@tuxteam.de wrote:
> On Fri, Oct 01, 2021 at 03:18:22PM -0500, David Wright wrote:
>
> [...]
>
> > I have a buster system that was up-to-date from the last point-release
> > and kernel (2021-09-10 18:22:47).
> >
> > The only certificate expiration problem I have observed (and still
> > observe, having taken no action) is with apt-listbugs:
>
> [...]
>
> > However, my next thought was to temporarily move my
> > /etc/apt/apt.conf file, which contains just the one proxy
> > line pointing at apt-cacher-ng, with the result that
> > apt-listbugs was able to run without any problem.
>
> I understand correctly: using apt-cacher-ng somehow breaks
> certificate validation for apt-listbugs?

Well, let's put it like this. Nothing has failed during the
acquisition of the packages through apt-cacher-ng's caching
mechanism. I tested this by choosing a brand-new package to
install, and attempting to install it. Sure enough, the
download step succeeded, and the package appeared in the
cache pool.

However, the apt-listbugs step failed on account of an expired
certificate, just like the example quoted. man apt-listbugs
says that it reads /etc/apt/apt.conf:

$ cat /etc/apt/apt.conf
Acquire::http::Proxy "http://192.168.1.14:3142/";
$

and temporarily removing that line allowed apt-listbugs to
succeed without provoking the error and ensuing dialogue.

> > I have no problem visiting bugs.debian.org with firefox
> > and lynx, so there's no urgency. I'm just happy that
> > two days ago I did a full install of bullseye on a
> > machine through my proxy and with listbugs turned on.
>
> That would be no surprise (provided I understood correctly
> above, that is): firefox and lynx don't proxy through
> apt-cacher-ng.

No, but look at the top of this thread:

"new certification problem, this time on buster.

"While looking why wget on Debian 8 does not work with lists.debian.org
i learn that it does not work on Debian 10 either:"

so in the interests of maximising the information in my post,
I checked that I didn't have that problem with either bug.d.o
or lists.d.o, or any other site.

I wanted to make this check, particular as my /etc/ssl/certs
contains a link to DST_Root_CA_X3.crt about which I know
nothing at all except that it's a certificate that had allegedly
just expired at midnight (see the older thread). The coincidence
in timing of this problem might be significant.

> > But it is odd that telling APT not to retrieve any bugs
> > should still say "Retrieving bug reports... Done".
>
> Proper messages are hard :)

You're right, but it would be fairly simple, I think, to turn
a two-way Fail/Done into a three-way Fail/Skipped/Done message,
only someone needs to point it out — like someone apparently
had already done for the message in the contemporaneous thread
https://lists.debian.org/debian-user/2021/10/msg00075.html

I know little about digital certificates except that:
. they cause trouble when they expire, and
. their keepers seem to forget that they're going to expire.

My ca-certificates_20200601~deb10u2_all.deb file is dated
Jan 29 2021 but the contents appear to be over a year old.
Does it contain the expired certificate? Do I rely on it?
Do I expect a new package soon? A routine upgrade, or one
caused by this specific problem? Bullseye's package is named
20210119, which just happens to be ten days before my file's
date. Does that mean my file is really just as up-to-date?
What does the string 20200601 actually mean?

Don't worry, those questions are just rhetorical, expressing
my ignorance.

Cheers,
David.

[David Wright]

On Sat 02 Oct 2021 at 08:15:59 (+0200), to...@tuxteam.de wrote:

> On Fri, Oct 01, 2021 at 03:18:22PM -0500, David Wright wrote:
>
> [...]
>
> > I have a buster system that was up-to-date from the last point-release
> > and kernel (2021-09-10 18:22:47).
> >
> > The only certificate expiration problem I have observed (and still
> > observe, having taken no action) is with apt-listbugs:
>
> [...]
>
> > However, my next thought was to temporarily move my
> > /etc/apt/apt.conf file, which contains just the one proxy
> > line pointing at apt-cacher-ng, with the result that
> > apt-listbugs was able to run without any problem.
>
> I understand correctly: using apt-cacher-ng somehow breaks
> certificate validation for apt-listbugs?

Yes. Here's a summary. Note that I had DST_Root_CA_X3.crt as
trusted in /etc/ssl/ until after it expired, and its expiry
caused no problems with browsers, package downloads etc, but
only this symptom. Since then, I have removed it from
/etc/ca-certificates.conf using the ! mechanism, yet this
symptom persists as shown here:

——✄——

# update-ca-certificates --fresh
Clearing symlinks in /etc/ssl/certs...
done.
Updating certificates in /etc/ssl/certs...
125 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...

Replacing debian:ACCVRAIZ1.pem
[ … ]
Replacing debian:DigiCert_Trusted_Root_G4.pem
Replacing debian:D-TRUST_Root_Class_3_CA_2_2009.pem
[ … ]
Replacing debian:ISRG_Root_X1.pem
[ … ]
Replacing debian:emSign_Root_CA_-_G1.pem
done.
done.
# logout
$ cat /etc/apt/apt.conf
#Acquire::http::Proxy "http://192.168.1.14:3142/";
$ apt-listbugs list base-files
Retrieving bug reports... Done
Parsing Found/Fixed information... Done

$ cat /etc/apt/apt.conf
Acquire::http::Proxy "http://192.168.1.14:3142/";

$ apt-listbugs list base-files

Retrieving bug reports... 0% Fail
Error retrieving bug reports from the server with the following error message:
E: SSL_connect returned=1 errno=0 state=error: certificate verify failed (certificate has expired)
It could be because your network is down, or because of broken proxy servers, or the BTS server itself is down. Check network configuration and try again

Retry downloading bug information? [Y/n] n
Continue the installation anyway? [y/N]

E: Exiting with error
$ cat /etc/debian_version
10.10
$

——✄——

AFAICT ISRG_Root_X1.pem points to the ordinary ISRG_Root_X1,
not the cross-signed one, BTW.

So I thought I would investigate a little further by running
apt-listbugs under strace -f. Note that though the apt-cacher-ng
service is running on the same machine, that service is AIUI
not being traced and so is opaque.

99% of the traces (without/with proxy) look the same,
until you pass the three lstat references to
/usr/lib/ruby/vendor_ruby/http/cookie_jar/hash_store.rb
whereupon they diverge.

Without proxy: after 47 lines of faffing about, it reads
/etc/hosts, does some DNS, and appears to communicate
with the University of Oregon, which would seem reasonable.

With proxy: after 222 lines of, I assume, chatting to the
proxy, it opens /usr/lib/ruby/vendor_ruby/httpclient/cacert.pem
and this contains a list of actual certificates, amongst
which, wouldn't you know, is DST Root CA X3.

So the problem lies in ruby, specifically the package
ruby-httpclient, version 2.8.3-2. This might explain
why "random" applications fail while other ones don't.

I verified the above by editing out the DST Root CA X3
certificate from /usr/lib/ruby/vendor_ruby/httpclient/cacert.pem
and restarting apt-cacher-ng.service, whereupon:

——✄——

$ apt-listbugs list base-files

Retrieving bug reports... 0% Fail
Error retrieving bug reports from the server with the following error message:

E: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)

It could be because your network is down, or because of broken proxy servers, or the BTS server itself is down. Check network configuration and try again
Retry downloading bug information? [Y/n] n
Continue the installation anyway? [y/N]

E: Exiting with error
$

——✄——

and the error changes from expired to missing as
cacert.pem does not include ISRG Root X1.

Cheers,
David.