Bug#912277: apache2: does not start any more: AH01903: Failed to configure CA certificate chain!
Thorsten Glaser
Version: 2.4.35-1
Severity: important
After a recent upgrade, apache2 does not start any more:
tglase@tglase:~ $ cat /var/log/apache2/error.log
[Mon Oct 29 20:18:58.090841 2018] [ssl:emerg] [pid 17306] AH01903: Failed to configure CA certificate chain!
[Mon Oct 29 20:18:58.090919 2018] [ssl:emerg] [pid 17306] AH02311: Fatal error initialising mod_ssl, exiting. See /var/log/apache2/error.log for more information
AH00016: Configuration Failed
The certificate itself and the chain are ok, though:
tglase@tglase:~ $ openssl verify -CApath /etc/ssl/certs -show_chain -purpose sslserver -verify_hostname tglase.lan.tarent.de -untrusted /etc/ssl/W_lan_tarent_de.ca /etc/ssl/W_lan_tarent_de.cer
/etc/ssl/W_lan_tarent_de.cer: OK
Chain:
depth=0: CN = *.lan.tarent.de (untrusted)
depth=1: C = US, O = DigiCert Inc, OU = www.digicert.com, CN = RapidSSL TLS RSA CA G1 (untrusted)
depth=2: C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root G2
Postfix, on the same system, using the same certificates…
smtpd_tls_cert_file = /etc/ssl/W_lan_tarent_de.cer
smtpd_tls_key_file = /etc/ssl/private/W_lan_tarent_de.key
smtpd_tls_CAfile = /etc/ssl/W_lan_tarent_de.ca
… runs fine, so this must be some regression in Apache2.
-- Package-specific info:
-- System Information:
Debian Release: buster/sid
APT prefers unreleased
APT policy: (500, 'unreleased'), (500, 'buildd-unstable'), (500, 'unstable')
Architecture: x32 (x86_64)
Foreign Architectures: i386, amd64
Kernel: Linux 4.18.0-2-amd64 (SMP w/8 CPU cores)
Locale: LANG=C, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8), LANGUAGE=C (charmap=UTF-8)
Shell: /bin/sh linked to /bin/lksh
Init: sysvinit (via /sbin/init)
Versions of packages apache2 depends on:
ii apache2-bin 2.4.35-1
ii apache2-data 2.4.35-1
ii apache2-utils 2.4.35-1
ii dpkg 1.19.2
ii lsb-base 9.20170808
ii mime-support 3.61
ii perl 5.26.2-7+b1
ii procps 2:3.3.15-2
Versions of packages apache2 recommends:
ii ssl-cert 1.0.39
Versions of packages apache2 suggests:
ii apache2-doc 2.4.35-1
pn apache2-suexec-pristine | apache2-suexec-custom <none>
ii dillo [www-browser] 3.0.5-4
ii links2 [www-browser] 2.17-1
ii lynx [www-browser] 2.8.9rel.1-2
ii opera-static [www-browser] 9.64.2480.gcc4.qt3
Versions of packages apache2-bin depends on:
ii libapr1 1.6.3-3
ii libaprutil1 1.6.1-3+b1
ii libaprutil1-dbd-pgsql 1.6.1-3+b1
ii libaprutil1-dbd-sqlite3 1.6.1-3+b1
ii libaprutil1-ldap 1.6.1-3+b1
ii libbrotli1 1.0.6-1
ii libc6 2.27-6
ii libcurl4 7.61.0-1
ii libjansson4 2.11-1
ii libldap-2.4-2 2.4.46+dfsg-5+x32.1
ii liblua5.2-0 5.2.4-1.1+b1
ii libnghttp2-14 1.34.0-1
ii libpcre3 2:8.39-11
ii libssl1.1 1.1.1-1
ii libxml2 2.9.4+dfsg1-7+b1
ii perl 5.26.2-7+b1
ii zlib1g 1:1.2.11.dfsg-1
Versions of packages apache2-bin suggests:
ii apache2-doc 2.4.35-1
pn apache2-suexec-pristine | apache2-suexec-custom <none>
ii dillo [www-browser] 3.0.5-4
ii links2 [www-browser] 2.17-1
ii lynx [www-browser] 2.8.9rel.1-2
ii opera-static [www-browser] 9.64.2480.gcc4.qt3
Versions of packages apache2 is related to:
ii apache2 2.4.35-1
ii apache2-bin 2.4.35-1
-- Configuration Files:
/etc/apache2/apache2.conf changed:
DefaultRuntimeDir ${APACHE_RUN_DIR}
PidFile ${APACHE_PID_FILE}
Timeout 300
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 5
User ${APACHE_RUN_USER}
Group ${APACHE_RUN_GROUP}
HostnameLookups Off
ErrorLog ${APACHE_LOG_DIR}/error.log
LogLevel warn
IncludeOptional mods-enabled/*.load
IncludeOptional mods-enabled/*.conf
Include ports.conf
<Directory />
Options FollowSymLinks
AllowOverride None
Require all denied
</Directory>
<Directory /usr/share>
AllowOverride None
Require all granted
</Directory>
<Directory /var/www/>
Options Indexes FollowSymLinks ExecCGI
AddHandler cgi-script .cgi
AllowOverride None
Require all granted
</Directory>
AccessFileName .htaccess
<FilesMatch "^\.ht">
Require all denied
</FilesMatch>
LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %O" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent
IncludeOptional conf-enabled/*.conf
IncludeOptional sites-enabled/*.conf
/etc/apache2/sites-available/default-ssl.conf changed:
<IfModule mod_ssl.c>
<VirtualHost _default_:443>
ServerAdmin webmaster@localhost
DocumentRoot /var/www/html
# Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
# error, crit, alert, emerg.
# It is also possible to configure the loglevel for particular
# modules, e.g.
#LogLevel info ssl:warn
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
# For most configuration files from conf-available/, which are
# enabled or disabled at a global level, it is possible to
# include a line for only one particular virtual host. For example the
# following line enables the CGI configuration for this host only
# after it has been globally disabled with "a2disconf".
#Include conf-available/serve-cgi-bin.conf
# SSL Engine Switch:
# Enable/Disable SSL for this virtual host.
SSLEngine on
# *.lan.tarent.de
SSLCertificateKeyFile /etc/ssl/private/W_lan_tarent_de.key
SSLCertificateFile /etc/ssl/W_lan_tarent_de.cer
SSLCertificateChainFile /etc/ssl/W_lan_tarent_de.ca
# A self-signed (snakeoil) certificate can be created by installing
# the ssl-cert package. See
# /usr/share/doc/apache2/README.Debian.gz for more info.
# If both key and certificate are stored in the same file, only the
# SSLCertificateFile directive is needed.
#SSLCertificateFile /etc/ssl/certs/ssl-cert-snakeoil.pem
#SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key
# Server Certificate Chain:
# Point SSLCertificateChainFile at a file containing the
# concatenation of PEM encoded CA certificates which form the
# certificate chain for the server certificate. Alternatively
# the referenced file can be the same as SSLCertificateFile
# when the CA certificates are directly appended to the server
# certificate for convinience.
#SSLCertificateChainFile /etc/apache2/ssl.crt/server-ca.crt
# Certificate Authority (CA):
# Set the CA certificate verification path where to find CA
# certificates for client authentication or alternatively one
# huge file containing all of them (file must be PEM encoded)
# Note: Inside SSLCACertificatePath you need hash symlinks
# to point to the certificate files. Use the provided
# Makefile to update the hash symlinks after changes.
#SSLCACertificatePath /etc/ssl/certs/
#SSLCACertificateFile /etc/apache2/ssl.crt/ca-bundle.crt
# Certificate Revocation Lists (CRL):
# Set the CA revocation path where to find CA CRLs for client
# authentication or alternatively one huge file containing all
# of them (file must be PEM encoded)
# Note: Inside SSLCARevocationPath you need hash symlinks
# to point to the certificate files. Use the provided
# Makefile to update the hash symlinks after changes.
#SSLCARevocationPath /etc/apache2/ssl.crl/
#SSLCARevocationFile /etc/apache2/ssl.crl/ca-bundle.crl
# Client Authentication (Type):
# Client certificate verification type and depth. Types are
# none, optional, require and optional_no_ca. Depth is a
# number which specifies how deeply to verify the certificate
# issuer chain before deciding the certificate is not valid.
#SSLVerifyClient require
#SSLVerifyDepth 10
# SSL Engine Options:
# Set various options for the SSL engine.
# o FakeBasicAuth:
# Translate the client X.509 into a Basic Authorisation. This means that
# the standard Auth/DBMAuth methods can be used for access control. The
# user name is the `one line' version of the client's X.509 certificate.
# Note that no password is obtained from the user. Every entry in the user
# file needs this password: `xxj31ZMTZzkVA'.
# o ExportCertData:
# This exports two additional environment variables: SSL_CLIENT_CERT and
# SSL_SERVER_CERT. These contain the PEM-encoded certificates of the
# server (always existing) and the client (only existing when client
# authentication is used). This can be used to import the certificates
# into CGI scripts.
# o StdEnvVars:
# This exports the standard SSL/TLS related `SSL_*' environment variables.
# Per default this exportation is switched off for performance reasons,
# because the extraction step is an expensive operation and is usually
# useless for serving static content. So one usually enables the
# exportation for CGI and SSI requests only.
# o OptRenegotiate:
# This enables optimized SSL connection renegotiation handling when SSL
# directives are used in per-directory context.
#SSLOptions +FakeBasicAuth +ExportCertData +StrictRequire
<FilesMatch "\.(cgi|shtml|phtml|php)$">
SSLOptions +StdEnvVars
</FilesMatch>
<Directory /usr/lib/cgi-bin>
SSLOptions +StdEnvVars
</Directory>
# SSL Protocol Adjustments:
# The safe and default but still SSL/TLS standard compliant shutdown
# approach is that mod_ssl sends the close notify alert but doesn't wait for
# the close notify alert from client. When you need a different shutdown
# approach you can use one of the following variables:
# o ssl-unclean-shutdown:
# This forces an unclean shutdown when the connection is closed, i.e. no
# SSL close notify alert is send or allowed to received. This violates
# the SSL/TLS standard but is needed for some brain-dead browsers. Use
# this when you receive I/O errors because of the standard approach where
# mod_ssl sends the close notify alert.
# o ssl-accurate-shutdown:
# This forces an accurate shutdown when the connection is closed, i.e. a
# SSL close notify alert is send and mod_ssl waits for the close notify
# alert of the client. This is 100% SSL/TLS standard compliant, but in
# practice often causes hanging connections with brain-dead browsers. Use
# this only for browsers where you know that their SSL implementation
# works correctly.
# Notice: Most problems of broken clients are also related to the HTTP
# keep-alive facility, so you usually additionally want to disable
# keep-alive for those clients, too. Use variable "nokeepalive" for this.
# Similarly, one has to force some clients to use HTTP/1.0 to workaround
# their broken HTTP/1.1 implementation. Use variables "downgrade-1.0" and
# "force-response-1.0" for this.
# BrowserMatch "MSIE [2-6]" \
# nokeepalive ssl-unclean-shutdown \
# downgrade-1.0 force-response-1.0
</VirtualHost>
</IfModule>
-- no debconf information
Stefan Fritsch
> tglase@tglase:~ $ cat /var/log/apache2/error.log
> [Mon Oct 29 20:18:58.090841 2018] [ssl:emerg] [pid 17306] AH01903: Failed to
> configure CA certificate chain!
> [Mon Oct 29 20:18:58.090919 2018] [ssl:emerg] [pid 17306] AH02311: Fatal
> error initialising mod_ssl, exiting.
> See /var/log/apache2/error.log for more information AH00016: Configuration
> Failed
If there is none, try adding loglevel ssl:debug and re-try.
Thorsten Glaser
> On Monday, 29 October 2018 20:31:54 CET Thorsten Glaser wrote:
> > tglase@tglase:~ $ cat /var/log/apache2/error.log
> > [Mon Oct 29 20:18:58.090841 2018] [ssl:emerg] [pid 17306] AH01903: Failed to
> > configure CA certificate chain!
> > [Mon Oct 29 20:18:58.090919 2018] [ssl:emerg] [pid 17306] AH02311: Fatal
> > error initialising mod_ssl, exiting.
> > See /var/log/apache2/error.log for more information AH00016: Configuration
> > Failed
>
> Have you looked into /var/log/apache2/error.log if there is more information?
as the “cat” showed…
I just hit this on another machine, it’s the 2.4.35-1 → 2.4.37-1 upgrade
that caused the failure.
> If there is none, try adding loglevel ssl:debug and re-try.
That gives:
[Sun Nov 04 17:05:02.839408 2018] [ssl:info] [pid 18196] AH01887: Init: Initializing (virtual) servers for SSL
[Sun Nov 04 17:05:02.839427 2018] [ssl:info] [pid 18196] AH01914: Configuring server ci-busyapps.lan.tarent.de:443 for SSL protocol
[Sun Nov 04 17:05:02.839433 2018] [ssl:debug] [pid 18196] ssl_engine_init.c(1748): AH10083: Init: (ci-busyapps.lan.tarent.de:443) mod_md support is unavailable.
[Sun Nov 04 17:05:02.839729 2018] [ssl:emerg] [pid 18196] AH01903: Failed to configure CA certificate chain!
[Sun Nov 04 17:05:02.839739 2018] [ssl:emerg] [pid 18196] AH02311: Fatal error initialising mod_ssl, exiting. See /var/log/apache2/error.log for more information
AH00016: Configuration Failed
So perhaps the mod_ssl backport / new feature was bad?
On a hunch, I tried a2enmod md, but that does not change much:
[Sun Nov 04 17:05:47.417353 2018] [ssl:info] [pid 18229] AH01887: Init: Initializing (virtual) servers for SSL
[Sun Nov 04 17:05:47.417371 2018] [ssl:info] [pid 18229] AH01914: Configuring server ci-busyapps.lan.tarent.de:443 for SSL protocol
[Sun Nov 04 17:05:47.417377 2018] [ssl:debug] [pid 18229] ssl_engine_init.c(1748): AH10083: Init: (ci-busyapps.lan.tarent.de:443) mod_md support is available.
[Sun Nov 04 17:05:47.417663 2018] [ssl:emerg] [pid 18229] AH01903: Failed to configure CA certificate chain!
[Sun Nov 04 17:05:47.417673 2018] [ssl:emerg] [pid 18229] AH02311: Fatal error initialising mod_ssl, exiting. See /var/log/apache2/error.log for more information
AH00016: Configuration Failed
More debugging data points: this did not occur immediately after
the package upgrade, only when I did an /etc/init.d/apache2 stop
followed by start.
Worse, this persists after downgrading apache2, apache2-bin,
apache2-data, apache2-utils to 2.4.35-1 (?!?!?!).
Dazed and confused,
//mirabilos
--
tarent solutions GmbH
Rochusstraße 2-4, D-53123 Bonn • http://www.tarent.de/
Tel: +49 228 54881-393 • Fax: +49 228 54881-235
HRB 5168 (AG Bonn) • USt-ID (VAT): DE122264941
Geschäftsführer: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Alexander Steeg
Thorsten Glaser
> I just hit this on another machine, it’s the 2.4.35-1 → 2.4.37-1 upgrade
> that caused the failure.
> More debugging data points: this did not occur immediately after
> the package upgrade, only when I did an /etc/init.d/apache2 stop
> followed by start.
>
> Worse, this persists after downgrading apache2, apache2-bin,
> apache2-data, apache2-utils to 2.4.35-1 (?!?!?!).
2.4.33-3+b1 is the oldest version I can downgrade to, and it
also exhibits the problem. WTF.
We did switch certificates recently, but OpenSSL accepts them…
Still puzzled,
Thorsten Glaser
thanks
> 2.4.33-3+b1 is the oldest version I can downgrade to, and it
> also exhibits the problem. WTF.
and thought “hey, Apache 2 still documents SSLCertificateChainFile,
plus it’s the proper way to specify the chain given it’s normally
separate from the certificates, and there’s no warning message about
that directive, but let’s give it a shot”.
So I did:
# cat /etc/ssl/W_lan_tarent_de.cer /etc/ssl/W_lan_tarent_de.ca >/etc/ssl/combined-cer-chain.pem
Then I edited /etc/apache2/sites-enabled/default-ssl.conf, commenting
out SSLCertificateFile and SSLCertificateChainFile, and adding
SSLCertificateFile /etc/ssl/combined-cer-chain.pem
tglase@tglase:~ $ sudo cleanenv / /etc/init.d/apache2 stop
Stopping Apache httpd web server: apache2.
Server was not running ... (warning).
tglase@tglase:~ $ sudo cleanenv / /etc/init.d/apache2 start
Starting Apache httpd web server: apache2 ..
.oO(wait, what?)
tglase@tglase:~ $ curl --head https://$(hostname -f)/
HTTP/1.1 200 OK
Date: Sun, 04 Nov 2018 17:34:29 GMT
Server: Apache/2.4.35 (Debian)
Content-Type: text/html;charset=UTF-8
.oO(what now?)
So it turns out that, ever since some upgrade, the directive
SSLCertificateChainFile is *silently* ignored, but this only
becomes apparent when you stop+start instead of restart (so
they are *still* not equivalent ☹).
I don’t think this acceptable. Ideally, the option would be
still supported; it does no harm and has worked for decades.
If that’s not desired, it MUST yield a warning.
bye,
Stefan Fritsch
> This is a real WTF. I found https://serverfault.com/a/892300/189656
> and thought “hey, Apache 2 still documents SSLCertificateChainFile,
> plus it’s the proper way to specify the chain given it’s normally
> separate from the certificates, and there’s no warning message about
> that directive, but let’s give it a shot”.
Maybe you are hitting https://bz.apache.org/bugzilla/show_bug.cgi?id=62880 .
That would cause non-deterministic behavior (depending on what went on before
the SSLCertificateChainFile is encountered). If you have time, you could try
the patch attached to that bug. Right now, I don't have time to prepare test
packages.
Cheers,
Stefan
Dorian Kwiatkowski
kontaktuję się z Państwem, ponieważ dostrzegam możliwość redukcji opłat za prąd.
Odpowiednio dobrana instalacja fotowoltaiczna to rozwiązanie, które pozwala wygenerować spore oszczędności w skali roku.
Chciałbym porozmawiać z Państwem o tego typu rozwiązaniu, a także przedstawić wstępne kalkulacje.
Czy są Państwo zainteresowani?
Pozdrawiam,
Dorian Kwiatkowski
Patryk Górecki
jakiś czas temu zgłosiła się do nas firma, której strona internetowa nie pozycjonowała się wysoko w wyszukiwarce Google.
Na podstawie wykonanego przez nas audytu SEO zoptymalizowaliśmy treści na stronie pod kątem wcześniej opracowanych słów kluczowych. Nasz wewnętrzny system codziennie analizuje prawidłowe działanie witryny. Dzięki indywidualnej strategii, firma zdobywa coraz więcej Klientów.
Czy chcieliby Państwo zwiększyć liczbę osób odwiedzających stronę internetową firmy? Mógłbym przedstawić ofertę?
Pozdrawiam serdecznie,
Patryk Górecki
Patryk Górecki
Dawid Rowicki
Czy mógłbym przedstawić rozwiązanie, które umożliwia monitoring każdego auta w czasie rzeczywistym w tym jego pozycję, zużycie paliwa i przebieg?
Dodatkowo nasze narzędzie minimalizuje koszty utrzymania samochodów, skraca czas przejazdów, a także tworzenie planu tras czy dostaw.
Z naszej wiedzy i doświadczenia korzysta już ponad 49 tys. Klientów. Monitorujemy 809 000 pojazdów na całym świecie, co jest naszą najlepszą wizytówką.
Bardzo proszę o e-maila zwrotnego, jeśli moglibyśmy wspólnie omówić potencjał wykorzystania takiego rozwiązania w Państwa firmie.
Z poważaniem,
Dawid Rowicki
Arkadiusz Stryj
kontaktuje się z Państwem, ponieważ jako osoba zajmująca się usprawnieniem procesów, chciałbym zaprezentować nowoczesne rozwiązanie dla Państwa firmy.
System został stworzony z myślą o przedsiębiorstwach z sektora MŚP, aby zapewnić bezpieczeństwo danych, niezawodność i optymalizację procesów. Dzięki bogatym funkcjonalnościom usprawnia i przyspiesza obsługę zleceń.
Jeśli chcieliby Państwo zwiększyć tempo rozwoju swojej działalności i poszerzyć rynek zbytu chętnie opowiem więcej. Kiedy mogę się skontaktować?
Pozdrawiam,
Arkadiusz Stryj
Adam Furgalski
zapoznałem się z Państwa ofertą i z przyjemnością przyznaję, że przyciąga uwagę i zachęca do dalszych rozmów.
Pomyślałem, że może mógłbym mieć swój wkład w Państwa rozwój i pomóc dotrzeć z tą ofertą do większego grona odbiorców. Pozycjonuję strony www, dzięki czemu generują świetny ruch w sieci.
Możemy porozmawiać w najbliższym czasie?
Pozdrawiam
Adam Furgalski