Bug#1026353: mariadb-server: mariadb does not start after bullseye point release 11.6
Matthew P Zagrabelny
Version: 1:10.5.18-0+deb11u1
Severity: important
Dear Maintainer,
Unattended upgrade upgraded mariadb this morning and now the service does not
start:
Start-Date: 2022-12-18 06:15:03
Commandline: /usr/bin/unattended-upgrade
Upgrade: mariadb-server:amd64 (1:10.5.15-0+deb11u1, 1:10.5.18-0+deb11u1), mariadb-server-10.5:amd64 (1:10.5.15-0+deb11u1, 1:10.5.18-0+deb11u1)
End-Date: 2022-12-18 06:15:09
# systemctl status mariadb.service
● mariadb.service - MariaDB 10.5.18 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sun 2022-12-18 15:25:38 CST; 38min ago
Docs: man:mariadbd(8)
https://mariadb.com/kb/en/library/systemd/
Process: 481 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Process: 498 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 507 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)
Process: 604 ExecStart=/usr/sbin/mariadbd $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
Main PID: 604 (code=exited, status=1/FAILURE)
Status: "MariaDB server is down"
CPU: 190ms
Dec 18 15:25:36 mariadb-test-system systemd[1]: Starting MariaDB 10.5.18 database server...
Dec 18 15:25:37 mariadb-test-system mariadbd[604]: 2022-12-18 15:25:37 0 [Note] /usr/sbin/mariadbd (mysqld 10.5.18-MariaDB-0+deb11u1) starting as process 604 ...
Dec 18 15:25:38 mariadb-test-system systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Dec 18 15:25:38 mariadb-test-system systemd[1]: mariadb.service: Failed with result 'exit-code'.
Dec 18 15:25:38 mariadb-test-system systemd[1]: Failed to start MariaDB 10.5.18 database server.
I've run:
# strace -f /usr/sbin/mariadbd
but there isn't anything that sticks out to me.
# journalctl -xe
Dec 18 16:09:48 mariadb-test-system systemd[1]: Starting MariaDB 10.5.18 database server...
░░ Subject: A start job for unit mariadb.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit mariadb.service has begun execution.
░░
░░ The job identifier is 789.
Dec 18 16:09:49 mariadb-test-system mariadbd[8369]: 2022-12-18 16:09:49 0 [Note] /usr/sbin/mariadbd (mysqld 10.5.18-MariaDB-0+deb11u1) starting as process 8369 ...
Dec 18 16:09:49 mariadb-test-system systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ An ExecStart= process belonging to unit mariadb.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 1.
Dec 18 16:09:49 mariadb-test-system systemd[1]: mariadb.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit mariadb.service has entered the 'failed' state with result 'exit-code'.
Dec 18 16:09:49 mariadb-test-system systemd[1]: Failed to start MariaDB 10.5.18 database server.
░░ Subject: A start job for unit mariadb.service has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit mariadb.service has finished with a failure.
░░
░░ The job identifier is 789 and the job result is failed.
-- System Information:
Debian Release: 11.6
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'stable-security'), (500, 'stable')
Architecture: amd64 (x86_64)
Kernel: Linux 5.10.0-20-amd64 (SMP w/1 CPU thread)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages mariadb-server depends on:
ii mariadb-server-10.5 1:10.5.18-0+deb11u1
mariadb-server recommends no packages.
mariadb-server suggests no packages.
-- no debconf information
Michael Prokop
> Version: 1:10.5.18-0+deb11u1
> Severity: important
>
> start:
>
> Dec 18 15:25:36 mariadb-test-system systemd[1]: Starting MariaDB 10.5.18 database server...
> Dec 18 15:25:37 mariadb-test-system mariadbd[604]: 2022-12-18 15:25:37 0 [Note] /usr/sbin/mariadbd (mysqld 10.5.18-MariaDB-0+deb11u1) starting as process 604 ...
> Dec 18 15:25:38 mariadb-test-system systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
> Dec 18 15:25:38 mariadb-test-system systemd[1]: mariadb.service: Failed with result 'exit-code'.
> Dec 18 15:25:38 mariadb-test-system systemd[1]: Failed to start MariaDB 10.5.18 database server.
referenced in your configuration, like:
| [mysqld]
| [...]
| ssl = false
| ssl-ca = /etc/mysql/cacert.pem
| ssl-cert = /etc/mysql/server-cert.pem
| ssl-key = /etc/mysql/server-key.pem
| [...]
So while it was even set to `ssl = false` on this system, it now
fails with:
| 2022-12-19 10:33:24 0 [ERROR] Failed to setup SSL
| 2022-12-19 10:33:24 0 [ERROR] SSL error: SSL_CTX_set_default_verify_paths failed
| 2022-12-19 10:33:24 0 [ERROR] Aborting
(FTR, removing the ssl-ca/ssl-cert/ssl-key settings fixed it for me,
those settings came from defaults of
https://github.com/puppetlabs/puppetlabs-mysql so I'm sure more
folks will be affected.)
This used to be a warning only until and including mariadb
version 1:10.5.15-0+deb11u1:
| 2022-11-22 7:42:46 0 [Warning] Failed to setup SSL
| 2022-11-22 7:42:46 0 [Warning] SSL error: SSL_CTX_set_default_verify_paths failed
| 2022-11-22 7:42:46 0 [Warning] SSL error: error:02001002:system library:fopen:No such file or directory
| 2022-11-22 7:42:46 0 [Warning] SSL error: error:2006D080:BIO routines:BIO_new_file:no such file
| 2022-11-22 7:42:46 0 [Warning] SSL error: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib
Now as of mariadb 1:10.5.18-0+deb11u1 (as present in bullseye as of
the latest point release) this ends up as hard failure:
| 2022-12-18 6:42:14 0 [ERROR] Failed to setup SSL
| 2022-12-18 6:42:14 0 [ERROR] SSL error: SSL_CTX_set_default_verify_paths failed
| 2022-12-18 6:42:14 0 [ERROR] Aborting
Possibly related to the OpenSSL 3.0 support introduced with 10.5.17
(see https://mariadb.com/kb/en/mariadb-10517-release-notes/), but
IMO this is a regression.
regards
-mika-
Michael Prokop
> > * Matthew P Zagrabelny [Sun Dec 18, 2022 at 04:11:54PM -0600]:
> > referenced in your configuration, like:
> >
> > | [mysqld]
> > | [...]
> > | ssl = false
> > | ssl-ca = /etc/mysql/cacert.pem
> > | ssl-cert = /etc/mysql/server-cert.pem
> > | ssl-key = /etc/mysql/server-key.pem
> > | [...]
> >
> > So while it was even set to `ssl = false` on this system, it now
> > fails with:
> >
> > | 2022-12-19 10:33:24 0 [ERROR] Failed to setup SSL
> > | 2022-12-19 10:33:24 0 [ERROR] SSL error:
> > SSL_CTX_set_default_verify_paths failed
> > | 2022-12-19 10:33:24 0 [ERROR] Aborting
> >
> > (FTR, removing the ssl-ca/ssl-cert/ssl-key settings fixed it for me,
> > those settings came from defaults of
> > https://github.com/puppetlabs/puppetlabs-mysql so I'm sure more
> > folks will be affected.)
>
>
> 'ssl-disable' => true
>
> in the puppet manifest, but that generates spurious warnings and mysqld
> refreshes:
>
> # puppet agent -t
[...]
> So... I'm not sure what the best way forward is.
[...]
See e.g. https://github.com/puppetlabs/puppetlabs-mysql/issues/1509,
so make sure to update to latest
https://github.com/puppetlabs/puppetlabs-mysql, which also includes
https://github.com/puppetlabs/puppetlabs-mysql/pull/1513 and takes
care of this, at least to get a working default again. (While I
still think that a stable update shouldn't include such a behavior
change. :))
HTH && regards
-mika-
Faustin Lammler
Otto Kekäläinen
bug in the sense that users need to fix their TLS configuration and
there is not anything the server should do differently at this point.
This was also related to
https://bugs.launchpad.net/ubuntu/+source/mariadb-10.3/+bug/1997916
and https://jira.mariadb.org/browse/MDEV-29811
Note that latest version of MariaDB in Debian
unstable/testing/Bookworm is 10.11.2. You might want to consider
testing it.
If you want to contribute in the open source way to fix this or any
other issue, see
https://salsa.debian.org/mariadb-team/mariadb-server/-/wikis/Contributing-to-MariaDB-packaging-in-Debian
on how to submit a Merge Request!
If you have time to help, please consider these (in order of importance):
1. Review current open MRs at
https://salsa.debian.org/mariadb-team/mariadb-server/-/merge_requests
2. Review items highlighted by Debian QA systems (Lintian, builds etc)
and submit a fix to improve the quality:
https://tracker.debian.org/pkg/mariadb
3. Review what testing we have at
https://salsa.debian.org/mariadb-team/mariadb-server/-/pipelines and
think about potential gaps - CI is very important as it is the only
way we can prevent regressions in a scalable way
4. Review/follow-up on existing bugs that currently need more
information: https://bugs.debian.org/cgi-bin/pkgreport.cgi?repeatmerged=no&src=mariadb&src=mariadb-10.6&src=mariadb-10.5&src=mariadb-10.3&src=mariadb-10.1
MariaDB and C++ skills are useful, but not required. For example
reviewing the NEWS for 10.11 requires no coding skills:
https://salsa.debian.org/mariadb-team/mariadb-server/-/merge_requests/37
My request for help from debian-devel in
https://lists.debian.org/debian-devel/2023/02/msg00272.html did not
get many responses, so the future of this package depends on how
active the users and people who previously reported bugs are in
participating in the maintenance of the package.
- Otto