Bug#1009290: mariadb-server-10.6: Fails to start on live system

34 views
Skip to first unread message

Daniel Lewart

unread,
Apr 11, 2022, 1:50:03 AMApr 11
to
Package: mariadb-server-10.6
Version: 1:10.6.7-3
Severity: important

Debian MariaDB Maintainers,

Live Build (--distribution testing) image from April 2022.
8 GB RAM, so should be plenty of disk space available.

$ sudo apt install mariadb-server-10.6-dbgsym
...
Setting up mariadb-server-10.6 (1:10.6.7-3) ...
Created symlink /etc/systemd/system/multi-user.target.wants/mariadb.service → /lib/systemd/system/mariadb.service.
Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142.
...

$ sudo journalctl | grep -i mariadb | cut -c58-
[See below]

Perhaps this is similar to:
#983002 - plocate: does not work with /var on overlayfs
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=983002

Please let me know any additional information that you could use.

Thank you!
Daniel Lewart
Urbana, Illinois
---
2022-04-10 22:53:02 0 [Warning] InnoDB: Retry attempts for writing partial data failed.
2022-04-10 22:53:02 0 [ERROR] InnoDB: Write to file ./ibdata1 failed at offset 0, 1048576 bytes should have been written, only 0 were written. Operating system error number 22. Check that your OS and file system support files of this size. Check also that the disk is not full or a disk quota exceeded.
2022-04-10 22:53:02 0 [ERROR] InnoDB: Error number 22 means 'Invalid argument'
2022-04-10 22:53:02 0 [ERROR] InnoDB: Could not set the file size of './ibdata1'. Probably out of disk space
2022-04-10 22:53:02 0 [ERROR] InnoDB: Database creation was aborted with error Generic error. You may need to delete the ibdata1 file before trying to start up again.
2022-04-10 22:53:02 0 [ERROR] InnoDB: Operating system error number 22 in a file operation.
2022-04-10 22:53:02 0 [ERROR] InnoDB: Error number 22 means 'Invalid argument'
2022-04-10 22:53:02 0 [ERROR] InnoDB: File (unknown): 'close' returned OS error 222. Cannot continue operation
220410 22:53:02 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.6.7-MariaDB-3
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=0
max_threads=153
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467957 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
2022-04-10 22:53:02 0 [ERROR] InnoDB: Operating system error number 9 in a file operation.
2022-04-10 22:53:02 0 [ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
2022-04-10 22:53:02 0 [ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation
Printing to addr2line failed
/usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x555d0aa77f0e]
/usr/sbin/mariadbd(handle_fatal_signal+0x478)[0x555d0a5c5268]
2022-04-10 22:53:02 0 [ERROR] InnoDB: Operating system error number 9 in a file operation.
2022-04-10 22:53:02 0 [ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
2022-04-10 22:53:02 0 [ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12200)[0x7f606ba96200]
2022-04-10 22:53:02 0 [ERROR] InnoDB: Operating system error number 9 in a file operation.
2022-04-10 22:53:02 0 [ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
2022-04-10 22:53:02 0 [ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x141)[0x7f606b57a8a1]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x112)[0x7f606b564546]
2022-04-10 22:53:02 0 [ERROR] InnoDB: Operating system error number 9 in a file operation.
2022-04-10 22:53:02 0 [ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
2022-04-10 22:53:02 0 [ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation
/usr/sbin/mariadbd(+0x63b8e9)[0x555d0a2638e9]
/usr/sbin/mariadbd(+0xcb66e5)[0x555d0a8de6e5]
/usr/sbin/mariadbd(+0xdbc308)[0x555d0a9e4308]
/usr/sbin/mariadbd(+0xdbc371)[0x555d0a9e4371]
/usr/sbin/mariadbd(+0xdbea20)[0x555d0a9e6a20]
/usr/sbin/mariadbd(+0xdbfb41)[0x555d0a9e7b41]
/usr/sbin/mariadbd(+0xd2585f)[0x555d0a94d85f]
/usr/sbin/mariadbd(+0xc64245)[0x555d0a88c245]
/usr/sbin/mariadbd(_Z24ha_initialize_handlertonP13st_plugin_int+0x7e)[0x555d0a5c839e]
/usr/sbin/mariadbd(+0x797a76)[0x555d0a3bfa76]
/usr/sbin/mariadbd(_Z11plugin_initPiPPci+0x828)[0x555d0a3c0a38]
/usr/sbin/mariadbd(+0x6c367a)[0x555d0a2eb67a]
/usr/sbin/mariadbd(_Z11mysqld_mainiPPc+0x404)[0x555d0a2f1224]
2022-04-10 22:53:02 0 [ERROR] InnoDB: Operating system error number 9 in a file operation.
2022-04-10 22:53:02 0 [ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
2022-04-10 22:53:02 0 [ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f606b5657fd]
2022-04-10 22:53:02 0 [ERROR] InnoDB: Operating system error number 9 in a file operation.
2022-04-10 22:53:02 0 [ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
2022-04-10 22:53:02 0 [ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation
/usr/sbin/mariadbd(_start+0x2a)[0x555d0a2e680a]
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /var/lib/mysql
Resource Limits:
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 0 bytes
Max resident set unlimited unlimited bytes
Max processes 29195 29195 processes
Max open files 32190 32190 files
Max locked memory 976257024 976257024 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 29195 29195 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
Core pattern: core

/usr/bin/mysql_install_db: line 541: 7508 Aborted "$mysqld_bootstrap" $defaults $defaults_group_suffix "$mysqld_opt" --bootstrap $silent_startup "--basedir=$basedir" "--datadir=$ldata" --log-warnings=0 --enforce-storage-engine="" "--plugin-dir=${plugindir}" $args --max_allowed_packet=8M --net_buffer_length=16K

Installation of system tables failed! Examine the logs in
/var/lib/mysql for more information.

The problem could be conflicting information in an external
my.cnf files. You can ignore these by doing:

shell> /usr/bin/mysql_install_db --defaults-file=~/.my.cnf

You can also try to start the mysqld daemon with:

shell> /usr/sbin/mariadbd --skip-grant-tables --general-log &

and use the command line tool /usr/bin/mariadb
to connect to the mysql database and look at the grant tables:

shell> /usr/bin/mysql -u root mysql
mysql> show tables;

Try 'mysqld --help' if you have problems with paths. Using
--general-log gives you a log in /var/lib/mysql that may be helpful.

The latest information about mysql_install_db is available at
https://mariadb.com/kb/en/installing-system-tables-mysql_install_db
You can find the latest source at https://downloads.mariadb.org and
the maria-discuss email list at https://launchpad.net/~maria-discuss

Please check all of the above before submitting a bug report
at https://mariadb.org/jira

###

Denys Nykula

unread,
May 18, 2022, 10:30:03 AMMay 18
to
Hello. The apt install of mariadb-server-10.6 fails for me too,
with the same line "Could not execute systemctl" but different
journal messages. I removed the existing setup in /etc and /var,
purged dependencies, but install still fails. How do I debug?

systemd[1]: Starting MariaDB 10.6.7 database server...
[Note] /usr/sbin/mariadbd (server 10.6.7-MariaDB-3+b1) starting as process 98614 ...
[Note] InnoDB: The first data file './ibdata1' did not exist. A new tablespace will be created!
[Note] InnoDB: Compressed tables use zlib 1.2.11
[Note] InnoDB: Number of pools: 1
[Note] InnoDB: Using crc32 + pclmulqdq instructions
[Note] InnoDB: Using liburing
[Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
[Note] InnoDB: Completed initialization of buffer pool
[Note] InnoDB: Setting file './ibdata1' size to 12 MB. Physically writing the file full; Please wait ...
[Note] InnoDB: File './ibdata1' size is now 12 MB.
[Note] InnoDB: Setting log file ./ib_logfile101 size to 100663296 bytes
[Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
[Note] InnoDB: New log file created, LSN=10329
[Note] InnoDB: Doublewrite buffer not found: creating new
[Note] InnoDB: Doublewrite buffer created
[Note] InnoDB: 128 rollback segments are active.
[Note] InnoDB: Creating shared tablespace for temporary tables
[Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
[Note] InnoDB: File './ibtmp1' size is now 12 MB.
[Note] InnoDB: 10.6.7 started; log sequence number 0; transaction id 3
[Note] Plugin 'FEEDBACK' is disabled.
[ERROR] Could not open mysql.plugin table: "Table 'mysql.plugin' doesn't exist". Some plugins may be not loaded
[Warning] You need to use --log-bin to make --expire-logs-days or --binlog-expire-logs-seconds work.
[ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist
[Note] Server socket created on IP: '127.0.0.1'.
[ERROR] Fatal error: Can't open and lock privilege tables: Table 'mysql.db' doesn't exist
[ERROR] Aborting
Warning: Memory not freed: 280
systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: mariadb.service: Failed with result 'exit-code'.
systemd[1]: Failed to start MariaDB 10.6.7 database server.

Faustin Lammler

unread,
May 23, 2022, 10:40:03 AMMay 23
to
Hi Denys,
I am not sure that the problem is the same, in your case it seems that
the datadir does not exist or need to be recreated.

If you are sure that nothing can be lost (or that you have backups), you
should try the following:
| sudo systemctl stop mariadb
| sudo rm -rf /var/lib/mysql/*
| sudo -u mysql mariadb-install-db
| Installing MariaDB/MySQL system tables in '/var/lib/mysql' ...
| 2022-05-23 14:18:53 0 [Warning] mariadbd: io_uring_queue_init() failed with errno 1
| 2022-05-23 14:18:53 0 [Warning] InnoDB: liburing disabled: falling back to innodb_use_native_aio=OFF
| 2022-05-23 14:18:53 0 [Warning] You need to use --log-bin to make --expire-logs-days or --binlog-expire-logs-seconds work.
| OK
| ...
| sudo systemctl start mariadb

Regards,

Denys Nykula <ve...@libre.net.ua>,
18/05/2022 – 17:16:00 (+0300):
> _______________________________________________
> pkg-mysql-maint mailing list
> pkg-mys...@alioth-lists.debian.net
> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-mysql-maint

--
Faustin
signature.asc

Faustin Lammler

unread,
May 23, 2022, 10:40:03 AMMay 23
to
Hi Daniel!
Thanks for your detailed report.

I am not able to reproduce this with the latest
mariadb-server-10.6-dbgsym version (10.6.8).

Can you try with the latest version and tell me?

Also, can you describe a bit better your setup? If this is a filesystem
related issue, I would like to try to reproduce it myself.

I have tested with a systemd container that I have created for these
kind of testing, here are the commands (using podman):

| podman run --name sys-test --rm -d fauust/docker-systemd:debian-sid
| podman exec -it sys-test bash -c "echo \"deb http://deb.debian.org/debian-debug/ sid-debug main\" >> /etc/apt/sources.list && apt update && apt install -y mariadb-server-10.6-dbgsym"

Cheers!

Daniel Lewart <lew...@gmail.com>,
11/04/2022 – 00:20:00 (-0500):
signature.asc

Denys Nykula

unread,
May 23, 2022, 11:00:03 AMMay 23
to
> > sudo rm -rf /var/lib/mysql/*
> > sudo -u mysql mariadb-install-db

Thanks Faustin, something was fixed between 10.6.7-3 and 10.6.8-1 that
made mariadb-install-db succeed for me since today morning.

With the March packages it kept failing every time during or after the
InnoDB initialization (as I mentioned, I started clean every time,
removing the mysql directories in /etc and /var).

Daniel Lewart

unread,
May 26, 2022, 12:30:03 AMMay 26
to
Faustin,

> I am not able to reproduce this with the latest
> mariadb-server-10.6-dbgsym version (10.6.8).

> Can you try with the latest version and tell me?

Yes, I tried 10.6.8-1 with the same result.

> Also, can you describe a bit better your setup? If this is a filesystem
> related issue, I would like to try to reproduce it myself.

Yes. This is a simplified version of my setup ...

1) sudo apt -y install apt-cacher-ng live-build

2) I used Live Build to create a live sid image:
#!/bin/sh
cd /dev/shm
sudo mount /dev/shm -odev,exec,remount,size=3G
lb config --apt-http-proxy http://localhost:3142 -d sid
echo '! Packages Priority standard' \
> config/package-lists/standard.list.chroot
echo 'deb http://deb.debian.org/debian-debug sid-debug main' \
> config/archives/live.list.binary
sudo lb build

3) qemu-system-x86_64 -enable-kvm -cdrom /dev/shm/live-image-amd64.hybrid.iso \
-cpu host -m 4G -smp 2

4) In the guest, it didn't matter what I tried:
sudo apt -y install mariadb-server
sudo apt -y install mariadb-server-10.6-dbgsym
mariadb-server-core-10.6-dbgsym

I'm sure it is an overlay filesystem issue:

$ sudo ls -l /var/lib/mysql/ibdata1
-rw-rw---- 1 mysql mysql 0 May 25 12:34 /var/lib/mysql/ibdata1

$ df -hT /var/lib/mysql
Filesystem Type Size Used Avail Use% Mounted on
overlay overlay 2.0G 355M 1.6G 19% /

$ findmnt -J /
{
"filesystems": [
{
"target": "/",
"source": "overlay",
"fstype": "overlay",
"options":
"rw,noatime,lowerdir=/run/live/rootfs/filesystem.squashfs/,upperdir=/run/live/overlay/rw,workdir=/run/live/overlay/work"
}
]
}

> I have tested with a systemd container that I have created for these
> kind of testing, here are the commands (using podman):
>
> | podman run --name sys-test --rm -d fauust/docker-systemd:debian-sid
> | podman exec -it sys-test bash -c "echo \"deb http://deb.debian.org/debian-debug/ sid-debug main\" >> /etc/apt/sources.list && apt update && apt install -y mariadb-server-10.6-dbgsym"

I will give this a try. I'm curious what filesystem the container uses.

Thank you!
Dan
Urbana, IL

Daniel Lewart

unread,
Jun 3, 2022, 4:10:03 AMJun 3
to
Faustin,

DL> I will give this a try. I'm curious what filesystem the container uses.

$ podman exec -it sys-test findmnt -J /
{
"filesystems": [
{
"target": "/",
"source": "fuse-overlayfs",
"fstype": "fuse.fuse-overlayfs",
"options":
"rw,nodev,noatime,user_id=0,group_id=0,default_permissions,allow_other"
}
]
}

Dan
Urbana, Illinois

Daniel Lewart

unread,
Jun 6, 2022, 1:50:04 AMJun 6
to
tags + upstream

Faustin,

I found a much simpler way to demonstrate my problem with
a Debian Bullseye Live image and MariaDB generic Linux server:
* debian-live-11.3.0-amd64-standard.iso
* mariadb-10.6.8-linux-systemd-x86_64.tar.gz

I decided to file a new bug upstream:
[DEV-28751] mariadb-install-db fails writing ibdata1 on overlayfs
https://jira.mariadb.org/browse/MDEV-28751

Also, strace shows a difference between 10.5.16 (good) and 10.6.8 (fails).

Merci beaucoup,
Dan
Urbana, Illinois

Faustin Lammler

unread,
Jun 7, 2022, 4:00:04 AMJun 7
to
Control: forwarded -1 https://jira.mariadb.org/browse/MDEV-28751

Hi Daniel!
Thank you very much for your detailed report, your tests and to have
filled a jira issue!

I am now also subscribed to MDEV-28751 and will talk with Daniel Black
to see how we can debug this and maybe check if this is an interesting
step to add in our CI system.

Cheers!

--
Faustin
signature.asc
Reply all
Reply to author
Forward
0 new messages