Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
Bug#1009290: mariadb-server-10.6: Fails to start on live system
1,616 views
Skip to first unread message
Daniel Lewart
Apr 11, 2022, 1:50:03 AM4/11/22
to
Package: mariadb-server-10.6
Version: 1:10.6.7-3
Severity: important
Version: 1:10.6.7-3
Severity: important
Debian MariaDB Maintainers,
Live Build (--distribution testing) image from April 2022.
8 GB RAM, so should be plenty of disk space available.
$ sudo apt install mariadb-server-10.6-dbgsym
...
Setting up mariadb-server-10.6 (1:10.6.7-3) ...
Created symlink /etc/systemd/system/multi-user.target.wants/mariadb.service → /lib/systemd/system/mariadb.service.
Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142.
...
$ sudo journalctl | grep -i mariadb | cut -c58-
[See below]
Perhaps this is similar to:
#983002 - plocate: does not work with /var on overlayfs
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=983002
Please let me know any additional information that you could use.
Thank you!
Daniel Lewart
Urbana, Illinois
---
2022-04-10 22:53:02 0 [Warning] InnoDB: Retry attempts for writing partial data failed.
2022-04-10 22:53:02 0 [ERROR] InnoDB: Write to file ./ibdata1 failed at offset 0, 1048576 bytes should have been written, only 0 were written. Operating system error number 22. Check that your OS and file system support files of this size. Check also that the disk is not full or a disk quota exceeded.
2022-04-10 22:53:02 0 [ERROR] InnoDB: Error number 22 means 'Invalid argument'
2022-04-10 22:53:02 0 [ERROR] InnoDB: Could not set the file size of './ibdata1'. Probably out of disk space
2022-04-10 22:53:02 0 [ERROR] InnoDB: Database creation was aborted with error Generic error. You may need to delete the ibdata1 file before trying to start up again.
2022-04-10 22:53:02 0 [ERROR] InnoDB: Operating system error number 22 in a file operation.
2022-04-10 22:53:02 0 [ERROR] InnoDB: Error number 22 means 'Invalid argument'
2022-04-10 22:53:02 0 [ERROR] InnoDB: File (unknown): 'close' returned OS error 222. Cannot continue operation
220410 22:53:02 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Server version: 10.6.7-MariaDB-3
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=0
max_threads=153
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467957 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
2022-04-10 22:53:02 0 [ERROR] InnoDB: Operating system error number 9 in a file operation.
2022-04-10 22:53:02 0 [ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
2022-04-10 22:53:02 0 [ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation
Printing to addr2line failed
/usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x555d0aa77f0e]
/usr/sbin/mariadbd(handle_fatal_signal+0x478)[0x555d0a5c5268]
2022-04-10 22:53:02 0 [ERROR] InnoDB: Operating system error number 9 in a file operation.
2022-04-10 22:53:02 0 [ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
2022-04-10 22:53:02 0 [ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12200)[0x7f606ba96200]
2022-04-10 22:53:02 0 [ERROR] InnoDB: Operating system error number 9 in a file operation.
2022-04-10 22:53:02 0 [ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
2022-04-10 22:53:02 0 [ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x141)[0x7f606b57a8a1]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x112)[0x7f606b564546]
2022-04-10 22:53:02 0 [ERROR] InnoDB: Operating system error number 9 in a file operation.
2022-04-10 22:53:02 0 [ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
2022-04-10 22:53:02 0 [ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation
/usr/sbin/mariadbd(+0x63b8e9)[0x555d0a2638e9]
/usr/sbin/mariadbd(+0xcb66e5)[0x555d0a8de6e5]
/usr/sbin/mariadbd(+0xdbc308)[0x555d0a9e4308]
/usr/sbin/mariadbd(+0xdbc371)[0x555d0a9e4371]
/usr/sbin/mariadbd(+0xdbea20)[0x555d0a9e6a20]
/usr/sbin/mariadbd(+0xdbfb41)[0x555d0a9e7b41]
/usr/sbin/mariadbd(+0xd2585f)[0x555d0a94d85f]
/usr/sbin/mariadbd(+0xc64245)[0x555d0a88c245]
/usr/sbin/mariadbd(_Z24ha_initialize_handlertonP13st_plugin_int+0x7e)[0x555d0a5c839e]
/usr/sbin/mariadbd(+0x797a76)[0x555d0a3bfa76]
/usr/sbin/mariadbd(_Z11plugin_initPiPPci+0x828)[0x555d0a3c0a38]
/usr/sbin/mariadbd(+0x6c367a)[0x555d0a2eb67a]
/usr/sbin/mariadbd(_Z11mysqld_mainiPPc+0x404)[0x555d0a2f1224]
2022-04-10 22:53:02 0 [ERROR] InnoDB: Operating system error number 9 in a file operation.
2022-04-10 22:53:02 0 [ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
2022-04-10 22:53:02 0 [ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xcd)[0x7f606b5657fd]
2022-04-10 22:53:02 0 [ERROR] InnoDB: Operating system error number 9 in a file operation.
2022-04-10 22:53:02 0 [ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
2022-04-10 22:53:02 0 [ERROR] InnoDB: File (unknown): 'close' returned OS error 209. Cannot continue operation
/usr/sbin/mariadbd(_start+0x2a)[0x555d0a2e680a]
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /var/lib/mysql
Resource Limits:
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 0 bytes
Max resident set unlimited unlimited bytes
Max processes 29195 29195 processes
Max open files 32190 32190 files
Max locked memory 976257024 976257024 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 29195 29195 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
Core pattern: core
/usr/bin/mysql_install_db: line 541: 7508 Aborted "$mysqld_bootstrap" $defaults $defaults_group_suffix "$mysqld_opt" --bootstrap $silent_startup "--basedir=$basedir" "--datadir=$ldata" --log-warnings=0 --enforce-storage-engine="" "--plugin-dir=${plugindir}" $args --max_allowed_packet=8M --net_buffer_length=16K
Installation of system tables failed! Examine the logs in
/var/lib/mysql for more information.
The problem could be conflicting information in an external
my.cnf files. You can ignore these by doing:
shell> /usr/bin/mysql_install_db --defaults-file=~/.my.cnf
You can also try to start the mysqld daemon with:
shell> /usr/sbin/mariadbd --skip-grant-tables --general-log &
and use the command line tool /usr/bin/mariadb
to connect to the mysql database and look at the grant tables:
shell> /usr/bin/mysql -u root mysql
mysql> show tables;
Try 'mysqld --help' if you have problems with paths. Using
--general-log gives you a log in /var/lib/mysql that may be helpful.
The latest information about mysql_install_db is available at
https://mariadb.com/kb/en/installing-system-tables-mysql_install_db
You can find the latest source at https://downloads.mariadb.org and
the maria-discuss email list at https://launchpad.net/~maria-discuss
Please check all of the above before submitting a bug report
at https://mariadb.org/jira
###
Denys Nykula
May 18, 2022, 10:30:03 AM5/18/22
to
Hello. The apt install of mariadb-server-10.6 fails for me too,
with the same line "Could not execute systemctl" but different
journal messages. I removed the existing setup in /etc and /var,
purged dependencies, but install still fails. How do I debug?
systemd[1]: Starting MariaDB 10.6.7 database server...
[Note] /usr/sbin/mariadbd (server 10.6.7-MariaDB-3+b1) starting as process 98614 ...
[Note] InnoDB: The first data file './ibdata1' did not exist. A new tablespace will be created!
[Note] InnoDB: Compressed tables use zlib 1.2.11
[Note] InnoDB: Number of pools: 1
[Note] InnoDB: Using crc32 + pclmulqdq instructions
[Note] InnoDB: Using liburing
[Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
[Note] InnoDB: Completed initialization of buffer pool
[Note] InnoDB: Setting file './ibdata1' size to 12 MB. Physically writing the file full; Please wait ...
[Note] InnoDB: File './ibdata1' size is now 12 MB.
[Note] InnoDB: Setting log file ./ib_logfile101 size to 100663296 bytes
[Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
[Note] InnoDB: New log file created, LSN=10329
[Note] InnoDB: Doublewrite buffer not found: creating new
[Note] InnoDB: Doublewrite buffer created
[Note] InnoDB: 128 rollback segments are active.
[Note] InnoDB: Creating shared tablespace for temporary tables
[Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
[Note] InnoDB: File './ibtmp1' size is now 12 MB.
[Note] InnoDB: 10.6.7 started; log sequence number 0; transaction id 3
[Note] Plugin 'FEEDBACK' is disabled.
[ERROR] Could not open mysql.plugin table: "Table 'mysql.plugin' doesn't exist". Some plugins may be not loaded
[Warning] You need to use --log-bin to make --expire-logs-days or --binlog-expire-logs-seconds work.
[ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist
[Note] Server socket created on IP: '127.0.0.1'.
[ERROR] Fatal error: Can't open and lock privilege tables: Table 'mysql.db' doesn't exist
[ERROR] Aborting
Warning: Memory not freed: 280
systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: mariadb.service: Failed with result 'exit-code'.
systemd[1]: Failed to start MariaDB 10.6.7 database server.
with the same line "Could not execute systemctl" but different
journal messages. I removed the existing setup in /etc and /var,
purged dependencies, but install still fails. How do I debug?
systemd[1]: Starting MariaDB 10.6.7 database server...
[Note] /usr/sbin/mariadbd (server 10.6.7-MariaDB-3+b1) starting as process 98614 ...
[Note] InnoDB: The first data file './ibdata1' did not exist. A new tablespace will be created!
[Note] InnoDB: Compressed tables use zlib 1.2.11
[Note] InnoDB: Number of pools: 1
[Note] InnoDB: Using crc32 + pclmulqdq instructions
[Note] InnoDB: Using liburing
[Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
[Note] InnoDB: Completed initialization of buffer pool
[Note] InnoDB: Setting file './ibdata1' size to 12 MB. Physically writing the file full; Please wait ...
[Note] InnoDB: File './ibdata1' size is now 12 MB.
[Note] InnoDB: Setting log file ./ib_logfile101 size to 100663296 bytes
[Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
[Note] InnoDB: New log file created, LSN=10329
[Note] InnoDB: Doublewrite buffer not found: creating new
[Note] InnoDB: Doublewrite buffer created
[Note] InnoDB: 128 rollback segments are active.
[Note] InnoDB: Creating shared tablespace for temporary tables
[Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
[Note] InnoDB: File './ibtmp1' size is now 12 MB.
[Note] InnoDB: 10.6.7 started; log sequence number 0; transaction id 3
[Note] Plugin 'FEEDBACK' is disabled.
[ERROR] Could not open mysql.plugin table: "Table 'mysql.plugin' doesn't exist". Some plugins may be not loaded
[Warning] You need to use --log-bin to make --expire-logs-days or --binlog-expire-logs-seconds work.
[ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist
[Note] Server socket created on IP: '127.0.0.1'.
[ERROR] Fatal error: Can't open and lock privilege tables: Table 'mysql.db' doesn't exist
[ERROR] Aborting
Warning: Memory not freed: 280
systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: mariadb.service: Failed with result 'exit-code'.
systemd[1]: Failed to start MariaDB 10.6.7 database server.
Faustin Lammler
May 23, 2022, 10:40:03 AM5/23/22
to
Hi Denys,
I am not sure that the problem is the same, in your case it seems that
the datadir does not exist or need to be recreated.
If you are sure that nothing can be lost (or that you have backups), you
should try the following:
| sudo systemctl stop mariadb
| sudo rm -rf /var/lib/mysql/*
| sudo -u mysql mariadb-install-db
| Installing MariaDB/MySQL system tables in '/var/lib/mysql' ...
| 2022-05-23 14:18:53 0 [Warning] mariadbd: io_uring_queue_init() failed with errno 1
| 2022-05-23 14:18:53 0 [Warning] InnoDB: liburing disabled: falling back to innodb_use_native_aio=OFF
| 2022-05-23 14:18:53 0 [Warning] You need to use --log-bin to make --expire-logs-days or --binlog-expire-logs-seconds work.
| OK
| ...
| sudo systemctl start mariadb
Regards,
Denys Nykula <ve...@libre.net.ua>,
18/05/2022 – 17:16:00 (+0300):
> _______________________________________________
> pkg-mysql-maint mailing list
> pkg-mys...@alioth-lists.debian.net
> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-mysql-maint
--
Faustin
I am not sure that the problem is the same, in your case it seems that
the datadir does not exist or need to be recreated.
If you are sure that nothing can be lost (or that you have backups), you
should try the following:
| sudo systemctl stop mariadb
| sudo rm -rf /var/lib/mysql/*
| sudo -u mysql mariadb-install-db
| Installing MariaDB/MySQL system tables in '/var/lib/mysql' ...
| 2022-05-23 14:18:53 0 [Warning] mariadbd: io_uring_queue_init() failed with errno 1
| 2022-05-23 14:18:53 0 [Warning] InnoDB: liburing disabled: falling back to innodb_use_native_aio=OFF
| 2022-05-23 14:18:53 0 [Warning] You need to use --log-bin to make --expire-logs-days or --binlog-expire-logs-seconds work.
| OK
| ...
| sudo systemctl start mariadb
Regards,
Denys Nykula <ve...@libre.net.ua>,
18/05/2022 – 17:16:00 (+0300):
> pkg-mysql-maint mailing list
> pkg-mys...@alioth-lists.debian.net
> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-mysql-maint
--
Faustin
Faustin Lammler
May 23, 2022, 10:40:03 AM5/23/22
to
Hi Daniel!
Thanks for your detailed report.
I am not able to reproduce this with the latest
mariadb-server-10.6-dbgsym version (10.6.8).
Can you try with the latest version and tell me?
Also, can you describe a bit better your setup? If this is a filesystem
related issue, I would like to try to reproduce it myself.
I have tested with a systemd container that I have created for these
kind of testing, here are the commands (using podman):
| podman run --name sys-test --rm -d fauust/docker-systemd:debian-sid
| podman exec -it sys-test bash -c "echo \"deb http://deb.debian.org/debian-debug/ sid-debug main\" >> /etc/apt/sources.list && apt update && apt install -y mariadb-server-10.6-dbgsym"
Cheers!
Daniel Lewart <lew...@gmail.com>,
11/04/2022 – 00:20:00 (-0500):
Thanks for your detailed report.
I am not able to reproduce this with the latest
mariadb-server-10.6-dbgsym version (10.6.8).
Can you try with the latest version and tell me?
Also, can you describe a bit better your setup? If this is a filesystem
related issue, I would like to try to reproduce it myself.
I have tested with a systemd container that I have created for these
kind of testing, here are the commands (using podman):
| podman run --name sys-test --rm -d fauust/docker-systemd:debian-sid
| podman exec -it sys-test bash -c "echo \"deb http://deb.debian.org/debian-debug/ sid-debug main\" >> /etc/apt/sources.list && apt update && apt install -y mariadb-server-10.6-dbgsym"
Cheers!
Daniel Lewart <lew...@gmail.com>,
11/04/2022 – 00:20:00 (-0500):
Denys Nykula
May 23, 2022, 11:00:03 AM5/23/22
to
> > sudo rm -rf /var/lib/mysql/*
> > sudo -u mysql mariadb-install-db
Thanks Faustin, something was fixed between 10.6.7-3 and 10.6.8-1 that
> > sudo -u mysql mariadb-install-db
made mariadb-install-db succeed for me since today morning.
With the March packages it kept failing every time during or after the
InnoDB initialization (as I mentioned, I started clean every time,
removing the mysql directories in /etc and /var).
Daniel Lewart
May 26, 2022, 12:30:03 AM5/26/22
to
Faustin,
> I am not able to reproduce this with the latest
> mariadb-server-10.6-dbgsym version (10.6.8).
> Can you try with the latest version and tell me?
Yes, I tried 10.6.8-1 with the same result.
> Also, can you describe a bit better your setup? If this is a filesystem
> related issue, I would like to try to reproduce it myself.
Yes. This is a simplified version of my setup ...
1) sudo apt -y install apt-cacher-ng live-build
2) I used Live Build to create a live sid image:
#!/bin/sh
cd /dev/shm
sudo mount /dev/shm -odev,exec,remount,size=3G
lb config --apt-http-proxy http://localhost:3142 -d sid
echo '! Packages Priority standard' \
> config/package-lists/standard.list.chroot
echo 'deb http://deb.debian.org/debian-debug sid-debug main' \
> config/archives/live.list.binary
sudo lb build
3) qemu-system-x86_64 -enable-kvm -cdrom /dev/shm/live-image-amd64.hybrid.iso \
-cpu host -m 4G -smp 2
4) In the guest, it didn't matter what I tried:
sudo apt -y install mariadb-server
sudo apt -y install mariadb-server-10.6-dbgsym
mariadb-server-core-10.6-dbgsym
I'm sure it is an overlay filesystem issue:
$ sudo ls -l /var/lib/mysql/ibdata1
-rw-rw---- 1 mysql mysql 0 May 25 12:34 /var/lib/mysql/ibdata1
$ df -hT /var/lib/mysql
Filesystem Type Size Used Avail Use% Mounted on
overlay overlay 2.0G 355M 1.6G 19% /
$ findmnt -J /
{
"filesystems": [
{
"target": "/",
"source": "overlay",
"fstype": "overlay",
"options":
"rw,noatime,lowerdir=/run/live/rootfs/filesystem.squashfs/,upperdir=/run/live/overlay/rw,workdir=/run/live/overlay/work"
}
]
Thank you!
Dan
Urbana, IL
> I am not able to reproduce this with the latest
> mariadb-server-10.6-dbgsym version (10.6.8).
> Can you try with the latest version and tell me?
> Also, can you describe a bit better your setup? If this is a filesystem
> related issue, I would like to try to reproduce it myself.
1) sudo apt -y install apt-cacher-ng live-build
2) I used Live Build to create a live sid image:
#!/bin/sh
cd /dev/shm
sudo mount /dev/shm -odev,exec,remount,size=3G
lb config --apt-http-proxy http://localhost:3142 -d sid
echo '! Packages Priority standard' \
> config/package-lists/standard.list.chroot
echo 'deb http://deb.debian.org/debian-debug sid-debug main' \
> config/archives/live.list.binary
sudo lb build
3) qemu-system-x86_64 -enable-kvm -cdrom /dev/shm/live-image-amd64.hybrid.iso \
-cpu host -m 4G -smp 2
4) In the guest, it didn't matter what I tried:
sudo apt -y install mariadb-server
sudo apt -y install mariadb-server-10.6-dbgsym
mariadb-server-core-10.6-dbgsym
I'm sure it is an overlay filesystem issue:
$ sudo ls -l /var/lib/mysql/ibdata1
-rw-rw---- 1 mysql mysql 0 May 25 12:34 /var/lib/mysql/ibdata1
$ df -hT /var/lib/mysql
Filesystem Type Size Used Avail Use% Mounted on
overlay overlay 2.0G 355M 1.6G 19% /
$ findmnt -J /
{
"filesystems": [
{
"target": "/",
"source": "overlay",
"fstype": "overlay",
"options":
"rw,noatime,lowerdir=/run/live/rootfs/filesystem.squashfs/,upperdir=/run/live/overlay/rw,workdir=/run/live/overlay/work"
}
]
}
> I have tested with a systemd container that I have created for these
> kind of testing, here are the commands (using podman):
>
> | podman run --name sys-test --rm -d fauust/docker-systemd:debian-sid
> | podman exec -it sys-test bash -c "echo \"deb http://deb.debian.org/debian-debug/ sid-debug main\" >> /etc/apt/sources.list && apt update && apt install -y mariadb-server-10.6-dbgsym"
I will give this a try. I'm curious what filesystem the container uses.
> I have tested with a systemd container that I have created for these
> kind of testing, here are the commands (using podman):
>
> | podman run --name sys-test --rm -d fauust/docker-systemd:debian-sid
> | podman exec -it sys-test bash -c "echo \"deb http://deb.debian.org/debian-debug/ sid-debug main\" >> /etc/apt/sources.list && apt update && apt install -y mariadb-server-10.6-dbgsym"
Thank you!
Dan
Urbana, IL
Daniel Lewart
Jun 3, 2022, 4:10:03 AM6/3/22
to
Faustin,
DL> I will give this a try. I'm curious what filesystem the container uses.
$ podman exec -it sys-test findmnt -J /
{
"filesystems": [
{
"target": "/",
"source": "fuse-overlayfs",
"fstype": "fuse.fuse-overlayfs",
"options":
"rw,nodev,noatime,user_id=0,group_id=0,default_permissions,allow_other"
}
]
}
Dan
Urbana, Illinois
DL> I will give this a try. I'm curious what filesystem the container uses.
$ podman exec -it sys-test findmnt -J /
{
"filesystems": [
{
"target": "/",
"source": "fuse-overlayfs",
"fstype": "fuse.fuse-overlayfs",
"options":
"rw,nodev,noatime,user_id=0,group_id=0,default_permissions,allow_other"
}
]
}
Dan
Urbana, Illinois
Daniel Lewart
Jun 6, 2022, 1:50:04 AM6/6/22
to
tags + upstream
Faustin,
I found a much simpler way to demonstrate my problem with
a Debian Bullseye Live image and MariaDB generic Linux server:
* debian-live-11.3.0-amd64-standard.iso
* mariadb-10.6.8-linux-systemd-x86_64.tar.gz
I decided to file a new bug upstream:
[DEV-28751] mariadb-install-db fails writing ibdata1 on overlayfs
https://jira.mariadb.org/browse/MDEV-28751
Also, strace shows a difference between 10.5.16 (good) and 10.6.8 (fails).
Merci beaucoup,
Dan
Urbana, Illinois
Faustin,
I found a much simpler way to demonstrate my problem with
a Debian Bullseye Live image and MariaDB generic Linux server:
* debian-live-11.3.0-amd64-standard.iso
* mariadb-10.6.8-linux-systemd-x86_64.tar.gz
I decided to file a new bug upstream:
[DEV-28751] mariadb-install-db fails writing ibdata1 on overlayfs
https://jira.mariadb.org/browse/MDEV-28751
Also, strace shows a difference between 10.5.16 (good) and 10.6.8 (fails).
Merci beaucoup,
Dan
Urbana, Illinois
Faustin Lammler
Jun 7, 2022, 4:00:04 AM6/7/22
to
Control: forwarded -1 https://jira.mariadb.org/browse/MDEV-28751
Hi Daniel!
Thank you very much for your detailed report, your tests and to have
filled a jira issue!
I am now also subscribed to MDEV-28751 and will talk with Daniel Black
to see how we can debug this and maybe check if this is an interesting
step to add in our CI system.
Cheers!
--
Faustin
Hi Daniel!
Thank you very much for your detailed report, your tests and to have
filled a jira issue!
I am now also subscribed to MDEV-28751 and will talk with Daniel Black
to see how we can debug this and maybe check if this is an interesting
step to add in our CI system.
Cheers!
--
Faustin
Otto Kekäläinen
Jan 21, 2023, 7:10:05 PM1/21/23
to
Hi!
I don't see anything new in upstream
https://jira.mariadb.org/browse/MDEV-28751 about this.
However we do have MariaDB 10.11.1 in Debian now. Maybe you Daniel can
for the sake of it check if the behaviour is still same on 10.11?
I would personally also be keen to figure out how to make the MariaDB
datadir be able to run off overlayfs or something similar so I can
implement my long time idea to have shadow upgrades where the old
mariadbd and old data is kept running and intact, and every new
version mariadbd would first run off overlayfs with real data to
simulate the upgrade before it is done for real.
I don't see anything new in upstream
https://jira.mariadb.org/browse/MDEV-28751 about this.
However we do have MariaDB 10.11.1 in Debian now. Maybe you Daniel can
for the sake of it check if the behaviour is still same on 10.11?
I would personally also be keen to figure out how to make the MariaDB
datadir be able to run off overlayfs or something similar so I can
implement my long time idea to have shadow upgrades where the old
mariadbd and old data is kept running and intact, and every new
version mariadbd would first run off overlayfs with real data to
simulate the upgrade before it is done for real.
Daniel Lewart
Jan 30, 2023, 3:20:04 AM1/30/23
to
Otto, et al,
> I don't see anything new in upstream
> https://jira.mariadb.org/browse/MDEV-28751 about this.
> However we do have MariaDB 10.11.1 in Debian now. Maybe you Daniel can
> for the sake of it check if the behaviour is still same on 10.11?
Unfortunately, still the same.
I created a live standard image with CODENAME=unstable:
$ uname -a
Linux debian 6.1.0-2-amd64 #1 SMP PREEMPT_DYNAMIC
Debian 6.1.7-1 (2023-01-18) x86_64 GNU/Linux
Then I did the following:
$ sudo apt -y install mariadb-server
Output included:
Setting up mariadb-server (1:10.11.1-1) ...
Created symlink
/etc/systemd/system/multi-user.target.wants/mariadb.service →
/lib/systemd/system/mariadb.service.
Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 145.
Below is the output from the systemd journal.
In the upstream bug:
Daniel Black added a comment - 2022-06-07 04:50
I'll be just pushing this to the kernel (overlayfs) developers.
I don't know what happened with that.
The good news is that Arnaud R's upstream "innodb_flush_method = fsync"
workaround still works.
Sorry!
2023-01-30 0:54:27 0 [Warning] InnoDB: Retry attempts for writing
partial data failed.
2023-01-30 0:54:27 0 [ERROR] InnoDB: Write to file ./ibdata1 failed
2023-01-30 0:54:27 0 [ERROR] InnoDB: Could not set the file size of
in a file operation.
2023-01-30 0:54:28 0 [ERROR] InnoDB: Error number 22 means 'Invalid argument'
2023-01-30 0:54:28 0 [ERROR] InnoDB: File (unknown): 'close' returned
in a file operation.
2023-01-30 0:54:28 0 [ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
2023-01-30 0:54:28 0 [ERROR] InnoDB: File (unknown): 'close' returned
/usr/sbin/mariadbd(handle_fatal_signal+0x409)[0x55ba61741c69]
2023-01-30 0:54:28 0 [ERROR] InnoDB: Operating system error number 9
in a file operation.
2023-01-30 0:54:28 0 [ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
2023-01-30 0:54:28 0 [ERROR] InnoDB: File (unknown): 'close' returned
/lib/x86_64-linux-gnu/libc.so.6(+0x8accc)[0x7f028b3e1ccc]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x12)[0x7f028b392ef2]
/lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7f028b37d472]
2023-01-30 0:54:28 0 [ERROR] InnoDB: Operating system error number 9
in a file operation.
2023-01-30 0:54:28 0 [ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
2023-01-30 0:54:28 0 [ERROR] InnoDB: File (unknown): 'close' returned
/usr/sbin/mariadbd(+0xd0a775)[0x55ba61a38775]
/usr/sbin/mariadbd(+0xe140c8)[0x55ba61b420c8]
/usr/sbin/mariadbd(+0xe14131)[0x55ba61b42131]
/usr/sbin/mariadbd(+0xe16060)[0x55ba61b44060]
/usr/sbin/mariadbd(+0xe17041)[0x55ba61b45041]
/usr/sbin/mariadbd(+0xd7bbae)[0x55ba61aa9bae]
/usr/sbin/mariadbd(+0xcb7545)[0x55ba619e5545]
/usr/sbin/mariadbd(_Z24ha_initialize_handlertonP13st_plugin_int+0x7e)[0x55ba61744e6e]
/usr/sbin/mariadbd(+0x802167)[0x55ba61530167]
/usr/sbin/mariadbd(_Z11plugin_initPiPPci+0x85a)[0x55ba6153112a]
/usr/sbin/mariadbd(+0x6f42e0)[0x55ba614222e0]
/usr/sbin/mariadbd(_Z11mysqld_mainiPPc+0x41d)[0x55ba61427e7d]
2023-01-30 0:54:28 0 [ERROR] InnoDB: Operating system error number 9
in a file operation.
2023-01-30 0:54:28 0 [ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
2023-01-30 0:54:28 0 [ERROR] InnoDB: File (unknown): 'close' returned
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85)[0x7f028b37e245]
2023-01-30 0:54:28 0 [ERROR] InnoDB: Operating system error number 9
in a file operation.
2023-01-30 0:54:28 0 [ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
2023-01-30 0:54:28 0 [ERROR] InnoDB: File (unknown): 'close' returned
Max open files 32184 32184 files
Max locked memory 258789376 258789376 bytes
(debian...@lists.debian.org) (gcc-12 (Debian 12.2.0-14) 12.2.0,
GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC Debian
6.1.7-1 (2023-01-18)
/usr/bin/mysql_install_db: line 554: 2096 Aborted
> I don't see anything new in upstream
> https://jira.mariadb.org/browse/MDEV-28751 about this.
> However we do have MariaDB 10.11.1 in Debian now. Maybe you Daniel can
> for the sake of it check if the behaviour is still same on 10.11?
I created a live standard image with CODENAME=unstable:
$ uname -a
Linux debian 6.1.0-2-amd64 #1 SMP PREEMPT_DYNAMIC
Debian 6.1.7-1 (2023-01-18) x86_64 GNU/Linux
Then I did the following:
$ sudo apt -y install mariadb-server
Output included:
Setting up mariadb-server (1:10.11.1-1) ...
Created symlink
/etc/systemd/system/multi-user.target.wants/mariadb.service →
/lib/systemd/system/mariadb.service.
Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 145.
Below is the output from the systemd journal.
In the upstream bug:
Daniel Black added a comment - 2022-06-07 04:50
I'll be just pushing this to the kernel (overlayfs) developers.
I don't know what happened with that.
The good news is that Arnaud R's upstream "innodb_flush_method = fsync"
workaround still works.
Sorry!
Daniel Lewart
Urbana, Illinois
---
$ sudo journalctl -t mariadb-server.postinst | cut -c55-
Urbana, Illinois
---
2023-01-30 0:54:27 0 [Warning] InnoDB: Retry attempts for writing
partial data failed.
2023-01-30 0:54:27 0 [ERROR] InnoDB: Write to file ./ibdata1 failed
at offset 0, 1048576 bytes should have been written, only 0 were
written. Operating system error number 22. Check that your OS and file
system support files of this size. Check also that the disk is not
full or a disk quota exceeded.
2023-01-30 0:54:27 0 [ERROR] InnoDB: Error number 22 means 'Invalid argument'
written. Operating system error number 22. Check that your OS and file
system support files of this size. Check also that the disk is not
full or a disk quota exceeded.
2023-01-30 0:54:27 0 [ERROR] InnoDB: Could not set the file size of
'./ibdata1'. Probably out of disk space
2023-01-30 0:54:27 0 [ERROR] InnoDB: Database creation was aborted
with error Generic error. You may need to delete the ibdata1 file
before trying to start up again.
2023-01-30 0:54:28 0 [ERROR] InnoDB: Operating system error number 22
before trying to start up again.
in a file operation.
2023-01-30 0:54:28 0 [ERROR] InnoDB: Error number 22 means 'Invalid argument'
2023-01-30 0:54:28 0 [ERROR] InnoDB: File (unknown): 'close' returned
OS error 222. Cannot continue operation
230130 0:54:28 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Server version: 10.11.1-MariaDB-1
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=0
max_threads=153
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads =
468018 K bytes of memory
read_buffer_size=131072
max_used_connections=0
max_threads=153
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads =
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
2023-01-30 0:54:28 0 [ERROR] InnoDB: Operating system error number 9
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
in a file operation.
2023-01-30 0:54:28 0 [ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
2023-01-30 0:54:28 0 [ERROR] InnoDB: File (unknown): 'close' returned
OS error 209. Cannot continue operation
Printing to addr2line failed
/usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x55ba61bd403e]
Printing to addr2line failed
/usr/sbin/mariadbd(handle_fatal_signal+0x409)[0x55ba61741c69]
2023-01-30 0:54:28 0 [ERROR] InnoDB: Operating system error number 9
in a file operation.
2023-01-30 0:54:28 0 [ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
2023-01-30 0:54:28 0 [ERROR] InnoDB: File (unknown): 'close' returned
OS error 209. Cannot continue operation
/lib/x86_64-linux-gnu/libc.so.6(+0x3bf90)[0x7f028b392f90]
/lib/x86_64-linux-gnu/libc.so.6(+0x8accc)[0x7f028b3e1ccc]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x12)[0x7f028b392ef2]
/lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7f028b37d472]
2023-01-30 0:54:28 0 [ERROR] InnoDB: Operating system error number 9
in a file operation.
2023-01-30 0:54:28 0 [ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
2023-01-30 0:54:28 0 [ERROR] InnoDB: File (unknown): 'close' returned
OS error 209. Cannot continue operation
/usr/sbin/mariadbd(+0x6692e1)[0x55ba613972e1]
/usr/sbin/mariadbd(+0xd0a775)[0x55ba61a38775]
/usr/sbin/mariadbd(+0xe140c8)[0x55ba61b420c8]
/usr/sbin/mariadbd(+0xe14131)[0x55ba61b42131]
/usr/sbin/mariadbd(+0xe16060)[0x55ba61b44060]
/usr/sbin/mariadbd(+0xe17041)[0x55ba61b45041]
/usr/sbin/mariadbd(+0xd7bbae)[0x55ba61aa9bae]
/usr/sbin/mariadbd(+0xcb7545)[0x55ba619e5545]
/usr/sbin/mariadbd(_Z24ha_initialize_handlertonP13st_plugin_int+0x7e)[0x55ba61744e6e]
/usr/sbin/mariadbd(+0x802167)[0x55ba61530167]
/usr/sbin/mariadbd(_Z11plugin_initPiPPci+0x85a)[0x55ba6153112a]
/usr/sbin/mariadbd(+0x6f42e0)[0x55ba614222e0]
/usr/sbin/mariadbd(_Z11mysqld_mainiPPc+0x41d)[0x55ba61427e7d]
2023-01-30 0:54:28 0 [ERROR] InnoDB: Operating system error number 9
in a file operation.
2023-01-30 0:54:28 0 [ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
2023-01-30 0:54:28 0 [ERROR] InnoDB: File (unknown): 'close' returned
OS error 209. Cannot continue operation
/lib/x86_64-linux-gnu/libc.so.6(+0x2718a)[0x7f028b37e18a]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85)[0x7f028b37e245]
2023-01-30 0:54:28 0 [ERROR] InnoDB: Operating system error number 9
in a file operation.
2023-01-30 0:54:28 0 [ERROR] InnoDB: Error number 9 means 'Bad file descriptor'
2023-01-30 0:54:28 0 [ERROR] InnoDB: File (unknown): 'close' returned
OS error 209. Cannot continue operation
/usr/sbin/mariadbd(_start+0x21)[0x55ba6141ceb1]
The manual page at
https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/
contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /var/lib/mysql
Resource Limits:
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 0 bytes
Max resident set unlimited unlimited bytes
Max processes 7693 7693 processes
https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/
contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /var/lib/mysql
Resource Limits:
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 0 bytes
Max resident set unlimited unlimited bytes
Max open files 32184 32184 files
Max locked memory 258789376 258789376 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 7693 7693 signals
Max file locks unlimited unlimited locks
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
Core pattern: core
Kernel version: Linux version 6.1.0-2-amd64
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
Core pattern: core
(debian...@lists.debian.org) (gcc-12 (Debian 12.2.0-14) 12.2.0,
GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC Debian
6.1.7-1 (2023-01-18)
/usr/bin/mysql_install_db: line 554: 2096 Aborted
Daniel Lewart
Jan 31, 2023, 2:40:05 AM1/31/23
to
Otto, et al,
How about temporarily inserting something like the implementation below of
Arnaud R's workaround somewhere (where?) in mariadb-server(-10.6).postinst?
Thank you!
Daniel Lewart
Urbana, Illinois
---
# Temporary workaround which should be removed after upstream fixes
# https://jira.mariadb.org/browse/MDEV-28751
fstype=$(findmnt -n -o FSTYPE -T /var/lib/mysql)
if [ "$fstype" = overlay ]; then
cat <<- EOF > "${mysql_cfgdir:-/etc/mysql}/mariadb.conf.d/90-live.cnf"
[mysqld]
innodb_flush_method = fsync
EOF
fi
How about temporarily inserting something like the implementation below of
Arnaud R's workaround somewhere (where?) in mariadb-server(-10.6).postinst?
Thank you!
Daniel Lewart
Urbana, Illinois
---
# https://jira.mariadb.org/browse/MDEV-28751
fstype=$(findmnt -n -o FSTYPE -T /var/lib/mysql)
if [ "$fstype" = overlay ]; then
cat <<- EOF > "${mysql_cfgdir:-/etc/mysql}/mariadb.conf.d/90-live.cnf"
[mysqld]
innodb_flush_method = fsync
EOF
fi
Otto Kekäläinen
Jan 31, 2023, 3:20:05 AM1/31/23
to
But I don't think this should be part of the maintainer scripts, maybe
something else? Feel free to open MR if you have a concrete
suggestion.
https://salsa.debian.org/mariadb-team/mariadb-server/-/wikis/Contributing-to-MariaDB-packaging-in-Debian
It would also be cool to have in the salsa-ci.yml a job that attempts
to use and run off overlayfs.
Antoni Villalonga
May 31, 2023, 10:10:05 AM5/31/23
to
Hi,
I've faced same problem on a fresh Bookworm live system (mariadb-server
1:10.11.2-1).
As a workaround I've mounted /var/lib/mysql as a ext4 filesystem.
FYI: It works fine on Bullseye.
Kind regards,
--
Antoni Villalonga
https://friki.cat/
I've faced same problem on a fresh Bookworm live system (mariadb-server
1:10.11.2-1).
As a workaround I've mounted /var/lib/mysql as a ext4 filesystem.
FYI: It works fine on Bullseye.
Kind regards,
--
Antoni Villalonga
https://friki.cat/
Witold Baryluk
Jun 24, 2023, 3:40:05 PM6/24/23
to
Package: mariadb-server
Version: 1:10.11.3-1
Followup-For: Bug #1009290
X-Debbugs-Cc: witold....@gmail.com
Still broken in 10.11.3.
This is fresh installation, zero modifications.
Jun 24 19:14:24 debian systemd[1]: Starting mariadb.service - MariaDB 10.11.3 database server...
Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [Note] Starting MariaDB 10.11.3-MariaDB-1 source revision as process 2685
Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [Note] InnoDB: Compressed tables use zlib 1.2.13
Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [Note] InnoDB: Number of transaction pools: 1
Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [Note] InnoDB: Using liburing
Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [Note] InnoDB: Completed initialization of buffer pool
Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [ERROR] InnoDB: The Auto-extending data file './ibdata1' is of a different size 0 pages than specified by innodb_data_file_path
Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [ERROR] InnoDB: Operating system error number 22 in a file operation.
Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [ERROR] InnoDB: Error number 22 means 'Invalid argument'
Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/
Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [ERROR] InnoDB: File (unknown): 'close' returned OS error 222. Cannot continue operation
Jun 24 19:14:25 debian mariadbd[2685]: 230624 19:14:25 [ERROR] mysqld got signal 6 ;
Jun 24 19:14:25 debian mariadbd[2685]: This could be because you hit a bug. It is also possible that this binary
Jun 24 19:14:25 debian mariadbd[2685]: or one of the libraries it was linked against is corrupt, improperly built,
Jun 24 19:14:25 debian mariadbd[2685]: or misconfigured. This error can also be caused by malfunctioning hardware.
Jun 24 19:14:25 debian mariadbd[2685]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
Jun 24 19:14:25 debian mariadbd[2685]: We will try our best to scrape up some info that will hopefully help
Jun 24 19:14:25 debian mariadbd[2685]: diagnose the problem, but since we have already crashed,
Jun 24 19:14:25 debian mariadbd[2685]: something is definitely wrong and this may fail.
Jun 24 19:14:25 debian mariadbd[2685]: Server version: 10.11.3-MariaDB-1 source revision:
Jun 24 19:14:25 debian mariadbd[2685]: key_buffer_size=134217728
Jun 24 19:14:25 debian mariadbd[2685]: read_buffer_size=131072
Jun 24 19:14:25 debian mariadbd[2685]: max_used_connections=0
Jun 24 19:14:25 debian mariadbd[2685]: max_threads=153
Jun 24 19:14:25 debian mariadbd[2685]: thread_count=0
Jun 24 19:14:25 debian mariadbd[2685]: It is possible that mysqld could use up to
Jun 24 19:14:25 debian mariadbd[2685]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 468022 K bytes of memory
Jun 24 19:14:25 debian mariadbd[2685]: Hope that's ok; if not, decrease some variables in the equation.
Jun 24 19:14:25 debian mariadbd[2685]: Thread pointer: 0x0
Jun 24 19:14:25 debian mariadbd[2685]: Attempting backtrace. You can use the following information to find out
Jun 24 19:14:25 debian mariadbd[2685]: where mysqld died. If you see no messages after this, something went
Jun 24 19:14:25 debian mariadbd[2685]: terribly wrong...
Jun 24 19:14:25 debian mariadbd[2685]: stack_bottom = 0x0 thread_stack 0x49000
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x559932b6ecfe]
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(handle_fatal_signal+0x409)[0x5599326dc7c9]
Jun 24 19:14:25 debian mariadbd[2685]: libc_sigaction.c:0(__restore_rt)[0x7efc03891f90]
Jun 24 19:14:25 debian mariadbd[2685]: nptl/pthread_kill.c:44(__pthread_kill_implementation)[0x7efc038e0ccc]
Jun 24 19:14:25 debian mariadbd[2685]: posix/raise.c:27(__GI_raise)[0x7efc03891ef2]
Jun 24 19:14:25 debian mariadbd[2685]: stdlib/abort.c:81(__GI_abort)[0x7efc0387c472]
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(+0x67483c)[0x55993232883c]
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(+0xd2a515)[0x5599329de515]
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(+0xe29078)[0x559932add078]
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(+0xe2c5cf)[0x559932ae05cf]
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(+0xe2c6bd)[0x559932ae06bd]
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(+0x685d85)[0x559932339d85]
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(+0xcd9051)[0x55993298d051]
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(_Z24ha_initialize_handlertonP13st_plugin_int+0x7e)[0x5599326dfafe]
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(+0x80f257)[0x5599324c3257]
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(_Z11plugin_initPiPPci+0x85a)[0x5599324c421a]
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(+0x7008f7)[0x5599323b48f7]
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(_Z11mysqld_mainiPPc+0x41d)[0x5599323ba31d]
Jun 24 19:14:25 debian mariadbd[2685]: x86/libc-start.c:74(__libc_start_call_main)[0x7efc0387d18a]
Jun 24 19:14:25 debian mariadbd[2685]: csu/libc-start.c:128(call_init)[0x7efc0387d245]
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(_start+0x21)[0x5599323af541]
Jun 24 19:14:25 debian mariadbd[2685]: The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
Jun 24 19:14:25 debian mariadbd[2685]: information that should help you find out what is causing the crash.
Jun 24 19:14:25 debian mariadbd[2685]: Writing a core file...
Jun 24 19:14:25 debian mariadbd[2685]: Working directory at /var/lib/mysql
Jun 24 19:14:25 debian mariadbd[2685]: Resource Limits:
Jun 24 19:14:25 debian mariadbd[2685]: Limit Soft Limit Hard Limit Units
Jun 24 19:14:25 debian mariadbd[2685]: Max cpu time unlimited unlimited seconds
Jun 24 19:14:25 debian mariadbd[2685]: Max file size unlimited unlimited bytes
Jun 24 19:14:25 debian mariadbd[2685]: Max data size unlimited unlimited bytes
Jun 24 19:14:25 debian mariadbd[2685]: Max stack size 8388608 unlimited bytes
Jun 24 19:14:25 debian mariadbd[2685]: Max core file size 0 unlimited bytes
Jun 24 19:14:25 debian mariadbd[2685]: Max resident set unlimited unlimited bytes
Jun 24 19:14:25 debian mariadbd[2685]: Max processes 1030035 1030035 processes
Jun 24 19:14:25 debian mariadbd[2685]: Max open files 32768 32768 files
Jun 24 19:14:25 debian mariadbd[2685]: Max locked memory 524288 524288 bytes
Jun 24 19:14:25 debian mariadbd[2685]: Max address space unlimited unlimited bytes
Jun 24 19:14:25 debian mariadbd[2685]: Max file locks unlimited unlimited locks
Jun 24 19:14:25 debian mariadbd[2685]: Max pending signals 1030035 1030035 signals
Jun 24 19:14:25 debian mariadbd[2685]: Max msgqueue size 819200 819200 bytes
Jun 24 19:14:25 debian mariadbd[2685]: Max nice priority 0 0
Jun 24 19:14:25 debian mariadbd[2685]: Max realtime priority 0 0
Jun 24 19:14:25 debian mariadbd[2685]: Max realtime timeout unlimited unlimited us
Jun 24 19:14:25 debian mariadbd[2685]: Core pattern: |/lib/systemd/systemd-coredump %P %u %g %s %t 9223372036854775808 %h
Jun 24 19:14:25 debian mariadbd[2685]: Kernel version: Linux version 6.1.0-9-amd64 (debian...@lists.debian.org) (gcc-12 (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC Debian 6.1.27-1 (2023-05-08)
Jun 24 19:14:25 debian systemd[1]: mariadb.service: Main process exited, code=dumped, status=6/ABRT
Jun 24 19:14:25 debian systemd[1]: mariadb.service: Failed with result 'core-dump'.
Jun 24 19:14:25 debian systemd[1]: Failed to start mariadb.service - MariaDB 10.11.3 database server.
Maybe mariadb does not like it being on overlayfs.
Here are my mounts:
user@debian:~$ mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=131844608k,nr_inodes=32961152,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=26378644k,mode=755,inode64)
/dev/sda1 on /run/live/medium type iso9660 (ro,noatime,nojoliet,check=s,map=n,blocksize=2048,iocharset=utf8)
/dev/loop0 on /run/live/rootfs/filesystem.squashfs type squashfs (ro,noatime,errors=continue)
tmpfs on /run/live/overlay type tmpfs (rw,noatime,size=131893220k,mode=755,inode64)
overlay on / type overlay (rw,noatime,lowerdir=/run/live/rootfs/filesystem.squashfs/,upperdir=/run/live/overlay/rw,workdir=/run/live/overlay/work,redirect_dir=on)
tmpfs on /usr/lib/live/mount type tmpfs (rw,nosuid,nodev,noexec,relatime,size=26378644k,mode=755,inode64)
/dev/sda1 on /usr/lib/live/mount/medium type iso9660 (ro,noatime,nojoliet,check=s,map=n,blocksize=2048,iocharset=utf8)
/dev/loop0 on /usr/lib/live/mount/rootfs/filesystem.squashfs type squashfs (ro,noatime,errors=continue)
tmpfs on /usr/lib/live/mount/overlay type tmpfs (rw,noatime,size=131893220k,mode=755,inode64)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=29782)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
ramfs on /run/credentials/systemd-sysctl.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-tmpfiles-setup-dev.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,relatime,inode64)
ramfs on /run/credentials/systemd-tmpfiles-setup.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
tracefs on /sys/kernel/debug/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=26378640k,nr_inodes=6594660,mode=700,uid=1000,gid=1000,inode64)
gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)
portal on /run/user/1000/doc type fuse.portal (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)
-- System Information:
Debian Release: 12.0
APT prefers stable-security
APT policy: (500, 'stable-security'), (500, 'stable-debug'), (500, 'bookworm'), (500, 'stable')
Architecture: amd64 (x86_64)
Foreign Architectures: i386
Kernel: Linux 6.1.0-9-amd64 (SMP w/32 CPU threads; PREEMPT)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages mariadb-server depends on:
ii adduser 3.134
ii debconf [debconf-2.0] 1.5.82
ii galera-4 26.4.13-1
ii gawk 1:5.2.1-2
ii iproute2 6.1.0-3
ii libc6 2.36-9
ii libdbi-perl 1.643-4
ii libpam0g 1.5.2-6
ii libssl3 3.0.9-1
ii libstdc++6 12.2.0-14
ii lsof 4.95.0-1
ii mariadb-client 1:10.11.3-1
ii mariadb-common 1:10.11.3-1
ii mariadb-server-core 1:10.11.3-1
ii passwd 1:4.13+dfsg1-1+b1
ii perl 5.36.0-7
ii procps 2:4.0.2-3
ii psmisc 23.6-1
ii rsync 3.2.7-1
ii socat 1.7.4.4-2
ii zlib1g 1:1.2.13.dfsg-1
Versions of packages mariadb-server recommends:
ii libhtml-template-perl 2.97-2
ii mariadb-plugin-provider-bzip2 1:10.11.3-1
ii mariadb-plugin-provider-lz4 1:10.11.3-1
ii mariadb-plugin-provider-lzma 1:10.11.3-1
ii mariadb-plugin-provider-lzo 1:10.11.3-1
ii mariadb-plugin-provider-snappy 1:10.11.3-1
ii pv 1.6.20-1
Versions of packages mariadb-server suggests:
ii bsd-mailx [mailx] 8.1.2-0.20220412cvs-1
ii mailutils [mailx] 1:3.15-4
pn mariadb-test <none>
ii netcat-openbsd 1.219-1
-- debconf information:
mariadb-server/nis_warning:
mariadb-server/postrm_remove_databases: false
mariadb-server/old_data_directory_saved:
Version: 1:10.11.3-1
Followup-For: Bug #1009290
X-Debbugs-Cc: witold....@gmail.com
Still broken in 10.11.3.
This is fresh installation, zero modifications.
Jun 24 19:14:24 debian systemd[1]: Starting mariadb.service - MariaDB 10.11.3 database server...
Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [Note] Starting MariaDB 10.11.3-MariaDB-1 source revision as process 2685
Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [Note] InnoDB: Compressed tables use zlib 1.2.13
Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [Note] InnoDB: Number of transaction pools: 1
Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [Note] InnoDB: Using liburing
Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [Note] InnoDB: Completed initialization of buffer pool
Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [ERROR] InnoDB: The Auto-extending data file './ibdata1' is of a different size 0 pages than specified by innodb_data_file_path
Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [ERROR] InnoDB: Operating system error number 22 in a file operation.
Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [ERROR] InnoDB: Error number 22 means 'Invalid argument'
Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [Note] InnoDB: Some operating system error numbers are described at https://mariadb.com/kb/en/library/operating-system-error-codes/
Jun 24 19:14:25 debian mariadbd[2685]: 2023-06-24 19:14:25 0 [ERROR] InnoDB: File (unknown): 'close' returned OS error 222. Cannot continue operation
Jun 24 19:14:25 debian mariadbd[2685]: 230624 19:14:25 [ERROR] mysqld got signal 6 ;
Jun 24 19:14:25 debian mariadbd[2685]: This could be because you hit a bug. It is also possible that this binary
Jun 24 19:14:25 debian mariadbd[2685]: or one of the libraries it was linked against is corrupt, improperly built,
Jun 24 19:14:25 debian mariadbd[2685]: or misconfigured. This error can also be caused by malfunctioning hardware.
Jun 24 19:14:25 debian mariadbd[2685]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
Jun 24 19:14:25 debian mariadbd[2685]: We will try our best to scrape up some info that will hopefully help
Jun 24 19:14:25 debian mariadbd[2685]: diagnose the problem, but since we have already crashed,
Jun 24 19:14:25 debian mariadbd[2685]: something is definitely wrong and this may fail.
Jun 24 19:14:25 debian mariadbd[2685]: Server version: 10.11.3-MariaDB-1 source revision:
Jun 24 19:14:25 debian mariadbd[2685]: key_buffer_size=134217728
Jun 24 19:14:25 debian mariadbd[2685]: read_buffer_size=131072
Jun 24 19:14:25 debian mariadbd[2685]: max_used_connections=0
Jun 24 19:14:25 debian mariadbd[2685]: max_threads=153
Jun 24 19:14:25 debian mariadbd[2685]: thread_count=0
Jun 24 19:14:25 debian mariadbd[2685]: It is possible that mysqld could use up to
Jun 24 19:14:25 debian mariadbd[2685]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 468022 K bytes of memory
Jun 24 19:14:25 debian mariadbd[2685]: Hope that's ok; if not, decrease some variables in the equation.
Jun 24 19:14:25 debian mariadbd[2685]: Thread pointer: 0x0
Jun 24 19:14:25 debian mariadbd[2685]: Attempting backtrace. You can use the following information to find out
Jun 24 19:14:25 debian mariadbd[2685]: where mysqld died. If you see no messages after this, something went
Jun 24 19:14:25 debian mariadbd[2685]: terribly wrong...
Jun 24 19:14:25 debian mariadbd[2685]: stack_bottom = 0x0 thread_stack 0x49000
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x559932b6ecfe]
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(handle_fatal_signal+0x409)[0x5599326dc7c9]
Jun 24 19:14:25 debian mariadbd[2685]: libc_sigaction.c:0(__restore_rt)[0x7efc03891f90]
Jun 24 19:14:25 debian mariadbd[2685]: nptl/pthread_kill.c:44(__pthread_kill_implementation)[0x7efc038e0ccc]
Jun 24 19:14:25 debian mariadbd[2685]: posix/raise.c:27(__GI_raise)[0x7efc03891ef2]
Jun 24 19:14:25 debian mariadbd[2685]: stdlib/abort.c:81(__GI_abort)[0x7efc0387c472]
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(+0x67483c)[0x55993232883c]
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(+0xd2a515)[0x5599329de515]
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(+0xe29078)[0x559932add078]
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(+0xe2c5cf)[0x559932ae05cf]
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(+0xe2c6bd)[0x559932ae06bd]
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(+0x685d85)[0x559932339d85]
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(+0xcd9051)[0x55993298d051]
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(_Z24ha_initialize_handlertonP13st_plugin_int+0x7e)[0x5599326dfafe]
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(+0x80f257)[0x5599324c3257]
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(_Z11plugin_initPiPPci+0x85a)[0x5599324c421a]
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(+0x7008f7)[0x5599323b48f7]
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(_Z11mysqld_mainiPPc+0x41d)[0x5599323ba31d]
Jun 24 19:14:25 debian mariadbd[2685]: x86/libc-start.c:74(__libc_start_call_main)[0x7efc0387d18a]
Jun 24 19:14:25 debian mariadbd[2685]: csu/libc-start.c:128(call_init)[0x7efc0387d245]
Jun 24 19:14:25 debian mariadbd[2685]: /usr/sbin/mariadbd(_start+0x21)[0x5599323af541]
Jun 24 19:14:25 debian mariadbd[2685]: The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
Jun 24 19:14:25 debian mariadbd[2685]: information that should help you find out what is causing the crash.
Jun 24 19:14:25 debian mariadbd[2685]: Writing a core file...
Jun 24 19:14:25 debian mariadbd[2685]: Working directory at /var/lib/mysql
Jun 24 19:14:25 debian mariadbd[2685]: Resource Limits:
Jun 24 19:14:25 debian mariadbd[2685]: Limit Soft Limit Hard Limit Units
Jun 24 19:14:25 debian mariadbd[2685]: Max cpu time unlimited unlimited seconds
Jun 24 19:14:25 debian mariadbd[2685]: Max file size unlimited unlimited bytes
Jun 24 19:14:25 debian mariadbd[2685]: Max data size unlimited unlimited bytes
Jun 24 19:14:25 debian mariadbd[2685]: Max stack size 8388608 unlimited bytes
Jun 24 19:14:25 debian mariadbd[2685]: Max core file size 0 unlimited bytes
Jun 24 19:14:25 debian mariadbd[2685]: Max resident set unlimited unlimited bytes
Jun 24 19:14:25 debian mariadbd[2685]: Max processes 1030035 1030035 processes
Jun 24 19:14:25 debian mariadbd[2685]: Max open files 32768 32768 files
Jun 24 19:14:25 debian mariadbd[2685]: Max locked memory 524288 524288 bytes
Jun 24 19:14:25 debian mariadbd[2685]: Max address space unlimited unlimited bytes
Jun 24 19:14:25 debian mariadbd[2685]: Max file locks unlimited unlimited locks
Jun 24 19:14:25 debian mariadbd[2685]: Max pending signals 1030035 1030035 signals
Jun 24 19:14:25 debian mariadbd[2685]: Max msgqueue size 819200 819200 bytes
Jun 24 19:14:25 debian mariadbd[2685]: Max nice priority 0 0
Jun 24 19:14:25 debian mariadbd[2685]: Max realtime priority 0 0
Jun 24 19:14:25 debian mariadbd[2685]: Max realtime timeout unlimited unlimited us
Jun 24 19:14:25 debian mariadbd[2685]: Core pattern: |/lib/systemd/systemd-coredump %P %u %g %s %t 9223372036854775808 %h
Jun 24 19:14:25 debian mariadbd[2685]: Kernel version: Linux version 6.1.0-9-amd64 (debian...@lists.debian.org) (gcc-12 (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC Debian 6.1.27-1 (2023-05-08)
Jun 24 19:14:25 debian systemd[1]: mariadb.service: Main process exited, code=dumped, status=6/ABRT
Jun 24 19:14:25 debian systemd[1]: mariadb.service: Failed with result 'core-dump'.
Jun 24 19:14:25 debian systemd[1]: Failed to start mariadb.service - MariaDB 10.11.3 database server.
Maybe mariadb does not like it being on overlayfs.
Here are my mounts:
user@debian:~$ mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=131844608k,nr_inodes=32961152,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=26378644k,mode=755,inode64)
/dev/sda1 on /run/live/medium type iso9660 (ro,noatime,nojoliet,check=s,map=n,blocksize=2048,iocharset=utf8)
/dev/loop0 on /run/live/rootfs/filesystem.squashfs type squashfs (ro,noatime,errors=continue)
tmpfs on /run/live/overlay type tmpfs (rw,noatime,size=131893220k,mode=755,inode64)
overlay on / type overlay (rw,noatime,lowerdir=/run/live/rootfs/filesystem.squashfs/,upperdir=/run/live/overlay/rw,workdir=/run/live/overlay/work,redirect_dir=on)
tmpfs on /usr/lib/live/mount type tmpfs (rw,nosuid,nodev,noexec,relatime,size=26378644k,mode=755,inode64)
/dev/sda1 on /usr/lib/live/mount/medium type iso9660 (ro,noatime,nojoliet,check=s,map=n,blocksize=2048,iocharset=utf8)
/dev/loop0 on /usr/lib/live/mount/rootfs/filesystem.squashfs type squashfs (ro,noatime,errors=continue)
tmpfs on /usr/lib/live/mount/overlay type tmpfs (rw,noatime,size=131893220k,mode=755,inode64)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=29782)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
ramfs on /run/credentials/systemd-sysctl.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-tmpfiles-setup-dev.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,relatime,inode64)
ramfs on /run/credentials/systemd-tmpfiles-setup.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
tracefs on /sys/kernel/debug/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=26378640k,nr_inodes=6594660,mode=700,uid=1000,gid=1000,inode64)
gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)
portal on /run/user/1000/doc type fuse.portal (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)
-- System Information:
Debian Release: 12.0
APT prefers stable-security
APT policy: (500, 'stable-security'), (500, 'stable-debug'), (500, 'bookworm'), (500, 'stable')
Architecture: amd64 (x86_64)
Foreign Architectures: i386
Kernel: Linux 6.1.0-9-amd64 (SMP w/32 CPU threads; PREEMPT)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages mariadb-server depends on:
ii adduser 3.134
ii debconf [debconf-2.0] 1.5.82
ii galera-4 26.4.13-1
ii gawk 1:5.2.1-2
ii iproute2 6.1.0-3
ii libc6 2.36-9
ii libdbi-perl 1.643-4
ii libpam0g 1.5.2-6
ii libssl3 3.0.9-1
ii libstdc++6 12.2.0-14
ii lsof 4.95.0-1
ii mariadb-client 1:10.11.3-1
ii mariadb-common 1:10.11.3-1
ii mariadb-server-core 1:10.11.3-1
ii passwd 1:4.13+dfsg1-1+b1
ii perl 5.36.0-7
ii procps 2:4.0.2-3
ii psmisc 23.6-1
ii rsync 3.2.7-1
ii socat 1.7.4.4-2
ii zlib1g 1:1.2.13.dfsg-1
Versions of packages mariadb-server recommends:
ii libhtml-template-perl 2.97-2
ii mariadb-plugin-provider-bzip2 1:10.11.3-1
ii mariadb-plugin-provider-lz4 1:10.11.3-1
ii mariadb-plugin-provider-lzma 1:10.11.3-1
ii mariadb-plugin-provider-lzo 1:10.11.3-1
ii mariadb-plugin-provider-snappy 1:10.11.3-1
ii pv 1.6.20-1
Versions of packages mariadb-server suggests:
ii bsd-mailx [mailx] 8.1.2-0.20220412cvs-1
ii mailutils [mailx] 1:3.15-4
pn mariadb-test <none>
ii netcat-openbsd 1.219-1
-- debconf information:
mariadb-server/nis_warning:
mariadb-server/postrm_remove_databases: false
mariadb-server/old_data_directory_saved:
Witold Baryluk
Jun 25, 2023, 7:50:04 PM6/25/23
to
Package: mariadb-server
Version: 1:10.11.3-1
Followup-For: Bug #1009290
X-Debbugs-Cc: witold....@gmail.com
Just wwitching to fsync, does not appear to solve the problem either:
Version: 1:10.11.3-1
Followup-For: Bug #1009290
X-Debbugs-Cc: witold....@gmail.com
root@debian:~# cat /etc/mysql/mariadb.conf.d/90-live.cnf
[mysqld]
innodb_flush_method = fsync
root@debian:~#
Jun 25 23:30:03 debian systemd[1]: Starting mariadb.service - MariaDB 10.11.3 database server...
Jun 25 23:30:03 debian mariadbd[1395434]: 2023-06-25 23:30:03 0 [Note] Starting MariaDB 10.11.3-MariaDB-1 source revision as process 1395434
Jun 25 23:30:03 debian mariadbd[1395434]: 2023-06-25 23:30:03 0 [Note] InnoDB: Compressed tables use zlib 1.2.13
Jun 25 23:30:03 debian mariadbd[1395434]: 2023-06-25 23:30:03 0 [Note] InnoDB: Number of transaction pools: 1
Jun 25 23:30:03 debian mariadbd[1395434]: 2023-06-25 23:30:03 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
Jun 25 23:30:03 debian mariadbd[1395434]: 2023-06-25 23:30:03 0 [Note] InnoDB: Using liburing
Jun 25 23:30:03 debian mariadbd[1395434]: 2023-06-25 23:30:03 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
Jun 25 23:30:03 debian mariadbd[1395434]: 2023-06-25 23:30:03 0 [Note] InnoDB: Completed initialization of buffer pool
Jun 25 23:30:03 debian mariadbd[1395434]: 2023-06-25 23:30:03 0 [ERROR] InnoDB: The Auto-extending data file './ibdata1' is of a different size 0 pages than specified by innodb_data_file_path
Jun 25 23:30:03 debian mariadbd[1395434]: 2023-06-25 23:30:03 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
Jun 25 23:30:03 debian mariadbd[1395434]: 2023-06-25 23:30:03 0 [Note] InnoDB: Starting shutdown...
Jun 25 23:30:03 debian mariadbd[1395434]: 2023-06-25 23:30:03 0 [ERROR] Plugin 'InnoDB' init function returned error.
Jun 25 23:30:03 debian mariadbd[1395434]: 2023-06-25 23:30:03 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
Jun 25 23:30:03 debian mariadbd[1395434]: 2023-06-25 23:30:03 0 [Note] Plugin 'FEEDBACK' is disabled.
Jun 25 23:30:03 debian mariadbd[1395434]: 2023-06-25 23:30:03 0 [ERROR] Could not open mysql.plugin table: "Table 'mysql.plugin' doesn't exist". Some plugins may be not loaded
Jun 25 23:30:03 debian mariadbd[1395434]: 2023-06-25 23:30:03 0 [ERROR] Unknown/unsupported storage engine: InnoDB
Jun 25 23:30:03 debian mariadbd[1395434]: 2023-06-25 23:30:03 0 [ERROR] Aborting
Jun 25 23:30:03 debian systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Jun 25 23:30:03 debian systemd[1]: mariadb.service: Failed with result 'exit-code'.
Jun 25 23:30:03 debian systemd[1]: Failed to start mariadb.service - MariaDB 10.11.3 database server.
This is probably, becuase postinst script probably (silently) failed, and
something was not initialized properly during install.
Adding runtime check in that script to detect overlay fs will not be
enough, because it is possible to install mariadb-server in chroot for
live image, which could be on non-overlay fs, but when actually started
it might be on overlay fs.
It would be preferable to do this in startup script instead, but then a
lot of logic would probably need to move from postinst, to startup script.
0 new messages