Bug#1037327: MariaDB upgrade fails if page compressed tables exist already due to plugin dependency

[Laurence Parry]

Package: mariadb-server

Version: 1:10.11.3-1

Dear Maintainer,

When trying to upgrade a (sysvinit) Debian Bullseye[-backports] server with MariaDB 10.5 to Bookworm, I ran into the following issue:

Setting up mariadb-server (1:10.11.3-1) ...
Stopping MariaDB database server: mariadbd.
Starting MariaDB database server: mariadbd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . failed!
invoke-rc.d: initscript mariadb, action "start" failed.
dpkg: error processing package mariadb-server (--configure):
 installed mariadb-server package post-installation script subprocess returned error exit status 1
[...]

dpkg: dependency problems prevent configuration of mariadb-plugin-provider-lz4:
 mariadb-plugin-provider-lz4 depends on mariadb-server (>= 1:10.11.1-1); however:
  Package mariadb-server is not configured yet.

dpkg: error processing package mariadb-plugin-provider-lz4 (--configure):
 dependency problems - leaving unconfigured

This occurred because the piwik/matamo database already contained lz4-compressed tables. MariaDB complained about lacking the plugin, and mariadb-server could not complete config. mariadb-plugin-provider-lz4 could not be installed as it depended on mariadb-server.

My my.cnf has (from https://mariadb.com/kb/en/innodb-page-compression/ ):

innodb_compression_algorithm=lz4
innodb_compression_default=ON

I ended up having to manually download from:

https://packages.debian.org/bookworm/amd64/mariadb-plugin-provider-lz4/download

and force it to ignore the dependency:

dpkg -i --force-depends mariadb-plugin-provider-lz4_10.11.3-1_amd64.deb

After this I could fix the installation and the server started. Removing the plugin package predictably breaks the server again - this also reproduces the "not available" error I got after trying to run the server manually, which wasn't surfaced during the install:

2023-06-11 13:08:58 0 [Note] Starting MariaDB 10.11.3-MariaDB-1 source revision  as process 16188
2023-06-11 13:08:58 0 [ERROR] mysqld: Can't open shared library '/usr/lib/mysql/plugin/provider_lz4.so' (errno: 2, cannot open shared object file: No such file or directory)
2023-06-11 13:08:58 0 [ERROR] Couldn't load plugins from 'provider_lz4.so'.
2023-06-11 13:08:58 0 [ERROR] mysqld: InnoDB: compression algorithm lz4 (2) is not available. Please, load the corresponding provider plugin.
2023-06-11 13:08:58 0 [ERROR] Plugin 'InnoDB' init function returned error.
2023-06-11 13:08:58 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2023-06-11 13:08:58 0 [Note] Plugin 'FEEDBACK' is disabled.
2023-06-11 13:08:58 0 [ERROR] mysqld: unknown variable 'provider_lz4=force_plus_permanent'
2023-06-11 13:08:58 0 [ERROR] Aborting

I can appreciate that the plugins may rely on mariadb-server to be installed to an extent. However it does not have to be installed the whole way otherwise dpkg wouldn't have worked. Perhaps the plugins could be installed despite mariadb-server not being fully configured, and in this case it should trigger the server to be reconfigured after the plugins are installed.

Sample table structure shows "PAGE_COMPRESSED":

| piwik_log_action | CREATE TABLE `piwik_log_action` (
  `idaction` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `name` varchar(4096) DEFAULT NULL,
  `hash` int(10) unsigned NOT NULL,
  `type` tinyint(3) unsigned DEFAULT NULL,
  `url_prefix` tinyint(2) DEFAULT NULL,
  PRIMARY KEY (`idaction`),
  KEY `index_type_hash` (`type`,`hash`)
) ENGINE=InnoDB AUTO_INCREMENT=116404687 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci `PAGE_COMPRESSED`=1 |

# uname -a

Linux acedia 6.1.27 #1 SMP Thu Jun  8 16:12:46 CEST 2023 x86_64 GNU/Linux

Libc: Version: 2.36-9

Best regards,

--

Laurence "GreenReaper" Parry
https://www.greenreaper.co.uk

[Otto Kekäläinen]

Thanks for the report!

We have gone back-and-forth on if the server should have 'depends' or 'recommends' or the compression plugins (eg. https://salsa.debian.org/mariadb-team/mariadb-server/-/merge_requests/36) but actually have not considered that the provider plugins indeed could also have adjusted if they depend or just weakly recommend the server.

I will do some testing on this.

[Otto Kekäläinen]

Hello!

It seems the issue is with dpkg not making the new config files
available using the correct name:

# apt install mariadb-plugin-provider-lz4
Setting up mariadb-server (1:10.11.4-1) ...

Stopping MariaDB database server: mariadbd.
Starting MariaDB database server: mariadbd . . . . .

Running '/etc/init.d/mariadb start' failed with error log:
230611 20:15:15 mysqld_safe Starting mariadbd daemon with databases
from /var/lib/mysql
2023-06-11 20:15:15 0 [Note] Starting MariaDB 10.11.4-MariaDB-1 source
revision as process 4570
2023-06-11 20:15:15 0 [ERROR] mariadbd: InnoDB: compression algorithm

lz4 (2) is not available. Please, load the corresponding provider
plugin.

2023-06-11 20:15:15 0 [ERROR] Plugin 'InnoDB' init function returned error.
2023-06-11 20:15:15 0 [ERROR] Plugin 'InnoDB' registration as a
STORAGE ENGINE failed.
2023-06-11 20:15:15 0 [Note] Plugin 'FEEDBACK' is disabled.
2023-06-11 20:15:15 0 [ERROR] Unknown/unsupported storage engine: InnoDB
2023-06-11 20:15:15 0 [ERROR] Aborting

# ls -la /usr/lib/mysql/plugin/provider_lz4.so
-rw-r--r-- 1 root root 14192 Jun 4 18:22 /usr/lib/mysql/plugin/provider_lz4.so

# find /etc/mysql/mariadb.conf.d
/etc/mysql/mariadb.conf.d/50-client.cnf
/etc/mysql/mariadb.conf.d/50-mysql-clients.cnf
/etc/mysql/mariadb.conf.d/60-galera.cnf
/etc/mysql/mariadb.conf.d/50-mysqld_safe.cnf
/etc/mysql/mariadb.conf.d/innodb-lz-compression.cnf
/etc/mysql/mariadb.conf.d/provider_bzip2.cnf.dpkg-new
/etc/mysql/mariadb.conf.d/provider_lz4.cnf.dpkg-new
/etc/mysql/mariadb.conf.d/provider_lzma.cnf.dpkg-new
/etc/mysql/mariadb.conf.d/provider_lzo.cnf.dpkg-new
/etc/mysql/mariadb.conf.d/provider_snappy.cnf.dpkg-new
/etc/mysql/mariadb.conf.d/50-server.cnf

# cat /etc/mysql/mariadb.conf.d/provider_lz4.cnf.dpkg-new
[server]
plugin_load_add=provider_lz4
provider_lz4=force_plus_permanent

# cp -a /etc/mysql/mariadb.conf.d/provider_lz4.cnf.dpkg-new
/etc/mysql/mariadb.conf.d/provider_lz4.cnf

# apt install mariadb-plugin-provider-lz4
mariadb-plugin-provider-lz4 is already the newest version (1:10.11.4-1).
Setting up mariadb-server (1:10.11.4-1) ...

Stopping MariaDB database server: mariadbd.
Starting MariaDB database server: mariadbd.

Setting up mariadb-plugin-provider-lz4 (1:10.11.4-1) ...
Setting up mariadb-server-10.5 (1:10.11.4-1) ...
Processing triggers for mariadb-server (1:10.11.4-1) ...
Stopping MariaDB database server: mariadbdERROR 2002 (HY000): Can't
connect to local server through socket '/run/mysqld/mysqld.sock' (2)
Starting MariaDB database server: mariadbd.

# find /etc/mysql/mariadb.conf.d
/etc/mysql/mariadb.conf.d
/etc/mysql/mariadb.conf.d/50-client.cnf
/etc/mysql/mariadb.conf.d/50-mysql-clients.cnf
/etc/mysql/mariadb.conf.d/60-galera.cnf
/etc/mysql/mariadb.conf.d/50-mysqld_safe.cnf
/etc/mysql/mariadb.conf.d/innodb-lz-compression.cnf
/etc/mysql/mariadb.conf.d/50-server.cnf
/etc/mysql/mariadb.conf.d/provider_lz4.cnf
/etc/mysql/mariadb.conf.d/provider_bzip2.cnf
/etc/mysql/mariadb.conf.d/provider_lzma.cnf
/etc/mysql/mariadb.conf.d/provider_lzo.cnf
/etc/mysql/mariadb.conf.d/provider_snappy.cnf

The are renamed from *.cnf.dpkg-new to *.cnf only after the
installation completes.

Perhaps the solution involves adjusting the time when the
restart/reload is triggered and somehow force it to happen as late as
possible, to ensure the 'Setting up mariadb-plugin-provider-lz4' has
happened before server restarts
(https://salsa.debian.org/mariadb-team/mariadb-server/-/blob/debian/latest/debian/rules#L239-243).


I added a test case for this in
https://salsa.debian.org/otto/mariadb-server/-/commits/bugfix/1037327-upgrades-need-provider-plugin-lz
and it fails (as expected) in
https://salsa.debian.org/otto/mariadb-server/-/jobs/4297516 but if we
come up with a solution, at least we have now a test to validate it.

[Otto Kekäläinen]

> I added a test case for this in
> https://salsa.debian.org/otto/mariadb-server/-/commits/bugfix/1037327-upgrades-need-provider-plugin-lz
> and it fails (as expected) in
> https://salsa.debian.org/otto/mariadb-server/-/jobs/4297516 but if we
> come up with a solution, at least we have now a test to validate it.

MR opened https://salsa.debian.org/mariadb-team/mariadb-server/-/merge_requests/51
about the test. I have not yet have time to figure out the best way to
avoid this.

If other people run into this issue, please tell about it by commenting here.

For the time being the manual workaround is to run:

```
cp -a /etc/mysql/mariadb.conf.d/provider_lz4.cnf.dpkg-new
/etc/mysql/mariadb.conf.d/provider_lz4.cnf
apt install mariadb-plugin-provider-lz4
```