Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
Bug#1027337: mariadb-server-10.5: Huge increase in memory consumption (leak?) since upgrade from 10.5.15 to 10.5.18
962 views
Skip to first unread message
Xan Charbonnet
Dec 30, 2022, 11:20:03 AM12/30/22
to
Package: mariadb-server-10.5
Version: 1:10.5.18-0+deb11u1
Severity: important
Dear Maintainer,
I run a pair of nameservers, and after a recent round of minor software
updates (see below for the complete list), mariadbd was OOM'd on both
machines.
These machines are minimal VMs: one has 512MB RAM, the other 1GB RAM. They've
been running various versions of Debian flawlessly for many years. I
temporarily upgraded the hardware to 4GB RAM on both in order to remove some
urgency from the situation. It does appear that mariadbd's memory consumption
is far higher than it should be, and slowly growing.
There isn't too much being demanded of this database. Its only client is
PowerDNS which runs locally. PowerDNS was not changed in this recent update.
Please let me know what additional information I can provide. Thank you!
Packages updated when the problem began:
base-files:amd64 from 11.1+deb11u5 to 11.1+deb11u6
nftables:amd64 from 0.9.8-3.1 to 0.9.8-3.1+deb11u1
libnftables1:amd64 from 0.9.8-3.1 to 0.9.8-3.1+deb11u1
mariadb-common:all from 1:10.5.15-0+deb11u1 to 1:10.5.18-0+deb11u1
libmariadb3:amd64 from 1:10.5.15-0+deb11u1 to 1:10.5.18-0+deb11u1
mariadb-client-core-10.5:amd64 from 1:10.5.15-0+deb11u1 to 1:10.5.18-0+deb11u1
mariadb-client-10.5:amd64 from 1:10.5.15-0+deb11u1 to 1:10.5.18-0+deb11u1
mariadb-server-core-10.5:amd64 from 1:10.5.15-0+deb11u1 to 1:10.5.18-0+deb11u1
mariadb-server-10.5:amd64 from 1:10.5.15-0+deb11u1 to 1:10.5.18-0+deb11u1
libtasn1-6:amd64 from 4.16.0-2 to 4.16.0-2+deb11u1
nano:amd64 from 5.4-2+deb11u1 to 5.4-2+deb11u2
distro-info-data:all from 0.51+deb11u2 to 0.51+deb11u3
grub2-common:amd64 from 2.06-3~deb11u4 to 2.06-3~deb11u5
grub-pc:amd64 from 2.06-3~deb11u4 to 2.06-3~deb11u5
grub-pc-bin:amd64 from 2.06-3~deb11u4 to 2.06-3~deb11u5
grub-common:amd64 from 2.06-3~deb11u4 to 2.06-3~deb11u5
libksba8:amd64 from 1.5.0-3+deb11u1 to 1.5.0-3+deb11u2
linux-image-amd64:amd64 from 5.10.149-2 to 5.10.158-2
linux-libc-dev:amd64 from 5.10.149-2 to 5.10.158-2
mariadb-server:all from 1:10.5.15-0+deb11u1 to 1:10.5.18-0+deb11u1
-- System Information:
Debian Release: 11.6
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'stable-security'), (500, 'stable')
Architecture: amd64 (x86_64)
Kernel: Linux 5.10.0-20-amd64 (SMP w/2 CPU threads)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages mariadb-server-10.5 depends on:
ii adduser 3.118
ii debconf [debconf-2.0] 1.5.77
ii galera-4 26.4.11-0+deb11u1
ii gawk 1:5.1.0-1
ii iproute2 5.10.0-4
ii libc6 2.31-13+deb11u5
ii libdbi-perl 1.643-3+b1
ii libpam0g 1.4.0-9+deb11u1
ii libssl1.1 1.1.1n-0+deb11u3
ii libstdc++6 10.2.1-6
ii lsb-base 11.1.0
ii lsof 4.93.2+dfsg-1.1
ii mariadb-client-10.5 1:10.5.18-0+deb11u1
ii mariadb-common 1:10.5.18-0+deb11u1
ii mariadb-server-core-10.5 1:10.5.18-0+deb11u1
ii passwd 1:4.8.1-1
ii perl 5.32.1-4+deb11u2
ii procps 2:3.3.17-5
ii psmisc 23.4-2
ii rsync 3.2.3-4+deb11u1
ii socat 1.7.4.1-3
ii zlib1g 1:1.2.11.dfsg-2+deb11u2
Versions of packages mariadb-server-10.5 recommends:
ii libhtml-template-perl 2.97-1.1
Versions of packages mariadb-server-10.5 suggests:
ii mailutils [mailx] 1:3.10-3+b1
pn mariadb-test <none>
pn netcat-openbsd <none>
-- debconf information excluded
Version: 1:10.5.18-0+deb11u1
Severity: important
Dear Maintainer,
I run a pair of nameservers, and after a recent round of minor software
updates (see below for the complete list), mariadbd was OOM'd on both
machines.
These machines are minimal VMs: one has 512MB RAM, the other 1GB RAM. They've
been running various versions of Debian flawlessly for many years. I
temporarily upgraded the hardware to 4GB RAM on both in order to remove some
urgency from the situation. It does appear that mariadbd's memory consumption
is far higher than it should be, and slowly growing.
There isn't too much being demanded of this database. Its only client is
PowerDNS which runs locally. PowerDNS was not changed in this recent update.
Please let me know what additional information I can provide. Thank you!
Packages updated when the problem began:
base-files:amd64 from 11.1+deb11u5 to 11.1+deb11u6
nftables:amd64 from 0.9.8-3.1 to 0.9.8-3.1+deb11u1
libnftables1:amd64 from 0.9.8-3.1 to 0.9.8-3.1+deb11u1
mariadb-common:all from 1:10.5.15-0+deb11u1 to 1:10.5.18-0+deb11u1
libmariadb3:amd64 from 1:10.5.15-0+deb11u1 to 1:10.5.18-0+deb11u1
mariadb-client-core-10.5:amd64 from 1:10.5.15-0+deb11u1 to 1:10.5.18-0+deb11u1
mariadb-client-10.5:amd64 from 1:10.5.15-0+deb11u1 to 1:10.5.18-0+deb11u1
mariadb-server-core-10.5:amd64 from 1:10.5.15-0+deb11u1 to 1:10.5.18-0+deb11u1
mariadb-server-10.5:amd64 from 1:10.5.15-0+deb11u1 to 1:10.5.18-0+deb11u1
libtasn1-6:amd64 from 4.16.0-2 to 4.16.0-2+deb11u1
nano:amd64 from 5.4-2+deb11u1 to 5.4-2+deb11u2
distro-info-data:all from 0.51+deb11u2 to 0.51+deb11u3
grub2-common:amd64 from 2.06-3~deb11u4 to 2.06-3~deb11u5
grub-pc:amd64 from 2.06-3~deb11u4 to 2.06-3~deb11u5
grub-pc-bin:amd64 from 2.06-3~deb11u4 to 2.06-3~deb11u5
grub-common:amd64 from 2.06-3~deb11u4 to 2.06-3~deb11u5
libksba8:amd64 from 1.5.0-3+deb11u1 to 1.5.0-3+deb11u2
linux-image-amd64:amd64 from 5.10.149-2 to 5.10.158-2
linux-libc-dev:amd64 from 5.10.149-2 to 5.10.158-2
mariadb-server:all from 1:10.5.15-0+deb11u1 to 1:10.5.18-0+deb11u1
-- System Information:
Debian Release: 11.6
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'stable-security'), (500, 'stable')
Architecture: amd64 (x86_64)
Kernel: Linux 5.10.0-20-amd64 (SMP w/2 CPU threads)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages mariadb-server-10.5 depends on:
ii adduser 3.118
ii debconf [debconf-2.0] 1.5.77
ii galera-4 26.4.11-0+deb11u1
ii gawk 1:5.1.0-1
ii iproute2 5.10.0-4
ii libc6 2.31-13+deb11u5
ii libdbi-perl 1.643-3+b1
ii libpam0g 1.4.0-9+deb11u1
ii libssl1.1 1.1.1n-0+deb11u3
ii libstdc++6 10.2.1-6
ii lsb-base 11.1.0
ii lsof 4.93.2+dfsg-1.1
ii mariadb-client-10.5 1:10.5.18-0+deb11u1
ii mariadb-common 1:10.5.18-0+deb11u1
ii mariadb-server-core-10.5 1:10.5.18-0+deb11u1
ii passwd 1:4.8.1-1
ii perl 5.32.1-4+deb11u2
ii procps 2:3.3.17-5
ii psmisc 23.4-2
ii rsync 3.2.3-4+deb11u1
ii socat 1.7.4.1-3
ii zlib1g 1:1.2.11.dfsg-2+deb11u2
Versions of packages mariadb-server-10.5 recommends:
ii libhtml-template-perl 2.97-1.1
Versions of packages mariadb-server-10.5 suggests:
ii mailutils [mailx] 1:3.10-3+b1
pn mariadb-test <none>
pn netcat-openbsd <none>
-- debconf information excluded
Xan Charbonnet
Dec 31, 2022, 5:10:07 PM12/31/22
to
Some memory statistics. Uptime, resident memory size, and % of memory
in the system for mariadbd:
30 hours -- 1.0g -- 26.7%
36 hours -- 1.2g -- 30.3%
54 hours -- 1.6g -- 41.5%
Looks like a pretty linear progression of ~33MB/hour being leaked. This
is on a system that previously had 512MB of total RAM. Now with 4GB it
appears it'll be able to stay up about 5 days total. There were no
configuration changes other than upgrading the packages.
in the system for mariadbd:
30 hours -- 1.0g -- 26.7%
36 hours -- 1.2g -- 30.3%
54 hours -- 1.6g -- 41.5%
Looks like a pretty linear progression of ~33MB/hour being leaked. This
is on a system that previously had 512MB of total RAM. Now with 4GB it
appears it'll be able to stay up about 5 days total. There were no
configuration changes other than upgrading the packages.
Xan Charbonnet
Jan 2, 2023, 2:10:03 PM1/2/23
to
At the 96 hour mark it hit 2.5GB, 64.6%. I reverted the seven MariaDB
packages back to 10.5.15, and the problem has disappeared. mariadbd is
holding steady at 135MB, 3.4%.
I'm happy to help participate in figuring out what's going wrong.
packages back to 10.5.15, and the problem has disappeared. mariadbd is
holding steady at 135MB, 3.4%.
I'm happy to help participate in figuring out what's going wrong.
Andreas Laut
Jan 5, 2023, 9:20:05 AM1/5/23
to
Hi all,
we'd like to say: We have hit the same bug in the same way and we have
no idea what happens here.
We are also running this mariadb server as powerdns backend (means small
dataset) and since the upgrade to 10.5.18 the ressources usage increase
constantly. Not only RSS of mariadbd. It's also the cpu load which
increases the same way. Until the memory limit is reached and the
database process gets oom killed.
we'd like to say: We have hit the same bug in the same way and we have
no idea what happens here.
We are also running this mariadb server as powerdns backend (means small
dataset) and since the upgrade to 10.5.18 the ressources usage increase
constantly. Not only RSS of mariadbd. It's also the cpu load which
increases the same way. Until the memory limit is reached and the
database process gets oom killed.
Julien Banchet
Jan 5, 2023, 10:20:04 AM1/5/23
to
Though, oddly, it doesn't seem to effect each and every instance of 10.5.18 (most were automatically upgraded)
For instance in one of the master-master setups we have for a powerdns+radius backend, it's mainly the second node that has little queries that hogs memory, while the first, much more solicited, is keeping in check
https://www.evernote.com/l/ACgw53fJdg5C_qBaMsAkst14aXRnI1tJjhg
Cordialement,
Julien Banchet
--
Cusae - 6 Allée Cardinal de Givry 21000 Dijon
Tél : +33951717101 / +33651886877
Andreas Laut
Jan 6, 2023, 3:10:03 AM1/6/23
to
What we tested yesterday:
dump the pdns database, drop it and push all data again into mariadb-server. To be sure that the data doesn't get messed up) After that we restarted pdns prozess.
mysqldump pdns > /tmp/pdns.sql; mysql -e "DROP DATABASE pdns"; mysql -e "CREATE DATABASE pdns"; cat /tmp/pdns.sql | mysql pdns; service pdns restart(command not tested so don't use this directly!!, just a sum up of the commands used)
You can see the effect on this graph. At point x it was done. CPU
load of mariadb-server process goes down but it started again to
increase. On the memory usage it has no effect, increase
constantly.
Xan Charbonnet
Jan 6, 2023, 9:30:03 AM1/6/23
to
Every few seconds (at least in my configuration), PDNS runs database
queries to see which domains need refreshing.
After running for a while with this problem, I can "catch" those queries
(at least one of them) with SHOW PROCESSLIST.
Excerpt:
| 117 | pdns | localhost | pdns | Execute | 0 | Starting cleanup |
SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records
WHERE disabled=0 and type=? and name=? | 0.000 |
It seems to me that the longer this problem has been going on, the
longer this query takes, at least the "starting cleanup" phase of it. I
think that's why CPU usage climbs: the longer it goes, the longer this
query takes and the more CPU is used to execute it.
Caught a different one just now, also in phase "starting cleanup":
| 128 | pdns | localhost | pdns | Execute | 0 | Starting cleanup |
SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records
WHERE disabled=0 and name=? and domain_id=? | 0.000 |
Yes, I can regularly "catch" many queries and they're always in state
"starting cleanup".
I went over to the machine that is not having this problem (because it's
still on 10.5.15) and was able to catch one such query in "starting
cleanup", and another in "commit". But it took maybe 25x more attempts
to catch such a query than on the one with the problem. The queries on
the "good" box finish very quickly, and they're slow on the "bad" box.
queries to see which domains need refreshing.
After running for a while with this problem, I can "catch" those queries
(at least one of them) with SHOW PROCESSLIST.
Excerpt:
| 117 | pdns | localhost | pdns | Execute | 0 | Starting cleanup |
SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records
WHERE disabled=0 and type=? and name=? | 0.000 |
It seems to me that the longer this problem has been going on, the
longer this query takes, at least the "starting cleanup" phase of it. I
think that's why CPU usage climbs: the longer it goes, the longer this
query takes and the more CPU is used to execute it.
Caught a different one just now, also in phase "starting cleanup":
| 128 | pdns | localhost | pdns | Execute | 0 | Starting cleanup |
SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records
WHERE disabled=0 and name=? and domain_id=? | 0.000 |
Yes, I can regularly "catch" many queries and they're always in state
"starting cleanup".
I went over to the machine that is not having this problem (because it's
still on 10.5.15) and was able to catch one such query in "starting
cleanup", and another in "commit". But it took maybe 25x more attempts
to catch such a query than on the one with the problem. The queries on
the "good" box finish very quickly, and they're slow on the "bad" box.
Faustin Lammler
Jan 10, 2023, 7:40:04 AM1/10/23
to
Hi,
first of all, thank you all for the excellent (and detailed) report!
I don't see anything that could be related with Debian packaging of
MariaDB and so I suggest you check directly upstream.
I have searched into https://jira.mariadb.org and the closest report
that I can come with is https://jira.mariadb.org/browse/MDEV-29988. I do
not forward this bug report to it because I am not sure that this is the
same problem.
There is also https://jira.mariadb.org/browse/MDEV-29097 that may give
some ideas on how to find a workaround until the next release happen.
Cheers!
--
Faustin
GPG: F652 BCD1 1AA8 8975 F010 48A5 390A 2F27 832A 5C79
first of all, thank you all for the excellent (and detailed) report!
I don't see anything that could be related with Debian packaging of
MariaDB and so I suggest you check directly upstream.
I have searched into https://jira.mariadb.org and the closest report
that I can come with is https://jira.mariadb.org/browse/MDEV-29988. I do
not forward this bug report to it because I am not sure that this is the
same problem.
There is also https://jira.mariadb.org/browse/MDEV-29097 that may give
some ideas on how to find a workaround until the next release happen.
Cheers!
--
Faustin
GPG: F652 BCD1 1AA8 8975 F010 48A5 390A 2F27 832A 5C79
Xan Charbonnet
Jan 10, 2023, 11:10:03 AM1/10/23
to
Faustin,
Thanks for looking into this, and for your work packaging MariaDB for
Debian.
It looks to me like https://jira.mariadb.org/browse/MDEV-29988 is very
likely to be the issue we're seeing. Fortunately (if I'm interpreting
upstream's JIRA correctly), it looks like a fix exists and will be
included in 10.5.19.
From their release history, 10.5.19 can be expected in early February.
Probably what I will do is downgrade all machines to 10.5.15 and look
forward to the Debian package of 10.5.19, which hopefully will not be
long after the upstream release.
Thanks for looking into this, and for your work packaging MariaDB for
Debian.
It looks to me like https://jira.mariadb.org/browse/MDEV-29988 is very
likely to be the issue we're seeing. Fortunately (if I'm interpreting
upstream's JIRA correctly), it looks like a fix exists and will be
included in 10.5.19.
From their release history, 10.5.19 can be expected in early February.
Probably what I will do is downgrade all machines to 10.5.15 and look
forward to the Debian package of 10.5.19, which hopefully will not be
long after the upstream release.
Faustin Lammler
Jan 10, 2023, 2:00:05 PM1/10/23
to
Hi!
Xan Charbonnet <x...@charbonnet.com>,
10/01/2023 – 10:07:02 (-0600):
> It looks to me like https://jira.mariadb.org/browse/MDEV-29988 is very
> likely to be the issue we're seeing. Fortunately (if I'm interpreting
> upstream's JIRA correctly), it looks like a fix exists and will be included
> in 10.5.19.
Indeed, MDEV-29988 is fixed in the next release.
> From their release history, 10.5.19 can be expected in early February.
Exact, maybe even by the end of January.
> Probably what I will do is downgrade all machines to 10.5.15 and look
> forward to the Debian package of 10.5.19, which hopefully will not be long
> after the upstream release.
I already sent an email to Otto about this critical issue and hopefully,
10.5.19 will hit Debian soon after it's GA upstream.
Cheers!
--
Faustin
Xan Charbonnet <x...@charbonnet.com>,
10/01/2023 – 10:07:02 (-0600):
> It looks to me like https://jira.mariadb.org/browse/MDEV-29988 is very
> likely to be the issue we're seeing. Fortunately (if I'm interpreting
> upstream's JIRA correctly), it looks like a fix exists and will be included
> in 10.5.19.
> From their release history, 10.5.19 can be expected in early February.
> Probably what I will do is downgrade all machines to 10.5.15 and look
> forward to the Debian package of 10.5.19, which hopefully will not be long
> after the upstream release.
10.5.19 will hit Debian soon after it's GA upstream.
Cheers!
--
Faustin
fmaesen
Feb 7, 2023, 10:40:04 AM2/7/23
to
Hi all, there is an upstream release 10.5.19 with hopefully a fix:
https://mariadb.com/kb/en/mariadb-10-5-19-release-notes/
On Thu, 12 Jan 2023 08:55:06 +0000 Matteo Valsasna <MValsasna@pdxeng.ch> wrote:
> Thanks a lot Faustin
>
>
>
> I found an useful description of MDEV-29988 in a comment to one of its duplicates, and it matches quite well our application scenario for the database connection that cause the memory leak according to information_schema.PROCESSLIST.memory_used
> (prepared statements executed maaany times, string comparisons)
>
> we reproduced the issue on a test machine, and we will try to downgrade to 10.5.15
>
>
>
> MAtteo
>
>
>
> https://jira.mariadb.org/browse/MDEV-30096?focusedCommentId=243814&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-243814
>
> [serg]Sergei Golubchik<https://jira.mariadb.org/secure/ViewProfile.jspa?name=serg> added a comment - 2022-11-28 22:20
>
> Yes, <https://jira.mariadb.org/browse/MDEV-29988> was in 10.6.11 too. But let's first check the preconditions:
>
> * you use prepared statements
> * you prepare once and then execute many thousands of times before deallocating the statement
> * your statements contains a string comparison with strings in different (but compatible) character sets (e.g. utf8 and latin1).
>
> if all that matches your use case you could be affected by <https://jira.mariadb.org/browse/MDEV-29988> . If you do, then
>
> * every execution of such a prepared statements will allocate more memory
> * all memory will be freed when a statements is deallocated (explicitly or when the client disconnects)
> * time to deallocate a prepared statement is proportional to the number of times it was executed
>
> and workarounds could be
>
> * deallocate prepared statements more often
> * rewrite statements to avoid comparing strings in different character sets
>
> or wait for the next release that will have it fixed
https://mariadb.com/kb/en/mariadb-10-5-19-release-notes/
On Thu, 12 Jan 2023 08:55:06 +0000 Matteo Valsasna <MValsasna@pdxeng.ch> wrote:
> Thanks a lot Faustin
>
>
>
> I found an useful description of MDEV-29988 in a comment to one of its duplicates, and it matches quite well our application scenario for the database connection that cause the memory leak according to information_schema.PROCESSLIST.memory_used
> (prepared statements executed maaany times, string comparisons)
>
> we reproduced the issue on a test machine, and we will try to downgrade to 10.5.15
>
>
>
> MAtteo
>
>
>
> https://jira.mariadb.org/browse/MDEV-30096?focusedCommentId=243814&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-243814
>
> [serg]Sergei Golubchik<https://jira.mariadb.org/secure/ViewProfile.jspa?name=serg> added a comment - 2022-11-28 22:20
>
> Yes, <https://jira.mariadb.org/browse/MDEV-29988> was in 10.6.11 too. But let's first check the preconditions:
>
> * you use prepared statements
> * you prepare once and then execute many thousands of times before deallocating the statement
> * your statements contains a string comparison with strings in different (but compatible) character sets (e.g. utf8 and latin1).
>
> if all that matches your use case you could be affected by <https://jira.mariadb.org/browse/MDEV-29988> . If you do, then
>
> * every execution of such a prepared statements will allocate more memory
> * all memory will be freed when a statements is deallocated (explicitly or when the client disconnects)
> * time to deallocate a prepared statement is proportional to the number of times it was executed
>
> and workarounds could be
>
> * deallocate prepared statements more often
> * rewrite statements to avoid comparing strings in different character sets
>
> or wait for the next release that will have it fixed
Otto Kekäläinen
Mar 1, 2023, 1:40:05 AM3/1/23
to
On Tue, 28 Feb 2023 at 09:25, Faustin Lammler <fau...@fala.red> wrote:
>
> Hi Matteo!
> I am not in charge of doing uploads to Debian FTP so, I don't know.
>
> @Otto do you know when 10.5.19 will be uploaded?
Only Debian release team knows the exact date. Page
>
> Hi Matteo!
> I am not in charge of doing uploads to Debian FTP so, I don't know.
>
> @Otto do you know when 10.5.19 will be uploaded?
https://release.debian.org/ says mid-February.
You can +1 on https://bugs.debian.org/1031042 if you want or you can
already now download binaries from the CI pipeline from commit
https://salsa.debian.org/mariadb-team/mariadb-10.5/-/commits/debian/1%2510.5.19-0+deb11u1
= https://salsa.debian.org/mariadb-team/mariadb-10.5/-/pipelines/498296
= https://salsa.debian.org/mariadb-team/mariadb-10.5/-/jobs/3929573/artifacts/browse/debian/output/
0 new messages