two identical galera nodes, yet completely different swap usage for mysqld process
cyusedfzfb
Hi all,
Hoping you've all enjoyed a good weekend!
I'm trying to understand why we see mysqld use 100% swap on one node galera-2, and 0% swap on other node galera-1.
Both machines have 62G RAM, and "free -g" shows plenty RAM available:
[user@galera-1 ~]$ free -gand
total used free shared buff/cache available
Mem: 62 35 7 2 19 23
Swap: 4 0 4
[user@galera-2 ~]$ free -gThe mariadb running configuration ("SHOW VARIABLES;") is the same on both, except for obvious differences like hostname, auto_increment_offset, server_id, and some internal variables.
total used free shared buff/cache available
Mem: 62 40 0 1 21 19
Swap: 4 4 0
Both have sysctl "vm.swappiness=1" (persistant as well as for the running conf)
TOP shows that mysql is using zero swap on galera-1:
top - 10:15:51 up 55 days, 21:49, 1 user, load average: 0.15, 0.25, 0.16and pretty much ALL available swap on the other galera-2:
Tasks: 266 total, 2 running, 264 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.6 us, 0.6 sy, 0.0 ni, 98.5 id, 0.1 wa, 0.1 hi, 0.1 si, 0.0 st
MiB Mem : 63876.6 total, 7282.5 free, 36448.4 used, 20145.7 buff/cache
MiB Swap: 5120.0 total, 4697.4 free, 422.6 used. 23985.1 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND SWAP
409260 mysql 20 0 48.7g 40.1g 5.0g S 0.9 64.2 177:00.68 mysqld 0.0g
top - 10:18:32 up 11 days, 19:30, 1 user, load average: 0.04, 0.08, 0.24
Tasks: 281 total, 1 running, 280 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.9 us, 0.5 sy, 0.0 ni, 98.4 id, 0.0 wa, 0.1 hi, 0.1 si, 0.0 st
MiB Mem : 63876.6 total, 816.4 free, 41305.4 used, 21754.8 buff/cache
MiB Swap: 5120.0 total, 34.9 free, 5085.1 used. 20562.6 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND SWAP
1344 mysql 20 0 48.9g 40.8g 4.2g S 0.7 65.3 218:14.51 mysqld 4.9g
Both machines are REPLICATING ONLY in the same galera cluster,
where all reading/writing is done on a third node. Both have no
trouble keeping up with the cluster.
Disk-wise also no difference, and both are running the same
mariadb on RHEL8.9 mysql Ver 15.1 Distrib 10.5.22-MariaDB, for
Linux (x86_64) using EditLine wrapper
Our monitoring doesn't like 100% used swap, but more fundamentally I want to understand the difference in behaviour.
Someone here with suggestions what to look at, and how to get a better understanding of what is happening?
Thanks!
For the record, ending this email with cat /proc/meminfo for both machines, hoping that that makes some sense to someone here. :-)
galera-1 | CHANGED | rc=0 >>and
MemTotal: 65409676 kB
MemFree: 7274944 kB
MemAvailable: 24479216 kB
Buffers: 6372 kB
Cached: 20620352 kB
SwapCached: 22064 kB
Active: 11728564 kB
Inactive: 45899640 kB
Active(anon): 6995028 kB
Inactive(anon): 32896636 kB
Active(file): 4733536 kB
Inactive(file): 13003004 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 5242872 kB
SwapFree: 4810116 kB
Dirty: 932 kB
Writeback: 0 kB
AnonPages: 36971484 kB
Mapped: 5428548 kB
Shmem: 2890100 kB
KReclaimable: 119628 kB
Slab: 176372 kB
SReclaimable: 119628 kB
SUnreclaim: 56744 kB
KernelStack: 5440 kB
PageTables: 95616 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 37947708 kB
Committed_AS: 47836332 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 131952 kB
VmallocChunk: 0 kB
Percpu: 2368 kB
HardwareCorrupted: 0 kB
AnonHugePages: 35928064 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
FileHugePages: 0 kB
FilePmdMapped: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 10468764 kB
DirectMap2M: 54542336 kB
DirectMap1G: 4194304 kB
galera-2 | CHANGED | rc=0 >>
MemTotal: 65409688 kB
MemFree: 839632 kB
MemAvailable: 21014964 kB
Buffers: 7416 kB
Cached: 22012092 kB
SwapCached: 3324808 kB
Active: 19884616 kB
Inactive: 44045340 kB
Active(anon): 9721772 kB
Inactive(anon): 33551148 kB
Active(file): 10162844 kB
Inactive(file): 10494192 kB
Unevictable: 26128 kB
Mlocked: 26128 kB
SwapTotal: 5242872 kB
SwapFree: 30304 kB
Dirty: 556 kB
Writeback: 0 kB
AnonPages: 38609448 kB
Mapped: 4674536 kB
Shmem: 1358228 kB
KReclaimable: 220688 kB
Slab: 280964 kB
SReclaimable: 220688 kB
SUnreclaim: 60276 kB
KernelStack: 5632 kB
PageTables: 109356 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 37947716 kB
Committed_AS: 45948600 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 132192 kB
VmallocChunk: 0 kB
Percpu: 2352 kB
HardwareCorrupted: 0 kB
AnonHugePages: 32049152 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
FileHugePages: 0 kB
FilePmdMapped: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 345500 kB
DirectMap2M: 29014016 kB
DirectMap1G: 39845888 kB
Colin Charles
cyusedfzfb
Hi all, Colin,
Nice to see you replying here! We're still benefitting from the lessons learned in your EMEA galera couse, 24/25 juni. :-) We appreciate your reply here.
Since our initial posting, we have learned that the unexpected swap usage is not galera-specific. Look at another example:
RHEL 8.10, running mariadb-server-utils.x86_64 3:10.5.22-1.module+el8.8.0+20134+a92c7654:
[user@db ~]$ free -g
total used free shared buff/cache available
Mem: 187 9 11 0 165 175and:
Swap: 3 3 0
top - 08:52:46 up 39 days, 15:50, 2 users, load average: 1.70, 2.05, 1.99
Tasks: 616 total, 2 running, 614 sleeping, 0 stopped, 0 zombie
%Cpu(s): 6.0 us, 0.8 sy, 0.0 ni, 88.2 id, 4.7 wa, 0.1 hi, 0.1 si, 0.0 st
MiB Mem : 191529.7 total, 11761.7 free, 9764.9 used, 170003.1 buff/cache
MiB Swap: 4096.0 total, 1006.1 free, 3089.9 used. 179429.5 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND SWAP
1952811 mysql 20 0 14.4g 5.0g 15172 S 65.7 2.7 9663:14 mysqld 1.7g
2934963 root 20 0 5208 2056 1408 S 8.9 0.0 10:39.29 gzip 0
2934962 root 20 0 33612 8268 6960 S 7.9 0.0 2:35.31 mysqldump 0
while:
[user@db ~]$ cat /proc/sys/vm/swappinessWe could (we should) allocate more ram to mysql, but the point is: there is plenty of RAM available, and has been, since boot, always. And yet, mariadb is using "a lot" of swap.
1
Why? Can you explain? Can anyone here explain? Swappiness set to
1...is mariadb ignoring swappiness..?
I guess this has no performance impact, probably the swapped
pages are not actually used, but zabbix does not like 100% swap
usage. And frankly: neither do I.
Anyone?
MJ
--
You received this message because you are subscribed to the Google Groups "codership" group.
To unsubscribe from this group and stop receiving emails from it, send an email to codership-tea...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/codership-team/67621191-b953-4db8-92c9-dd99800fe607n%40googlegroups.com.
mj | galera
Why is this happening?
Vinicius Grippa
Antes de imprimir pense em seu compromisso com o meio ambiente.
Before printing, think about your commitment to the Environment.
To view this discussion on the web visit https://groups.google.com/d/msgid/codership-team/c02381d7-b2c9-436f-88ec-699af7cd0a8fn%40googlegroups.com.
cyusedfzfb
Hi all,
Just to follow up this thread, so others can benefit as well.
This turns out to be an actual issue on RHEL8: https://access.redhat.com/solutions/6785021
We have set the suggested sysctl vm.force_cgroup_v2_swappiness=1 and all unexpected swapping has disappeared.
Perhaps (I hope) this info will help others with similar issues.
MJ
To view this discussion on the web visit https://groups.google.com/d/msgid/codership-team/c02381d7-b2c9-436f-88ec-699af7cd0a8fn%40googlegroups.com.