Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
AIX system memory consumption is killing the nerves on DB servers?
1,154 views
Skip to first unread message
Stefan K
Jul 15, 2013, 2:31:50 PM7/15/13
to
Hi all,
has anyone else noticed the huge amount of system memory since AIX6?
It's a while ago since AIX 6.1 came out, but today the problem is still the same.
Take this example. A dedicated DB server with 50 GB ram. Tablespace containers are DMS file on JFS2 filesystems, most of them are CIO enabled.
DB2 is consuming statically 30 GB, taken from db2top: memory. There are no big variations, utility heap and sorting are nothing compared to the size of bufferpools.
This system is providing a NFS share needed for some file exchange, most of them are 1-time accessed files.
Apart from that there are no other significant memory consumers.
The system memory consumption is increasing over time. DB2 and the system are up and running since 200+ days.
By looking at /proc/sys/fs/jfs2/memory_usage, it can be seen that 8 GB ram are used for just inode and metadata cache. 8 GB, on a system with ca. 8000-9000 open files. This is 900kbytes per open file in average?
AIX PMRs just recommend to leave the settings as they are for best performance, and anyway this is all working as designed.
But the system goes quite a lot into paging space, and DB2 is just consuming 60% of the total ram.
On DB servers we really want to use the memory for DB2, as far as possible.
It's clear AIX needs some memory to be able to run.
But the memory consumption for these caches just go far beyond of what is required.
I really want IBM to update the defaults and have them to be more open for different customer requirements.
Does anyone agree? Thanks a lot for any comment.
Best regards / Stefan
$ uptime
06:09PM up 219 days
--topas_nmon--c=CPU--------------Host=mydbhost-------Refresh=2 secs---17:35.30--
Memory -----------------------------------------------------------------------
Physical PageSpace | pages/sec In Out | FileSystemCache
% Used 96.4% 46.6% | to Paging Space 0.0 0.0 | (numperm) 1.7%
% Free 3.6% 53.4% | to File System 278.5 43.5 | Process 72.6%
MB Used 51309.0MB 19103.8MB | Page Scans 0.0 | System 22.1%
MB Free 1939.0MB 21856.2MB | Page Cycles 0.0 | Free 3.6%
Total(MB) 53248.0MB 40960.0MB | Page Steals 0.0 | ------
| Page Faults 242.5 | Total 100.0%
------------------------------------------------------------ | numclient 1.7%
Min/Maxperm 1546MB( 3%) 46387MB( 87%) <--% of RAM | maxclient 87.1%
Min/Maxfree 4096 4608 Total Virtual 92.0GB | User 71.1%
Min/Maxpgahead 2 8 Accessed Virtual 49.1GB 53.4%| Pinned 23.2%
-------------------------------------------------------------------------------
$ cat /proc/sys/fs/jfs2/memory_usage
metadata cache: 2.476.605.440
inode cache: 5.538.643.968
total: 8.015.249.408
$ oslevel -s
6100-06-06-1140
$ root ioo -L | grep 400
j2_inodeCacheSize 400 400 400 1 1000 D
j2_metadataCacheSize 400 400 400 1 1000 D
$ root lsof > hirn
$ wc -l hirn
8991 hirn
8200 open files without TCP connections.
Reduce the two settings for a test and be surprised how much free ram you are going to have ! e.g.
root ioo -o j2_inodeCacheSize=40
root ioo -o j2_metadataCacheSize=40
has anyone else noticed the huge amount of system memory since AIX6?
It's a while ago since AIX 6.1 came out, but today the problem is still the same.
Take this example. A dedicated DB server with 50 GB ram. Tablespace containers are DMS file on JFS2 filesystems, most of them are CIO enabled.
DB2 is consuming statically 30 GB, taken from db2top: memory. There are no big variations, utility heap and sorting are nothing compared to the size of bufferpools.
This system is providing a NFS share needed for some file exchange, most of them are 1-time accessed files.
Apart from that there are no other significant memory consumers.
The system memory consumption is increasing over time. DB2 and the system are up and running since 200+ days.
By looking at /proc/sys/fs/jfs2/memory_usage, it can be seen that 8 GB ram are used for just inode and metadata cache. 8 GB, on a system with ca. 8000-9000 open files. This is 900kbytes per open file in average?
AIX PMRs just recommend to leave the settings as they are for best performance, and anyway this is all working as designed.
But the system goes quite a lot into paging space, and DB2 is just consuming 60% of the total ram.
On DB servers we really want to use the memory for DB2, as far as possible.
It's clear AIX needs some memory to be able to run.
But the memory consumption for these caches just go far beyond of what is required.
I really want IBM to update the defaults and have them to be more open for different customer requirements.
Does anyone agree? Thanks a lot for any comment.
Best regards / Stefan
$ uptime
06:09PM up 219 days
--topas_nmon--c=CPU--------------Host=mydbhost-------Refresh=2 secs---17:35.30--
Memory -----------------------------------------------------------------------
Physical PageSpace | pages/sec In Out | FileSystemCache
% Used 96.4% 46.6% | to Paging Space 0.0 0.0 | (numperm) 1.7%
% Free 3.6% 53.4% | to File System 278.5 43.5 | Process 72.6%
MB Used 51309.0MB 19103.8MB | Page Scans 0.0 | System 22.1%
MB Free 1939.0MB 21856.2MB | Page Cycles 0.0 | Free 3.6%
Total(MB) 53248.0MB 40960.0MB | Page Steals 0.0 | ------
| Page Faults 242.5 | Total 100.0%
------------------------------------------------------------ | numclient 1.7%
Min/Maxperm 1546MB( 3%) 46387MB( 87%) <--% of RAM | maxclient 87.1%
Min/Maxfree 4096 4608 Total Virtual 92.0GB | User 71.1%
Min/Maxpgahead 2 8 Accessed Virtual 49.1GB 53.4%| Pinned 23.2%
-------------------------------------------------------------------------------
$ cat /proc/sys/fs/jfs2/memory_usage
metadata cache: 2.476.605.440
inode cache: 5.538.643.968
total: 8.015.249.408
$ oslevel -s
6100-06-06-1140
$ root ioo -L | grep 400
j2_inodeCacheSize 400 400 400 1 1000 D
j2_metadataCacheSize 400 400 400 1 1000 D
$ root lsof > hirn
$ wc -l hirn
8991 hirn
8200 open files without TCP connections.
Reduce the two settings for a test and be surprised how much free ram you are going to have ! e.g.
root ioo -o j2_inodeCacheSize=40
root ioo -o j2_metadataCacheSize=40
Rakesh Gupta
Jul 16, 2013, 1:34:24 PM7/16/13
to
http://pic.dhe.ibm.com/infocenter/aix/v6r1/topic/com.ibm.aix.prftungd/doc/prftungd/support_pinned_mem.htm
Hopes that solves the mystery of high memory usage
Stefan K
Jul 17, 2013, 9:49:13 AM7/17/13
to
Thanks for your reply. I understand the pinned pages.
But the key fact is that AIX is caching an useless amount of inodes and metadata, and consuming far too much memory for this.
I just saw that the AIX 7 default for these parameters is half’ed. To me this seems still to be too much.
Somehow these caches should be intelligently managed, so that the volume correspond to the number of currently open and used files, and maybe the number of newly opened files. Another factor to take into consideration is the available memory. I think it makes no sense to cache millions of 1-time used inodes, if the system goes into paging space at the same time.
Best regards / Stefan
But the key fact is that AIX is caching an useless amount of inodes and metadata, and consuming far too much memory for this.
I just saw that the AIX 7 default for these parameters is half’ed. To me this seems still to be too much.
Somehow these caches should be intelligently managed, so that the volume correspond to the number of currently open and used files, and maybe the number of newly opened files. Another factor to take into consideration is the available memory. I think it makes no sense to cache millions of 1-time used inodes, if the system goes into paging space at the same time.
Best regards / Stefan
davebw...@gmail.com
Apr 13, 2015, 9:41:23 AM4/13/15
to
Hi Stefan
Thanks for sharing this.
I am wondering what the outcome of this was if you can remember ? ie. did you manage to reduce the memory consumption on AIX 6.1 ?
We have similar issues with oracle and AIX 6.1 - our servers are constantly paging.
Thanks
Dave Browne
Thanks for sharing this.
I am wondering what the outcome of this was if you can remember ? ie. did you manage to reduce the memory consumption on AIX 6.1 ?
We have similar issues with oracle and AIX 6.1 - our servers are constantly paging.
Thanks
Dave Browne
Stefan K
Apr 13, 2015, 10:57:43 AM4/13/15
to
Hi Dave, I saw IBM halfed the value for AIX 7. The meaning of the values is a bit of black AIX magic. It may still consume 10% of ram.
If a server is used as DB server, meaning it shouldn´t use millions of files, and you notice too high consumption, just lower it and keep monitoring the system. Not sure what process is maintaining this list, but you don´t want to see e.g. lrud 100% busy.
Cheers / Stefan
If a server is used as DB server, meaning it shouldn´t use millions of files, and you notice too high consumption, just lower it and keep monitoring the system. Not sure what process is maintaining this list, but you don´t want to see e.g. lrud 100% busy.
Cheers / Stefan
0 new messages