Swap Usage
Doug Hardie
_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stabl...@freebsd.org"
Chris H
> I have several FreeBSD 9.3 systems that are using swap and I can’t figure
> out what is doing it. The key system has 6GB swap and currently it has over
> 2GB in use. ps shows only a kernel module [intr] with a W status. Obviously
> that isn’t using the space. No other process shows a W in its status. I
> suspect this is somewhat related to the use of mmap in one application.
> However, all of the mmaps in that application are file backed and thus
> shouldn’t use swap. How do I figure out what that swap space is being used
> for?
Maybe top(1)?
top -P
for example. At least you could see who's chewing all your memory. Which
should be a good clue as to who's responsible for swap usage.
--Chris
Doug Hardie
> On 29 July 2015, at 18:57, Chris H <bsd-...@bsdforge.com> wrote:
>
> On Wed, 29 Jul 2015 17:41:33 -0700 Doug Hardie <bc...@lafn.org> wrote
>
>> I have several FreeBSD 9.3 systems that are using swap and I can’t figure
>> out what is doing it. The key system has 6GB swap and currently it has over
>> 2GB in use. ps shows only a kernel module [intr] with a W status. Obviously
>> that isn’t using the space. No other process shows a W in its status. I
>> suspect this is somewhat related to the use of mmap in one application.
>> However, all of the mmaps in that application are file backed and thus
>> shouldn’t use swap. How do I figure out what that swap space is being used
>> for?
> Maybe top(1)?
> top -P
> for example. At least you could see who's chewing all your memory. Which
> should be a good clue as to who's responsible for swap usage.
UFS although I don’t see how that could make a difference. Swap doesn’t use a regular file system. However, the kernel must track the actual usage using something like a simple file system. There must be a way to investigate it.
Top doesn’t show anything unusual. The most VM used is by the process that uses a lot of mmap space that is all file backed. If it was another process that was partially swapped then the ps status should show a W. Only the one does.
Doug Hardie
> On 29 July 2015, at 17:41, Doug Hardie <bc...@lafn.org> wrote:
>
> I have several FreeBSD 9.3 systems that are using swap and I can’t figure out what is doing it. The key system has 6GB swap and currently it has over 2GB in use. ps shows only a kernel module [intr] with a W status. Obviously that isn’t using the space. No other process shows a W in its status. I suspect this is somewhat related to the use of mmap in one application. However, all of the mmaps in that application are file backed and thus shouldn’t use swap. How do I figure out what that swap space is being used for?
UFS although that shouldn’t matter. Swap does not use a file system. If it did, it would be easy to figure out.
Top doesn’t show much.
Peter Jeremy
On 2015-Jul-29 17:41:33 -0700, Doug Hardie <bc...@lafn.org> wrote:
>I have several FreeBSD 9.3 systems that are using swap and I can’t
>figure out what is doing it. The key system has 6GB swap and
>currently it has over 2GB in use.
this)? If not, this just means that at some time in the past, the
system was under memory pressure and paged some process memory out.
Since then, that memory hasn't been touched so the system hasn't paged
it in.
>ps shows only a kernel module
>[intr] with a W status.
under severe RAM pressure. Normally, the system will just page out
inactive parts of a processes address space - and none of the ps flags
will show this.
> How do I figure out what that swap space is being used for?
the number of resident pages within each swap-backed region, any
pages in that region that have been touched but are not resident
are on the swap device but any pages that have never been touched
aren't counted at all.
--
Peter Jeremy
Doug Hardie
Bingo. procstat shows the problem. The process that I suspected has a large number of entries like:
650 0x834c00000 0x835800000 rw- 0 0 1 0 ---- sw
650 0x835800000 0x835c00000 rw- 0 0 1 0 ---- sw
650 0x835c00000 0x837c00000 rw- 1 0 1 0 ---- sw
I don’t know whats in those areas yet. If I were to kill the process with SIGABRT would the core dump show those areas? I might be able to figure out what they are from that.
Thanks for the pointer.
Doug Hardie
I believe I have found a bug in FreeBSD 9.3 mmap. The following test program (named test.c) does an mmap to itself and monitors the memory objects created with procstat. After the mmap, munamp is called and the objects are checked again. The objects created from the mmap do not get purged, but remain. Eventually, after enough have been created, they start becoming type sw and eventually the system runs out of swap.
The output:
zool# ./test
31459 0x8049000 0x804a000 rw- 1 0 1 0 CN-- df
31459 0x2805f000 0x28069000 rw- 10 0 1 0 C--- df
31459 0x28186000 0x281ad000 rw- 13 0 1 0 CN-- df
31459 0xbfbdf000 0xbfbff000 rwx 3 0 1 0 C--D df
----------------------------------
31459 0x8049000 0x804a000 rw- 1 0 1 0 CN-- df
31459 0x804a000 0x8400000 rw- 1 0 1 0 CN-- df
31459 0x2805f000 0x28069000 rw- 10 0 1 0 CN-- df
31459 0x28186000 0x281ad000 rw- 14 0 1 0 CN-- df
31459 0x28400000 0x28800000 rw- 6 0 1 0 CN-- df
31459 0xbfbdf000 0xbfbff000 rwx 3 0 1 0 C--D df
----------------------------------
31459 0x8049000 0x804a000 rw- 1 0 1 0 CN-- df
31459 0x804a000 0x8400000 rw- 1 0 1 0 CN-- df
31459 0x2805f000 0x28069000 rw- 10 0 1 0 CN-- df
31459 0x28186000 0x281ad000 rw- 14 0 1 0 CN-- df
31459 0x28400000 0x28800000 rw- 6 0 1 0 CN-- df
31459 0xbfbdf000 0xbfbff000 rwx 3 0 1 0 C--D df
----------------------------------
31459 0x8049000 0x804a000 rw- 1 0 1 0 CN-- df
31459 0x804a000 0x8400000 rw- 1 0 1 0 CN-- df
31459 0x2805f000 0x28069000 rw- 10 0 1 0 CN-- df
31459 0x28186000 0x281ad000 rw- 14 0 1 0 CN-- df
31459 0x28400000 0x28800000 rw- 6 0 1 0 CN-- df
31459 0xbfbdf000 0xbfbff000 rwx 3 0 1 0 C--D df
The program:
int main (int argc, char *argv[])
{
int rc, pid, fd;
char *cp;
char cmd[1024];
pid = getpid ();
sprintf (cmd, "procstat -v %d | grep df", pid);
fflush (stdout);
rc = system (cmd);
fflush (stdout);
printf ("----------------------------------\n");
fd = open ("./test.c", O_RDWR);
cp = mmap (0, 100, PROT_READ|PROT_WRITE, MAP_FILE, fd, 0);
if (cp == MAP_FAILED)
{
printf ("mmap error %s\n", strerror(errno));
exit (1);
}
fflush (stdout);
rc = system (cmd);
fflush (stdout);
printf ("----------------------------------\n");
rc = munmap (cp, 100);
if (rc == -1)
{
printf ("munmap error %s\n", strerror(errno));
exit (1);
}
fflush (stdout);
rc = system (cmd);
fflush (stdout);
printf ("----------------------------------\n");
close (fd);
fflush (stdout);
rc = system (cmd);
fflush (stdout);
Doug Hardie
zool# more test.c
#include <stdio.h>
#include <fcntl.h>
#include <string.h>
#include <stdlib.h>
#include <errno.h>
#include <sys/mman.h>
int main (int argc, char *argv[])
{
int rc, pid, fd;
char *cp;
char cmd[1024];
pid = getpid ();
sprintf (cmd, "procstat -v %d | grep df", pid);
fflush (stdout);
rc = system (cmd);
fflush (stdout);
printf ("----------------------------------\n");
fd = open ("./test.c", O_RDWR);
fflush (stdout);
rc = system (cmd);
fflush (stdout);
printf ("----------------------------------\n");
close (fd);
fflush (stdout);
rc = system (cmd);
fflush (stdout);
}
The output is:
zool# ./test
85860 0x8049000 0x804a000 rw- 1 0 1 0 CN-- df
85860 0x2805f000 0x28069000 rw- 10 0 1 0 C--- df
85860 0x28186000 0x281ad000 rw- 13 0 1 0 CN-- df
85860 0xbfbdf000 0xbfbff000 rwx 3 0 1 0 C--D df
----------------------------------
85860 0x8049000 0x804a000 rw- 1 0 1 0 CN-- df
85860 0x804a000 0x8400000 rw- 1 0 1 0 CN-- df
85860 0x2805f000 0x28069000 rw- 10 0 1 0 CN-- df
85860 0x28186000 0x281ad000 rw- 14 0 1 0 CN-- df
85860 0x28400000 0x28800000 rw- 6 0 1 0 CN-- df
85860 0xbfbdf000 0xbfbff000 rwx 3 0 1 0 C--D df
----------------------------------
85860 0x8049000 0x804a000 rw- 1 0 1 0 CN-- df
85860 0x804a000 0x8400000 rw- 1 0 1 0 CN-- df
85860 0x2805f000 0x28069000 rw- 10 0 1 0 CN-- df
85860 0x28186000 0x281ad000 rw- 14 0 1 0 CN-- df
85860 0x28400000 0x28800000 rw- 6 0 1 0 CN-- df
85860 0xbfbdf000 0xbfbff000 rwx 3 0 1 0 C--D df
Open creates 2 memory allocations, one in low memory and one in high. Close does not remove them. Somewhere in the source I found a note that said that close on files didn’t require any action. However, those two memory allocations do need to get freed.