lsyncd uses too much memory

[rodericliu]

I am using lsyncd to watch my mail directory. There are many
directories in the mail directory. After lsyncd has been running for
several days, I found that the lsyncd uses large amount of the memory
and it's still increasing.

Here are some snapshot from my server:

Top:

top - 06:56:45 up 382 days, 16:57, 1 user, load average: 0.81, 0.75,
0.70
Tasks: 127 total, 1 running, 126 sleeping, 0 stopped, 0 zombie
Cpu(s): 7.2%us, 18.8%sy, 0.0%ni, 74.0%id, 0.0%wa, 0.0%hi,
0.0%si, 0.0%st
Mem: 5985620k total, 5946760k used, 38860k free, 316736k
buffers
Swap: 2096472k total, 118364k used, 1978108k free, 2073572k
cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
28741 root 20 0 2953m 2.9g 652 S 23 50.4 2779:50
lsyncd

Lsyncd Status:

Lsyncd status report at Tue Jul 19 07:09:47 2011

Sync1 source=/mnt/data/vmail/
There are 0 delays
Excluding:
nothing.

Inotify watching 763434 directories

I add the following to lsyncd startup script:

echo 2000000 > /proc/sys/fs/inotify/max_user_watches
echo 163840 > /proc/sys/fs/inotify/max_queued_events

Why would lsyncd uses so much memory? Can you give me some information
about this? Thank you!

[Sven Hartrumpf]

Hi.

Mon, 18 Jul 2011 22:19:18 -0700 (PDT), rodericliu wrote:
> top - 06:56:45 up 382 days

Don't you need any kernel patches? :-)

> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 28741 root 20 0 2953m 2.9g 652 S 23 50.4 2779:50 lsyncd

...
> Inotify watching 763434 directories
...

> Why would lsyncd uses so much memory?

Each directory watch needs up to 1 KB,
see https://groups.google.com/d/msg/lsyncd/KZfQE9VUZFw/twTMjxWa41wJ
That would explain 0.8 GB of your 2.9 GB, the rest is
really strange.

Sven

[rodericliu]

Yes, the server has been running for a long time.

uname -a

Linux rsid-a-30 2.6.27.19-5-default #1 SMP 2009-02-28 04:40:21 +0100 x86_64 x86_64 x86_64 GNU/Linux

Maybe it's too old, I will do some test on the new kernel version server.

Thanks for ur replay!

[Axel Kittenberger]

Its the Linux´ inotify interface. Every directories needs an inotify
watch, which needs 1K memory on a 64-bit system. Additionally Lsyncd
saves for every directory it watches a) the inotify handle (4 bytes)
b) the directories full path (size depends on the length of the
string) then add a few bytes for the hash tables to find the path
names quickly by the handle and vice versa.

For a while fanotify looked a very interesting alternative which
allows to make one watch for a whole filesystem. But unfortunately it
does not report move events at all, making it unusable for Lsyncd.

Currently we have two major notification systems on Linux` both
tailored to be a specific use case:
* inotify: made for GUIs who display one directory and are only
interested in a few directories
* fanotify: made for malware protection, and trimmed very much to those needs.

None is really suitable for indexing or syncing mechanisms like
Lsyncd, like for example Mac OSX´ fsevents interface which allows with
one handle to watch everything on the system. Apple wrote it for their
hotspot index machine, and on OSX 10.5.5 Lsyncd can hijacks that
interface.

Inotify works, but uses tons of memory for large trees.

Get the Linux´ kernel fixed and Lsyncd will improve on that ;-)

kind regards, Axel

> --
> You received this message because you are subscribed to the Google Groups "lsyncd" group.
> To post to this group, send email to lsy...@googlegroups.com.
> To unsubscribe from this group, send email to lsyncd+un...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/lsyncd?hl=en.
>
>