inotify/lsyncd limitations

[Adi Andreias]

Hello,

What do you think are the limitations of inotify/lsyncd in terms of:
(1) number of files/directories watched
(2) change frequency

I know that this is a pretty broad question and there's no clear
answer for every case. More specific:
- is it feasible to lsyncd on 10 million files? or 1 million
directories? If not what would be a safe number and what kind of
resources are eaten? lsyncd memory buffers, anything else?
- is feasible to run lsyncd in optimal conditions if there are
10/100/1000 file changes per second?

I'm currently running lsyncd test in these conditions:
- 1.1 million files
- 84 000 directories
- peak at 7 file changes per second, probably an average of 1 changes
per second (a lot of no-changes-second). These numbers are from
looking at tail -f /var/log/lsyncd (not precise numbers).

lsyncd process uses about 6MB of RES memory and processor usage is
virtually zero.
lsyncd initializations (after the very first rsync was done) takes
about 5 minutes to build file list and about 3 minutes to do the
rsync. During file list building rsync uses less than 8% of processor
and rare peaks of 43%. With an I/O around 25%. But system is loaded
with the actual web sites as well, of course.

Would it be safe to use it in these conditions:
- about 20 million files
- about 340 000 directories
- peak at 30 file changes per second, average of 10 files changes per
second

And are there some real world use cases with some numbers where lsyncd
runs successfully?

Thank you

[Axel Kittenberger]

I'm personally using lsyncd in office with 24k files, so far below
your requirements.

Honestly about limitations: I don't know! You tell me :-)

For most parts I consider linux kernel limits to be more striking than
lsyncd limits. Older kernerls had an watch lmit of 8k that is
directories not files, which has already been lifted... on my desktop
computer its now 512k directories. This can be viewn/configured in

/proc/sys/fs/inotify/max_user_watches

The event queue is another limited.

The max watches limit is a hard limit, that is, if lsyncd would need
to watch more directories than this it can't do more than to give up.

The event queue is a soft limit, if events get raised faster than
lsyncd attending to them, the kernel creates an overflow. Its soft as
it would not be much of a drama, since all lsyncd would have to do is
to clear the event queue, and just resync everything. However this has
not been coded yet. Currently overflowed queues result into lost
syncs.

About events/second I consider rsync processing time to be the bottle
neck. Since lsyncd halts until an rsync operation is finished it
really depends on rsyncs runtimes, if rsync takes longer to process
than events showing up, its like a basin where water input is faster
than drainage.

Kind regards, Axel

[Adi Andreias]

Hi Axel,

Thanks for your reply.

Didn't find any reference of /proc/sys/fs/inotify/max_user_watches
upper limit.
I mean, can I configure it to monitor 2 million directories?

I guess it would be safer for lsyncd just to write changed files/
directories to a file and have separate process that reads the file
and does rsync.

I'm not familiar with Linux programming event model, but I assume the
"action" can run in a separate thread so that it doesn't block the
event handler that gets system notifications.
This action thread can run continuously and have some communication
between main (even handler) thread and "action" thread, a queue for
the action thread to process.

[Axel Kittenberger]

> Didn't find any reference of /proc/sys/fs/inotify/max_user_watches
> upper limit.
> I mean, can I configure it to monitor 2 million directories?

Depends on kernel, I can.

----
root@woodstock:/home/axel# echo "4000000" >
/proc/sys/fs/inotify/max_user_watches
root@woodstock:/home/axel# cat /proc/sys/fs/inotify/max_user_watches
4000000
----
My limit is somewhere at 800 billion directories,
---
root@woodstock:/home/axel# echo "800000000000" >
/proc/sys/fs/inotify/max_user_watches
root@woodstock:/home/axel# echo "810000000000" >
/proc/sys/fs/inotify/max_user_watches
bash: echo: write error: Invalid argument
------

I however do not know how many kernel memory is eaten by the lot of
directories. About your usecase, if any possible I would just try it
out and see what happens :-)

> I guess it would be safer for lsyncd just to write changed files/
> directories to a file and have separate process that reads the file
> and does rsync.
>
> I'm not familiar with Linux programming event model, but I assume the
> "action" can run in a separate thread so that it doesn't block the
> event handler that gets system notifications.
> This action thread can run continuously and have some communication
> between main (even handler) thread and "action" thread, a queue for
> the action thread to process.

Well it depends, as long as you only want 1 rsync operation to run at
the same time, you don't gain anything, altough you would retrain
control a tad earlier, lsyncd would nevertheless have to wait for the
first rsync operation to complete.

In case you do not want to wait for rsync to finish, but start several
operations at once, you do not need to change the code big time also,
just skip the waiting routine. Altough this cannot be commandline
configured at the moment, this is just a minor code edit. This is a
design decission, I for one wanted to limit load by limiting to 1
rsync operation at a time. I would advice against using several at
once, as you would need special logic to determine if the operations
would get in conflict, like syncing a directory and a parent direcoty
or so... this logic is not coded.

- Axel

[Adi Andreias]

Hi Axel,

From some old release notes of inotify (2004-2005, didn't find newer
releases included in the kernel),
inotify_add_watch uses 40 bytes (for watch data) + 512 bytes (for
inode data), which is 552 bytes per watch.

On 2 million directories watched, it's more than 1GB of memory used.
I have asked on several mailing lists if this is true in kernel
2.6.18.

Do you happen to know this info?

Thanks

[Axel Kittenberger]

Sorry, I don't know.

Kind regards, Axel

[Adi Andreias]

In case someone will seek this later, in kenel 2.6.29.1 file
Documentation\filesystems\proc.txt says:

max_user_watches
----------------

Every epoll file descriptor can store a number of files to be
monitored
for event readiness. Each one of these monitored files constitutes a
"watch".
This configuration option sets the maximum number of "watches" that
are
allowed for each user.
Each "watch" costs roughly 90 bytes on a 32bit kernel, and roughly 160
bytes
on a 64bit one.
The current default value for max_user_watches is the 1/32 of the
available
low memory, divided for the "watch" cost in bytes.

[Axel Kittenberger]

Thank you very much for looking this up!

Note: lsyncd does not watch singular files, but whole directories. So
its total number of directories that counts, not files.

Kind regards, Axel