Linux implementation of UV_FS_EVENT_RECURSIVE?

[Joshua Warner]

Hi,

It looks like, at least in the latest release (0.11.14), UV_FS_EVENT_RECURSIVE isn't implemented on linux.  I'd be interested in taking on the implementation, if the community is responsive.

On the surface, the implementation should be pretty straight-forward:

* watch the base directory

* list it's children

** recurse as necessary

The only thing left is to un-watch and re-watch directories as we get add and remove notifications for them (non-trivial, but straight-forward).

On the other hand, is this something that should be handled by the application built on top of libuv (or on top of node)?

Thoughts?

Thanks,

Joshua

[Fedor Indutny]

Hello Joshua,

While having no doubts in usefulness of this feature for some libuv
users, I have strong objections for it to be in the libuv itself,
because it is: quite high-level, will require complicated C code to
implement it, and doesn't require any interaction with internals.

Nevertheless, it'd be good to see a third-party library that would add
this feature! Probably, after seeing it I'll reconsider my comments
above.

Thank you for reaching us out,
Fedor.

> --
> You received this message because you are subscribed to the Google Groups
> "libuv" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to libuv+un...@googlegroups.com.
> To post to this group, send email to li...@googlegroups.com.
> Visit this group at http://groups.google.com/group/libuv.
> For more options, visit https://groups.google.com/groups/opt_out.

[Ben Noordhuis]

There are some practical issues with your proposal, the most important
one being that the numbers of watchers you can create is quite
limited:

# sysctl -a | grep fs.inotify.max_user
fs.inotify.max_user_instances = 128
fs.inotify.max_user_watches = 8192

Those are the defaults on my Fedora system and they're fairly typical
for most Linux installs. (fanotify doesn't have those limitations but
it has others: you need to be root in order to use it and it's not
available in older kernels.)

I appreciate that you want to take the time to implement this but I
don't think it's really feasible.

[Joshua Warner]

Fedor and Ben,

 

There are some practical issues with your proposal, the most important
one being that the numbers of watchers you can create is quite
limited

By analogy, I know that the MacOSx FSEvents API docs say you shouldn't completely depend on it anyway - there are apparently various cases where the API can fail to forward all change events to the application.  Thus, a correctly implemented application, whether using that native API or built on top of libuv, shouldn't depend on the completeness of the notifications, and should occasionally rescan the directory manually for changes.  Though I cringe to say it, we could document this as being "one of those cases" where the API breaks down.

Alternatively, since we actually know on linux exactly what directories we failed to watch because of limiting (presumably the inotify APIs return errors when you go over the limit, though I haven't verified) - we can actually tell the API user what parts of the tree we failed to recursively watch.  In addition to UV_RENAME and UV_CHANGE, we add UV_WATCH_ERROR (I'm not picky on the name), which indicates a problem watching the contents of the directory.

 

I appreciate that you want to take the time to implement this but I
don't think it's really feasible.

 

In the simple cases, it's not very difficult.  I have a toy implementation of (most of) this, and the added complexity of the recursive watch is ~300 lines (of c++, which is probably a little smaller than the equivalent C).

Yes, it's not a fully reliable API - but as I've mentioned above, you already lost that with the FSEvents implementation.

> Nevertheless, it'd be good to see a third-party library that would add 
> this feature! Probably, after seeing it I'll reconsider my comments 
> above. 

As far as third-party implementations go, there's the TUP build system - which implements very close to the algorithm I've described.  It's not libuv-compatible, but it shows that software in the wild is already using this approach.  Specifically: https://github.com/gittup/tup/blob/2365023235a35721d3f713e0816e23c8c4e3db4a/src/tup/monitor/inotify.c

To be clear, the question is not whether I'm going to implement a recursive watch API on top of inotify - it's whether I'll implement it in libuv or separately.  I think that API, even flawed, could add a lot of value to libuv.

-Joshua

[Fedor Indutny]

Ben,

Your word on it?

I haven't really changed my mind after looking at that code :) I still
think that it's too complex for libuv, and, also, I don't want to
maintain it (and know that I would, if we will merge it) ;)

[Ben Noordhuis]

Sorry, but the answer has to be 'no'. I strongly dislike
functionality that only works some of the time. If anything, you've
convinced me to remove the UV_FS_EVENT_RECURSIVE flag altogether.

[Joshua Warner]

Sorry, but the answer has to be 'no'.  I strongly dislike
functionality that only works some of the time.  If anything, you've
convinced me to remove the UV_FS_EVENT_RECURSIVE flag altogether.

Fair enough.  It's unfortunate that the platform APIs are in such a poor state at the moment.

Thanks for your consideration.