how do I optimize watchdog to reduce filtering logic

0 views

Skip to first unread message

David Bull

unread,

12:15 AM (10 hours ago) 12:15 AM

to watchdog-python

For example purposes say I want to watch a folder ParentFolder that has subfolders Sub1, Sub2, Sub3.

Part 1

If I want to create a watch on ParentFolder and all the subfolders I could add a watch with:

observer.schedule(handler, "ParentFolder", recursive=True)

or I could add a watch on each folder independently:

observer.schedule(handler, "ParentFolder") observer.schedule(handler, "ParentFolder/Sub1")

observer.schedule(handler, "ParentFolder/Sub2")

observer.schedule(handler, "ParentFolder/Sub3")

Functionally these do the same thing but I'm wondering if the second method uses more resources under the hood by creating separate schedules for each instead of just one?

The motivation behind this question is: say I didn't want to watch Sub2. I could add a recursive watch to ParentFolder and add some of my own filtering logic in a FileSystemEventHandler (ie. if event.src_path == "ParentFolder/Sub2": pass) or use one of the MatchingEventHandlers to do the same.

Or I could manually subscribe to the parent folder and subfolders independently leaving out `Sub2'.

observer.schedule(handler, "ParentFolder", recursive=False)

observer.schedule(handler, "ParentFolder/Sub1", recursive=True) observer.schedule(handler, "ParentFolder/Sub3", recursive=True)

I don't like the idea of scheduling a watch on a folder only to then have to run filtering logic in the event handler just to ignore it. I would rather do the filtering work up front in scheduling the watches and would like to understand the finder details of how scheduling works under the hood.

Of course this example is trivial but for large file systems optimizations can add up.

Part 2

Along the same lines of thinking as above and now considering the different event types. When it comes to the types of events (ie. created, moved, deleted, etc) is there any kind of filtering available on the side of scheduling watches/notifications with the underlying OS? It seems to me that pretty much all of the filtering is done within the event handlers (after the fact). In this case event types are dealt with or ignored through implementations on the event handler methods (ie. on_any_event(), on_moved(), on_created(), etc). As above I would prefer to do the filtering on the scheduling side rather than the event handling side.

For example: In my case I'm not so interested in modified events for folders but I am interested in modified events for files so it's not as simple as not implementing a on_modified() method in the event handler. I would rather just not be notified by the OS in the first place for folder modified events.

When scheduling a watch on the file system with the operating system (in my case I'm interested in inotify with linux OSs) does the watchdog backend automatically get notifications for all event types? Or is it possible when scheduling a watch for the OS to only notify watchdog for certain event types? and if so is it possible and/or practical to do this on a per folder/file basis?