timeline for os/fsnotify

[Nathan Youngman]

Greetings,

I'm wondering where os/fsnotify is at and where it needs to be to make the March 1st code freeze... and with ample time to also use it in cmd/go.

My effort towards this has been fairly sporadic thus far. Sorry about that.

The CL/pull request that I started on GitHub in September is getting fairly close. It has recursive watching and various filters that many people (including myself) have implemented on top in the past. There are still some optimizations we can do to save file handles when filters are specified, or to take advantage of OS level support for recursive watching on Windows and OS X (with a future FSEvents adapter).

My CL also has some interfaces to make it easier to write unit tests. Until now it only had 5 seconds worth of integration tests (which I'm skipping on the short run). 

I'd like to get these changes into the hands of people prior to Go 1.3... so that it's well tested prior to release. There's also the small matter of getting all this over to Mercurial (go.exp or os/fsnotify) and the issues into the Go issue tracker. How do we want to do this, and when?

Finally, Russ Cox mentioned doing an API design doc. Do we have a template/starting point? Can anyone just start the Google Doc?

Nathan.

[Ian Lance Taylor]

On Fri, Jan 3, 2014 at 11:24 PM, Nathan Youngman <n...@nathany.com> wrote:
>
> Finally, Russ Cox mentioned doing an API design doc. Do we have a
> template/starting point? Can anyone just start the Google Doc?

I'm not aware of any existing design doc for os/fsnotify. Anyone can
start it. There is no specific template, but there are a few existing
docs, such as http://golang.org/s/go12nil.

Ian

[Russ Cox]

I would like to get started on os/fsnotify. Let's work out what the API should be and then it will just be a matter of getting it implemented on all the operating systems.

I have started CL https://codereview.appspot.com/48310043 for the API discussion. Please comment there (you can add yourself to the CC list by clicking 'reply' and then please uncheck 'send mail' if you're not also saying anything).

Thanks.

Russ

[Nathan Youngman]

Hi Russ,

Thanks for starting a CL. I left a lengthy comment there. Would it make sense to still do up an API design document with some high level goals and links to reference material for various operating systems? I'm willing to take a stab at a first draft, though I suspect it will have more questions than answers.

All the feedback on your CL is great to see. That's one reason I can't wait to fully move fsnotify development from GitHub over to go.exp or os/fsnotify. Having CI against multiple OSes with the Build Status dasboard is also a big plus, and something I never found a great solution for. There are still a few things I'm working on that would be nice to wrap up before transferring over though. I believe that can happen well within the time we use the API design planned out.

Nathan.

[Russ Cox]

On Tue, Jan 7, 2014 at 12:08 AM, Nathan Youngman <n...@nathany.com> wrote:

Hi Russ,

Thanks for starting a CL. I left a lengthy comment there. Would it make sense to still do up an API design document with some high level goals and links to reference material for various operating systems? I'm willing to take a stab at a first draft, though I suspect it will have more questions than answers.

Sure, that'd be great.

Thanks.

Russ

[Nathan Youngman]

This evening I took a stab at a first draft for an API proposal. Hopefully it's a useful place to start:

http://goo.gl/MrYxyA

I never got too far into the implementation/code, just pasting some of the API I had been working on. There is a link to your CL.

Just seven weeks to code freeze. Yikes!

Nathan.

[minux]

There are some issues with the proposed API that I think it's worthwhile to discuss on

this list.

Background info (copied from the proposal):

Adapter    OS       File  Dir  Recursive

inotify     Linux       Y     Y    N

kqueue   BSD        Y     N    N

RDCW    Windows  Y    Y     Y

FSEvents OS X     ?     Y    Always

Issues:

1. If kqueue doesn't support watching directories, supporting watching directories will not be race-free on

*BSD. And we need to determine a policy when to rescan for directory changes (e.g. when to do readdir

again?). This problem bugs me enough that I even think of dropping dir watching support for *BSD until

they supported watching dir changes.

2. Because of the kqueue's inability to watch directories, for directory watches, we'd better filter which files

to watch before adding the actual watch for that file. I'd suggest we take a user supplied Filter function to

do that. This way seems better and more flexible than adding explicit options for ignoring hidden files/dirs,

and shell wildcard filtering.

Implementation issues:

should we dynamically switch between FSEvents and kqueue implementation on OS X?

Other issues:

1. IIRC iOS doesn't support FSEvents.

2. We need to consider Solaris support.

On Sat, Jan 11, 2014 at 1:02 AM, Nathan Youngman <n...@nathany.com> wrote:

This evening I took a stab at a first draft for an API proposal. Hopefully it's a useful place to start:

http://goo.gl/MrYxyA

I never got too far into the implementation/code, just pasting some of the API I had been working on. There is a link to your CL.

Just seven weeks to code freeze. Yikes!

IIUC as long as the work (that is important for 1.3) begins before the code freeze it's ok

for code to go in after the freeze according to the release planing docs.

[Nathan Youngman]

minux,

Thanks for going over my proposal. 

I should've mentioned that I'm no expert on kqueue or any of the other notification adapters. This is just based on my reading of the current fsnotify code base, commit messages and issues. 

Just like I wasn't aware of Mkdir events and was using os.Stat on Create events to do an IsDir() test. Whoops. 

I do like the idea of passing in a filter function. It's something I was thinking about recently, but I've yet to sketch out what the code would look like. 

Afaik, it should be possible to do everything with FSEvents on OS X, which should be generally more efficient. 

Watching a single file with a recursive directory watcher might feel a bit silly. Still, manually filtering out irrelevant events should work just fine, and it can coalesce events with a latency setting. I suspect watching an entire directory recursively will be much more common anyway, particularly for cmd/go. https://github.com/howeyc/fsnotify/issues/54

Nathan. 

--
Nathan Youngman
Email: n...@nathany.com
Web: http://www.nathany.com

[Nathan Youngman]

Hi,

So that's a start, but I still have a lot of questions around the API (Events channel vs. Iterator, etc.). Minux gave some good comments to the document already. http://goo.gl/MrYxyA

My other questions are around the process of getting this implemented.

If we're adding adapters for FSEvents, Solaris and a user-space recursive watcher, what will be our plan to have users test fsnotify before Go 1.3 beta/rc/final? Will we publish the new API/implementation at go.exp/fsnotify for use in Go 1.2?

Will we be starting with the code in go.exp/fsnotify? The API needs to be altered and it doesn't yet implement FSEvents, among other things. Or is the preference to write a new implementation, moving code over from go.exp/fsnotify as needed?

Would you like me (or Chris Howey) to submit a CL to get go.exp/fsnotify up-to-date with what's currently on GitHub?

I have an implementation of a user-space recursive watcher and a set of event filters with unit tests. The API will change and some code may get deleted. There are also a few TODOs left in the code (eg. RemoveRecursiveWatcher). Should I get this code over to go.exp as a starting point? If so, is it okay if I break it down into a few CLs for independent code review?

Would you like me to copy the fsnotify issues currently on GitHub to the Go issue tracker? We can also encourage contributors to get involved here instead.

Nathan