I think we are making good progress on this, but I do not believe we are in a position to commit to an API that we cannot change later. We don't have enough experience and shouldn't be designing against a deadline. I don't believe Go 1.3 should include os/fsnotify.
As an experiment, I started looking at what the go command would need to do its cache. I have built the seed of each interesting piece but have not grown them enough to connect. Perhaps I will get that done for Go 1.3 (with some OS-specific code in what I am calling 'go tool buildcache' instead of importing a portable package named 'os/fsnotify'), but perhaps not. Here is what I learned from the experiment.
The go command needs to watch for changes to source files but also for changes to the directories containing source files, so that it knows when a file has been added or removed, or when a new directory has been created that might shadow another. For example, if we cache the build result for $GOPATH/src/asdf, but then $GOROOT/src/pkg/asdf is created, the build result for $GOPATH/src/asdf must be dropped.
Godoc is a relatively small program. It is built from 102 packages built from 582 source files. We certainly want to be able to build programs larger than godoc. It appears that Linux inotify will let you watch individual directories for changes within that directory, so for godoc you are looking at a little over 102 inotify watches. For OS X, fsevents will let you watch whole subtrees, so for godoc built from 1 GOROOT and 1 GOPATH entry you are looking at 2 fsevents watches. For Microsoft Windows, FindFirstChangeNotification looks like it might be usable similar to fsevents. For Solaris, file event notifications (FEN) only apply to individual files or directories, and watching a directory inode does not appear to tell you about modifications made to files in that directory, so for godoc you are looking at almost 700 FEN watches. That might take a little while to set up, but assuming the kernel has no hard limits, it should be fine. Speaking of limits... For BSD (or OS X if you don't like fsevents), kqueue has the same "enumerate every file or directory" requirement as Solaris FEN, but you give them to the kernel not as file names but as file descriptors. It appears that the file descriptor must remain open while you are watching that file, so the per-process file descriptor limit imposes a limit on the number of things you can watch. Worse, the per-system file descriptor limit imposes a limit on the number of things anyone on the system can watch. The typical kernel limit on a BSD is only on the order of 10,000 file descriptors for the entire machine. Put a few users building significantly larger programs than godoc on the machine - even at different times, assuming the cache sits in the background - and the machine is out of file descriptors. If we go for the least common denominator, using inotify or fsevents as if it were FEN or kqueue, then we create significant unnecessary work for the system. If we don't, then fsnotify possibly can't be used on FEN or kqueue. Perhaps those systems should be ignored until they provide a more scalable notification system. Plan 9 won't have one (probably ever) so fsnotify is always going to be a best effort, might say I can't help you kind of package.
There is also a question of synchronization. Ideally you want to have a "Sync()" method that guarantees that when it returns, all events for modifications made before the call to Sync have now been delivered. Then a go command can have the buildcache call Sync before starting to give results, and we know it will not have missed recent changes and be delivering stale results. I know how to build a Sync from the fsevents API. I am less sure about the others; probably it is possible but it requires some thought. It also affects the API, because you need to know when the events triggered by the Sync have stopped. If the events are coming over a channel, you need some kind of sentinel event marking the sync position. This is all unclear to me. I think fsevents gives some clear guarantees about delivery that let you build Sync. Windows FindFirstChangeNotification seems not to give any at all. For example, you find out about file changes in a directory by watching the mtime on the files, but the docs say: "The operating system detects a change to the last write-time only when the file is written to the disk. For operating systems that use extensive caching, detection occurs only when the cache is sufficiently flushed." That's clearly useless for interactive go command build result caching.
In the non-recursive watch model provided by inotify, the idea of watching for directories that might yet be created is also quite subtle. You need to watch the parents of those directories, but the parents might not exist yet either, so you need to walk up the tree until you find a parent that does exist, and then you need to watch for subdirectories there, and then only some of them you care about starting to watch their children. And then maybe one of those directories will be removed, and you need to put the watch back on the parent. I've sat down to write the logic multiple times and backed off each time. I can't quite wrap my head around it. I'm sure it's possible, but it's so subtle it probably needs to be provided by the library, not reimplemented incorrectly by each user. We can't go full-blown recursive by default, because I still believe the cost there is too expensive.
Perhaps the right middle ground API is to allow watching only of directories, and to be told only "something in that directory changed". Fsevents can watch the root, inotify can watch individual directories, FEN can be told about directories and all the files in it. Maybe kqueue can try until it runs the machine out of file descriptors and then back off. Or maybe not.
In all honesty, we might have to decide that systems that don't at least provide a directory-based watching just lose out, so that BSD and maybe even Solaris systems just aren't supported. It would be nice to know if BSD or Solaris are thinking about higher-level watching (and in the case of BSD, not tying watches to file descriptors). I did not appreciate the wide variation in APIs and the complexity of turning the APIs into something you can actually program correctly against.
These are the sorts of questions I see about the os/fsnotify API and the reasons I think we can't commit to something in the next two weeks. I do hope we can get something done for the next release, and I encourage work to continue in the go.exp subrepo.
Russ