Kqueue file monitoring on freebsd and darwin

[Dave Cheney]

Dear golang-dev,

I would like to expose the use of kqueue for file change monitoring on
freebsd and darwin. Currently, on freebsd and darwin syscall.Kqueue is
wrapped in net.pollster which is then wrapped in a more general
net.pollserver. Before I invest too much time I wanted to ask the
opinion of the group on possible approaches, possibly:

a. Refactor net/fd_darwin.go and net/fd_freebsd into a new package,
os/kqueue, then refactor the os/kqueue package to support other types
of event flags enabling file change monitoring. This would probably
result in the creation of os/epoll for linux (although this wouldn't
be used to replace os/ionotify)
-or-
b. Build the required functionality to support file change monitoring
in a new package, os/kqueue, then at a later stage port net/fd_darwin
and net/fd_freebsd to use os/kqueue.

My preference is for the first option, but the second has merit as
well. I would be very appreciative of any advice the group could give.

Cheers

Dave

[Russ Cox]

Let's not refactor anything in net just yet.
I would suggest writing a new package os/kqueue
and then we should look at merging that and os/inotify
into a single os/fsnotify package.

Russ

[Dave Cheney]

Understood, touching net is very high risk. I'll look to adding the
minimum needed to implement file notifications in os/kqueue.

Cheers

Dave

[Hector Chu]

Okay, so now there will be a BSD specific API in os to do file change
notification in addition to the Linux inotify that is already there.
I think that now is the best time to also do the os/fsnotify API for
OS-agnostic file change notification. I tried to submit a change
implementing inotify for Windows last month and was told that we
should define os/fsnotify and implement against that. Otherwise
Windows will be left out in the cold.

[Dave Cheney]

Hi Hector,

I agree this isn't optimal. As my original proposal was to implement file change monitoring rather than exposing a generalised kqueue implementation, does it make sense to instead implement os/fsnotify for kqueue OSs with a view to incorporating win32 and Linux thereafter?

Cheers

Dace

Sent from my C64

[Hector Chu]

Hi Dave,

That sounds better, and we should agree on the set of flags supported
by os/fsnotify as not all OSes expose the same flags. The following
is what I wrote on 17 Feb concerning this matter:

"I think a sensible interface for Go would have us only watching
directories. We probably shouldn't allow recursive directory
watching.
Then we should expose a limited subset of the most useful flags in
inotify.
These would probably be create, delete, modify and rename (within the
directory only). Conveniently this is the intersection of flags that
Linux
and Windows supports (of course we should check that these
notifications are
also supported under the other OSes). We should also merge the Event
and
Error channels. Is there anything else that we need to address?"

[Dave Cheney]

> "I think a sensible interface for Go would have us only watching
> directories.

Agreed

> We probably shouldn't allow recursive directory
> watching.

If possible recursive directory watching would be ideal, think dropbox.go

> Then we should expose a limited subset of the most useful flags in
> inotify.
> These would probably be create, delete, modify and rename (within the
> directory only).  Conveniently this is the intersection of flags that
> Linux
> and Windows supports

Having come from a Java world where a large part of any codebase is
copying fields from one object to another I would be interested in
seeing if the API could avoid copying data from the internal structure
like syscall.Kevent_t to a portable structure that normalises the
returned data. I'm hoping something in the style of os.Waitmsg is
possible.

Cheers

Dave

[Hector Chu]

> > We probably shouldn't allow recursive directory
> > watching.
>
> If possible recursive directory watching would be ideal, think dropbox.go

I read that Linux doesn't support recursive directory watching. Try
searching for the phrase:
"Inotify monitoring of directories is not recursive: to monitor
subdirectories under a directory, additional watches must be created."

> > Then we should expose a limited subset of the most useful flags in
> > inotify.
> > These would probably be create, delete, modify and rename (within the
> > directory only).  Conveniently this is the intersection of flags that
> > Linux
> > and Windows supports
>
> Having come from a Java world where a large part of any codebase is
> copying fields from one object to another I would be interested in
> seeing if the API could avoid copying data from the internal structure
> like syscall.Kevent_t to a portable structure that normalises the
> returned data. I'm hoping something in the style of os.Waitmsg is
> possible.

How would we write OS-agnostic code against such an interface?