fcntl, wstat, and O_NONBLOCK

[Barret Rhoden]

last week ron and i talked about implementing parts of fcntl with
wstat. there's still a few issues with it.

for the most part, there's no problem with passing O_NONBLOCK with open
calls. the issue pops up when we want to change flags like O_NONBLOCK
after a file/chan has been opened.

fcntl setfl option is supposed to change OS file flags. that OS struct
file* is mostly equiv to the chan in plan 9. i think wstat is for
changing what i call "disk files", i.e. /foo/bar or an #ip
conversation. they're different objects, so using wstat might be wrong.

it could be possible to allow fcntl to set chan flags. i actually do
that right now, but i blacklist some flags. i.e:

int fd_setfl(int fd, int flags)
{
ERRSTACK(1);
struct chan *c;

if (waserror()) {
poperror();
return -1;
}
c = fdtochan(&current->open_files, fd, -1, 0, 1);
if (cexternal_flags_differ(flags, c->flag, O_CLOEXEC)) {
/* TODO: The whole CCEXEC / O_CLOEXEC on 9ns needs work */
set_errno(EINVAL);
error("can't toggle O_CLOEXEC with setfl");
}
if (cexternal_flags_differ(flags, c->flag, O_PATH)) {
set_errno(EINVAL);
error("can't toggle O_PATH with setfl");
}
if (cexternal_flags_differ(flags, c->flag, O_NONBLOCK)) {
/* If we want to let them toggle NONBLOCK, it'd require a device op */
set_errno(EINVAL);
error("can't set O_NONBLOCK, use a device-specific ctl command");
}
c->flag = (c->flag & ~CEXTERNAL_FLAGS) | (flags & CEXTERNAL_FLAGS);
cclose(c);
poperror();
return 0;
}


the reason i did this was because #ip's non-blocking applies to the
conversation, not the chan. we could set the c->flag, but it would
have no effect (c->flag is set, but #ip won't change the
conversation). or we could come up with a way to talk to the device
and tell it its fcntl/chan flags are changing.

for other devices, like the upcoming #eventfd, simply changing the chan
flag is sufficient (if we allowed it), since that's how that device
works.

a fix to make this issue go away would be to have #ip be able to do
non-blocking on a per-chan basis, instead of per-conversation, but
that actually avoids the issue. and i'm not sure if that's doable or
not without a lot of pain, or even if it's the right thing (might not
be).

any ideas of a nice 9p way to tell a device about a chan flag change,
but not a device file change? does that even make sense across 9p?

barring that, i'm tempted to allow toggling of O_NONBLOCK, and just
deal with the fact that some devices won't allow a toggle. right now,
#ip toggles get a helpful errstr. barring some per-device logic, you
won't know if the toggle was effective or not without knowing about the
underlying device's rules.

btw, the reason i keep bringing up 9p is that any device level interface
(i.e. a function pointer in struct dev) requires thinking about 9p and
mounts, since many (but not all) of the functions in struct dev are
equivalent to 9p commands.

it's worth thinking about O_APPEND too. you could do it on the chan on
the host only, but that's lousy if you have multiple writers. 9p stat
has a DMAPPEND bit, which appears to apply to the file as a whole, so
it looks like you can't have some writers APPEND, others not. not a
huge deal either way, but in this case DMAPPEND is a device file
attribute, not a chan flag. which is similar to NONBLOCK in #ip - it
applies to the device file, and not the chan flag.

so if APPEND is a device file attribute in plan 9 and a 'chan flag' in
linux, we're already off and doing our own thing, so we can pick what
makes the most sense.

that bring up the question of whether or not NONBLOCK should be a
device file attribute, a channel flag, or both. for #ip, it wants
nonblock on the device file. other devices may be similar. for
#eventfd, it's fine with either. in the linux world (and possibly for
apps we want to port), it's in the chan flag.

going back to #mnt for a second, let's think about a couple scenarios:

say we have a regular disk FS on a remote machine, mounted locally
through #mnt. if we say NONBLOCK, we actually care about that in
#mnt. the RPC for #mnt can block, independently of whatever is on the
other side of 9p. so we might want to support NONBLOCK within #mnt
(details are tricky), which could apply to the chan flag. (as a
side note, O_NONBLOCK doesn't prevent *all* forms of blocking in #ip,
it prevents blocking on the underlying conversation qio. some form of
"never block no matter what" would probably require a chan flag.)

but say instead we have a remote machine's #ip stack mounted locally
through our #mnt on /net. we might want to set up a nonblocking
conversation, which requires passing NONBLOCK through to the device on
the other end (either through wstat for the device file or through some
other message to apply to the chan).

anyway, if anyone has any insights, let me know. for now, i'm probably
going to leave it where you can set O_NONBLOCK during open, or via ctl
message for #ip, and otherwise you get an error. but as we fix up
parts of 9ns and integrate more with the glibc/linux world, these are
things we'll need to address.

btw, this has all been about the kernel interface. if you make a glibc
fcntl/setfl call on a BSD socket, i put something together that
intercepts the call and writes a "set nonblock" ctl command. so socket
apps should be okay. but if you try to create an eventfd, don't create
it with NONBLOCK, and later try to fcntl it, you'll fail.

barret