Strange EPOLLOUT|EPOLLET behaviour (spurious events)

[Christof Meerwald]

Hi,

I am seeing some unexpected behaviour with EPOLLOUT in edge-triggered
mode - some example code is available from
http://svn.cmeerw.net/src/nginetd/trunk/test/eptest-out.c

What seems to be happening is that I get an EPOLLOUT event after each
send on the socket (even in edge-triggered mode). This is what strace
shows me:

epoll_create(1024) = 3
socketpair(PF_FILE, SOCK_DGRAM, 0, [4, 5]) = 0
fcntl(5, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(5, F_SETFL, O_RDWR|O_NONBLOCK) = 0
epoll_ctl(3, EPOLL_CTL_ADD, 5, {EPOLLIN|EPOLLOUT|EPOLLET, {u32=5, u64=5}}) = 0
sendto(5, "\0\0\0\0", 4, 0, NULL, 0) = 4
recvfrom(4, "\0\0\0\0", 4, 0, NULL, NULL) = 4
epoll_wait(3, {{EPOLLOUT, {u32=5, u64=5}}}, 16, 0) = 1
sendto(5, "\1\0\0\0", 4, 0, NULL, 0) = 4
recvfrom(4, "\1\0\0\0", 4, 0, NULL, NULL) = 4
epoll_wait(3, {{EPOLLOUT, {u32=5, u64=5}}}, 16, 0) = 1
sendto(5, "\2\0\0\0", 4, 0, NULL, 0) = 4
recvfrom(4, "\2\0\0\0", 4, 0, NULL, NULL) = 4
epoll_wait(3, {{EPOLLOUT, {u32=5, u64=5}}}, 16, 0) = 1

But as I am using EPOLLET, I wouldn't expect to get any EPOLLOUT
events on that socket as send never returns EAGAIN (and there isn't
any state transition).

BTW, I have seen https://lkml.org/lkml/2011/11/17/48 , but my case is
different (as there isn't any EPOLLIN event on that socket either).

Am I missing something here?


Christof

--
http://cmeerw.org sip:cmeerw at cmeerw.org
mailto:cmeerw at cmeerw.org xmpp:cmeerw at cmeerw.org

[Rainer Weikusat]

Christof Meerwald <NOSPAM-see...@usenet.cmeerw.org> writes:
> I am seeing some unexpected behaviour with EPOLLOUT in edge-triggered
> mode - some example code is available from
> http://svn.cmeerw.net/src/nginetd/trunk/test/eptest-out.c
>
> What seems to be happening is that I get an EPOLLOUT event after each
> send on the socket (even in edge-triggered mode). This is what strace
> shows me:

[...]

> But as I am using EPOLLET, I wouldn't expect to get any EPOLLOUT
> events on that socket as send never returns EAGAIN (and there isn't
> any state transition).

That's expected behaviour: You get EPOLLOUT as soon as 'a send
operation' completed. You will also get EPOLLIN whenever the state
transitions from 'buffer empty' to 'buffer not empty' irregardless of
read ever returning EAGAIN. The kernel doesn't keep track of which
state changes where communicated to a userspace application. If this
is a problem, you should initially have all notifications disabled and
only enable them after 'something failed with EAGAIN'. It is necessary
to be somewhat careful in this cases so that state transitions aren't
lost. In pseudo-code, the logic could look like this:

1. perform operation
2. EAGAIN? if not, goto done
3. enable event
4. perform operation again
5. EAGAIN? if yes, goto 'wait for notification'
6. disable event
7. done

This may, of course, result in lots and lots of epoll_ctl calls ...

[Christof Meerwald]

On Wed, 25 Apr 2012 23:11:51 +0100, Rainer Weikusat wrote:
> Christof Meerwald <NOSPAM-see...@usenet.cmeerw.org> writes:
>> But as I am using EPOLLET, I wouldn't expect to get any EPOLLOUT
>> events on that socket as send never returns EAGAIN (and there isn't
>> any state transition).
> That's expected behaviour: You get EPOLLOUT as soon as 'a send
> operation' completed.

Well, that's what appears to be happening, but is it documented
anywhere?

Looking at http://linux.die.net/man/7/epoll I see:

"The suggested way to use epoll as an edge-triggered (EPOLLET)
interface is as follows: with nonblocking file descriptors; and by
waiting for an event only after read(2) or write(2) return EAGAIN."

and

"Do I need to continuously read/write a file descriptor until EAGAIN
when using the EPOLLET flag (edge-triggered behavior)? Receiving an
event from epoll_wait(2) should suggest to you that such file
descriptor is ready for the requested I/O operation. You must consider
it ready until the next (nonblocking) read/write yields EAGAIN."

Nothing there suggests that I should get an EPOLLOUT as soon as "a
send operation" completed.

[Rainer Weikusat]

I'm seriously sorry that the technology you are fumbling around with
doesn't work like you believe it should and apologize for assuming
that your misdirected complaint about the contents of some Linux
manpage was actually a question. I'm going to take suitable technical
measures for ensuring that this never happens again.

[Christof Meerwald]

On 26 Apr 2012 06:01:40 GMT, Christof Meerwald wrote:
> On Wed, 25 Apr 2012 23:11:51 +0100, Rainer Weikusat wrote:
>> Christof Meerwald <NOSPAM-see...@usenet.cmeerw.org> writes:
>>> But as I am using EPOLLET, I wouldn't expect to get any EPOLLOUT
>>> events on that socket as send never returns EAGAIN (and there isn't
>>> any state transition).
>> That's expected behaviour: You get EPOLLOUT as soon as 'a send
>> operation' completed.
> Well, that's what appears to be happening, but is it documented
> anywhere?

Actually, it's not as simple as that...