associate multiple sources with a single socket file,descriptor within kqueue

[Marius Zwicker]

There was a report on the xdispatch mailing list concerning a bug in
libkqueue discovered when re-using a socket fd within different dispatch
source objects [1].

You can find the detailed description below. From my understanding the
desired behaviour should not be an error but the old event should be
overridden, as documented in the man pages [2]:

>/ EV_ADD Adds the event to the kqueue. Re-adding an existing event will modify the parameters of
/>/ the original event, and not result in a duplicate entry. Adding an event automatically
/>/ enables it, unless overridden by the EV_DISABLE flag./


I will try to create a unit test exposing that bug and hopefully showing
the behaviour on Mac OS X. What's your opinion on that?

--

Marius Zwicker B.Sc.


MLBA - TEAM
www.mlba-team.de

PGP 8843 4AEF 1993 DD73 9358 22A1 9FBB 936E E09F 097D
X.509 C3:23:E9:B3:C1:23:4F:A6:20:DF:86:A8:7B:30:9E:30:49:C5:9E:53


[1] http://opensource.mlba-team.de/pipermail/libxdispatch/2012-September/000049.html
[2] https://developer.apple.com/library/mac/#documentation/Darwin/Reference/ManPages/man2/kqueue.2.html



We are using xdispatch for some worthy causes and encountered a rather bad
(from functionality standpoiont) breakage in libkqueue.

Namely, when try to associate multiple sources with a single socket file
descriptor, libkqueue breaks badly, the best case being this assertion:

----

bug1: xdispatch_0.7.0_src/libkqueue/src/common/knote.c:65:
void knote_release(struct knote *): Assertion
`(&kn->kn_mtx)->mtx_status == 1 && pthread_equal((&kn->kn_mtx)->mtx_owner,
(pthread_self()))' failed.
----


Our usage pattern should be perfectly valid and rather common:
-----

int sock_fd;
...
...

disp_read = dispatch_source_create(DISPATCH_SOURCE_TYPE_READ, sock_fd,
0, conn->queue);
...
disp_write = dispatch_source_create(DISPATCH_SOURCE_TYPE_WRITE, sock_fd,
0, conn->queue);

-----

What really leads to that assertion, as far as we can see, is shortcoming
in libkqueue design, as indicated by the following error:

----

KQ [47334214375184]: epoll_update(): op=1 fd=6 events= { data = 0x1301310,
events = EPOLLOUT EPOLLONESHOT }

KQ [47334214375184]: epoll_update(): epoll_ctl(2): File exists
----


It appears, that libkqueue always does EPOLL_CTL_ADD when event is registered,
even when the proper operation will be EPOLL_CTL_MOD with appropriate arguments.

[Mark Heily]

This problem was fixed in the stable branch a while back. Your bug
report references code that is only found in the trunk, so I assume
you are using the trunk. The trunk is buggy and not usable in it's
current state. It contains some experimental performance enhancements
that are not ready for prime time.

Please stop using the libkqueue trunk in xdispatch, and switch over to
the stable branch. I could use your help getting the Windows-specific
stuff backported into the stable branch.

Thanks,

- Mark

Long term, I am looking at

[Mark Heily]

On Thu, Oct 25, 2012 at 9:30 PM, Mark Heily <ma...@heily.com> wrote:
>
> Long term, I am looking at

Oops, accidentally sent the message before the postscript was finished :)

I was saying that long-term, I am looking at introducing a new API in
libkqueue 2.0 that will be a very thin wrapper around native OS
functionality, rather than providing a complete implementation of
kqueue for all platforms.

Here is the basic idea:


#ifdef BSD
#define kqueue_t int
#define kqueue_lite kqueue
#define kevent_lite kevent
#elif defined(__linux__)

/* Very thin wrapper around epoll() */

#elif defined(__SunOS)

/* Very thin wrapper around port_wait() */

#elif defined (__Win32)

/* Very thin wrapper around WaitForMultipleObjects() */

#else

/* Map to heavyweight libkqueue 1.0 functions */

#endif