kern/50094: not all threads woken up when multiple threads waiting with keven on same kqueue

[cme...@cmeerw.org]

>Number: 50094
>Category: kern
>Synopsis: not all threads woken up when multiple threads waiting with keven on same kqueue
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: kern-bug-people
>State: open
>Class: sw-bug
>Submitter-Id: net
>Arrival-Date: Sun Jul 26 19:10:00 +0000 2015
>Originator: Christof Meerwald
>Release: netbsd-7
>Organization:
>Environment:
NetBSD droid 7.0_RC1 NetBSD 7.0_RC1 (ODROID-C1.201507211940Z) evbarm
>Description:
as reported in http://mail-index.netbsd.org/tech-userlevel/2015/07/26/msg009262.html

I have got multiple threads waiting on a single kqueue and want to
wake up all these threads at some point (before exiting). I believe
the way to do it is to have a dummy socket/pipe added to the kqueue
and close the other end (I have also found
http://thread.gmane.org/gmane.os.netbsd.devel.kernel/28051 which seems
to agree on that approach).

But when I am actually testing this approach, some threads don't seem
to be woken up and keep waiting in the kevent syscall. Source code for
a test case is available from
http://svn.cmeerw.net/src/nginetd/trunk/test/kqtest-wakeup.cc

I have tested on my ODROID-C1 (quad core ARMv7) with netbsd-7 (build
from a few days ago) and seen the problem occur with 3 or more
threads.

This has also been confirmed in netbsd-6 on x86 in http://mail-index.netbsd.org/tech-userlevel/2015/07/26/msg009257.html
>How-To-Repeat:
compile with g++ -pthread and run specifying the number of threads (e.g. 16) as the argument. Test case should consistently terminate after a 1 second sleep.


#include <stdio.h>
#include <stdlib.h>

#include <errno.h>
#include <fcntl.h>
#include <unistd.h>

#include <arpa/inet.h>
#include <sys/event.h>
#include <sys/fcntl.h>
#include <sys/socket.h>
#include <sys/time.h>

#include <pthread.h>


namespace
{
extern "C" void *worker(void *arg)
{
const int kqid = *static_cast<int *>(arg);

struct kevent events[16];
bool done = false;

while (!done)
{
int rc_kevent = kevent(kqid, NULL, 0,
events, sizeof(events)/sizeof(*events),
NULL);
done = rc_kevent != 0;
}
}
}


int main(int argc, const char * const argv[])
{
const int nr_threads = (argc > 1) ? atoi(argv[1]) : 1;

int kqid(kqueue());

int syncsockets[2];
socketpair(AF_UNIX, SOCK_STREAM, 0, syncsockets);
{
struct kevent const events[] = {
{ syncsockets[1], EVFILT_READ, EV_ADD, 0, 0, 0 },
};
kevent(kqid, events, 1, NULL, 0, NULL);
}

pthread_t *threads = new pthread_t[nr_threads];
for (int i = 0; i < nr_threads; ++i)
{
pthread_create(threads + i, NULL, worker, &kqid);
}

sleep(1);

close(syncsockets[0]);

for (int i = 0; i < nr_threads; ++i)
{
pthread_join(threads[i], NULL);
}

close(syncsockets[1]);
close(kqid);

delete [] threads;

return 0;
}
>Fix:


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-...@muc.de

[Christof Meerwald]

The following reply was made to PR kern/50094; it has been noted by GNATS.

From: Christof Meerwald <cme...@cmeerw.org>
To: gnats...@NetBSD.org
Cc:
Subject: Re: kern/50094: not all threads woken up when multiple threads
waiting with keven on same kqueue

Date: Mon, 27 Jul 2015 21:17:58 +0200

I think I now know what is going on here.

For events that have neither EV_ONESHOT and EV_CLEAR set, the knote is
removed from the kqueue, the kqueue is unlocked, the event is
re-checked, the kqueue is locked again and the knote is re-added to
the kqueue.

The problem here is that there is a good chance that some of the
threads will see the kqueue being empty when woken up (while another
thread has removed the knote from the kqueue and is re-checking the
event) and go back to sleep.

A potential straightforward fix appears to be to add a cv_broadcast
call after the knote has been re-added to the kqueue.


Index: kern_event.c
===================================================================
RCS file: /cvsroot/src/sys/kern/kern_event.c,v
retrieving revision 1.83
diff -u -r1.83 kern_event.c
--- kern_event.c 2 Mar 2015 19:24:53 -0000 1.83
+++ kern_event.c 27 Jul 2015 19:05:30 -0000
@@ -1266,6 +1266,7 @@
kq->kq_count++;
kn->kn_status |= KN_QUEUED;
kq_check(kq);
+ cv_broadcast(&kq->kq_cv);
}
if (nkev == kevcnt) {
/* do copyouts in kevcnt chunks */


Christof

--

http://cmeerw.org sip:cmeerw at cmeerw.org
mailto:cmeerw at cmeerw.org xmpp:cmeerw at cmeerw.org