Kqueue and threading

[Willem Jan Withagen]

Hi,

Ceph uses a lot of threading, and for any part of it communication it
uses epoll(), which FreeBSD does not use. For that there was already a
EvenKqueue implementation.

But I think I'm now running into:
The kqueue() system call creates a new kernel event queue and
returns a descriptor. The queue is not inherited by a child
created with fork(2). However, if rfork(2) is called without the
RFFDG flag, then the descriptor table is shared, which will allow
sharing of the kqueue between two processes.

Kqueue descriptors are created and events are added, but then the
process starts other threads and expects the kqueue-id to be valid there.

However adding more events returns an error, also waiting on the ID for
events to happen returns an error (descriptor invalid)

Threading is done with 2 different constructions:
std::thread
and creating Workers

Would any of these qualify with the quoted warning? and invalidate the
descriptor table?

If not, how can I (easily) debug the reason why my descriptors go invalid?


thanx,
--WjW



_______________________________________________
freebsd...@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hacke...@freebsd.org"

[Eric Badger]

On 10/01/2016 13:02, Willem Jan Withagen wrote:
> Hi,
>
> Ceph uses a lot of threading, and for any part of it communication it
> uses epoll(), which FreeBSD does not use. For that there was already a
> EvenKqueue implementation.
>
> But I think I'm now running into:
> The kqueue() system call creates a new kernel event queue and
> returns a descriptor. The queue is not inherited by a child
> created with fork(2). However, if rfork(2) is called without the
> RFFDG flag, then the descriptor table is shared, which will allow
> sharing of the kqueue between two processes.
>
> Kqueue descriptors are created and events are added, but then the
> process starts other threads and expects the kqueue-id to be valid there.
>
> However adding more events returns an error, also waiting on the ID for
> events to happen returns an error (descriptor invalid)
>
> Threading is done with 2 different constructions:
> std::thread
> and creating Workers
>
> Would any of these qualify with the quoted warning? and invalidate the
> descriptor table?
>
> If not, how can I (easily) debug the reason why my descriptors go invalid?
>

Sharing a kqueue between threads of a process works. Are the workers
created using rfork without RFFDG as suggested in the manpage? I've
never had reason to do this, but a quick test seems to indicate that it
works as advertised. A normal fork closes the kqueue file descriptor. If
you suspect that's what's happening, you might run "procstat -f {worker
pid}" to see if file descriptors with a "k" (kqueue) in the "T" (type)
column appear (if not, they were presumably closed on the fork).

Eric

[Slawa Olhovchenkov]

On Mon, Oct 03, 2016 at 12:48:03PM -0500, Eric Badger wrote:

> Sharing a kqueue between threads of a process works. Are the workers

Can you comment PR207261?

> created using rfork without RFFDG as suggested in the manpage? I've
> never had reason to do this, but a quick test seems to indicate that it
> works as advertised. A normal fork closes the kqueue file descriptor. If
> you suspect that's what's happening, you might run "procstat -f {worker
> pid}" to see if file descriptors with a "k" (kqueue) in the "T" (type)
> column appear (if not, they were presumably closed on the fork).
>
> Eric
>

[Eric Badger]

On 10/03/2016 01:09 PM, Slawa Olhovchenkov wrote:
> On Mon, Oct 03, 2016 at 12:48:03PM -0500, Eric Badger wrote:
>
>> Sharing a kqueue between threads of a process works. Are the workers
>
> Can you comment PR207261?

Hmm, I don't think I have the netmap expertise to comment.

Eric

[Willem Jan Withagen]

Yup, I know what the man page writes. :) But now the trick question
is: how does the C++ worker/thread library function....

Some of the code actually does: std:move on a lambda-function, which
should instantiate a complete fresh environment. And that would
certainly qualify for expected closure of the kq descriptors.
OTOH it keeps regular file descriptors available in the thread.

I guess I'll have to start testing with a few simple C++ programs to
get things clear.

[Adrian Chadd]

netmap is a special snowflake case, as the last time I checked a whole
bunch of assumptions are made about the process/thread contexts doing
the work.



-adrian