Multiple loops per process on Windows and IOCP
Stephen Halter
int uv_tcp_write(uv_loop_t* loop,
uv_write_t* req,
uv_tcp_t* handle,
const uv_buf_t bufs[],
unsigned int nbufs,
uv_write_cb cb) {
int result;
DWORD bytes;
uv_req_init(loop, (uv_req_t*) req);
req->type = UV_WRITE;
req->handle = (uv_stream_t*) handle;
req->cb = cb;
/* Prepare the overlapped structure. */
memset(&(req->u.io.overlapped), 0, sizeof(req->u.io.overlapped));
if (handle->flags & UV_HANDLE_EMULATE_IOCP) {
req->event_handle = CreateEvent(NULL, 0, 0, NULL);
if (!req->event_handle) {
uv_fatal_error(GetLastError(), "CreateEvent");
}
req->u.io.overlapped.hEvent = (HANDLE) ((ULONG_PTR) req->event_handle | 1);
req->wait_handle = INVALID_HANDLE_VALUE;
}
...
if (handle->flags & UV_HANDLE_EMULATE_IOCP &&
!RegisterWaitForSingleObject(&req->wait_handle,
req->event_handle, post_write_completion, (void*) req,
INFINITE, WT_EXECUTEINWAITTHREAD | WT_EXECUTEONLYONCE)) {
SET_REQ_ERROR(req, GetLastError());
uv_insert_pending_req(loop, (uv_req_t*)req);
}
This seems to occur because we accept the socket on both the primary loop and secondary loop. The first call to uv_tcp_accept associates the socket with the primary loop's completion port which causes the second call to CreateIoCompletionPort (via uv_pipe_accept->uv_tcp_import) to fail on the duplicated socket on the secondary thread. This sequence of events is what causes the UV_HANDLE_EMULATE_IOCP flag to be set.
static int uv_tcp_set_socket(uv_loop_t* loop,
uv_tcp_t* handle,
SOCKET socket,
int family,
int imported) {
...
/* Associate it with the I/O completion port. */
/* Use uv_handle_t pointer as completion key. */
if (CreateIoCompletionPort((HANDLE)socket,
loop->iocp,
(ULONG_PTR)socket,
0) == NULL) {
if (imported) {
handle->flags |= UV_HANDLE_EMULATE_IOCP;
} else {
return GetLastError();
}
}
We were able to work around this issue on Windows by not using pipes, but instead accepting connections directly on the secondary loops. To prevent race conditions, we block inside the listen callback in the primary loop until the secondary loop finishes accepting the new connection. We still use Unix domain sockets on non-Windows platforms to support multiple loops.
The PR including this change can be found here: https://github.com/aspnet/KestrelHttpServer/pull/602/files#diff-5cfac6c0d3b3c415c8884d8696ff11f5R15
This gives us about a 15% performance improvement in our plaintext benchmark: https://github.com/aspnet/Benchmarks#the-scenarios It also happens to work around an issue we have with named pipes on Azure Web Apps, but it's entirely possible we could find a way to make named pipes work if need be
I understand this use of libuv is off-label, and that the underlying implementation could change, but is there any concrete reason we shouldn't do this on Windows? Is there a better method of supporting multiple event loops on Windows without emulating IOCP?
Thanks,
Stephen
Saúl Ibarra Corretgé
[snip]
> the duplicated socket on the secondary threads are using event handles
> instead of IOCP for async completion on windows, e.g.:
>
what your architecture is, can you describe it a little? Do you have one
thread accepting connections and then send them over pipes to other
loops using uv_write2?
[snip]
>
> I understand this use of libuv is off-label, and that the underlying
> implementation could change, but is there any concrete reason we
> shouldn't do this on Windows? Is there a better method of supporting
> multiple event loops on Windows without emulating IOCP?
>
down here only works with Windows >= 8.1) you could try this:
- accept connections in a thread
- once the connection is accepted, use uv_fileno to grab the socket
handle and detach it from the loop's IOCP with [0], see
FileReplaceCompletionInformation.
- send the handle using uv_write2
- since the socket is not associated with any IOCP the duplicated one
shouldn't be either, thus allowing the receiving loop to use IOCP and
not the emulated mode
- profit?
After writing this I wonder if we shouldn't do this automagically in
libuv when sending a handle...
[0]:
https://msdn.microsoft.com/en-us/library/windows/hardware/ff567096%28v=vs.85%29.aspx
Hope that helps!
Regards,
--
Saúl Ibarra Corretgé
bettercallsaghul.com