Questions about epoll or linux-aio in seastar

[john phate]

I have some questions about the epoll/inux-aio behavior when it is called by reactor::sleep which calls reactor_backend::wait_and_process_events. because epoll_wait/io_getevents is blocked, so this function call will block the reactor main thread, if there're new events come, for example, a disk io operation completes. but the thread blocks on epoll_wait/io_getevents, result in can't handle the disk io operation's callback. i wonder my understanding about this is true or false, if it is true, how to solve this problems. 

[john phate]

the same question for reactor_backend_aio::reap_kernel_completions which call by poll function

[Nadav Har'El]

On Wed, Dec 8, 2021 at 8:46 AM john phate <phat...@gmail.com> wrote:

I have some questions about the epoll/inux-aio behavior when it is called by reactor::sleep which calls reactor_backend::wait_and_process_events. because epoll_wait/io_getevents is blocked, so this function call will block the reactor main thread, if there're new events come, for example, a disk io operation completes. but the thread blocks on epoll_wait/io_getevents, result in can't handle the disk io operation's callback. i wonder my understanding about this is true or false, if it is true, how to solve this problems. 

There are two options here. One option is to use an "eventfd", a file descriptor that can wake up a sleeping epoll when an aio event happens. The other option is to not sleep at all, and continue a polling loop. The function aio_storage_context::can_sleep() supports both options.

 

The _aio_eventfd reactor field chooses between these two options (its value is the file object of that eventfd). It is determined by configuration options:

    if (!vm["poll-aio"].as<bool>()
            || (vm["poll-aio"].defaulted() && vm.count("overprovisioned"))) {
        _aio_eventfd = pollable_fd(file_desc::eventfd(0, 0));
    }

So for example, in "overprovisioned" mode if the only thing we're waiting for is IO, we go to sleep knowing full well that this is more expensive than continuing to busy-loop for this IO's completion, because we know that there are other processes who want to run on this CPU (that's the point of the overprovisioned mode). In "normal" mode, Seastar fully owns this CPU, and continuing to poll is more efficient than going to sleep.