How to choose reactor_backend_uring?

[­신희원 / 학생 / 컴퓨터공학부]

Hi,

I just want to run seastar with reactor_backend_uring,

but I don't know how to.

Please let me know how to .

Thanks.

Heewon Shin.

ᐧ

[Piotr Sarna]

Hi, you can configure Seastar compilation with the `--enable-io_uring` parameter, e.g. `./configure.py --enable-io_uring`. If you use CMake, there's a Seastar_IO_URING=1 flag you can set, serving the same purpose. But please note that uring will only be available when liburing is also available on your operating system (and io_uring is supported by your kernel).

--
You received this message because you are subscribed to the Google Groups "seastar-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to seastar-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/seastar-dev/CAG6M7EPiBxTgH%2BuuT-5jsyzWL5JbV5-XzQq0AYCqqwoaGCj4-g%40mail.gmail.com.

[Avi Kivity]

You also need to select --reactor-backend=io_uring.

To view this discussion on the web visit https://groups.google.com/d/msgid/seastar-dev/2ebe71e0-7117-c0f1-18b3-9de6327f44b8%40scylladb.com.

[­신희원 / 학생 / 컴퓨터공학부]

Thanks for your assistance. I appreciate it. 

ᐧ

2022년 9월 7일 (수) 오후 5:03, Avi Kivity <a...@scylladb.com>님이 작성:

[Gavin Ray]

Could anyone share a working, minimal CMake project that enables the io_uring backend?

I've tried the below and it still seems to target AIO (from the log output at least, I'm not super familiar with Seastar yet, maybe I could be mistaken):

project(seastar_demo)
include(FetchContent)

set(Seastar_IO_URING ON CACHE INTERNAL "")
FetchContent_Declare(seastar
GIT_REPOSITORY https://github.com/GavinRay97/seastar
GIT_TAG master)
FetchContent_MakeAvailable(seastar)

add_executable(seastar_demo main.cpp)
target_link_libraries(seastar_demo PRIVATE seastar)

When I run this, the log output I get is:

WARNING: debug mode. Not for benchmarking or production
WARN  2022-10-25 22:38:44,164 seastar - Requested AIO slots too large, please increase request capacity in /proc/sys/fs/aio-max-nr. available:65536 requested:176416
WARN  2022-10-25 22:38:44,164 seastar - max-networking-io-control-blocks adjusted from 10000 to 3070, since AIO slots are unavailable
WARN  2022-10-25 22:38:44,165 [shard  0] seastar - Creation of perf_event based stall detector failed, falling back to posix timer: std::system_error (error system:13, perf_event_open() failed: Permission denied)
INFO  2022-10-25 22:38:44,168 [shard  0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000
INFO  2022-10-25 22:38:44,168 [shard  0] seastar - IO queue uses 0.75ms latency goal for device 0
INFO  2022-10-25 22:38:44,168 [shard  0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647
INFO  2022-10-25 22:38:44,168 [shard  0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000

[Gavin Ray]

(Just to clarify because I forgot to mention it above, I have LibUring in my system packages, and my kernel version is 6.0)

[user@MSI seastar-demo]$ uname -a
Linux MSI 6.0.1-locietta-WSL2-xanmod1 #1 SMP Wed Oct 12 22:14:49 UTC 2022 x86_64 GNU/Linux

[user@MSI seastar-demo]$ ls /usr/lib64/liburing*
/usr/lib64/liburing.so  /usr/lib64/liburing.so.2  /usr/lib64/liburing.so.2.2

[tcha...@gmail.com]

hi Gavin,

there are couple things you might need to check first:

1. check the io_uring support in kernel:

  grep CONFIG_IO_URING /boot/config-$(uname -r)

2. check if io_uring header and library are available

  cmake -DSeastar_IO_URING=ON ...

3. and might want to double check in the build directory

  grep Seastar_IO_URING CMakeCache.txt
4. as Avi pointed out, to pass --reactor-backend=io_uring to your application. assuming you are using `app_template` and passing the argc and argv to `app.run()`.

i think https://github.com/scylladb/seastar/blob/master/demos/hello-world.cc is a minimal example.  despite that it does not do I/O, though. in my case:

$ demos/hello-world_demo --help-seastar 2>/dev/null | grep reactor-backend -A2 | head -n2
  --reactor-backend arg (=io_uring)     Internal reactor implementation
                                        ([io_uring, linux-aio, epoll])

HTH,

[Gavin Ray]

Thanks a bunch for the comprehensive instructions!

I was missing the understanding that I needed to also pass the "reactor" flag to the application at runtime.

Unfortunately, I checked all of these things and passed the --reactor-backend=io_uring flag but it still seems not to change it from AIO =/

Maybe I am still doing something wrong:

1. "Check the io_uring support in kernel"

[user@MSI  seastar-demo]$ cat /proc/config.gz | gunzip | grep -i uring
CONFIG_IO_URING=y

2. check if io_uring header and library are available

[user@MSI seastar-demo]$ cmake -DSeastar_IO_URING=ON  -G Ninja --debug-output --fresh

-- Performing Test HAVE_IOURING_FEATURES - Success
   Called from: [6]     /usr/share/cmake/Modules/Internal/CheckSourceCompiles.cmake
                [5]     /usr/share/cmake/Modules/CheckCXXSourceCompiles.cmake
                [4]     /usr/share/cmake/Modules/CheckStructHasMember.cmake
                [3]     /home/user/projects/seastar-demo/cmake-build-debug/_deps/seastar-src/cmake/FindLibUring.cmake
                [2]     /home/user/projects/seastar-demo/cmake-build-debug/_deps/seastar-src/cmake/SeastarDependencies.cmake
                [1]     /home/user/projects/seastar-demo/cmake-build-debug/_deps/seastar-src/CMakeLists.txt
-- Found LibUring: /usr/lib64/liburing.so (Required is at least version "2.0")
   Called from: [5]     /usr/share/cmake/Modules/FindPackageMessage.cmake
                [4]     /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake
                [3]     /home/user/projects/seastar-demo/cmake-build-debug/_deps/seastar-src/cmake/FindLibUring.cmake
                [2]     /home/user/projects/seastar-demo/cmake-build-debug/_deps/seastar-src/cmake/SeastarDependencies.cmake
                [1]     /home/user/projects/seastar-demo/cmake-build-debug/_deps/seastar-src/CMakeLists.txt

3. and might want to double check in the build directory

[user@MSI cmake-build-debug]$ grep Seastar_IO_URING CMakeCache.txt
Seastar_IO_URING:BOOL=ON

4. as Avi pointed out, to pass --reactor-backend=io_uring to your application. assuming you are using `app_template` and passing the argc and argv to `app.run()`.

I am using the below as my application (Redpanda starter, using FetchContent from my fork instead of submodule):

https://gist.github.com/GavinRay97/4e3a85780b4dfa0938101e326d71ef9e

Unfortunately, even with all these settings, the below still happens when passing "--reactor-backend=io_uring":

[user@MSI cmake-build-debug]$ ./seastar_demo --reactor-backend=io_uring

WARNING: debug mode. Not for benchmarking or production

WARN  2022-10-26 11:18:01,239 seastar - Requested AIO slots too large, please increase request capacity in /proc/sys/fs/aio-max-nr. available:65536 requested:176416
WARN  2022-10-26 11:18:01,239 seastar - max-networking-io-control-blocks adjusted from 10000 to 3070, since AIO slots are unavailable
WARN  2022-10-26 11:18:01,240 [shard  0] seastar - Creation of perf_event based stall detector failed, falling back to posix timer: std::system_error (error system:13, perf_event_open() failed: Permission denied)
INFO  2022-10-26 11:18:01,242 [shard  0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000
INFO  2022-10-26 11:18:01,242 [shard  0] seastar - IO queue uses 0.75ms latency goal for device 0
INFO  2022-10-26 11:18:01,242 [shard  0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647
INFO  2022-10-26 11:18:01,242 [shard  0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000

===============================================================

I will try modifying the Seastar source on my fork to log the reasons why it is choosing the current Reactor backend, to get some visibility into what is going on.

Maybe that could help diagnose what is happening?

[tcha...@gmail.com]

reply inlined.

On Wednesday, October 26, 2022 at 11:20:20 PM UTC+8 Gavin Ray wrote:

Thanks a bunch for the comprehensive instructions!

I was missing the understanding that I needed to also pass the "reactor" flag to the application at runtime.

Unfortunately, I checked all of these things and passed the --reactor-backend=io_uring flag but it still seems not to change it from AIO =/

Maybe I am still doing something wrong:

1. "Check the io_uring support in kernel"

[user@MSI  seastar-demo]$ cat /proc/config.gz | gunzip | grep -i uring
CONFIG_IO_URING=y

2. check if io_uring header and library are available

[user@MSI seastar-demo]$ cmake -DSeastar_IO_URING=ON  -G Ninja --debug-output --fresh

-- Performing Test HAVE_IOURING_FEATURES - Success
   Called from: [6]     /usr/share/cmake/Modules/Internal/CheckSourceCompiles.cmake
                [5]     /usr/share/cmake/Modules/CheckCXXSourceCompiles.cmake
                [4]     /usr/share/cmake/Modules/CheckStructHasMember.cmake
                [3]     /home/user/projects/seastar-demo/cmake-build-debug/_deps/seastar-src/cmake/FindLibUring.cmake
                [2]     /home/user/projects/seastar-demo/cmake-build-debug/_deps/seastar-src/cmake/SeastarDependencies.cmake
                [1]     /home/user/projects/seastar-demo/cmake-build-debug/_deps/seastar-src/CMakeLists.txt
-- Found LibUring: /usr/lib64/liburing.so (Required is at least version "2.0")

this looks suspicious. and https://github.com/scylladb/seastar/pull/1267 should fix it. the expected output should looks like:

-- Found LibUring: /usr/lib/x86_64-linux-gnu/liburing.so (found suitable version "2.2", minimum required is "2.0")

[tcha...@gmail.com]

Oh, sorry, i missed your point. probably the logging message is kind of confusing, 

> WARN  2022-10-25 22:38:44,164 seastar - Requested AIO slots too large, please increase request capacity in /proc/sys/fs/aio-max-nr. available:65536 requested:176416

this log message is always printed as long as the available aio nr is less than requested aio nr, even if liburing backend is picked instead of linux-aio one. see https://github.com/scylladb/seastar/blob/3c88c17d480c54c7d8160e060d106bd640f864e8/src/core/reactor.cc#L4017

[Gavin Ray]

Ahh, okay. So that line has nothing to do with Linux AIO at all.

Thanks for that CMake change btw, I merged it into my fork and indeed it does log this now:

 (found suitable version "2.2", minimum required is "2.0")

Stepping through the reactor backend code, it seems like io_uring is available, and maybe it's even being selected/used:

Maybe an INFO log line could be added to startup displaying which reactor backend has been chosen so that users know?

[Gavin Ray]

Ah wow, yeah it was using io_uring this whole time I think! 🤦

I added these lines:

// reactor.cc

auto backend_selector = reactor_opts.reactor_backend.get_selected_candidate();
std::cout << "Seastar reactor backend: " << backend_selector << std::endl;

// reactor_backend.cc

std::unique_ptr<reactor_backend> reactor_backend_selector::create(reactor& r) {
  if (_name == "io_uring") {
    #ifdef SEASTAR_HAVE_URING
    std::cout << "io_uring backend chosen" << std::endl;
    return std::make_unique<reactor_backend_uring>(r);
    #else
    throw std::runtime_error("io_uring backend not compiled in");
    #endif
  }
  if (_name == "linux-aio") {
    std::cout << "linux-aio backend chosen" << std::endl;
    return std::make_unique<reactor_backend_aio>(r);
  } else if (_name == "epoll") {
    std::cout << "epoll backend chosen" << std::endl;
    return std::make_unique<reactor_backend_epoll>(r);
  }
  throw std::logic_error("bad reactor backend");

}

And then when I booted up the app again:

Seastar reactor backend: io_uring
io_uring backend chosen

io_uring backend chosen
io_uring backend chosen

...

LOL

[Avi Kivity]

Yes, that aio message is misleading. We should fix it to only depend on the backend.

Note that linux-aio is still used with the io_uring backend to provide preemption flags.

To view this discussion on the web visit https://groups.google.com/d/msgid/seastar-dev/a0aa7f1c-0123-4a68-82b6-e8cc7afc109cn%40googlegroups.com.

[Gavin Ray]

Whoops, I'm not very good at email and I think I accidentally DM'ed the last reply directly to Avi

Sending this in the hopes it also attached my previous email, to the rest of the Google group

On Thu, Nov 3, 2022 at 10:41 AM Gavin Ray <ray.g...@gmail.com> wrote:

> Note that linux-aio is still used with the io_uring backend to provide preemption flags.

Curious what you mean by this? I'm not intimately familiar with Linux kernel I/O internal

[Avi Kivity]

Look at need_preempt(), class preemption_monitor, and class preempt_io_context. It's a hacky way to get the kernel to write to a user memory location without having some user thread wait for it.

[Gavin Ray]

Got it, will do -- thanks for the tip!