How can I wait for a seastar::parallel_for_each or seastar::async finishing?

[ka mof]

I tried get() and wait(), it throw "void seastar::internal::future_base::do_wait(): Assertion `thread' failed.".

[Benny Halevy]

There are several ways to wait on future<> returned by
e.g. parallel_for_each().

The trivial way is just return it up the stack,
e.g. app.run(argc, argv, [data] {
return parallel_for_each(data, [] (auto& x) {});
});

.get() blocks until the future<> is resolved
so it indeed can be used only in a seastar thread.
See for example run_compute_intensive_tasks in scheduling_group_demo.cc

Similary, future<> can also be used in coroutines,
where you can `co_await parallel_for_each(...)`
as done in coroutines_demo.cc

There the future<> returned by parallel_for_each is assigned to a variable:
auto f = seastar::parallel_for_each(...
and later in main() it is co_await:ed:
co_await std::move(f);

On Sun, 2022-02-27 at 11:44 -0800, ka mof wrote:
> I tried get() and wait(), it throw "void seastar::internal::future_base::do_wait(): Assertion `thread' failed.".

> --
> You received this message because you are subscribed to the Google Groups "seastar-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to seastar-dev...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/seastar-dev/998699ac-0eeb-4749-bc71-7360bd8da860n%40googlegroups.com.

[Gavin Ray]

I ran into this issue too and found this thread by googling.

Say you have something like the below code:

==================================

struct disk_io_service final

{

    seastar::file           db_file;

    explicit disk_io_service(seastar::sstring db_file_path)

    {

        db_file = seastar::open_file_dma(db_file_path, seastar::open_flags::rw | seastar::open_flags::create).get0();

    }

}

struct application_service final

{

    disk_io_service     disk_io_svc;

    explicit application_service()

        : disk_io_svc(fmt::format("db.stripe_{}.dat", shard_id))

};

auto main(int argc, char** argv) -> int

{

    seastar::app_template                 app;

    seastar::sharded<application_service> app_svc;

    return app.run(argc, argv, [&app, &app_svc] {

        return app_svc.start().then([&app_svc] {

            // ...

        });

    });

}

==================================

This fails during runtime for me because of the above-mentioned "thread" assertion

I what's the proper way to initialize the DB file?

[Benny Halevy]

On Sat, 2022-10-29 at 14:01 -0700, Gavin Ray wrote:
> I ran into this issue too and found this thread by googling.
> Say you have something like the below code:
>
> ==================================
>
> struct disk_io_service final
> {
>     seastar::file           db_file;
>
>     explicit disk_io_service(seastar::sstring db_file_path)
>     {
>         db_file = seastar::open_file_dma(db_file_path, seastar::open_flags::rw | seastar::open_flags::create).get0();

This is required to run a seastar thread context.

>     }
> }
>
> struct application_service final
> {
>     disk_io_service     disk_io_svc;
>
>     explicit application_service()
>         : disk_io_svc(fmt::format("db.stripe_{}.dat", shard_id))
> };
>
> auto main(int argc, char** argv) -> int
> {
>     seastar::app_template                 app;
>     seastar::sharded<application_service> app_svc;
>
>     return app.run(argc, argv, [&app, &app_svc] {

It looks like you want to run the code below in a thread, like:

return async([] {
app_svc.start.get();
});

> To view this discussion on the web visit https://groups.google.com/d/msgid/seastar-dev/2102e37f-f2d8-4f73-9dc0-e5ee4c10bcd2n%40googlegroups.com.

[Gavin Ray]

Thank you, that did the trick!

[Gavin Ray]

Well, interestingly it seemed to solve it in one case, but it still persists in another

So what I currently have is:

========================================

auto main(int argc, char** argv) -> int

{

    seastar::app_template app;

    return app.run(argc, argv, [&] {

        return seastar::async([&] {

            seastar::sharded<application_service> app_svc;

            app_svc.start().get();

        });

    });

}

struct application_service final

{

    explicit application_service()

        : disk_io_svc(format("db.stripe_{}.dat", shard_id)) {}

}

struct disk_io_service final

{

    seastar::file           db_file;

    explicit disk_io_service(seastar::sstring db_file_path)

    {

        db_file = seastar::open_file_dma(db_file_path,  open_flags::rw | open_flags::create).get0();

    }

}

========================================

What is strange is that it seems to succeed on some cores, but not others?

When I count the "Assertion `thread' failed." messages below, I get 15, which is NUM_CPUS - 1 (not sure if that has any significance)

Below I log the shard ID during the constructor of each class:

INFO  2022-10-30 15:44:56,116 [shard  0] learning-db - disk_io_service: shard_id = 0
INFO  2022-10-30 15:44:56,118 [shard  0] learning-db - Created buffer pool service on core 0
INFO  2022-10-30 15:44:56,118 [shard  0] learning-db - Created transaction service on core 0
INFO  2022-10-30 15:44:56,118 [shard  0] learning-db - Created database service on core 0
INFO  2022-10-30 15:44:56,119 [shard  0] learning-db - Created application on core 0
INFO  2022-10-30 15:44:56,123 [shard 15] learning-db - disk_io_service: shard_id = 15
INFO  2022-10-30 15:44:56,123 [shard  8] learning-db - disk_io_service: shard_id = 8
INFO  2022-10-30 15:44:56,123 [shard  7] learning-db - disk_io_service: shard_id = 7
seastar_demo: /home/user/projects/seastar-demo/build/_deps/seastar-src/src/core/future.cc:248: void seastar::internal::future_base::do_wait(): Assertion `thread' failed.
INFO  2022-10-30 15:44:56,123 [shard 10] learning-db - disk_io_service: shard_id = 10
seastar_demo: /home/user/projects/seastar-demo/build/_deps/seastar-src/src/core/future.cc:248: void seastar::internal::future_base::do_wait(): Assertion `thread' failed.
INFO  2022-10-30 15:44:56,123 [shard  1] learning-db - disk_io_service: shard_id = 1
INFO  2022-10-30 15:44:56,123 [shard  9] learning-db - disk_io_service: shard_id = 9
seastar_demo: /home/user/projects/seastar-demo/build/_deps/seastar-src/src/core/future.cc:248: void seastar::internal::future_base::do_wait(): Assertion `thread' failed.
INFO  2022-10-30 15:44:56,123 [shard  3] learning-db - disk_io_service: shard_id = 3
seastar_demo: /home/user/projects/seastar-demo/build/_deps/seastar-src/src/core/future.cc:248: void seastar::internal::future_base::do_wait(): Assertion `thread' failed.
INFO  2022-10-30 15:44:56,123 [shard 13] learning-db - disk_io_service: shard_id = 13
seastar_demo: /home/user/projects/seastar-demo/build/_deps/seastar-src/src/core/future.cc:248: void seastar::internal::future_base::do_wait(): Assertion `thread' failed.
seastar_demo: /home/user/projects/seastar-demo/build/_deps/seastar-src/src/core/future.cc:248: void seastar::internal::future_base::do_wait(): Assertion `thread' failed.
INFO  2022-10-30 15:44:56,123 [shard  4] learning-db - disk_io_service: shard_id = 4
INFO  2022-10-30 15:44:56,122 [shard  5] learning-db - disk_io_service: shard_id = 5
INFO  2022-10-30 15:44:56,122 [shard  6] learning-db - disk_io_service: shard_id = 6
INFO  2022-10-30 15:44:56,123 [shard 11] learning-db - disk_io_service: shard_id = 11
seastar_demo: /home/user/projects/seastar-demo/build/_deps/seastar-src/src/core/future.cc:248: void seastar::internal::future_base::do_wait(): Assertion `thread' failed.
seastar_demo: /home/user/projects/seastar-demo/build/_deps/seastar-src/src/core/future.cc:248: void seastar::internal::future_base::do_wait(): Assertion `thread' failed.
seastar_demo: /home/user/projects/seastar-demo/build/_deps/seastar-src/src/core/future.cc:248: void seastar::internal::future_base::do_wait(): Assertion `thread' failed.
INFO  2022-10-30 15:44:56,123 [shard 12] learning-db - disk_io_service: shard_id = 12
seastar_demo: /home/user/projects/seastar-demo/build/_deps/seastar-src/src/core/future.cc:248: void seastar::internal::future_base::do_wait(): Assertion `thread' failed.
seastar_demo: /home/user/projects/seastar-demo/build/_deps/seastar-src/src/core/future.cc:248: void seastar::internal::future_base::do_wait(): Assertion `thread' failed.
seastar_demo: /home/user/projects/seastar-demo/build/_deps/seastar-src/src/core/future.cc:248: void seastar::internal::future_base::do_wait(): Assertion `thread' failed.
INFO  2022-10-30 15:44:56,124 [shard  2] learning-db - disk_io_service: shard_id = 2
Aborting on shard 7.
Backtrace:
    <SNIPPED>
seastar_demo: /home/user/projects/seastar-demo/build/_deps/seastar-src/src/core/future.cc:248: void seastar::internal::future_base::do_wait(): Assertion `thread' failed.
seastar_demo: /home/user/projects/seastar-demo/build/_deps/seastar-src/src/core/future.cc:248: void seastar::internal::future_base::do_wait(): Assertion `thread' failed.
INFO  2022-10-30 15:44:56,125 [shard 14] learning-db - disk_io_service: shard_id = 14
seastar_demo: /home/user/projects/seastar-demo/build/_deps/seastar-src/src/core/future.cc:248: void seastar::internal::future_base::do_wait(): Assertion `thread' failed.
Aborted

[Benny Halevy]

On Sun, Oct 30, 2022, 21:47 Gavin Ray <ray.g...@gmail.com> wrote:

Well, interestingly it seemed to solve it in one case, but it still persists in another

So what I currently have is:

========================================

auto main(int argc, char** argv) -> int

{

    seastar::app_template app;

    return app.run(argc, argv, [&] {

        return seastar::async([&] {

            seastar::sharded<application_service> app_svc;

            app_svc.start().get();

        });

    });

}

struct application_service final

{

    explicit application_service()

        : disk_io_svc(format("db.stripe_{}.dat", shard_id)) {}

}

struct disk_io_service final

{

    seastar::file           db_file;

    explicit disk_io_service(seastar::sstring db_file_path)

    {

        db_file = seastar::open_file_dma(db_file_path,  open_flags::rw | open_flags::create).get0();

    }

}

========================================

What is strange is that it seems to succeed on some cores, but not others?

When I count the "Assertion `thread' failed." messages below, I get 15, which is NUM_CPUS - 1 (not sure if that has any significance)

Yhe thread is created only on shard 0, the one that starts the app, while sharded<T>::start constructs the service instances on all shards.

It's worth noting that a typical design for a seastar application will stick to synchronous calls in constructors and put the async initialization in another method, e.g. future<> start() or future<> init() that call the async path.

With c++20 it could very well be a coroutine which looks and feels more similar to a synchronous function, without the need to run in a thread.

the main thread could do something like

dist_svc.invoke_on_all([] (svc& s) {

    return svc.init();

}).get();

the way init is called on each shard and returns a future<> that invoke_on_all wait on.

The future<> that is returned by invoke_on_all is waited on using .get() in the thread context.

To view this discussion on the web visit https://groups.google.com/d/msgid/seastar-dev/6e635cd0-9cd7-403e-97d6-d7c2775db2d2n%40googlegroups.com.

[Gavin Ray]

> the thread is created only on shard 0, the one that starts the app, while sharded<T>::start constructs the service instances on all shards.
> ... 

> It's worth noting that a typical design for a seastar application will stick to synchronous calls in constructors and put the async initialization in another method

> e.g. future<> start() or future<> init() that call the async path.
> With c++20 it could very well be a coroutine which looks and feels more similar to a synchronous function, without the need to run in a thread.

Ahh okay, this makes a lot of sense -- thank you!

I've refactored my app to use this pattern.

On this note, there seem to be few examples of coroutines and C++ 20/23 codebases using Seastar

I've been building a toy relational database with Seastar for my own learning/hobby purposes

I'd love to contribute a PR for at least something like the buffer manager + disk I/O service that shows how to write a seastar app

with modern C++ and coroutines that perform async disk I/O with io_uring and implement a sharded buffer pool manager.

Seastar seems to have a heavy emphasis on the networking side of things so I think this could be valuable for folks looking to build DBMS

or other tools/services that are heavily file IO-bound.

(The gotcha here being that the PR may need a bit of hand-holding on best-practices. I've never professionally written C++, or a database)

[Avi Kivity]

On Thu, 2022-11-03 at 10:49 -0400, Gavin Ray wrote:

> the thread is created only on shard 0, the one that starts the app, while sharded<T>::start constructs the service instances on all shards.
> ... 

> It's worth noting that a typical design for a seastar application will stick to synchronous calls in constructors and put the async initialization in another method
> e.g. future<> start() or future<> init() that call the async path.
> With c++20 it could very well be a coroutine which looks and feels more similar to a synchronous function, without the need to run in a thread.

Ahh okay, this makes a lot of sense -- thank you!

I've refactored my app to use this pattern.

On this note, there seem to be few examples of coroutines and C++ 20/23 codebases using Seastar

I've been building a toy relational database with Seastar for my own learning/hobby purposes

I'd love to contribute a PR for at least something like the buffer manager + disk I/O service that shows how to write a seastar app

with modern C++ and coroutines that perform async disk I/O with io_uring and implement a sharded buffer pool manager.

Seastar seems to have a heavy emphasis on the networking side of things so I think this could be valuable for folks looking to build DBMS

or other tools/services that are heavily file IO-bound.

Actually, Seastar's main users (ScyllaDB, Ceph, RedPanda) are all disk intensive services.

(The gotcha here being that the PR may need a bit of hand-holding on best-practices. I've never professionally written C++, or a database)

No problem, that's what reviews are for. We have the apps/ directory for "real" applications and demos/ for demonstrations. Another way to contribute is to expand the tutorial.