How does the smp message queue work?

[Vanegicloh J]

Hello! I'm investigating smp logic in seastar and trying to understand how shards communicate with each other.

I get it like this:
1. Program starts.
2. Seastar allocates reactor on each shard.
3. On shard 0 seastar creates (smp::count ** smp::count) matrix of smp_message_queues.
4. Each smp_message_queue has pointer to reactors: receiver and transmitter.
5. On every shard there is a thread_local pointer(_qs) to the smp_message_queue matrix that is stored in memory of 0 shard.
6. On every shard in it's memory seastar allocates pending_fifo deque (this line: void tx_side::init() { new (&a) aa; } ).

When I call submit_to(2, foo) from the shard 3 this happens:
1. smp_message_queue wrappes foo to the worker_item and moves it to the memory of shard 2.
2. Now my foo is in pending_fifo on shard 2.
3. Reactor on shard 2 polls all related to it smp_message_queues and finds the worker_item with my foo in pending_fifo.
4. Reactor schedules the task. Now my foo is completed and work_item got a result.
5. Reactor on shard 3 polls all related to it smp_message_queues and finds completed worker.
7. Reactor on shard 3 moves the result to a promise and bring it to a caller of submit_to

My questions:
1. Is this flow correct? Could you please correct me if it's wrong.
2. All smp_message_queues are located in memory of shard 0. So, every time I call submit_to from another shard I access memory from shard 0. Is it correct?
3. If it's correct, should I minimize submit_to in my code to prevent "atomic instructions, cache line bouncing and memory fences"?

Thank you!

[Pavel Emelyanov]

On 31.05.2023 16:50, Vanegicloh J wrote:
> Hello! I'm investigating smp logic in seastar and trying to understand how shards communicate with each other.
>
> I get it like this:
> 1. Program starts.
> 2. Seastar allocates reactor on each shard.
> 3. On shard 0 seastar creates (smp::count ** smp::count) matrix of smp_message_queues.
> 4. Each smp_message_queue has pointer to reactors: receiver and transmitter.
> 5. On every shard there is a thread_local pointer(_qs) to the smp_message_queue matrix that is stored in memory of 0 shard.
> 6. On every shard in it's memory seastar allocates pending_fifo deque (this line: void tx_side::init() { new (&a) aa; } ).
>
> When I call submit_to(2, foo) from the shard 3 this happens:
> 1. smp_message_queue wrappes foo to the worker_item and moves it to the memory of shard 2.
> 2. Now my foo is in pending_fifo on shard 2.
> 3. Reactor on shard 2 polls all related to it smp_message_queues and finds the worker_item with my foo in pending_fifo.
> 4. Reactor schedules the task. Now my foo is completed and work_item got a result.
> 5. Reactor on shard 3 polls all related to it smp_message_queues and finds completed worker.
> 7. Reactor on shard 3 moves the result to a promise and bring it to a caller of submit_to
>
> My questions:
> 1. Is this flow correct? Could you please correct me if it's wrong.

Yes, with the except "moves ... to the memory of shard" is not very accurate.
The work_item-s are allocated as unique_ptr-s and they remain in the memory
they were allocated at.

> 2. All smp_message_queues are located in memory of shard 0. So, every time I call submit_to from another shard I access memory from shard 0. Is it correct?

Yes

> 3. If it's correct, should I minimize submit_to in my code to prevent "atomic instructions, cache line bouncing and memory fences"?

Minimizing sumit_to()-s is generally good practice, but it's two fold. If you
happen to have one shard loaded with work and another one idling it's highly
questionable if saving yourself from extra submit_to is worth it.

> Thank you!
>
> --
> You received this message because you are subscribed to the Google Groups "seastar-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to seastar-dev...@googlegroups.com <mailto:seastar-dev...@googlegroups.com>.
> To view this discussion on the web visit https://groups.google.com/d/msgid/seastar-dev/e53a21ae-254c-4e41-b552-ad54d746d393n%40googlegroups.com <https://groups.google.com/d/msgid/seastar-dev/e53a21ae-254c-4e41-b552-ad54d746d393n%40googlegroups.com?utm_medium=email&utm_source=footer>.