[RFC] allow client connections to target a specific shard

[Ultrabug]

Hello

I hope everyone is safe and getting through this moment as smoohtly as possible.

--

As a starter, I'd like to point out that - to my understanding - the Java driver also suffers from the points that I will be making here.

While working with Israel et al on the Python shard aware driver I found out that client connections get assigned shards in a round-robin manner (see system.clients table).

Since in the current protocol clients have no way to target a specific shard, they have to implement what we could call an optimistic mechanism which basically tries connecting until they get a connection to all shards.

- Java initial implementation: https://github.com/scylladb/java-driver/commit/2bd2e25a21d989a3d3c480cf6af9722a407436df#diff-d24ed3449944c678ae13b8da5294e80eR592

- Java optimization: https://github.com/scylladb/java-driver/commit/b925924fcd490ebaac020e02c02e0a918dd3cd1b

- Python initial implementation: https://github.com/scylladb/python-driver/pull/6/commits/dc05677cbdeea9679dbe1db30bcbe7f93f5ec356

- Python optimization: https://github.com/scylladb/python-driver/pull/6/commits/215056951562f95b223718a3918faa043719120c

- Python optimization optimization: https://github.com/scylladb/python-driver/pull/6/commits/32822c9a6c4d3a1b9db11cd79b408f9ed4a599f1

Needless to say that while this is inefficient, it also means that drivers are not fully shard aware until they luckily manage to get a connection to all shards!

Good news is : if we were somehow satisfied of the performance of the drivers, we will do even better by addressing this issue! :)

I'd like your point of view on two options I can see on my limited knowledge please.

---

Option 1- extend the protocol to allow clients to specify a shard_id to connect to

Maybe we could add a key in the protocol so that clients could specify the shard_id they want to connect to making connections-to-shard predictable.

I have no clue how hard it is or the consequences, so please go ahead.

---

Option 2 -change the way nodes assign shards to client connections on scylla

Maybe we could have nodes assign shards in a round-robin manner but per client.

This would save us from this eternal race and competition between multiple connections originating from multiple clients.

Thanks for considering this optimization <3

[Piotr Jastrzębski]

I'm not an expert by any means but it seems that the load balancing of
connections is actually part of Seastar.
Scylla only uses seastar::listen in multiple threads:
https://github.com/scylladb/scylla/blob/4e6f5436767ef84072af7964b49fe56b0ffbc6c4/transport/server.cc#L219
https://github.com/scylladb/scylla/blob/8da235e440cfb40b3ca897b2eff8958812fa6043/service/storage_service.cc#L2323

> --
> You received this message because you are subscribed to the Google Groups "ScyllaDB development" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-dev...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-dev/b51b0192-3c32-47e5-a25b-1ba88c9af362%40googlegroups.com.

[Piotr Jastrzębski]

One solution could be for different shards to listen on different
ports. For example shard X would listen on a port BasePort + X. That
would be equivalent to Ultrabug option 1. Client would be able to
decide which shard to connect to.

[Gleb Natapov]

On Thu, Apr 09, 2020 at 09:44:09AM -0700, Ultrabug wrote:
> Option 1- extend the protocol to allow clients to specify a shard_id to
> connect to
>
> Maybe we could add a key in the protocol so that clients could specify the
> shard_id they want to connect to making connections-to-shard predictable.
> I have no clue how hard it is or the consequences, so please go ahead.
>

How do you specify a shard_id while connecting? A shard is chosen
during connection (in case of DPDK by HW hash). Moving a connection after it was
created is pretty hard especially if we want to continue supporting DPDK.

> ---
>
> Option 2 -change the way nodes assign shards to client connections on scylla
>
> Maybe we could have nodes assign shards in a round-robin manner but per
> client.
> This would save us from this eternal race and competition between multiple
> connections originating from multiple clients.
>

Connection is created before a client is knows. Load balancing already
works per server port which is the only thing that identifies a
connection during connect.

Seastar supports "port" load balancer that allows a client to connect to
a specific shard by choosing a local port carefully. A connection is
created to a shard_id = src_port % smp::count. This is how internal rpc
connects to a right shard if a cluster is homogeneous. CQL port right
now does not uses this load balancer in the fear that it will negatively
affect non shard aware drivers).

--
Gleb.

[Avi Kivity]

Maybe we can quantify this. How does Linux choose source ports?


We can consider more complicated balancing schemes, e.h. (hash(source
IP) + (source port)) % nr_shards if the choice of source port will
always lead to the same shards.

[Benny Halevy]

On Fri, 2020-04-10 at 10:52 +0300, Gleb Natapov wrote:
> On Thu, Apr 09, 2020 at 09:44:09AM -0700, Ultrabug wrote:
> > Option 1- extend the protocol to allow clients to specify a shard_id to
> > connect to
> >
> > Maybe we could add a key in the protocol so that clients could specify the
> > shard_id they want to connect to making connections-to-shard predictable.
> > I have no clue how hard it is or the consequences, so please go ahead.
> >
> How do you specify a shard_id while connecting? A shard is chosen
> during connection (in case of DPDK by HW hash). Moving a connection after it was
> created is pretty hard especially if we want to continue supporting DPDK.

We can use the good old FTP approach by having a control verb on the known
server port, using which, the client can query the server about the number
of shards available and on which specific ports they listen.

[Avi Kivity]

On 4/13/20 3:08 PM, Benny Halevy wrote:
> On Fri, 2020-04-10 at 10:52 +0300, Gleb Natapov wrote:
>> On Thu, Apr 09, 2020 at 09:44:09AM -0700, Ultrabug wrote:
>>> Option 1- extend the protocol to allow clients to specify a shard_id to
>>> connect to
>>>
>>> Maybe we could add a key in the protocol so that clients could specify the
>>> shard_id they want to connect to making connections-to-shard predictable.
>>> I have no clue how hard it is or the consequences, so please go ahead.
>>>
>> How do you specify a shard_id while connecting? A shard is chosen
>> during connection (in case of DPDK by HW hash). Moving a connection after it was
>> created is pretty hard especially if we want to continue supporting DPDK.
> We can use the good old FTP approach by having a control verb on the known
> server port, using which, the client can query the server about the number
> of shards available and on which specific ports they listen.

We fall into the same problem as FTP: suddenly which ports need to be
opened in the firewall (or its new name: cloud security group) is not
known in advance. This can be problematic, especially in enterprise
deployments where you have to ask another group to poke holes between
networks.

[Benny Halevy]

That's a good point

[Ultrabug]

Hi

I wanted to give a follow-up on this thread since discussions took place in other ML lists and PRs

Current consensus based on recent discussions [1]:

- implement a source-based algorithm on scylla so that clients will be able to target a shard id by setting up their connection socket source port

- add two new shard-aware listening ports (+ options) on scylla where this source-based algorithm will be enabled

- modify shard aware scylla client drivers accordingly [2]

[1] : https://github.com/scylladb/scylla/pull/6781
[2] : https://github.com/scylladb/python-driver/pull/54

[Juliusz Stasiewicz]

On Fri, Jul 17, 2020 at 1:22 PM Ultrabug <ultr...@gmail.com> wrote:

Hi

I wanted to give a follow-up on this thread since discussions took place in other ML lists and PRs

Current consensus based on recent discussions [1]:

- implement a source-based algorithm on scylla so that clients will be able to target a shard id by setting up their connection socket source port

- add two new shard-aware listening ports (+ options) on scylla where this source-based algorithm will be enabled

- modify shard aware scylla client drivers accordingly [2]

Exactly. I'm working on it now (server-side + cpp-driver + java driver, in this order), so stay tuned!

/JS

[Ultrabug]

Hello everyone

"Scylla now listens on port 19042 as well as port 9042 for the native transport. This new port is used for shard-aware drivers, allowing them to quickly bind to a shard by selecting the source port (on the client itself)."

Juliusz, thanks for making this happen. I guess that we can start trying our drivers on the nightly builds!

How about the cpp and java drivers, did you also complete these?

I'm asking so that we can "close" this RFC :)

Cheers

[Piotr Jastrzębski]

On Tue, Aug 11, 2020 at 9:37 AM Ultrabug <ultr...@gmail.com> wrote:
>
> Hello everyone
>
> "Scylla now listens on port 19042 as well as port 9042 for the native transport. This new port is used for shard-aware drivers, allowing them to quickly bind to a shard by selecting the source port (on the client itself)."
>
> Juliusz, thanks for making this happen. I guess that we can start trying our drivers on the nightly builds!
>
> How about the cpp and java drivers, did you also complete these?
>
> I'm asking so that we can "close" this RFC :)

CPP driver is under development and Java is queued just after that.

>
> Cheers
>
> On Friday, July 17, 2020 at 7:27:40 PM UTC+2 juliusz.s...@scylladb.com wrote:
>>
>> On Fri, Jul 17, 2020 at 1:22 PM Ultrabug <ultr...@gmail.com> wrote:
>>>
>>> Hi
>>>
>>> I wanted to give a follow-up on this thread since discussions took place in other ML lists and PRs
>>>
>>> Current consensus based on recent discussions [1]:
>>>
>>> - implement a source-based algorithm on scylla so that clients will be able to target a shard id by setting up their connection socket source port
>>> - add two new shard-aware listening ports (+ options) on scylla where this source-based algorithm will be enabled
>>> - modify shard aware scylla client drivers accordingly [2]
>>
>>
>> Exactly. I'm working on it now (server-side + cpp-driver + java driver, in this order), so stay tuned!
>>
>> /JS
>

> --
> You received this message because you are subscribed to the Google Groups "ScyllaDB development" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-dev...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-dev/9d479fe7-2f4c-4be6-a060-6069de9c4047n%40googlegroups.com.