using gRPC for nodes communication within cluster

[lich...@gmail.com]

Hey experts,

As a newbie to gRPC, i was wondering how, if it's feasible at all, I could use gRPC to implement intra nodes communication within a cluster. Imagine each node would both send and receive some msgs from other nodes. Such msg could be say heartbeat or command that A want B to execute. Imagine this is to implement some sort of consensus algorithm(say Paxos or Raft).

From gRPC "hello world" example I can see there are clear client/server relationship between nodes sitting at two ends of service. But how could i manage to achieve above goal of letting each node talk to each other? is there any code/doc example that you guys suggest i should look into?

Thanks a lot!

[Spencer Fang]

Hi lichen, can you elaborate on what you mean by "letting each node talk to each other"? Is this a service discovery problem that you're describing? 

gRPC understands how to connect to target hosts directly, and how to resolve DNS names to a set of equivalent backend hosts (in Java this is via the DnsNameResolver). It is also possible to have custom NameResolver classes that are powered by mechanisms other than DNS, but this plug in API is an experimental feature that may be change in the future. These two classes may be useful in solving your problem.

If A knows the hostname for B, then it can create a direct connect to it, as shown in the examples. If A knows of shard_c, and just needs some way to talk to any machine in shard_c, then using the NameResolver may be the solution.

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscribe@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/7f70cebd-31ea-4863-9a98-bb0f80946318%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Spencer Fang

[Jerry]

thanks, Spencer! Sorry I didn't make it clearer. I myself was actually not fully understand what I might want to ask at that time. Let me try to rephrase it after further thinking/searching...

I will start by your clarifying what you asked:) yes, I feel "service discovery" if you call it or something like that, is part of problem. because when nodes talk to each others, there must be able to establish bidirectional communications. in this sense, nodes within this cluster need to "discover" each other and then set up server-client connection, right?

To better understand it, here're some more background.

1. 1. I have my own app logic to do some bookkeeping work on each node;
2. 2. a node would need to communicate with others to reach some consensus(part of app logic) and this means a node need to both have client and server; Note I only need gRPC to serve RPC/communication of my app logic;
3. 3. so how could I achieve this? server seems to be blocking after I call server.awaitTerminate(), right? but do we also have the async version of the gRPC server in java? I bet yes, but I am not yet sure how could I leverage it.

Let's look at an example. I have node A, B, C. I will need to have gRPC serverA, serverB, serverC start first, and for each server say A, I need client to connect to B and C. and in addition to communication part, app (say in node A)logic would be able to send out msg to other nodes(say B and C) via corresponding clients(to server B and C) if needed; and of course app logic would be notified when requests coming from B and C(because itself is a server).

I've been searching online for days and have gone through grpc/grpc-java related material and code example. however, i find there's not that many code examples to show what is best practice and pattern to leverage gRPC...i'd really like to hear whatever suggestion you may have...

thanks in advance!

On Saturday, July 1, 2017 at 10:50:29 AM UTC-7, Spencer Fang wrote:

Hi lichen, can you elaborate on what you mean by "letting each node talk to each other"? Is this a service discovery problem that you're describing? 

gRPC understands how to connect to target hosts directly, and how to resolve DNS names to a set of equivalent backend hosts (in Java this is via the DnsNameResolver). It is also possible to have custom NameResolver classes that are powered by mechanisms other than DNS, but this plug in API is an experimental feature that may be change in the future. These two classes may be useful in solving your problem.

If A knows the hostname for B, then it can create a direct connect to it, as shown in the examples. If A knows of shard_c, and just needs some way to talk to any machine in shard_c, then using the NameResolver may be the solution.

On Tue, Jun 27, 2017 at 11:02 PM, <lich...@gmail.com> wrote:

Hey experts,

As a newbie to gRPC, i was wondering how, if it's feasible at all, I could use gRPC to implement intra nodes communication within a cluster. Imagine each node would both send and receive some msgs from other nodes. Such msg could be say heartbeat or command that A want B to execute. Imagine this is to implement some sort of consensus algorithm(say Paxos or Raft).

From gRPC "hello world" example I can see there are clear client/server relationship between nodes sitting at two ends of service. But how could i manage to achieve above goal of letting each node talk to each other? is there any code/doc example that you guys suggest i should look into?

Thanks a lot!

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.

To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.

To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/7f70cebd-31ea-4863-9a98-bb0f80946318%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Spencer Fang

[Spencer Fang]

Are you using the Java version of gRPC? My response below will assume grpc-java:

How to discover the nodes in your cluster:

grpc does not have a built in way to solve your cluster membership problem. For your example of 3 nodes (A, B, C), I like to think of this as a sub-problem that can be initially solved with a hard coded list. So upon startup, each node can run through the list, and for each hostname that is not the local hostname, create a ManagedChannel that connects to it. A hard coded list has some obvious limitations, but should be enough to at least get you up and running.

How to send messages:

Assuming you've created a ManagedChannel for each host, you can use the generated stubs to send messages to the host that you want. If you want to use async, then use the async stub. The included RouteGuideClient example covers how to do this.

How to receive messages:

Look at the RouteGuideServer example on how to set up a service that can handle incoming requests. For your use case, it sounds like it matters where the request came from. I would recommend putting this information inside the message protobuf. Also, note that the server will handle requests concurrently, so if your application's bookkeeping has shared data structures you will need to implement synchronization.

To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscribe@googlegroups.com.

To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/52ec3f51-7afe-4634-8926-ae8c9f71f106%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Spencer Fang

[Jerry]

Thanks a lot, Spencer! Your explanation makes it clearer:) only a bit still remains unclear to me... could you please see my inline comments? looking forward to your suggestions!

thanks again!

 

On Wednesday, July 5, 2017 at 9:24:05 AM UTC-7, Spencer Fang wrote:

Are you using the Java version of gRPC? My response below will assume grpc-java:

yes, i am using gRPC-java.

 

How to discover the nodes in your cluster:

grpc does not have a built in way to solve your cluster membership problem. For your example of 3 nodes (A, B, C), I like to think of this as a sub-problem that can be initially solved with a hard coded list. So upon startup, each node can run through the list, and for each hostname that is not the local hostname, create a ManagedChannel that connects to it. A hard coded list has some obvious limitations, but should be enough to at least get you up and running.

how could i initialize them then? actually i am not fully clear about thread model of gRPC and best practice about how to integrate it with business logic. say i want to start a server(for incoming messages) first, but at where i do this? in a separated thread and let server.awaitTermination() so JVM is not shutdown, and I could initialize client(for outgoing messages) in another thread? and client keeps trying to connect to rest of nodes until successfully(because server may not be ready yet)? in all examples they are either server or client but not both. so i haven't seen an example showing this use case/pattern.

 

How to send messages:

Assuming you've created a ManagedChannel for each host, you can use the generated stubs to send messages to the host that you want. If you want to use async, then use the async stub. The included RouteGuideClient example covers how to do this.

How to receive messages:

Look at the RouteGuideServer example on how to set up a service that can handle incoming requests. For your use case, it sounds like it matters where the request came from. I would recommend putting this information inside the message protobuf. Also, note that the server will handle requests concurrently, so if your application's bookkeeping has shared data structures you will need to implement synchronization.

or can i retrieve those sender info from meta data of header? or putting into app payload is recommended?   

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/52ec3f51-7afe-4634-8926-ae8c9f71f106%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Spencer Fang

[Spencer Fang]

Regarding initialization:

You can initialize the server and clients as a part of your normal application start up process, nothing special here. In fact, the examples set everything up in the `main` method.

Regarding sender info:

The metadata is not exposed in the code generated stub API, which happens to be the simplest API to use. This is the API used by the included examples, note how `RouteGuideService extends RouteGuideGrpc.RouteGuideImplBase`, which is a Java class automatically generated from proto IDLs. Metadata is exposed in the lower level ServerCall and ClientCall API, which I consider a more advanced API.