Managing linux devices

[johnpea...@gmail.com]

I manage a fleet of linux devices and I need a way to send telemetry data(CPU, memory, drives, etc.)  from devices to a centralized server and also send commands(linux service restarts for example) from server to devices to execute on the device. The 15 different linux devices are bare metal boxes in 15 different locations access to WAN with a dynamic IP address. In a year the number of devices could jump to 100.

I'm looking for suggestions on:

- how to gather telemetry data to send to grpc server

- how authenticate and identify devices

- how to run commands locally on each device and report back

Thanks.

[Carl Mastrangelo]

Sounds like you want to use a client streaming rpc.  The client will start an RPC, and periodically send messages to the server, which will never respond.  This works well if the time between updates is small perhaps no more than a minute apart.   If it's longer, it would be better to just make it unary because the connection will be at risk of being torn down by intermediary things.

If you trust the hardware you are running on, a common way of authing would be client side certificates, but we leave the problem of certificate rotation and revocation up to you.  In gRPC, auth is more commonly (always?) done at the RPC level.  Each RPC would need an auth token, like a JWT. 

If you want a command and control style system, you'll need to use a bidirectional streaming RPC.   This would let the client phone home, and wait for messages to come down from the server to be executed.  The client must initiate the connection and the RPC, but after that each side is a peer.  

There are examples on how to do each style of these commands in the github repos.  Look for "Route Guide" examples on how to do it in each language. 

[John Pearson]

Thanks for the detailed response.

I do trust the hardware running on and I will read more about client side certificates. Currently reading https://github.com/grpc/grpc-go/tree/master/examples/route_guide as you recommended and will figure out bi-directional streaming RPC 

--
You received this message because you are subscribed to a topic in the Google Groups "grpc.io" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/grpc-io/YB0HErGYZPw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to grpc-io+unsubscribe@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/66ed336c-6de9-4f80-b0e0-55b05338e45a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[John Pearson]

Question regarding bi-directional streaming:

Is this issue(https://github.com/grpc/grpc-go/issues/484#issuecomment-288880402) solved because of bidirectional rpc?  -- it is almost exactly the same as my use case.

On Mon, Feb 5, 2018 at 3:41 PM, 'Carl Mastrangelo' via grpc.io <grp...@googlegroups.com> wrote:

--

You received this message because you are subscribed to a topic in the Google Groups "grpc.io" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/grpc-io/YB0HErGYZPw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to grpc-io+unsubscribe@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.

[Carl Mastrangelo]

It's unlikely that gRPC will be able to have servers initiate connections / rpcs to clients.   The initial headers need to be sent by the client to the server, and only the server can send trailers to indicate the RPC is done.   But, after these two, the client and server are peers.  That's why that issue, (and my post), suggest treating the message as inverted requests and responses.  

 

On Monday, February 5, 2018 at 8:55:46 PM UTC-8, John Pearson wrote:

Question regarding bi-directional streaming:

Is this issue(https://github.com/grpc/grpc-go/issues/484#issuecomment-288880402) solved because of bidirectional rpc?  -- it is almost exactly the same as my use case.

On Mon, Feb 5, 2018 at 3:41 PM, 'Carl Mastrangelo' via grpc.io <grp...@googlegroups.com> wrote:

Sounds like you want to use a client streaming rpc.  The client will start an RPC, and periodically send messages to the server, which will never respond.  This works well if the time between updates is small perhaps no more than a minute apart.   If it's longer, it would be better to just make it unary because the connection will be at risk of being torn down by intermediary things.

If you trust the hardware you are running on, a common way of authing would be client side certificates, but we leave the problem of certificate rotation and revocation up to you.  In gRPC, auth is more commonly (always?) done at the RPC level.  Each RPC would need an auth token, like a JWT. 

If you want a command and control style system, you'll need to use a bidirectional streaming RPC.   This would let the client phone home, and wait for messages to come down from the server to be executed.  The client must initiate the connection and the RPC, but after that each side is a peer.  

There are examples on how to do each style of these commands in the github repos.  Look for "Route Guide" examples on how to do it in each language. 

On Sunday, February 4, 2018 at 5:37:40 PM UTC-8, johnpea...@gmail.com wrote:

I manage a fleet of linux devices and I need a way to send telemetry data(CPU, memory, drives, etc.)  from devices to a centralized server and also send commands(linux service restarts for example) from server to devices to execute on the device. The 15 different linux devices are bare metal boxes in 15 different locations access to WAN with a dynamic IP address. In a year the number of devices could jump to 100.

I'm looking for suggestions on:

- how to gather telemetry data to send to grpc server

- how authenticate and identify devices

- how to run commands locally on each device and report back

Thanks.

--
You received this message because you are subscribed to a topic in the Google Groups "grpc.io" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/grpc-io/YB0HErGYZPw/unsubscribe.

To unsubscribe from this group and all its topics, send an email to grpc-io+u...@googlegroups.com.

To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.

[John Pearson]

I'm unclear on what the last part means: "That's why that issue, (and my post), suggest treating the message as inverted requests and responses."

Do you mean rpc Link (stream UpLink) returns (stream DownLink) {} ? With a bidirectional stream?

Thanks

To unsubscribe from this group and all its topics, send an email to grpc-io+unsubscribe@googlegroups.com.

To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/bc74b58e-8ae7-4538-9c95-e2fad7d66413%40googlegroups.com.

[Carl Mastrangelo]

Yes, that is correct.  (It's bidirectional because the word "stream" is in both the request (UpLink) and response (DownLink).)

I mistakenly thought this was another thread (which I replied to yesterday) which had almost the same issue.   Take a look at https://groups.google.com/d/msg/grpc-io/G4eYs1zNMjE/Yh2WJS7TBwAJ  where I describe how to do what you want.

[John Pearson]

Got it. So this isn't considered bad form for RPC? Technically it isn't RPC? In my case:

- Server implements rpc Link (stream UpLink) returns (stream DownLink) {}

- Client initiates connection

- Client waits for messages to come down from the server to be executed

- Clients parses the strings/bytes streamed down and uses a case statement to execute commands

To unsubscribe from this group and all its topics, send an email to grpc-io+unsubscribe@googlegroups.com.

To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.

[Carl Mastrangelo]

Your flow is correct.     Is it RPC?  I guess that's a more philosophical, but I don't think there is anything wrong with it.  Streaming RPCs are already kind of weird anyway.

A more pure solution would be something akin to tunneling, where a client connects to a server, but then runs a gRPC server _on top of_ the client connection allowing the actual server to make client-like requests.  This has been discussed before, but wasn't implemented due to time constraints.  gRPC is fairly young so more advanced use cases like this don't have solutions yet.  

Note that even though the flow is reversed, you still get a lot of the benefits of gRPC.  The real client still can do intelligent fail over across all your servers.   You get all the stats and trace capabilities.   Lastly, gRPC is pretty fast!

[John Pearson]

Makes sense, thanks for the explanation. I see tunneling is being worked on: https://github.com/grpc/grpc/issues/14101

Question: When I have 15 clients all connected in a bidirectional stream, how does the Server send a message to a particular client? For example in my case: Server needs to tell a client12 to run a particular command, how to identify the right connection/channel Server side?

I've tried searching for this: 

- https://groups.google.com/forum/#!searchin/grpc-io/identify$20stream/grpc-io/MAjt9cE_uCU/VMAjo_KiAQAJ

- https://groups.google.com/forum/#!searchin/grpc-io/identify$20stream/grpc-io/z6aEEiaeopM/AVSZRFPTCAAJ

To unsubscribe from this group and all its topics, send an email to grpc-io+unsubscribe@googlegroups.com.

To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.

[Carl Mastrangelo]

The client would need to self identify by setting a header when it initiates the RPC.   If you are using auth tokens, you could maybe put the identity in the token.  When the client makes an RPC, the server decodes the token and associates the stream with the client.  

Another option (as mentioned in my first post) would be to use client side certificates.  When the server gets the RPC, it can determine the security information from the connection and associate it with the stream.

Lastly (and probably easiest) is to add the identity in a custom header fields (also called "metadata").   You can do this directly with some gRPC APIs and indirectly by making a client side RPC interceptor.  I am more familiar with Java, which favors the interceptor approach.  It will depend on what language of gRPC library you are using.

I would personally suggest the latter, since it is the most flexible, and works well with proxies.   It also and expandable, so you can put more info in the headers to be identified by.  

 

[John Pearson]

I've been checking out the interceptor pattern. Tokens, certificates, metadata make sense to me. I'm confused about "the server decodes the token and associates the stream with the client". Once a client ID info in any way, it's unclear to me how the server associates a stream with a client and then when needed invokes a command on that particular stream.

I'm using go-lang.

To unsubscribe from this group and all its topics, send an email to grpc-io+unsubscribe@googlegroups.com.

To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.

[Carl Mastrangelo]

In go (and I may be mistaken), I believe authentication is pulled out of the headers and put into the Context.   When you get the initial metadata on the server side, you can look in the Context for the auth.   

To issue commands to the particular client, the server would need to maintain a map of client id to stream.  Your server goroutine would select on a channel for commands from your application, and then call Stream.SendMsg.   A rough sketch of the idea would be:

var clientmap map[string]chan interface{}

 

streamhandler := func(srv interface{}, stream ServerStream) error {

    ctxx:= stream.Context()

    auth := context.Value(authKey{})

    ch := make(chan interface{})

    clientmap[auth.ID()] = ch

    defer delete(clientmap, auth.ID())

    for {

      select {

        case d := <- ch:

          stream.WriteMsg(d)

        case <-ctx.Done():

          break

      }

    }

  return nil

}

When any goroutine wants to contact a particular client, it attempts to look up the client by its ID in the clientmap and then send a message to it.    Right now this is very simple and doesn't wait for a response, but you can add that in.  You'll need to fix the threadsafety too, and error handling, but this is the rough idea.

[John Pearson]

Awesome. Thanks for taking the time to code that up. I plan on trying to implement this today/tomorrow. I will report back.

To unsubscribe from this group and all its topics, send an email to grpc-io+unsubscribe@googlegroups.com.

To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.