GRPC Server implementation: How to Read data from a file using Python?

[Raja Omer]

I am new to GRPC, and want to know how should a GRCP server which can read from a file and send the data to a GRCP client in python? 

Scenario implementation: I have an excel file with two columns, ID and value. My understanding for implementation of this scenario is :
1- I will have to define a message in a protocol buffer file having these two attributes. 
2- Inside my protocol buffer file, I will define a service which will send the data to the client.

3- Using GRPC tools I will then create _pb2.py and _pb2_grpc.py files. 

4- Now I will write the GRPC server file in python. Inside this file I will override or define the service I have written in the protocol buffer file. My confusion is regarding this step. Do I have to read the excel in this step using python or do I have to make a database connection and pull the excel data from a database?

Many thanks!

[Richard Belleville]

> Do I have to read the excel in this step using python or do I have to make a database connection and pull the excel data from a database?

This question doesn't actually seem to have anything to do with gRPC. Is there something I'm missing?

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/a59b9544-13c0-42fd-afd8-284be0ea34cfn%40googlegroups.com.

[Raja Omer]

My confusion is regarding my approach, specifically that how to serve a client call for excel data using a  GRPC server? Do I have to store the data in some database and then read it into a message and send back on a row by row basis or can I just read from the excel directly in the GRCP server python file and send the entire excel data as one message response? Would any of these approaches be correct and will the server python file be the right place for this implementation?

[Richard Belleville]

I see. Both of these approaches are equally valid. The latter approach would be a unary RPC and require the server to buffer the entire spreadsheet in memory, meaning that it wouldn't be able to scale to very large datasets. However, this would also be the simpler approach in terms of implementation -- everything can be written synchronously.

The former approach is more robust if you need to scale to large datasets. This would be a client-unary, server-streaming RPC and the server wouldn't have to buffer the whole dataset in memory.

A couple of pitfalls specific to Python. The sync stack works with a fixed sized thread pool, which limits the number of clients that can connect to your backend at once. In the case of a unary RPC, this is less of an issue. If you go with the streaming approach, however, misbehaving clients could eat up a thread indefinitely until you have none left to service requests. If this is a concern for you, you could use the asyncio stack instead. A last caveat for Python is that it will be slower than other languages you could choose, including Go, Java, and C++, all of which are well supported gRPC implementations which will be more performant.

Thanks,

Richard Belleville 

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/23a8d925-1420-44a4-9b9c-8c35d29c86a7n%40googlegroups.com.

[Amit Saha]

On 15 May 2021, at 6:35 am, Raja Omer <code.j...@gmail.com> wrote:

 Do I have to read the excel in this step using python or do I have to make a database connection and pull the excel data from a database?

You can do this in both ways - one of them (say one protobuf message per row - for example) is scalable, but admittedly slightly complicated. The other is simple, but wouldn’t scale well. 

For the first approach, you would define a message as follows (for eg):

message Row {

  Col1 string

  Col2 int

}

Combine it with client side streaming.

For the second approach, you would define instead:

message excelData {

  data bytes

}

In the first approach, you do more work on both the client side (reading the excel file, construing a protobuf message) and the same work in reverse on the server side.

In the second approach, you do all the work on the server side.

However, the recommended approach is the first one.

Hope that helps.

[Raja Omer]

Hi guys, Thanks for your suggestions. So, seemingly, I have been able to implement a GRCP server which reads from a csv file and encodes it into JSON. I also have a GRCP client which can make the request and gets back the JSON representation.

In my second part, I want to create a http server which makes a request to the GRCP client for the JSON which gets that via a request to the GRCP server. The final result of the http request is to be displayed on a single page html. What approach or web framework should I use for GRCP? Will flask be the right one or Django or should I read up on something else?

Many thanks!

[Raja Omer]

I managed to write a flask application which triggers the GRCP client automatically because I am importing the result of the client's call to the GRCP server in the flask application python file. However, I have noticed that the data received by the GRCP client from the GRCP server loses its JSON formatting. This is what is happening on the server side to send back data to the GRCP client:

json_string = json_format.MessageToJson()  // This works fine and the result is as expected when I print it on the server side

return metricDetails_pb2.metricResponse(jsonString=json_string) // This return value messes things up and destroys the json formatting from above.

           

metricResponse is defined as a string in the .proto file as below:

message metricResponse {

string jsonString=1;

}

Similarly, the timeSeriesData is a repeated field in the timeSeries message of type metric which has two variables inside it:

message metric{

string time=1;

string usage=2;

}

message timeSeries {

repeated metric data=1;

}

As a result, when the flask application simply prints the answer received from the client the JSON formatting is lost. What can I do to fix this?

The below is the faulty output: