Sending pickle object as client request

[Jatin Sharma]

I want to send a class to the grpc server. So, I am pickling the class and sharing as a bytes format in the message.

The .proto file(serverdata.proto) looks like below:

syntax = "proto3";

service DataProvider{

    rpc Get_Data(client_class) returns (result);

}

message client_class{

    bytes class_str = 1;

}

message result{

    int64 res = 1;

}

client.py file looks like below:

import grpc

import serverdata_pb2

import serverdata_pb2_grpc

import pickle

import pandas as pd

class Get_Hash():

    def get_hash(self,df):

        return pd.util.hash_pandas_object(df).sum()

    

a = serverdata_pb2.client_class()

a.class_str = pickle.dumps(Get_Hash)

channel = grpc.insecure_channel('localhost:50051')

# create a stub (client)

stub = serverdata_pb2_grpc.DataProviderStub(channel)

response = stub.Get_Data(a)

print(response.res)

On running this client, I'm getting the following error:

Traceback (most recent call last):

  File "client.py", line 18, in <module>

    response = stub.Get_Data(a)

  File "/home/jatin/.local/lib/python3.8/site-packages/grpc/_channel.py", line 826, in __call__

    return _end_unary_response_blocking(state, call, False, None)

  File "/home/jatin/.local/lib/python3.8/site-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking

    raise _InactiveRpcError(state)

grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:

status = StatusCode.UNKNOWN

details = "Exception calling application: client_class"

debug_error_string = "{"created":"@1600867086.557068214","description":"Error received from peer ipv6:[::1]:50051","file":"src/core/lib/surface/call.cc","file_line":1061,"grpc_message":"Exception calling application: client_class","grpc_status":2}"

>

I'm unable to resolve this error. I checked the type of the pickle file. It was 'bytes'. So, I changed the type in the .proto file to bytes. I'd be grateful if someone can help me resolve this error.

Regards,
Jatin

[Richard Belleville]

I would expect there to be more to the error message than just "client_class". I'm assuming there's an indentation problem in your original post and that the instantiation of the "client_class" message is happening within the Get_Data handler. What happens if you try to instantiate a "client_class" outside of the handler, on the main thread. Do you get a more illuminating error message?

[Jatin Sharma]

I don't understand the handle terminology. Can you please explain what you mean by handle. Thanks.

[Russell Wu]

Please also post the log on the server side. It does look like some error happened in Get_Data handler on the server. Maybe mismatch of generated client_class code between client and server? 

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/65656b59-cc4d-40d7-a00f-eef9b8f785ecn%40googlegroups.com.

[Jatin Sharma]

Hi Russell,

Please find below the code for server.

import grpc
from concurrent import futures
import time

import pickle

import pandas as pd

import serverdata_pb2
import serverdata_pb2_grpc

def get_df():
    cars = {'Brand': ['Honda Civic','Toyota Corolla','Ford Focus','Audi A4'],
            'Price': [22000,25000,27000,35000]
            }
   
    df = pd.DataFrame(cars, columns = ['Brand', 'Price'])
    return df

class DataProviderServicer(serverdata_pb2_grpc.DataProviderServicer):
    def Get_Data(self,request,context):
        get_func = pickle.loads(request.client_class)
        a = get_func()
        df = get_df()
        ans = a.get_hash(df)
        response = serverdata_pb2.result()
        response.result = ans
        return response        
       
server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
   
serverdata_pb2_grpc.add_DataProviderServicer_to_server(
        DataProviderServicer(), server)
   
print('Starting server. Listening on port 50051.')
server.add_insecure_port('[::]:50051')
server.start()
   
try:
    while True:
        time.sleep(86400)
except KeyboardInterrupt:
    server.stop(0)

[Russell Wu]

It seems to me that on the server when the `loads` trys to restore the Python object Get_Hash it cannot find it. Because for classes, pickle doesn't serialize its implementation but its import path (in this case `main.Get_Hash` if you run client.py directly). I suggest you put a try...except clause around the logic in Get_Data handler and print it 

[Jatin Sharma]

I rewrote the program to pass a class. Now I'm getting the following error:

Traceback (most recent call last):

  File "client.py", line 25, in <module>
    proc_res = stub.SquareProc(serverdata_pb2.client_class(class_str = pick_sq))

  File "/home/jatin/.local/lib/python3.8/site-packages/grpc/_channel.py", line 826, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/home/jatin/.local/lib/python3.8/site-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNKNOWN

details = "Exception calling application: Can't get attribute 'Square_Loc' on <module '__main__' from 'server.py'>"
debug_error_string = "{"created":"@1601012690.097357367","description":"Error received from peer ipv6:[::1]:50051","file":"src/core/lib/surface/call.cc","file_line":1061,"grpc_message":"Exception calling application: Can't get attribute 'Square_Loc' on <module '__main__' from 'server.py'>","grpc_status":2}"

So, Square_Loc is the class that I have defined in the client and sent to the server. Can you suggest to me how I can debug this? Thanks

[Russell Wu]

You will need to put your Square_Loc in a shared module, whose import path must be identical to your client and server, for example:

```

# in client.py

from .loc import Square_Loc

# in server.py

from .loc import Square_Loc

```

[Jatin Sharma]

Hi Russell,

It worked.

However, I believe this may not be the solution that I am looking for. The idea that I'm trying to apply is that when the server contains a database/pandas and the client sends a request to the server. The request includes the function that needs to operate on the database.

In the suggested solution, I have to make the class available on the server. If I have to make the class available on the server, it beats the purpose of using a client to send the pickled procedure. Is there a workaround such that I will not have to make the class available on the server?

[Russell Wu]

Generally, allowing users to run arbitrary code on the server is a huge security risk. But if you really wanna do it, in python it's relatively easy. 

What you need to send to the server is the code itself. To do that, you first get the code of the function/class, and send it to the server. On the server, you use eval to run it with a dict as local scope, then your method will be available in that dict/scope. Pseudo code as below

```

# in client.py

import inspect

from .loc import SquareLoc

code = inspect.getsource(SquareLoc)

req = myrequest_pb2.Request()

req.code = code

stub.Get_Data(req)

# in server.py handler code

ctx = {}

eval(req.code, {}, ctx)

SquareLoc = ctx['SquareLoc']

...

```

[Jatin Sharma]

Thanks for the suggestion.

I tried it. I passed it as a string. I get the following error:

  File "client.py", line 27, in <module>
    proc_res = stub.SquareProc(req)

  File "/home/jatin/.local/lib/python3.8/site-packages/grpc/_channel.py", line 826, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/home/jatin/.local/lib/python3.8/site-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNKNOWN

details = "Exception calling application: invalid syntax (<string>, line 1)"
debug_error_string = "{"created":"@1601270472.165351495","description":"Error received from peer ipv6:[::1]:8080","file":"src/core/lib/surface/call.cc","file_line":1061,"grpc_message":"Exception calling application: invalid syntax (<string>, line 1)","grpc_status":2}"

And when I passed it as bytes after changing it to pickle format, I got the following error:

Traceback (most recent call last):

  File "client.py", line 19, in <module>
    req.class_str = code
TypeError: "class Square_Loc():\n    def proc(self,df):\n        print('This is the client function being sent  has type str, but expected one of: bytes