grpc streaming large file

[Weidong Lian]

Hello grpcs,

I have the following task sent from client to server.

service applis {

  rpc GenerateVoxelMesh(VoxelMeshRequest) returns (VoxelMeshReply) {}

}

message VoxelMeshRequest {

  any image_input = 1;

  bool smooth = 4;        

  int32 iterations = 5;   

  double mu = 6;          

  double lambda = 7;   

  double scale = 8;      

  repeat int32 part_ids = 9; 

}

message VoxelMeshReply {

  any nas_output = 1; // text file

}

The image_input, nas_output are the binary files that can be fairly large sometimes. I would guess the `any` is not a recommended type. 

It is preferred to use stream chunk bytes to send and receive the image and nas files. However, if we stream chunk file, we can not send

the other request parameters at one call. We will have to make multiple calls and make server side a state machine. It increases the complexity.

I am just wondering if there any more element design or what the idiomatic way of doing this in grpc? 

the possible design like below.

service applis {

  rpc GenerateVoxelMesh(stream VoxelMeshRequest) returns (stream VoxelMeshReply) {}

}

message VoxelMeshRequest {

    oneof test_oneof {

       FileChunk image_input = 1;

       VoxelMeshParamters mesh_params = 2;

    }

}

message FileChunk {

 bytes chunk = 1;

}

message VoxelMeshParameters {

  bool smooth = 4;        

  int32 iterations = 5;   

  double mu = 6;          

  double lambda = 7;   

  double scale = 8;      

  repeat int32 part_ids = 9; 

}

message VoxelMeshReply {

  FileChunk nas_output = 1; // text file

}

Any suggestion will be appreciated. 

Thanks in advance,

Weidong

[Eric Gribkoff]

Either approach could be appropriate: dividing the large message into chunks or sending it all in one message (note: gRPC has a default max message size that varies by language but can be configured). Which one performs best for your use case will depend on a number of other factors. There was some earlier discussion around this in https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/grpc-io/MbDTqNXhv7o/cvPjrhwCAgAJ.

Thanks,

Eric

[Raul Sampedro]

I recently had to implement a similar solution to send files. I considered these options:

a) Your option, rpc call that sends back chunks

b) RPC call that replies with a port number where the server will listen for the file (or in your case, some other response to the rpc call, like a file identifier that will be requested in other way)

c) Do not use rpc calls, and use regular sockets for file transfers, handling other types of communication via rpc

After researching 'a)', and implementing 'b)', I ended up going back to 'c)'. My server has two ports open, one for RPC calls, and one for FTP calls. They are separate threads that know how to talk to each other, but files are not served through RPC anymore.

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscribe@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/a4c82a7c-6542-4d51-be35-7c2b9e583dc9%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Srini Polavarapu]

You could set request parameters as metadata for the file upload RPC. Server receives metadata before the streaming data is received.

To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.

To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.

[Rudi Chiarito]

For large files, could you send a first request without any chunks, so that the server gets all the parameters first? Then the rest of the stream would be requests with just chunks that could you process progressively, as they come in.

For small files, you'd just send a stream with one request that has both the single chunk and the parameters.

--

You received this message because you are subscribed to the Google Groups "grpc.io" group.

To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscribe@googlegroups.com.

To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/05b404bb-b7a4-4f93-9e21-ea301da5b4d2%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Rudi Chiarito — Infrastructure — Clarifai, Inc.

"Trust me, I know what I'm doing." (Sledge Hammer!)

[Weidong Lian]

I kinda agree with you. After googling and testing, the grpc may not be a proper tool to handle the client-server with tight logic, e.g. both sides are state-machines, like in desktop client and remote server. We can still use protobuf, but your own socket/websocket might be easier solutions than grpc. 

grpc is a great tool to replace the REST APIs among microservices.

Your option c) sounds reasonable, but adds complications to the whole system. 

[Weidong Lian]

Thanks. This is exactly my current plan. The first message will be the parameters and file size info, then I will start to receive the subsequent streamed chunks. However, I do not distinguish the small or large files for simplicity. 

Thanks again for your suggestion, I will think about if it is worth having two APIs for small and large files. 

Weidong

[Weidong Lian]

Thanks Eric. The link is great to read. 

Weidong

[mu...@iadaptime.com]

Hi ,

        Can you share the image upload code.i m tried to image upload