Large Files not able to upload even as chunks

[VigneshDhanraj G]

Hi Team,

I am not able to upload large files more than 2GB even after chunk. I have found the file is uploaded still exeception raised? Please help me to understand the issue here and way to fix this problem.

Traceback (most recent call last):
  File "demo_client.py", line 12, in <module>
    client.upload(in_file_name)
  File "/home/vigneshdhanraj/Project/grpc-upload/grpc-file-transfer/src/lib.py", line 34, in upload
    response = self.stub.upload(chunks_generator)
  File "/home/vigneshdhanraj/Project/grpc-upload/myenv/lib/python3.6/site-packages/grpc/_channel.py", line 1011, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/home/vigneshdhanraj/Project/grpc-upload/myenv/lib/python3.6/site-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    status = StatusCode.UNKNOWN
    details = "Exception calling application: Value out of range: 2147483648"
    debug_error_string = "{"created":"@1587649701.194973268","description":"Error received from peer ipv6:[::1]:8888","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Exception calling application: Value out of range: 2147483648","grpc_status":2}"

Regards,

VigneshDhanraj G

[Mya Pitzeruse]

Is there an associated error on the server? The error message suggests a server error with an index out of bounds exception: 2^31. Only way I could see something like that happening is if the file was being buffered on the server side in memory.

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/cbc945c2-6d3a-4d2b-a52f-2a6b59a9dfba%40googlegroups.com.

--

Mya Pitzeruse

Principal Software Engineer - Service Infrastructure

Gender Pronouns: She, Her, Hers

mjp...@indeed.com

Indeed - We help people get jobs.

Indeed.com

Facebook  |  Twitter  |  Instagram

[VigneshDhanraj G]

Thanks Mya, 

There is no any error on the server and  i am getting error if i upload more than 2GB file, i am constanly uploading chunk on the storge.

On Monday, April 27, 2020 at 8:42:50 PM UTC+5:30, Mya Pitzeruse wrote:

Is there an associated error on the server? The error message suggests a server error with an index out of bounds exception: 2^31. Only way I could see something like that happening is if the file was being buffered on the server side in memory.

On Mon, Apr 27, 2020 at 6:56 AM VigneshDhanraj G <vignesh...@gmail.com> wrote:

Hi Team,

I am not able to upload large files more than 2GB even after chunk. I have found the file is uploaded still exeception raised? Please help me to understand the issue here and way to fix this problem.

Traceback (most recent call last):
  File "demo_client.py", line 12, in <module>
    client.upload(in_file_name)
  File "/home/vigneshdhanraj/Project/grpc-upload/grpc-file-transfer/src/lib.py", line 34, in upload
    response = self.stub.upload(chunks_generator)
  File "/home/vigneshdhanraj/Project/grpc-upload/myenv/lib/python3.6/site-packages/grpc/_channel.py", line 1011, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/home/vigneshdhanraj/Project/grpc-upload/myenv/lib/python3.6/site-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    status = StatusCode.UNKNOWN
    details = "Exception calling application: Value out of range: 2147483648"
    debug_error_string = "{"created":"@1587649701.194973268","description":"Error received from peer ipv6:[::1]:8888","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Exception calling application: Value out of range: 2147483648","grpc_status":2}"

Regards,

VigneshDhanraj G

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.

To unsubscribe from this group and stop receiving emails from it, send an email to grp...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/cbc945c2-6d3a-4d2b-a52f-2a6b59a9dfba%40googlegroups.com.

[Michael Webster]

Do you have any snippets you can share from your code?  I use python-grpc in another project for transferring files and I've never had an issue with files > 2gb, with 1mb chunks (local network operations).

[VigneshDhanraj G]

Thanks Michael,

syntax = "proto3";

service FileServer {

  rpc upload(stream Chunk) returns (Reply) {}

  rpc download(Request) returns (stream Chunk) {}

}

message Chunk {

  bytes buffer = 1;

}

message Request {

  string name = 1;

}

message Reply {

  int32 length = 1;

and my client.py

def get_file_chunks(filename):

    with open(filename, 'rb') as f:

        while True:

            piece = f.read(CHUNK_SIZE);

            if len(piece) == 0:

                return

            yield chunk_pb2.Chunk(buffer=piece)

def save_chunks_to_file(chunks, filename):

    with open(filename, 'wb') as f:

        for chunk in chunks:

            f.write(chunk.buffer)

class FileClient:

    def __init__(self, address):

        channel = grpc.insecure_channel(address)

        self.stub = chunk_pb2_grpc.FileServerStub(channel)

    def upload(self, in_file_name):

        chunks_generator = get_file_chunks(in_file_name)

        response = self.stub.upload(chunks_generator)

        assert response.length == os.path.getsize(in_file_name)

    def download(self, target_name, out_file_name):

        response = self.stub.download(chunk_pb2.Request(name=target_name))

        save_chunks_to_file(response, out_file_name)

if __name__ == '__main__':

    client = FileClient('localhost:8888')

    # demo for file uploading

    in_file_name = sys.argv[1]

    client.upload(in_file_name)

[Thomas Mercier]

The maximum value of an int32 is 2,147,483,647. Why are you using a signed type for length in the first place though?

To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/e3f6c515-71cd-40ca-bafb-b409788cb1fb%40googlegroups.com.

[vignesh...@gmail.com]

Thanks Thomas,

Thought like upload request needs to be bytes and for reply used int, that is the issue thanks a lot. Issue solved :)

Thanks Team, it is my mistake.

On Tuesday, April 28, 2020 at 12:37:06 AM UTC+5:30, Thomas Mercier wrote:

The maximum value of an int32 is 2,147,483,647. Why are you using a signed type for length in the first place though?

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/e3f6c515-71cd-40ca-bafb-b409788cb1fb%40googlegroups.com.