About the size of chunked_buffer

[Cong Wang]

Hi

I am just trying to transfer some data from server to client .

My code is :https://github.com/congfairy/transfer.git

What I have encountered is this problem:

.

it shows that the chunk body is too large.

Actually in the server.py, I have just set the chunk size to 1024*1024,which means the chunk size is 1M. What I would like to ask is that :

1.how do I know the _max_chunked_size of my connection?I have checked the code in /usr/local/lib/python3.5/site-packages/tornado/http1connection.py,so how can I set my _max_chunked_size?

2.the size of the chunk can only be set by the server?what the client get is based on the size that the server has set?In the client.py chunky function,I would like to know what is the chunk?it is set by the server?

3.what I would like to do is transfer some binary file almost 500M, to the client.The chunked size should be 1M or more.What I fulfilled is in multithread,so how could I fulfill the multi-stream?could you give me some suggestion?Or is there something I should improve in my code?

Thanks 

best regards

[Cong Wang]

And also  the fourth question:

The chunk size I set in server.py is 1024*1024,but the client get different chunk size,which is.What is wrong with the chunk size. Shouldn't the chunk size be the same from server to client?Thanks.

[Cong Wang]

I am so eager to hear from you ,please send me the answer  as early as possible ,thanks!

[Ben Darnell]

1. The "chunked body too large" message is a safety check, which protects you against a server that sends you a larger file than you're willing to handle. To download large files with SimpleAsyncHTTPClient, you must pass `max_body_size` to the AsyncHTTPClient constructor (be aware of the magic that allows AsyncHTTPClient objects to be reused; you may need to pass force_instance=True to disable this magic or ensure that you only create http clients in one place). 

2. TCP is a byte-stream protocol; chunk boundaries on the server do not necessarily correspond to chunk boundaries on the client. There are many levels of "chunking" here: your code reads 1MB from the file at a time and passes it to the IOStream via self.write(). Then the IOStream sends it to the socket in smaller chunks (typically around 100KB). The network then breaks it up into packets (1500 bytes). On the other side, the client reassembles packets into larger chunks (64KB, and this is not currently configurable). 

3. Why does the HTTP client's chunk size matter to you? You can take multiple 64KB chunks and assemble them into 1MB chunks if you want. This would be a little more convenient if the HTTP client's chunk size were configurable, but the performance difference should be small. 

I don't understand your question about multithread or multistream. HTTP uses a single stream per request, and Tornado is a single-threaded framework. 

4. The client's 64KB limit is an upper bound on chunk size. It may give you smaller chunks if that's what's in the network buffer. (this is one reason why increasing the client's chunk size may not do what you want)

On Fri, Dec 16, 2016 at 3:24 PM Cong Wang <congfa...@gmail.com> wrote:

And also  the fourth question:

The chunk size I set in server.py is 1024*1024,but the client get different chunk size,which is.What is wrong with the chunk size. Shouldn't the chunk size be the same from server to client?Thanks.

2016-12-16 15:16 GMT+08:00 Cong Wang <congfa...@gmail.com>:

Hi

I am just trying to transfer some data from server to client .

My code is :https://github.com/congfairy/transfer.git

What I have encountered is this problem:

.

it shows that the chunk body is too large.

Actually in the server.py, I have just set the chunk size to 1024*1024,which means the chunk size is 1M. What I would like to ask is that :

1.how do I know the _max_chunked_size of my connection?I have checked the code in /usr/local/lib/python3.5/site-packages/tornado/http1connection.py,so how can I set my _max_chunked_size?

2.the size of the chunk can only be set by the server?what the client get is based on the size that the server has set?In the client.py chunky function,I would like to know what is the chunk?it is set by the server?

[Cong Wang]

Hi

1.I have changed the client to SimpleAsyncHTTPClient(max_body_size),and there is no buffer problem.

But how could I pass the parameter force_instance=True to SimpleAsyncHTTPClient,since I have checked that force_instance is not a parameter of SimpleAsyncHTTPClient but a parameter of AsyncHTTPClient,so how could I use it?

2.for the downloading of file ,tornado client will load all the file to the buffer?And then write it to the filesystem?That is why I should change the max_body_size of SimpleAsyncHTTPClient?I am a little confused.

Thanks 

best regards,

cong

[Ben Darnell]

On Sat, Dec 17, 2016 at 4:08 PM Cong Wang <congfa...@gmail.com> wrote:

Hi

1.I have changed the client to SimpleAsyncHTTPClient(max_body_size),and there is no buffer problem.

But how could I pass the parameter force_instance=True to SimpleAsyncHTTPClient,since I have checked that force_instance is not a parameter of SimpleAsyncHTTPClient but a parameter of AsyncHTTPClient,so how could I use it?

The `force_instance` parameter is defined in `AsyncHTTPClient.__new__`, which is called when you create any AsyncHTTPClient subclass, including SimpleAsyncHTTPClient. The `__new__` method is a relatively obscure and magical feature of python that we use to share http clients (including connection limits and pools).

 

2.for the downloading of file ,tornado client will load all the file to the buffer?And then write it to the filesystem?That is why I should change the max_body_size of SimpleAsyncHTTPClient?I am a little confused.

No, Tornado does not load the whole file at once. As long as you use flush() on the server and streaming_callback on the client, only a small piece of the file will be in memory at once. The max_body_size limit is just a safety check that should probably be documented better. It makes it harder for you to accidentally run out of disk space if your streaming_callback writes everything it receives to disk.

-Ben

[Cong Wang]

with the sentence above ,I can't set the max_body_size parameter to http_client(the max_body_size dones't change ,it is also the default one).how could I pass force_instance and max_body_size to http_client?

I am so eager to know that ,thanks!

[Ben Darnell]

configure() is a class method, not an instance method. Call it with `AsyncHTTPClient.configure('tornado.simple_httpclient.SimpleAsyncHTTPClient', max_body_size=512*1024*1024)`. If you use `configure()` you don't also need `force_instance`.