Increase chunk size for auto-chunked file upload requests

[james isaac]

Hi,

I have a working piece of code that is able to upload files upto ~1.5 GB size.

Beyond this size, I get a "Content-Length > Int.Max Size not supported for non-(auto)-chunked requests" error response.

So I tried modifying the spray-can properties. These are the contents of my application.conf file:

spray.can.server {

  request-timeout = 124s

  idle-timeout = 125s

  request-chunk-aggregation-limit = 0

  parsing = ${spray.can.parsing}

}

spray.can.parsing {

  max-content-length = 8g

  max-chunk-size =100m

#  incoming-auto-chunking-threshold-size = 20

}

spray.can.host-connector {

  max-connections = 8

}

If I uncomment the "incoming-auto-chunking-threshold-size = 20" line, the file gets uploaded, but it takes a huge amount of time to upload. 

I noticed that even though I had set the chunk size to 10MB in the code, the actual chunk size that is being used at runtime is much smaller - sometimes around 500 0r 1000 bytes. The size also varies for each chunk. I expect each chunk to be 10MB since I specify it in the code like this:

 val parts = req.asPartStream(10485760)

Please note that the effective runtime chunk size becomes 10 MB when I comment the "incoming-auto-chunking-threshold-size = 20" line in the conf file.

Is there any other property that I need to change?

Regards,

James 

[Johannes Rudolph]

Hi James,

On Fri, May 29, 2015 at 6:56 AM, james isaac <jamesi...@gmail.com> wrote:
> I noticed that even though I had set the chunk size to 10MB in the code, the
> actual chunk size that is being used at runtime is much smaller - sometimes
> around 500 0r 1000 bytes.
> The size also varies for each chunk.

Yes, data will be delivered as soon as it comes in. The minimum sizes
then only depend on what's currently in the socket receive buffer.
It's a bit strange if you receive less than the MTU of your network in
one go, though.

> I expect each
> chunk to be 10MB since I specify it in the code like this:
>
> val parts = req.asPartStream(10485760)

No, this doesn't work for chunked requests. This method can be used if
you have a request with a non-chunked entity and want to create a
chunked request out of it but not the other way round.

> Please note that the effective runtime chunk size becomes 10 MB when I
> comment the "incoming-auto-chunking-threshold-size = 20" line in the conf
> file.

I guess, that's because first spray collects all the (auto-)chunks
into a request with a non-chunked entity which you then manually chunk
with `asPartStream`.

You should try to find out why receiving things in smaller chunks is
so much slower than receiving it in bigger chunks. E.g. if you are
writing data to disk make sure you buffer the writes.

HTH
Johannes

-----------------------------------------------
Johannes Rudolph
http://virtual-void.net