Multipart boundary calculation time for large files

[Peter Stokes]

Hi,

I am attempting to use Mojo::UserAgent from Mojolicious 3.30 with Perl 5.16.1 on Windows to perform file uploads with potentially substantial file sizes (10s of megabytes) based upon the documentation provided at http://mojolicio.us/perldoc/Mojolicious/Guides/Cookbook#Large_file_upload.

I have discovered that the time taken to calculate a unique boundary identifier within Mojo::Content::Multipart::build_boundary can be considerable for large files due to the resultant call to Mojo::Asset::File::contains that is used to ensure that the potential boundary identifier chosen does not occur within the file to be incorporated into the HTTP request.

When attempting to use the following ojo one-liner:

perl -Mojo -e “f($ARGV[0], {file => {file => $ARGV[1]}})” <url> <file>

a problem is encountered that the time take to determine a multipart boundary that does not occur within the file provided occurs after a connection is established, which may result in an idle connection timeout from the server.

Modifying the code to be:

my $agent = Mojo::UserAgent->new;

my $tx = $agent->build_form_tx($ARGV[0], {file => {file => $ARGV[1]}});

$tx->fix_headers;

$agent->start($tx);

alleviates part of the problem by ensuring that the time penalty associated with determining the multipart boundary identifier occurs prior to establishing a connection with the server. However, this version is not quite as catchy as a one-liner.

I notice that this issue has been encountered before (see: https://groups.google.com/d/topic/mojolicious/6DNYc3nKAvA/discussion) but the proposed solution does not take into account uniqueness of the chosen boundary identifier with respect to the data being sent, so I assume that it is invalid?

I am afraid that I am somewhat new to Mojolicious and web development. So I was wondering whether anyone can comment on the practicalities of re-ordering the processing within Mojolicous so that the ojo one-liner may pay the cost of determining a unique multipart boundary prior to establishing a connection with the server and/or speeding up the process of determining uniqueness of the chosen boundary identifier (i.e. scanning for the occurrence of a pattern within large asset files).

Thanks in advance.

Peter

[sri]

You might be interested in this change.

https://github.com/kraih/mojo/commit/b51393489f42c8f5f6e76a0a3459a0d60fb54c57

--
sebastian

[Peter Stokes]

Hi Sebastian,

 

That's brilliant, Thank You!

 

Peter