Problem with large file upload
Steven Saunderson
I am writing a Perl/CGI script to upload files from a HTTP client. It
works with small files but fails with 'Internal Server Error' with files
greater than about 512kB. The abort happens after the file is uploaded
but before the output file is opened. I have set MAX_POST to 16MB.
Can anybody offer any clues on fixing this problem ? Is there a maximum
input time parameter that I can change ?
Thanks,
--
Steven
--
PLEASE NOTE: comp.infosystems.www.authoring.cgi is a
SELF-MODERATED newsgroup. aa.net and boutell.com are
NOT the originators of the articles and are NOT responsible
for their content.
HOW TO POST to comp.infosystems.www.authoring.cgi:
http://www.thinkspot.net/ciwac/howtopost.html
Gunnar Hjalmarsson
> I am writing a Perl/CGI script to upload files from a HTTP client. It
> works with small files but fails with 'Internal Server Error' with files
> greater than about 512kB. The abort happens after the file is uploaded
> but before the output file is opened.
Where is your code? What does the error log say?
Without that info, it's hard to help. There is a CPAN module I wrote to
facilitate file uploads, CGI::UploadEasy, which may or may not make a
difference.
--
Gunnar Hjalmarsson
Email: http://www.gunnar.cc/cgi-bin/contact.pl
Steven Saunderson
<nor...@gunnar.cc> wrote:
> Steven Saunderson wrote:
> > I am writing a Perl/CGI script to upload files from a HTTP client. It
> > works with small files but fails with 'Internal Server Error' with files
> > greater than about 512kB. The abort happens after the file is uploaded
> > but before the output file is opened.
>
> Where is your code? What does the error log say?
Hi Gunnar, thanks for the reply. The script is 'upload.pl' and the
source is at <http://phelum.com/files/upload.pls>. I don't have access
to any server log that may assist.
> Without that info, it's hard to help. There is a CPAN module I wrote to
> facilitate file uploads, CGI::UploadEasy, which may or may not make a
> difference.
I will investigate. Thanks for the info.
--
Steven
Gunnar Hjalmarsson
> Gunnar Hjalmarsson wrote:
>>Steven Saunderson wrote:
>>>works with small files but fails with 'Internal Server Error' with files
>>>greater than about 512kB. The abort happens after the file is uploaded
>>>but before the output file is opened.
>>
>>Where is your code? What does the error log say?
>
> Hi Gunnar, thanks for the reply. The script is 'upload.pl' and the
> source is at <http://phelum.com/files/upload.pls>. I don't have access
> to any server log that may assist.
Well, since you use CGI::Carp 'fatalsToBrowser', the error message
should be displayed on screen.
>>There is a CPAN module I wrote to
>>facilitate file uploads, CGI::UploadEasy, which may or may not make a
>>difference.
>
> I will investigate. Thanks for the info.
Now, when I see your code, I think that CGI::UploadEasy might be
helpful. This is a script I wrote to demonstrate how the module can be
used: http://www.gunnar.cc/programs/upload.pl.txt
--
Gunnar Hjalmarsson
Email: http://www.gunnar.cc/cgi-bin/contact.pl
--
Steven Saunderson
<nor...@gunnar.cc> wrote:
> Well, since you use CGI::Carp 'fatalsToBrowser', the error message
> should be displayed on screen.
It helps in that some errors (e.g. syntax) are reported. But all I get
when the upload fails is 'internal server error'.
> Now, when I see your code, I think that CGI::UploadEasy might be
> helpful. This is a script I wrote to demonstrate how the module can be
> used: http://www.gunnar.cc/programs/upload.pl.txt
I'm not sure how or if I can install UploadEasy so I can use it. I have
only been learning Perl for about a week now and my only reference is a
few chapters in a Goerzen Linux book. I will get a better reference
book and learn the language.
I changed my script to use sysread/syswrite instead of read/write. This
might have helped (not sure why) because sometimes I can upload a 1MB
file. But it is not reliable and sometimes fails on a 20kB file. Large
files sometimes get 'connection timed out' instead of 'internal server
error'. The server will accept large uploads using PHP so I should be
able to do the same using Perl.
Thanks for your help,
--
Steven
Malcolm Dew-Jones
: Hi,
: I am writing a Perl/CGI script to upload files from a HTTP client. It
: works with small files but fails with 'Internal Server Error' with files
: greater than about 512kB. The abort happens after the file is uploaded
: but before the output file is opened. I have set MAX_POST to 16MB.
: Can anybody offer any clues on fixing this problem ? Is there a maximum
: input time parameter that I can change ?
I think you mean you're using CGI.pm
It saves the upload into a temporary file. Perhaps the server is not
allowing your script to save a temporary file larger than 1/2 Meg.
The script may not output an error since the server may just kill it.
According to perldoc CGI there is a function called
CGI::upload_hook(\&hook,$data);
It sounds like that can be used to control how the upload data is saved,
such as into a file of your choosing (which may get around a size
restriction in the temp directory).
The default temp directory appears to be found as follows...
1. if the current user (e.g. "nobody") has a directory named
"tmp" in its home directory, use that (Unix systems only).
2. if the environment variable TMPDIR exists, use the location
indicated.
3. Otherwise try the locations /usr/tmp, /var/tmp, C:\temp,
/tmp, /temp, ::Temporary Items, and \WWW_ROOT.
So you could check those directories to see if there is some kind of
quote involved.
Gunnar Hjalmarsson
> Gunnar Hjalmarsson wrote:
>>helpful. This is a script I wrote to demonstrate how the module can be
>>used: http://www.gunnar.cc/programs/upload.pl.txt
>
> I'm not sure how or if I can install UploadEasy so I can use it. I have
> only been learning Perl for about a week now and my only reference is a
> few chapters in a Goerzen Linux book. I will get a better reference
> book and learn the language.
Start here: http://learn.perl.org/
> I changed my script to use sysread/syswrite instead of read/write.
^^^^^^^^ ^^^^^
Use print() instead of write/syswrite.
--
Gunnar Hjalmarsson
Email: http://www.gunnar.cc/cgi-bin/contact.pl
--
Steven Saunderson
<nor...@gunnar.cc> wrote:
> Start here: http://learn.perl.org/
Okay, thanks for the link.
> > I changed my script to use sysread/syswrite instead of read/write.
> ^^^^^^^^ ^^^^^
> Use print() instead of write/syswrite.
Shall do, and I will also find out why.
--
Steven
Steven Saunderson
Dew-Jones) wrote:
> I think you mean you're using CGI.pm
Yes, precisely, I wasn't sure how to best describe it.
> It saves the upload into a temporary file. Perhaps the server is not
> allowing your script to save a temporary file larger than 1/2 Meg.
>
> The script may not output an error since the server may just kill it.
This sounds right. I'm not sure about the size limit because tests are
inconsistent. Sometimes I can upload a 800kB file, sometimes it will
fail on a much smaller file. I was wondering about a maximum time limit
(similar to PHP) but I can't find any mention of such a limit.
> According to perldoc CGI there is a function called
> CGI::upload_hook(\&hook,$data);
>
> It sounds like that can be used to control how the upload data is saved,
> such as into a file of your choosing (which may get around a size
> restriction in the temp directory).
Thanks, I will investigate this. I assumed this hook was for progress
reports but didn't check any further.
--
Steven
Steven Saunderson
<nor...@gunnar.cc> wrote:
> > I'm not sure how or if I can install UploadEasy so I can use it. I have
> > only been learning Perl for about a week now and my only reference is a
> > few chapters in a Goerzen Linux book. I will get a better reference
> > book and learn the language.
>
> Start here: http://learn.perl.org/
Thanks. I bought Perl by Example which has been a great help and also
learnt from <http://stein.cshl.org/WWW/CGI/>.
> Use print() instead of write/syswrite.
Done and understood.
All this didn't help my original problem (upload script aborting on
large upload. Tonight I found apache log errors "Filtering against POST
payload requested but payload is not available". It looks like there is
an Apache module called mod_security that scans POST data and this seems
to be causing the abort. I found a .htaccess option "SecFilterScanPOST
Off" that seems to help. Tonight I have managed to upload files larger
than 4MB so things look promising.
My current script is at <http://phelum.com/files/upload.pl> if anybody
is interested. It will be delivered as a text file.
Thanks,
--
Steven
Steven Saunderson
Dew-Jones) wrote:
> I think you mean you're using CGI.pm
>
> It saves the upload into a temporary file. Perhaps the server is not
> allowing your script to save a temporary file larger than 1/2 Meg.
Hi Malcolm,
I have done some more testing and checked CGI.pm. Mysteriously,
everything now works so I suspect the problem was due to recent security
changes in the server.
When I couldn't get the Perl script (using CGI.pm) to work I wrote a C++
version which uploads 20MB reliably. Then I wrote a Perl version
(without CGI.pm) but this didn't work. The next day it did upload
correctly and the original also worked.
The links to the files are :
<http://phelum.com/files/upload.pl> - Perl using CGI.pm
<http://phelum.com/files/upload1.pl> - Perl without CGI.pm
<http://phelum.com/files/upload.cgi> - C++ executable
<http://phelum.com/files/upload.cpp> - C++ source
<http://phelum.com/files/CGI.inc> - C++ include
The original purpose for the upload script has now passed but I have
certainly learnt much more about Perl, CGI, and POST streams.
Thanks for your input. It did inspire me to search through CGI.pm and
see how it works.
--
Steven