[AOLSERVER] Problem uploading HUGE files. Possible AOLServer bug

[Eduardo Santos]

Hi everybody,

My system have very specific needs to upload really large files sometimes. My upload limit is actually 400MB, but I've received a request to raise this value to 2GB. 

I'm using AOLServer 4.5.1 and I know this is based on ns_limits command. So, I used the following piece of code to raise the upload size:

set max_file_upload_mb 2500

set size [expr $max_file_upload_mb * 1024 * 1024]

ns_limits set default -maxupload $size

However, the response was completely unexpected:

nrunning 3 nwaiting 0 ntimeout 0 ndropped 0 noverflow 0 maxwait 100 maxupload -1673527296 timeout 60 maxrun 100

maxupload had a negative size! I started digging up in C source code from AOLServer, and I guess I've found the problem. In the file nsd/limits.c we have the following piece of code:

    int i, val;

    static CONST char *opts[] = {

        "get", "set", "list", "register", NULL

    };

    enum {

        LGetIdx, LSetIdx, LListIdx, LRegisterIdx

    } opt;

    static CONST char *cfgs[] = {

        "-maxrun", "-maxwait", "-maxupload", "-timeout", NULL

    };

(...)

            switch (cfg) {

                case LCUploadIdx:

                    limitsPtr->maxupload = val;

                    break;

            }

        }

It seems like maxupload receives the value from val, wich is an int var. If I'm not wrong, the maximum value for C int vars is 2,147,483,647, and I was trying to setup this value to 2,621,440,000, wich seems to be causing an overflow.

Considering all this, if I just change the line below everything should work, right?

    int i, val;

I could just use:

    int i;

    bigint val;

Did I say something wrong or this line of thinking make any sense?

--
Eduardo Santos
Analista de Sistemas

http://eduardosan.wordpress.com
http://twitter.com/eduardosan

--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <list...@listserv.aol.com> with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.

[Jeff Rogers]

You could set the max upload value to INT_MAX (2^31-1, as you have
below), and that should work, but going higher will be difficult.
Simply changing the variable to a long in one place is insufficient,
because that value is handled in quite a few places, and you'd need to
change all of them; the obvious one is the Limits structure referenced
in the code snippet you have, but there's also buffer sizes and scanf
formats (to read the upload size from the http request) to handle, and
probably a few others as well.

-J

Eduardo Santos wrote:
> Hi everybody,
>
> My system have very specific needs to upload really large files
> sometimes. My upload limit is actually 400MB, but I've received a
> request to raise this value to 2GB.
>
> I'm using AOLServer 4.5.1 and I know this is based on ns_limits command.
> So, I used the following piece of code to raise the upload size:
>
> set max_file_upload_mb 2500
> set size [expr $max_file_upload_mb * 1024 * 1024]
> ns_limits set default -maxupload $size
>
> However, the response was completely unexpected:
>
> nrunning 3 nwaiting 0 ntimeout 0 ndropped 0 noverflow 0 maxwait 100
> maxupload -1673527296 timeout 60 maxrun 100
>
>

> /maxupload/ had a negative size! I started digging up in C source code

> from AOLServer, and I guess I've found the problem. In the file
> nsd/limits.c we have the following piece of code:
>
> int i, val;
> static CONST char *opts[] = {
> "get", "set", "list", "register", NULL
> };
> enum {
> LGetIdx, LSetIdx, LListIdx, LRegisterIdx
> } opt;
> static CONST char *cfgs[] = {
> "-maxrun", "-maxwait", "-maxupload", "-timeout", NULL
> };
>
> (...)
> switch (cfg) {
>
> case LCUploadIdx:
> limitsPtr->maxupload = val;
> break;
>
> }
> }
>
>

> It seems like /maxupload/ receives the value from /val/, wich is an int

[Eduardo Santos]

Hi Jeff,

Thank you for the answer. I was thiking that such a problem really exists, and that's why I decided not to touch the source code on this. 

It seems to me a bad thing to have a such low limit for file uploads. As I don't know this implementation, do you think there's any other way we can increase the file upload limit?

2011/5/16 Jeff Rogers <dv...@diphi.com>

[Jeff Rogers]

Eduardo Santos wrote:
> Hi Jeff,
>
> Thank you for the answer. I was thiking that such a problem really
> exists, and that's why I decided not to touch the source code on this.
>
> It seems to me a bad thing to have a such low limit for file uploads. As
> I don't know this implementation, do you think there's any other way we
> can increase the file upload limit?

I think someone else was looking at uploading large files last year.
The problem they were running into was that unless your link is fast the
browser (particularly IE) times out waiting for a response, even though
it hasn't finished uploading yet. I would not be surprised if you
encounter similar problems.

You (and your users) may find it more pleasant to use a upload tool that
automatically breaks the file into multiple parts for uploading, and the
server re-combines them after the upload is complete. For example,
Plupload appears to support this. http://www.plupload.com/

(I've not used plupload, it was just the first hit I got from a search).

-J