[AOLSERVER] Problem with file uploads larger than maxinput

[Fenton, Brian]

Hi

When I upload a file larger than the maxinput I get the Firefox browser page with a "the connection to the server was reset while the page was loading" message. Looking in the access logs it seems to suggest that it doesn't even hit the website as there is no log entry. A file smaller than the maxinput uploads with no problems and everything is correctly logged. Increasing maxinput allows the file to upload and also is everything is correctly logged (so it's not a browser or network issue).

I don't fully understand this - if there is no entry in the access log, does this mean that I'm not even connnecting to AOLserver? As the success of the upload is dependent on maxinput I imagine there is some sort of connection to the server. Assuming there is a connection how can I log it or intercept it?

Can anyone advise me on what to try? Ideally, I'd like to be able to capture the fact that the uploaded file is larger than our limit, and feedback to the user. Even more ideally, I'd like to be able to tell them BEFORE they upload. :-)

I've got the following settings in my config.tcl (this is AOLserver 4.5.1 but also happens on 4.0.10 and on Windows version)

set max_file_upload_mb 10
set max_file_upload_min 5
ns_section ns/server/${server}/module/nssock
ns_param maxinput [expr {$max_file_upload_mb * 1024 * 1024}] ;# Maximum File Size for uploads in bytes
ns_param maxpost [expr {$max_file_upload_mb * 1024 * 1024}] ;# Maximum File Size for uploads in bytes
ns_param recvwait [expr {$max_file_upload_min * 60}] ;# Maximum request time in minutes

--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <list...@listserv.aol.com> with the
body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.

[Jim Davidson]

Hi,

The short answer is no, there's no access log entry although there may be a server log message buried in the chatter.

The reason is the access log is a "trace" that fires at the end of an HTTP connection and the request isn't a connection until all the content has been read and the data structures hooked up and passed over to a connection thread. In retrospect, transaction logging should be a lower-level built-in that can deal with logging these aborted transactions.

You can see what may be logged in the server log by looking at the LogReadError function at the end of nsd/driver.c, maybe a E_RRANGE, "max request exceeded".

-Jim

[Dossy Shiobara]

IIRC, in AOLserver 4.5+, until the whole HTTP request is received by the
server, it's being handled entirely by the driver thread -- and if the
maxinput ns_limit is exceeded, the driver thread drops the connection
before it's ever dispatched to a "connection thread" (which might be
better referred to as a "request handler thread").

I could be wrong, it's been a few years since I looked at that code ...
but that's what I seem to recall.

On 6/23/11 8:04 AM, Fenton, Brian wrote:
> I don't fully understand this - if there is no entry in the access log, does this mean that I'm not even connnecting to AOLserver? As the success of the upload is dependent on maxinput I imagine there is some sort of connection to the server. Assuming there is a connection how can I log it or intercept it?

--
Dossy Shiobara | "He realized the fastest way to change
do...@panoptic.com | is to laugh at your own folly -- then you
http://panoptic.com/ | can let go and quickly move on." (p. 70)
* WordPress * jQuery * MySQL * Security * Business Continuity *

[Fenton, Brian]

Thanks Jim

I should have mentioned that there was nothing in the error log either.

So, how would you recommend I capture the fact that the uploaded file is larger than our limit, and feedback to the user?

thanks
Brian

________________________________________
From: AOLserver Discussion [AOLS...@LISTSERV.AOL.COM] On Behalf Of Jim Davidson [jgdav...@MAC.COM]
Sent: 23 June 2011 13:47
To: AOLS...@LISTSERV.AOL.COM
Subject: Re: [AOLSERVER] Problem with file uploads larger than maxinput

[Dossy Shiobara]

Alternatively, instead of the driver thread tossing the connection on
the floor, it could mark the HTTP request as being truncated in some
way, and passing up to maxinput bytes through to the connection thread.
Then, the connection thread can decide how to handle the request -- but,
the driver thread still makes sure that the hard limits are enforced on
the request that will be passed on to the conn thread ... ?

On 6/23/11 8:47 AM, Jim Davidson wrote:
> The reason is the access log is a "trace" that fires at the end of an HTTP connection and the request isn't a connection until all the content has been read and the data structures hooked up and passed over to a connection thread. In retrospect, transaction logging should be a lower-level built-in that can deal with logging these aborted transactions.

--

Dossy Shiobara | "He realized the fastest way to change
do...@panoptic.com | is to laugh at your own folly -- then you
http://panoptic.com/ | can let go and quickly move on." (p. 70)
* WordPress * jQuery * MySQL * Security * Business Continuity *

[Majid Khan]

I think if its possible in your case use the javascript and calculate the size of the upload file and return it right away if its  exceeds the limit this way you dont' need to worry about sending the request to the server, waiting for a while and ended up with nothing returned.

Regards,

Majid.

[Jim Davidson]

Howdy,

Appears you need to set driver "debug" mode for the driver for the given socket module thing:

ns_section "ns/server/server1/module/nssock"
ns_param debug 1

Hopefully that's not too much muck in the server log.

-Jim

[Jim Davidson]

I think this could work, i.e., dummy up a "runt" or "truncated" connection to exercise the rest of the code. But, it's possible something would go uninitialized or assume goodness, i.e., just run off the end of the truncated content. Hmm...

I think today, in 2011, some of the flexibility we imagined back in 1995 isn't really so needed. This includes general purpose logging plugins and network drivers when there really is just the common log format, ordinary and SSL sockets. More in the core for the base HTTP processing could make things easier to maintain.

-Jim

[Fenton, Brian]

Thanks Majid

well there are Flash based on front end solutions e.g. SWFUpload, but it's not possible in Javascript. I think it's still important to have a server solution as well. I do agree though that it's kind of crazy to find out AFTER you upload a file, that the file was too big!

Brian

________________________________________
From: AOLserver Discussion [AOLS...@LISTSERV.AOL.COM] On Behalf Of Majid Khan [majid...@GMAIL.COM]
Sent: 23 June 2011 14:34

To: AOLS...@LISTSERV.AOL.COM
Subject: Re: [AOLSERVER] Problem with file uploads larger than maxinput

I think if its possible in your case use the javascript and calculate the size of the upload file and return it right away if its exceeds the limit this way you dont' need to worry about sending the request to the server, waiting for a while and ended up with nothing returned.

Regards,

Majid.

On Thu, Jun 23, 2011 at 5:16 PM, Fenton, Brian <Brian....@quest.ie<mailto:Brian....@quest.ie>> wrote:
Thanks Jim

I should have mentioned that there was nothing in the error log either.

So, how would you recommend I capture the fact that the uploaded file is larger than our limit, and feedback to the user?

thanks
Brian

________________________________________
From: AOLserver Discussion [AOLS...@LISTSERV.AOL.COM<mailto:AOLS...@LISTSERV.AOL.COM>] On Behalf Of Jim Davidson [jgdav...@MAC.COM<mailto:jgdav...@MAC.COM>]

Sent: 23 June 2011 13:47

To: AOLS...@LISTSERV.AOL.COM<mailto:AOLS...@LISTSERV.AOL.COM>

Subject: Re: [AOLSERVER] Problem with file uploads larger than maxinput

Hi,

The short answer is no, there's no access log entry although there may be a server log message buried in the chatter.

The reason is the access log is a "trace" that fires at the end of an HTTP connection and the request isn't a connection until all the content has been read and the data structures hooked up and passed over to a connection thread. In retrospect, transaction logging should be a lower-level built-in that can deal with logging these aborted transactions.

You can see what may be logged in the server log by looking at the LogReadError function at the end of nsd/driver.c, maybe a E_RRANGE, "max request exceeded".

-Jim

On Jun 23, 2011, at 6:04 AM, Fenton, Brian wrote:

> Hi
>
> When I upload a file larger than the maxinput I get the Firefox browser page with a "the connection to the server was reset while the page was loading" message. Looking in the access logs it seems to suggest that it doesn't even hit the website as there is no log entry. A file smaller than the maxinput uploads with no problems and everything is correctly logged. Increasing maxinput allows the file to upload and also is everything is correctly logged (so it's not a browser or network issue).
>
> I don't fully understand this - if there is no entry in the access log, does this mean that I'm not even connnecting to AOLserver? As the success of the upload is dependent on maxinput I imagine there is some sort of connection to the server. Assuming there is a connection how can I log it or intercept it?
>
> Can anyone advise me on what to try? Ideally, I'd like to be able to capture the fact that the uploaded file is larger than our limit, and feedback to the user. Even more ideally, I'd like to be able to tell them BEFORE they upload. :-)
>
> I've got the following settings in my config.tcl (this is AOLserver 4.5.1 but also happens on 4.0.10 and on Windows version)
>
> set max_file_upload_mb 10
> set max_file_upload_min 5
> ns_section ns/server/${server}/module/nssock
> ns_param maxinput [expr {$max_file_upload_mb * 1024 * 1024}] ;# Maximum File Size for uploads in bytes
> ns_param maxpost [expr {$max_file_upload_mb * 1024 * 1024}] ;# Maximum File Size for uploads in bytes
> ns_param recvwait [expr {$max_file_upload_min * 60}] ;# Maximum request time in minutes
>
>
> --
> AOLserver - http://www.aolserver.com/
>

> To Remove yourself from this list, simply send an email to <list...@listserv.aol.com<mailto:list...@listserv.aol.com>> with the

> body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.

--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <list...@listserv.aol.com<mailto:list...@listserv.aol.com>> with the

body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.

--
AOLserver - http://www.aolserver.com/

To Remove yourself from this list, simply send an email to <list...@listserv.aol.com<mailto:list...@listserv.aol.com>> with the

[Fenton, Brian]

Thanks Jim

yes that now logs the fact that the file was too big (multiple times in fact), but how can I access this fact in TCL?

[23/Jun/2011:15:05:50][27869.163851][-nssock:driver-] Error: conn[38]: max content exceeded
[23/Jun/2011:15:05:50][27869.163851][-nssock:driver-] Error: conn[39]: max content exceeded
[23/Jun/2011:15:05:50][27869.163851][-nssock:driver-] Error: conn[40]: max content exceeded
[23/Jun/2011:15:05:50][27869.163851][-nssock:driver-] Error: conn[41]: max content exceeded
[23/Jun/2011:15:05:50][27869.163851][-nssock:driver-] Error: conn[42]: max content exceeded
[23/Jun/2011:15:05:50][27869.163851][-nssock:driver-] Error: conn[43]: max content exceeded
[23/Jun/2011:15:05:50][27869.163851][-nssock:driver-] Error: conn[44]: max content exceeded

Brian

________________________________________
From: AOLserver Discussion [AOLS...@LISTSERV.AOL.COM] On Behalf Of Jim Davidson [jgdav...@MAC.COM]

Sent: 23 June 2011 14:44

[Jim Davidson]

I think the short answer is there is no way.

Checking the code and your error message, this is error condition E_CRANGE. It's returned from SockReadLine which is called repeatedly to read the request line ("GET /url/ ...") and headers. As it reads lines, it parses them for some special conditions. If it sees a "content-length" header, it's parsed and verified the content to be sent isn't out of range. If it is, it returns E_CRANGE and the connection is aborted. So, there's no Tcl environment available to fiddle with.

There is some Tcl framework in there for "que wait" callbacks but it's not exposed for error conditions. Good idea.

Question: What are you trying to do with Tcl? Maybe you could hack the driver.c code to do something smart at CRANGE alone? I think it's the most common limit-related error.

-Jim

[Fenton, Brian]

I'm using OpenACS hence the TCL. I just want to let the user know that their file is over the size limit. Could we, in driver.c instead of closing the connection socket, return a custom template (like Apache does) ?

Brian

________________________________________
From: AOLserver Discussion [AOLS...@LISTSERV.AOL.COM] On Behalf Of Jim Davidson [jgdav...@MAC.COM]

Sent: 23 June 2011 15:59

[Dossy Shiobara]

Yes, on E_CRANGE, the driver thread should return HTTP 413 Request too
large, IMHO. And, we should be able to configure a custom response for
status code 413 ...

Patches welcome ;)

On 6/23/11 11:28 AM, Fenton, Brian wrote:
> I'm using OpenACS hence the TCL. I just want to let the user know that their file is over the size limit. Could we, in driver.c instead of closing the connection socket, return a custom template (like Apache does) ?

--

Dossy Shiobara | "He realized the fastest way to change
do...@panoptic.com | is to laugh at your own folly -- then you
http://panoptic.com/ | can let go and quickly move on." (p. 70)
* WordPress * jQuery * MySQL * Security * Business Continuity *

[Jim Davidson]

Agreed. Should go on the bug/feature list. Is there a list ? :)

Sent from a phone

[Enrique Catalan]

Hi,

The final goal is to have a server-side validation and return a page
template explaining what the problem is rather than just closing the
connection. Having client-side validation ( JavaScript/Flash like
gmail ) helps in terms of UI but I presume wouldn't be safe enough
agains DoS.

IMHO, I agree with Dossy, to use the driver thread to check the hard
limits and instead of dropping the connection, just mark the HTTP
request and let the request handler to return the 413. I also think
the template could be configured in the 'ns_section
ns/server/${server}/redirects' , isn't it ?

If you all agree with this, Brian and I can help to get a patch.

Thanks for the great feedback!,
Enrique.

[Dossy Shiobara]

In theory, it could go on SourceForge's issue tracker ... or opened as a
issue on GitHub ... or ...

On 6/23/11 12:13 PM, Jim Davidson wrote:
> Agreed. Should go on the bug/feature list. Is there a list ? :)
>
> Sent from a phone
>
> On Jun 23, 2011, at 9:58 AM, Dossy Shiobara<do...@PANOPTIC.COM> wrote:
>
>> > Yes, on E_CRANGE, the driver thread should return HTTP 413 Request too large, IMHO. And, we should be able to configure a custom response for status code 413 ...
>> >
>> > Patches welcome ;)

--

[Dossy Shiobara]

That was what I was thinking -- driver marks the request as exceeding
the limit, and setting the response status code to 413. The benefits
that I see (if implemented the way I'm imagining) --

1) access logging of requests w/ 413 status code

2) custom response page via ns/server/${server}/redirects.

If you and Brian could work up a patch, that'd be wonderful!

Does anyone see any problems with this approach? Any reason not to do
it? I don't suppose it can possibly break any kind of backward
compatibility, as no code right now can even be written to handle such a
scenario, anyway.

Of course, once we decide on a behavior, and folks code against the
implementation ... changing/fixing it becomes more "expensive" for
everyone, so if there's any kind of worry about how this is going to
work, lets iron out the details now ;)

On 6/23/11 12:29 PM, Enrique Catalan wrote:
> IMHO, I agree with Dossy, to use the driver thread to check the hard
> limits and instead of dropping the connection, just mark the HTTP
> request and let the request handler to return the 413. I also think
> the template could be configured in the 'ns_section
> ns/server/${server}/redirects' , isn't it ?
>
> If you all agree with this, Brian and I can help to get a patch.

--

Dossy Shiobara | "He realized the fastest way to change
do...@panoptic.com | is to laugh at your own folly -- then you
http://panoptic.com/ | can let go and quickly move on." (p. 70)
* WordPress * jQuery * MySQL * Security * Business Continuity *

[Jim Davidson]

I think this one case can be handled this way because you could truncate the request (some content may have been read ahead) and then flag it for special handling in the normal connection processing code/threads. The reason is the server has a full request, you're just choosing to not read all the content, so if you do a little touch up of the conn structure it can flow through the connection threads without tripping up. I'm cautious here because I've spent a lot of time in the past debugging half-baked conn problems. So, this patch makes sense for the E_CRANGE error alone, the most annoying problem.

Later we could kick around the idea of getting the code in the driver to handle much more detailed logging, dumping the access log module, etc.

-Jim

[Gustaf Neumann]

Dear all,

As a small contribution, i added a patch to sourceforge cvs.
The patch returns the 413 error message via a new function
Ns_ConnReturnEntityToLarge(), which is written in the same
style as Ns_ConnReturnNotFound(). The patch is somewhat
minimal and handles this issue entirely in the driver (also,
forcing the log entry). I have paid no special attention to
the custom response page.

-gn

--

[Jim Davidson]

Cool. The driver thread is technically a single threaded thing which shouldn't block and the connection return functions are blocking but as the response is small, it probably just works. Also, the connection code hasn't been verified to run in the driver (filters, traces, etc.) but again it likely just works unless there's a really goofy, blocking, involved filter lurking in some installation.

A more involved, perhaps unnecessary solution would be to either move all the response writing back to the driver in non-blocking I/O mode for all requests and/or taking error condition connections and simply tightening them up and sending them to a connection thread to generate the response safely. Hmm.... The later is probably safer/easier and more backward compatible but the former concept always intrigued me.

Anyway, this patch sounds like a smart first step and can stop the bleeding on the most common error response you'd want to know about, exceed content size.

-Jim

[=?utf-8?Q?Maurizio_Martignano?=]

Dear all,
Most of the times Aolserver is used with a reverse proxy like
Pound or Nginx in front of it. I would just leave Aolserver as it is...
and make sure that the reverse proxy checks the size of the objects to be
uploaded and provides a proper error message.

For Nginx:

server {
listen 80;
client_max_body_size 12M; <-- here the maximum size is 12M

Then for the error message:
To redirect to a simple 413 object too big page for a specific location:
location /my_page {
error_page 413 = /my_page_object_too_big.html;}
A site wide 413 page:
server {
listen 80;
error_page 413 /website_object_too_big.html;
...

Nothing to implement... everything is already done for us....

Ciao,
Maurizio

[Fenton, Brian]

Thanks for this Gustaf,

I'm rolling up my sleeves, and have just put on my new C hat (never worn before). Let's see how I get on....

Brian

________________________________________
From: AOLserver Discussion [AOLS...@LISTSERV.AOL.COM] On Behalf Of Gustaf Neumann [neu...@WU-WIEN.AC.AT]
Sent: 24 June 2011 08:00

To: AOLS...@LISTSERV.AOL.COM
Subject: Re: [AOLSERVER] Problem with file uploads larger than maxinput

Dear all,

[Jeff Rogers]

A thought for an alternate way to handle this - if the content-length is
too large, the just don't read the content, but set an error flag on the
conn. That way, an adventurous system could return a custom error page,
or even handle reading the content through custom code. I think this
could also allow handling of uploaded files larger than 2G as someone
else was having.

The default handling by returning a 413 error could be implemented as a
pre-queue filter, which would still handle the error without needing a
conn thread.

If anyone's messing around in there, this might be a good time to also
think about how to handle http/1.1's 'Expect: 100-continue', which
should be most beneficial for large entity bodies anyway.

-J

Dossy Shiobara wrote:
> Yes, on E_CRANGE, the driver thread should return HTTP 413 Request too
> large, IMHO. And, we should be able to configure a custom response for
> status code 413 ...
>
> Patches welcome ;)
>
>
> On 6/23/11 11:28 AM, Fenton, Brian wrote:
>> I'm using OpenACS hence the TCL. I just want to let the user know that
>> their file is over the size limit. Could we, in driver.c instead of
>> closing the connection socket, return a custom template (like Apache
>> does) ?
>

--

[Jeff Rogers]

Jim Davidson wrote:
> Cool. The driver thread is technically a single threaded thing which
> shouldn't block and the connection return functions are blocking but
> as the response is small, it probably just works. Also, the
> connection code hasn't been verified to run in the driver (filters,
> traces, etc.) but again it likely just works unless there's a really
> goofy, blocking, involved filter lurking in some installation.

This has always worried me about using pre-queue filters - they run in
the driver thread which you really don't want to block, and a lot of
interesting things to do there a web service can easily block or need to
be carefully written to not block.

Naturally, I have ideas for a solution :)

One way would be to (optionally) have a separate thread pool for
pre-queue filters, similar to how reading can be done in separate
threads. I don't think this would be terribly difficult, but then I
haven't tried it yet.

My other idea is to let connections "restart" after finishing up in
their conn thread. That is, after processing is done, rather than
cleaning up the conn and closing the socket, kick it back to the
driver's prequeue state and run it again from the very beginning, the
only difference being that the conn processing may have setup quewaits
or cls data, or changed the url. This approach could let you set up
complex state machines with lots of processing in each state. I looked
into doing this a little, and it's complicated, because you have to
abort the cleanup that normally happens when the conn is done, which can
happen in several places. This complexity is probably unwarranted for
most things, but it would allow 'expect: 100-continue' handling after
authentication (if the conn was moved back to the readwait state).

> A more involved, perhaps unnecessary solution would be to either move
> all the response writing back to the driver in non-blocking I/O mode
> for all requests and/or taking error condition connections and simply
> tightening them up and sending them to a connection thread to
> generate the response safely. Hmm.... The later is probably
> safer/easier and more backward compatible but the former concept
> always intrigued me.

I really like the idea of making the response writing nonblocking. It
would make everything symmetrical, and could avoid the possibility of
tying up conn threads writing a large response (like a file download).
I know there are ways of handling that particular problem, but having it
not be a problem in the first place would be better.

-J

[Jeff Rogers]

Jim Davidson wrote:

> I think today, in 2011, some of the flexibility we imagined back in
> 1995 isn't really so needed. This includes general purpose logging
> plugins and network drivers when there really is just the common log
> format, ordinary and SSL sockets. More in the core for the base HTTP
> processing could make things easier to maintain.

I may be the only one, but I miss some of the flexibilty that has
already been removed; in particular the nsunix driver that let me run
multiple servers with different credentials.

There are also arguments for leaving flexibility in other things. I
suspect they're the same arguments you had in 1995, but I'll offer my
thoughts too. Some people do want more than common log format written
to a file (distributed logging, or writing to a database). There are
extensions to HTTP on the horizon (websockets and SPDY come to mind).
These things don't need to be directly supported, but a framework for
letting someone add in support without rewriting major parts of the
server is, IMHO, a good thing.

-J

[Fenton, Brian]

Hello

I've been struggling a little with this change. I took the HEAD version including Gustaf's changes and compiled it. When I uploaded a file larger than the max, I could see that it was calling Gustaf's new function Ns_ConnReturnEntityToLarge (correctly triggered by the E_CRANGE) - however, the user still got the closed connection, so even though the function was running, it wasn't sending any message back to the user. So then I decided to do what Dossy/Enrique suggested: when the E_CRANGE condition was met, mark the socket state as SOCK_OVERFLOW, and then in driver.c in the "Process sockets ready to run" while loop, if socket state is SOCK_OVERFLOW then call Ns_ConnReturnEntityToLarge. However, this still isn't sending anything back to the user and the connection is still getting closed.

To do a sanity check that the Ns_ConnReturnEntityToLarge function actually works, I created a TCL proc called ns_returntoolarge which mapped to Ns_ConnReturnEntityToLarge - and yes this works fine when called directly from a TCL program (correctly returning the "The request entity (e.g. file to be uploaded) is too large." message to the browser). So knowing that the C function works fine, my conclusion then was that I must be calling it from the wrong place (or passing in the wrong conn pointer or whatever).

So the question is - what's going on here and where is the right place to call Ns_ConnReturnEntityToLarge? I noticed that none of the similar Ns_ConnReturn* functions ever get called in driver.c, so I started to suspect that driver.c was the wrong place. The other Ns_ConnReturn* functions seem to mainly get called in op.c and queue.c, however neither of those seem to be involved in this particular situation. I don't see any other obvious place, so I'm a little bit stumped at the moment.

Any suggestions?

many thanks
Brian

________________________________________

From: AOLserver Discussion [AOLS...@LISTSERV.AOL.COM] On Behalf Of Gustaf Neumann [neu...@WU-WIEN.AC.AT]
Sent: 24 June 2011 08:00

To: AOLS...@LISTSERV.AOL.COM
Subject: Re: [AOLSERVER] Problem with file uploads larger than maxinput

Dear all,

[Gustaf Neumann]

Strange, it works for me. can you send me your config file?
Concerning "the right place": As it was discussed here, it
would be certainly better to move the reply-sending to a
request handling thread (or a spooling thread like in
naviserver), simply to be sure that the driver is never
blocking. This should not be complicated, but i simply did
not have to time, and i see no reason, why replying from the
driver should not work.

-gustaf

[Fenton, Brian]

Dear Gustaf

thanks for the reply. I have tried it with the latest OpenACS config.tcl and also another older one I use (both attached). I just assumed I was doing something wrong, didn't want to sound as though I was criticising your work.

kind regards
Brian

________________________________________
From: AOLserver Discussion [AOLS...@LISTSERV.AOL.COM] On Behalf Of Gustaf Neumann [neu...@WU-WIEN.AC.AT]

Sent: 05 July 2011 23:47

To: AOLS...@LISTSERV.AOL.COM
Subject: Re: [AOLSERVER] Problem with file uploads larger than maxinput

Strange, it works for me. can you send me your config file?

[Gustaf Neumann]

Dear Brian,

Thanks with the config files, they are fine. With some more
testing, i found some more insights: The version in CVS head
returns always the right result to a client (tested with "nc
-v localhost 8006 < POST.txt"), no matter of the size of the
uploaded content, but works only reliably in browsers
(firefox, chrome) for rather small requests (set your
maxupload to 5k, upload a 10 k file), and does reliable NOT
work with upload files > 20MB (which is the interesting
case). So, the problem looks like a timing/buffering problem
to me. RFC 2616 says e.g.

If the server chooses to close the connection immediately after sending the
response, it SHOULD send a Connection header including the
connection-token close.

The reply of aolserver contains the "connection: close", so
this seems standard compliant. The driver has to close the
connection immediately after sending the reply, so, the
moral of the story seems to be that we have to get the
response handling out of the driver and require a solution
via the request handling thread - what's the best solution
anyhow. So far, i don't understand, why the size of the
upload request changes the behavior.

maybe, i can look into this next week.

-gustaf neumann

[Fenton, Brian]

Dear Gustaf

many thanks for the reply. Yes, I can confirm your results - if I set a limit of 10k and upload a file of 15k, it works perfectly for me.

Would be great if you could look into this. If not, maybe you could give me a few pointers where to change the "request handling thread".

many thanks
Brian

________________________________________
From: AOLserver Discussion [AOLS...@LISTSERV.AOL.COM] On Behalf Of Gustaf Neumann [neu...@WU-WIEN.AC.AT]

Sent: 08 July 2011 12:20

[Gustaf Neumann]

Dear all,

A few minutes ago, i have committed a change to the
sourceforge aolserver repository that should should fix the
problem finally. The primary problem was not that the driver
replied directly to the client, but the fact that client
(e.g. firefox) has not yet sent the full request while the
server had already replied and closed the connection. Since
the handling of the header happens in aolserver before the
full request from the client is received, the server could
close the connection before the client has sent the full
request. So, although the client (e.g. firefox) has already
received a proper reply from aolserver, it complained about
the fact, that it could not sent the full request.

This explains, why the error handling worked reliably with
"small" upload files but not with large ones in current
browsers; it explains as well, that simple clients (like
"nc") worked always reliable.

Additionally, i have moved the reply handling to the
connection threads, and made some more cosmetical changes.

Brian, please check again.

-gustaf neumann

[Fenton, Brian]

Dear Gustaf (and all)

thanks Gustaf for the fixes - reading your code, it's interesting to see how AOLserver works internally. And it's all so elegant, meaning even I can follow it! I can confirm your fix has resolved the 413 problem. I attempted to take it a step further and add a 413 redirect to the config.tcl as follows:

ns_section ns/server/${server}/redirects
ns_param 413 "/global/file-too-large.html"

However, this caused AOLserver to crash when uploading large files. After some debugging, I figured out a solution to unmark the connection once the redirect has been called (patch to op.c attached). Obviously, please let me know if there is a better way.

I also added a new TCL proc 'ns_returntoolarge' which maps to the Ns_ConnReturnEntityTooLarge C function. I find these TCL procs handy as an alternative to the config.tcl redirects, as they can be overloaded if needed. Patches to tclcmds.c and tclresp.c attached.

I'd love to hear any feedback that people have on these patches. As it's my first time, I'd be very grateful if you could review them.

Best wishes
Brian

________________________________________
From: Gustaf Neumann [neu...@wu-wien.ac.at]
Sent: 09 July 2011 11:25
To: AOLserver Discussion
Cc: Fenton, Brian

[Gustaf Neumann]

Brian many thanks, i commited your change to the CVS on
sourceforge.
-gustaf neumann