Incomplete POST data passed by nginx / passenger to rails

[Steven Hartland]

We're having a problem where by the rails / rack hander is receiving
incomplete POST data for file uploads intermittently.

I've added debugging to the RewindableInput class which populates the
temporary file used to contain the POST data and its always missing
large amounts of data when this issues occurs. For example for a 24MB
file upload it contains only ~16K of data.

The setup we have is:-
machine 1: nginx loadbalancer -> machine 2: nginx + passenger

If I access the nginx + passenger instance directly the request is
processed normally, for the current test case I have, but I believe
this is more luck than judgement.

I've traced the connection data when accessed via the loadbalancer the
request to the nginx + passenger instance on machine 2 contains the
full amount of data so I currently believe there is either a problem
in the nginx passenger module or the helper app.

While going through the code looking for potential issues I came
across the following comment:
"HISTORIC NOTE:
We used to register passenger_content_handler as a default content
handler, instead of setting ngx_http_core_loc_conf_t->handler.
However, if ngx_http_read_client_request_body (and thus
passenger_content_handler) returns NGX_AGAIN, then Nginx will pass the
not-fully-receive file upload data to the upstream handler even though
it shouldn't. Is this an Nginx bug? In any case, setting
ngx_http_core_loc_conf_t->handler fixed the problem."

This seems to describe the issue we're seeing so I'm wondering if its
still not fixed fully?

So the question is this possible and if so any tips on how to confirm
this is the problem or on how to identify where the issue lies?

[Steven Hartland]

Some more information which may be key; the size of the data received
is that reported by nginx as the "http client request body preread" in
this case:-
2010/12/24 15:53:02 [debug] 17432#0: *21 http client request body
preread 15939.

[Steven Hartland]

I believe this is now an issue with the processing of sendfile calls
with regards to retry on EAGAIN.

Disabling sendfile support in the main nginx config appears to fix
this problem.

The question is is this specific to the passenger module or to all
nginx modules and hence a bug in the core nginx code?

Here's a debug log which seems to indicate this:-
2010/12/24 18:04:08 [debug] 27169#0: *21 connected
2010/12/24 18:04:08 [debug] 27169#0: *21 http upstream connect: 0
2010/12/24 18:04:08 [debug] 27169#0: *21 http upstream send request
2010/12/24 18:04:08 [debug] 27169#0: *21 chain writer buf fl:0 s:2405
2010/12/24 18:04:08 [debug] 27169#0: *21 chain writer buf fl:0 s:15939
2010/12/24 18:04:08 [debug] 27169#0: *21 chain writer buf fl:1 s:
23658449
2010/12/24 18:04:08 [debug] 27169#0: *21 chain writer in:
0000000801885F78
2010/12/24 18:04:08 [debug] 27169#0: *21 sendfile() sent only 18344
bytes (35: Resource temporarily unavailable)
2010/12/24 18:04:08 [debug] 27169#0: *21 sendfile: -1, 1, 0, @0
18344:23658449
2010/12/24 18:04:08 [debug] 27169#0: *21 chain writer out:
0000000801885F98
2010/12/24 18:04:08 [debug] 27169#0: *21 event timer add: 23:
600000:1293214448446
2010/12/24 18:04:08 [debug] 27169#0: *21 kevent set event: 23: ft:-2
fl:0025
2010/12/24 18:04:08 [debug] 27169#0: *21 http run request: "/
uploadfile"
2010/12/24 18:04:08 [debug] 27169#0: *21 http upstream check client,
write event:1, "/uploadfile"
2010/12/24 18:06:08 [debug] 27169#0: *21 http run request: "/
uploadfile"

As you can see sendfile returns after sending the first two buffer
with EAGAIN, but a retry is never triggered.