Strange connection drops from Trac via mod_wsgi

34 views
Skip to first unread message

Matt Stevens

unread,
Jul 14, 2008, 3:27:54 PM7/14/08
to modwsgi
I'm running into a pretty strange issue with mod_wsgi on Windows that
I thought I'd mention here just in case anyone has any ideas.

Running Trac via mod_wsgi on Apache 2.2, if a user attempts to
download a large attachment the connection appears to be closed about
halfway through the transfer. So if a 4mb file is requested, after
about 2mb has been transferred the connection is simply closed.
Content-Length and all other headers are appear to be valid and the
initial data transferred is correct.

At first this seemed like a Trac bug, but when running under
mod_python or CGI the issue disappears. The connection drop only
occurs when the transfer is going through mod_wsgi.

Using mod_wsgi on Windows has proven to be much more stable than
mod_python so I'd really love to get this one sorted out. Does anyone
have any ideas as to what might be causing this?

Graham Dumpleton

unread,
Jul 14, 2008, 9:16:28 PM7/14/08
to mod...@googlegroups.com
Exactly which version of mod_wsgi?

Can you change the WSGI script file from having:

application = trac.web.main.dispatch_request

to:

_application = trac.web.main.dispatch_request

def application(environ, start_response):
del environ['wsgi.file_wrapper']
return _application(environ, start_response)

I just want to eliminate wsgi.file_wrapper extension as being issue.

This presumes you are using mod_wsgi 2.0 or later.

Graham

2008/7/15 Matt Stevens <smas...@gmail.com>:

Graham Dumpleton

unread,
Jul 14, 2008, 9:18:12 PM7/14/08
to mod...@googlegroups.com
Cancel that request. Just realised that wsgi.file_wrapper not
available on Windows anyway. :-)

So just let me know which version of mod_wsgi is being used and I'll
think of other tests you can do.

In interim at least, enable 'debug' LogLevel for Apache. See:

http://code.google.com/p/modwsgi/wiki/DebuggingTechniques

Documentation says 'info', but I want 'debug'.

Look for anything strange in log file for the problematic request.

Graham

2008/7/15 Graham Dumpleton <graham.d...@gmail.com>:

Graham Dumpleton

unread,
Jul 15, 2008, 8:50:45 AM7/15/08
to mod...@googlegroups.com
Sorry, want to reinstate that request for wsgi.file_wrapper test. It
does exist, but it isn't optimised to use direct access to file
descriptor. If Trac is using wsgi.file_wrapper, entirely possible that
it could still be an issue in wsg.file_wrapper. The code paths in this
case would be some not normally exercised on UNIX but would be on
Windows.

Also, can you use example code in that debugging document I referenced
to capture request WSGI environ and response headers and post them so
can see what the request looks like and what response headers also
look like so can get a feel for what the application is doing.

Matt Stevens

unread,
Jul 15, 2008, 3:44:04 PM7/15/08
to modwsgi
You nailed it, the wsgi.file_wrapper extension is the problem. After
disabling it the transfers started working perfectly.

I'm using the 2.0 binary from http://adal.chiriliuc.com/mod_wsgi/revision_878_2.0/.
When turning on debug logging with wsgi.file_wrapper enabled I do see
an error generated:

[Tue Jul 15 13:32:58 2008] [info] [client 127.0.0.1] mod_wsgi
(pid=252, process='', application=''): Loading WSGI script 'C:/Trac/
endpoint.db/apache/endpoint.wsgi'., referer:
http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1] ('REQUEST',,
referer: http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1] {'COMSPEC': 'C:\
\\\WINDOWS\\\\system32\\\\cmd.exe',, referer:
http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
'DOCUMENT_ROOT': 'C:/Program Files/xampp/htdocs',, referer:
http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
'GATEWAY_INTERFACE': 'CGI/1.1',, referer:
http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1] 'HTTP_ACCEPT':
'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',,
referer: http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
'HTTP_ACCEPT_CHARSET': 'ISO-8859-1,utf-8;q=0.7,*;q=0.7',, referer:
http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
'HTTP_ACCEPT_ENCODING': 'gzip,deflate',, referer:
http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
'HTTP_ACCEPT_LANGUAGE': 'en-us,en;q=0.5',, referer:
http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
'HTTP_CONNECTION': 'keep-alive',, referer:
http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1] 'HTTP_COOKIE':
'trac_form_token=9fc197adbdff9308572e6f1f;
trac_session=113aeeb671a563bfd9a543ec',, referer:
http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1] 'HTTP_HOST':
'127.0.0.1',, referer: http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
'HTTP_KEEP_ALIVE': '300',, referer:
http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
'HTTP_REFERER': 'http://127.0.0.1/endpoint/attachment/wiki/DownTest/
nant-0.86-beta1-bin.zip',, referer:
http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
'HTTP_USER_AGENT': 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:
1.9) Gecko/2008052906 Firefox/3.0',, referer:
http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1] 'PATH': 'C:\\\
\Program Files\\\\Windows Resource Kits\\\\Tools\\\\;C:\\\\WINDOWS\\\
\system32;C:\\\\WINDOWS;C:\\\\WINDOWS\\\\System32\\\\Wbem;C:\\\
\Program Files\\\\Common Files\\\\Adaptec Shared\\\\System;C:\\\
\Program Files\\\\Microsoft Visual Studio .NET 2003\\\\SDK\\\\v1.1\\\
\Bin;C:\\\\ssh_client\\\\bin;C:\\\\j2sdk1.4.2_06\\\\bin;C:\\\\Windows\\
\\Microsoft.NET\\\\Framework\\\\v1.1.4322;C:\\\\nant\\\\nant-0.85-rc3\\
\\bin;C:\\\\ssh_client\\\\ssh\\\\bin;C:\\\\Program Files\\\\Subversion\
\\\bin;C:\\\\Python25;C:\\\\Program Files\\\\GnuWin32\\\\bin;C:\\\
\Program Files\\\\Rational\\\\ClearCase\\\\etc\\\\utils;C:\\\\Program
Files\\\\doxygen\\\\bin;%WIND_BASE%\\\\host\\\\%WIND_HOST_TYPE%\\\
\bin;C:\\\\PROGRA~1\\\\ATT\\\\Graphviz\\\\bin;C:\\\\Program Files\\\
\Rational\\\\ClearCase\\\\bin;C:\\\\Program Files\\\\Rational\\\
\common;C:\\\\Program Files\\\\Common Files\\\\Compuware;c:\\\\Program
Files\\\\Microsoft SQL Server\\\\90\\\\Tools\\\\binn\\\\;C:\\\\android\
\\\tools;C:\\\\Qt\\\\4.4.0\\\\bin;C:\\\\Program Files\\\\QuickTime\\\
\QTSystem\\\\',, referer: http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1] 'PATHEXT':
'.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.VBS',, referer:
http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1] 'PATH_INFO': '/
raw-attachment/wiki/DownTest/nant-0.86-beta1-bin.zip',, referer:
http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
'PATH_TRANSLATED': 'C:\\\\Program Files\\\\xampp\\\\htdocs\\\\raw-
attachment\\\\wiki\\\\DownTest\\\\nant-0.86-beta1-bin.zip',, referer:
http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
'QUERY_STRING': '',, referer: http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1] 'REMOTE_ADDR':
'127.0.0.1',, referer: http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1] 'REMOTE_PORT':
'2561',, referer: http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
'REQUEST_METHOD': 'GET',, referer:
http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1] 'REQUEST_URI':
'/endpoint/raw-attachment/wiki/DownTest/nant-0.86-beta1-bin.zip',,
referer: http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
'SCRIPT_FILENAME': 'C:/Trac/endpoint.db/apache/endpoint.wsgi',,
referer: http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1] 'SCRIPT_NAME':
'/endpoint',, referer: http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1] 'SERVER_ADDR':
'127.0.0.1',, referer: http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
'SERVER_ADMIN': 'admin@localhost',, referer:
http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1] 'SERVER_NAME':
'127.0.0.1',, referer: http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1] 'SERVER_PORT':
'80',, referer: http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
'SERVER_PROTOCOL': 'HTTP/1.1',, referer:
http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
'SERVER_SIGNATURE': '<address>Apache/2.2.8 (Win32) DAV/2 mod_ssl/2.2.8
OpenSSL/0.9.8g mod_autoindex_color mod_auth_sspi/1.0.4 PHP/5.2.5
mod_wsgi/2.0 Python/2.5.2 Server at 127.0.0.1 Port 80</address>\\n',,
referer: http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
'SERVER_SOFTWARE': 'Apache/2.2.8 (Win32) DAV/2 mod_ssl/2.2.8 OpenSSL/
0.9.8g mod_autoindex_color mod_auth_sspi/1.0.4 PHP/5.2.5 mod_wsgi/2.0
Python/2.5.2',, referer: http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1] 'SystemRoot':
'C:\\\\WINDOWS',, referer: http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1] 'WINDIR': 'C:\\
\\WINDOWS',, referer: http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
'mod_wsgi.application_group': '',, referer:
http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
'mod_wsgi.callable_object': 'application',, referer:
http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
'mod_wsgi.process_group': '',, referer:
http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
'mod_wsgi.reload_mechanism': '0',, referer:
http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
'mod_wsgi.script_reloading': '1',, referer:
http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1] 'wsgi.errors':
<mod_wsgi.Log object at 0x022AAEA8>,, referer:
http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
'wsgi.file_wrapper': <built-in method file_wrapper of mod_wsgi.Adapter
object at 0x0232C578>,, referer: http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1] 'wsgi.input':
<mod_wsgi.Input object at 0x06B83A98>,, referer:
http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
'wsgi.multiprocess': False,, referer:
http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
'wsgi.multithread': True,, referer:
http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
'wsgi.run_once': False,, referer: http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
'wsgi.url_scheme': 'http',, referer:
http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
'wsgi.version': (1, 0)}), referer:
http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1] ('RESPONSE',,
referer: http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1] '200 Ok',,
referer: http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1] [('ETag',
'W/"anonymous/Tue, 08 Jul 2008 21:39:14 GMT/False"'),, referer:
http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1] ('Content-
Disposition', 'attachment'),, referer:
http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1] ('Content-
Type', 'application/zip; charset=iso-8859-15'),, referer:
http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1] ('Content-
Length', '4963893'),, referer: http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1] ('Last-
Modified', 'Tue, 08 Jul 2008 21:39:15 GMT')]), referer:
http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip
[Tue Jul 15 13:32:58 2008] [debug] mod_wsgi.c(3139): [client
127.0.0.1] mod_wsgi (pid=252): Content length mismatch, expected
4963893, response generated 1613824: C:/Trac/endpoint.db/apache/
endpoint.wsgi, referer: http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bin.zip

With wsgi.file_wrapper off the error is not generated and the transfer
goes through properly. Let me know if there's any other information I
can provide, and thanks for the help!

Matt

On Jul 15, 6:50 am, "Graham Dumpleton" <graham.dumple...@gmail.com>
wrote:
> Sorry, want to reinstate that request for wsgi.file_wrapper test. It
> does exist, but it isn't optimised to use direct access to file
> descriptor. If Trac is using wsgi.file_wrapper, entirely possible that
> it could still be an issue in wsg.file_wrapper. The code paths in this
> case would be some not normally exercised on UNIX but would be on
> Windows.
>
> Also, can you use example code in that debugging document I referenced
> to capture request WSGI environ and response headers and post them so
> can see what the request looks like and what response headers also
> look like so can get a feel for what the application is doing.
>
> Graham
>
> 2008/7/15 Graham Dumpleton <graham.dumple...@gmail.com>:
>
> > Cancel that request. Just realised that wsgi.file_wrapper not
> > available on Windows anyway. :-)
>
> > So just let me know which version of mod_wsgi is being used and I'll
> > think of other tests you can do.
>
> > In interim at least, enable 'debug' LogLevel for Apache. See:
>
> >  http://code.google.com/p/modwsgi/wiki/DebuggingTechniques
>
> > Documentation says 'info', but I want 'debug'.
>
> > Look for anything strange in log file for the problematic request.
>
> > Graham
>
> > 2008/7/15 Graham Dumpleton <graham.dumple...@gmail.com>:
> >> Exactly which version of mod_wsgi?
>
> >> Can you change the WSGI script file from having:
>
> >>  application = trac.web.main.dispatch_request
>
> >> to:
>
> >>  _application = trac.web.main.dispatch_request
>
> >>  def application(environ, start_response):
> >>    del environ['wsgi.file_wrapper']
> >>    return _application(environ, start_response)
>
> >> I just want to eliminate wsgi.file_wrapper extension as being issue.
>
> >> This presumes you are using mod_wsgi 2.0 or later.
>
> >> Graham
>
> >> 2008/7/15 Matt Stevens <smashc...@gmail.com>:

Graham Dumpleton

unread,
Jul 15, 2008, 9:54:05 PM7/15/08
to mod...@googlegroups.com
Thanks, I'll do some checking of that code, although unfortunately
don't have Windows.

The problem could be one of two things.

The first is that mod_wsgi is simply broken somehow.

The second is that Trac is not opening files in 'raw' module. The
consequences of this are that if the file contains CRLF pairs, they
would get returned as just LF. For each of these this would have the
effect of decreasing the actual returned content length. Since the
expected content length and actual returned content length are so
different, that however would have to be a lot of CRLF pairs in the
original file, which seems unlikely unless zip algorithm generates a
lot for a reason.

If you really wanted to try something, you could in a Python script do:

fd = open("nant-0.86-beta1-bin.zip", 'r')
data = fd.read()
print len(data)

fd = open("nant-0.86-beta1-bin.zip", 'rb')
data = fd.read()
print len(data)

and tell me what it outputs.

Graham

2008/7/16 Matt Stevens <smas...@gmail.com>:

Matt Stevens

unread,
Jul 16, 2008, 2:34:05 PM7/16/08
to modwsgi
Not using binary mode to open the binary file definitely produced some
bad results, but these numbers don't seem to line up either:

r: 303
rb: 4963893

Taking a quick look at the Trac code, it does specify a _FileWrapper
object, not sure if this is useful:
http://trac.edgewall.org/browser/trunk/trac/web/wsgi.py

Matt

On Jul 15, 7:54 pm, "Graham Dumpleton" <graham.dumple...@gmail.com>
wrote:
> Thanks, I'll do some checking of that code, although unfortunately
> don't have Windows.
>
> The problem could be one of two things.
>
> The first is that mod_wsgi is simply broken somehow.
>
> The second is that Trac is not opening files in 'raw' module. The
> consequences of this are that if the file contains CRLF pairs, they
> would get returned as just LF. For each of these this would have the
> effect of decreasing the actual returned content length. Since the
> expected content length and actual returned content length are so
> different, that however would have to be a lot of CRLF pairs in the
> original file, which seems unlikely unless zip algorithm generates a
> lot for a reason.
>
> If you really wanted to try something, you could in a Python script do:
>
>   fd = open("nant-0.86-beta1-bin.zip", 'r')
>   data = fd.read()
>   print len(data)
>
>   fd = open("nant-0.86-beta1-bin.zip", 'rb')
>   data = fd.read()
>   print len(data)
>
> and tell me what it outputs.
>
> Graham
>
> 2008/7/16 Matt Stevens <smashc...@gmail.com>:
>
>
>
> > You nailed it, the wsgi.file_wrapper extension is the problem. After
> > disabling it the transfers started working perfectly.
>
> > I'm using the 2.0 binary fromhttp://adal.chiriliuc.com/mod_wsgi/revision_878_2.0/.
> > When turning on debug logging with wsgi.file_wrapper enabled I do see
> > an error generated:
>
> > [Tue Jul 15 13:32:58 2008] [info] [client 127.0.0.1] mod_wsgi
> > (pid=252, process='', application=''): Loading WSGI script 'C:/Trac/
> > endpoint.db/apache/endpoint.wsgi'., referer:
> >http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bi...
> > [Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1] ('REQUEST',,
> > referer:http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bi...
> > [Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]  {'COMSPEC': 'C:\
> > \\\WINDOWS\\\\system32\\\\cmd.exe',, referer:
> >http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bi...
> > [Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
> > 'DOCUMENT_ROOT': 'C:/Program Files/xampp/htdocs',, referer:
> >http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bi...
> > [Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
> > 'GATEWAY_INTERFACE': 'CGI/1.1',, referer:
> >http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bi...
> > [Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]   'HTTP_ACCEPT':
> > 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',,
> > referer:http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bi...
> > [Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
> > 'HTTP_ACCEPT_CHARSET': 'ISO-8859-1,utf-8;q=0.7,*;q=0.7',, referer:
> >http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bi...
> > [Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
> > 'HTTP_ACCEPT_ENCODING': 'gzip,deflate',, referer:
> >http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bi...
> > [Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
> > 'HTTP_ACCEPT_LANGUAGE': 'en-us,en;q=0.5',, referer:
> >http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bi...
> > [Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
> > 'HTTP_CONNECTION': 'keep-alive',, referer:
> >http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bi...
> > [Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]   'HTTP_COOKIE':
> > 'trac_form_token=9fc197adbdff9308572e6f1f;
> > trac_session=113aeeb671a563bfd9a543ec',, referer:
> >http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bi...
> > [Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]   'HTTP_HOST':
> > '127.0.0.1',, referer:http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bi...
> > [Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
> > 'HTTP_KEEP_ALIVE': '300',, referer:
> >http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bi...
> > [Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
> > 'HTTP_REFERER': 'http://127.0.0.1/endpoint/attachment/wiki/DownTest/
> > nant-0.86-beta1-bin.zip',, referer:
> >http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bi...
> > [Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
> > 'HTTP_USER_AGENT': 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:
> > 1.9) Gecko/2008052906 Firefox/3.0',, referer:
> >http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bi...
> > [Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]   'PATH': 'C:\\\
> > \Program Files\\\\Windows Resource Kits\\\\Tools\\\\;C:\\\\WINDOWS\\\
> > \system32;C:\\\\WINDOWS;C:\\\\WINDOWS\\\\System32\\\\Wbem;C:\\\
> > \Program Files\\\\Common Files\\\\Adaptec Shared\\\\System;C:\\\
> > \Program Files\\\\Microsoft Visual Studio .NET 2003\\\\SDK\\\\v1.1\\\
> > \Bin;C:\\\\ssh_client\\\\bin;C:\\\\j2sdk1.4.2_06\\\\bin;C:\\\\Windows\\
> > \\Microsoft.NET\\\\Framework\\\\v1.1.4322;C:\\\\nant\\\\nant-0.85-rc3\\
> > \\bin;C:\\\\ssh_client\\\\ssh\\\\bin;C:\\\\Program Files\\\\Subversion\
> > \\\bin;C:\\\\Python25;C:\\\\Program Files\\\\GnuWin32\\\\bin;C:\\\
> > \Program Files\\\\Rational\\\\ClearCase\\\\etc\\\\utils;C:\\\\Program
> > Files\\\\doxygen\\\\bin;%WIND_BASE%\\\\host\\\\%WIND_HOST_TYPE%\\\
> > \bin;C:\\\\PROGRA~1\\\\ATT\\\\Graphviz\\\\bin;C:\\\\Program Files\\\
> > \Rational\\\\ClearCase\\\\bin;C:\\\\Program Files\\\\Rational\\\
> > \common;C:\\\\Program Files\\\\Common Files\\\\Compuware;c:\\\\Program
> > Files\\\\Microsoft SQL Server\\\\90\\\\Tools\\\\binn\\\\;C:\\\\android\
> > \\\tools;C:\\\\Qt\\\\4.4.0\\\\bin;C:\\\\Program Files\\\\QuickTime\\\
> > \QTSystem\\\\',, referer:http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bi...
> > [Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]   'PATHEXT':
> > '.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.VBS',, referer:
> >http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bi...
> > [Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]   'PATH_INFO': '/
> > raw-attachment/wiki/DownTest/nant-0.86-beta1-bin.zip',, referer:
> >http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bi...
> > [Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
> > 'PATH_TRANSLATED': 'C:\\\\Program Files\\\\xampp\\\\htdocs\\\\raw-
> > attachment\\\\wiki\\\\DownTest\\\\nant-0.86-beta1-bin.zip',, referer:
> >http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bi...
> > [Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
> > 'QUERY_STRING': '',, referer:http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bi...
> > [Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]   'REMOTE_ADDR':
> > '127.0.0.1',, referer:http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bi...
> > [Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]   'REMOTE_PORT':
> > '2561',, referer:http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bi...
> > [Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
> > 'REQUEST_METHOD': 'GET',, referer:
> >http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bi...
> > [Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]   'REQUEST_URI':
> > '/endpoint/raw-attachment/wiki/DownTest/nant-0.86-beta1-bin.zip',,
> > referer:http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bi...
> > [Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
> > 'SCRIPT_FILENAME': 'C:/Trac/endpoint.db/apache/endpoint.wsgi',,
> > referer:http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bi...
> > [Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]   'SCRIPT_NAME':
> > '/endpoint',, referer:http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bi...
> > [Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]   'SERVER_ADDR':
> > '127.0.0.1',, referer:http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bi...
> > [Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
> > 'SERVER_ADMIN': 'admin@localhost',, referer:
> >http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bi...
> > [Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]   'SERVER_NAME':
> > '127.0.0.1',, referer:http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bi...
> > [Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]   'SERVER_PORT':
> > '80',, referer:http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bi...
> > [Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
> > 'SERVER_PROTOCOL': 'HTTP/1.1',, referer:
> >http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bi...
> > [Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
> > 'SERVER_SIGNATURE': '<address>Apache/2.2.8 (Win32) DAV/2 mod_ssl/2.2.8
> > OpenSSL/0.9.8g mod_autoindex_color mod_auth_sspi/1.0.4 PHP/5.2.5
> > mod_wsgi/2.0 Python/2.5.2 Server at 127.0.0.1 Port 80</address>\\n',,
> > referer:http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bi...
> > [Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
> > 'SERVER_SOFTWARE': 'Apache/2.2.8 (Win32) DAV/2 mod_ssl/2.2.8 OpenSSL/
> > 0.9.8g mod_autoindex_color mod_auth_sspi/1.0.4 PHP/5.2.5 mod_wsgi/2.0
> > Python/2.5.2',, referer:http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bi...
> > [Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]   'SystemRoot':
> > 'C:\\\\WINDOWS',, referer:http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bi...
> > [Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]   'WINDIR': 'C:\\
> > \\WINDOWS',, referer:http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bi...
> > [Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
> > 'mod_wsgi.application_group': '',, referer:
> >http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bi...
> > [Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
> > 'mod_wsgi.callable_object': 'application',, referer:
> >http://127.0.0.1/endpoint/attachment/wiki/DownTest/nant-0.86-beta1-bi...
> > [Tue Jul 15 13:32:58 2008] [error] [client 127.0.0.1]
> > 'mod_wsgi.process_group': '',, referer:
>
> ...
>
> read more »

Graham Dumpleton

unread,
Jul 18, 2008, 11:12:45 PM7/18/08
to mod...@googlegroups.com
2008/7/17 Matt Stevens <smas...@gmail.com>:

>
> Not using binary mode to open the binary file definitely produced some
> bad results, but these numbers don't seem to line up either:
>
> r: 303
> rb: 4963893
>
> Taking a quick look at the Trac code, it does specify a _FileWrapper
> object, not sure if this is useful:
> http://trac.edgewall.org/browser/trunk/trac/web/wsgi.py

Have been slowly looking at this, so don't think I have forgotten.

What is weird is that it appears that Trac doesn't appear to use
_FileWrapper when there is no wsgi.file_wrapper when working through a
true WSGI server. It does however use _FileWrapper when running with
its own internal web server.

I need to pose this as a question on Trac developers list to confirm
my reading of source code as don't have version of Trac to run where
am at the moment to try myself.

What you might do for me is use the debugging middleware in section
'Poorly Performing Code' of:

http://code.google.com/p/modwsgi/wiki/DebuggingTechniques

to wrap Trac application. Modify the middleware to remove
'wsgi.file_wrapper' from environ like you did before. Then do your
requests and monitor Apache error log file. Tell me if the middleware
complains about Trac returning an actual file object for the
attachment response. If it is, it can perform badly as described in
that section.

The code in Trac which looks suspect is in web/api.py.

if self.method != 'HEAD':
self._response = file(path, 'rb')
file_wrapper = self.environ.get('wsgi.file_wrapper')
if file_wrapper:
self._response = file_wrapper(self._response, 4096)
raise RequestDone

What this is saying, is if wsgi.file_wrapper exists then use it,
otherwise just return file object. I believe that code should use
_FileWrapper when it doesn't exist.

Other than finding that, my tests on MacOS X haven't found a problem
with returning StringIO instance with wsgi.file_wrapper, which would
trigger same code as on Windows. So, when get a chance, may get you to
try some other things for me. Will need to work out what to suggest
first though. :-)

Graham

Graham Dumpleton

unread,
Jul 19, 2008, 10:17:27 PM7/19/08
to mod...@googlegroups.com
2008/7/19 Graham Dumpleton <graham.d...@gmail.com>:

> 2008/7/17 Matt Stevens <smas...@gmail.com>:
>>
>> Not using binary mode to open the binary file definitely produced some
>> bad results, but these numbers don't seem to line up either:
>>
>> r: 303
>> rb: 4963893

BTW, forgot to mention that the reason that non binary is very short,
is that text mode in Windows from memory will still stop reading when
it encounters a ctrl-z in file. This harks back to old DOS days. At
least I think this still happens.

More below .....

As well as that test, can you try test application:

def application(environ, start_response):
status = '200 OK'

response_headers = [('Content-Type', 'text/plain'),]
start_response(status, response_headers)

file = open('/some/absolute/path/nant-0.86-beta1-bin.zip', 'rb')

length = 0
wrapper = environ['wsgi.file_wrapper'](file, 4096)

for block in wrapper:
length += len(block)

return 'LENGTH %d' % length

Fix up path as appropriate. Is the length displayed as expected?

Graham

Graham Dumpleton

unread,
Jul 20, 2008, 10:30:50 PM7/20/08
to mod...@googlegroups.com
I have posted a query onto Trac Development list about sub optimal
file (attachment) support:

http://groups.google.com/group/trac-dev/browse_frm/thread/a813b85ed3073b93?hl=en

No response at this time.

Graham

2008/7/20 Graham Dumpleton <graham.d...@gmail.com>:

Graham Dumpleton

unread,
Jul 23, 2008, 10:33:25 AM7/23/08
to mod...@googlegroups.com
Trac has been patched in Subversion now for the separate problem I
identified there.

Haven't had a chance to look further at original issue as reported and
haven't see a response to my request to run the test and provided
result.

Graham

2008/7/21 Graham Dumpleton <graham.d...@gmail.com>:

Graham Dumpleton

unread,
Jul 30, 2008, 9:20:13 PM7/30/08
to mod...@googlegroups.com
Since I haven't got any feedback about test I requested in relation to
this problem, am unable to pursue it further. Hopefully if someone
else sees issue they will help debug the problem, otherwise can do
nothing at this point as all the tests and checks I have done on a
UNIX system check out okay. I don't see why Windows would work any
differently provided that raw mode is used when opening original file.

Graham

2008/7/20 Graham Dumpleton <graham.d...@gmail.com>:

illuminarti

unread,
Aug 21, 2008, 6:10:48 PM8/21/08
to modwsgi
FWIW, I am seeing this same problem on OS X. So it seems that the
Windows part of the equation may not be relevant.


I have some binary files not downloading completely out of trac: I
have a couple of .jpg's and an .xls file that exhibit problems. Other
jpegs (bigger, and smaller) work fine. So it seems to be something
about the content of the file.

I am using mod_wsgi 2.1, with Apache 1.3, and Trac 0.11 on OS X Tiger
10.4.11.



Adding the change to the .wsgi file to eliminate the file wrapper
fixed the problem. These are not very large files, and they seem to
break in predictable ways - the download always stops at the same
point - so it would seem to be something about the files themselves -
but hexdumping the files, I don't see any suspect patterns in the data
at or around the point where the download stops. (Although the first
byte that isn't sent is 0x00 - but there are plenty of nulls in the
file before that point). And as far as it gets, the broken file is
identical to the original file. Interestingly perhaps (or perhaps not)
the successfully downloaded part of the file is exactly 44kB. So
perhaps it's something about the next chunk that stops it being
sent?...

Which of the steps above should I pick this up at to try and help
debug?









On Jul 30, 8:20 pm, "Graham Dumpleton" <graham.dumple...@gmail.com>
wrote:

illuminarti

unread,
Aug 21, 2008, 6:45:00 PM8/21/08
to modwsgi
Some more information - in every one of my three problematic files,
the download cuts out on a 4kB boundary (44k in one case, 20k in the
second, and 4k in the third). And in every case, the next character
that should be transmitted is 0x00.

Seems more than a coincidence... :-)


illuminarti

unread,
Aug 21, 2008, 8:26:55 PM8/21/08
to modwsgi
OK, I can reproduce this from scratch. If I have a pure text file, 8k
long (it just says 'The quick brown fox jumped over the lazy dog!'
over and over again.)
And roughly in the middle I put a single 0x00 character.

If the null character is not exactly on the 4k boundary, I can
download it from Trac with no problems.

If I take out a few characters to align the null as the first byte of
the second 4k of the file, the attempt to download the file stalls and
eventually times out after exactly 4k has been downloaded.

In both cases it uploads just fine as an attachment to a ticket, and
can be displayed inline (since it's a text file, Trac is happy to
display it - the problem happens when I click on the 'Download in
original format' link).

Wow. That's a doozy, isn't it? I have no idea why it happens, but at
least we know what triggers it. But since it's a real pain to try and
make sure the screenshots I'm uploading for the training materials I'm
writing don't have a zero byte every 4k characters, I really hope we
can find a solution :-)

Graham Dumpleton

unread,
Aug 21, 2008, 8:33:01 PM8/21/08
to mod...@googlegroups.com
Thanks for the detailed information.

I was offline yesterday, so just catching up on emails etc. With this
I should be able to duplicate it with just wsgi.file_wrapper and not
even require Trac. I'll get on to this one today.

Thanks again.

Graham

2008/8/22 illuminarti <si...@illuminarti.com>:

Simon J. Oliver

unread,
Aug 21, 2008, 8:43:43 PM8/21/08
to mod...@googlegroups.com
Thanks Graham!

To save you the bother, I'm attaching said text file. This is the one
with it aligned, so it breaks the download.

Cheers,

Simon

--

Simon J. Oliver
CISSP-ISSAP, ISSMP
Applied Information Technology Center
University of Memphis

testtext2.txt

Graham Dumpleton

unread,
Aug 21, 2008, 9:02:23 PM8/21/08
to mod...@googlegroups.com
2008/8/22 Simon J. Oliver <si...@illuminarti.com>:

> Thanks Graham!
>
> To save you the bother, I'm attaching said text file. This is the one
> with it aligned, so it breaks the download.

Very odd. I have no problem with your file, but can trigger problems
in other ways. Just the following is sufficient:

def application(environ, start_response):
status = '200 OK'

response_headers = [('Content-Type', 'text/plain'),]
start_response(status, response_headers)

path = os.path.join('/tmp/null.txt')
fd = open(path, 'w')
#fd.write(8192*'\0')
fd.write('\0')
fd.seek(0)
return environ['wsgi.file_wrapper'](fd)

Doing HEAD using telnet get:

$ telnet localhost 8224Trying ::1...
Connected to localhost.
Escape character is '^]'.
HEAD /wsgi/scripts/file.py HTTP/1.0

HTTP/1.1 200 OK
Date: Fri, 22 Aug 2008 00:57:38 GMT
Server: Apache/2.2.4 (Unix) mod_wsgi/3.0-TRUNK Python/2.3.5
Content-Length: 1
Connection: close
Content-Type: text/plain

Connection closed by foreign host.

If use curl it just hangs waiting for data.

Don't even need 4k of data. The first character of any 4k block,
including the first, being a null is possibly enough.

I'll have to check the code path which is used, but the whole point of
wsgi.file_wrapper is that it passes control for sending file off to
Apache functions to do. So first impression would be that I can't see
how this can be an issue in mod_wsgi and that it would have to be in
Apache.

Anyway, time for some digging.

Graham

> Cheers,
>
> Simon
>
> --
>
> Simon J. Oliver
> CISSP-ISSAP, ISSMP
> Applied Information Technology Center
> University of Memphis
>
>
> >
>

> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!
>
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234
> The quick brown fox jumped over the lazy dog!12345The quick brown fox jumped over the lazy dog!1234

>> --~--~---------~--~----~------------~-------~--~----~
>> This message is part of the topic "Strange connection drops from Trac
>> via mod_wsgi" in the Google Group "modwsgi" for which you requested
>> email
>> updates.
>> To stop receiving email updates for this topic, please visit the topic
>> at http://groups.google.com/group/modwsgi/t/90d169459a651120
>> -~----------~----~----~----~------~----~------~--~---
>>
>
>
>

Graham Dumpleton

unread,
Aug 21, 2008, 9:19:14 PM8/21/08
to mod...@googlegroups.com
Ignore that, my test code below is somehow flawed.

Even if I write a file containing 8192 null characters, can't reproduce it.

This is with Python 2.3, Apache 2.2.4 and MacOS X 10.4.11.

And then something clicks. You are using Apache 1.3. For both Apache
1.3 and Windows Apache 2.X, use of sendfile techniques isn't used and
so any problem would lie in mod_wsgi somewhere. I'll need to disable
the sendfile code manually and then see what happens.

Graham

2008/8/22 Graham Dumpleton <graham.d...@gmail.com>:

Graham Dumpleton

unread,
Aug 21, 2008, 9:50:12 PM8/21/08
to mod...@googlegroups.com
Duplicated using mod_wsgi 2.1 tar ball source code, Apache 1.3.41 and
Python 2.3 as shipped with Tiger. Doesn't occur with Apache 2.2
though.

With your file and test harness of:

import os

def application(environ, start_response):
status = '200 OK'

response_headers = [('Content-Type', 'text/plain'),]
start_response(status, response_headers)

path = os.path.join(os.path.dirname(__file__), 'null4k.txt')
return environ['wsgi.file_wrapper'](open(path, 'rb'), 4096)

Whatever is causing it, it is already fixed in mod_wsgi 3.0
development version as don't have the problem there. :-)

Just need to work out what changed.

Simon J. Oliver

unread,
Aug 21, 2008, 9:59:02 PM8/21/08
to mod...@googlegroups.com
Glad you're making some progress with it :-)

As it happens. I'm using the macports python 2.5 on that that machine,
but that probably doesn't make any difference, given what you're
seeing. I am still using the 1.3.41 apache that came with Tiger, though.


Simon

Graham Dumpleton

unread,
Aug 21, 2008, 10:12:15 PM8/21/08
to mod...@googlegroups.com
Following code in mod_wsgi is wrong:

if (*PyString_AsString(result) == '\0') {
PyErr_SetObject(PyExc_StopIteration, Py_None);
Py_DECREF(args);
Py_DECREF(result);
return 0;
}

Meant to be checking length of string, not whether first character is a null.

Thus:

if (PyString_Size(result) == 0) {
PyErr_SetObject(PyExc_StopIteration, Py_None);
Py_DECREF(args);
Py_DECREF(result);
return 0;
}

It was done correctly in mod_wsgi 3.0 although code in that area had
been changed around to handle Python 3.0 unicode strings. Not sure why
I didn't notice that it was wrong and fix it in mod_wsgi 2.X.

Anyway, the problem would affect mod_wsgi 2.0 and mod_wsgi 2.1 on
Windows or Apache 1.3 on UNIX.

Working version of mod_wsgi 2.2 which includes this and other fixes
can be checked out from subversion at:

https://modwsgi.googlecode.com/svn/branches/mod_wsgi-2.X

More details in:

http://code.google.com/p/modwsgi/wiki/ChangesInVersion0202

Haven't yet updated this with details of this fix yet though.

Graham

Graham Dumpleton

unread,
Aug 21, 2008, 11:12:37 PM8/21/08
to mod...@googlegroups.com
On Apache 2.X on UNIX, you would also have had problems if file like
object supplied to wsgi.file_wrapper wasn't actually an open file with
associated file descriptor, but some other file like object. For
example:

import StringIO
data = StringIO.StringIO(8192*'\0')
return environ['wsgi.file_wrapper'](data, 4096)

Simon J. Oliver

unread,
Aug 21, 2008, 11:50:39 PM8/21/08
to mod...@googlegroups.com
Graham -

Thank you so much. I downloaded 2.2 and it does indeed seem to have
fixed the problem.

I appreciate the prompt attention to this issue.

Simon

Graham Dumpleton

unread,
Aug 22, 2008, 12:06:00 AM8/22/08
to mod...@googlegroups.com
2008/8/22 Simon J. Oliver <si...@illuminarti.com>:
>
> Graham -
>
> Thank you so much. I downloaded 2.2 and it does indeed seem to have
> fixed the problem.
>
> I appreciate the prompt attention to this issue.

It always helps when people can give a good pointer, like you did, as
to what the problem is and are also happy to do a bit of debugging if
I have further questions. There have been separate cases in the past
where people have indicated they were having a problem but did little
to debug it themselves nor were they supplying the exact information I
needed when I asked various questions. In one case person came out and
said I was being too difficult when I kept asking questions.

So, it gets frustrating sometimes, but generally more than happy to
help sort problems out, especially when it is a problem in mod_wsgi
code itself. :-)

BTW, since I haven't heard any complaints about other fixes pending
there in 2.2 and haven't had time to investigate further some of the
graceful shutdown issues, I'll try and roll out a version 2.2 now.

I know there are some other potentially worrying issues outstanding at
the moment, but don't have simple answers for those now, plus haven't
had a huge amount of time lately to work on it.

Graham

Matt Stevens

unread,
Sep 3, 2008, 5:23:43 PM9/3/08
to modwsgi
I know I'm very late coming back to this thread, but just wanted to
report that your fix did indeed resolve the original issue on Windows
as well - using mod_wsgi 2.3 file transfers are now working perfectly.
Nice work!

Matt

On Aug 21, 10:06 pm, "Graham Dumpleton" <graham.dumple...@gmail.com>
wrote:
> >> 2008/8/22 Graham Dumpleton <graham.dumple...@gmail.com>:
> >>>>> 2008/8/22 Graham Dumpleton <graham.dumple...@gmail.com>:
> >>>>>> Ignore that, my test code below is somehow flawed.
>
> >>>>>> Even if I write a file containing 8192 null characters, can't
> >>>>>> reproduce it.
>
> >>>>>> This is with Python 2.3, Apache 2.2.4 and MacOS X 10.4.11.
>
> >>>>>> And then something clicks. You are using Apache 1.3. For both
> >>>>>> Apache
> >>>>>> 1.3 and Windows Apache 2.X, use of sendfile techniques isn't
> >>>>>> used and
> >>>>>> so any problem would lie in mod_wsgi somewhere. I'll need to
> >>>>>> disable
> >>>>>> the sendfile code manually and then see what happens.
>
> >>>>>> Graham
>
> >>>>>> 2008/8/22 Graham Dumpleton <graham.dumple...@gmail.com>:
Reply all
Reply to author
Forward
0 new messages