HTTPS and file uploads (a bug? Or is it just me?)
280 views
Skip to first unread message
alanp
Oct 26, 2010, 7:16:43 AM10/26/10
to cherrypy-users
I now have a prototype site using a mixture of HTTP and HTTPS (for the
secure bits). It all seems to be coming on fine, except with respect
to one current problem in file uploads. I've reduced the code to the
example below (hopefully not having thrown bits of baby out with the
bathwater...)
import os, cherrypy
class Root:
@cherrypy.expose
def index(self):
return """<html><body>
<form action="upload" method="post" enctype="multipart/
form-data">
filename: <input type="file" name="myFile"/><br/><input
type="submit"/>
</form></body></html>"""
@cherrypy.expose
def upload(self, myFile):
return """<html><body>myFile filename: %s<br/>myFile mime-
type: %s</body></html>
""" % ( myFile.filename, myFile.type)
global_conf = {'global': {
'server.socket_host': '0.0.0.0',
'server.socket_port': 8080,
#'server.ssl_certificate': 'prg/cms.aculab.com.crt',
#'server.ssl_private_key': 'prg/cms.aculab.com.key',
'log.screen': True}
}
cherrypy.config.update( global_conf)
cherrypy.quickstart( Root())
Run exactly as shown with HTTP on 8080, I can upload files (well, they
get into CP and print the little summary, but my example makes no
effort to save 'myFile' from there...) - files of any practical
size...
If I comment out the two ssl_ config elements, then everything will
work just the same using HTTPS on 8080, but will only successfully
upload files < 8192 bytes. A 100k file, for example causes the
following complaint (at the screen - the page itself never appears)...
<stacks of stack trace...>
File "/usr/local/lib/python2.6/dist-packages/cherrypy/
_cpreqbody.py", line 560, in read
data = self.fp.read(chunksize)
File "/usr/local/lib/python2.6/dist-packages/cherrypy/wsgiserver/
__init__.py", line 265, in read
data = self.rfile.read(size)
File "/usr/local/lib/python2.6/dist-packages/cherrypy/wsgiserver/
__init__.py", line 932, in read
assert n <= left, "recv(%d) returned %d bytes" % (left, n)
AssertionError: recv(8192) returned 16384 bytes
A 10.2k file causes a similar complaint that 10606 bytes have been
received....
Yet everything else I have done under SSL (downloads, for example)
seem to work just fine...
CP version 3.2.0rc1 on python 2.6.5 on Ubuntu 10.4.
Again, thanks for your help
Alan
secure bits). It all seems to be coming on fine, except with respect
to one current problem in file uploads. I've reduced the code to the
example below (hopefully not having thrown bits of baby out with the
bathwater...)
import os, cherrypy
class Root:
@cherrypy.expose
def index(self):
return """<html><body>
<form action="upload" method="post" enctype="multipart/
form-data">
filename: <input type="file" name="myFile"/><br/><input
type="submit"/>
</form></body></html>"""
@cherrypy.expose
def upload(self, myFile):
return """<html><body>myFile filename: %s<br/>myFile mime-
type: %s</body></html>
""" % ( myFile.filename, myFile.type)
global_conf = {'global': {
'server.socket_host': '0.0.0.0',
'server.socket_port': 8080,
#'server.ssl_certificate': 'prg/cms.aculab.com.crt',
#'server.ssl_private_key': 'prg/cms.aculab.com.key',
'log.screen': True}
}
cherrypy.config.update( global_conf)
cherrypy.quickstart( Root())
Run exactly as shown with HTTP on 8080, I can upload files (well, they
get into CP and print the little summary, but my example makes no
effort to save 'myFile' from there...) - files of any practical
size...
If I comment out the two ssl_ config elements, then everything will
work just the same using HTTPS on 8080, but will only successfully
upload files < 8192 bytes. A 100k file, for example causes the
following complaint (at the screen - the page itself never appears)...
<stacks of stack trace...>
File "/usr/local/lib/python2.6/dist-packages/cherrypy/
_cpreqbody.py", line 560, in read
data = self.fp.read(chunksize)
File "/usr/local/lib/python2.6/dist-packages/cherrypy/wsgiserver/
__init__.py", line 265, in read
data = self.rfile.read(size)
File "/usr/local/lib/python2.6/dist-packages/cherrypy/wsgiserver/
__init__.py", line 932, in read
assert n <= left, "recv(%d) returned %d bytes" % (left, n)
AssertionError: recv(8192) returned 16384 bytes
A 10.2k file causes a similar complaint that 10606 bytes have been
received....
Yet everything else I have done under SSL (downloads, for example)
seem to work just fine...
CP version 3.2.0rc1 on python 2.6.5 on Ubuntu 10.4.
Again, thanks for your help
Alan
alanp
Oct 26, 2010, 12:18:18 PM10/26/10
to cherrypy-users
A little more info...
I've spotted there is an open ticket on this very issue (#954 - about
a year old), so I'm not too hopeful of a quick resolution
here ;~( However, a couple of additional observations:
i) I had OpenSSL v0.10 installed (ubuntu 10.4 considers this the
'current' version). I've installed the latest 0.12a2, and *no
difference* that I can see...
ii) We have a Fedora 8 system here that doesn't exhibit the problem
(under apparently identical conditions!!!)
... perplexed ...
I've spotted there is an open ticket on this very issue (#954 - about
a year old), so I'm not too hopeful of a quick resolution
here ;~( However, a couple of additional observations:
i) I had OpenSSL v0.10 installed (ubuntu 10.4 considers this the
'current' version). I've installed the latest 0.12a2, and *no
difference* that I can see...
ii) We have a Fedora 8 system here that doesn't exhibit the problem
(under apparently identical conditions!!!)
... perplexed ...
Iztok Kavkler
Oct 27, 2010, 7:25:58 AM10/27/10
to cherryp...@googlegroups.com
I don't know if this is the same problem, but: Cherrypy has serious
issues with timeouts when reading the body of a large post request. Just
try adding
server.socket_timeout = <some large number>
to the global section of your config.
There is more about it on the page
http://nickmurdoch.livejournal.com/382648.html
Best,
Iztok
aculab\alanp
Oct 27, 2010, 7:45:18 AM10/27/10
to cherryp...@googlegroups.com
On Wed, 2010-10-27 at 13:25 +0200, Iztok Kavkler wrote:
> I don't know if this is the same problem, but: Cherrypy has serious
> issues with timeouts when reading the body of a large post request. Just
> try adding
>
> server.socket_timeout = <some large number>
>
> to the global section of your config.
>
> There is more about it on the page
>
> http://nickmurdoch.livejournal.com/382648.html
> I don't know if this is the same problem, but: Cherrypy has serious
> issues with timeouts when reading the body of a large post request. Just
> try adding
>
> server.socket_timeout = <some large number>
>
> to the global section of your config.
>
> There is more about it on the page
>
> http://nickmurdoch.livejournal.com/382648.html
Hey thanks - that's a great tip.
However, I'm certain it's not my issue.... I now have (what for the
moment appears to be) a decent work-around... I'll post it back here
when I have more certainty (I've already eaten too many words in my life
to be in a hurry...)
Rgrds, Alan
alanp
Oct 27, 2010, 10:37:37 AM10/27/10
to cherrypy-users
So I have a "work-around" (aka a really horrible cruft!)
The first observation is that someone put the assert in the read()
function "__init__.py" to detect precisely this problematic condition
- that the recv( 8192) call returns more than 8192... And yes, under
certain conditions, under ssl, it clearly does (but maybe not for
everybody!)
I put stack trace into the recv function and tested uploading a 10k
file, and it is clear that when read() calls self.recv(8192), the call
initially visits a recv() fn in the ssl library, which then calls down
to the __init__ recv() function *twice*, first getting 8192 bytes,
then getting a further 2k, all of which (10k) it then returns to the
read function.
So it seems to be the ssl library that isn't honouring the (8192)
constraint.
So, as I don't at all understand the overall design of this stack, in
the best tradition of modern medicine, my "cure" is to treat the
symptoms (duh!)...
In the read() routine (around line 900 in __init__.py) in the event
there is more than the requested data, instead of running straight
into the assert, I write out the requested amount (typically 8192),
and stash the rest away in a variable "self._datastash", which I
create on the fly (I said it was nasty!) On entering read(), before
fetching any new data, I check if there is anything in
self._datastash, and if so, I return it....
I'm sure that if I came understand exactly how the buffers in that
routine are properly being used, I could forge a more elegant
solution, but my head hurts too much already...
<snip somewhere_around_line=900, comments=mostly_removed>
if size < 0:
self._rbuf = StringIO.StringIO() # reset _rbuf. we
consume it via buf.
while True:
### start fixup
try:
if self._datastash:
data = self._datastash
self._datastash = ''
else:
data = self.recv(rbufsize)
except:
self._datastash = ''
### end fixup (plus indented next line)
data = self.recv(rbufsize)
if not data:
break
buf.write(data)
return buf.getvalue()
else:
<snippy />
while True:
left = size - buf_len
### start fixup
try:
if self._datastash:
data = self._datastash
self._datastash = ''
else:
data = self.recv( left)
except:
self._datastash = ''
### end fixup (plus indented next line)
data = self.recv( left)
if not data:
break
n = len(data)
if n == size and not buf_len:
return data
if n == left:
buf.write(data)
del data # explicit free
break
### start fixup
if n > left:
buf.write(data[ :left])
self._datastash = data[ left:]
del data # explicit free
break
### end fixup
buf_len += n
del data # explicit free
#assert buf_len == buf.tell()
return buf.getvalue()
</snip>
I hope this is useful to someone, with luck, to sort it out properly
(somehow) for the next release...
Thanks, Alan
The first observation is that someone put the assert in the read()
function "__init__.py" to detect precisely this problematic condition
- that the recv( 8192) call returns more than 8192... And yes, under
certain conditions, under ssl, it clearly does (but maybe not for
everybody!)
I put stack trace into the recv function and tested uploading a 10k
file, and it is clear that when read() calls self.recv(8192), the call
initially visits a recv() fn in the ssl library, which then calls down
to the __init__ recv() function *twice*, first getting 8192 bytes,
then getting a further 2k, all of which (10k) it then returns to the
read function.
So it seems to be the ssl library that isn't honouring the (8192)
constraint.
So, as I don't at all understand the overall design of this stack, in
the best tradition of modern medicine, my "cure" is to treat the
symptoms (duh!)...
In the read() routine (around line 900 in __init__.py) in the event
there is more than the requested data, instead of running straight
into the assert, I write out the requested amount (typically 8192),
and stash the rest away in a variable "self._datastash", which I
create on the fly (I said it was nasty!) On entering read(), before
fetching any new data, I check if there is anything in
self._datastash, and if so, I return it....
I'm sure that if I came understand exactly how the buffers in that
routine are properly being used, I could forge a more elegant
solution, but my head hurts too much already...
<snip somewhere_around_line=900, comments=mostly_removed>
if size < 0:
self._rbuf = StringIO.StringIO() # reset _rbuf. we
consume it via buf.
while True:
### start fixup
try:
if self._datastash:
data = self._datastash
self._datastash = ''
else:
data = self.recv(rbufsize)
except:
self._datastash = ''
### end fixup (plus indented next line)
data = self.recv(rbufsize)
if not data:
break
buf.write(data)
return buf.getvalue()
else:
<snippy />
while True:
left = size - buf_len
### start fixup
try:
if self._datastash:
data = self._datastash
self._datastash = ''
else:
data = self.recv( left)
except:
self._datastash = ''
### end fixup (plus indented next line)
data = self.recv( left)
if not data:
break
n = len(data)
if n == size and not buf_len:
return data
if n == left:
buf.write(data)
del data # explicit free
break
### start fixup
if n > left:
buf.write(data[ :left])
self._datastash = data[ left:]
del data # explicit free
break
### end fixup
assert n <= left, "recv(%d) returned %d bytes" %
(left, n)
buf.write(data)
(left, n)
buf_len += n
del data # explicit free
#assert buf_len == buf.tell()
return buf.getvalue()
</snip>
I hope this is useful to someone, with luck, to sort it out properly
(somehow) for the next release...
Thanks, Alan
Tim
Nov 21, 2012, 3:10:25 PM11/21/12
to cherryp...@googlegroups.com
A fix for this has been committed see https://bitbucket.org/stpierre/cherrypy/changeset/1001a34c0fd52f91353b4b8e66ff444026f95ed5#chg-cherrypy/wsgiserver/wsgiserver2.py I replaced my wsgiserver2.py and my uploads are working fine with https now.
Stephan Semerad
Jun 22, 2017, 3:55:43 PM6/22/17
to cherrypy-users
You guys are my heros! :) thanks a lot
Reply all
Reply to author
Forward
0 new messages