potential issue re in memory django file uploading.
david b.
django/core/files/uploadhandler.py) :
FileUploadHandler
...
def new_file(self, field_name, file_name, content_type,
content_length, charset=None):
"""
Signal that a new file has been started.
Warning: As with any data from the client, you should not trust
content_length (and sometimes won't even get it).
"""
So the content_length we control right? - Maybe I missed something but
... I can say I want to upload a small file then upload a file that
triggers an oom condition / use a lot of memory no ? ...
And then this.
class MemoryFileUploadHandler(FileUploadHandler):
"""
File upload handler to stream uploads into memory (used for small
files).
"""
def handle_raw_input(self, input_data, META, content_length,
boundary, encoding=None):
"""
Use the content_length to signal whether or not this handler
should be in use.
"""
# Check the content-length header to see if we should
# If the post is too large, we cannot use the Memory handler.
if content_length > settings.FILE_UPLOAD_MAX_MEMORY_SIZE:
self.activated = False
else:
self.activated = True
def new_file(self, *args, **kwargs):
super(MemoryFileUploadHandler, self).new_file(*args, **kwargs)
if self.activated:
self.file = StringIO()
raise StopFutureHandlers()
def receive_data_chunk(self, raw_data, start):
"""
Add the data to the StringIO file.
"""
if self.activated:
self.file.write(raw_data)
else:
return raw_data
def file_complete(self, file_size):
"""
Return a file object if we're activated.
"""
if not self.activated:
return
self.file.seek(0)
return InMemoryUploadedFile(
file = self.file,
field_name = self.field_name,
name = self.file_name,
content_type = self.content_type,
size = file_size,
There is a regression test for this BUT --> in the test suite there
is # A small file (under the 5M quota)
which is governed by
(django/tests/regressiontests/file_uploads/uploadhandler.py)
def receive_data_chunk(self, raw_data, start):
self.total_upload += len(raw_data)
if self.total_upload >= self.QUOTA:
raise StopUpload(connection_reset=True)
return raw_data
So obviously my proposed attack is to simply say "content length is
tiny" and "this file is actually HUGE".
I hope I missed something :) I don't really want this to occur ...
Graham Dumpleton
the original content at the content length specified in the request.
Thus not possible to force more than content length down to the
application level.
Graham
dave b
The documentation and code in django suggests that this is not the
case. So lets assume we are not using apache but another httpd of some
sort - then this problem will be present.
http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.4 seems to
say otherwise from my reading.
Also as per the django code comment (in django/core/files/uploadhandler.py):
def new_file(self, field_name, file_name, content_type,
content_length, charset=None):
"""
Signal that a new file has been started.
Warning: As with any data from the client, you should not trust
content_length (and sometimes won't even get it).
"""
self.field_name = field_name
self.file_name = file_name
self.content_type = content_type
self.content_length = content_length
self.charset = charset
dave b
Just to clarify this - I meant that the http content length header
item is *not* required - as per
http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.4 (also see
4.4.2), so I do not believe that apache would do what you said :) -
there is a default limit in a apache of around 2gb for the attackers
file to reach though.
dave b
Woop I hit send a bit early, I meant to also include:
So now the attack can now just not specify the content length and um...
>>> None < 2621440
True
We pass the check ;) as far as I know. Look if I really have missed
something do tell me :)
Steve Holden
> On 28 August 2010 23:21, dave b <db.pu...@gmail.com> wrote:
>> On 28 August 2010 23:09, dave b <db.pu...@gmail.com> wrote:
>>> The documentation and code in django suggests that this is not the
>>> case. So lets assume we are not using apache but another httpd of some
>>> sort - then this problem will be present.
>>> http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.4 seems to
>>> say otherwise from my reading.
>>
>> Just to clarify this - I meant that the http content length header
>> item is *not* required - as per
>> http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.4 (also see
>> 4.4.2), so I do not believe that apache would do what you said :) -
>> there is a default limit in a apache of around 2gb for the attackers
>> file to reach though.
>>
>
> Woop I hit send a bit early, I meant to also include:
> So now the attack can now just not specify the content length and um...
>
>>>> None < 2621440
> True
>
>
> We pass the check ;) as far as I know. Look if I really have missed
> something do tell me :)
>
from earlier in the thread. That way, I might actually read them ... ;-)
regards
Steve
--
DjangoCon US 2010 September 7-9 http://djangocon.us/
Graham Dumpleton
On Aug 28, 11:21 pm, dave b <db.pub.m...@gmail.com> wrote:
> >>> So obviously my proposed attack is to simply say "content length is
> >>> tiny" and "this file is actually HUGE".
> >>> I hope I missed something :) I don't really want this to occur ...
>
> >> A decent web server such as Apache (under mod_wsgi) will stop reading
> >> the original content at the content length specified in the request.
> >> Thus not possible to force more than content length down to the
> >> application level.
>
> >> Graham
>
> > The documentation and code in django suggests that this is not the
> > case. So lets assume we are not using apache but another httpd of some
> > sort - then this problem will be present.
>
> Just to clarify this - I meant that the http content length header
> there is a default limit in a apache of around 2gb for the attackers
> file to reach though.
only way you can not have a content length.
Chunked transfer encoding on request content isn't well supported.
Specifically, chunked transfer encoding is not supported by:
mod_python
FASTCGI
SCGI
AJP
CGI
WSGI
So, basically every standard way that Django can be hosted doesn't
support chunked transfer encoding on request content.
What you should instead see is an immediate HTTP_LENGTH_REQUIRED error
response from web server.
That said, some pure Python WSGI servers, eg., wsgiref in Python
standard library, may be broken and don't properly error when no
content length. However, for WSGI applications at least, they will be
broken if they try and read more than content length anyway, where it
takes on value of 0 if not specified, because reading more than
content length is a violation of the WSGI specification.
All up, I would suggest you are getting worked up over nothing.
BTW, have you actually tried to test your hypothesis to see whether it
is correct? If you have, post what server arrangement you are using.
Graham
Steve Holden
> On Aug 28, 11:21 pm, dave b <db.pub.m...@gmail.com> wrote:
>>>>> So obviously my proposed attack is to simply say "content length is
>>>>> tiny" and "this file is actually HUGE".
> All up, I would suggest you are getting worked up over nothing.
dave b
> On 8/28/2010 6:10 PM, Graham Dumpleton wrote:
>> On Aug 28, 11:21 pm, dave b <db.pub.m...@gmail.com> wrote:
>>>>>> So obviously my proposed attack is to simply say "content length is
>>>>>> tiny" and "this file is actually HUGE".
> [...]
>> All up, I would suggest you are getting worked up over nothing.
> +1
>
Yes I have :) it "works for me tm".
Also, you have to consider the other problem. If the file is > 2.5 mb
it can be put in /tmp and this has no size limits which again is going
to make the system slower and can be used to attack it? in either case
there seem to be real protections against this in django core as far
as I can see.
dave b
I meant "no" real protections against this in django core.
Graham Dumpleton
On Aug 29, 1:17 pm, dave b <db.pub.m...@gmail.com> wrote:
LimitRequestBody 1000000
and Apache/mod_wsgi will give back a HTTP_REQUEST_ENTITY_TOO_LARGE
error when it goes over that size before it even passes the request to
Django and even before any of the request content is read by Apache.
So, add the protections where most appropriate if you want to outright
block requests with large content. If your issue is efficient handling
of large posts, where you do want to handle them, then that is an
issue for Django.
Note that other Apache modules by which you can host Django may not
work properly in honouring LimitRequestBody directive of Apache. The
mod_python module for example doesn't really get it right, causing an
exception when request content tries to be read by Django application,
resulting in a malformed error response.
Graham
dave b
Yes I understand that both apache and some the mods have limits.
However, this doesn't stop a persistent attacker abusing these
relatively high limits.
IMHO I feel that django should be able to put a cap on the largest
size temporary file size possible. In addition, /tmp is a fine place
to store temporary files if the size may not be known.
I propose the following in psudo code:
1. start reading the file if we are receiving it (regardless of the
default 2.5 mb limit)
2. once it goes over 2.5mb / the configured default shift to the next
available storage option.
In addition, as you are pointing out that it is really not possible
for django to handle chunked requests and the content length field
must be specified then you should have no problems with the following
patch (in either case this is safe because by default it will revert
back to the temporary file storage) :
--- django/core/files/uploadhandler.py.orig 2010-08-29 13:50:17.000000000 +1000
+++ django/core/files/uploadhandler.py 2010-08-29 14:01:15.000000000 +1000
@@ -153,7 +153,7 @@
"""
# Check the content-length header to see if we should
# If the post is too large, we cannot use the Memory handler.
- if content_length > settings.FILE_UPLOAD_MAX_MEMORY_SIZE:
+ if content_length is None or content_length >
settings.FILE_UPLOAD_MAX_MEMORY_SIZE:
self.activated = False
else:
self.activated = True
@@ -170,6 +170,7 @@
"""
if self.activated:
self.file.write(raw_data)
+ self.file.truncate(settings.FILE_UPLOAD_MAX_MEMORY_SIZE)
else:
return raw_data
Steve Holden
the many issues a Django sit will face this is a relatively low
probability attack. That's fair enough - a vulnerability is a
vulnerability, after all, no matter how improbable, and not everyone
will set up their production systems to be correctly protected by a
front-end server.
If you really want to see this in Django you would be better advised to
post it in the issue tracker, where it will not get lost.
dave b
> the many issues a Django sit will face this is a relatively low
> probability attack. That's fair enough - a vulnerability is a
> vulnerability, after all, no matter how improbable, and not everyone
> will set up their production systems to be correctly protected by a
> front-end server.
>
> If you really want to see this in Django you would be better advised to
> post it in the issue tracker, where it will not get lost.
>
>
Ok I will do this :-)
Sure Also, Wow gmail is a horrible email client. I am so used to $else.
The patch I included in my previous email should not break any *real*
clients, only evil ones(potentially).
As I understand it an attacker can abuse gzip user requests, if
mod_deflate is enabled (AND configured to decompress incoming user
requests - this is not the default) in apache2 with a user gziped
request body.
So an attack could do effectively have a file like this:
f = open("rar", "w")
string = ""
for i in range(0, 10000000):
string += " " + "1"
f.write(string)
f.close()
ls -lah 20M 2010-08-29 17:15 rar
(except replace write with append and do it a lot more ;) ) and then
send it gziped as in the request body.
Just for fun ;)
gzip rar
ls -lah 19K 2010-08-29 17:15 rar.gz
So django will receive the original 20M file (as the httpd has
uncompressed it for django ) afaik.
see Input Decompression at http://httpd.apache.org/docs/2.0/mod/mod_deflate.html
dave b
> it near enough your hosting service to take your site down. What
> counter-measures are you going to take against that?
Good question. I have two cats and they like to lick people ^^
They are a bit friendly I guess. Do you think I can train them to
pounce on strangers?
> You perhaps feel people aren't taking your proposal seriously enough.
> The fact of the matter is that security is never absolute, and on a
> threat scale of 0 to 10 this issue comes in at about 1.7. If you are
> running a professional service and you are monitoring it correctly then
> you ought to notice an attack of this nature before it does too much damage.
Look I send you guys an email, I send a patch for one of the problems
and point out the second one. I just can't make you guys happy can I ?
http://cwe.mitre.org/top25/
[22] 145 CWE-770 Allocation of Resources Without Limits or Throttling
http://cwe.mitre.org/top25/#CWE-770
But ok if you say so ;)
I wonder how much ram most django commonly found django installations
that allow file upload on the internet have? now remember a lot of
those have a fairly fast download and upload ;)
bonus points if they have mod_deflate decompressing the user body request ;)
> I repeat, you may be correct in treating this as a vulnerability, but
> your estimate of its seriousness appears to disagree with that of
> others. If you want to have your code seriously considered for inclusion
> (and why not?) you should raise it in the Django issue tracker - see
> "Reporting Bugs" in
I did as I was suggested to do so :)
Some one first told me on irc in #django that I should raise it here first :)
Please see http://code.djangoproject.com/ticket/14192
Steve Holden
That's cool. Sorry I nagged you unnecessarily. Yes, it might be a
problem. But you'll notice it's #22 on a list of 25 ...
Anyway, since you have done your civic duty there's a good chance that a
fix will find its way into some future version. Thanks for being a good
citizen.
dave b
> fix will find its way into some future version. Thanks for being a good
> citizen.
Django is an awesome project and. However, a bug is a bug. I don't
care if it is a security bug or not, a bug *should* get fixed.
FYI: I sent this information originally to the django security contact
email address as per the website.
This is what I got back.
"I'll be on vacation and offline until December 27th.
I likely won't be responding to the majority of email received between
now and then. If it's important, please re-send the email after the
27th.
Revolution Systems clients should contact <sup...@revsys.com> and/or
Frank Wiles at <fr...@revsys.com>.
In an emergency, I can be contacted at the Ranguana Lodge in
Placencia, Belize. The local number is 523-3112; callers from the US
should call 011-501-523-3112.
Thanks!
Jacob"
So I understand people are busy etc. but ... django is a large project
why did I receive this email?
Required <sarcasm> That's ok all attackers will not look for security
problems in django will not look for bugs while Jacob is away.
</sarcasm>
--
The ripest fruit falls first. -- William Shakespeare, "Richard II"
Graham Dumpleton
On Aug 29, 9:43 pm, dave b <db.pub.m...@gmail.com> wrote:
> > OK, so you don't believe the advice you are getting, which is that of
> > the many issues a Django sit will face this is a relatively low
> > probability attack. That's fair enough - a vulnerability is a
> > vulnerability, after all, no matter how improbable, and not everyone
> > will set up their production systems to be correctly protected by a
> > front-end server.
>
> > If you really want to see this in Django you would be better advised to
> > post it in the issue tracker, where it will not get lost.
>
> Ok I will do this :-)
>
> Sure Also, Wow gmail is a horrible email client. I am so used to $else.
> The patch I included in my previous email should not break any *real*
> clients, only evil ones(potentially).
>
> As I understand it an attacker can abuse gzip user requests, if
> mod_deflate is enabled (AND configured to decompress incoming user
> requests - this is not the default) in apache2 with a user gziped
> request body.
This is because use of a mutating input filter such as mod_deflate
with a WSGI based application is not possible.
As I said before, a WSGI application is not supposed to read more
content than is specified by the request content length, if it is, it
is a non compliant WSGI application.
For the case of mod_deflate, in decompressing the request content, it
cannot know in advance what the new content length will be. As such,
it leaves the original content length header intact. This means that a
conforming WSGI application would still only read up to that original
content length even if you were to enable mod_deflate. The result
being that the decompressed request content would actually be
truncated at the content length specified for the original compressed
content.
In short, you can not use mod_deflate on request content with
conforming WSGI applications as that would only read up to the
original content length. I have blogged about this limitation of WSGI
in the past, so maybe you should go search through postings on my
blog.
As I asked you before, have you actually run tests to prove your
hypothesis that this is all broken? If you have, then post what
hosting configuration and software you are using.
Graham
Russell Keith-Magee
>> Anyway, since you have done your civic duty there's a good chance that a
>> fix will find its way into some future version. Thanks for being a good
>> citizen.
>
> Django is an awesome project and. However, a bug is a bug. I don't
> care if it is a security bug or not, a bug *should* get fixed.
>
> FYI: I sent this information originally to the django security contact
> email address as per the website.
> This is what I got back.
>
>
> "I'll be on vacation and offline until December 27th.
This is deeply concerning. This is the second time in very recent
memory that someone has said they've had difficulty getting through to
secu...@djangoproject.org.
I need to do some investigation to determine exactly what is going on
here. Allow me to assure you that we take security reports *very*
seriously, and if people are having problems contacting security@,
then this is a problem we need to solve as a very high priority.
As for the problem: At this point, I'm thoroughly confused about what
you're actually describing. In order for this to be something that
needs a fix, it needs to be:
1) An actual problem where you can clearly describe the circumstances
or sequence of events that would allow an attack to occur, and
2) Something that is actually Django's problem -- by which I mean,
something that is actually Django's responsibility to solve, rather
than something that is a webserver configuration issue.
At this point, it's not clear to me that either of these two things
are true. Based on your messages and the feedback from Graham and
Steve, it sounds like you're describing an attack that *could* exist,
but only if you've got a misconfigured (or badly implemented) web
server.
If you believe that I'm wrong, and there *is* an actual problem, you
need to convince us. This doesn't mean posting large wads of Django's
source code and proposed patches over multiple messages. It means
describing in clear, concise language exactly what conditions need to
exist for a problem to occur.
Yours,
Russ Magee %-)
dave b
Morning. Will do so below here.
Just do remember, there is more than one way to run a httpd, some of
us run our own custom stuff ;) and not everyone is using a setup like
you have.
-------------------
Feature: Attacker crashes your django installation via file uploading
As attacker
I want to crash your django installation
To take your site down or reduce its availability, so I can steal the
underpants and then profit!!!!
Background:
Given I am an attacker
And you have uploads enabled with the default settings (memory and
temporary file).
And you are running on a platform with /tmp
Feature: I upload a 1gb file and have this go into system memory
Given I have a 1gb file
When I uploaded it to the website
Then I should see that your system now has used an additional 1gb of /tmp
And available system memory is now reduced
So basically I was saying there are two problems.
One is if the httpd isn't behaving properly(this is probably not
entirely true) with respect to the content length field and abusing
memory limitation.
The second issue is that there is no *default* set limit on temporary
file uploads, so any file larger than 2.5mb can find its way to /tmp
and there is no limit on the size of these files in django core.
That is there is no set limit on the size of a temporary file upload.
The second problem is going to exist within the bounds of the set
limits of the webserver and the various mods that are used with
django.
In an extreme and very unlikely case, the httpd may ungzip the data
from the attacker and modify the content length (when it knows what it
should be - the connection is terminated ) with django getting a large
amount of data to store from a much smaller user body request.
--
Let me take you a button-hole lower. -- William Shakespeare, "Love's
Labour's Lost"
Graham Dumpleton
On Aug 30, 1:54 pm, dave b <db.pub.m...@gmail.com> wrote:
> On 30 August 2010 11:04, Russell Keith-Magee <russ...@keith-magee.com> wrote:
not memory and since data is streamed when being written to disk, the
only memory impact on the application would be what is required to
hold one block of the data in memory while it is being read/written.
The whole file would not be getting stored in memory because when >
2.5MB, it wouldn't be using in memory file uploader, but the disk
based one.
> So basically I was saying there are two problems.
> memory limitation.
all request content into memory at one time, they stream data, usually
through the web application actually pulling it through a chunk at a
time when the application wants it. Thus is matters little what
content length is to the web server.
> The second issue is that there is no *default* set limit on temporary
> file uploads, so any file larger than 2.5mb can find its way to /tmp
> and there is no limit on the size of these files in django core.
> That is there is no set limit on the size of a temporary file upload.
isn't an equivalent of LimitRequestBody definable within Django unless
can be done with middleware. But then it is as I pointed out the wrong
place to be doing it. The better place is in the web server. This is
because the web server can reject it immediately. If you do it in the
application, then for any hosting mechanism that has to proxy data
through to the application, eg., mod_wsgi daemon mode, fastcgi, scgi,
uWSGI etc, then the web server will have already started proxying
through the data to the application process, thus wasting cycles and
resources, by the time the application decides it doesn't want to
handle it. You are thus always better off rejecting large requests as
early as possible in the pipeline.
So, yes it may make sense for Django to have a fail safe and allow you
to specify a maximum on upload size if it doesn't already, but that is
only of use where you haven't set up your production web server
properly to protect you from abuses, something you should be doing.
> The second problem is going to exist within the bounds of the set
> limits of the webserver and the various mods that are used with
> django.
>
> In an extreme and very unlikely case, the httpd may ungzip the data
> from the attacker and modify the content length (when it knows what it
> should be - the connection is terminated ) with django getting a large
> amount of data to store from a much smaller user body request.
The mod_deflate filter does not modify the content length as it cant.
This is because it has to stream the decompressed data and by the time
it knows the actual content length, the original content length header
has long ago been passed through to the application and there is no
opportunity to update it. As such. any application will only see the
length as being what the original content length said it was and would
truncate the data at that. As I said before, it is technically not
possible to use mod_deflate on request content with a WSGI
application.
Anyway, I would have to agree with Russell, you are simply not making
yourself clear enough and to added to that seem to keep echoing
statements that have been refuted.
For the third time I ask you whether you have actually gone and tested
your hypothesis and can provide a working test case that demonstrates
the problem.
FWIW, there are much simpler ways to bring a site down than this. I
suggest you go research Slowloris.
http://en.wikipedia.org/wiki/Slowloris
Graham
dave b
You have a valid point re /tmp, sorry I am used to mounting /tmp as
/tmpfs - my mistake :)
Ok lets be *really* clear the security problem still exists.
An attack can in the limits set on the maximum post by the httpd /
module in use upload a large file.
> I don't actually use Django so not 100% sure, but yes there possibly
> isn't an equivalent of LimitRequestBody definable within Django unless
> can be done with middleware.
Ok so you don't even use django, ok...
You know I think I missed your presentation at pycon-au.
>
> So, yes it may make sense for Django to have a fail safe and allow you
> to specify a maximum on upload size if it doesn't already, but that is
> only of use where you haven't set up your production web server
> properly to protect you from abuses, something you should be doing.
Yes and imho it should be in django by default, not up to end django
users to figure out.
Secure by default please!
>
> Anyway, I would have to agree with Russell, you are simply not making
> yourself clear enough and to added to that seem to keep echoing
> statements that have been refuted.
If you say so. I was pushing some other(more aggressive) impacts in
exotic configurations with custom httpd etc. .
> For the third time I ask you whether you have actually gone and tested
> your hypothesis and can provide a working test case that demonstrates
> the problem.
Ok. Look. You don't use django.
1. Try this - go to the django website
http://docs.djangoproject.com/en/dev/intro/tutorial01/
2. and follow the tutorial 1 (and also do 2 ) when it says put the
poll file like this:
from django.db import models
class Poll(models.Model):
question = models.CharField(max_length=200)
pub_date = models.DateTimeField('date published')
class Choice(models.Model):
poll = models.ForeignKey(Poll)
choice = models.CharField(max_length=200)
votes = models.IntegerField()
put this instead:
from django.db import models
import datetime
class Poll(models.Model):
question = models.CharField(max_length=200)
pub_date = models.DateTimeField('date published')
filed = models.FileField(upload_to="tmp/")
def __unicode__(self):
return self.question
def was_published_today(self):
return self.pub_date.date() == datetime.date.today()
class Choice(models.Model):
poll = models.ForeignKey(Poll)
choice = models.CharField(max_length=200)
votes = models.IntegerField()
filed = models.FileField(upload_to="tmp/")
def __unicode__(self):
return self.choice
Ok still following?
well you finish the tutorial(s) now and then you try to upload a file right?
So you start uploading the file. Now because (I assume you are still
using the django built in webserver) why don't you play with this a
bit, start uploading say 10 1gb files(all at once) then stop them(all)
at around say 700mb~ in.
Have fun! (obviously you should go further than this and try with
apache setup etc.).
> FWIW, there are much simpler ways to bring a site down than this. I
> suggest you go research Slowloris.
I know about this attack, but I can use my attack against those who
are not using apache.
What do you say to this ?
Here you should get one of these -->
http://www.flickr.com/photos/chrisjrn/4740021871/sizes/l/
Isn't it cute?
--
Small things make base men proud. -- William Shakespeare, "Henry VI"
Graham Dumpleton
On Aug 31, 1:09 am, dave b <db.pub.m...@gmail.com> wrote:
> /me rolls eyes.
> You have a valid point re /tmp, sorry I am used to mounting /tmp as
> /tmpfs - my mistake :)
> Ok lets be *really* clear the security problem still exists.
> An attack can in the limits set on the maximum post by the httpd /
> module in use upload a large file.
This is because they are partitions dependent upon by other
applications. If you fill up those partitions you will impact those
other applications and they could fail and stuff up your system in
other ways. A decent production setup would put uploads in a dedicated
partition where use can be properly controlled.
> > I don't actually use Django so not 100% sure, but yes there possibly
> > isn't an equivalent of LimitRequestBody definable within Django unless
> > can be done with middleware.
>
> Ok so you don't even use django, ok...
> You know I think I missed your presentation at pycon-au.
>
> > So, yes it may make sense for Django to have a fail safe and allow you
> > to specify a maximum on upload size if it doesn't already, but that is
> > only of use where you haven't set up your production web server
> > properly to protect you from abuses, something you should be doing.
>
> Yes and imho it should be in django by default, not up to end django
> users to figure out.
> Secure by default please!
single global value. This is because in practice, what is a valid
maximum for an form post/upload depends on the URL.
This is why using front end web server like Apache is preferred,
because its Location directive allows you to implement more fine
grained control than what a single value allows.
For example, you could have a default of:
LimitRequestBody 1000000
but then for a specific URL designed to handle large uploads, you
could have:
LimitRequestBody 100000000
Allowing this in Django itself would be harder.
The fact still remains that you should do these things using what is
the most appropriate tool and Django isn't going to be the right
place.
It certainly would be very dangerous to rely purely on Django defaults
in a production setting and if that is the approach you take for your
systems, I would be quite concerned as through ignoring the abilities
of the web server to protect you, you may be opening yourself up to
even more danger. You always want to be cutting off things as early as
possible where it can be done efficiently. Doing it in your Python web
application is the worst place to do it as you still clog up the
pipeline and are also having to do it with a programming language
which isn't going to be as efficient in dealing things as pure C code
in the web server.
> > Anyway, I would have to agree with Russell, you are simply not making
> > yourself clear enough and to added to that seem to keep echoing
> > statements that have been refuted.
>
> If you say so. I was pushing some other(more aggressive) impacts in
> exotic configurations with custom httpd etc. .
>
> > For the third time I ask you whether you have actually gone and tested
> > your hypothesis and can provide a working test case that demonstrates
> > the problem.
>
> Ok. Look. You don't use django.
> 1. Try this - go to the django websitehttp://docs.djangoproject.com/en/dev/intro/tutorial01/
probably know more than most about the internal workings of Django and
Python web frameworks in general, and most definitely would know more
in regard to the interface between web server and web framework/
application. I am also well capable of reading code to get an
understanding things.
> 2. and follow the tutorial 1 (and also do 2 ) when it says put the
> poll file like this:
> well you finish the tutorial(s) now and then you try to upload a file right?
> So you start uploading the file. Now because (I assume you are still
> using the django built in webserver) why don't you play with this a
> bit, start uploading say 10 1gb files(all at once) then stop them(all)
> at around say 700mb~ in.
do what I expect you think it is. If I cut them all off while still
processing the first, the latter requests wouldn't even have been
accepted by the web server at that point and so do nothing.
Using the development server as a guide is also stupid. If you use a
real server, even one that can handle concurrent requests, if the
client connection is cut off, the application should se an exception
for the broken connection prior to all content being received.
Presumably in that case the upload file will be removed as wasn't
complete. Also, if you do manage to fill up the disk partition then
the write of data to file will fail and again an exception will occur
and file should be removed. If Django isn't removing files when
exceptions occur while writing it I would be quite surprised. Thus
impacts of filling up disk using a rogue upload should in some
respects be transitory as it should recover.
So, does this mean you have actually run this test and proven
conclusively that there is an issue, or again is this just
speculation? If you are finding that Django isn't cleaning up failed
uploads due to truncated client connection or running out of disk
space, then that may be an issue that needs solving, but you are not
be clear enough about the actual problems to know whether it is all
speculation or is instead fact.
> Have fun! (obviously you should go further than this and try with
> apache setup etc.).
I also don't care about that specific case, I am taking you to task
about your other hypothetical cases around trying to stuff down more
content than specified by content length, use of mod_deflate etc. That
is there area where I get the impression you haven't actually done any
testing.
Also, you posted patches previously. One was:
+++ django/core/files/uploadhandler.py 2010-08-29 14:01:15.000000000
+1000
@@ -153,7 +153,7 @@
"""
# Check the content-length header to see if we should
# If the post is too large, we cannot use the Memory
handler.
- if content_length > settings.FILE_UPLOAD_MAX_MEMORY_SIZE:
+ if content_length is None or content_length >
settings.FILE_UPLOAD_MAX_MEMORY_SIZE:
self.activated = False
else:
self.activated = True
look at the code of Django where it uses those classes, such as in
'django/http/multipartparser.py', you will see:
content_length = int(META.get('HTTP_CONTENT_LENGTH',
META.get('CONTENT_LENGTH',0)))
So, the users of those upload handler have already handled the case of
there being no CONTENT_LENGTH, making it 0 if not supplied. The
assumption there in code is therefore that you will always supply a
valid content length, and not None. Thus the patch serves no purpose.
The other patch is also not well thought out either. You have:
@@ -170,6 +170,7 @@
"""
if self.activated:
self.file.write(raw_data)
+ self.file.truncate(settings.FILE_UPLOAD_MAX_MEMORY_SIZE)
else:
return raw_data
going to be inefficient for a start. It is also bad for other reasons
as well.
First is that it silently throws away data in excess of a value
totally unrelated to the content length. Thus higher level code will
believe the upload worked quite okay, yet what has actually happened
is a loss of data.
Second is that this isn't actually avoiding the Django application
having to read the data in in the first place.
> > FWIW, there are much simpler ways to bring a site down than this. I
> > suggest you go research Slowloris.
>
> I know about this attack, but I can use my attack against those who
> are not using apache.
> What do you say to this ?
targeting slow delivery of bytes making up the original request
headers. It therefore is attacking the initial web server code.
Granted that it mainly affects process/thread based web servers and
not asynchronous servers, but think what happens if you do the same
slow delivery of data for the actual request content.
In this latter case you are hitting the actual web application itself
and because WSGI applications require a process/thread blocking model,
then applying this type of attack to actual request content will cause
problems for a Django application no matter what web server you are
using.
Some web servers used in front of an application may help to avoid
this. For example, when using nginx as a proxy front end it will
buffer up to 1MB of data in memory before actually deciding to proxy
the request through to backend. In other words, it attempts to only
hand over a request when pretty sure have the data needed to satisfy
the request. Because nginx is asynchronous, it itself will not suffer
too much from the problem except if it buffers in memory and memory
use starts increasing. You can still defeat it though by saying
content length is going to be more than 1MB, because in that case
nginx will proxy immediately and stream data and so backend can still
be attacked.
Anyway, enough said, I am not going to post about this again as little
I say seems to be sinking in at all. If you still think there is a
problem, submit a ticket to the Django issue tracker with clear
description, a test script to prove it along with results from test
script, plus your patch, plus your patch for that test script, plus
the documentation. Good luck.
Graham
Russell Keith-Magee
>> I don't actually use Django so not 100% sure, but yes there possibly
>> isn't an equivalent of LimitRequestBody definable within Django unless
>> can be done with middleware.
>
> Ok so you don't even use django, ok...
> You know I think I missed your presentation at pycon-au.
Whether Graham uses Django isn't especially relevant here. You've been
making a bunch of claims about mod_wsgi and Apache behavior. Graham is
most certainly an expert on those two tools.
>> So, yes it may make sense for Django to have a fail safe and allow you
>> to specify a maximum on upload size if it doesn't already, but that is
>> only of use where you haven't set up your production web server
>> properly to protect you from abuses, something you should be doing.
>
> Yes and imho it should be in django by default, not up to end django
> users to figure out.
> Secure by default please!
That's an easy epithet to throw around, but I disagree that it is
appropriate here. "Security" doesn't mean "stops the user from making
mistakes".
Also, consider this from a practical standpoint. If we introduced this
setting, we would need to introduce it with a default value. If we set
that limit to a low value (say, 10MB), lots of existing users would
discover that their website suddenly stopped accepting files. If we
set the value high (say 1GB), the attack still exists -- a malicious
user could upload a series of 999MB files. The only truly
backwards-compatibly option is to introduce the setting, but disable
it by default (i.e., no size limit). However, this relys on people
reading the documentation and determining an appropriate value for the
setting -- at which point, we've just duplicated the functionality of
Apache without actually changing anything.
IMHO, Graham is completely right when he says that the webserver is
the right place to catch this. Django isn't about to start introducing
more settings to duplicate functionality that is better provided by
other parts of the tool chain.
This is not an isolated policy decision, either. For example, Django
*could* provide a database connection pool -- but we have resolutely
refused to do so, on the grounds that there are excellent third party
tools that implement this functionality. Just because we *can* doesn't
mean we *should*.
That said, there has been discussion recently about adding a section
to Django's docs talking about security issues -- things that may not
be immediately obvious about project design and configuration, but
would behoove users to think about. A discussion of this problem
sounds like it would be a good addition.
Look -- Graham may not use Django on a daily basis, but he's not a
fool. For the record, neither am I. A cursory examination of his
history on this mailing list would indicate that saying "Add a
FileField uploading to /tmp to an existing model" would be more than
enough detail to describe your setup here.
The part of this problem that you continue to refuse to describe is
*THE ONLY PART THAT MATTERS* - the web server configuration that
you're using to make your assertion.
> well you finish the tutorial(s) now and then you try to upload a file right?
> So you start uploading the file. Now because (I assume you are still
> using the django built in webserver) why don't you play with this a
> bit, start uploading say 10 1gb files(all at once) then stop them(all)
> at around say 700mb~ in.
> Have fun! (obviously you should go further than this and try with
> apache setup etc.).
The implication here is that you *haven't* tried this with Apache.
Worse still, it sounds like you might be trying to use the Django
development server as your test case to validate that this is a
problem.
If this is the case, I'm going to start ignoring this thread -- the
development server *IS NOT* a production ready server [1], and *IS
NOT* a useful indication of anything approximating real-world server
behaviour, beyond the fact that it implements HTTP and WSGI. If you're
trying to use the dev server as proof of anything at the server level,
you're on your own.
[1] http://docs.djangoproject.com/en/dev/ref/django-admin/?from=olddocs#runserver-port-or-ipaddr-port
>> FWIW, there are much simpler ways to bring a site down than this. I
>> suggest you go research Slowloris.
> I know about this attack, but I can use my attack against those who
> are not using apache.
> What do you say to this ?
I say that this is the reason we're getting frustrated.
So far, you haven't actually told us which web server you *are* using.
You keep alluding to this "not-apache" webserver, but you haven't
actually said which webserver you *are* using.
Apache has it's faults, but it is feature complete, battle hardened,
widely available, and almost certainly the most widely used webserver
for Django deployment. It's also the webserver that virtually every
Django user and developer will have some experience in using. If a
potential problem exists in at the server level, Apache is the best
lingua franca to use in demonstrating it.
If the problem is actually that you're using a custom HTTP server, and
that custom HTTP server isn't providing a feature that you need (such
as a way to reject large uploads), then my argument to you is that you
need to get a better web server. Django isn't going to add features to
compensate for inadequate HTTP servers -- especially when there are a
wide range of HTTP servers out there that *do* provide a full feature
set.
Yours,
Russ Magee %-)
dave b
>
> That's an easy epithet to throw around, but I disagree that it is
> appropriate here. "Security" doesn't mean "stops the user from making
> mistakes".
Look like wsgi, apache2 and django all on ubuntu PLACE no size limits
at all by default. Isn't that neat?
I think debian is the same too!
Seriously, are you silly enough to think I was just using the
development server for testing?
Do not pass go do not collect profit!
> it by default (i.e., no size limit). However, this relys on people
> reading the documentation and determining an appropriate value for the
> setting -- at which point, we've just duplicated the functionality of
> Apache without actually changing anything.
Incorrect. Put it in the changelog etc.
I meant people are supposed to know about apache and its setup surely
they should read the changelog when django changes? right.
> IMHO, Graham is completely right when he says that the webserver is
> the right place to catch this. Django isn't about to start introducing
> more settings to duplicate functionality that is better provided by
> other parts of the tool chain.
I disagree. Very much so. Stop saying this isn't a django issue and
start fixing it.
> That said, there has been discussion recently about adding a section
> to Django's docs talking about security issues -- things that may not
> be immediately obvious about project design and configuration, but
> would behoove users to think about. A discussion of this problem
> sounds like it would be a good addition.
Look you guys are saying that django is secure and then not willing to
say "ok django might want to do something here". That's a great idea!
>> Ok still following?
>
> Look -- Graham may not use Django on a daily basis, but he's not a
> fool. For the record, neither am I. A cursory examination of his
> history on this mailing list would indicate that saying "Add a
> FileField uploading to /tmp to an existing model" would be more than
> enough detail to describe your setup here.
>
> The part of this problem that you continue to refuse to describe is
> *THE ONLY PART THAT MATTERS* - the web server configuration that
> you're using to make your assertion.
The default wsgi apache2 setup on $distro is a good testing place for
this if you want to test how people will likely have it setup, or was
said to me in prior emails here. I have tested it against that setup
of course and *against* *many* others.
>> apache setup etc.).
>
> The implication here is that you *haven't* tried this with Apache.
> Worse still, it sounds like you might be trying to use the Django
> development server as your test case to validate that this is a
> problem.
I can assure you I have tested this against apache2 with wsgi running
with django.
>> What do you say to this ?
>
> I say that this is the reason we're getting frustrated.
Put your hands up in the air like you just don't care!
> So far, you haven't actually told us which web server you *are* using.
> You keep alluding to this "not-apache" webserver, but you haven't
> actually said which webserver you *are* using.
This doesn't even matter here, my attack works against the default
apache wsgi and django setup K THX BIA.
> Apache has it's faults, but it is feature complete, battle hardened,
> widely available, and almost certainly the most widely used webserver
> for Django deployment. It's also the webserver that virtually every
> Django user and developer will have some experience in using. If a
> potential problem exists in at the server level, Apache is the best
> lingua franca to use in demonstrating it.
Done that :)
blahblahblalbha sssh listen. No Of course I have tested it against that.
How confident are you in that in every possible setup for django used
in production, this issue isn't a problem?
> If the problem is actually that you're using a custom HTTP server, and
> that custom HTTP server isn't providing a feature that you need (such
> as a way to reject large uploads)
Look. Should I post this to somewhere more security related on the internet?
Summary:
In the default setup of wsgi, apache and django (on distributions like
ubuntu and debian) by default there are no limits on the size of a
file that an attacker can upload.
http://cwe.mitre.org/top25/#CWE-770 and see example 2 at
http://cwe.mitre.org/data/definitions/770.html
--
Conscience doth make cowards of us all. -- Shakespeare
dave b
> From my testing (granted this was run against something pre-1.2 so things
> may have changed since then), as soon as you initiate the first file upload,
> you're monopolizing the devserver process, preventing further attempts to do
> the following 9 uploads until the first has completed (successfully or
> aborted).
>
> Not that you'd want to use devserver in a production environment anyways, as
> Russell points out in the documentation[1]. Note the "DO NOT USE THIS
> SERVER IN A PRODUCTION SETTING." It's not very subtle about that. :)
Exactly!
So test against wsgi and apache2 ;)
--
There's small choice in rotten apples. -- William Shakespeare, "The
Taming of the Shrew"
Steve Holden
>>> Secure by default please!
>>
>> That's an easy epithet to throw around, but I disagree that it is
>> appropriate here. "Security" doesn't mean "stops the user from making
>> mistakes".
>
> Look like wsgi, apache2 and django all on ubuntu PLACE no size limits
> at all by default. Isn't that neat?
> I think debian is the same too!
> Seriously, are you silly enough to think I was just using the
> development server for testing?
> Do not pass go do not collect profit!
>
troll. If you think there is even a possibility that he is that silly
then you are clearly sadly deluded. Russell is well know throughout the
Django community (and outside it) for his deep understanding of the
issues and compromises involved in building a professional web framework.
You, on the other hand, are a complete unknown who has walked in off the
street and started to make a lot of noise. All this has achieved is to
demonstrate your total incomprehension of how to achieve practical
results, which despite your focus on only the technical issues that
interest you involves knowing how to interact with people as well.
It's kind of cute that you appear to think you might be smarter than me
(though you certainly don't have as much experience, and frankly I think
I would be likely to win in sheer sex appeal too). It's sheer lunacy to
think you might be smarter than Keith-Magee and Dumpleton (though you
may *just* beat Graham on people skills, he has the definite advantage
that he has demonstrated he knows what he's talking about).
>
>
>> it by default (i.e., no size limit). However, this relys on people
>> reading the documentation and determining an appropriate value for the
>> setting -- at which point, we've just duplicated the functionality of
>> Apache without actually changing anything.
>
> Incorrect. Put it in the changelog etc.
> I meant people are supposed to know about apache and its setup surely
> they should read the changelog when django changes? right.
>
>> IMHO, Graham is completely right when he says that the webserver is
>> the right place to catch this. Django isn't about to start introducing
>> more settings to duplicate functionality that is better provided by
>> other parts of the tool chain.
>
> I disagree. Very much so. Stop saying this isn't a django issue and
> start fixing it.
>
Who are you to tell the Django developers what to be doing? If you have
a demonstrated public record of making sound engineering decisions that
resulted in well-constructed systems, *then* your opinion might count
for something. You don't just need brains, you need to now how to use
them as well.
>
>> That said, there has been discussion recently about adding a section
>> to Django's docs talking about security issues -- things that may not
>> be immediately obvious about project design and configuration, but
>> would behoove users to think about. A discussion of this problem
>> sounds like it would be a good addition.
>
> Look you guys are saying that django is secure and then not willing to
> say "ok django might want to do something here". That's a great idea!
>
>
>>> Ok still following?
>>
>> Look -- Graham may not use Django on a daily basis, but he's not a
>> fool. For the record, neither am I. A cursory examination of his
>> history on this mailing list would indicate that saying "Add a
>> FileField uploading to /tmp to an existing model" would be more than
>> enough detail to describe your setup here.
>>
>> The part of this problem that you continue to refuse to describe is
>> *THE ONLY PART THAT MATTERS* - the web server configuration that
>> you're using to make your assertion.
>
> The default wsgi apache2 setup on $distro is a good testing place for
> this if you want to test how people will likely have it setup, or was
> said to me in prior emails here. I have tested it against that setup
> of course and *against* *many* others.
>
Sigh. It has already been clearly explained to you that the default
installation of anything isn't a reliable way to measure a platform's
fitness for purpose. If you want to run a production server on the
default installation of Apache how far do you think you will get?
The point is, serious people with serious production problems well
understand the issues that can come up, and the don't expect that the
products they use will meet their needs out of the box.
Go to Google and look up the phrase "one-trick pony". That's where I
have you filed right now.
>
>
>>> apache setup etc.).
>>
>> The implication here is that you *haven't* tried this with Apache.
>> Worse still, it sounds like you might be trying to use the Django
>> development server as your test case to validate that this is a
>> problem.
>
> I can assure you I have tested this against apache2 with wsgi running
> with django.
>
But still you refuse to show us the results? That seems a bit perverse.
>
>>> What do you say to this ?
>>
>> I say that this is the reason we're getting frustrated.
>
> Put your hands up in the air like you just don't care!
>
It isn't our responsibility to understand your drivel. Make sense or
stop making this noise.
>> So far, you haven't actually told us which web server you *are* using.
>> You keep alluding to this "not-apache" webserver, but you haven't
>> actually said which webserver you *are* using.
> This doesn't even matter here, my attack works against the default
> apache wsgi and django setup K THX BIA.
>
>> Apache has it's faults, but it is feature complete, battle hardened,
>> widely available, and almost certainly the most widely used webserver
>> for Django deployment. It's also the webserver that virtually every
>> Django user and developer will have some experience in using. If a
>> potential problem exists in at the server level, Apache is the best
>> lingua franca to use in demonstrating it.
>
> Done that :)
> blahblahblalbha sssh listen. No Of course I have tested it against that.
> How confident are you in that in every possible setup for django used
> in production, this issue isn't a problem?
>
I don't know about Keith, but I am pretty confident that the Washington
Post, the New York Times, and all the rest of the large organizations
that rely on Django for their daily bread wouldn't consider your output
worth $30 an hour. Please read people's replies more carefully.
>> If the problem is actually that you're using a custom HTTP server, and
>> that custom HTTP server isn't providing a feature that you need (such
>> as a way to reject large uploads)
>
> Look. Should I post this to somewhere more security related on the internet?
>
Frankly, at this stage you can stick it up your ass and set fire to it
as far as I'm concerned. I like to delude myself that I am pretty
tolerant, but an ego the size of yours rubs me up the wrong way and I
start to forget my manners.
> Summary:
> In the default setup of wsgi, apache and django (on distributions like
> ubuntu and debian) by default there are no limits on the size of a
> file that an attacker can upload.
> http://cwe.mitre.org/top25/#CWE-770 and see example 2 at
> http://cwe.mitre.org/data/definitions/770.html
>
Summary:
If you are stupid enough to run a production web server on the default
setup of wsgi, apache and django when you are serving more than 5,000
impressions a day you are an idiot who will never deserve gainful
employment in the web world.
How many hits a day does your site get?
Look, get off your high horse. Stop trying to prove to everyone how
clever you are, join the community and start to seriously learn from the
many bright people who regularly contribute to this list (which is why I
hang out here. As I learn, I also try to answer a few questions,
sometimes wrongly, to take the strain off the guys who've been doing it
since I joined, and before).
If you can do that, this episode will blow over and soon you will be as
welcome as everyone else. Carry on like you are doing and people will
start to run the other way at the sight of your name.
regards
Steve
>
>
> --
> Conscience doth make cowards of us all. -- Shakespeare
>
A fool thinks himself to be wise, but a wise man knows himself to be a
fool. -- also Shakespeare
PS: Since you are such a stickler, try adding a space to the double dash
that delineates your sig block. That way sensible mailers won't try and
quote it as part of your message.
Tim Chase
> well you finish the tutorial(s) now and then you try to upload
> a file right? So you start uploading the file. Now because (I
> assume you are still using the django built in webserver) why
> don't you play with this a bit, start uploading say 10 1gb
> files(all at once) then stop them(all) at around say 700mb~
> in.
Is it just me, or will this fail to make a difference because the
devserver is single-threaded, so it will only respond to one
request at a time? ...so you can't concurrently upload 10 1gb
files...
From my testing (granted this was run against something pre-1.2
so things may have changed since then), as soon as you initiate
the first file upload, you're monopolizing the devserver process,
preventing further attempts to do the following 9 uploads until
the first has completed (successfully or aborted).
Not that you'd want to use devserver in a production environment
anyways, as Russell points out in the documentation[1]. Note the
"DO NOT USE THIS SERVER IN A PRODUCTION SETTING." It's not very
subtle about that. :)
-tkc
[1]
http://docs.djangoproject.com/en/dev/ref/django-admin/?from=olddocs#runserver-port-or-ipaddr-port
Russell Keith-Magee
Do whatever floats your boat. Graham and I have both explained the
situation. If anyone asks the same questions, we'll give them the same
answer.
> Summary:
> In the default setup of wsgi, apache and django (on distributions like
> ubuntu and debian) by default there are no limits on the size of a
> file that an attacker can upload.
> http://cwe.mitre.org/top25/#CWE-770 and see example 2 at
> http://cwe.mitre.org/data/definitions/770.html
One last try for your benefit:
If you have your Apache install configured to accept arbitrarily sized
uploads, your Apache install will accept arbitrarily sized uploads.
This should not be surprising behavior.
If Debian or Ubuntu are packaging Apache with this as the default
behavior, that's an issue for Debian and Ubuntu to manage. This
doesn't make *Django* insecure -- it makes the default install of
*Apache* insecure on those platforms. *Any* application deployed using
*any* framework would be subject to the same attack. And the attack
can be completely avoided by correctly configuring Apache.
And, for the record, the fact that Ubuntu or Debian have chosen these
defaults doesn't make Apache insecure either. System defaults exist to
make it easy and obvious to get something started. A responsible
sysadmin for a public-facing webserver shouldn't be using *any*
OS-provided defaults without auditing them. To aid that process, the
Django project is in a position to describe the sorts of issues that a
sysadmin should watch out for; hence, documenting deployment-related
security issues such as this one is in scope.
However, at the end of the day, as Graham and I have *repeatedly* told
you -- this is an issue that should be caught at the webserver, not at
the application level. Even if we weren't talking about duplicating
functionality (and we are), there are both practical and technical
reasons why it is inappropriate for Django to implement file-upload
size restrictions. This problem can be avoided with appropriate
configuration of your web server, and therefore should be.
Yours,
Russ Magee %-)
dave b
> defaults doesn't make Apache insecure either. System defaults exist to
> make it easy and obvious to get something started. A responsible
> sysadmin for a public-facing webserver shouldn't be using *any*
> OS-provided defaults without auditing them. To aid that process, the
> Django project is in a position to describe the sorts of issues that a
> sysadmin should watch out for; hence, documenting deployment-related
> security issues such as this one is in scope.
>
> However, at the end of the day, as Graham and I have *repeatedly* told
> you -- this is an issue that should be caught at the webserver, not at
> the application level. Even if we weren't talking about duplicating
> functionality (and we are), there are both practical and technical
> reasons why it is inappropriate for Django to implement file-upload
> size restrictions. This problem can be avoided with appropriate
> configuration of your web server, and therefore should be.
> If you can do that, this episode will blow over and soon you will be as
> welcome as everyone else. Carry on like you are doing and people will
> start to run the other way at the sight of your name.
Ok look. I am trying to be a nice guy here. You are making my job
harder than it should be.
Fact I reported a potential security bug, which did have the incorrect
description(sort of ;) ) at the start but then I have provided a clear
summary of the issue.
So let me tell you a story. A guy reports a bug to the django security
email, he gets a vacation reply. So he asks on irc in #django where he
should take the issue to next, he is told the django users mailing
list.
So he emails the list and they don't take him seriously. They claim
that the problem is that people shouldn't be using the default
webserver configurations. They claim the problem isn't our fault. They
also claim that the guy thinks he is smarter than others. They claim
that this guy is wrong because in the "real" world people don't use
defaults without auditing *all* of them right?
I can tell you that this guy feels really stupid right now.
Here something from the default php5 setup on debian(for apache).:
; Maximum size of POST data that PHP will accept.
post_max_size = 8M
...
; Maximum allowed size for uploaded files.
upload_max_filesize = 2M
; Maximum number of files that can be uploaded via a single request
max_file_uploads = 50
--
How apt the poor are to be proud. -- William Shakespeare, "Twelfth-Night"
Russell Keith-Magee
>> Do not pass go do not collect profit!
>> Put your hands up in the air like you just don't care!
>> blahblahblalbha sssh listen.
...
On Tue, Aug 31, 2010 at 9:42 AM, Steve Holden <hold...@gmail.com> wrote:
> Frankly, at this stage you can stick it up your ass and set fire to it
> as far as I'm concerned. I like to delude myself that I am pretty
> tolerant, but an ego the size of yours rubs me up the wrong way and I
> start to forget my manners.
Ok - before tempers get out of control, let's nip this in the bud.
I don't care if the your weapon of choice is passive-aggressive
expressions of pop culture, or detailed explorations of the
appropriate application of incendiary products to bodily orifices --
this kind of tone doesn't cast the community in a good light.
We try to maintain a civil tone here. If you don't feel that you're
able to maintain that tone, please walk away from this discussion.
Yours
Russ Magee %-)
Steve Holden
Thanks for the reminder. I apologize.
regards
Steve
> --
> You received this message because you are subscribed to the Google Groups "Django users" group.
> To post to this group, send email to django...@googlegroups.com.
> To unsubscribe from this group, send email to django-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
>
dave b
Agreed, I apologize for my part in this.
Just a heads up have sent an email to bugtraq with this little issue
outlined in it.
While you wait for it to go through moderation here are some quotes
from much smarter people than myself:
"Those who cannot learn from history are doomed to repeat it."
"History is written by the victors. "
' "No comment" is a splendid expression. I am using it again and again. '
"Given enough eyeballs, all bugs are shallow. "
--
My only love sprung from my only hate!Too early seen unknown, and
known too late! -- William Shakespeare, "Romeo and Juliet"
Russell Keith-Magee
I've already apologised for this - and I'm still investigating. The
immediate problem is that the person I need to talk to is on vacation.
Once he returns, I should be able to get the situation resolved.
> So he asks on irc in #django where he
> should take the issue to next, he is told the django users mailing
> list.
> So he emails the list and they don't take him seriously. They claim
> that the problem is that people shouldn't be using the default
> webserver configurations. They claim the problem isn't our fault. They
> also claim that the guy thinks he is smarter than others. They claim
> that this guy is wrong because in the "real" world people don't use
> defaults without auditing *all* of them right?
>
> I can tell you that this guy feels really stupid right now.
Ok - lets spin this around to our side.
A guy posts a large chunk of Django's source code to the users mailing
list, making a vague claim about some sort of file upload-based
attack. It takes him *9* follow up messages (not counting the
responses from others in the community) to narrow down that he is
reporting 2 problems -- one of which he has already been explicitly
told isn't actually a problem. He intersperses his voluminous comments
with vague pop-culturesque exclamations, makes demands that we should
stop arguing and just do what he says, provides excessive detail where
it isn't required, and insufficient detail where it is required, and
keeps making veiled references to "custom HTTP" webservers without
ever documenting which webserver he's actually concerned about.
Ultimately, the actual problem is narrowed down to "The default
install of Apache on Ubuntu allows you to upload a file of arbitrary
size". The guy is told "Apache provides a setting to control this". He
is also given a number of practical and technical reasons why the
webserver is the right place to solve the problem.
His response is to say he will escalate this to some other security
forum. We can only assume that this is a threat that he will raise
merry hell until we do what he says.
Our intention is not to make anyone feel stupid. As I've said
previously, we take security seriously. However, extraordinary claims
require extraordinary proof. When software X uses web server Y, and Y
explicitly provides settings to avoid the specific problem you're
describing, and your "attack" is predicated on those settings not
being used in your use of Y, it's hard to make the case that you've
found a security hole in X. You have, at best, found a weakness in the
default configuration of Y on a specific platform -- which is exactly
what we've told you.
As for our claim that you should be auditing the settings of the
software you use -- I'm unapologetic about that. Default values on any
platform are selected to provide maximum utility for the general case,
not maximum utility for a specific case.
> Here something from the default php5 setup on debian(for apache).:
Argumentum ad verecundiam (or possibly ad antiquitatem, depending on
how polite you're being). The fact that PHP does something doesn't
necessarily mean it's a good idea. After all, Django exists
specifically because the original developers were unhappy with the
design choices made by PHP.
Yours,
Russ Magee %-)
dave b
> forum. We can only assume that this is a threat that he will raise
> merry hell until we do what he says.
Right first: Yes I am sorry for the 9 or so posts :) I am only human.
Right. Um no that's not a threat.
That's being responsible imho, instead of just those looking at
django-users, others will now also know about the problem. Knowing
about the problem, they can apply a workaround or fix.
Do remember, fix it anyway you like, the problem still exists.
> Our intention is not to make anyone feel stupid. As I've said
> previously, we take security seriously. However, extraordinary claims
> require extraordinary proof. When software X uses web server Y, and Y
> explicitly provides settings to avoid the specific problem you're
> describing, and your "attack" is predicated on those settings not
> being used in your use of Y, it's hard to make the case that you've
> found a security hole in X. You have, at best, found a weakness in the
> default configuration of Y on a specific platform -- which is exactly
> what we've told you.
I am not about to see what the default are on other commonly used
platforms, that is a total waste of my time.
http://httpd.apache.org/docs/2.2/mod/core.html#limitrequestbody (the
default is 0).
Apparently LimitRequestBody is not touched by mod_wsgi so I assume
this means the default remains 0, unlimited.
> As for our claim that you should be auditing the settings of the
> software you use -- I'm unapologetic about that. Default values on any
> platform are selected to provide maximum utility for the general case,
> not maximum utility for a specific case.
I am going to suggest
1. this is fixed in django (through a default size limit)
and
2. wsgi sets the default LimitRequestBody to be a sane value.
As Graham Dumpleton[0] does a lot of work on mod_wsgi. So perhaps he
can introduce this into mod_wsgi.
This will mean that
1. django running in different configurations will be protected.
2. other python programs will be protected to a degree (under mod_wsgi).
Also, Graham Dumpleton keep up the good work on mod_wsgi!
[0] http://code.google.com/p/modwsgi/source/list
--
Talkers are no good doers. -- William Shakespeare, "Henry VI"
dave b
Lighttpd has a default limit of 2gb, cherokee seems to have the same.
Pin it on the httpd all you like - but the default apache has no limit
(0 - unlimited :) ).
http://httpd.apache.org/docs/2.0/mod/core.html#limitrequestbody
--
The better part of valor is discretion. -- William Shakespeare, "Henry IV"
James Bennett
> Ok no movement :)
Nor is there likely to be. Insofar as you've identified a problem at
all, it's a problem in a piece of software that isn't Django, and
you've ignored multiple people who've pointed that fact out to you
(along with the fact that Apache behaves exactly as documented and
that we expect people to know how to configure their web server to
suit their needs, use cases and threat models).
So what, precisely, are you hoping to gain from continuing this thread?
--
"Bureaucrat Conrad, you are technically correct -- the best kind of correct."
Steve Holden
An attacker could also assemble a powerful explosive device and detonate
it near enough your hosting service to take your site down. What
counter-measures are you going to take against that?
You perhaps feel people aren't taking your proposal seriously enough.
The fact of the matter is that security is never absolute, and on a
threat scale of 0 to 10 this issue comes in at about 1.7. If you are
running a professional service and you are monitoring it correctly then
you ought to notice an attack of this nature before it does too much damage.
I repeat, you may be correct in treating this as a vulnerability, but
your estimate of its seriousness appears to disagree with that of
others. If you want to have your code seriously considered for inclusion
(and why not?) you should raise it in the Django issue tracker - see
"Reporting Bugs" in
http://docs.djangoproject.com/en/1.2/internals/contributing/