If there was massive security hole found in Django, are there plans in place to deal with it?
Jason Huggins
blasted right now for not having a complete policy for releasing and
communicating urgent security flaws. I'm not poking fun, this is pretty
serious stuff.
Read here for some of the comments they're getting today via
Reddit.com:
"Mandatory security patch issued for Rails, but the vulnerability is
not disclosed. "
http://programming.reddit.com/info/cvlr/comments
Rather than poke fun or make light of the today's situation with Rails,
I'd rather like to turn the focus on the Django project and play the
"But what if it happened to you?" game.
I heard way-back-when that Django was security-audited, and found to be
secure, but that was (I think) before Django was open-sourced, and I
don't remember seeing many details.. However, doing a quick search for
"security" of the Dev list shows this message from Jacob:
http://groups.google.com/group/django-developers/msg/682543e6c084ff75
Should someone commission a new audit given the amount of code changes
in Django since last year? (It would be a great opportunity for some
company <cough Google cough ThoughtWorks cough WorldOnline /> to
sponsor.
A few questions:
1) If there was critical security flaw found in Django (any version)
today, are there plans in place on how to deal with it? If so, are
those plans posted anywhere? If not, let's roll up our sleaves and do
it! :-)
2) Is there a Django mailing list for security alerts? If not, let's
create one!
3) What are some projects to emulate in terms of "They do security
right"? I've heard Debian has good policies and execution on those
plans. Any others?
Having an emergency plan ready to go the moment a security issue crops
up is one of those things that will really ease the mind of an
"enterprise" flirting with the idea of going Django. It's also an area
where Django can really shine compared to other web frameworks (Python
or otherwise).
-Jason
Ian Holsman
Malcolm Tredinnick
[...]
> A few questions:
> 1) If there was critical security flaw found in Django (any version)
> today, are there plans in place on how to deal with it? If so, are
> those plans posted anywhere? If not, let's roll up our sleaves and do
> it! :-)
>
> 2) Is there a Django mailing list for security alerts? If not, let's
> create one!
See
http://www.djangoproject.com/documentation/contributing/#reporting-security-issues
If somebody did find a security problem, I would hope they would have
the presence of mind to look in the "how to contribute" document if they
could find no other way to contact people. Also note that the email
address there is the "obvious" one in some sense for reporting security
problems.
Finally, a little while ago somebody did report what they thought was a
security hole via Trac (an unfortunate choice, but that's the way it
goes) and it was looked at immediately.
> 3) What are some projects to emulate in terms of "They do security
> right"? I've heard Debian has good policies and execution on those
> plans. Any others?
Which policies in particular are you thinking about?
To provide some background about how distros work: Each professional
distribution (Debian, Suse, Mandrake, Red Hat, Ubuntu, Slackware,
*BSD, ...) has a security team that has representatives on the contacts
list specifically for notifying vendors of security problems in advance
of a release. It's why when you read about, e.g., an Apache release, you
also see most major distributions putting out update packages on the
same day. It isn't magic -- it's coordination and it's a very
professional group of people to work with.
> Having an emergency plan ready to go the moment a security issue crops
> up is one of those things that will really ease the mind of an
> "enterprise" flirting with the idea of going Django. It's also an area
> where Django can really shine compared to other web frameworks (Python
> or otherwise).
Revolutionary ideas aren't really needed here, since the current
system(s) work fairly well and using proven practices in cases like this
is absolutely the right thing to do. The thing that seems to have bitten
Ruby on Rails (and is applicable to us as well) is that they have a
broad installation base via grass-roots support, rather than via
distributions. Python, for example, has had security flaws in the past
(usually due to third-party libraries) and they have handled the release
and disclosure process very well.
I'm not completely sure I agree with the way the Ruby team are handling
this release, but since I don't know the details yet, I can't really
work out what is happening; they may have very good justification for
the way they are doing it, or they may just be slightly shell-shocked.
It just didn't cross my radar on any of the security-related lists I'm
on that normally cover these types of security announcements. They are
the places that this "enterprise" group watch for results. Presumably
there was a post to the Ruby-users mailing list, wherever that is and
that is the other place people are going to be watching: if you're using
something that is essentially pre-release for mission-critical work, you
are implicitly taking on the responsibility of watching carefully for
things like this.
Regards,
Malcolm
Jeremy Dunck
> I'm not completely sure I agree with the way the Ruby team are handling
> this release, but since I don't know the details yet, I can't really
> work out what is happening; they may have very good justification for
> the way they are doing it, or they may just be slightly shell-shocked.
Dave Thomas, who is heavily involved with Ruby yet not Rails-core, and
who I think has very good judgement, thinks the issue is severe enough
that they're doing the right thing to let people hear and upgrade
before full disclosure.
> something that is essentially pre-release for mission-critical work, you
> are implicitly taking on the responsibility of watching carefully for
> things like this.
True, but Rails had lots of buzz and has -lots- of prod systems. Of
the 2 people I talked to with prod rails systems, neither had heard of
this 3 hours after the posting. I only knew because of luck on
prog.reddit.
Security is freaking hard.
Jason Huggins
> See
> http://www.djangoproject.com/documentation/contributing/#reporting-security-issues
Sorry I didn't read that first before posting here. Though I did a Trac
search for "security" and that page didn't come up in the first few
search results...
Though, looking at the guidelines, I wonder how the Django devs would
know who to contact in one-on-one private emails about the
vulnerability and the fix.
> Which policies in particular are you thinking about?
I guess I'm focused on three things:
1) How do Django devs handle security issues when notified? This looks
well covered in the existing reporting-security-issues doc.
2) How should the affected users be notified? Having read the above
doc, I think this could use some more detail.
3) Is there any sort of policy or promise on how many versions back
Django devs are willing to go back and support?
The implicit assumption of using pre 1.0 code (especially open source)
is that there are no promises... But there might need to be something
more explicit like ("we'll make our best effort to support the creation
of any critical security patches for version 'X' for at least Y number
of years after initial release). In the Rails case, they first said
"upgrade to the latest version to get the patch", which made many
people unhappy. Then they back-peddaled and said they'd work on patches
for the older versions, too. Might be better to state up-front how long
Django devs expect to support various versions so people can weigh the
risks of staying on an old release later on. This is a good thing to
decide on when 1.0 ships. By comparison, the Ubuntu project is very
upfront in how they promise to support each release.
- Jason
James Bennett
> 2) How should the affected users be notified? Having read the above
> doc, I think this could use some more detail.
One would hope that anyone who's using Django is subscribed to
django-users and/or watches the Django blog (or that a company which
uses Django has a designated person doing that), because those would
be the obvious channels for notifying users that a security-related
patch/release had been issued.
> 3) Is there any sort of policy or promise on how many versions back
> Django devs are willing to go back and support?
The documentation page Malcolm linked states that patches will be
developed for the current release and the two releases previous to it.
That seems like a fairly sane policy, is roughly in line with what
some of the more popular Linux distros do.
--
"May the forces of evil become confused on the way to your house."
-- George Carlin
Jason Huggins
> True, but Rails had lots of buzz and has -lots- of prod systems. Of
> the 2 people I talked to with prod rails systems, neither had heard of
> this 3 hours after the posting. I only knew because of luck on
> prog.reddit.
Same here, programming.reddit.com is my most hit site these days...
But all the more reason for letting Django users know *before-hand*
where they should look for stuff like this (which list they should be
subscribed to or RSS feed to check). As Django user/dev, I would *not*
want to *first* hear about something like this on reddit. :-)
Maybe there should be some guideline like.. "If you're going to deploy
Django on a server accessible by the general public, subscribe to our
security RSS feed or mailing list to be notified as needed". Even then,
I can see how a policy like that is "tricky"... What's to keep an evil
blackhat from subscribing to the very same list so he he knows when to
get busy cracking sites using the same information?
-Jason
Jason Huggins
> > 3) Is there any sort of policy or promise on how many versions back
> > Django devs are willing to go back and support?
>
> The documentation page Malcolm linked states that patches will be
> developed for the current release and the two releases previous to it.
> That seems like a fairly sane policy, is roughly in line with what
> some of the more popular Linux distros do.
Ugh. May bad... I must have missed that. Right there it says, "[For
security issues, we'll] Halt all other development as long as is needed
to develop a fix, including patches against the current and two
previous releases."
Thanks for answering my questions, everyone. I'll try to read the docs
more closely next time... :-)
- Jason
James Bennett
> blackhat from subscribing to the very same list so he he knows when to
> get busy cracking sites using the same information?
I've been watching people go round and round about this in various
places today, and I have to say that I can respect the Rails team's
policy of not releasing full details of the vulnerability until after
their users have had a little time to patch. Full disclosure still
happens, but it's slightly safer for end users this way. A couple
other open-source projects I've used/been involved with have followed
a similar policy of "update ASAP, and we'll release details once
people have had time to patch", and it's caused no harm that I've
seen.
And as much as some people I've talked to have been wailing and
gnashing teeth about Rails being into Mac OS X 10.5 while Django
isn't, well, I don't envy somebody who gets shipped as part of a major
operating system when it comes time to issue security updates :)
Malcolm Tredinnick
> [...]
> And as much as some people I've talked to have been wailing and
> gnashing teeth about Rails being into Mac OS X 10.5 while Django
> isn't, well, I don't envy somebody who gets shipped as part of a major
> operating system when it comes time to issue security updates :)
This is pretty much a solved problem. It is coordinated through the
vendor security contacts lists that Ian was talking about. It happens
more often than you may realise: Apache or OpenSSL or the Linux kernel
or some other pervasive, critical component has a security hole
discovered and the release of the updates is coordinated and
simultaneous. So Apple would release the updates on the same day as
everybody else. If you do it well, you don't end up where people look at
you like Microsoft and think they can't trust the update (another
advantage of Open Source, too). Often the upstream source can supply the
patch, so the vendors need only audit it and do package rebuilds and
rush it through release QA (again, they'll often have priority paths
internally for security fixes).
Regards,
Malcolm
Ivan Sagalaev
> One would hope that anyone who's using Django is subscribed to
> django-users and/or watches the Django blog
This would be less and less true as time goes because Django will spread
beyond early adopters to a new forming local communities. For example
there is russian Django's LiveJournal community where I suspect many
people read just that list and not django-users because either they're
already happy with local list or just don't feel comfortable enough with
English to read it on a dialy basis.
si...@simon.net.nz
mailing list specifically for disclosures and point version upgrades?
This gives something for vendors etc to subscribe to and follow, and
patches etc can be announced in here before djangoproject.com or, say,
reddit.
--Simon
Jyrki Pulliainen
I second Simon on this one.
I've been an admin for different platforms and softwares for a long
time now. Propably the one of open source projects people on this list
might know is the Gallery.
What gallery has is an separate announce list. The clear advantage is
the fact that when the announce list is a read-only (eg. no discussion
on that list), all the vital information does not get lost between
other posts. Also, the announce list doesn't fill your mail account to
the limits with "How does the feature xyz work? How can I change the
background?"-posts, it only has all the vital information about new
versions and possible security flaws (which, sadly, the Gallery has a
_lot_).
So, a separate announce-list would make it easier for the end user or
vendor to filter all update and security related material related to
Django. You all propably agree with me, that catching that kind of
information from the django-users list is almost impossible.
--
Jyrki // jyrki.pu...@gmail.com
Jason Huggins
Jyrki Pulliainen wrote:
> On 8/10/06, si...@simon.net.nz <si...@simon.net.nz> wrote:
> >
> > For notification what about a low-volume django-announce group /
> > mailing list specifically for disclosures and point version upgrades?
> > This gives something for vendors etc to subscribe to and follow, and
> > patches etc can be announced in here before djangoproject.com or, say,
> > reddit.
>
> I second Simon on this one.
And I third it. :-)...
At this point, I'll leave it to the project admins to decide how to
procede. But a new "django-announce" Google group sounds like the
logicial next step.
- Jason
Adrian Holovaty
> At this point, I'll leave it to the project admins to decide how to
> procede. But a new "django-announce" Google group sounds like the
> logicial next step.
I've created the django-announce mailing list:
http://groups.google.com/group/django-announce
django-announce is a low-traffic, read-only mailing list that
distributes announcements
about the Django Web framework. Topics include new Django
releases, significant
feature additions and security alerts.
Adrian
--
Adrian Holovaty
holovaty.com | djangoproject.com
Simon Willison
> On 8/9/06, Jason Huggins <jrhu...@gmail.com> wrote:
> > I can see how a policy like that is "tricky"... What's to keep an evil
> > blackhat from subscribing to the very same list so he he knows when to
> > get busy cracking sites using the same information?
>
> I've been watching people go round and round about this in various
> places today, and I have to say that I can respect the Rails team's
> policy of not releasing full details of the vulnerability until after
> their users have had a little time to patch. Full disclosure still
> happens, but it's slightly safer for end users this way. A couple
> other open-source projects I've used/been involved with have followed
> a similar policy of "update ASAP, and we'll release details once
> people have had time to patch", and it's caused no harm that I've
> seen.
There are two principle reasons the Rails issue has gone over badly
with the community:
1. Forced upgrade to 1.1.5. This caused big problems for users on
versions of Rails older than 1.1.4 - people still using Rails 1.0 for
example. With hindsight, I think it's obvious to everyone that the
Rails team should have prepared patches for other affected versions
rather than forcing people to upgrade. Thankfull, Django security
policy already takes this in to account:
http://www.djangoproject.com/documentation/contributing/#reporting-security-issues
"Halt all other development as long as is needed to develop a fix,
including patches against the current and two previous releases."
2. The full disclosure problem. The argument against fully disclosing
the problem seems pretty solid: it's a serious exploit (it appears to
allow remote Ruby code execution) and they wanted to give Rails users a
head-start on the attackers. Here's the argument against (for those who
haven't been following along):
- The flaw can be found by running diff against 1.1.4 and 1.1.5. Any
half decent attacker could do this. The Rails team actually included a
red herring (an SQL injection unit test) to try to throw people off the
scent, but that was never going to hold back anyone with a modicum of
talent. Once one black hat had figured it out it would spread quickly
via underground mailing lists and IRC channels, leaving white hats at a
disadvantage.
- Because users didn't know what the flaw was, they couldn't evaluate
the solution to confirm that it was going to work. More importantly,
they had nothing to judge how urgent the upgrade was or if there was
some other short-term measure they could take to protect themselves.
The biggest complaints are due to a combination of the above factors.
The initial announcement stated that more versions of Rails were
affected than was actually the case. This resulted in some users
spending hours upgrading their applications, only to later find that
the version of Rails they were running was safe. This would not have
been an issue had full disclosure let them (or the community) analyse
the problem in more depth. It also wouldn't have been an issue if
patches had been released for older Rails versions.
Based on all that, I think the principle lesson from the whole thing is
that getting patches out for older releases is critically important.
Luckily this is already listed as Django policy. I doubt that full
disclosure would have been as much of an issue as it has if they had
got those patches out straight away.
Like Jeremy said, this stuff is hard.
Some good background reading on the Rails situation:
http://weblog.rubyonrails.org/2006/8/10/security-update-rails-1-0-not-affected
(the comments)
http://www.ruby-forum.com/topic/76671
Cheers,
Simon
Jason Huggins
> On 8/10/06, Jason Huggins <jrhu...@gmail.com> wrote:
> > At this point, I'll leave it to the project admins to decide how to
> > procede. But a new "django-announce" Google group sounds like the
> > logicial next step.
>
> I've created the django-announce mailing list:
Thanks, Adrian. :-)
- Jason
e
Hopefully this is not out-of-line in this thread. I am a Rails person,
not a Django person, although I have written a lot of Python in the
past. I can give you some more information about the fallout in the
rails community which might help you formulate your policy.
I agree with Simon, except that I think that close to full disclosure
would have been much better considering the circumstances.
What the problem came down to was that the patches offered by Rails
core were very shoddy. They introduced new vulnerabilities where
previously there were none, and broke a lot of applications. If they
had kept quiet about the issue for a few more days and then released
all the patches along with a small note "recommended security update",
and a non-alarmist discussion of the source of the issue, there would
not have been so much trouble. The primary cause of the problem had
been reported in the ticket system for some months already, so it
shouldn't have been a surprise. But suddenly someone in rails core
realized you could combine that with another unrelated feature to cause
data loss.
If the patches had worked, people wouldn't have minded the zero
disclosure so much. But without good patches and also without full
disclosure everyone was left high and dry. Even partial disclosure
would have helped a lot (and it was definitely a possibility, since
exploiting the flaw requires a combination of unrelated parts of the
application stack). People in situations requiring risk management had
absolutely no options, and everyone lost a lot of time.
Also, the response of some community members to those pressing for full
disclosure was very nasty, along the lines of "it worked fine on my
blog, suck it up and upgrade", "diff it yourself" (for a problem deep
inside the application stack), and "we don't want the enterprise here
anyway".
Evan
I am sure there are other sides to the story, but here is how I
perceived it.
Timeline (inexact)
5 pm EST: I was working on a plugin and the latest Rails svn co
suddenly broke some auto-loading functionality. Inspection of the
source showed sweeping changes in the routing and library loading.
7 pm: weblog.rubyonrails.org posts a hand-wringing warning:
"This is a MANDATORY upgrade for anyone not running on a very recent
edge (which isn't affected by this). If you have a public Rails site,
you MUST upgrade to Rails 1.1.5. The security issue is severe and you
do not want to be caught unpatched.The issue is in fact of such a
criticality that we're not going to dig into the specifics. No need
to arm would-be assalients."
Channel #rubyonrails on freenode goes nuts.
11 pm: incompatibilities and breakages are being widely reported with
the patch for recent Rails installs on Windows, and for applications
that use a type of scaffolder called "Engines". Also, there are many
known incompatibilities between Rails 1 and Rails 1.1.5. Response from
core seems to be "suck it up". Everyone starts manually upgrading their
old apps.
Not wanting to upgrade one of my apps from a May svn co, but obviously
not having a specific patch for that random revision, I investigate the
changes along with some others on IRC.
3 am: first reports from diff'ers willing to publicly disclose appear
on ruby-forum and on personal blogs. Accuracy increases as the night
goes on, but basically we had it right. The vulnerability seems severe
but not impossible to migitate without resorting to patches.
4 am: same people report that there is no reason to think that Rails 1
is affected. Also, the 1.1.5 patch is incomplete and apparently
introduces part of the vulnerability into previously safe Rails 1
installations.
7 am: Rails core confirms that the vulnerability does not effect 1.
7 pm: Rails core posts a new patch, 1.1.6, to fix problems in the
previous patch, as well as backports to all affected tags. Testing on
all combinations of webservers and rails versions is still going on,
and the patch might not be final. The patch does still not support
Rails Engines. They also partially verify what the actual problem is
and suggest that their hand was forced by the reports circulating on
the web.
Adrian Holovaty
> Hopefully this is not out-of-line in this thread. I am a Rails person,
> not a Django person, although I have written a lot of Python in the
> past. I can give you some more information about the fallout in the
> rails community which might help you formulate your policy.
Thanks very much for that detailed writeup, Evan! It's very helpful, indeed.
Julio Nobrega
> Even partial disclosure
> would have helped a lot (and it was definitely a possibility, since
> exploiting the flaw requires a combination of unrelated parts of the
> application stack).
For what's worth, my 0.02 cents about this part. The good thing
about partial/full disclosure is that it gives server admins more
choices, because they have more information.
I know zero about Rails, the patch, or the security hole. But I'll
be damned if there weren't several ways to mitigate the vulnerability
without applying the patches, something that doesn't happen instantly
on a production environment. From minimal servers running only RoR and
the database on another machine with a firewall filtering packages, to
Apache/Ruby permissions, logs, and backups, up to disabling some
application features while the patch is tested, leaving users without
their lovely screens for 30 minutes but preventing server damages,
there's a lot that can be done when you have more information.
I just hope that, when a security hole is found on Django (not IF.
When), we get a good overview of the problem. There are occasions when
the official patch is not even worth to apply, because it will break
features and there's another way around to fix it.
There's a lot of smart folks out there, believe in that :)
--
Julio Nobrega - http://www.inerciasensorial.com.br