I think "whooami" did a decent job in responding so far.
It seems to me that if there's anywhere that "we" (WP dev/hacker community) dropped the ball, it's the period of time between 12/12/2007 (when trac ticket #5313 was closed) and 2/2/2008 (when it was reopened after an exploit had been published).
Judging by the timeline illustrated by that ticket's history, in combination with the ongoing discussion in the linked support forum thread insisting that the threat was real, there is a nearly two month period of time in which potentially no one was looking at this very closely. I could be wrong on that, and maybe people were working hard behind the scenes and just not finding anything, but it seems to me (after looking at the exploit) that one of us should have been able to definitively confirm or deny that issue within the last four months.
It's easy for me to say that, since I obviously didn't take the time to look into it either, but I think maybe we should at least acknowledge this situation as a mistake and resolve to learn from it.
As to the age old debate on whether full disclosure is appropriate or not, the trac ticket history in this case again gives evidence to its "effective motivating value" (for lack of a better term): closed on 12/12 for lack of POC, reopened and quickly fixed on 2/2 when the POC code was released. Of course, he could have / should have sent the POC exploit in privately, but it's just surprising to me that none of us was able to reproduce it without the exploit in these last four months. _______________________________________________ wp-hackers mailing list wp-hack...@lists.automattic.com http://lists.automattic.com/mailman/listinfo/wp-hackers
Fwiw, the discoverer of the problem and the POC (atleast not the one thats on my site and is functional [it's actually an exploit]) didnt come from the same source. Not that that matters, but I wanted to make that clear. I wont express my frustration here since I think it's clear within the actual thread how I feel....
On Feb 2, 2008 4:16 PM, Jared Bangs <ja...@pacific22.com> wrote:
> On Feb 2, 2008 4:55 AM, chays <who...@gmail.com> wrote:
> I think "whooami" did a decent job in responding so far.
> It seems to me that if there's anywhere that "we" (WP dev/hacker > community) > dropped the ball, it's the period of time between 12/12/2007 (when trac > ticket #5313 was closed) and 2/2/2008 (when it was reopened after an > exploit > had been published).
> Judging by the timeline illustrated by that ticket's history, in > combination > with the ongoing discussion in the linked support forum thread insisting > that the threat was real, there is a nearly two month period of time in > which potentially no one was looking at this very closely. I could be > wrong > on that, and maybe people were working hard behind the scenes and just not > finding anything, but it seems to me (after looking at the exploit) that > one > of us should have been able to definitively confirm or deny that issue > within the last four months.
> It's easy for me to say that, since I obviously didn't take the time to > look > into it either, but I think maybe we should at least acknowledge this > situation as a mistake and resolve to learn from it.
> As to the age old debate on whether full disclosure is appropriate or not, > the trac ticket history in this case again gives evidence to its > "effective > motivating value" (for lack of a better term): closed on 12/12 for lack of > POC, reopened and quickly fixed on 2/2 when the POC code was released. Of > course, he could have / should have sent the POC exploit in privately, but > it's just surprising to me that none of us was able to reproduce it > without > the exploit in these last four months. > _______________________________________________ > wp-hackers mailing list > wp-hack...@lists.automattic.com > http://lists.automattic.com/mailman/listinfo/wp-hackers
> I think "whooami" did a decent job in responding so far.
> It seems to me that if there's anywhere that "we" (WP dev/hacker community) > dropped the ball, it's the period of time between 12/12/2007 (when trac > ticket #5313 was closed) and 2/2/2008 (when it was reopened after an exploit > had been published).
Jared, instead of speaking generally, could you share what evidence was overlooked?
On Feb 2, 2008 4:25 PM, Lloyd Budd <lloydomat...@gmail.com> wrote:
> On Feb 2, 2008 1:16 PM, Jared Bangs <ja...@pacific22.com> wrote:
> > It seems to me that if there's anywhere that "we" (WP dev/hacker > community) > > dropped the ball, it's the period of time between 12/12/2007 (when trac > > ticket #5313 was closed) and 2/2/2008 (when it was reopened after an > exploit > > had been published).
> Jared, instead of speaking generally, could you share what evidence > was overlooked?
> Thank you, > Lloyd
I wasn't saying we overlooked any evidence, just that we didn't follow up on it as well as we could have.
The evidence was basically just the reports of people's posts being compromised in this manner. Since they were pretty serious, I think we could have done more to either confirm or deny that there was a vulnerability that caused this to be possible. I didn't say anyone overlooked this; I was only suggesting that perhaps the issue shouldn't have been dropped as soon as it was when a cause could not originally be identified.
BTW, I'm intentionally using language like "we", etc. because I'm not intending to bash anyone or start flame wars. My simple point was that if more of "us" in the WP dev community looked more closely at this issue I believe that the root cause would have been discovered. Of course, that's easy to say in hindsight, but since there are a limited number of places in the code where a post can be modified like this (outside of SQL injection, etc.) we theoretically could have found this one if we had enough people seriously looking for it, IMHO.
Also, I think I was clear in lumping myself into the group of people who didn't put enough effort into working on this problem. I have no good excuses (other than the standard "not enough time"), but like I said, perhaps we can just learn from this and do better next time. Perhaps more of us can dedicate our time to this type of stuff instead of more "user facing" / recognizable stuff like adding more features. _______________________________________________ wp-hackers mailing list wp-hack...@lists.automattic.com http://lists.automattic.com/mailman/listinfo/wp-hackers
On Feb 2, 2008 5:39 PM, Jared Bangs <ja...@pacific22.com> wrote (and I trimmed):
> I wasn't saying we overlooked any evidence, just that we didn't follow up on > it as well as we could have.
The perception that WordPress has a poor security record is an issue close to my heart.
I'm not not certian what should have been followed up on? whooami and otto42 and others were proactive and tried to get additional information and pursue the issue. Maybe, could you provide an timeline with people's actions describing how the issue could have been pursued more proactively?
My feeling is there probably isn't many specific insights in this scenario, but you are correct there is great opportunity to contribute to WordPress' security profile.
I imagine more interesting is analysing characteristics of individual and classes of WordPress security problems to see if there are more lurking, opportunity for programmatic protection, or training.
Unfortunately, for me, I have little programming juice, and none in security.
Aside, I find http://blogsecurity.net/ awkward participation, because I don't think I've ever seen a reference to a trac ticket number in any of the posts, or updates when issues are resolved.
> My simple point was that if > more of "us" in the WP dev community looked more closely at this issue I > believe that the root cause would have been discovered. Of course, that's > easy to say in hindsight, but since there are a limited number of places in > the code where a post can be modified like this (outside of SQL injection, > etc.) we theoretically could have found this one if we had enough people > seriously looking for it, IMHO.
That is no more or less true than any other exploit discovered or yet to be discovered. There is only a short list of goals of compromising a system.
> Perhaps more of us can dedicate our time to this type of stuff instead of more > "user facing" / recognizable stuff like adding more features.
I don't think there is any excess of people working on "user facing" stuff either unfortunately.
Are there specific things that you are now working on related to this now?
this, posted to the forum thread by Matt, and followed by my being banned from the forums "whoami, your fix does not. I would rather not have people think they're safe and really not be, and there is a release coming shortly anyway. If you'd like to post more to this thread please reply to the email I sent you this morning. If anyone is scared and wants a fix NOW, they should either turn off registration (which is off by default) or delete xmlrpc.php.
1. the fix isnt mine. It's securiteam's. 2. Its tested, and yes, it does work, for the exploit provided. 3. I never got an e-mail you from this morning and I'll stop short of calling you out on that fact except to say that I was at my computer most of the day, and would NOT have missed an e-mail from you had I received it.
I think your apparent anger is a little misdirected Matt , especially given all the dates that have been tossed about regarding this.
whoo
On Feb 2, 2008 9:31 PM, Lloyd Budd <lloydomat...@gmail.com> wrote:
> On Feb 2, 2008 5:39 PM, Jared Bangs <ja...@pacific22.com> wrote (and I > trimmed):
> > I wasn't saying we overlooked any evidence, just that we didn't follow > up on > > it as well as we could have.
> The perception that WordPress has a poor security record is an issue > close to my heart.
> I'm not not certian what should have been followed up on? whooami and > otto42 and others were proactive and tried to get additional > information and pursue the issue. Maybe, could you provide an timeline > with people's actions describing how the issue could have been pursued > more proactively?
> My feeling is there probably isn't many specific insights in this > scenario, but you are correct there is great opportunity to contribute > to WordPress' security profile.
> I imagine more interesting is analysing characteristics of individual > and classes of WordPress security problems to see if there are more > lurking, opportunity for programmatic protection, or training.
> Unfortunately, for me, I have little programming juice, and none in > security.
> Aside, I find http://blogsecurity.net/ awkward participation, because > I don't think I've ever seen a reference to a trac ticket number in > any of the posts, or updates when issues are resolved.
> > My simple point was that if > > more of "us" in the WP dev community looked more closely at this issue I > > believe that the root cause would have been discovered. Of course, > that's > > easy to say in hindsight, but since there are a limited number of places > in > > the code where a post can be modified like this (outside of SQL > injection, > > etc.) we theoretically could have found this one if we had enough people > > seriously looking for it, IMHO.
> That is no more or less true than any other exploit discovered or yet > to be discovered. There is only a short list of goals of compromising > a system.
> > Perhaps more of us can dedicate our time to this type of stuff instead > of more > > "user facing" / recognizable stuff like adding more features.
> I don't think there is any excess of people working on "user facing" > stuff either unfortunately.
> Are there specific things that you are now working on related to this now?
chays wrote: > 1. the fix isnt mine. It's securiteam's.
Okay, well then "Paul (Yabba) Jones" is wrong and it's still not recommended and shouldn't be promoted. I recommended two temporary fixes that work.
> 2. Its tested, and yes, it does work, for the exploit provided.
Yes but we think there's a different issue it doesn't address and it's generally bad policy to have people mucking around editing a huge PHP file. They should do one of the fixes above, both non-harmful, or wait for the release.
> 3. I never got an e-mail you from this morning and I'll stop short of > calling you out on that fact except to say that I was at my computer most of > the day, and would NOT have missed an e-mail from you had I received it.
My mistake, it was sent to the user "whoami" not "whooami".
> I think your apparent anger is a little misdirected Matt , especially given > all the dates that have been tossed about regarding this.
I'm not angry, just trying to resolve the issue quickly and safely.
On Feb 2, 2008 6:31 PM, Lloyd Budd <lloydomat...@gmail.com> wrote:
> On Feb 2, 2008 5:39 PM, Jared Bangs <ja...@pacific22.com> wrote (and I > trimmed):
> > I wasn't saying we overlooked any evidence, just that we didn't follow > up on > > it as well as we could have.
> The perception that WordPress has a poor security record is an issue > close to my heart.
I didn't mention anything about that, outside of the context of this particular issue. I didn't mean to imply that anyone on this list (including you) don't care about security problems (only encouraging that we could all do better); sorry if it came across that way.
> I'm not not certian what should have been followed up on? whooami and > otto42 and others were proactive and tried to get additional > information and pursue the issue. Maybe, could you provide an timeline > with people's actions describing how the issue could have been pursued > more proactively?
I'm really not interested in making this an argument at all, so I'd rather not do a full on reconstruction of the events here. You may disagree with my view on this, and I certainly respect your opinion.
To keep it as short as possible: the original report may have been lacking technical details (which is understandable for most users). Lots of people responded, asking good questions to try to get to the heart of the matter. Lots more people chimed in stating that they had been hit as well.
"rawalex" posted one month ago pointing directly to xml-rpc as the cause of the problem. Between that point and now is primarily where I feel that we could have found this, if enough people were concerned.
I believe that the fact that we didn't until an exploit was finally published (even though it has apparently been in "private" use for months) may send a message to some that disclosure is what it takes to get moving on this type of issue.
> My feeling is there probably isn't many specific insights in this > scenario, but you are correct there is great opportunity to contribute > to WordPress' security profile.
> I imagine more interesting is analysing characteristics of individual > and classes of WordPress security problems to see if there are more > lurking, opportunity for programmatic protection, or training.
I couldn't agree more. This is what I was alluding to earlier in my closing remark about new features usually taking a much more prominent focus.
> Unfortunately, for me, I have little programming juice, and none in > security.
<LightHeartedJoke>Maybe with the latest round of funding, Automattic can invest in a couple full time security oriented "hackers" to hammer on it and try to flush this stuff out.</LightHeartedJoke>
> I don't think I've ever seen a reference to a trac ticket number in > any of the posts, or updates when issues are resolved.
> > My simple point was that if > > more of "us" in the WP dev community looked more closely at this issue I > > believe that the root cause would have been discovered. Of course, > that's > > easy to say in hindsight, but since there are a limited number of places > in > > the code where a post can be modified like this (outside of SQL > injection, > > etc.) we theoretically could have found this one if we had enough people > > seriously looking for it, IMHO.
> That is no more or less true than any other exploit discovered or yet > to be discovered. There is only a short list of goals of compromising > a system.
This is where I disagree. I think we had enough info in this case (see my comment above about rawlex's post from a month ago) to make this different than just a random 0-day security breach that we couldn't be expected to prepare for.
> > Perhaps more of us can dedicate our time to this type of stuff instead > of more > > "user facing" / recognizable stuff like adding more features.
> I don't think there is any excess of people working on "user facing" > stuff either unfortunately.
Perhaps, but even as a topic of conversation (on this list) it doesn't seem to come up much, outside of the context of addressing a specific known vulnerability. I suppose it's like that with most open source projects, though, and I certainly don't have an answer for how to change that.
> Are there specific things that you are now working on related to this now?
As time permits, but unfortunately it usually doesn't. Perhaps in the eyes of some, that disqualifies my statements here, and I recognize that. That's why I was sure to include myself in the list of people who could have done more on this issue and didn't.
But since you asked, the last thing I did regarding security was submitting a patch to WPMU ticket #528, which ported over Ryan's password salting and cookie authentication fixes from the standard WP trunk. It didn't get commited, and the ticket was closed with an indication that it would wait until the 2.5 sync. I'll also note that I didn't come back and whine about it (here or elsewhere) or get into a close/reopen battle on Trac.
I do really believe that those security related changes were important (arguably moreso with MU since there are likely to be more registered users per install), but if the commiters and/or enough users disagree, I'm not going to spend any time arguing about it. I barely had the free time to write it, and definitely don't have the excess time to debate and campaign for it.
Lastly, I got the feeling that you (perhaps on behalf of the project as a whole) were feeling attacked or criticized. Let me clarify again that that was not the intent of my comments. My only hope is to motivate others here in the community to do more than we have been regarding security, especially in cases where there are multiple reports of active exploits with strong hints as to the source of the problem. _______________________________________________ wp-hackers mailing list wp-hack...@lists.automattic.com http://lists.automattic.com/mailman/listinfo/wp-hackers
On Sat, 2008-02-02 at 17:39 -0800, Jared Bangs wrote: > The evidence was basically just the reports of people's posts being > compromised in this manner. Since they were pretty serious, I think we could > have done more to either confirm or deny that there was a vulnerability that > caused this to be possible. I didn't say anyone overlooked this; I was only > suggesting that perhaps the issue shouldn't have been dropped as soon as it > was when a cause could not originally be identified.
I feel I should chime in as the original owner of that trac ticket. I hope this is evidence that I was taking the issue seriously and wanted to work to fix it.
At the time the ticket was closed there was no evidence that what was being seen was anything anything more than people noticing a past exploitation through a known, fixed, issue. No one could provide even a rough guess as to when their site was exploited, only when they had noticed it. There was no evidence as to the vector the exploit used.
No one doubted that users were experiencing problems, one of my own installations had been exploited. My logs didn't go back far enough to pin point when so I couldn't confirm an issue with the current release. I extended my logging across all my installations with the hope of catching it again. I've been monitoring my logs over the past month.
I didn't see anything wrong with closing the ticket as until actual evidence was found, there was little to do beyond stare at the code and hope for enlightenment.
