I think "whooami" did a decent job in responding so far.
It seems to me that if there's anywhere that "we" (WP dev/hacker community) dropped the ball, it's the period of time between 12/12/2007 (when trac ticket #5313 was closed) and 2/2/2008 (when it was reopened after an exploit had been published).
Judging by the timeline illustrated by that ticket's history, in combination with the ongoing discussion in the linked support forum thread insisting that the threat was real, there is a nearly two month period of time in which potentially no one was looking at this very closely. I could be wrong on that, and maybe people were working hard behind the scenes and just not finding anything, but it seems to me (after looking at the exploit) that one of us should have been able to definitively confirm or deny that issue within the last four months.
It's easy for me to say that, since I obviously didn't take the time to look into it either, but I think maybe we should at least acknowledge this situation as a mistake and resolve to learn from it.
As to the age old debate on whether full disclosure is appropriate or not, the trac ticket history in this case again gives evidence to its "effective motivating value" (for lack of a better term): closed on 12/12 for lack of POC, reopened and quickly fixed on 2/2 when the POC code was released. Of course, he could have / should have sent the POC exploit in privately, but it's just surprising to me that none of us was able to reproduce it without the exploit in these last four months. _______________________________________________ wp-hackers mailing list wp-hack...@lists.automattic.com http://lists.automattic.com/mailman/listinfo/wp-hackers
Fwiw, the discoverer of the problem and the POC (atleast not the one thats on my site and is functional [it's actually an exploit]) didnt come from the same source. Not that that matters, but I wanted to make that clear. I wont express my frustration here since I think it's clear within the actual thread how I feel....
On Feb 2, 2008 4:16 PM, Jared Bangs <ja...@pacific22.com> wrote:
> On Feb 2, 2008 4:55 AM, chays <who...@gmail.com> wrote:
> I think "whooami" did a decent job in responding so far.
> It seems to me that if there's anywhere that "we" (WP dev/hacker > community) > dropped the ball, it's the period of time between 12/12/2007 (when trac > ticket #5313 was closed) and 2/2/2008 (when it was reopened after an > exploit > had been published).
> Judging by the timeline illustrated by that ticket's history, in > combination > with the ongoing discussion in the linked support forum thread insisting > that the threat was real, there is a nearly two month period of time in > which potentially no one was looking at this very closely. I could be > wrong > on that, and maybe people were working hard behind the scenes and just not > finding anything, but it seems to me (after looking at the exploit) that > one > of us should have been able to definitively confirm or deny that issue > within the last four months.
> It's easy for me to say that, since I obviously didn't take the time to > look > into it either, but I think maybe we should at least acknowledge this > situation as a mistake and resolve to learn from it.
> As to the age old debate on whether full disclosure is appropriate or not, > the trac ticket history in this case again gives evidence to its > "effective > motivating value" (for lack of a better term): closed on 12/12 for lack of > POC, reopened and quickly fixed on 2/2 when the POC code was released. Of > course, he could have / should have sent the POC exploit in privately, but > it's just surprising to me that none of us was able to reproduce it > without > the exploit in these last four months. > _______________________________________________ > wp-hackers mailing list > wp-hack...@lists.automattic.com > http://lists.automattic.com/mailman/listinfo/wp-hackers
> I think "whooami" did a decent job in responding so far.
> It seems to me that if there's anywhere that "we" (WP dev/hacker community) > dropped the ball, it's the period of time between 12/12/2007 (when trac > ticket #5313 was closed) and 2/2/2008 (when it was reopened after an exploit > had been published).
Jared, instead of speaking generally, could you share what evidence was overlooked?
On Feb 2, 2008 4:25 PM, Lloyd Budd <lloydomat...@gmail.com> wrote:
> On Feb 2, 2008 1:16 PM, Jared Bangs <ja...@pacific22.com> wrote:
> > It seems to me that if there's anywhere that "we" (WP dev/hacker > community) > > dropped the ball, it's the period of time between 12/12/2007 (when trac > > ticket #5313 was closed) and 2/2/2008 (when it was reopened after an > exploit > > had been published).
> Jared, instead of speaking generally, could you share what evidence > was overlooked?
> Thank you, > Lloyd
I wasn't saying we overlooked any evidence, just that we didn't follow up on it as well as we could have.
The evidence was basically just the reports of people's posts being compromised in this manner. Since they were pretty serious, I think we could have done more to either confirm or deny that there was a vulnerability that caused this to be possible. I didn't say anyone overlooked this; I was only suggesting that perhaps the issue shouldn't have been dropped as soon as it was when a cause could not originally be identified.
BTW, I'm intentionally using language like "we", etc. because I'm not intending to bash anyone or start flame wars. My simple point was that if more of "us" in the WP dev community looked more closely at this issue I believe that the root cause would have been discovered. Of course, that's easy to say in hindsight, but since there are a limited number of places in the code where a post can be modified like this (outside of SQL injection, etc.) we theoretically could have found this one if we had enough people seriously looking for it, IMHO.
Also, I think I was clear in lumping myself into the group of people who didn't put enough effort into working on this problem. I have no good excuses (other than the standard "not enough time"), but like I said, perhaps we can just learn from this and do better next time. Perhaps more of us can dedicate our time to this type of stuff instead of more "user facing" / recognizable stuff like adding more features. _______________________________________________ wp-hackers mailing list wp-hack...@lists.automattic.com http://lists.automattic.com/mailman/listinfo/wp-hackers
On Feb 2, 2008 5:39 PM, Jared Bangs <ja...@pacific22.com> wrote (and I trimmed):
> I wasn't saying we overlooked any evidence, just that we didn't follow up on > it as well as we could have.
The perception that WordPress has a poor security record is an issue close to my heart.
I'm not not certian what should have been followed up on? whooami and otto42 and others were proactive and tried to get additional information and pursue the issue. Maybe, could you provide an timeline with people's actions describing how the issue could have been pursued more proactively?
My feeling is there probably isn't many specific insights in this scenario, but you are correct there is great opportunity to contribute to WordPress' security profile.
I imagine more interesting is analysing characteristics of individual and classes of WordPress security problems to see if there are more lurking, opportunity for programmatic protection, or training.
Unfortunately, for me, I have little programming juice, and none in security.
Aside, I find http://blogsecurity.net/ awkward participation, because I don't think I've ever seen a reference to a trac ticket number in any of the posts, or updates when issues are resolved.
> My simple point was that if > more of "us" in the WP dev community looked more closely at this issue I > believe that the root cause would have been discovered. Of course, that's > easy to say in hindsight, but since there are a limited number of places in > the code where a post can be modified like this (outside of SQL injection, > etc.) we theoretically could have found this one if we had enough people > seriously looking for it, IMHO.
That is no more or less true than any other exploit discovered or yet to be discovered. There is only a short list of goals of compromising a system.
> Perhaps more of us can dedicate our time to this type of stuff instead of more > "user facing" / recognizable stuff like adding more features.
I don't think there is any excess of people working on "user facing" stuff either unfortunately.
Are there specific things that you are now working on related to this now?
this, posted to the forum thread by Matt, and followed by my being banned from the forums "whoami, your fix does not. I would rather not have people think they're safe and really not be, and there is a release coming shortly anyway. If you'd like to post more to this thread please reply to the email I sent you this morning. If anyone is scared and wants a fix NOW, they should either turn off registration (which is off by default) or delete xmlrpc.php.
1. the fix isnt mine. It's securiteam's. 2. Its tested, and yes, it does work, for the exploit provided. 3. I never got an e-mail you from this morning and I'll stop short of calling you out on that fact except to say that I was at my computer most of the day, and would NOT have missed an e-mail from you had I received it.
I think your apparent anger is a little misdirected Matt , especially given all the dates that have been tossed about regarding this.
whoo
On Feb 2, 2008 9:31 PM, Lloyd Budd <lloydomat...@gmail.com> wrote:
> On Feb 2, 2008 5:39 PM, Jared Bangs <ja...@pacific22.com> wrote (and I > trimmed):
> > I wasn't saying we overlooked any evidence, just that we didn't follow > up on > > it as well as we could have.
> The perception that WordPress has a poor security record is an issue > close to my heart.
> I'm not not certian what should have been followed up on? whooami and > otto42 and others were proactive and tried to get additional > information and pursue the issue. Maybe, could you provide an timeline > with people's actions describing how the issue could have been pursued > more proactively?
> My feeling is there probably isn't many specific insights in this > scenario, but you are correct there is great opportunity to contribute > to WordPress' security profile.
> I imagine more interesting is analysing characteristics of individual > and classes of WordPress security problems to see if there are more > lurking, opportunity for programmatic protection, or training.
> Unfortunately, for me, I have little programming juice, and none in > security.
> Aside, I find http://blogsecurity.net/ awkward participation, because > I don't think I've ever seen a reference to a trac ticket number in > any of the posts, or updates when issues are resolved.
> > My simple point was that if > > more of "us" in the WP dev community looked more closely at this issue I > > believe that the root cause would have been discovered. Of course, > that's > > easy to say in hindsight, but since there are a limited number of places > in > > the code where a post can be modified like this (outside of SQL > injection, > > etc.) we theoretically could have found this one if we had enough people > > seriously looking for it, IMHO.
> That is no more or less true than any other exploit discovered or yet > to be discovered. There is only a short list of goals of compromising > a system.
> > Perhaps more of us can dedicate our time to this type of stuff instead > of more > > "user facing" / recognizable stuff like adding more features.
> I don't think there is any excess of people working on "user facing" > stuff either unfortunately.
> Are there specific things that you are now working on related to this now?
chays wrote: > 1. the fix isnt mine. It's securiteam's.
Okay, well then "Paul (Yabba) Jones" is wrong and it's still not recommended and shouldn't be promoted. I recommended two temporary fixes that work.
> 2. Its tested, and yes, it does work, for the exploit provided.
Yes but we think there's a different issue it doesn't address and it's generally bad policy to have people mucking around editing a huge PHP file. They should do one of the fixes above, both non-harmful, or wait for the release.
> 3. I never got an e-mail you from this morning and I'll stop short of > calling you out on that fact except to say that I was at my computer most of > the day, and would NOT have missed an e-mail from you had I received it.
My mistake, it was sent to the user "whoami" not "whooami".
> I think your apparent anger is a little misdirected Matt , especially given > all the dates that have been tossed about regarding this.
I'm not angry, just trying to resolve the issue quickly and safely.
On Feb 2, 2008 6:31 PM, Lloyd Budd <lloydomat...@gmail.com> wrote:
> On Feb 2, 2008 5:39 PM, Jared Bangs <ja...@pacific22.com> wrote (and I > trimmed):
> > I wasn't saying we overlooked any evidence, just that we didn't follow > up on > > it as well as we could have.
> The perception that WordPress has a poor security record is an issue > close to my heart.
I didn't mention anything about that, outside of the context of this particular issue. I didn't mean to imply that anyone on this list (including you) don't care about security problems (only encouraging that we could all do better); sorry if it came across that way.
> I'm not not certian what should have been followed up on? whooami and > otto42 and others were proactive and tried to get additional > information and pursue the issue. Maybe, could you provide an timeline > with people's actions describing how the issue could have been pursued > more proactively?
I'm really not interested in making this an argument at all, so I'd rather not do a full on reconstruction of the events here. You may disagree with my view on this, and I certainly respect your opinion.
To keep it as short as possible: the original report may have been lacking technical details (which is understandable for most users). Lots of people responded, asking good questions to try to get to the heart of the matter. Lots more people chimed in stating that they had been hit as well.
"rawalex" posted one month ago pointing directly to xml-rpc as the cause of the problem. Between that point and now is primarily where I feel that we could have found this, if enough people were concerned.
I believe that the fact that we didn't until an exploit was finally published (even though it has apparently been in "private" use for months) may send a message to some that disclosure is what it takes to get moving on this type of issue.
> My feeling is there probably isn't many specific insights in this > scenario, but you are correct there is great opportunity to contribute > to WordPress' security profile.
> I imagine more interesting is analysing characteristics of individual > and classes of WordPress security problems to see if there are more > lurking, opportunity for programmatic protection, or training.
I couldn't agree more. This is what I was alluding to earlier in my closing remark about new features usually taking a much more prominent focus.
> Unfortunately, for me, I have little programming juice, and none in > security.
<LightHeartedJoke>Maybe with the latest round of funding, Automattic can invest in a couple full time security oriented "hackers" to hammer on it and try to flush this stuff out.</LightHeartedJoke>
> I don't think I've ever seen a reference to a trac ticket number in > any of the posts, or updates when issues are resolved.
> > My simple point was that if > > more of "us" in the WP dev community looked more closely at this issue I > > believe that the root cause would have been discovered. Of course, > that's > > easy to say in hindsight, but since there are a limited number of places > in > > the code where a post can be modified like this (outside of SQL > injection, > > etc.) we theoretically could have found this one if we had enough people > > seriously looking for it, IMHO.
> That is no more or less true than any other exploit discovered or yet > to be discovered. There is only a short list of goals of compromising > a system.
This is where I disagree. I think we had enough info in this case (see my comment above about rawlex's post from a month ago) to make this different than just a random 0-day security breach that we couldn't be expected to prepare for.
> > Perhaps more of us can dedicate our time to this type of stuff instead > of more > > "user facing" / recognizable stuff like adding more features.
> I don't think there is any excess of people working on "user facing" > stuff either unfortunately.
Perhaps, but even as a topic of conversation (on this list) it doesn't seem to come up much, outside of the context of addressing a specific known vulnerability. I suppose it's like that with most open source projects, though, and I certainly don't have an answer for how to change that.
> Are there specific things that you are now working on related to this now?
As time permits, but unfortunately it usually doesn't. Perhaps in the eyes of some, that disqualifies my statements here, and I recognize that. That's why I was sure to include myself in the list of people who could have done more on this issue and didn't.
But since you asked, the last thing I did regarding security was submitting a patch to WPMU ticket #528, which ported over Ryan's password salting and cookie authentication fixes from the standard WP trunk. It didn't get commited, and the ticket was closed with an indication that it would wait until the 2.5 sync. I'll also note that I didn't come back and whine about it (here or elsewhere) or get into a close/reopen battle on Trac.
I do really believe that those security related changes were important (arguably moreso with MU since there are likely to be more registered users per install), but if the commiters and/or enough users disagree, I'm not going to spend any time arguing about it. I barely had the free time to write it, and definitely don't have the excess time to debate and campaign for it.
Lastly, I got the feeling that you (perhaps on behalf of the project as a whole) were feeling attacked or criticized. Let me clarify again that that was not the intent of my comments. My only hope is to motivate others here in the community to do more than we have been regarding security, especially in cases where there are multiple reports of active exploits with strong hints as to the source of the problem. _______________________________________________ wp-hackers mailing list wp-hack...@lists.automattic.com http://lists.automattic.com/mailman/listinfo/wp-hackers
On Sat, 2008-02-02 at 17:39 -0800, Jared Bangs wrote: > The evidence was basically just the reports of people's posts being > compromised in this manner. Since they were pretty serious, I think we could > have done more to either confirm or deny that there was a vulnerability that > caused this to be possible. I didn't say anyone overlooked this; I was only > suggesting that perhaps the issue shouldn't have been dropped as soon as it > was when a cause could not originally be identified.
I feel I should chime in as the original owner of that trac ticket. I hope this is evidence that I was taking the issue seriously and wanted to work to fix it.
At the time the ticket was closed there was no evidence that what was being seen was anything anything more than people noticing a past exploitation through a known, fixed, issue. No one could provide even a rough guess as to when their site was exploited, only when they had noticed it. There was no evidence as to the vector the exploit used.
No one doubted that users were experiencing problems, one of my own installations had been exploited. My logs didn't go back far enough to pin point when so I couldn't confirm an issue with the current release. I extended my logging across all my installations with the hope of catching it again. I've been monitoring my logs over the past month.
I didn't see anything wrong with closing the ticket as until actual evidence was found, there was little to do beyond stare at the code and hope for enlightenment.
Jared, I disagree that there is evidence that anyone "dropped the ball", and assuming that was the case, I don't see solutions being presented.
"rawalex" did provide some great clues, and I hope that he sent that information and all the details he had to secur...@wordpress.org.
If we look at the log of xmlrpc.php, we will see a number of security improvements around the time: http://trac.wordpress.org/log/trunk/xmlrpc.php . Clearly an exploit wasn't identified, or there would have been a release, but opportunities to harden the code were found.
http://trac.mu.wordpress.org/ticket/528 is an awesome example of participating Jared! And I know first hand how hard it is to find the time, and how frustrating it is when the same priority I give to an issue isn't shared by others. The original ticket by "drmike" isn't a good one. No ones mother knows "how easy it is to lift a password hash in wordpress", because the first requirement is getting access to or a copy of the database, which if happens your mother is already really upset with you. It is still a very important issue, and the collaboration that has taken place around resolving the issue is brilliant!
I long ago learned that if you don't have the time to champion an issue, there is no point in reporting it, and even less in providing a patch. The exception is the bug whose stink has be smelt in the next room. You seem to be suggesting that donncha disagreed with the importance of this issue, but I don't think he wrote that or thinks that. He wrote "it's such an invasive change to the users table it's better to wait until it's 100% reliable." Donncha's belief that the fix needed more venting is sound, and a number of fixes to it have been made since then.
Maybe becomes I'm sensitive to it after being the QA owner for security issues on Netscape 8 and then Flock, but the perception that WordPress has a poor security record is an issue close to my heart. I say "perception" as I don't have the expertise in PHP or web apps of this nature to know how founded it is, but it still hurts and I hope people with the expertise and passion will shape the path to changing that perception (it is safe to assume that the core participants write more secure code today than yesterday, as they have been to the school of hard knocks, and take the security issues personally).
You can lead the horses to water, but if they aren't as thirsty as you they won't drink. Or, actions inspire more than words, and mostly I just got words, Lloyd _______________________________________________ wp-hackers mailing list wp-hack...@lists.automattic.com http://lists.automattic.com/mailman/listinfo/wp-hackers
> I feel I should chime in as the original owner of that trac ticket. I > hope this is evidence that I was taking the issue seriously and wanted > to work to fix it.
> At the time the ticket was closed there was no evidence that what was > being seen was anything anything more than people noticing a past > exploitation through a known, fixed, issue. No one could provide even a > rough guess as to when their site was exploited, only when they had > noticed it. There was no evidence as to the vector the exploit used.
> No one doubted that users were experiencing problems, one of my own > installations had been exploited. My logs didn't go back far enough to > pin point when so I couldn't confirm an issue with the current release. > I extended my logging across all my installations with the hope of > catching it again. I've been monitoring my logs over the past month.
> I didn't see anything wrong with closing the ticket as until actual > evidence was found, there was little to do beyond stare at the code and > hope for enlightenment.
I get your point, in that since no one was able to definitively point to a specific time & version when their installations had been compromised there was no solid proof yet that the very latest version still contained the vulnerability. I'd still disagree with closing the ticket so quickly under those circumstances, but I respect your differing opinion. _______________________________________________ wp-hackers mailing list wp-hack...@lists.automattic.com http://lists.automattic.com/mailman/listinfo/wp-hackers
On Feb 3, 2008 9:15 AM, Lloyd Budd <lloydomat...@gmail.com> wrote:
> Jared, I disagree that there is evidence that anyone "dropped the > ball", and assuming that was the case, I don't see solutions being > presented.
> "rawalex" did provide some great clues, and I hope that he sent that > information and all the details he had to secur...@wordpress.org.
> <trim>
> You can lead the horses to water, but if they aren't as thirsty as you > they won't drink. Or, actions inspire more than words, and mostly I > just got words,
OK, my only point in my comments (that I keep repeating) is that in general I think that we (including myself) could do better when it comes to proactively hardening security in WordPress.
I'm not sure what else I can personally do in terms of "presenting solutions" other than participating in these discussions in hopes of encouraging people to that end, and contributing patches when I'm able, but I'm open to suggestions.
If you disagree with my point (that we could do better in this area) that's fine. You certainly invest a lot more time in it than I do, so perhaps you're right. Either way, I'd rather not argue about it, since I don't think that will be productive. I appreciate your responses, and I respect your point of view.