DDoS update: Friday 8PM PDT
Chad Etzel
Here is the state of things as we know them:
- The DDoS attack is still ongoing, and the intensity has not
decreased at all. Because of this, interaction with the site and with
the API will continue to be shaky due to the defenses that have been
put in place by our Ops team. At this point, removing any of those
defenses is not an option.
- Whitelisted IPs that have a restricted rate-limit is a *known
issue,* and we are still working on restoring increased rate-limiting.
- OAuth funkiness is a *known issue* which seems to be exacerbated by
the whole DDoS thing.
- Automatic blacklisting of "valid" or "innocent" IPs is a *known
issue* and a result of the DDoS defenses. These blacklistings are
temporary, though the amount of time they "stick" is variant upon the
number of requests being made. The best thing to do to avoid this is
throttle back your requests. We know that this may not be an option
for everyone, but if you can, it will help.
- Keep respecting 302's as you get them.
THIS IS THE MOST IMPORTANT POINT. PLEASE READ IT TWICE:
*There is no ETA on fixing any of this*
*There is no ETA on fixing any of this*
I know that sounds harsh and cold, but if you want us to be perfectly
honest with you, that's the truth. Things will continue to be rocky as
long as this attack continues. They may get worse, they may get
better. That should not be read as "we don't care about fixing it" or
"we're not going to fix it until everything blows over" but instead as
"we can't promise when things will be back to normal, but in the
meantime we are working on fixing is ASAP."
Ops is going to be working around the clock this weekend.
We will also be monitoring the situation and giving out new
information as we have it. Please remain patient with us. As much as
you want it to be fixed, we want it to be fixed more. Some of my
personal apps are completely borked as well. We're all going to have
to ride this out together. Communications may be slowed over the
weekend, but please know that we are not ignoring the situation.
Thanks,
-Chad
Chris Corriveau
resolution telling us this helps and gives us some stuff to tell our
users. Good luck and keep the updates coming.
-----------------
Chris-
Josh Roesslein
on battling this onslaught. :)
Josh
Mike Champion
must be a challenging situation, as it helps those of us downstream
make more informed decisions.
-mike
On Aug 7, 11:34 pm, Josh Roesslein <jroessl...@gmail.com> wrote:
> Let's just hope the attackers behind this get bored and move on soon. Great
> work guys
> on battling this onslaught. :)
>
> On Fri, Aug 7, 2009 at 10:29 PM, Chris Corriveau
>
>
>
>
> > Thank you Chad appreciate this update. Even though there is no real
> > resolution telling us this helps and gives us some stuff to tell our users.
> > Good luck and keep the updates coming.
>
> > -----------------
> > Chris-
>
fastest963
Dewald Pretorius
You guys at Twitter need to realize something extremely important:
a) We support you 100%, and
b) It's these types of communications that keep temperatures down, and
enable us to keep our users informed.
So, hang in there and just keep us posted. That's all we're asking
for. And if there's anything we can do to help, just holler.
Dewald
davidzimm
We're rooting for you- not because it benefits us but because of the
in justice against you.
Michael E. Carluen
As its been already said but deserves another mention... THANK YOU for the
unsugar-coated honesty. You know what you know, and we know what you know.
I am sure you guys are working in overdrive to make these all go away. Just
keep us developers informed.
Michael
Dusty
chinaski007
Thanks for the update... keep 'em comin!
Stephen Weierman
TNTIran
you are all working on it every minute. We will hang in with you!!
Rodney Bryant
someone always wants to rain on your parade.
Cheers!
James Salsman
>
> Here is the state of things as we know them:
>
> - The DDoS attack is still ongoing, and the intensity has not
Has anyone had a close enough look at the botnet infection to deduce
the command channel traffic? For better or worse (time will tell)
there are plenty of government grey hats with wiretap-ready Narus
access who may not be able to contact you directly, but who would sure
know what to do and would be willing to do it if you could describe
the botnet command channel characteristics.
I remember not very long ago a botnet was described, by one of
Felton's students if I remember right, on some blog post, and then a
week later someone else who had captured an infection in a vm debugger
got to watch as it received a very nicely crafted command to unlink
from the host boot sequence and exit. If you want that kind of help
from the shadows, you gotta help the spooks figure out the control
channel behind the attack. Good luck, and remember it won't be long
after it passes before you can look back and laugh, so keep your chins
up!
richardhenry
100%.
Richard
Buzz
I will translate your messagein french and add it to my blog because
many users are wondering wat's going on.
Cheers
tweezy-e
I'm hear to join the chorus of support for continuing to keep us
informed and wish you well in the battle to restore service.
My services are down with 408s, and I'd love to have them back, but as
long as I know the why's and what's, that's a huge help.
Thanks and good luck. Well handled.
Steve Andaz
understand is how come its taking so long. . . . Over the past 18
months with the extensive popularity of twitter shouldnt you have
invested in the security used by other social networking sites?
Afterall,some of them were attacked and are still fully operational.
Also, rather than have some users unable to acess twitter, would it
not be more reasonable to just suspend the entire system and work on
the problem in house?
EnviroChem
DDoS attacks are not "real" security breaches of the target service
and security "fixes" won't solve the problem. A DDoS is accomplished
by infecting countless computers (e.g. unsecured home PCs) all over
the Internet and turning them into a massive zombie army. That is
then directed to barrage the target of an attack with a ceaseless
flood of requests saturating the target's ability to handle and
respond to requests. The goal is to bring down the target service and
force it offline. In this case to silence everyone who uses Twitter.
The way a DDoS has to be countered is:
1) increasing the target's resources such that it is able to handle
massively larger numbers of requests (this is countered by increasing
the size of the zombie army attacking);
2) identifying signatures of the bogus requests from the zombie army
higher upstream from the target of the attack and filtering those
requests out so that they never reach the intended target (can be
difficult to do for a service like Twitter);
3) tracking down how the zombie army is controlled and taking control
of the zombie army to shut it down;
4) getting people to properly secure their own PCs such that they
never become infected in the first place and/or taking their own
computers offline until they are fixed once they do become infected
(good luck with that).