OK Seriously People

[Joe Bowman]

Disclaimer: I'm not affiliated with Twitter in any way, other than as
a consumer of their API and a user of their product. Currently, my app
running on appengine is having the same difficulty as many of you.

There's a lot of false statements and accusations being made against
the Twitter folk right now. These are being made by unhappy people
who's products that rely on Twitter aren't working, and don't
understand what's going on.

First off, to people stating that Twitter Ops needs to work 30 hour
shifts, and any ops person who hasn't, isn't a real ops person. A real
ops person knows that after about 15 hours their mental capacity to
solve problems begins to deteriorate and they have to rest in order to
not cause further damage to their organization. Maybe in the little
mom and pop shops where you run the show you're stuck with those 30
hour marathons, but in larger shops you have team members, and you
schedule with them.

Also, for a lot of what's going, there is likely nothing Twitter can
do. Most of the packets that are being filtered are more than likely
being filtered upstream from Twitter, by their provider(s). This DDoS
will also be having an impact on their provider, who will be taking
actions themselves. Since they don't exist primarily just to serve
Twitter, they are going to be taking action in the best interest of
all their customers, which means whitelisting IPs just because Twitter
says so probably isn't going to happen "like that". Those of us using
cloud applications, welcome to the first incident showing how we can
be a victim of our choice to use the cloud. For all we know, the
attacks are coming from the cloud we are a member of.

Twitter has done very well with communication on this issue. The only
suggestion I can make is to sticky their most recent updates in the
hopes of more people reading them. They've had updates every day.
That's a lot.

The only fault I can really see with Twitter is that they haven't
scaled out their infrastructure to better handle an event like this.
They really should look at what Google and Facebook (who both survived
the attacks better) are doing and work towards building something
similar. As I'm completely ignorant of what their infrastructure looks
like, I won't start offering suggestions on what to do. I've never had
to support something the scale of Twitter, but I'm sure their Ops
department are already working on ideas, because that's what ops does.
They solve problems and make sure they don't happen again.

Yes, it is frustrating that this problem still exists, and we all
would have liked for Twitter to be better prepared. IMHO they are a
victim of their own success and growth, and scaling to meet demand
became more of a priority than scaling to manage their demand. It
happens all the time. But give their ops a break. They are
communicating, they are working on the issues they can work on, and
the real problem is whoever is behind the attack in the first place.
Bashing Twitter does nothing but make people who are tired,
frustrated, and working extremely hard feel like crap because you want
to whine and point fingers when you have no idea what you're talking
about.

[Stuart]

Too bloody right!

-Stuart

[Cameron Kaiser]

> There's a lot of false statements and accusations being made against
> the Twitter folk right now. These are being made by unhappy people
> who's products that rely on Twitter aren't working, and don't
> understand what's going on.

Well put.

--
------------------------------------ personal: http://www.cameronkaiser.com/ --
Cameron Kaiser * Floodgap Systems * www.floodgap.com * cka...@floodgap.com
-- Po-Ching Lives! ------------------------------------------------------------

[Paul McDonald]

VERY well put, my app is as affected as everyone (UberTwitter) and
many of my users are blaming my app. That said, I'm very satisfied
at the way Twitter has handled this.

-Paul

On Aug 9, 10:14 am, Cameron Kaiser <spec...@floodgap.com> wrote:
> > There's a lot of false statements and accusations being made against
> > the Twitter folk right now. These are being made by unhappy people
> > who's products that rely on Twitter aren't working, and don't
> > understand what's going on.
>
> Well put.
>
> --
> ------------------------------------ personal:http://www.cameronkaiser.com/--

>   Cameron Kaiser * Floodgap Systems *www.floodgap.com* ckai...@floodgap.com
> -- Po-Ching Lives! ------------------------------------------------------------

[Dossy Shiobara]

On 8/9/09 10:00 AM, Joe Bowman wrote:
> First off, to people stating that Twitter Ops needs to work 30 hour
> shifts, and any ops person who hasn't, isn't a real ops person. A real
> ops person knows that after about 15 hours their mental capacity to
> solve problems begins to deteriorate and they have to rest in order to
> not cause further damage to their organization. Maybe in the little
> mom and pop shops where you run the show you're stuck with those 30
> hour marathons, but in larger shops you have team members, and you
> schedule with them.

"People" didn't state it. I did.

I've worked in Ops in small 3-person companies as well as for 5,000+
employee companies with an IT team of 60+ people. In both situations,
there have been exceptional times where yes, I've taken a 30 minute nap
on the floor to recharge through a 30+ hour marathon emergency situation.

I'm not suggesting that these people don't need to sleep (unless they
have access to some Provigil and/or Adderall), but if Twitter isn't
treating this outage as a #1 priority - and I mean literally, not
figuratively - then it's a clear message to us in the third-party
ecosystem that we'd better not make them our primary focus because we
can't rely on them being here tomorrow if things get really bad.

Here's an interesting game to play: how many Twitter people (Ops or
otherwise) do you think are actively working on fixing the problem at
this very moment? (Don't include people "just communicating" or
otherwise not able to directly affect the situation.)

a) 0
b) 1-5
c) 6-10
d) 11-15
e) 16-20
f) 20+

I'm guessing (c), 6-10. And, that's me being optimistic for a change.

--
Dossy Shiobara | do...@panoptic.com | http://dossy.org/
Panoptic Computer Network | http://panoptic.com/
"He realized the fastest way to change is to laugh at your own
folly -- then you can let go and quickly move on." (p. 70)

[CaMason]

"it's a clear message to us in the third-party
ecosystem that we'd better not make them our primary focus because we
can't rely on them being here tomorrow if things get really bad."

Surely, that's a wise move anyway, considering Twitter is a third-
party supplier of free data. All of us app developers are at the whim
of Twitter's operations and business decisions.

Relying on third-parties for any business is risky - especially ones
that have no contractual obligations to you.

[Stuart]

On 9 Aug 2009, at 16:35, Dossy Shiobara <do...@panoptic.com> wrote:

On 8/9/09 10:00 AM, Joe Bowman wrote:

First off, to people stating that Twitter Ops needs to work 30 hour

shifts, and any ops person who hasn't, isn't a real ops person. A real

ops person knows that after about 15 hours their mental capacity to

solve problems begins to deteriorate and they have to rest in order to

not cause further damage to their organization. Maybe in the little

mom and pop shops where you run the show you're stuck with those 30

hour marathons, but in larger shops you have team members, and you

schedule with them.

"People" didn't state it.  I did.

I've worked in Ops in small 3-person companies as well as for 5,000+ employee companies with an IT team of 60+ people.  In both situations, there have been exceptional times where yes, I've taken a 30 minute nap on the floor to recharge through a 30+ hour marathon emergency situation.

I'm not suggesting that these people don't need to sleep (unless they have access to some Provigil and/or Adderall), but if Twitter isn't treating this outage as a #1 priority - and I mean literally, not figuratively - then it's a clear message to us in the third-party ecosystem that we'd better not make them our primary focus because we can't rely on them being here tomorrow if things get really bad.

Here's an interesting game to play: how many Twitter people (Ops or otherwise) do you think are actively working on fixing the problem at this very moment?  (Don't include people "just communicating" or otherwise not able to directly affect the situation.)

   a) 0
   b) 1-5
   c) 6-10
   d) 11-15
   e) 16-20
   f) 20+

I'm guessing (c), 6-10.  And, that's me being optimistic for a change.

If you really think the number of people they have working on this has any relevance to the effectiveness of their defence then you clearly have very little understanding of what's happening.

Here's all I have to say on this...

* Any possibility that they're not taking this seriously is rediculous!

* Twitter don't owe you anything, even if *you* chose to build *your* business around *their* service.

* Yes, 30 hour shifts are sometimes needed in situations like this, but you're not running Twitter so let them get on with it and stop playing back-seat manager.

* I can't believe you lot don't realise that constantly demanding status updates, while certainly important to you, is little more than a distraction for those who are actually fighting the good fight. When they know something they think should be shared, you can bet they'll share it. In the meantime grab a Guinness and be patient - all good things come to those who wait!

Now put your keyboard down and go play outside; it's a beautiful day.

-Stuart

-- 

http://3ft9.com/  

[Terry Jones]

>>>>> "Stuart" == Stuart <stu...@gmail.com> writes:
Stuart> * I can't believe you lot don't realise that constantly demanding
Stuart> status updates, while certainly important to you, is little more
Stuart> than a distraction for those who are actually fighting the good
Stuart> fight.

I woke up this morning with the thought that the Twitter mailing list has
now become part of the DDoS.

What percentage of the people complaining loudly and increasing the general
stress/pressure level are actually bots? :-)

Terry

[Adam Cloud]

***Scenario***

A band broadcasts their music on a radio station all the time, and people are able to freely tune into it, or go buy their music. They go and play in a city park for free every day just because it's a much nicer experience for the listener then to be just sitting at home listening on their radio.

You as an up and coming entrepreneur go buy a hotdog & drink stand and setup camp in that park to make some cash off of the flow of people who come to see this free event every day. You being there, giving the ability for people to eat & drink without leaving the park allows for more of this bands songs to be heard, in effect increasing the chance that their music might be purchased. So you're essentially helping them, by taking advantage of them for your business.

The band gets in a car crash, and alot of equipment is damaged to the point of not being able to be used, along with their main source of transportation. The band starts working to find and replace all that is damaged in their equipment and for their car. 

Now you can imagine that little hotdog stand guy standing on their doorstep while they recover yelling profanities and how they should be skipping the shipping company who's delivering their parts and get their parts themselves to save time. Yelling that they shouldn't be sleeping, they should be working on their band van right now to make sure it can take them back to the park so he can make some money. "People aren't coming to my stand anymore!!! They're going to fast food restaurants and going home. WTF i sold my wife for this stand!!!"

Now of course, this little hotdog stand man may not have really sold his wife, depending upon which one of you people who are still up in arms about this was put in his place, but i think you get my points.

The band could easily move to a venue that has their own hotdog/drink stand making your services not necessary, but instead of doing that and capitalizing on the profit they could get from that, they're still planning on going back to the same park they do their free shows at, and allowing you to continue earning your money.

And this concludes storytime. :)

Happy sunday! (Relax!)

[Rich]

I agree, and I know the Twitter Ops guys are probably exhausted and
working around the clock to keep the system running. I have the
utmost respect for it having been responsible for a website before
that suffered from DDoS attacks on occasion.

Twitter.com seems to be running pretty well right now, although I just
can't get Mobile Safari to work properly with oAuth in my simulator.
Obviously this isn't everyone as my user figures do keep going up
which is good news.

I would like it if there were more of a Dev API status page though
maybe rather than the general Twitter status page.

Occasionally my cron jobs from my whitelisted ips get blocked. I
monitor it, and throttle the cron a bit. I was running every 5 mins,
at the moment I've dropped it to every 10 and this exact moment I'm
monitoring it manually as it had totally dropped for a while (although
it seems to be connecting again now when I run it.

I tried the 302 redirect for curl code posted but that code didn't
actually re-post the request back to twitter for POST requests. At
the moment my library IS using 302s for all GET requests (which
actually is most of them!)

On Aug 9, 5:58 pm, Terry Jones <te...@jon.es> wrote:

[Neil Ellis]

Nice story Adam, however the band are actually trying to run a business, not doing this for love/free. I can assure you the investors in Twitter will be looking to turn profit. Of course if the band are laid up then the danger is the hotdog man (and all his customers) will go to another band that are still playing and have fans.

That's why I'm 100% confident all that can be done is being done, cos plenty of people at Twitter will know how fickle a user base can be.

Good luck guys, I know what these situations are like and it's hard on you all - I actually hope you guys are getting some rest because it doesn't sound like this is a 100 yard sprint.

I also hope someone is making sure the ops/devs aren't reading this list (or getting emails etc) - stress doesn't help productivity in my experience. Knowing what is at stake does.

Again good luck chaps, I know how the trenches feel :-)

And of course it does suck for the rest of us too, alas that is business.

ATB

Neil

[David Fisher]

A few of you are acting like real children and a few of you still have
your heads screwed on right.

I'm confident they are doing everything they can. Chill and enjoy your
weekend. They'll get it sorted out.

What did you guys do in 2007? Twitter was down all the time then. Your
blood pressure must have been through the roof with weekly visits to a
shrink if you responded this way every time it went down.

dave

[Sean Callahan]

Agree with what you said. Very well put. It is affecting most all of
us. Our photo sharing service (TweetPhoto) is tied into 20 apps whose
users aren't able to upload photo onto our platform. I've communicated
by adding an alert to our homepage about the issues which broadcasts
the message and hopefully helps manage user expectations. Twitter,
you'll figure it out and find a solution. I'm also confident you'll
keep us in the loop going foward. Thanks!

Sean

[Andrew Badera]

On Sun, Aug 9, 2009 at 2:52 PM, Sean Callahan <seanca...@gmail.com> wrote:

Agree with what you said. Very well put. It is affecting most all of
us. Our photo sharing service (TweetPhoto) is tied into 20 apps whose
users aren't able to upload photo onto our platform. I've communicated
by adding an alert to our homepage about the issues which broadcasts
the message and hopefully helps manage user expectations. Twitter,
you'll figure it out and find a solution. I'm also confident you'll
keep us in the loop going foward. Thanks!

Sean

Also, if you're serving as a bus or any sort of infrastructure, you should probably have both in-band _and_ out-of-band channels for communicating this stuff to YOUR third-parties. Any information you need to convey about Twitter status and your status should be wrapped into every call you return. Saves you this hassle in the future.

Thanks-
- Andy Badera
- and...@badera.us
- Google me: http://www.google.com/search?q=andrew+badera
- This email is: [ ] bloggable [x] ask first [ ] private

[Dossy Shiobara]

On 8/9/09 12:47 PM, Stuart wrote:
> * I can't believe you lot don't realise that constantly demanding status
> updates, while certainly important to you, is little more than a
> distraction for those who are actually fighting the good fight.

I hear there's this popular service that makes it easy to send out short
status updates ... what's it called again?

[Dean Collins]

On 8/9/09 12:47 PM, Stuart wrote:
> * I can't believe you lot don't realise that constantly demanding
status
> updates, while certainly important to you, is little more than a
> distraction for those who are actually fighting the good fight.

>Dossy Shiobara | do...@panoptic.com | http://dossy.org/

>I hear there's this popular service that makes it easy to send out
short
>status updates ... what's it called again?

Ha ha funniest thing I've heard about this whole debacle.

Regards,

Dean Collins
Cognation Inc
de...@cognation.net
+1-212-203-4357 New York
+61-2-9016-5642 (Sydney in-dial).
+44-20-3129-6001 (London in-dial).

[Neil Ellis]

>
>
> What did you guys do in 2007? Twitter was down all the time then. Your
> blood pressure must have been through the roof with weekly visits to a
> shrink if you responded this way every time it went down.

How many people had bet their business/livelihood on Twitter in 2007?

Compassion all-round Dave, I feel for the businesses who are suffering
as
much as Twitter themselves. The amount of hassle Twitter are getting is
probably no less than the downstream hassle to the 3rd parties. We
live in a very demanding society (in the West at least).

The wrongdoers in this are the DDoS people. Everyone else is
collateral in
someone else's stupid war. I'm surprised no government has stepped
in to
be honest as it's setting a hell of a precedence (who knows maybe they
have).

It is like a surreal tech soap opera on this list though :-) Hey maybe I
could pitch this to Fox .....

All the best
Neil

[David Fisher]

> It is like a surreal tech soap opera on this list though :-) Hey maybe I
> could pitch this to Fox .....
>
> All the best
> Neil
>

DO IT! Maybe Twitter would have made a good reality show. Lots of
super-dramatic angles and queues pointing at sysadmins typing at a
console. :)

The mailing list would be like the random room they walk into to talk
to the camera.

dave

[Neil Ellis]

Well there was the Twitter TV program, maybe this is it. Maybe we're
in a
Borat style movie already ;-)

I think it should be in the style of 24 : I can see it now, clock
counting
down - Jack Bauer torturing sysadmin; developers being exposed
as double agents .... oh the intrigue :-)

Of course if it was 24 it would turn out that Biz Stone works for
Facebook.

LOL

ATB
Neil

[Ryan]

I have just started writing a Twitter App for a small company (great
timing lol). Part of it works and part of it is down. Is there any
where else I can check to see which parts of the API are down? I have
continually checked status.twitter.com, however I want to make sure
that any errors I'm seeing currently are caused by the twitter outtage
and not my own (possibly errant) code.

Basically I'm under a deadline and if I need to contact my superiors I
want to make sure that I'm right in assuming sections of the API are
truly down.

[Scott C. Lemon]

Uh ... for all of you hotdog men, threatening to move on to another
"band that are still playing and have fans." ... PLEASE DO!

I can promise you that you will not be missed by twitter, or anyone
here.

For all of the whiny developers who didn't plan for a rainy day ...
maybe this is time for you to think about a real business plan,
instead of making easy money off of an impressive free ride. Or maybe
you're more worried that you are now looking bad to your fan club ...
in that case you'll be a business failure no matter what.

In any case, if people actually think that the appropriate people and
resources aren't being applied to this situation ... you ARE
ignorant. The people committed to twitter - including employees and
investors - won't allow that to be the case. They have FAR more at
stake than all of us combined.

Lastly ... if you want to complain about problems, please take that
somewhere else and share it with negative people who want to rag and
complain with each other to make themselves feel better and smarter.
If you have some serious questions and requests for HOW YOU CAN HELP?
Well ask away ... provide suggestions ... or fly out to San Fran and
beg to be let in the door and assist.

And so, my fellow twitter developers: ask not what twitter can do for
you - ask what you can do for twitter. :-)

[Scott Haneda]

In all honesty, I think you are stuck. The responsible thing to do is follow this list and wait until the API is reported 100% functional. 

Writing a new app is bound to have testing issues, currently you have zero way of knowing if it is your app, the API, routing blocks, or even congestion. 

You will waste time, sanity, and money by trying to develop in this environment. 

I think all you can do, if wanting to do the right thing by your employer, is tell them no future work can be done at this time. 

Of course, there may be other areas you can work on, like UI. There is enough data in the API wiki you can mimic some of the results and hobble along. I still think the only smart thing to do is wait. 

This may be a good lead in to a conversation about relying entirely on another service for a source of a business model. 

-- 

Scott

Iphone says hello.

[Neil Ellis]

Or you could actually read my email, my goodness people are so rude.

No threats in my email, please read again in context.

Peace
Neil