Platform downtime is expected

[Doug Williams]

Hi all --

If you have been monitoring our status blog [1] or been to Twitter.com today you have noticed that we are once again experiencing problems due to external causes. The issues causing the downtime require that we once again take measures to bring the site back online.

The first step our operations team must take will likely cause API downtime, especially affecting OAuth. We apologize for the inconvenience and we will work quickly to reduce the impact to the API. We appreciate your patience and I will update you as soon as we know more.

Thanks,

Doug

[dougw]

Looks like I forgot the link to the status blog.

[1] http://status.twitter.com/post/163603406/working-on-unexpected-downtime

Thanks,
Doug

On Aug 15, 11:08 am, Doug Williams <d...@twitter.com> wrote:
> Hi all --If you have been monitoring our status blog [1] or been to

[Sam Street]

I haven't experienced any downtime or lack of connectivity so far.

[Sean Callahan]

Thanks for the update Doug. Users on TweetPhoto are not able to login.
I've added an alert notification on our homepage, http://TweetPhoto.com,
to make them aware of the issues linking to the Twitter status blog.
Will you need our IPs again to whitelist them or are you good to go.
Please let me know how I can be of service.

-Sean

[Doug Williams]

Sean,

At this time we are monitoring the situation and containing issues as we see them. Let's hold off on restoration requests until things stabilize.

Thanks,

Doug

[Sean Callahan]

Thanks for the reply Doug. Any new news? Still not able to login
using basic auth on TweetPhoto. Do you have any ETA as to when we'll
be restored?


On Aug 15, 1:29 pm, Doug Williams <d...@twitter.com> wrote:
> Sean,At this time we are monitoring the situation and containing issues as

> we see them. Let's hold off on restoration requests until things stabilize.
>
> Thanks,
> Doug
>

> On Sat, Aug 15, 2009 at 1:17 PM, Sean Callahan <seancalla...@gmail.com>wrote:
>
>
>
>
>
> > Thanks for the update Doug. Users on TweetPhoto are not able to login.

> > I've added an alert notification on our homepage,http://TweetPhoto.com,

[Ritvvij]

Hi could this be the reason why?

On clicking authenticate user, my application moves to twitter. When
the user presses APPROVE, it waits like forever and then goes to a
blank page with url as "http://twitter.com/oauth/authorize " instead
of my call back URL. ??

Please advise.
Till last night my app was working, and since today morning, even
right now, its down. :-(

[goodtest]

Same here. New users can't authorize. After pressing 'Approve', it
takes for ever and ends up in a blank-page with http://twitter.com/oauth/authorize
url.

[Ritvvij]

Phew! thanks a lot.. i spent whole day thinking where I messed up the
code. :-)

On Aug 15, 10:03 pm, goodtest <goodtest...@gmail.com> wrote:
> Same here. New users can't authorize. After pressing 'Approve', it

> takes for ever and ends up in a blank-page withhttp://twitter.com/oauth/authorize

[bosher]

Noticing the downtime with oAuth now...

[Sean Callahan]

I was a bit confused about this downtime as the Twitter status blog
says it was unexpected, but the post Doug wrote here says it was
expected. Nevertheless, standing by and awaiting restoration of the
API so users can log into their favorite Twitter apps.

-Sean

[Andrew Badera]

On Sun, Aug 16, 2009 at 4:30 AM, Sean Callahan<seanca...@gmail.com> wrote:
>
> I was a bit confused about this downtime as the Twitter status blog
> says it was unexpected, but the post Doug wrote here says it was
> expected. Nevertheless, standing by and awaiting restoration of the
> API so users can log into their favorite Twitter apps.
>
> -Sean

Blog entry came first. By the time Doug posted here, they knew there
was an issue, and expected it could/would cause downtime. But the
incident(s) it/themselves were unexpected.

∞ Andy Badera
∞ This email is: [ ] bloggable [x] ask first [ ] private
∞ Google me: http://www.google.com/search?q=(andrew+badera)+OR+(andy+badera)

[Costa Rica]

Hello
I am perceiving search API downtime as well

http://search.twitter.com/search?q=Indonesia returns results, but the
XML, ATOM, JSON are returning me a blank page...
TCI

On Aug 15, 12:08 pm, Doug Williams <d...@twitter.com> wrote:
> Hi all --If you have been monitoring our status blog [1] or been to

[Hwee-Boon Yar]

I was trying to be patient with this, but seeing that no one is
complaining, I'm afraid I might be alone here.

One of SimplyTweet's server had not been able to access the API *at
all*. Even /rate_limit_status (nothing to do with OAuth) timeout every
time. Is this expected? No response when I tweeted @twitterapi, as
usual.

wget "http://twitter.com/account/rate_limit_status.xml?
screen_name=hboon"


--
Hwee-Boon

On Aug 16, 4:40 pm, Costa Rica <ticoconid...@gmail.com> wrote:
> Hello
> I am perceiving search API downtime as well
>

> http://search.twitter.com/search?q=Indonesiareturns results, but the

[Jonathan George]

1. It's been roughly 10 hours. How about an update?

2. It'd be great if you would post this to status.twitter.com, in
addition to the developer mailing list. Status is seen by more users,
and the last update you have on it is rather ambiguous. When users
see that the Twitter web interface is up and running, they expect the
apps to be up and running as well.

best,
jonathan

On Aug 15, 1:08 pm, Doug Williams <d...@twitter.com> wrote:
> Hi all --If you have been monitoring our status blog [1] or been to

[Sean Callahan]

Yeah, no one can login to http://TweetPhoto.com just tried minutes ago
and still no dice. Twitter, do I need to provide anything to you or do
something on my end to allow users to login through basic
authentication on our site? I tried moments ago and could not login,
but was able to login to a competitors site no problem. Please
advise.

-Sean

[twitscoop]

Can't agree more with 2.

Please guys, you need to convey a consistant and *visible* message to
your users.

[twitscoop]

Same for us, please let us know if there is anything we can provide /
do to restore oAuth access to our app. Some other comp. web-based apps
work 100%, so this is kind of embarrassing...

Many thanks in advance.

On 16 août, 17:41, Sean Callahan <seancalla...@gmail.com> wrote:
> Yeah, no one can login tohttp://TweetPhoto.comjust tried minutes ago

[johan pretorius]

Doug:

> The first step our operations team must take will likely cause API downtime,
> especially affecting OAuth. We apologize for the inconvenience and we will
> work quickly to reduce the impact to the API. We appreciate your patience
> and I will update you as soon as we know more.

Do you guys have any news on the Oauth outage?

We can still not authenticate our users (tweetfunnel.com).

thanks,

Johan

[Ritvvij]

Me neither

On Aug 16, 12:32 pm, johan pretorius <johan.pretor...@gmail.com>
wrote:

[bosher]

How is it that all the oAuth apps out there are down, but others like
TweetMeMe are not? TweetMeMe works just fine, how is that possible?

[Andrew Badera]

On Sun, Aug 16, 2009 at 1:40 PM, bosher<bhel...@gmail.com> wrote:
>
> How is it that all the oAuth apps out there are down, but others like
> TweetMeMe are not? TweetMeMe works just fine, how is that possible?
>

HTTP Basic Auth still works I believe, as do any pre-problem OAuth
tokens issued.

[Chad Etzel]

Hi all,

The API team is actively debugging the OAuth issues as we speak.
Please be patient as we nail down the problems.

Thanks,
-Chad

[Hwee-Boon Yar]

> The API team is actively debugging the OAuth issues as we speak.

This is what worries very much.

I can't even do this without timing out:

curl "http://twitter.com/account/rate_limit_status.xml?
screen_name=hboon"

I have no API access *at all*.

--
Hwee-Boon

[bosher]

Thanks for the update Chad..

On Aug 16, 10:52 am, Chad Etzel <jazzyc...@gmail.com> wrote:
> Hi all,
>
> The API team is actively debugging the OAuth issues as we speak.
> Please be patient as we nail down the problems.
>
> Thanks,
> -Chad
>
>
>
> On Sun, Aug 16, 2009 at 1:43 PM, Andrew Badera<and...@badera.us> wrote:
>

[bosher]

Not sure if this will help or not, regarding oAuth Sign in With
Twitter

Works - Sign in With Twitter works if the user isn't logged into
Twitter and has to enter their L&P on Twitter
Doesn't Work - If the user is already signed into Twitter, clicks
Allow, it just hangs...

[Dewald Pretorius]

Can you at the very least PLEASE publish something on
status.twitter.com about the API being down and/or very unresponsive
at times, so that I have a link where I can refer my users, so that
they can see I am not shitting them?

Dewald

[Chad Etzel]

We've asked the keeper-o-the-blog to post something to that effect.
Hopefully it will appear soon.
-Chad

[Hwee-Boon Yar]

Can you confirm if OAuth access is the only known issue? I feel silly
repeating the same question over and over again: "Even /
rate_limit_status calls are timing out on my server. I have no API

access *at all*".

--
Hwee-Boon

[Ryan Sarver]

Everyone,

Please see the updated post on status.twitter.com -
http://status.twitter.com/post/164410057/trouble-with-oauth-and-api-clients.

We are continuing to assess the issue and will report back when we know more.

Thanks for your patience, Ryan

[Dewald Pretorius]

Thanks Ryan, but unfortunately that does not help me. It does not tell
my users that most, if not all calls to the API are being rejected
with connection refused. In other words, from my perspective, the API
is completely down / inoperative / unresponsive.

My issue has nothing to do with OAuth.

Dewald

On Aug 16, 7:41 pm, Ryan Sarver <rsar...@twitter.com> wrote:
> Everyone,
>

> Please see the updated post on status.twitter.com -http://status.twitter.com/post/164410057/trouble-with-oauth-and-api-c....

>
> We are continuing to assess the issue and will report back when we know more.
>
> Thanks for your patience, Ryan
>

[Dewald Pretorius]

From my side, it looks as if Operations are once again not allowing
volume white-listed API traffic through the defenses.

It's almost as if last weekend did not happen, and no lessons were
learned from it.

I am not once again going to throw my toys out of the cot. It was only
on Friday that I found my favorite rubber duck underneath my mom's
bed.

Dewald

PS. Please ask them to fix it ASAP. The way things are going now, it
is neither easy, nor attractive to provide services that use the
Twitter API.

[Josh Roesslein]

Anyone having troubles also with profile image / background update API endpoints? I'm getting 500 errors so
I'm guessing the error is on twitter's end. Just want to be sure its not my code.

Josh

[goodtest]

I agree. Its hard to convince people to use twitter API if it goes
down for days and that too this frequently :(

[Ryan Sarver]

Dewald,

What exact issues are you having? Can you please provide packet dumps
or more information so we can debug?

Thanks, Ryan

[Ryan Sarver]

Please test your OAuth apps to see if you are still having issues. We
have made a number of changes in the network and they should be
operational again.

Please let me know if you are still having any other problems.

Ryan

[Ritvvij]

Oauth integration started working for me.

[Dewald Pretorius]

Ryan,

I am having exactly the same issue as last Saturday and early Sunday.
Low volume API calls go through, but the moment you make a lot of
calls from the same IP address, you are completely blocked and get
connection refused.

Operations need to do the same that they did last Sunday, when they
allowed high volume calls from white-listed IP addresses through the
defenses.

Dewald

[Dewald Pretorius]

Is anyone at Twitter going to look at this and get back with an update
or answer?

[Jonathan George]

Please post an update to status.twitter.com, so our users can be as
patient as we are.

best,
jonathan

[Aaron Forgue]

Is this downtime specific to only oAuth related requests, or all API
requests? My site has not been functioning for the past day and all I
am getting back are http codes of "0".

Is my app perhaps blocked?

[Nick L.]

Hey All,

Definitely experiencing OAuth issues as well. Still in development
on my local machine but I keep getting time out errors upon trying
authenticate Oauth here:

http://twitter.com/oauth/authenticate

*Work-A-Round*

So I could keep things going (and keep coding), my quick workaround
was to sign into twitter.com first, and then let my app authorize
oauth that way. This has worked, but I think its because I've set my
app to "sign-in-with twitter". You can do so on your app config page
on twitter (http://www.twitter.com/apps/). Not sure if this will help
some.

The big issue is that I know other peoples (and their users) are
still unable to authorize the Oauth tokens, so hopefully that gets
resolved soon.

Keep up the good work Twitts. Crossing my fingers this isn't an
ongoing issue.

On Aug 16, 6:41 pm, Ryan Sarver <rsar...@twitter.com> wrote:
> Everyone,
>

> Please see the updated post on status.twitter.com -http://status.twitter.com/post/164410057/trouble-with-oauth-and-api-c....

>
> We are continuing to assess the issue and will report back when we know more.
>
> Thanks for your patience, Ryan
>

[Dewald Pretorius]

I'm not so sure that one can describe the relative silence on this
thread as patience.

It can probably be ascribed to developers abandoning Twitter and being
too busy coding for other platforms to even bother writing on this
thread.

To have exactly the same thing happen to third-party apps for two
weekends in a row, which causes high-volume API calls to be blocked
and the apps to be broken for days on end, is just simply pathetic.
That is NOT "developer support" in any way, shape or form.

Not to mention the fact that again nothing is posted about it on the
Twitter Status blog, even after a request for such a post was made by
us.

We are again left swinging in the wind by Twitter.

Dewald

[Nick L.]

The OAuth issue seems to be resolved on this end. Upon *not* being
already signed into twitter, I can now use oAuth to authenticate and
sucessfully fire back towards the callback ur li've set it up. So it
seems like it's back working.

I"m on a development/local machine though so I'm not sure how this
effects in production.

[Dewald Pretorius]

*bump*

[mikeo]

We are still experiencing timeout errors when trying to OAuth. We have
been down since Saturday and I have not seen an update yet today.

If the Oauth issues have been resolved, are there any IP ranges being
blocked from API usage. Our application is hosted on Rackspace.

An update on this would be great.

Thank You,

Mike

On Aug 15, 2:08 pm, Doug Williams <d...@twitter.com> wrote:
> Hi all --If you have been monitoring our status blog [1] or been to
> Twitter.com today you have noticed that we are once again experiencing
> problems due to external causes. The issues causing the downtime require
> that we once again take measures to bring the site back online.

>
> The first step our operations team must take will likely cause API downtime,
> especially affecting OAuth. We apologize for the inconvenience and we will
> work quickly to reduce the impact to the API. We appreciate your patience
> and I will update you as soon as we know more.
>

> Thanks,
> Doug

[Adam]

Ryan,

We have been experiencing OAuth issues since Saturday and would like
an update on status.

We are currently receiving timeout errors when trying to OAuth.

If the OAuth issue has been resolved are there currently any IP ranges
being blocked from the API. Our app is hosted on Rackspace.

An update would be appreciated!

Adam

[Jonathan@ScribbleLive]

The Twitter OAuth login is working about 50% of the time for us.
Sometimes we get failures making the initial request for a token. If
that works, and the user gets to the Decline/Accept screen, when they
click "Accept" the connection often times out before they get a
response.

[Ryan Sarver]

Everyone,

I am meeting with Ops right now to get a status update and will follow
up with the list as soon as we are done.

Stay tuned.

Best, Ryan

[Mike O]

We are still experiencing OAuth Issues. I haven't seen an official
status update this morning. Have these issues been resolved?

We have been down since Saturday and are receiving timeout exceptions
when trying OAuth. If the issues have been resolved are any IP ranges
currently being blocked. Our application is hosted on Rackspace.

Regards,

Mike

On Aug 16, 9:11 pm, Ryan Sarver <rsar...@twitter.com> wrote:
> Dewald,
>
> What exact issues are you having? Can you please provide packet dumps
> or more information so we can debug?
>

[Adam]

Hi Ryan,

Still meeting? Anxiously waiting since we are now dead in the water
for 3 days now.

Thanks.

Adam

On Aug 17, 1:39 pm, Ryan Sarver <rsar...@twitter.com> wrote:
> Everyone,
>
> I am meeting with Ops right now to get a status update and will follow
> up with the list as soon as we are done.
>
> Stay tuned.
>
> Best, Ryan
>

[Alex Payne]

There are other threads going about these ongoing issues, including
instructions for what to send us to help troubleshoot.

--
Alex Payne - Platform Lead, Twitter, Inc.
http://twitter.com/al3x

[Sean Callahan]

Hi Ryan,

I just replied to your email and also will post here in case you read
this first. Maybe others will have an idea too as why basic auth is
not working. Long story short - we were down for 3 days, up for the
last 3 hours, and now down once again - no one can login using basic
auth on our site http://tweetphoto.com.

It so happens when I responded to your email about 3 hours ago
everything was working fine - meaning I was able to login to
TweetPhoto as were other users. Since replying to your second email
basic authentication on http://tweetphoto.com is not working an no one
can login to their account on our domain.

I'm not sure why basic auth is not working when you said only OAuth
should have been affected. With that said, I have an idea as I've
spoken to a few Rackspace customers. Is it possible that a range of
IPs at Rackspace are being blocked/throttled preventing users from
logging in through basic auth?

I'm not sure why basic authentication came back online this morning
and now it does not work. You asked me to respond to an email Alex
sent, but I have not seen that. Please let me know what information
you need to better troubleshoot this issue so we can resume service.

As I mentioned in my email to you, I am willing to pay a monthly
service fee as I'm sure other Twitter Developers are, to keep service
running to TweetPhoto.com up 100% of the time.

Please advise.

Best Regards,

Sean Callahan

On Aug 16, 11:40 am, bosher <bhellm...@gmail.com> wrote:
> Thanks for the update Chad..
>
> On Aug 16, 10:52 am, Chad Etzel <jazzyc...@gmail.com> wrote:
>
>
>
> > Hi all,
>
> > The API team is actively debugging the OAuth issues as we speak.
> > Please be patient as we nail down the problems.
>
> > Thanks,

> > -Chad
>

> > On Sun, Aug 16, 2009 at 1:43 PM, Andrew Badera<and...@badera.us> wrote:
>
> > > On Sun, Aug 16, 2009 at 1:40 PM, bosher<bhellm...@gmail.com> wrote:
>
> > >> How is it that all the oAuth apps out there are down, but others like
> > >> TweetMeMe are not? TweetMeMe works just fine, how is that possible?
>
> > > HTTP Basic Auth still works I believe, as do any pre-problem OAuth
> > > tokens issued.
>
> > > ∞ Andy Badera
> > > ∞ This email is: [ ] bloggable [x] ask first [ ] private

> > > ∞ Google me:http://www.google.com/search?q=(andrew+badera)+OR+(andy+badera)- Hide quoted text -
>
> - Show quoted text -

[Dewald Pretorius]

On Aug 17, 7:33 pm, Sean Callahan <seancalla...@gmail.com> wrote:
> As I mentioned in my email to you, I am willing to pay a monthly
> service fee as I'm sure other Twitter Developers are, to keep service
> running to TweetPhoto.com up 100% of the time.

Amen. And so am I.

Dewald

[Ryan Sarver]

Sean,

I was just able to log into TweetPhoto using my basic auth credentials
with no problem. Please test again and provide any of the additional
details that you can:

*Copying from Alex email to make sure its consistent
1. The IP of the machine making requests to the Twitter API. If you're
behind NAT, please be sure to send us your *external* IP.

2. The IP address of the machine you're contacting in the Twitter
cluster. You can find this on UNIX machines via the "host" or
"nslookup" commands, and on Windows machines via the nslookup" command.

3. The Twitter API URL (method) you're requesting and any other
details about the request (GET vs. POST, parameters, headers, etc.).

4. Your host operating system, browser (including version), relevant
cookies, and any other pertinent information about your environment.

5. What kind of network connection you have and from which provider,
and what kind of network connectivity devices you're using.
</copy>

Best, Ryan

[Sean Callahan]

Hi Ryan,

When I sent the email 30 minutes ago I was not able to login. Now,
like you, I was able to login. Now it is a sporadic issue. I will get
you the info you need to those 5 questions. Because our current
environment at the Rackspace Cloud is shared I need to call them.
Basically, it is simply a basic auth issue. We're not calling
anything, but that. User's cannot login. The IPs of the Rackspace
Cloud that I am on are (Primary) 74.205.61.228 and (Secondary)
74.205.61.229 Not sure if this helps. Please advise.


Sean

> > auth on our sitehttp://tweetphoto.com.

>
> > It so happens when I responded to your email about 3 hours ago
> > everything was working fine - meaning I was able to login to
> > TweetPhoto as were other users. Since replying to your second email

> > basic authentication onhttp://tweetphoto.comis not working an no one

> >> > > ∞ Google me:http://www.google.com/search?q=(andrew+badera)+OR+(andy+badera)-Hide quoted text -
>
> >> - Show quoted text -- Hide quoted text -

[benn]

The oauth login timeout seems to be fixed for me. Thanks
twittereenies!

Ben

> > > basic authentication onhttp://tweetphoto.comisnot working an no one

> > >> > > ∞ Google me:http://www.google.com/search?q=(andrew+badera)+OR+(andy+badera)-Hidequoted text -