I know it's a little late in asking, but should we switch off cron
jobs that make a lot of API calls while this DoS is going on, or while
you are recovering from it?
I don't want my IP addresses to be blocked because they are making a
lot of calls! I've seen in the past that Ops lay down carpet bombing
with cluster munitions when under attack.
Will it help you to recover if we switched off the cron jobs?
Right now most of my connections are just being refused.
Do you guys at least check against the list of white listed IP
addresses before you block an IP address in times like these?
Will there be innocent bystanders caught in the cross-fire again?
This is the kind of info that we developers need...
> I know it's a little late in asking, but should we switch off cron
> jobs that make a lot of API calls while this DoS is going on, or while
> you are recovering from it?
> I don't want my IP addresses to be blocked because they are making a
> lot of calls! I've seen in the past that Ops lay down carpet bombing
> with cluster munitions when under attack.
> Will it help you to recover if we switched off the cron jobs?
> Right now most of my connections are just being refused.
> Do you guys at least check against the list of white listed IP
> addresses before you block an IP address in times like these?
> Will there be innocent bystanders caught in the cross-fire again?
> This is the kind of info that we developers need...
I turned our crons off, just to be safe. Plus there isn't much of a
point of running them when the majority of the api calls still aren't
getting through.
On Aug 6, 1:35 pm, Dewald Pretorius <dpr...@gmail.com> wrote:
> I know it's a little late in asking, but should we switch off cron
> jobs that make a lot of API calls while this DoS is going on, or while
> you are recovering from it?
> I don't want my IP addresses to be blocked because they are making a
> lot of calls! I've seen in the past that Ops lay down carpet bombing
> with cluster munitions when under attack.
> Will it help you to recover if we switched off the cron jobs?
> Right now most of my connections are just being refused.
> Do you guys at least check against the list of white listed IP
> addresses before you block an IP address in times like these?
> Will there be innocent bystanders caught in the cross-fire again?
> This is the kind of info that we developers need...
We're talking to our operations team about it, who in turn is talking to our hosting provider. It seems that some aggressive IP filtering may have been catching some web-based third-party Twitter applications, as well as data centers used by mobile providers.
> returns nothing at all from my production server, which seems like a > sign that its IP has been blocked.
> My app works fine from my dev box.
> -jonathan
> On Aug 6, 1:35 pm, Dewald Pretorius <dpr...@gmail.com> wrote: >> Chad,
>> I know it's a little late in asking, but should we switch off cron >> jobs that make a lot of API calls while this DoS is going on, or while >> you are recovering from it?
>> I don't want my IP addresses to be blocked because they are making a >> lot of calls! I've seen in the past that Ops lay down carpet bombing >> with cluster munitions when under attack.
>> Will it help you to recover if we switched off the cron jobs?
>> Right now most of my connections are just being refused.
>> Do you guys at least check against the list of white listed IP >> addresses before you block an IP address in times like these?
>> Will there be innocent bystanders caught in the cross-fire again?
>> This is the kind of info that we developers need...
Thanks Alex - just to confirm, no requests from twitterfeed have been getting though ever since the DOS attack. It does appear to be IP based, as requests from non-production machines (ironically the non-whitelisted IPs) get through, but all production IPs appear to be blocked.
On Thu, Aug 6, 2009 at 9:40 PM, Alex Payne <a...@twitter.com> wrote:
> We're talking to our operations team about it, who in turn is talking > to our hosting provider. It seems that some aggressive IP filtering > may have been catching some web-based third-party Twitter > applications, as well as data centers used by mobile providers.
> On Thu, Aug 6, 2009 at 12:52, Jonathan<twitcaps.develo...@gmail.com> > wrote:
> > I would also appreciate an answer to this question. My calls to the > > Search API are failing because of circular redirection, and
> > returns nothing at all from my production server, which seems like a > > sign that its IP has been blocked.
> > My app works fine from my dev box.
> > -jonathan
> > On Aug 6, 1:35 pm, Dewald Pretorius <dpr...@gmail.com> wrote: > >> Chad,
> >> I know it's a little late in asking, but should we switch off cron > >> jobs that make a lot of API calls while this DoS is going on, or while > >> you are recovering from it?
> >> I don't want my IP addresses to be blocked because they are making a > >> lot of calls! I've seen in the past that Ops lay down carpet bombing > >> with cluster munitions when under attack.
> >> Will it help you to recover if we switched off the cron jobs?
> >> Right now most of my connections are just being refused.
> >> Do you guys at least check against the list of white listed IP > >> addresses before you block an IP address in times like these?
> >> Will there be innocent bystanders caught in the cross-fire again?
> >> This is the kind of info that we developers need...
> We're talking to our operations team about it, who in turn is talking
> to our hosting provider. It seems that some aggressive IP filtering
> may have been catching some web-based third-party Twitter
> applications, as well as data centers used by mobile providers.
> On Thu, Aug 6, 2009 at 12:52, Jonathan<twitcaps.develo...@gmail.com> wrote:
> > I would also appreciate an answer to this question. My calls to the
> > Search API are failing because of circular redirection, and
> > returns nothing at all from my production server, which seems like a
> > sign that its IP has been blocked.
> > My app works fine from my dev box.
> > -jonathan
> > On Aug 6, 1:35 pm, Dewald Pretorius <dpr...@gmail.com> wrote:
> >> Chad,
> >> I know it's a little late in asking, but should we switch off cron
> >> jobs that make a lot of API calls while this DoS is going on, or while
> >> you are recovering from it?
> >> I don't want my IP addresses to be blocked because they are making a
> >> lot of calls! I've seen in the past that Ops lay down carpet bombing
> >> with cluster munitions when under attack.
> >> Will it help you to recover if we switched off the cron jobs?
> >> Right now most of my connections are just being refused.
> >> Do you guys at least check against the list of white listed IP
> >> addresses before you block an IP address in times like these?
> >> Will there be innocent bystanders caught in the cross-fire again?
> >> This is the kind of info that we developers need...
I'm also seeing this same behavior for my whitelisted production IPs for CheapTweet.com and TweetReach.com. (Those were whitelisted under the @CheapTweet and @appozite accounts, respectively.) It works in development, but no requests are getting through to twitter.com on our production servers.
I know you all have a lot on your plate right now but let us know what we can do to get un-blocked.
On Thu, Aug 6, 2009 at 3:56 PM, Mario Menti <mme...@gmail.com> wrote: > Thanks Alex - just to confirm, no requests from twitterfeed have been > getting though ever since the DOS attack. It does appear to be IP based, as > requests from non-production machines (ironically the non-whitelisted IPs) > get through, but all production IPs appear to be blocked.
> On Thu, Aug 6, 2009 at 9:40 PM, Alex Payne <a...@twitter.com> wrote:
>> We're talking to our operations team about it, who in turn is talking >> to our hosting provider. It seems that some aggressive IP filtering >> may have been catching some web-based third-party Twitter >> applications, as well as data centers used by mobile providers.
>> On Thu, Aug 6, 2009 at 12:52, Jonathan<twitcaps.develo...@gmail.com> >> wrote:
>> > I would also appreciate an answer to this question. My calls to the >> > Search API are failing because of circular redirection, and
>> > returns nothing at all from my production server, which seems like a >> > sign that its IP has been blocked.
>> > My app works fine from my dev box.
>> > -jonathan
>> > On Aug 6, 1:35 pm, Dewald Pretorius <dpr...@gmail.com> wrote: >> >> Chad,
>> >> I know it's a little late in asking, but should we switch off cron >> >> jobs that make a lot of API calls while this DoS is going on, or while >> >> you are recovering from it?
>> >> I don't want my IP addresses to be blocked because they are making a >> >> lot of calls! I've seen in the past that Ops lay down carpet bombing >> >> with cluster munitions when under attack.
>> >> Will it help you to recover if we switched off the cron jobs?
>> >> Right now most of my connections are just being refused.
>> >> Do you guys at least check against the list of white listed IP >> >> addresses before you block an IP address in times like these?
>> >> Will there be innocent bystanders caught in the cross-fire again?
>> >> This is the kind of info that we developers need...
> I'm also seeing this same behavior for my whitelisted production IPs for
> CheapTweet.com and TweetReach.com. (Those were whitelisted under the
> @CheapTweet and @appozite accounts, respectively.) It works in development,
> but no requests are getting through to twitter.com on our production
> servers.
> I know you all have a lot on your plate right now but let us know what we
> can do to get un-blocked.
> On Thu, Aug 6, 2009 at 3:56 PM, Mario Menti <mme...@gmail.com> wrote:
> > Thanks Alex - just to confirm, no requests from twitterfeed have been
> > getting though ever since the DOS attack. It does appear to be IP based, as
> > requests from non-production machines (ironically the non-whitelisted IPs)
> > get through, but all production IPs appear to be blocked.
> > On Thu, Aug 6, 2009 at 9:40 PM, Alex Payne <a...@twitter.com> wrote:
> >> We're talking to our operations team about it, who in turn is talking
> >> to our hosting provider. It seems that some aggressive IP filtering
> >> may have been catching some web-based third-party Twitter
> >> applications, as well as data centers used by mobile providers.
> >> On Thu, Aug 6, 2009 at 12:52, Jonathan<twitcaps.develo...@gmail.com>
> >> wrote:
> >> > I would also appreciate an answer to this question. My calls to the
> >> > Search API are failing because of circular redirection, and
> >> > returns nothing at all from my production server, which seems like a
> >> > sign that its IP has been blocked.
> >> > My app works fine from my dev box.
> >> > -jonathan
> >> > On Aug 6, 1:35 pm, Dewald Pretorius <dpr...@gmail.com> wrote:
> >> >> Chad,
> >> >> I know it's a little late in asking, but should we switch off cron
> >> >> jobs that make a lot of API calls while this DoS is going on, or while
> >> >> you are recovering from it?
> >> >> I don't want my IP addresses to be blocked because they are making a
> >> >> lot of calls! I've seen in the past that Ops lay down carpet bombing
> >> >> with cluster munitions when under attack.
> >> >> Will it help you to recover if we switched off the cron jobs?
> >> >> Right now most of my connections are just being refused.
> >> >> Do you guys at least check against the list of white listed IP
> >> >> addresses before you block an IP address in times like these?
> >> >> Will there be innocent bystanders caught in the cross-fire again?
> >> >> This is the kind of info that we developers need...
> On Aug 6, 2:30 pm, Hayes Davis <ha...@appozite.com> wrote: > > I'm also seeing this same behavior for my whitelisted production IPs for > > CheapTweet.com and TweetReach.com. (Those were whitelisted under the > > @CheapTweet and @appozite accounts, respectively.) It works in > development, > > but no requests are getting through to twitter.com on our production > > servers.
> > I know you all have a lot on your plate right now but let us know what we > > can do to get un-blocked.
> > On Thu, Aug 6, 2009 at 3:56 PM, Mario Menti <mme...@gmail.com> wrote: > > > Thanks Alex - just to confirm, no requests from twitterfeed have been > > > getting though ever since the DOS attack. It does appear to be IP > based, as > > > requests from non-production machines (ironically the non-whitelisted > IPs) > > > get through, but all production IPs appear to be blocked.
> > > On Thu, Aug 6, 2009 at 9:40 PM, Alex Payne <a...@twitter.com> wrote:
> > >> We're talking to our operations team about it, who in turn is talking > > >> to our hosting provider. It seems that some aggressive IP filtering > > >> may have been catching some web-based third-party Twitter > > >> applications, as well as data centers used by mobile providers.
> > >> On Thu, Aug 6, 2009 at 12:52, Jonathan<twitcaps.develo...@gmail.com> > > >> wrote:
> > >> > I would also appreciate an answer to this question. My calls to the > > >> > Search API are failing because of circular redirection, and
> > >> > returns nothing at all from my production server, which seems like a > > >> > sign that its IP has been blocked.
> > >> > My app works fine from my dev box.
> > >> > -jonathan
> > >> > On Aug 6, 1:35 pm, Dewald Pretorius <dpr...@gmail.com> wrote: > > >> >> Chad,
> > >> >> I know it's a little late in asking, but should we switch off cron > > >> >> jobs that make a lot of API calls while this DoS is going on, or > while > > >> >> you are recovering from it?
> > >> >> I don't want my IP addresses to be blocked because they are making > a > > >> >> lot of calls! I've seen in the past that Ops lay down carpet > bombing > > >> >> with cluster munitions when under attack.
> > >> >> Will it help you to recover if we switched off the cron jobs?
> > >> >> Right now most of my connections are just being refused.
> > >> >> Do you guys at least check against the list of white listed IP > > >> >> addresses before you block an IP address in times like these?
> > >> >> Will there be innocent bystanders caught in the cross-fire again?
> > >> >> This is the kind of info that we developers need...
Yes seems like this is some sort of IP based blocking that they introduced, since one of my production servers started failing yesterday, then the other server, on a different IP, which was consistantly working, started failing later in the evening.
Any suggestions on who can I contact directly to get this resolved? I filled out the 'whitelisting form' just now, but never had to worry about it in the past as my application is not abusive with rate limits, and not sure if this is the best channel anyway, since its more of an incorrect / misapplied blacklisting issue, it would seem?
> Yes seems like this is some sort of IP based blocking that they introduced,
> since one of my production servers started failing yesterday, then the other
> server, on a different IP, which was consistantly working, started failing
> later in the evening.
> Any suggestions on who can I contact directly to get this resolved? I
> filled out the 'whitelisting form' just now, but never had to worry about it
> in the past as my application is not abusive with rate limits, and not sure
> if this is the best channel anyway, since its more of an incorrect /
> misapplied blacklisting issue, it would seem?
> Hedley
> On Fri, Aug 7, 2009 at 3:42 AM, Dewald Pretorius <dpr...@gmail.com> wrote:
> > They are definitely still actively blocking all volume requests.
> > I noticed this morning that my website was working. Checked, and my
> > rate limit was back to 20,000.
> > So, I switched on one of my cron jobs, and within less than 5 minutes
> > all requests from my IP were being completely blocked again.
> > Wonder just how big are these woods that Twitter has to come out of.
> I'm getting the ame problem with bullsonwallstreet.com - previous
> whitelisted rates of 20000 now down to 150... not recovered yet.
> And I throttle all requests to a pretty low level for the REST API...
> but still down at 150!
> Let's hope that this attack ends soon, and honest users can have the
> performance needed back again soon!
> Simon
> On Aug 7, 7:48 am, Hedley Robertson <hedley.robert...@gmail.com>
> wrote:
> > Yes seems like this is some sort of IP based blocking that they introduced,
> > since one of my production servers started failing yesterday, then the other
> > server, on a different IP, which was consistantly working, started failing
> > later in the evening.
> > Any suggestions on who can I contact directly to get this resolved? I
> > filled out the 'whitelisting form' just now, but never had to worry about it
> > in the past as my application is not abusive with rate limits, and not sure
> > if this is the best channel anyway, since its more of an incorrect /
> > misapplied blacklisting issue, it would seem?
> > Hedley
> > On Fri, Aug 7, 2009 at 3:42 AM, Dewald Pretorius <dpr...@gmail.com> wrote:
> > > They are definitely still actively blocking all volume requests.
> > > I noticed this morning that my website was working. Checked, and my
> > > rate limit was back to 20,000.
> > > So, I switched on one of my cron jobs, and within less than 5 minutes
> > > all requests from my IP were being completely blocked again.
> > > Wonder just how big are these woods that Twitter has to come out of.
We have seen the rates for our app go from 20,000 to 150 and back to 20,000 over a short interval. It is causing complete havoc to our traffic as 150 requests are used up in a matter of minutes and we have no notice about the change happening.
This is not affecting an optional cron job, this is for normal usage to make requests on behalf of our users. If we are limited then the user feels it immediately.
Can you ring fence those white-listed addresses that you recognise as totally legitimate - even if it requires an intensive manual exercise - and then just stabilise things for these sites? Is that being attempted at all? The IP addresses of every app for users of this thread would be a great start!
The IP address I am most concerned about is for Twibbon.com: 174.129.249.253
Our site (tunein.com) is getting 408s from the OAuth API; also, our
daemons that do friend timeline calls have been getting empty results
since 11 PM last night.
> We have seen the rates for our app go from 20,000 to 150 and back to 20,000
> over a short interval. It is causing complete havoc to our traffic as 150
> requests are used up in a matter of minutes and we have no notice about the
> change happening.
> This is not affecting an optional cron job, this is for normal usage to make
> requests on behalf of our users. If we are limited then the user feels it
> immediately.
> Can you ring fence those white-listed addresses that you recognise as
> totally legitimate - even if it requires an intensive manual exercise - and
> then just stabilise things for these sites? Is that being attempted at all?
> The IP addresses of every app for users of this thread would be a great
> start!
> The IP address I am most concerned about is for Twibbon.com: 174.129.249.253
I've noticed that friends_timeline when supplied a count parameter will return nothing. Other parameters seem to work okay.
On Fri, Aug 7, 2009 at 10:54 AM, AdamHertz <adamdhe...@gmail.com> wrote:
> Our site (tunein.com) is getting 408s from the OAuth API; also, our > daemons that do friend timeline calls have been getting empty results > since 11 PM last night.
> On 8月8日, 午前1:15, Jonathan Joyce <jonathan.jo...@gmail.com> wrote:
> > We have seen the rates for our app go from 20,000 to 150 and back to 20,000
> > over a short interval. It is causing complete havoc to our traffic as 150
> > requests are used up in a matter of minutes and we have no notice about the
> > change happening.
> > This is not affecting an optional cron job, this is for normal usage to make
> > requests on behalf of our users. If we are limited then the user feels it
> > immediately.
> > Can you ring fence those white-listed addresses that you recognise as
> > totally legitimate - even if it requires an intensive manual exercise - and
> > then just stabilise things for these sites? Is that being attempted at all?
> > The IP addresses of every app for users of this thread would be a great
> > start!
> > The IP address I am most concerned about is for Twibbon.com: 174.129.249.253
> > I appreciate these are difficult times.
> > Anything you can do would be much appreciated.
