Default throttling and the ability to alter it

[Ethan Pooley]

I am running into throttling on the San Diego MTA OneBusAway server that I don't experience on instances my organization hosts itself. Neither we nor they are aware of why this difference exists or how to change it. I'm looking for clarification on whether any throttling is built into the web services by default, whether it has changed in the last few years, and whether there is a way to control it with server-side configuration options.

When requests are close together (less than one or two tenths of a second) we receive a Permission Denied error just as described here: https://groups.google.com/forum/#!searchin/onebusaway-api/throttling/onebusaway-api/emv-ba0NpAI/fKAlsBqqEhcJ

[Sean Barbeau]

Ethan,
It depends on how they are adding the API key to their OBA server.  If they are using the admin console within the onebusaway-webapp to generate a random key (see screenshot), then I believe the default Min API request interval is 100 ms.  To change this, they need to go into the admin console and edit your key, and change the interval to 0ms.

If they are adding it via the data-sources.xml config when they set up OBA, then I believe the default interval is 0ms, and you shouldn't see this issue.  Interval should be explicitly settable via data-sources.xml I believe as well.

Sean

[Colin McGlynn]

Hi Sean, 

Thanks, that explains a couple different issues we were having!  I've got a couple of suggestions for how I think we could improve this behavior and make it more transparent.

1) Switch from HTTP 401 to HTTP 429 (To Many Requests). 

It is more precise and I think will help future API consumers track down the issue faster when they encounter it.

2) Switch the default Min API request interval to 0

There are a lot of scenarios where an API consumer might make calls in quick succession that aren't abusive. For example, any app that needs information on more than one stop is highly likely to send the requests very near to one another, in response to a user action.

What do you think? If there are no objections I'll make a Github issue.

[Barbeau, Sean]

Colin,

I agree with both #1 and #2.  I hit this same issue when building OBA Alexa – it hits a few APIs quickly (stops-for-location and arrivals-and-departures-for-stop, IIRC) to get all the info needed to present it to the user.

 

Yes, Github issues please!  Thanks!

 

Sean

--
You received this message because you are subscribed to a topic in the Google Groups "OneBusAway API" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/onebusaway-api/652bzEFzlhs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to onebusaway-ap...@googlegroups.com.
To post to this group, send email to onebusa...@googlegroups.com.
Visit this group at https://groups.google.com/group/onebusaway-api.
For more options, visit https://groups.google.com/d/optout.

[Sheldon A. Brown]

for 1)
Brian's reasoning for the 401 is a generic unauthorized. RFC 6585
that defines HTTP 429 hadn't come out yet when this code was written
:)

It's implemented here:
https://github.com/camsys/onebusaway-application-modules/blob/unified/onebusaway-api-webapp/src/main/java/org/onebusaway/api/actions/api/ApiKeyInterceptor.java#L88

for 2)
I'm interested to hear how you envision protecting server resources.
From my perspective, asking a client not to request resources more
than 10 times a second by default is quite reasonable. Sound Transit
is actively working on a terms of service policy, so now is the time
to speak up.

Sheldon

> --
> You received this message because you are subscribed to the Google Groups
> "OneBusAway API" group.
> To unsubscribe from this group and stop receiving emails from it, send an

[cooki...@gmail.com]

Rolling windows would be great. For example, if a user is allowed 500
requests per 60 seconds, any request will be blocked if 500 requests
have already occurred in the preceding 60 seconds, without any set
points at which this interval resets. This requires some extra
implementation so that the user cannot make 500 requests at 0:59 and
another 500 requests at 1:01.

[Sheldon A. Brown]

I'd love to see a pull request for that. Especially one that would
scale across a load balancer without shared state. Its an interesting
problem to solve....

Sheldon

[cooki...@gmail.com]

I've implemented it before, though not in Java. The downside to my
implementation is that requests after the limit kicks in count towards
the total number of requests. Oh, and you'll need an additional piece
of software.
http://slides.tim-bond.com/redis/#/20

[Sheldon A. Brown]

You missed my shared state comment :)

[Sheldon A. Brown]

Seriously though -- what makes OBA appealing to smaller agencies is
its lack of dependencies. Adding Redis or Elasticache may make sense
for very large deployments, but not your average deployment. That
could be handled by optional Spring configuration though.

Sheldon

[Cookie Guru]

I don't follow.  Which doesn't have shared state: the load balancer or the rate limiter?

>>>>> email to onebusaway-api+unsubscribe@googlegroups.com.
>>>>> To post to this group, send email to onebusaway-api@googlegroups.com.

>>>>> Visit this group at https://groups.google.com/group/onebusaway-api.
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>>
>>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "OneBusAway API" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an

>>> email to onebusaway-api+unsubscribe@googlegroups.com.
>>> To post to this group, send email to onebusaway-api@googlegroups.com.

>>> Visit this group at https://groups.google.com/group/onebusaway-api.
>>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "OneBusAway API" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to onebusaway-api+unsubscribe@googlegroups.com.
> To post to this group, send email to onebusaway-api@googlegroups.com.

> Visit this group at https://groups.google.com/group/onebusaway-api.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "OneBusAway API" group.

To unsubscribe from this group and stop receiving emails from it, send an email to onebusaway-api+unsubscribe@googlegroups.com.
To post to this group, send email to onebusaway-api@googlegroups.com.

[Sheldon A. Brown]

I'm just pointing out that your algorithm for rate limiting requires
shared state -- and you've side stepped that problem by requiring
redis.

What's currently implemented now is simple, mostly fair, and requires
no shared state. That's why its appealing.

Sheldon

>> >>>>> email to onebusaway-ap...@googlegroups.com.

>> >>>>> To post to this group, send email to

>> >>>>> onebusa...@googlegroups.com.

>> >>>>> Visit this group at https://groups.google.com/group/onebusaway-api.
>> >>>>> For more options, visit https://groups.google.com/d/optout.
>> >>>>
>> >>>>
>> >>>>
>> >>>
>> >>> --
>> >>> You received this message because you are subscribed to the Google
>> >>> Groups
>> >>> "OneBusAway API" group.
>> >>> To unsubscribe from this group and stop receiving emails from it, send
>> >>> an

>> >>> email to onebusaway-ap...@googlegroups.com.
>> >>> To post to this group, send email to onebusa...@googlegroups.com.

>> >>> Visit this group at https://groups.google.com/group/onebusaway-api.
>> >>> For more options, visit https://groups.google.com/d/optout.
>> >>
>> >>
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "OneBusAway API" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an

>> > email to onebusaway-ap...@googlegroups.com.
>> > To post to this group, send email to onebusa...@googlegroups.com.

>> > Visit this group at https://groups.google.com/group/onebusaway-api.
>> > For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "OneBusAway API" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to onebusaway-ap...@googlegroups.com.
>> To post to this group, send email to onebusa...@googlegroups.com.

>> Visit this group at https://groups.google.com/group/onebusaway-api.
>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "OneBusAway API" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to onebusaway-ap...@googlegroups.com.
> To post to this group, send email to onebusa...@googlegroups.com.

[Cookie Guru]

But if you wanted to validate a dozen requests per second, could you get that out of Hibernate?  No doubt there are some rate limiter applications using Hibernate as a backend.

>> >>>>> email to onebusaway-api+unsubscribe@googlegroups.com.

>> >>>>> To post to this group, send email to

>> >>>>> onebusaway-api@googlegroups.com.

>> >>>>> Visit this group at https://groups.google.com/group/onebusaway-api.
>> >>>>> For more options, visit https://groups.google.com/d/optout.
>> >>>>
>> >>>>
>> >>>>
>> >>>
>> >>> --
>> >>> You received this message because you are subscribed to the Google
>> >>> Groups
>> >>> "OneBusAway API" group.
>> >>> To unsubscribe from this group and stop receiving emails from it, send
>> >>> an

>> >>> email to onebusaway-api+unsubscribe@googlegroups.com.
>> >>> To post to this group, send email to onebusaway-api@googlegroups.com.

>> >>> Visit this group at https://groups.google.com/group/onebusaway-api.
>> >>> For more options, visit https://groups.google.com/d/optout.
>> >>
>> >>
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "OneBusAway API" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an

>> > email to onebusaway-api+unsubscribe@googlegroups.com.
>> > To post to this group, send email to onebusaway-api@googlegroups.com.

>> > Visit this group at https://groups.google.com/group/onebusaway-api.
>> > For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "OneBusAway API" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to onebusaway-api+unsubscribe@googlegroups.com.
>> To post to this group, send email to onebusaway-api@googlegroups.com.

>> Visit this group at https://groups.google.com/group/onebusaway-api.
>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "OneBusAway API" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to onebusaway-api+unsubscribe@googlegroups.com.
> To post to this group, send email to onebusaway-api@googlegroups.com.

> Visit this group at https://groups.google.com/group/onebusaway-api.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "OneBusAway API" group.

To unsubscribe from this group and stop receiving emails from it, send an email to onebusaway-api+unsubscribe@googlegroups.com.
To post to this group, send email to onebusaway-api@googlegroups.com.

[Sheldon A. Brown]

While you probably could do that, part of OBA's strength is its
minimal reliance on a database. Currently the API keys are read in
and then cached, and the rate limiting is all in memory and does not
hit the database. There would be some significant performance
implications to that, or some highly optimized code.

But I'm not saying this can't be done, or a better rate limiting
shouldn't be implemented. I'm just pointing out that it's an
interesting problem to solve.

Pull requests are always welcome :)

>>> >>>>> email to onebusaway-ap...@googlegroups.com.

>>> >>>>> To post to this group, send email to

>>> >>>>> onebusa...@googlegroups.com.

>>> >>>>> Visit this group at https://groups.google.com/group/onebusaway-api.
>>> >>>>> For more options, visit https://groups.google.com/d/optout.
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>
>>> >>> --
>>> >>> You received this message because you are subscribed to the Google
>>> >>> Groups
>>> >>> "OneBusAway API" group.
>>> >>> To unsubscribe from this group and stop receiving emails from it,
>>> >>> send
>>> >>> an

>>> >>> email to onebusaway-ap...@googlegroups.com.
>>> >>> To post to this group, send email to onebusa...@googlegroups.com.

>>> >>> Visit this group at https://groups.google.com/group/onebusaway-api.
>>> >>> For more options, visit https://groups.google.com/d/optout.
>>> >>
>>> >>
>>> >
>>> > --
>>> > You received this message because you are subscribed to the Google
>>> > Groups
>>> > "OneBusAway API" group.
>>> > To unsubscribe from this group and stop receiving emails from it, send
>>> > an

>>> > email to onebusaway-ap...@googlegroups.com.
>>> > To post to this group, send email to onebusa...@googlegroups.com.

>>> > Visit this group at https://groups.google.com/group/onebusaway-api.
>>> > For more options, visit https://groups.google.com/d/optout.
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "OneBusAway API" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an

>>> email to onebusaway-ap...@googlegroups.com.
>>> To post to this group, send email to onebusa...@googlegroups.com.

>>> Visit this group at https://groups.google.com/group/onebusaway-api.
>>> For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "OneBusAway API" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to onebusaway-ap...@googlegroups.com.
>> To post to this group, send email to onebusa...@googlegroups.com.

>> Visit this group at https://groups.google.com/group/onebusaway-api.
>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "OneBusAway API" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to onebusaway-ap...@googlegroups.com.
> To post to this group, send email to onebusa...@googlegroups.com.

> Visit this group at https://groups.google.com/group/onebusaway-api.
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "OneBusAway API" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to onebusaway-ap...@googlegroups.com.
> To post to this group, send email to onebusa...@googlegroups.com.

[Barbeau, Sean]

I think the objection to #2 is fair from a server resource perspective (I tend to think app-first :)), and I concur that keeping things simple in terms of rate limiting is strongly desired.

If #1 is made more transparent to developers (i.e., the cause of the error is rate limiting) I think that would go a long way by itself. We could also use some better documentation surrounding this as well for developers, along with a defined process and contact list if you want to ask to get your limit removed.

Sean

You received this message because you are subscribed to a topic in the Google Groups "OneBusAway API" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/onebusaway-api/652bzEFzlhs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to onebusaway-ap...@googlegroups.com.

[Colin McGlynn]

Looking back to the start of this issue, both the app developers and server administrators were unclear on what was happening and why.  So I agree with Sean, making this more transparent to everyone is the first step regardless of how/if rate limiting is changed and doing so will improve the situation significantly.   

I also agree that rolling windows are a better way to rate limit and are common for a lot of Transit APIs. I don't have the Java knowledge to know what the best way to implement one would be so I won't comment on that aspect.

[Colin McGlynn]

Side question, will the server need to be rebooted before the API rate limiting change takes effect?

San Diego MTS and Sound Transit have been kind enough to set our 'API limit Min API request interval' to 0 but we are still seeing the the 401s

[Sheldon A. Brown]

No, the value is cached but the cache does expire (5-10 minutes from
memory). So I believe it will sort itself out. If you are still
seeing issues after an hour let me know and I'll dig deeper.

Sheldon

[Sheldon A. Brown]

Ok -- that sounds great to me. So we can start with an issue
representing better documentation on the subject, and an issue to
perform some refactoring to propagate the 429 back to the user when
appropriate.

Did you want to file those Colin?

Sheldon

[Colin McGlynn]

Sure thing, https://github.com/OneBusAway/onebusaway-application-modules/issues/189 is posted with my best guess at where the documentation should go

[Colin McGlynn]

HTTP 429 issue https://github.com/OneBusAway/onebusaway-application-modules/issues/190

[Sheldon A. Brown]

Thanks!

On Tue, Nov 1, 2016 at 2:37 PM, Colin McGlynn <co...@transitscreen.com> wrote:
> HTTP 429 issue
> https://github.com/OneBusAway/onebusaway-application-modules/issues/190

[Colin McGlynn]

Hey Sheldon, I'm still getting fairly frequent 401s from PugetSound. Any help is appreciated, thanks!

[Sheldon A. Brown]

I'm looking into this. I'll contact you directly when I have more info.

Sheldon