[ANNOUNCE] Etsy API v2 Rate Limits Change

[Bethany Macri]

Hi all,

We are altering the way rate limits for applications are calculated. Instead of each user of a given application being allocated a set rate limit, applications will now have a single rate limit based upon the number of users of that application. We do not expect this to reduce the total number of allowable calls for a given app and in fact should help applications that have a minority of users that require a large volume of API calls.

This change will be happening within the next few weeks.  We will let you know before the change goes out.  This will be ramped up first to 20% of random requests, then to 50% of random requests and finally to 100% of all requests.  After that you will be seeing different rate limit information in the header of your responses.  

Thanks,
Bethany

[Graeme Grant]

I do have a concern regarding this - whilst many users have 100 to 500 listings, there are users that have over 4,000 listings. To collect data for Listings, it can take more than one call per listings to collect all the data. How can we ensure that regular users are not disadvantaged by users with a large number of listings?
 
G.

 

---

Date: Wed, 16 Apr 2014 12:35:36 -0400
Subject: [ANNOUNCE] Etsy API v2 Rate Limits Change
From: bma...@etsy.com
To: etsy-...@googlegroups.com

--
You received this message because you are subscribed to the Google Groups "Etsy API" group.
To unsubscribe from this group and stop receiving emails from it, send an email to etsy-api-v2...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nicole]

I have the same concern as Grahame, as we have a similar situation - we have to do a number of calls per listing / transaction which can take some daily imports well into the many thousands of calls.

Can you please let us know what the new limits are likely to be so we can start planning for this change?

[david olick]

Hey Etsy API team!  I have some questions regarding this change.

If 10 users each had a limit of 10k each, is the new limit for an API key going to be 100k?  Also I have some concerns for abuse... previously if a single user was over-using a website/app/tool, only that user would lose API privileges, but under the new paradigm it sounds like a single user (or a small minority) can take down a website/app/tool for everyone.

Although I see the advantage of being able to manage API use rather than you do it for us... If your answer to my first question is "no", I am a bit worried for apps that aren't website-based because they have no idea how extensive other users are using the API (including themselves if they have the same app installed on multiple computers/devices).  At the very least, can you offer an API method (which doesn't add to the throttling limit) that lets us query how much of the throttling limit a user has consumed?  I feel this would be necessary to properly manage API use and for debugging reasons.

--

David Olick

Oriku Inc.

[Nicole]

Etsy API Team: Could we please get a reply to our questions before the Easter break if possible? Many of us don't work full time on our apps so the Easter long weekend is going to be the only lengthly period of time before this change is implemented to make the adjustments we may need to make. 

This is potentially a huge deal to our apps depending on what the new limits are going to be, and may involve quite a bit of reworking of our import logic :(

[Bethany]

Hi all,

What specifically are your concerns?  This should not impact your applications negatively.  The change will not decrease the rate limit of your apps.  If a limit is reached, you as the developer can still ask for a rate limit increase.  If you are not hitting your rate limits now, this change will not cause you to reach the limit.  In fact, the change will make it less likely that you hit your limit.  If apps suddenly start hitting rate limits at a higher rate than usual, we would consider this a bug and make an emergency fix to temporarily restore the old behavior.

David, your example is roughly correct.

Bethany

[Bethany]

Additionally, v3 is already using these limits.  If you are curious about what your rate limit will be, you can make a v3 call and see the rate limits in the response header.

[Nicole]

My concern is firstly that the rate limit change wasn't explained, we have just been advised that it is "going to change in the next couple of weeks" to an "application wide limit" - what the change actually is, and why this is happening wasn't explained enough for us to figure out how this change is going to affect our apps so this is why we have concerns and are asking questions.

Rate Limits are a huge deal to most of us with applications that have a number of users with large numbers of API calls as I'm sure we have all had our early issues with trying to find ways to ensure that users never hit these limits. If we do happen to hit these limits, asking for a rate increase is often too late as the user has already experienced the ceiling and has a failed import as a result (which often leads to a new user assuming the fault is with us and we lose the sale) so we really do need to know how this new system works so we can make sure our apps can gracefully handle the change. If this is now application wide instead of user based, once we hit our limit we are effectively completely incapacitated as an app and could potentially lose many sales from this, not just one - so this is incredibly important.

If these new limits going to be calculated based on users (or OAuth access tokens I'm assuming here), when does the new application limit calculate? e.g. Daily or instantly and dynamically as the user authenticates with our application? When does the limit reset?

Does this affect the 10 queries per second rate or is this still based on the individual OAuth access token? If this is also application based, this is going to cause a huge amount of problems for us as we often have many concurrent users doing full imports of their data. We have quite a bit of logic that ensures that we don't hit our individual token limits, and if this is application wide this is going to massively slow down our app as it's now all going to need to go through the one rate limiter.

If we could just get all the pertinent information about this change explained officially from Etsy (rather than leading us on a logic treasure hunt around the discussion group to find "David's response" and via calls to the v3 API to ascertain the exact rules) that would be much appreciated. 

I'm really baffled and frustrated as to why we keep getting obtuse and run-around answers to what I see as quite valid questions :(

[Bethany Macri]

Hi all,

A few points of clarification:

If at any point your application's rate limit is reached, you can request that the limit be increased and in most cases we will increase it.

The limit is calculated when a a user authenticates or revokes an OAuth token.

The rate limits will be application-wide for both the daily and the per-second limits.

We do not expect this change to cause problems for applications.

Bethany

--

[david olick]

Hey Bethany!

What we really need is specifics and examples.  To properly update our applications we NEED specifics.  Rolling it out slowly is great for finding bugs, but it's a horrible way at learning how a new system works.

--

David Olick

Oriku Inc.

[Bethany Macri]

Hi everyone,

You should not need to change your applications because of this change.

Besides letting you know when and at what rate this change will be rolled out, answering David's earlier question on this email chain and informing you that you can see your application's rate limit by making v3 calls, answering Nicole's questions about when rate limits are calculated and whether or not they will affect both per-second and per-day limits, assuring you that we neither intend nor expect this change to affect your applications negatively, what helpful examples and/or specifics can I provide?

We are not obligated to disclose the way these limits are calculated.

Thanks,

Bethany

[Graeme Grant]

Hi Bethany,
 
From what I have read, the calculations are per user (aka private usage). How about those apps/websites that don't use private calls and only public calls? How does the new system cater for these services?
 
G.
 

---

Date: Mon, 21 Apr 2014 15:31:26 -0400
Subject: Re: [ANNOUNCE] Etsy API v2 Rate Limits Change
From: bma...@etsy.com
To: etsy-...@googlegroups.com

[Nicole]

So if we could use an example to clarify this as I am still not completely clear on the ramifications of this change, if these are our current rate limits:

SERVICE	PUBLIC RATE LIMITS		OAUTH RATE LIMITS (PER USER)		STATUS

Production API

35 per sec.

10,000 per day

35 per sec.

10,000 per day

Full Access

When the new rate limits are implemented, if I have 1000 users for example would I have a total daily application limit of 10000000 (1000 x 10,000) and a total per second limit of 35,000 (1000 x 35)?

Or, are the per second limits to remain as is at 35 per sec (as I am imagining this is most probably the case)? If so, to reiterate my comments above: this will cause quite a number of problems for my app as we heavily rely on our per second request limit to be per OAuth token at present.

Could please clarify this point please? I'm sure you are making this change with the best of intentions, and you don't intend that this change will impact our apps negatively but we need to assess this change for ourselves to ensure that it doesn't as we have businesses that depend on providing this service to our users.

[Bethany Macri]

Hi all,

The upcoming change will affect the per-second rate limits in the same way it affects the daily limits.  So in Nicole's example, after this change, the per-second limit for that application would be roughly 35,000 (it would not remain at 35 per second for users across the application).  I hope this provides clarity and also allays some of your concerns.

Please let me know what other specifics you'd like answered about this change.

Bethany

[Nicole]

Bethany: thanks for the clarification, it goes a long way to helping us plan for this change. 

I'm sure this has all been figured out on the Etsy infrastructure end to ensure you can handle the increase in API traffic as 35,000 requests a second for an app with only 1000 users seems like an extremely generous amount that I'm sure most of us will be now going to take full advantage of! 

I'm sure that there are Etsy apps that have active OAuth tokens in the tens of thousands if not hundreds of thousands, so the rate limits per second for these applications will be absolutely incredible...

When this is implemented, we will be definitely trying to take advantage of our new per second rate limits as we can now scale up our delayed job processes and add extra processing servers to make our importing awesomely fast, and finally look at implementing a scheduled automatic daily updater for our users.

Just wanted to double check on this as it sounded too good to be true from our perspective! :)

[Cameron]

Hi Bethany - 

Has this change been rolled out yet, or is it still to come?

We have recently been getting errors when processing sales for some large shops that we have exceeded our quota for that particular shop:

"You have exceeded your quota of: xxxxxx requests per 86400 second(s) per user for authorized requests."

So it seems like the old quota system is still in place, unless the error message is just out of date.

We are very excited about the new system of one bulk quota and a huge increase in per-second requests.  If you can share any updates that would be great.

-Cameron

[Nicole]

Are any updates on when this is due to be rolled out? We would love to start using the new limits!

On Wednesday, 16 April 2014 17:35:36 UTC+1, Bethany wrote:

[Nicole]

Should we assume then that this rate limit change is not going to happen anytime soon? 

In April, Bethany said that the rate change was due to happen in the "next couple of weeks" and it is now 5 months later with no further update. If this has now been completely shelved it would be nice to at least know as we can then take our performance improvements task off our schedule.

Thanks!

[david olick]

Bump.

Cameron and Nicole have been asking questions since July with no response.

--

David Olick

Oriku Inc.

--