[ANNOUNCE] Upcoming change to BillCharge findAllUserCharges API call

101 views
Skip to first unread message

Paul Wright

unread,
Jul 2, 2012, 8:42:27 AM7/2/12
to etsy-...@googlegroups.com
Hi everyone,

To improve the response speed for API calls to BillCharge findAllUserCharges (/users/:user_id/charges) we are planning to alter the behaviour such that the min_created and max_created parameters become mandatory and can be no more than 31 days apart. If only one of these _created parameters is submitted in a request then the other will default to +/-31 days as appropriate.


As this is a behaviour change to this API call we intend to roll it out over the following timeline:

* 05/02/12, today: notify the API mailing list and add a note to this effect to the developer's documentation.
* 05/16/12, two weeks from now: monitor API requests and notify any developers relying on the old behavior.
* 05/25/12, three weeks from now: switch over to the new behavior, any requests that do not include min_created and max_created will be returned a HTTP 400 error response.

We realise that this might result in some additional work but hope you can appreciate that the current unrestricted approach slows down our response and, indirectly, your applications.

Cheers,

Paul.

Paul Wright

unread,
Jul 2, 2012, 9:46:43 AM7/2/12
to etsy-...@googlegroups.com
Apologies, those dates are July, rather than May. The actual time line is:

* 07/02/12, today: notify the API mailing list and add a note to this effect to the developer's documentation.
* 07/16/12, two weeks from now: monitor API requests and notify any developers relying on the old behavior.
* 07/25/12, three weeks from now: switch over to the new behavior, any requests that do not include min_created and max_created will be returned a HTTP 400 error response.

Cheers,

Paul.

nic

unread,
Jul 2, 2012, 5:25:41 PM7/2/12
to etsy-...@googlegroups.com
Are these dates set in stone? These changes are going to require a fairly significant rewrite of my codebase and I'm not 100% positive I will be able to get these changes into production in the space of a mere 3 weeks from now  :( 

Please keep in mind that for many of us the weekends are the only time we have to work on our apps, so from now gives us approximately 6 "working days" to code, test and release - providing we don't actually have a spare day off before now and the deadline - sigh :(

I'd really appreciate at least a month at the bare minimum for notice on these sort of big changes, especially if we are going to get hit with 400 error responses if we fail to hit this date (which is obviously going to affect our relationships with our users) - this approach seems quite harsh to me to be honest?


Nicole
Craftybase

Paul Wright

unread,
Jul 5, 2012, 5:34:34 AM7/5/12
to etsy-...@googlegroups.com
Hi Nicole,

Firstly, thanks for getting in touch.  I appreciate that not every app has an army of developers working full time on it.  

We're trying to limit the scope of impact with this issue to the paging of results - switching from large offsets to narrower timestamp windows as the desired access pattern for a user's bill charges.

It's very possible this isn't the only use case of this API call and if this is the case I'd love to hear how developers are using it, that would help us.

Cheers,

Paul.

--
You received this message because you are subscribed to the Google Groups "Etsy API V2" group.
To view this discussion on the web visit https://groups.google.com/d/msg/etsy-api-v2/-/9-ik8BHFR98J.
To post to this group, send email to etsy-...@googlegroups.com.
To unsubscribe from this group, send email to etsy-api-v2...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/etsy-api-v2?hl=en.

Gabriel Fielding

unread,
Jul 6, 2012, 11:36:17 AM7/6/12
to etsy-...@googlegroups.com
Hi Paul,

I have questions regarding this change.

Is the API currently accepting and using those new parameters?

I want to be sure of the appropriate meaning of the "min_" and "max_" prefix with regards to orientation in time. I assume "min_" is temporally before "max_". Is that correct?

Your post says only one of "min_created" and "max_created" parameters is required but the API synopsis states both are mandatory.  I assume your post is correct. So for example, if I have a call now in which I list the most recent 25 user charges, I send the parameters: 'limit' => '25','offset' => "0",'sort_order' => "down".  Can you confirm that in order to make that call conform to the new change, I would just need to add one parameter:  'max_created' => strtotime("now").  

If you are trying to improve response speed on your side, would tightening the date range when I make calls reduce your load?  That is, would the inclusion of 'min_created' => strtotime("-1 day") be an improvement over letting it default to "-31 days"? 

thanks,
Gabe

Paul Wright

unread,
Jul 6, 2012, 12:02:09 PM7/6/12
to etsy-...@googlegroups.com
Hi Gabriel,

Thanks for getting in contact!  (Answers inline.)

On Fri, Jul 6, 2012 at 4:36 PM, Gabriel Fielding <gabriel....@gmail.com> wrote:
Hi Paul,

I have questions regarding this change.

Is the API currently accepting and using those new parameters?

Yes, these parameters have been optional for a while.  Currently there is no limit for length of time between min_created and max_created though that will change when they become mandatory.

I want to be sure of the appropriate meaning of the "min_" and "max_" prefix with regards to orientation in time. I assume "min_" is temporally before "max_". Is that correct?

Your interpretation is correct, min_created must be earlier than max_created.  If this is not the case you will receive an error response.

Your post says only one of "min_created" and "max_created" parameters is required but the API synopsis states both are mandatory.  I assume your post is correct. So for example, if I have a call now in which I list the most recent 25 user charges, I send the parameters: 'limit' => '25','offset' => "0",'sort_order' => "down".  Can you confirm that in order to make that call conform to the new change, I would just need to add one parameter:  'max_created' => strtotime("now").  

Ideally clients would control both min_created and max_created with each request but we want to be flexible in this requirement so there's the additional defaulting behaviour described if one of them is not supplied.  With your example if you include the max_created parameter as the current timestamp then you will get back up to 25 charges that have occurred in the last 31 day period.  (In this transitional period you would get the 25 most recent charges regardless of when they were created.)

If you are trying to improve response speed on your side, would tightening the date range when I make calls reduce your load?  That is, would the inclusion of 'min_created' => strtotime("-1 day") be an improvement over letting it default to "-31 days"? 

If you're happy to only get results for the last 1 day period then this will indeed reduce the work we need to do to respond.

Hope that helps,

Paul. 

Nicole Pascoe

unread,
Jul 6, 2012, 12:11:31 PM7/6/12
to etsy-...@googlegroups.com
Thanks for the reply - it's the narrower query window, not the min / max parameters that is going to cause the most amount of rework for me - I essentially need some way of retrieving all of a user's historical bill payments when a new user starts with our app so that they initially have their full bill charge history ready for analysis. If they don't have this when they signup, then they are not going to see the full benefit of my app :(

If this window is going to be set, I'll have to change my code to repeatedly cycle back in 31 day increments until I hit the initial join date for the user (which is obviously going to cause more API calls in total than the current solution)....unless there is a different way that is recommended by Etsy?

Graeme Grant

unread,
Jul 6, 2012, 12:14:51 PM7/6/12
to EtsyAPI Forum
If you take a peek at the number of records to retrieve, you could run several data pipes simultaneously to pull down the data ... up to 10 calls/second.


Date: Fri, 6 Jul 2012 17:11:31 +0100
Subject: Re: [ANNOUNCE] Upcoming change to BillCharge findAllUserCharges API call
From: nicole...@gmail.com
To: etsy-...@googlegroups.com

Nicole (Craftybase)

unread,
Jul 8, 2012, 2:26:43 PM7/8/12
to etsy-...@googlegroups.com
Thanks for the suggestion :) I use delayed jobs at present to handle simultaneous API calls - the issue isn't so much how to handle performance on my end, its more that I don't want to implement something that a) means that I use more of my API limits and b) is something that Etsy doesn't want me to do so I'm kinda stuck until I get some guidance on what Etsy wants me to implement.

It would be really great if someone from Etsy could let back to me asap on this as I really now effectively have 1 weekend to implement this before my app ceases to function correctly...

Nicole
Paul.


To unsubscribe from this group, send email to etsy-api-v2+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/etsy-api-v2?hl=en.

--
You received this message because you are subscribed to the Google Groups "Etsy API V2" group.
To post to this group, send email to etsy-...@googlegroups.com.
To unsubscribe from this group, send email to etsy-api-v2+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/etsy-api-v2?hl=en.

--
You received this message because you are subscribed to the Google Groups "Etsy API V2" group.
To post to this group, send email to etsy-...@googlegroups.com.
To unsubscribe from this group, send email to etsy-api-v2+unsubscribe@googlegroups.com.

Nicole (Craftybase)

unread,
Jul 13, 2012, 11:51:46 AM7/13/12
to etsy-...@googlegroups.com
I really need to know what Etsy now recommends as an approach now that the large offset is deprecated - changing my code to cycle right back to the user's join date is going to substantially add to the API calls I make as part of a user's initial import. 

As I only really now have this weekend to make these changes, can I please get a reply for this today if possible?

Thanks,

Nicole

Paul.


To unsubscribe from this group, send email to etsy-api-v2+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/etsy-api-v2?hl=en.

--
You received this message because you are subscribed to the Google Groups "Etsy API V2" group.
To post to this group, send email to etsy-...@googlegroups.com.
To unsubscribe from this group, send email to etsy-api-v2+unsubscribe@googlegroups.com.

Paul Wright

unread,
Jul 13, 2012, 12:11:29 PM7/13/12
to etsy-...@googlegroups.com
Hi Nicole,

Just wanted to confirm that we're okay with the number of calls
increasing when doing an initial import.

It's a trade off between a super flexible API method that's slow to
respond or a faster but slightly more restrictive method. In this
instance we're going with the latter as it speeds up our response. We
understand it may inflate the number of API calls, especially in your
situation of an initial import.

Hope that helps!

Paul.
>>>> etsy-api-v2...@googlegroups.com.
>>>> For more options, visit this group at
>>>> http://groups.google.com/group/etsy-api-v2?hl=en.
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Etsy API V2" group.
>>> To post to this group, send email to etsy-...@googlegroups.com.
>>> To unsubscribe from this group, send email to
>>> etsy-api-v2...@googlegroups.com.
>>> For more options, visit this group at
>>> http://groups.google.com/group/etsy-api-v2?hl=en.
>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Etsy API V2" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/etsy-api-v2/-/G_bds7XWFWYJ.
>
> To post to this group, send email to etsy-...@googlegroups.com.
> To unsubscribe from this group, send email to
> etsy-api-v2...@googlegroups.com.

Katrina

unread,
Jul 13, 2012, 6:46:56 PM7/13/12
to etsy-...@googlegroups.com


On Monday, July 2, 2012 2:42:27 PM UTC+2, Paul Wright wrote:
To improve the response speed for API calls to BillCharge findAllUserCharges (/users/:user_id/charges) we are planning to alter the behaviour such that the min_created and max_created parameters become mandatory and can be no more than 31 days apart. If only one of these _created parameters is submitted in a request then the other will default to +/-31 days as appropriate.

Previously when we called the BillCharge findAllUserCharges endpoint, the result said:

{
  count: 50346,
  results: [ { ...}, {...} ]
}

This tells us how many expenses the user has.

Now, since we restrain the call to BillCharge findAllUserCharges by min_date and max_date, the count that comes back in the result only refers to the number of expenses that the user has *within that timeframe*.

We've been using this number to sanity check that we've actually managed to fetch all of a user's expenses.

What call can we make now in order to get that number?

Cheers,
Katrina

Nicole (Craftybase)

unread,
Jul 14, 2012, 7:52:16 AM7/14/12
to etsy-...@googlegroups.com
This is also going to be a big problem for my app too - I'm going to need to know the total number of bill charges before I start processing them or I'll have no idea when the import is complete...the alternative is to hit every 31 day call first to add up the totals, THEN kick off the same calls again as async to start processing the data. This doesn't sound right at all....

Is there any chance of a reconsideration on this API change - this feels like we are moving backwards in functionality? 

Chris Dary

unread,
Jul 14, 2012, 10:58:07 AM7/14/12
to etsy-...@googlegroups.com
Hi Nicole, Katrina,

Let me confer with the team in regard to total counts.

I can say that we need to move forward with these changes. As a little background, this method as it's defined currently is one of the most expensive in our system. These changes are designed to make it a lot faster, which is better in the long run for all of us. I know having to switch to the new syntax is a pain, but hopefully you'll see faster syncs as a result, especially on large sets of charges.

Nicole, you should be able to detect when the import is complete by going backward through time up to the user's join date. Once you've hit the join date, you'll know you've got them all. Does that make sense?

Hope this helps,

-Chris

Nicole (Craftybase)

unread,
Jul 14, 2012, 12:04:22 PM7/14/12
to etsy-...@googlegroups.com
While I understand that the API needs to evolve to ensure performance, this change is removing functionality that we were previously able to rely on in our apps, and is going to result in slower performance on our side as we are going to have to make significantly more API calls than we were previously. 

We have been performance tuning our Etsy import process for the best part of 18 months now - as a result, we have a complex system of delayed jobs and other async processes to ensure that the Etsy import is as quick as we can make it as this is really important to our users. Your proposed solution to my problem assumes a single thread in a very simple environment in real time, which is not the case at all.

This isn't a just a "switch in syntax" - this is asking us to inherently change the way our apps work - with only 3 weeks notice to do so. This most recent change wasn't consultative, it was dictated to us with a threat to return error codes if we didn't comply within your hard date deadline. There is a very long history of this sort of thing going on here (you only have to read the first page of this group to get a feel for things), but at least in the past we had been given a fair amount of time to make breaking changes.

It would be great now that we have a new API lead that they started communicating with us as fellow developers that are also working on high traffic consumer web apps, rather than a group of random hackers that are doing this for a bit of fun. We have customers that rely on our software, so we need to be able to trust and rely on the API team here that they have our interests in mind.

Chris Dary

unread,
Jul 14, 2012, 1:25:37 PM7/14/12
to etsy-...@googlegroups.com
Hey Nicole,

I have a couple thoughts here. Before I speak to them though: I want to say that I hear your frustration, and not in a please-hold-we'll-be-with-you-shortly kind of way. I wouldn't be here if there weren't improvements to be made.

Feedback (and criticism!) is valuable, and I'm happy to have it: I hope this can start out in a working-together context, because I genuinely want to help you make awesome things. It's pretty much my job. Does that make sense? If not, feel totally free to email me personally: cd...@etsy.com.

OK, my thoughts:

Re: the removal of counts: I hear that. I'm talking to the team now about how we can get the count information back out there in a simple way. It is the weekend though, so I might not have anything to say here until Monday or Tuesday.

Re: communication and timeframes: Honestly it'd be my preference that backwards incompatible changes never occur. But when they have to, in the future I'll try to make sure ample, ample notice is given. If we end up providing an alternate resource for count information, it would seem reasonable to me to extend this deadline a bit. I'll see what we can do, no promises yet though.

I hope this is helpful. Let me know,

-Chris

--
You received this message because you are subscribed to the Google Groups "Etsy API V2" group.
To view this discussion on the web visit https://groups.google.com/d/msg/etsy-api-v2/-/ZWAhiz36deQJ.

Chris Dary

unread,
Jul 17, 2012, 5:33:26 PM7/17/12
to etsy-...@googlegroups.com
Hi folks,

I just wanted to follow up here as promised. We're finalizing a new resource which should solve the count problem for you and hopefully be a bit more useful as well. I'll follow up when it's become a concrete spec (within the next day or so I would suspect).

In addition, some hopefully good news: we'll be pushing out this deadline a bit to help you guys wrap your heads around the new resource before we have to go dark on the old syntax. When I follow up with the spec I'll be able to give you some hard timelines.

Katrina/Nicole, thanks for your feedback on this.

-Chris

On Saturday, July 14, 2012 1:25:37 PM UTC-4, Chris Dary wrote:
Hey Nicole,

I have a couple thoughts here. Before I speak to them though: I want to say that I hear your frustration, and not in a please-hold-we'll-be-with-you-shortly kind of way. I wouldn't be here if there weren't improvements to be made.

Feedback (and criticism!) is valuable, and I'm happy to have it: I hope this can start out in a working-together context, because I genuinely want to help you make awesome things. It's pretty much my job. Does that make sense? If not, feel totally free to email me personally: cd...@etsy.com.

OK, my thoughts:

Re: the removal of counts: I hear that. I'm talking to the team now about how we can get the count information back out there in a simple way. It is the weekend though, so I might not have anything to say here until Monday or Tuesday.

Re: communication and timeframes: Honestly it'd be my preference that backwards incompatible changes never occur. But when they have to, in the future I'll try to make sure ample, ample notice is given. If we end up providing an alternate resource for count information, it would seem reasonable to me to extend this deadline a bit. I'll see what we can do, no promises yet though.

I hope this is helpful. Let me know,

-Chris

To unsubscribe from this group, send email to etsy-api-v2+unsubscribe@googlegroups.com.

Katrina Owen

unread,
Jul 18, 2012, 7:52:41 AM7/18/12
to etsy-...@googlegroups.com
Hi Chris,

Thanks for the update.

We're all good to go on the new syntax, and we can simply ignore the
sanity-checks/counts for the moment.

Cheers,
Katrina
>>> etsy-api-v2...@googlegroups.com.
>>> For more options, visit this group at
>>> http://groups.google.com/group/etsy-api-v2?hl=en.
>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Etsy API V2" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/etsy-api-v2/-/yEynGtyYHm0J.
> To post to this group, send email to etsy-...@googlegroups.com.
> To unsubscribe from this group, send email to
> etsy-api-v2...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages