AWS API Request Limits

[Andrew Langhorn]

Hi all,

Recently, we've been hitting the API request limits for certain calls in our AWS account. Is anyone free and willing to share their experiences in working with API request limits and Terraform? 

CloudTrail tells me that most of our API calls are from Terraform, so I'd like to work out good ways of cutting down on API calls made, if possible.

Thanks,

Andrew

[David Adams]

My understanding is that AWS API request limits are typically scaled on pretty small periods (from sub-minute to a 15 minute window, IIRC). If you are hitting them mostly with Terraform then you must be doing a lot of Terraform (or have very complex Terraform). What calls are you hitting  limits on and from your CloudTrail data what rate are you calling those APIs at?

One potential way to reduce this problem would be to turn down Terraform's parallel operations with the `--parallelism` flag. The default is to have 10 workers running so you have some leeway to slow TF down there.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/terraform/issues
IRC: #terraform-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Terraform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to terraform-too...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/CAEpa1DJe6Z8G0E5QP9roBqj0%3D3KUGhzPg8inrXpw6%3DhUDXkN4g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Andrew Langhorn]

We have a lot of Terraform, and some of its reasonably complex. Most actions called are Describe* actions, so I suspect that most of the offending calls are actually during state refresh. There are eight environments, including production, so we're actually exacerbating the problem eight-fold in that way.

As a temporary workaround, we've tried blocking Terraform jobs in our CI pipeline on each other. Previously, we would allow one pipeline to start before another had hit production as long as the new pipeline didn't surpass the previous one (thus taking all of the previous one's changes in), unless there was a good reason.

I didn't know about the parallelism flag; thanks for pointing it out. I'll see if we can tone down by a few workers, and hopefully get that will get us past this a bit easier.

Ultimately, if that doesn't work, the end result might have to be an account migration for certain environments, but if I can avoid that, I'd rather do so... :)

Of course, not knowing what the limits are doesn't help. (Mind, Amazon did point me in the direction of a long mathematical equation to help compute back-off and retry intervals.)

Andrew

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/CAN3s8za%2BnFW2Gf4HjqX_yqH%3Dyp83oNxmn4HzeYGVgZtH0PLsMw%40mail.gmail.com.

[David Adams]

Interesting information. Since you're running your TF thru CI, I can understand why you might start hitting limits. I'm sure the AWS provider in Terraform is very bad from a minimizing queries standpoint, given the internal architecture of Terraform. Ultimately, though, the one thing TF definitely should be doing is to implement the backoff formula AWS recommends internally, rather than just fail on an API ratelimit error. I suspect the AWS Go SDK used by TF has some capabilities for doing so built-in--at least some other official SDKs have idiomatic mechanisms for exponential API backoff--but potentially there would need to be coordination between the parallel goroutines.

Seems like someone out there ought to have built a caching AWS API proxy...

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/CAEpa1DKum52KjuHu_5hhMd8mvVsEV5yed%2B9L41tkBqyPUSJ3pQ%40mail.gmail.com.

[Paul Draper]

Seems like someone out there ought to have built a caching AWS API proxy...

I don't own the code, but I did.

Just Nginx with the URL and body content as the cache key, and maybe a couple other things.

As with any cache, that will circumvent access controls if multiple access levels use it. 

To unsubscribe from this group and stop receiving emails from it, send an email to terrafo...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/CAEpa1DJe6Z8G0E5QP9roBqj0%3D3KUGhzPg8inrXpw6%3DhUDXkN4g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/terraform/issues
IRC: #terraform-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Terraform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to terrafo...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/CAN3s8za%2BnFW2Gf4HjqX_yqH%3Dyp83oNxmn4HzeYGVgZtH0PLsMw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/terraform/issues
IRC: #terraform-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Terraform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to terrafo...@googlegroups.com.