Cloud usage APIs throttling exception

[Charles GERY]

Hi All,

We have been trying to install the emission dashboard on our AWS Project using the cloud usage APIs approach. We managed to make the dashboard work with the classical approach, but had troubles with this second approach.

We followed the instructions described in https://www.cloudcarbonfootprint.org/docs/alternative-data-approaches. We added the required permissions and completed the AWS_ACCOUNTS variable with our account id and name in the api/.env (and the cli/.env) file and set the AWS_USE_BILLING_DATA variable to false.

Everytime we try to run the dashboard we get the following exception : Unable to process footprint request.ThrottlingException: Rate exceeded

Have you ever had this issue before and do you know how to solve it?

Best,

Charles

[Dan Lewis-Toakley]

Hi Charles, 

Thanks for reaching out and using CCF -- apologies for the delayed reply. So we currently aren't actively supporting the high accuracy approach via Cloud Usage APIs. This is mainly due to the primary approach providing a better, more holistic view of your energy and carbon emissions. We're going to update our documentation to make that clearer as well.  

The cloud usage APIs being very service dependent (meaning it is confined only to the services that we've implemented so far in the application), in addition to the strict rate limitations on the Cloud Usage APIs that you're experiencing due to the need of making a request to the API for each individual service. If possible, we recommend sticking with the classical (billing data) approach as it should be enough to get you the accurate estimates that you need.

That said, I spent some time testing this and was able to get a similar error: request.LimitExceededException: Account maximum query concurrency limit of [10] reached. (Service: AWSLogs; Status Code: 400; Error Code: LimitExceededException; Request ID: fa4f37fb-df9e-4d4a-abc7-702c06597739; Proxy: null)

I'm not sure if that is exactly the same one as you, but it looks potentially related. I was able to narrow down the problem to our Lamba implementation. If you don't need energy/carbon estimations for Lambda, then you can remove lines 161-164 from packages/common/src/Config.ts and see how that goes. 

Alternatively if you need Lambda, I have pushed a fix for this that reduces the number of concurrent CloudWatch queries that works for me locally when pulling data for the past year. You could pull that update down and try again. 

Let us know how it goes! 

Best,

Dan

--
We're excited to hear from you and hope you're enjoying Cloud Carbon Footprint.
Please fill out our feedback form: https://forms.gle/nFzkRioryy4R1DGB6
Add your name to ADOPTERS.md: https://github.com/cloud-carbon-footprint/cloud-carbon-footprint/blob/trunk/ADOPTERS.md
Give us a star on the github if you're enjoying the tool!
---
You received this message because you are subscribed to the Google Groups "Cloud Carbon Footprint" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-carbon-foot...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-carbon-footprint/bf1a01c7-e25f-4d57-9cfe-b2127390df01n%40googlegroups.com.

--

Dan Lewis-Toakley Green Cloud Lead, North America Pronouns  He/Him Email  dan.lewi...@thoughtworks.com Telephone  +19172545068

[Charles GERY]

Hi Dan,

Thanks for your reply and for pushing a fix !

Thanks for the heads up regarding the support of the Cloud Usage APIs. We’ll stick to the more holistic version for now then.

Regardless of that, we pulled your fix and tried it locally.  Unfortunately, we still got some errors. We either get the error I described in my previous message, or the one you had.

We did not remove the Lambda implementation in the code, because it is specifically the one we wanted to test ;)

Let us know if you have any updates regarding the Cloud Usage APIs !

Best,

Charles