GCP lookup table coefficients

[Alireza Razaghi]

Hi,

First of all, thank you very much for the hard work you are putting in this tool.

In our company, we use AWS, GCP, and Azure and in order to calculate the carbon foot print, I'm using the look up table approach.

I started with AWS and after creating the lookup table I simply multiply our usage amount with the co2e values in the lookup table and the results make sense. 

Then I tried to do the same thing for GCP, but here the results do not make sense as they are very high compared to AWS and we have much more usage in AWS. 

So my question is about GCP usage units in the lookup table. For instance, if in the lookup table the unit is seconds, do I need to convert the usage amount to hours and then multiply it by the coefficient? Because if I keep it in seconds the result is too high. What about byte-seconds, do I need to conver them into Gb-month?

Thank you in advance for your help,

Ali

[Cloud Carbon Footprint]

Hi Ali,

Thank you for reaching out!

You may notice here that when calculating vCpu hours for GCP, we are simply dividing the usage amount by 3600 to convert the usage data in seconds to hours. In this commit, you will see that we originally had the total number of vCPU Hours calculated as a function of compute usage amount in hours multiplied by the number of vCpus from the Billing Export Data query. We determined that this is not correct. The usage amount is already factored in the number of vCpus in the query results, so we were overestimating compute resources.

We are very open to feedback on this approach, as we have acknowledged this creates some slight differences in estimations among other CSP's.

Thanks,
The Cloud Carbon Footprint Team

[Alireza Razaghi]

Hi,

Thank you for your answer.

It's still not clear to me though. Should I convert our usage to hours and then multiply by coefficients in the look-up table or leave them in seconds? What about other units like byte-seconds vs. Gb-month?

I really appreciate your help here.

Cheers,

Ali

[Cameron Casher]

Hi Ali,

Apologies, I misunderstood your question. The result for compute resources should be multiplied by hours. I can see how this would be deceiving since we return the usageUnit value of seconds for GCP.

We will plan to remove the usageUnit column from the Lookup Table output, as well as provide extra documentation on the correct category units to avoid further confusion.

Thanks,

--
We're excited to hear from you and hope you're enjoying Cloud Carbon Footprint.
Please fill out our feedback form: https://forms.gle/nFzkRioryy4R1DGB6
Add your name to ADOPTERS.md: https://github.com/cloud-carbon-footprint/cloud-carbon-footprint/blob/trunk/ADOPTERS.md
Give us a star on the github if you're enjoying the tool!
---
You received this message because you are subscribed to the Google Groups "Cloud Carbon Footprint" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-carbon-foot...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-carbon-footprint/cfa97fce-99d6-4811-90c4-18a3afa433b1n%40googlegroups.com.

--

Cameron Casher
Senior Software Engineer & Clean Tech Strategist
Email	cameron...@thoughtworks.com
Telephone	+1 (517) 896-2968

[Alireza Razaghi]

Thank you very much Cameron. That would help a lot.

In the meantime is there any way to find out correct units for other services like storage (for example Gb-month)?

On another note, I'm creating a lookup table for Azure and I'm getting following error:

Azure ConsumptionManagementClient UsageDetailRow paging failed. Reason: Too many requests. Please retry after 60 seconds.

Do you know how can I fix it? I'm trying to extract the lookup table for 1 year

Cheers,

Ali

[Cloud Carbon Footprint]

Hi Ali,

The following are the correct units for each service category:
Compute - Hours

Storage - Terabyte-hours

Memory - Gigabyte-hours

Networking - Gigabytes

Also, when generating a lookup table using Azure, you may see that log appear. Due to rate limiting, Azure will likely need to retry many times. For that reason, we have logged the 'too many requests' message as a warning rather than an error. The process should finish running after some time. Please let us know if you receive any errors that prevent the lookup table generation.

Thanks,

The Cloud Carbon Footprint Team

[Alireza Razaghi]

Hi,

Thanks again for the valuable information. 

Regarding the azure lookup table, when I use  yarn create-azure-lookup command without any dates I was getting lots of errors:

"Azure ConsumptionManagementClient UsageDetailRow paging failed. Reason: Too many requests. Please retry after 60 seconds."

But when I specified the dates for the last year (which is way more than default value last 30 days) it worked like a charm and no errors. It took a long time but it worked. So I don't know if there is a bug or I just got lucky!

Also I have a question about Azure. I work in a game company and 70% of our usage is PlayFab but it seems that in the lookup table there is no co2 coefficient for PlayFab (It's zero). Is there any reason for that and any solution?

Cheers,

Ali

[Cloud Carbon Footprint]

Hi Ali,

Thanks for the update on the create-azure-lookup attempt! It is nice to know that you got successful results after specifying the correct dates. Hopefully others can refer to this thread that might face a similar situation.

We think that the likely result of PlayFab being excluded from the estimations is because CCF is not able to categorize it based on the usage data line item details. When that is the case, which is true for most managed services, the estimation is grouped as an 'Unknown Estimation'. You can find more information on that here. If you would be willing to send us the line-item detail of this service (scrubbed of sensitive company data), then we would be happy to confirm this theory.

Thanks,
The Cloud Carbon Footprint Team

[Alireza Razaghi]

Hi,

This is coming from the lookup table if that helps:

Thanks,

Ali

[Cloud Carbon Footprint]

Hi Ali,

It appears that the reason PlayFab is not accounted for, is because the usageUnit is simply '1'. The logic beyond CCF for Azure determines how to categorize a service as compute, storage, networking, or memory based on the usageUnit. As you can see here, if there is an unsupported usageUnit, we do not return an estimation.For Azure, the usageUnit corresponds to the meter details unit of measure within the consumption management API. As of today, we have been unable to find the appropriate documentation to explain what a value of '1' should represent, and therefore, we are unable to accurately estimate it. We will continue looking into this particular usage unit, but I would compel you to provide any feedback you may have to correctly categorize this service by the usage unit.

Thanks,

The Cloud Carbon Footprint Team

[Hao Liu]

Hi,

Thanks for the maintenance of this LUT function. But I'm just a little bit confused about the mentioning of the MD:
For each service categorization, here are the standard usage units that the lookup table results represent and how you should multiply your usage amount:
- Compute: _Hours_
- Storage: _Terabyte-hours_
- Memory: _Gigabyte-hours_
- Networking: _Gigabytes_

So how should I use the usageAmount to multiply the LUT output? For example:
I have such usage data:
| product_region    | line_item_product_code | line_item_usage_type | pricing_unit | line_item_line_item_type | line_item_usage_amount
| -------- | ------- | -------- | ------- | -------- | ------- |
| us-west-2 | AmazonS3 | USW2-TimedStorage-ByteHrs | GB-Hours | Usage | 0.689

In this case, should I use 0.689 * lookupTableOutput.kilowattHours? Or 0.689 / 1000 * lookupTableOutput.kilowattHours? I want to know whether I need convert the usageAmount to TB-Hours again.
And I suppose the lookupTableOutput.co2e is in MT, but not KG, am I right?
 

I also questioned this in the GitHub Issue linked to this.

Thanks in advance~

Cameron CasherSenior Software Engineer & Clean Tech StrategistEmailcameron...@thoughtworks.comTelephone+1 (517) 896-2968