Wrong CO2 estimation for Azure Logic Apps

[C. G.]

Hi everyone ! 

I'm trying to connect Azure data for my company but i meet some issues.

First, with 386 Azure accounts, I can't set correctly my "/packages/api/.env", requests are too long and i get this error : 
" Azure ConsumptionManagementClient.usageDetails.list failed. Reason Too many requests"

So i choose to set only my Azure account into "/packages/cli/.env" and start the command : yarn seed-cache-file and export 1 month for example and it works but i've got incorrect data. See results below.

"cloudProvider": "AZURE",

                "kilowattHours": 1340710.4783253025,

                "co2e": 423.6645111507954,

                "usesAverageCPUConstant": false,

                "serviceName": "Logic Apps",

                "accountId": "xxx",

                "accountName": "xxx",

                "region": "northeurope",

                "cost": 1973.4540393325624

And now, let's compare with data provided by Microsoft Azure PowerBI app, "Emission Impact Dashboard"

You can see that Microsoft estimates CO2 consumption for 1 month to 0.09 Tons of CO2, and "Cloud Carbon Footprint" app estimates for 423 Tons of CO2 ! See also for kilowatthours consumption, above 1 million kWh for 1 month for only 1 account ! 
There is definitively a problem of units...

I also add this issue : 
" [ConsumptionManagement] warn: Unexpected usage type for storage service: B DTUs " and so many others... I add as mentionned in "/packages/azure/src/lib/ConsumptionTypes.ts" 

export const STORAGE_USAGE_TYPES: string[] = [

  'Data Stored',

  'Metadata',

  'Registry Unit',

  'ZRS',

  'LRS',

  'GRS',

  'GZRS',

  'Data Retention',

  'Pay-as-you-go Data at Rest',

  'Standard Instances',

  'Node',

  '10 DTUs',

  'S0 DTUs',

  'B DTUs',

  'eDTUs',

  'On Premises Server Protected Instances',

  'Standard Trial Nodes',

  'Standard Nodes',

  'Azure VM Protected Instances',

  'Standard User',

  'Microsoft-hosted CI',

  'Basic User',

  'Multi-step Web Test',

  'S0 Secondary Active DTUs',

  'Resource Monitored at 5 Minute Frequency',

  'VM Replicated to Azure',

]

I also tried to add 'Logic Apps' into  STORAGE_USAGE_TYPES but it changed nothing for the co2e results.

If someone had also problem to estimates CO2 consumption for Azure Logic apps, can you help me to have same results as Microsoft estimations displayed in PowerBI ?

Thanks in advance,

Clement

[C. G.]

I'm actually try to connect my data with branch "azure-subscription-chunks". It takes a long time to collect 10 accounts by 10 accounts in total of 386... I warn you if it works better with this method and if yes, i will merge for good modifications into my app !

[C. G.]

UPDATE : 

it seems that because of long time (or too many subscription ?), i can't query data no more than 15 accounts before i get an error and restart my query. So i stopped using this branch.

I also determine that "Bandwidth" CO2 estimation is wrong. Cloud Carbon footprint estimates CO2 consumption for January by 15,4 Tons of CO2 for all my accounts and Microsoft estimates to 0.05 MTCO2 (for scope 1/2/3) for all my accounts.

I add bandwidth into NETWORKING_USAGE_TYPES into "\packages\azure\src\lib\ConsumptionTypes.ts" and results doesn't match Microsoft's results. It is the same as if it was not added into NETWORKING_USAGE_TYPES.

[C. G.]

UPDATE v2 :

I tried another time with "azure-subscription-chunks". I launch yarn start at 5:20PM and it stopped at 3:11 AM. So after 12 hours of working to generate a cache file filtered by month at 5:11AM.

So i compared another time results with PowerBI ( https://docs.microsoft.com/en-us/power-bi/connect-data/service-connect-to-emissions-impact-dashboard ) and my results are :

(date - 1st January to 31 March 2022)

• Virtual Machines
  • Estimates more CO2 in CCF than Power BI : 3.74 MTCO2 difference
    

• SQL Databse
  • Estimates more CO2 in CCF than Power BI : 2.276 MTCO2 difference

• Storage
  • Estimates more CO2 in PowerBI than CCF : 3.79 MTCO2 difference

• Bandwidth
  • Estimates more CO2 in CCF than PowerBI : 35.01 MTCO2 difference (the biggest !)
    

• Azure App Service
  • Estimates more CO2 in PowerBI than CCF : 3.78 MTCO2 difference

• Total Emission
  • PowerBI : 67.26 MTCO2
  • CCF :
    
    • With Bandwidth : 95.4 MTCO2
      
    • Without Bandwidth : 60.4 MTCO2
      

In conclusion :

Bandwidth service CO2 estimation is oversized but i realize that i didn't add name of service in NETWORKING_USAGE_TYPES or maybe in UNKNOWN_SERVICES ... I will retry another time to generate a cache-file and compare results.

I got CO2 difference between 1.41 MTCO2 and 3.32 MTCO2 for this period that accumulates every month. So, in prevision at the end of the year, a gap of 24 Tons should appear. So I have to keep in mind that i may consume more CO2 than I thought.

Branch "Azure-subscription-chunks" work well and I will use these modifications to run with my application. But I have to be patient because of the high number of account ! So to answer my first post, I will not use anymore "yarn-seed-cache-file" to generate Azure Data.

[Cloud Carbon Footprint]

Hi Clement,

Thank you so much for using Cloud Carbon Footprint as well as for sharing your insights and findings that you've come across during your setup! The level of detail is extremely helpful and we appreciate when people find additional unsupported usage types to add for Azure.

Here are a few notes I have in regards to some of the things that came up:

• We have been actively investigating a more optimized approach for querying billing data from Azure. Due to the rate limits of the Consumption Management API, we've noticed that performance has been a concern for those with large Azure usage. We're hoping to have a refactored approach within the near future and will release more information soon. A quick fix for now may be to increase the retry limit for your app or configuring your application to query for only a specific number of subscriptions at a time. Let me know if you have a deeper interest or trouble with either of these workarounds and I'd be happy to provide more detail.
• I've noticed that you were attempting to pull estimates using the 'azure-subscriptions-chunks' branch of the CCF app. I would highly recommend not doing this it is a developmental branch that was a previous spike attempt at improving the performance of the Azure API calls, and the branch is over several commits/releases behind the main version of the app. As a result, it is not intended for production use as you may not be getting the most up-to-date estimates due to missing several improvements that have been made to out methodology since the creation of that branch. I'd recommend retrying with the main branch using one of the fixes above to see if you get better estimates.
• Thanks for sharing your findings on the difference between Azure Emission's Impact Dashboard! You'll find that Power BI, as well as the other first-party carbon footprint tools, will not have exact matching numbers for their carbon estimates. This is due to a difference in methodology amongst other factors that would not make this possible. We do, however, try to aim to be within a reasonable ballpark of their estimates. Please keep this in mind as you continue to compare findings. We aim on releasing a more in-depth analysis regarding the comparison of each tool in the near future! It is actually recommended that you use CCF in combination with first-party cloud provider tools to have the most insight into your carbon impact.
• Going off of my last point, the overestimates of some of your services are definitely concerning. My initial impression is that Azure Logic Apps for instance could be classified under the wrong usage type. Based on the pricing unit, it seems to be better considered as a Compute usage type. However, I'm happy to dive into this some more.

Hopefully, this clears up some of your questions. If you're interested, we'd greatly appreciate it if you could consider creating a pull request for the usage types that you've added with appropriate estimates! Otherwise, we'd be happy to add them on your behalf. Additionally, our team is open to setting up a call if you're interested in discussing or troubleshooting further. Just let us know and we can discuss options for a time.

Best,

Arik

[C. G.]

Hello Arik,

thank you for your response ! 

I'm very interested about your first point concerning the retry limit and to query for a specific number of subscription. I will appreciate your help about this point. 

Ok for the branch "azure-subscription-chunks". I hope the modifications will work so i will definitively stop use this branch for my Azure Account. Actually, I cloned the Application to use one with master branch, and another with "azure-chunks" branch to create a "estimation.cache.month.json" file to copy into the application that use "master" branch. 
The problem actually is that i kept AWS credentials into my /api/.env, so it write twice dates and can't show AWS + Azure line chart ... it is ok when I want to show independently AWS or Azure. I hope later that with one query, it could resolve this issue.

For the Pull Request about usage types, you can add them on my behalf. 

And another request, I want to deploy the application into a Web App (not a VM) in Azure. I'm trying with my team to deploy it but it is not working actually. Do you/someone already deployed it and can help me ? 

Thank you very much ! 

[C. G.]

Hello Arik,

I hope you're doing well.

I'm still looking for help for query less subscription at a time you mentionned in your first point.

I hope your work on improving Azure part is on the good way.

Thank you for your help !

[Cloud Carbon Footprint]

Hi Clement,

My apologies for the delayed response and thanks for following up on this! We are hard at work at several performance improvements at the moment and hope to share details soon.

In regards to Azure, yes, there is still a manual fix for querying specific subscriptions at a time. This can be done by going to the packages/azure/src/application/AzureAccount.ts then navigating to the getDataFromConsumptionManagement function (currently line 52 on the main branch). In this function, there is a constant array called subscriptions that is eventually populated using the for-loop that follows it to add all of the subscriptions associated with the Azure account you provided. You can instead comment out that for-loop, and initialize the subscriptions variable with your own list of subscription ID strings. Doing so will allow for the application to query usage only for the subscriptions you provided, which should yield faster results due to the smaller scope of data. You can use this method to query for certain subscriptions at a time, which in turn should populate the cache, and eventually get you the wider scope of data you need without the throttling.

As a side note, a contributor is currently working on a more user-friendly approach for doing this type of filtering. It will be doing the same thing, but via an API param with provided IDs rather than a manual code change. You can track the progress of that PR here: https://github.com/cloud-carbon-footprint/cloud-carbon-footprint/pull/853

You're welcome to wait on this PR to be merged if you don't feel comfortable making the change yourself. Let me know if that helps or if you have other questions!

Best, 

Arik

[C. G.]

Hello Arik and Thoughtworks ! 

Do you have any updates concerning the application ? 

About the PR N°853 concerning the feature for Azure Subscriptions quering, is someone is working on it ? Because the last update mentionned " ccasher unassigned 4upz on 2 Aug " and no one seem to work on it. We are a lot of people waiting for this feature :) 

Another point concerning co2 estimations between the latest branch & azure-subscription-chunks as i mentionned earlier in this conversation.
I'm trying to use only CLI command to retrieve data. But i met some issues. (command yarn start-cli )

Actually, I : 

• Only with CLI with branch "azure-subscription-chunks"
  • Collect good Azure data
  • Can't collect AWS data
• Only with CLI with branch "latest"
  • Collect wrong Azure data
  • Collect good AWS data

I hope in the future that CLI function in latest branch will display data that are on "azure-subscription-chunks" :) 

[Cloud Carbon Footprint]

Hi Clement,

Thanks for your patience.
We have been busy working on the improvements to add a Mongo implementation to aid with performance issues.
We will review the Azure Subscription issue/PR as soon as we can and investigate the CLI issue you described.

Thanks,

The Cloud Carbon Footprint team at Thoughtworks

[C. G.]

Hello Cloud Carbon Footprint Team ! :D 

I hope your work on MongoDB is improving and on a good way ! 👍

I also see Arik create PR #906 appear just after my message on 8th September, thank you !  : https://github.com/cloud-carbon-footprint/cloud-carbon-footprint/issues/906 . I just post the link here to find it easily :) 

There is something new about the PR 853 & PR 906 ? 

Thanks,

Clement.

[Cloud Carbon Footprint]

Hey Clement,

Thanks - Mongo is here and we will be sending release notes very shortly!
The tickets you reference are still on the board and we are updating priorities for our team daily. We invite you to follow those tickets so you're updated if they move!

Cheers,
The Cloud Carbon Footprint Team at Thoughtworks