Deployment of Cloud Carbon Footprint - Scaling issues

[Dan Laird]

Good afternoon. Here at OVO we are committed to reducing our Carbon Footprint and being Carbon Neutral in our operations by 2030.  
Cloud represents a growing amount of our Carbon footprint and this tool really jumped out at us.
We have run 2 'Green Day/Hackathons' with a small team and made real progress in getting the Dashboard running, and more recently getting the Backstage plugin running.

However, we have run into some difficulties and rather than raise GitHub Issues - we wondered if there might be a better way to share our challenges and see if together some progress could be made.
We are primarily concerned about the size of our data set, and whether the app can scale to support us.
We currently are linking to just our AWS usage (400+ accounts) - and the dashboard is not able to operate - for example the largest issue we are encountering is NodeJS issues (Heap Out Of Memory Exceptions)
We also hit issues with Seed File generation (when grouping by Day - FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory)

We have explored some of the recent changes with MongoDB caching, and other improvements but they have not allowed us to make much more progress.

If you would be open to a conversation to see how we might make progress then please let us know.

Many thanks
Dan

[Cloud Carbon Footprint]

Hi Dan,

Thanks for reaching out! It is exciting to hear about your efforts at OVO in the Green Cloud space! Currently, our efforts to build around scalability and combat the memory issues for NodeJS revolve around the implementation of MongoDB. We are close to issuing a CCF release that will provide full integration for MongoDB, which will enable you to gather data in chunks by paginating the API request. This feature will not improve the length of time it takes to query AWS, which is why we suggest seeding your cache file (MongoDB) one day at a time for large amounts of data. The implementation of MongoDB should also improve seeding the cache process.

Regarding our UI, it is not currently built to scale with large amounts of data. For this reason at this time, we would suggest only using the UI dashboards to analyze smaller data time frames, or potentially using a separate tool like Data Studio to consume the CCF data and handle larger data sizes.

We would love to learn more about your goals and commitments at OVO and would be very interested to connect if you would like to hop on a call and discuss in more detail!

Thanks,

CCF Team