Using MongoDB as caching option together with Docker Compose

[Mukmin Pitoyo]

Hi CCF team,

First I would like to say thank you for your great work so far! 

I would like to start using MongoDB as my main caching method as recommended by you guys. In my current setup, I am using the Docker approach that connects to my AWS data, and it uses the Docker Secrets, etc. as laid out in the "Run with Docker" page of the documentation.

I have already setup the MongoDB variables in my packages/api/.env file as shown below:

And when I run 'yarn create-docker-secrets' again, I verified that the respective variables have been changed correctly:

However, when I run 'docker-compose up' again, I see that the API Docker container still uses the local cache file within the container:

Which I double-checked it to be true because when I SSH into the Docker API container, I see the "estimates.cache.day.json" file containing all my recent emissions data that I assume was pulled from CUR reports through AWS Athena. 

My question would be how can I direct the Docker API container to use MongoDB's caching instead of the local cache? Or is the current setup only restricted to using MongoDB caching through the yarn approach? 

My organization's direction would probably prefer the Docker containers approach since we are concerned about scalability issues in the future since there will be a large number of accounts we intend to add to this CCF, and prefer to run the API container and client container separately

Thank you and I hope to hear from you soon!

Best Regards,

Mukmin

[Cloud Carbon Footprint]

Hi Mukmin,

 The reason is that when MongoDB support was added to CCF, we didn't update the docker infrastructure. The `docker-compose.yml` file didn't carry credentials nor the configuration necessary to enable MongoDB. We updated the `docker-compose.yml` to show an example of setting the connection with MongoDB. In short, CCF relies on `CACHE_MODE` configuration to select the caching method and the default is 'local'. By setting the environment variable `CACHE_MODE` to `MONGODB` CCF will attempt to use MongoDB.

Thanks, sorry for the delayed response,

The Cloud Carbon Footprint team at Thoughtworks.

[Mukmin Pitoyo]

Hi CCF Team,

Thank you and I appreciate the clarification. May I double-check on the 'docker-compose.yml' file you are referring to, is it already pushed onto the GitHub repo? Currently on GitHub it looks the same as the previous version, I don't see the changes in setting the connection with MongoDB yet.

Thanks!

Best Regards,

Mukmin

[Cloud Carbon Footprint]

I'm so sorry, I forgot to push the changes. The updated file can be found here: https://github.com/cloud-carbon-footprint/cloud-carbon-footprint/blob/c7231420289985f8e00a0815bb0672812af5dcfb/docker-compose.yml

Cheers,

The Cloud Carbon Footprint Team at Thoughtworks

[Mukmin Pitoyo]

Thank you for the prompt response, appreciate it very much!

Using the updated docker-compose.yml file with the changes, I can see that the Docker API container now uses the MongoDB caching method instead of the local cache. 

I have a follow-up question, after setting up MongoDB Atlas and having set the correct URI, how does CCF pull estimates from Athena and store the cache in MongoDB for the first time? According to the documentation, it says:

 "After calculating estimates for the first time, the app will create a new collection titled “ccf”. Estimates will be separated into timestamps and stored into a collection that is named according to the current grouping method (i.e. “estimates-by-month)."

I assume CCF will call Athena DB at least once to get fresh estimates and create a new collection/database called "ccf"? But how would I know it would be successful?

I suspect it was not successful the first time, which leads to this error now:

Thank you!

[Cloud Carbon Footprint]

Hi Mukmin,

Glad to hear that your Docker issue was resolved!

To clarify the documentation on the data saving process: usage data is retrieved from the cloud provider (e.g. Athena), then used to calculates estimates, then the resulting estimates are stored to your MongoDB instance before being returned as a response.

You can verify if it's successful by checking your MongoDB instance and checking for a database named "ccf". Inside that database, there should be a collection called "estimates-by-[grouping-method]" where your estimates should be stored.

Judging by the warnings leading up to your error, it seems there is something wrong with the MongoDB URI that you configured. The message says: "URI cannot contain options with no value". This could be leading to the connection error that you're seeing on the last line -- meaning that it was most likely unsuccessful in saving your estimates. Try to double-check that your URI is valid and give it another go. 

Hope that helps!

The Cloud Carbon Footprint Team at Thoughtworks

[Mukmin Pitoyo]

Hi CCF Team,

Thank you for your reply, everything is working now! Docker containers setup can now work with MongoDB as the caching option.

I've double-checked, and yes you are right, seems like the issue is that when I run 'yarn create-docker-secrets', the MongoDB URI is not reflected exactly as what I configured in the .env file.

For e.g. in the .env file, the MongoDB URI is mongodb+srv://ABCD123:<password>@ABCD123.ABCD123.mongodb.net/?retryWrites=true&w=majority

But after running the 'yarn create-docker-secrets' command, it returns mongodb+srv://ABCD123:<password>@ABCD123.ABCD123.mongodb.net/?retryWrites=

But yes just a small issue, updated the link without the '/?retryWrites=true&w=majority' variable string and it works now.

Thanks for the help, really appreciate it!

[Cloud Carbon Footprint]

Thanks for pointing that out, we've fixed the script to generate the credentials to account for multiple "=" in configuration.

Thanks,

The Cloud Carbon Footprint Team at Thoughtworks.