Using tool to estimate CO2e for AWS DataSync

[Mark T]

Hi,

My team is looking at trying to estimate the carbon footprint of using AWS DataSync (https://aws.amazon.com/datasync), and are wondering how to account for the DataSync service itself. We can do a network estimate for the data transfer, and a storage estimate for the source and target data stores, but are not sure how to account for the DataSync service itself (which must be using some compute power).

Any recommendations on how to handle this?

Thanks

Mark

[Dan Lewis-Toakley]

Hi Mark, 

Apologies for the delay. This is a tough one... Because you are only charged based on the number of GBs transferred (source), there doesn't seem to be any cost and usage rows other than that for DataSync. This is also what I'm seeing via Athena - a single row returns from the query Arelys provided:

AWSDataSync

APS3-Transferred-Bytes

GB

Now the docs also state: "You are charged standard request, storage, and data transfer rates to read from and write to AWS services, such as Amazon Simple Storage Service (Amazon S3), Amazon Elastic File System (Amazon  EFS), Amazon FSx for Windows File Server, and AWS Key Management Service (KMS)." -- so usage rows would also show up for those resources, but it may be difficult to tie it back to DataSync unless you have some tags set up to identify this. Right now, CCF doesn't support tags but we hope to in the future. 

However that still doesn't cover the "use of network acceleration technology, managed cloud infrastructure, data validation, and automation capabilities. " (same source). My advice would probably be either: 

1. Decide on a proxy for the number of vCPUs you expect might cover this, and work out how long the sync takes. With that information you could estimate the compute resources required. 
2. To potentially get more accurate, you should deploy the DataSync EC2 agent to an instance, and measure the time it takes to sync data between services for a given instance size. This should help you estimate the resources needed for the underlying managed service. But it would be more involved and maybe not worth it. 

I hope that helps a bit -- sorry it's not a clear cut answer. 

Best,

Dan  

On Tue, Oct 5, 2021 at 5:11 AM Mark Taylor <mta...@thoughtworks.com> wrote:

Hi Arelys,

Since we're trying to estimate what it would cost, we don't have any usage logs to query at the moment.

We are looking to run some trials, however, so I will see if it's possible to run that query then, though I'll also have to figure out how to generate the usage logs!

Thanks

Mark

On Thu, 30 Sept 2021 at 17:05, Arelys Alvarez <arelys....@thoughtworks.com> wrote:

Hi Mark, 

Sorry for the late response. In order for us to share the approach for estimating DataSync, would you be comfortable running the query below and providing us with the result so we can better help you with this.

Here is the query: 

`SELECT line_item_product_code AS serviceName, line_item_usage_type AS usageType, pricing_unit AS usageUnit, product_vcpu AS vCpus FROM <your-table-name> WHERE line_item_line_item_type IN ('Usage', 'DiscountedUsage', 'SavingsPlanCoveredUsage') AND line_item_product_code = 'AWSDataSync' GROUP BY 1,2,3,4`

Thank you, 

Arelys 

--
You received this message because you are subscribed to the Google Groups "Cloud Carbon Footprint" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-carbon-foot...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-carbon-footprint/6f10f4fb-01ac-4e2f-bbf7-740f86836153n%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "green-cloud-team" group.
To unsubscribe from this group and stop receiving emails from it, send an email to green-cloud-te...@thoughtworks.com.
To view this discussion on the web visit https://groups.google.com/a/thoughtworks.com/d/msgid/green-cloud-team/CA%2Br31fcKWjBzguOxmEHFeLUJdcoq7T-FgZkP_88p_sk-%2BsRuOw%40mail.gmail.com.

--

Dan Lewis-Toakley Green Cloud Lead, North America Pronouns  He/Him Email  dan.lewi...@thoughtworks.com Telephone  +19172545068

[Mark Taylor]

Thanks for those ideas Dan!