cloud composer vcpu time

[salessteals]

I changed my DAGs a little bit this month to add some tasks to account for different edge cases and i noticed that our cloud composer costs have significantly increased, mostly because of Cloud Composer SQL vCPU time which is in addition to our normal Cloud Composer vCPU time. I don't use Cloud SQL at all, I am using the beta version of airflow 1.1 and i wonder if that is why the change in costs? Does anyone know why this is and how I can reduce this cost or is it not possible if I'm using the beta version (if that is why there's an increase in costs). 

[Feng Lu]

Could you share with me your environment details?

Thanks.

Feng 

On Thu, Feb 7, 2019 at 6:07 PM salessteals <saless...@gmail.com> wrote:

I changed my DAGs a little bit this month to add some tasks to account for different edge cases and i noticed that our cloud composer costs have significantly increased, mostly because of Cloud Composer SQL vCPU time which is in addition to our normal Cloud Composer vCPU time. I don't use Cloud SQL at all, I am using the beta version of airflow 1.1 and i wonder if that is why the change in costs? Does anyone know why this is and how I can reduce this cost or is it not possible if I'm using the beta version (if that is why there's an increase in costs). 

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/85d2168c-7ced-4426-920c-052816b90594%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[salessteals]

The one thing besides the airflow runtime i changed was max_active_runs_per_dag to avoid duplicate runs of the same file when triggered by a cloud function. This usually means i have a bunch of dag runs queued at a time because if 10 files come in at once, 9 are queued waiting to be run. If this is causing an upsurge in cost i can change this back. 

• Image version: composer-1.4.2-airflow-1.10.0
• Zone: us-central1-a
• Python version: 2

• Worker nodes

  Disk size (GB)

  20

  Machine type

  n1-standard-1

  Node count

  3

  Network ID

  default

  Subnetwork ID

  -

Image version

composer-1.4.2-airflow-1.10.0

Max_active_runs_per_dag : 1

The other airflow configuration settings are the ones that airflow 1.1 beta came with

Thanks so much!

On Friday, February 8, 2019 at 5:24:24 AM UTC-5, Feng Lu wrote:

Could you share with me your environment details?

Thanks.

Feng 

[salessteals]

any update on this? 

[Feng Lu]

You are running with Composer 1.4.2 (post-GA version) and we do use Cloud SQL for storing Airflow metadata database. Here's the how we bill Composer environments.

Hope it helps to clarify. 

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To post to this group, send email to cloud-compo...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/aec45197-e2a4-487c-bd89-7b25ece74a09%40googlegroups.com.

[Sydney Sarachek]

Hey Feng, thanks so much. How would I reduce this cost? 

[Sydney Sarachek]

I.e. would I decrease the # of tasks (I have some dummy tasks so I'm not sure if that makes it more costly) or I'm not sure 

[Feng Lu]

Unfortunately that won't help, we currently don't dynamically scale Composer environment with respect to DAG workload.