How to keep composer (airflow) DAGs metadata ?

[Jason Chen]

 Hi,


With GCP composer, I know it's using airflow and DB to save DAGs' metadata.
Is the metadata DB in GCS ? If the composer environment is gone (deleted),
is there a way to migrate the metadata DB to another new composer environment? 
I know the DAGs Python file is in GCS/dags/ and we can copy Python files from there,
but I am wondering the metadata DB.


Thanks.

[James Malone]

Hi Jason,

The metadata database for Cloud Composer lives in a tenant project and is managed and monitored by the Composer service. When an environment is deleted, so is that metadata database. Is your goal here to recover the database for a deleted environment, or to do an in-place switch to copy a (non-deleted) environment's metadata to a new environment?

The DAGs live in the Cloud Storage so they are not ephemeral with the environment.

Cheers,

James

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/b11de514-fa5d-4c35-965b-84063bfe9cde%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Raghuram Amala]

Hi James,

Can you advise how to clean up the metadata ie to delete dag runs history for better performance? Thanks !!

On Wednesday, May 30, 2018 at 11:40:43 AM UTC+10, James Malone wrote:

Hi Jason,

The metadata database for Cloud Composer lives in a tenant project and is managed and monitored by the Composer service. When an environment is deleted, so is that metadata database. Is your goal here to recover the database for a deleted environment, or to do an in-place switch to copy a (non-deleted) environment's metadata to a new environment?

The DAGs live in the Cloud Storage so they are not ephemeral with the environment.

Cheers,

James

On Tue, May 29, 2018 at 11:38 AM Jason Chen <chingch...@gmail.com> wrote:

 Hi,


With GCP composer, I know it's using airflow and DB to save DAGs' metadata.
Is the metadata DB in GCS ? If the composer environment is gone (deleted),
is there a way to migrate the metadata DB to another new composer environment? 
I know the DAGs Python file is in GCS/dags/ and we can copy Python files from there,
but I am wondering the metadata DB.


Thanks.

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.

[Wilson Lian]

Hi Raghuram,

You can delete rows from the Airflow database using a raw SQL query in the "Data Profiling" > "Ad Hoc Query" page in the Airflow UI. Select the "airflow_db" database from the dropdown menu and run "DELETE FROM dag_run;" to delete all rows. Make sure no DAG runs are in-flight so you don't orphan any task instances.

best,

Wilson

On Tue, Aug 6, 2019 at 10:06 PM Raghuram Amala <raghura...@gmail.com> wrote:

Hi James,

Can you advise how to clean up the metadata ie to delete dag runs history for better performance? Thanks !!

On Wednesday, May 30, 2018 at 11:40:43 AM UTC+10, James Malone wrote:

Hi Jason,

The metadata database for Cloud Composer lives in a tenant project and is managed and monitored by the Composer service. When an environment is deleted, so is that metadata database. Is your goal here to recover the database for a deleted environment, or to do an in-place switch to copy a (non-deleted) environment's metadata to a new environment?

The DAGs live in the Cloud Storage so they are not ephemeral with the environment.

Cheers,

James

On Tue, May 29, 2018 at 11:38 AM Jason Chen <chingch...@gmail.com> wrote:

 Hi,


With GCP composer, I know it's using airflow and DB to save DAGs' metadata.
Is the metadata DB in GCS ? If the composer environment is gone (deleted),
is there a way to migrate the metadata DB to another new composer environment? 
I know the DAGs Python file is in GCS/dags/ and we can copy Python files from there,
but I am wondering the metadata DB.


Thanks.

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.

To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/b11de514-fa5d-4c35-965b-84063bfe9cde%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/4a7320e6-1a52-4b9d-9ea1-65a0e3719997%40googlegroups.com.

[Raghuram Amala]

Hi Wilson,

Thanks for writing back on this. I somehow missed this email.

Can you advise how could i retain last 30 days data and delete the rest?

"DELETE FROM dag_run;"