Dags not populating the in webserver's DagBag Object

[sanjay...@shopkick.com]

Hi,

I am attempting to migrate to Beta and created an environment yesterday with my code.

My dag's don't seem to be loading properly on the web server for some reason. I see that the dag

This DAG isn't available in the web server's DagBag object. It shows up in this list because the scheduler marked it as active in the metadata database.

Is there a way for me to restart the webserver so that the dag bag can repopulate ?

[sanjay...@shopkick.com]

I think this could be an issue raised with airflow.

https://stackoverflow.com/questions/41560614/airflow-this-dag-isnt-available-in-the-webserver-dagbag-object

I think restarting the webserver helps (it helped when I created a managed webserver) . I dont know if there is any way to do that for the TP UI though.

Any information would be helpful

Regards,

Sanjay

[Trevor Edwards]

I've only seen this issue occur either (a) for a short duration after uploading a new DAG to the webserver as the DAG propagates or (b) if the DAG is removed.

Is the webserver still showing this issue, despite the DAG existing in your DAGs folder?

[Sanjay Kumar]

Yes. I still see this issue on the GAE UI. But it doesn't seem to happen in the managed webserver UI .

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/e05e7faf-dcdf-4a1a-b2f2-bd23c1533c3b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Sanjay Kumar | Data Platform Engineer

o:  m:  

[Trevor Edwards]

Interesting. Does the DAG depend on any programmatic generation for it structure? I.e. making an API call to define tasks

On Tuesday, July 17, 2018 at 8:59:41 AM UTC-7, Sanjay Kumar wrote:

Yes. I still see this issue on the GAE UI. But it doesn't seem to happen in the managed webserver UI .

On Tue, Jul 17, 2018 at 11:56 AM 'Trevor Edwards' via cloud-composer-discuss <cloud-compo...@googlegroups.com> wrote:

I've only seen this issue occur either (a) for a short duration after uploading a new DAG to the webserver as the DAG propagates or (b) if the DAG is removed.

Is the webserver still showing this issue, despite the DAG existing in your DAGs folder?

On Thursday, July 12, 2018 at 9:58:23 AM UTC-7, sanjay...@shopkick.com wrote:

I think this could be an issue raised with airflow.

https://stackoverflow.com/questions/41560614/airflow-this-dag-isnt-available-in-the-webserver-dagbag-object

I think restarting the webserver helps (it helped when I created a managed webserver) . I dont know if there is any way to do that for the TP UI though.

Any information would be helpful

Regards,

Sanjay

On Thursday, July 12, 2018 at 11:15:27 AM UTC-4, sanjay...@shopkick.com wrote:

Hi,

I am attempting to migrate to Beta and created an environment yesterday with my code.

My dag's don't seem to be loading properly on the web server for some reason. I see that the dag

This DAG isn't available in the web server's DagBag object. It shows up in this list because the scheduler marked it as active in the metadata database.

Is there a way for me to restart the webserver so that the dag bag can repopulate ?

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.

To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/e05e7faf-dcdf-4a1a-b2f2-bd23c1533c3b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sanjay Kumar]

We are building the dag of configs present in a config file which is stored in data folder. There is no API call to any other application to build the dag.

On Tue, Jul 17, 2018 at 12:03 PM 'Trevor Edwards' via cloud-composer-discuss <cloud-compo...@googlegroups.com> wrote:

Interesting. Does the DAG depend on any programmatic generation for it structure? I.e. making an API call to define tasks

On Tuesday, July 17, 2018 at 8:59:41 AM UTC-7, Sanjay Kumar wrote:

Yes. I still see this issue on the GAE UI. But it doesn't seem to happen in the managed webserver UI .

On Tue, Jul 17, 2018 at 11:56 AM 'Trevor Edwards' via cloud-composer-discuss <cloud-compo...@googlegroups.com> wrote:

I've only seen this issue occur either (a) for a short duration after uploading a new DAG to the webserver as the DAG propagates or (b) if the DAG is removed.

Is the webserver still showing this issue, despite the DAG existing in your DAGs folder?

On Thursday, July 12, 2018 at 9:58:23 AM UTC-7, sanjay...@shopkick.com wrote:

I think this could be an issue raised with airflow.

https://stackoverflow.com/questions/41560614/airflow-this-dag-isnt-available-in-the-webserver-dagbag-object

I think restarting the webserver helps (it helped when I created a managed webserver) . I dont know if there is any way to do that for the TP UI though.

Any information would be helpful

Regards,

Sanjay

On Thursday, July 12, 2018 at 11:15:27 AM UTC-4, sanjay...@shopkick.com wrote:

Hi,

I am attempting to migrate to Beta and created an environment yesterday with my code.

My dag's don't seem to be loading properly on the web server for some reason. I see that the dag

This DAG isn't available in the web server's DagBag object. It shows up in this list because the scheduler marked it as active in the metadata database.

Is there a way for me to restart the webserver so that the dag bag can repopulate ?

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.

To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/e05e7faf-dcdf-4a1a-b2f2-bd23c1533c3b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Sanjay Kumar | Data Platform Engineer

o:  m:  

--

You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.

To post to this group, send email to cloud-compo...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/ab2d3cb5-1bac-4eff-9ec1-0f377818c090%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Trevor Edwards]

The data folder is only mounted on workers. You will have to put the config file in your DAGs folder for the webserver to see it.

On Tuesday, July 17, 2018 at 9:34:21 AM UTC-7, Sanjay Kumar wrote:

We are building the dag of configs present in a config file which is stored in data folder. There is no API call to any other application to build the dag.

On Tue, Jul 17, 2018 at 12:03 PM 'Trevor Edwards' via cloud-composer-discuss <cloud-compo...@googlegroups.com> wrote:

Interesting. Does the DAG depend on any programmatic generation for it structure? I.e. making an API call to define tasks

On Tuesday, July 17, 2018 at 8:59:41 AM UTC-7, Sanjay Kumar wrote:

Yes. I still see this issue on the GAE UI. But it doesn't seem to happen in the managed webserver UI .

On Tue, Jul 17, 2018 at 11:56 AM 'Trevor Edwards' via cloud-composer-discuss <cloud-compo...@googlegroups.com> wrote:

I've only seen this issue occur either (a) for a short duration after uploading a new DAG to the webserver as the DAG propagates or (b) if the DAG is removed.

Is the webserver still showing this issue, despite the DAG existing in your DAGs folder?

On Thursday, July 12, 2018 at 9:58:23 AM UTC-7, sanjay...@shopkick.com wrote:

I think this could be an issue raised with airflow.

https://stackoverflow.com/questions/41560614/airflow-this-dag-isnt-available-in-the-webserver-dagbag-object

I think restarting the webserver helps (it helped when I created a managed webserver) . I dont know if there is any way to do that for the TP UI though.

Any information would be helpful

Regards,

Sanjay

On Thursday, July 12, 2018 at 11:15:27 AM UTC-4, sanjay...@shopkick.com wrote:

Hi,

I am attempting to migrate to Beta and created an environment yesterday with my code.

My dag's don't seem to be loading properly on the web server for some reason. I see that the dag

This DAG isn't available in the web server's DagBag object. It shows up in this list because the scheduler marked it as active in the metadata database.

Is there a way for me to restart the webserver so that the dag bag can repopulate ?

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.

To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/e05e7faf-dcdf-4a1a-b2f2-bd23c1533c3b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Sanjay Kumar | Data Platform Engineer

o:  m:  

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.

To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/ab2d3cb5-1bac-4eff-9ec1-0f377818c090%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Scott Jungwirth]

On Wednesday, July 18, 2018 at 7:57:31 AM UTC-7, Trevor Edwards wrote:

The data folder is only mounted on workers. You will have to put the config file in your DAGs folder for the webserver to see it.

Wow I never would have figured this out myself. I had a firebase credentials file in the data folder and was iterating over some config data to create dynamic DAGs and I figured there was a problem pulling the firebase data but couldn't find any logs or other info to confirm and testing locally worked fine.