Question/Issue: Subdag not runnable

Tyrone Hinderson

unread,

Mar 15, 2016, 4:55:49 AM3/15/16

to Airflow

I am trying to run a simple subdag example similar to the one suggested in the airflow docs. Unfortunately, I am unable to "zoom into subdag" in the manner I would expect--by clicking the button indicated in the screenshot:

Furthermore, the dag as a whole doesn't seem to work. The child dag can't seem to run, let alone complete, and failed tasks have errors like this:

Enter code here...

I'm not quite sure how to proceed, since I'm running an example quite similar to that which is found in the docs (except mine can be interpreted using airflow 1.6.2). Does anyone have ideas as to what's going on or how best to find out?

Thanks!

Tyrone Hinderson

unread,

Mar 15, 2016, 4:56:42 AM3/15/16

to Airflow

So sorry, I meant to copy the error message above:

[2016-03-15 04:23:56,021] {models.py:974} INFO - Executing <Task(SubDagOperator): child_dag> on 2016-03-13 00:00:00
[2016-03-15 04:23:56,032] {models.py:1041} ERROR - [Errno 35] Resource temporarily unavailable
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1000, in run
    result = task_copy.execute(context=context)
  File "/usr/local/lib/python2.7/site-packages/airflow/operators/subdag_operator.py", line 45, in execute
    executor=self.executor)
  File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 2472, in run
    job.run()
  File "/usr/local/lib/python2.7/site-packages/airflow/jobs.py", line 165, in run
    self._execute()
  File "/usr/local/lib/python2.7/site-packages/airflow/jobs.py", line 711, in _execute
    executor.start()
  File "/usr/local/lib/python2.7/site-packages/airflow/executors/local_executor.py", line 59, in start
    w.start()
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 130, in start
    self._popen = Popen(self)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/forking.py", line 121, in __init__
    self.pid = os.fork()

Chowduri Manjunatha

unread,

Mar 16, 2016, 4:59:46 AM3/16/16

to Airflow

Hi Tyrone,

I think you are getting "OSError: [Errno 12] Cannot allocate memory" error, If this is the error you are looking please go through the below URL to create swap space in your machine.

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-store-swap-volumes.html

Please let me know if you need more details.

Thanks,

Chowduri M

Tyrone Hinderson

unread,

Mar 16, 2016, 10:32:06 AM3/16/16

to Airflow

Hi Chowduri,

I have seen no indication that memory allocation is my problem--what gave you that idea?

Moreover, while memory allocation may have something to do with the task failure, it doesn't explain why the webserver can't display my child dag.

Chowduri Manjunatha

unread,

Mar 16, 2016, 10:39:30 AM3/16/16

to Airflow

When we were trying we got Memory allocation issue and I could see similar kind of error in your comment.

I think Maxi can help you, lets wait for the reply.

Maxime Beauchemin

unread,

Mar 16, 2016, 11:11:36 AM3/16/16

to Airflow

It's beyond me, looks like some system policy on that box is preventing from using Python's multiprocessing module. Being able to use that module is a requirement for Airflow.

Arthur Wiedmer

unread,

Mar 16, 2016, 11:58:04 AM3/16/16

to Airflow

Tyrone,

I might take a wild guess and say that you are running this on Mac OSX, correct?

If so, the usual limit is 100 processes. It is possible to bump into that limit. ulimit -a might show you the actual limit for your system.

A thing that might explain this : You set a high limit on concurrency for the localexecutor which works by spawning processes.

http://stackoverflow.com/questions/20075975/python-multiprocessing-pool-recovery-after-resource-temporarily-unavailable

You can up the limit of processes that can be forked, but I do not remember how to do it with OSX.

Best,

Arthur

Tyrone Hinderson

unread,

Mar 16, 2016, 2:17:28 PM3/16/16

to Airflow

Hi Arthur,

Indeed I am running the flow on Mac OS X. I have modified the DAG, making the subdag actually do something visible and allowing only one DAG instance to run at a time.

I am seeing the following logs, which lead me to suspect that the dag is being run as a backfill job (this blog post warns against this possibility):

[2016-03-16 13:48:44,507] {models.py:974} INFO - Executing <Task(SubDagOperator): child_dag> on 2016-03-01 00:00:00
[2016-03-16 13:48:44,568] {base_executor.py:34} INFO - Adding to queue: airflow run parent_dag.child_dag parent_dag.child_dag.print_task 2016-03-01T00:00:00 --local -s 2016-03-01T00:00:00 
[2016-03-16 13:48:49,521] {jobs.py:802} INFO - [backfill progress] waiting: 1 | succeeded: 0 | kicked_off: 1 | failed: 0 | wont_run: 0 
[2016-03-16 13:48:49,524] {local_executor.py:29} INFO - LocalWorker running airflow run parent_dag.child_dag parent_dag.child_dag.print_task 2016-03-01T00:00:00 --local -s 2016-03-01T00:00:00 
[2016-03-16 13:48:54,527] {jobs.py:788} ERROR - The airflow run command failed at reporting an error. This should not occur in normal circumstances. Task state is 'running',reported state is 'success'. TI is <TaskInstance: parent_dag.child_dag.parent_dag.child_dag.print_task 2016-03-01 00:00:00 [running]>
[2016-03-16 13:48:54,527] {jobs.py:802} INFO - [backfill progress] waiting: 1 | succeeded: 0 | kicked_off: 1 | failed: 0 | wont_run: 0

Nothing actually happens as a result of running the child task--it simply hangs there. The only reason I saw the issues above was that at one point I was running too many instances of the disfunctional task.

Tyrone Hinderson

unread,

Mar 16, 2016, 2:47:03 PM3/16/16

to Airflow

Here is evidence that something is amiss with respect to my subdag being recognized by the scheduler:

I started the scheduler after completely resetting the airflow db, and saw the following in the logs:

[2016-03-16 14:41:35,878] {models.py:222} INFO - Finding 'running' jobs without a recent heartbeat
[2016-03-16 14:41:35,879] {models.py:228} INFO - Failing jobs without heartbeat after 2016-03-16 14:39:20.879235
[2016-03-16 14:41:37,950] {__init__.py:36} INFO - Using executor LocalExecutor
[2016-03-16 14:41:37,953] {__init__.py:36} INFO - Using executor LocalExecutor

Traceback (most recent call last):

  File "/usr/local/bin/airflow", line 15, in <module>
    args.func(args)
  File "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 134, in run
    raise AirflowException(msg)
airflow.utils.AirflowException: DAG [parent_dag.child_dag] could not be found in /Users/thinderson/airflow/dags

Traceback (most recent call last):

  File "/usr/local/bin/airflow", line 15, in <module>
    args.func(args)
  File "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 134, in run
    raise AirflowException(msg)
airflow.utils.AirflowException: DAG [parent_dag.child_dag] could not be found in /Users/thinderson/airflow/dags

I believe my code avoids the pitfalls mentioned in the blog post, but perhaps someone here can see something wrong with what I'm doing? It may be worth noting that while the above error message only appears once when the parent dag is initially discovered, I do not believe the issue is resolved without intervention.

Tyrone Hinderson

unread,

Mar 16, 2016, 3:53:52 PM3/16/16

to Airflow

I believe I've found the problem--

from airflow.operators.subdag_operator import SubDagOperator

renders an operator with very different behavior from

from airflow.operators import SubDagOperator

Just try it yourself. The latter works as expected, whereas the former leads to the following, very confusing error:

Logging into: /Users/thinderson/airflow/logs/parent_dag/child_dag/2016-03-11T00:00:00
[2016-03-16 15:49:13,725] {__init__.py:36} INFO - Using executor SequentialExecutor
[2016-03-16 15:49:24,579] {__init__.py:36} INFO - Using executor SequentialExecutor

Traceback (most recent call last):

  File "/usr/local/bin/airflow", line 4, in <module>
    __import__('pkg_resources').run_script('airflow==1.6.2', 'airflow')
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 726, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1484, in run_script
    exec(code, namespace, namespace)
  File "/usr/local/lib/python2.7/site-packages/airflow-1.6.2-py2.7.egg/EGG-INFO/scripts/airflow", line 15, in <module>
    args.func(args)
  File "/usr/local/lib/python2.7/site-packages/airflow-1.6.2-py2.7.egg/airflow/bin/cli.py", line 167, in run

    raise AirflowException(msg)
airflow.utils.AirflowException: DAG [parent_dag.child_dag] could not be found in /Users/thinderson/airflow/dags

I am able to use other operators seemingly without incident, so I am perplexed as to why this import dichotomy exists for SubDagOperator. I am currently running airflow from the HEAD of its master branch--formerly I was using the version shipped with `pip install`, but after reading about some recent subdag-related bugfixes, and otherwise at the end of my rope, I opted to try the bleeding edge.

I hope someone can shed light on this strange behavior, and I will use the working import in the meantime.

Thanks for the help up until now!

Reply all

Reply to author

Forward