Deepcopy error when trying to airflow clear

573 views

Skip to first unread message

Laura Lorenz

unread,

Dec 22, 2015, 1:40:02 PM12/22/15

to Airflow

Hello all! I am running Airflow v 1.5.1 (still working on upgrading to 1.6.x) and have run into an error on our production machine. When I try to clear tasks either through the UI or through the command line with `airflow clear`, I get an error with deepcopy. The traceback (cleaned up a bit) is below:

root@730602c41b19:/datadive# airflow clear -t t28_push_audience_campaign_to_neo -d -s 2015-12-21T00:00:00 -e 2015-12-21T00:00:00 data_etl


[ ... took out a bunch of DAG loading stuff like below for other DAGs ... ]


2015-12-22 18:29:43,870 INFO - Loaded DAG <DAG: data_etl>

2015-12-22 18:29:43,870 INFO - Importing /datadive/dags/ingestion_DAG.py


[ ... took out more DAG loading stuff as above for more DAGs ... ]


Traceback (most recent call last):

  File "/usr/local/bin/airflow", line 10, in <module>

    args.func(args)

  File "/usr/lib/python2.7/dist-packages/airflow/bin/cli.py", line 226, in clear

    include_upstream=args.upstream,

  File "/usr/lib/python2.7/dist-packages/airflow/models.py", line 2072, in sub_dag

    dag = copy.deepcopy(self)

  File "/usr/lib/python2.7/copy.py", line 174, in deepcopy

    y = copier(memo)

  File "/usr/lib/python2.7/dist-packages/airflow/models.py", line 2057, in __deepcopy__

    setattr(result, k, copy.deepcopy(v, memo))

  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy

    y = copier(x, memo)

  File "/usr/lib/python2.7/copy.py", line 230, in _deepcopy_list

    y.append(deepcopy(a, memo))

  File "/usr/lib/python2.7/copy.py", line 174, in deepcopy

    y = copier(memo)

  File "/usr/lib/python2.7/dist-packages/airflow/models.py", line 1482, in __deepcopy__

    setattr(result, k, copy.deepcopy(v, memo))

  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy

    y = copier(x, memo)

  File "/usr/lib/python2.7/copy.py", line 230, in _deepcopy_list

    y.append(deepcopy(a, memo))

  File "/usr/lib/python2.7/copy.py", line 174, in deepcopy

    y = copier(memo)

  File "/usr/lib/python2.7/dist-packages/airflow/models.py", line 1482, in __deepcopy__

    setattr(result, k, copy.deepcopy(v, memo))

  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy

    y = _reconstruct(x, rv, 1, memo)

  File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct

    state = deepcopy(state, memo)

  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy

    y = copier(x, memo)

  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict

    y[deepcopy(key, memo)] = deepcopy(value, memo)

  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy

    y = _reconstruct(x, rv, 1, memo)

  File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct

    state = deepcopy(state, memo)

  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy

    y = copier(x, memo)

  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict

    y[deepcopy(key, memo)] = deepcopy(value, memo)

  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy

    y = _reconstruct(x, rv, 1, memo)

  File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct

    state = deepcopy(state, memo)

  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy

    y = copier(x, memo)

  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict

    y[deepcopy(key, memo)] = deepcopy(value, memo)

  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy

    y = _reconstruct(x, rv, 1, memo)

  File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct

    state = deepcopy(state, memo)

  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy

    y = copier(x, memo)

  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict

    y[deepcopy(key, memo)] = deepcopy(value, memo)

  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy

    y = copier(x, memo)

  File "/usr/lib/python2.7/copy.py", line 298, in _deepcopy_inst

    state = deepcopy(state, memo)

  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy

    y = copier(x, memo)

  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict

    y[deepcopy(key, memo)] = deepcopy(value, memo)

  File "/usr/lib/python2.7/copy.py", line 174, in deepcopy

    y = copier(memo)

TypeError: cannot deepcopy this pattern object

Same via the UI. Here are the web logs:









2015-12-22 18:22:14,752 - tornado.access - INFO - 200 GET /admin/airflow/graph?execution_date=2015-12-21+00%3A00%3A00&arrange=TB&root=&dag_id=data_etl (73.212.236.134) 31.69ms

2015-12-22 18:22:19,515 - airflow.www.app - ERROR - Exception on /admin/airflow/action [GET]

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1817, in wsgi_app

    response = self.full_dispatch_request()

  File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1477, in full_dispatch_request

    rv = self.handle_user_exception(e)

  File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1381, in handle_user_exception

    reraise(exc_type, exc_value, tb)

  File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1475, in full_dispatch_request

    rv = self.dispatch_request()

  File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1461, in dispatch_request

    return self.view_functions[rule.endpoint](**req.view_args)

  File "/usr/lib/python2.7/dist-packages/flask_admin/base.py", line 68, in inner

    return self._run_view(f, *args, **kwargs)

  File "/usr/lib/python2.7/dist-packages/flask_admin/base.py", line 359, in _run_view

    return fn(self, *args, **kwargs)

  File "/usr/lib/python2.7/dist-packages/airflow/www/app.py", line 925, in action

    include_upstream=upstream)

  File "/usr/lib/python2.7/dist-packages/airflow/models.py", line 2072, in sub_dag

    dag = copy.deepcopy(self)

  File "/usr/lib/python2.7/copy.py", line 174, in deepcopy

    y = copier(memo)

  File "/usr/lib/python2.7/dist-packages/airflow/models.py", line 2057, in __deepcopy__

    setattr(result, k, copy.deepcopy(v, memo))

  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy

    y = copier(x, memo)

  File "/usr/lib/python2.7/copy.py", line 230, in _deepcopy_list

    y.append(deepcopy(a, memo))

  File "/usr/lib/python2.7/copy.py", line 174, in deepcopy

    y = copier(memo)

  File "/usr/lib/python2.7/dist-packages/airflow/models.py", line 1482, in __deepcopy__

    setattr(result, k, copy.deepcopy(v, memo))

  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy

    y = copier(x, memo)

  File "/usr/lib/python2.7/copy.py", line 230, in _deepcopy_list

    y.append(deepcopy(a, memo))

  File "/usr/lib/python2.7/copy.py", line 174, in deepcopy

    y = copier(memo)

  File "/usr/lib/python2.7/dist-packages/airflow/models.py", line 1482, in __deepcopy__

    setattr(result, k, copy.deepcopy(v, memo))

  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy

    y = _reconstruct(x, rv, 1, memo)

  File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct

    state = deepcopy(state, memo)

  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy

    y = copier(x, memo)

  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict

    y[deepcopy(key, memo)] = deepcopy(value, memo)

  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy

    y = _reconstruct(x, rv, 1, memo)

  File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct

    state = deepcopy(state, memo)

  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy

    y = copier(x, memo)

  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict

    y[deepcopy(key, memo)] = deepcopy(value, memo)

  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy

    y = _reconstruct(x, rv, 1, memo)

  File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct

    state = deepcopy(state, memo)

  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy

    y = copier(x, memo)

  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict

    y[deepcopy(key, memo)] = deepcopy(value, memo)

  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy

    y = _reconstruct(x, rv, 1, memo)

  File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct

    state = deepcopy(state, memo)

  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy

    y = copier(x, memo)

  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict

    y[deepcopy(key, memo)] = deepcopy(value, memo)

  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy

    y = copier(x, memo)

  File "/usr/lib/python2.7/copy.py", line 298, in _deepcopy_inst

    state = deepcopy(state, memo)

  File "/usr/lib/python2.7/copy.py", line 163, in deepcopy

    y = copier(x, memo)

  File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict

    y[deepcopy(key, memo)] = deepcopy(value, memo)

  File "/usr/lib/python2.7/copy.py", line 174, in deepcopy

    y = copier(memo)

TypeError: cannot deepcopy this pattern object

2015-12-22 18:22:19,516 - tornado.access - ERROR - 500 GET /admin/airflow/action?action=clear&task_id=t28_push_audience_campaign_to_neo&dag_id=data_etl&future=false&past=false&upstream=false&downstream=true&execution_date=2015-12-21T00:00:00&3A8080%2Fadmin%2Fairflow%2Fgraph%3Fexecution_date%3D2015-12-21%2B00%253A00%253A00%26arrange%3DTB%26root%3D%26dag_id%3Ddata_etl (73.212.236.134) 4.84ms

My QA server (also running airflow 1.5.1) can clear fine from UI or command line. According to my google fu this has something to do with the task instance regex/re in general, but anyone with intricate knowledge of the airflow clear procedure please let me know if you have seen this before or what it means.

Thanks!! As always much thanks for airflow :D

Laura

Maxime Beauchemin

unread,

Dec 22, 2015, 5:54:32 PM12/22/15

to Airflow

There's an object that's not deep-copyable in your DAG. Probably something you passed to params or callback.