Clearing task in UI causes TypeError: can't pickle thread.lock objects
2,792 views
Skip to first unread message
Terrence Szymanski
May 16, 2018, 7:04:22 PM5/16/18
to cloud-composer-discuss
Every once in a while when I try to clear a task in the web UI (click on a task in the graph view, then click "Clear"), I get an error message like the one below. It is intermittent and sometimes if I try again, I won't get this error, and sometimes it seems like the clear was successful despite getting this error message.
____/ ( ( ) ) \___ /( ( ( ) _ )) ) )\ (( ( )( ) ) ( ) ) ((/ ( _( ) ( _) ) ( () ) ) ( ( ( (_) (( ( ) .((_ ) . )_ ( ( ) ( ( ) ) ) . ) ( ) ( ( ( ( ) ( _ ( _) ). ) . ) ) ( ) ( ( ( ) ( ) ( )) ) _)( ) ) ) ( ( ( \ ) ( (_ ( ) ( ) ) ) ) )) ( ) ( ( ( ( (_ ( ) ( _ ) ) ( ) ) ) ( ( ( ( ( ) (_ ) ) ) _) ) _( ( ) (( ( )( ( _ ) _) _(_ ( (_ ) (_((__(_(__(( ( ( | ) ) ) )_))__))_)___) ((__) \\||lll|l||/// \_)) ( /(/ ( ) ) )\ ) ( ( ( ( | | ) ) )\ ) ( /(| / ( )) ) ) )) ) ( ( ((((_(|)_))))) ) ( ||\(|(|)|/|| ) ( |(||(||)|||| ) ( //|/l|||)|\\ \ ) (/ / // /|//||||\\ \ \ \ _) ------------------------------------------------------------------------------- Node: airflow-webserver-7b48bcf85c-snw28 ------------------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1988, in wsgi_app response = self.full_dispatch_request() File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1641, in full_dispatch_request rv = self.handle_user_exception(e) File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1544, in handle_user_exception reraise(exc_type, exc_value, tb) File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1639, in full_dispatch_request rv = self.dispatch_request() File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1625, in dispatch_request return self.view_functions[rule.endpoint](**req.view_args) File "/usr/local/lib/python2.7/site-packages/flask_admin/base.py", line 69, in inner return self._run_view(f, *args, **kwargs) File "/usr/local/lib/python2.7/site-packages/flask_admin/base.py", line 368, in _run_view return fn(self, *args, **kwargs) File "/usr/local/lib/python2.7/site-packages/flask_login.py", line 755, in decorated_view return func(*args, **kwargs) File "/usr/local/lib/python2.7/site-packages/airflow/www/utils.py", line 262, in wrapper return f(*args, **kwargs) File "/usr/local/lib/python2.7/site-packages/airflow/www/utils.py", line 309, in wrapper return f(*args, **kwargs) File "/usr/local/lib/python2.7/site-packages/airflow/www/views.py", line 989, in clear include_upstream=upstream) File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 3526, in sub_dag dag = copy.deepcopy(self) File "/usr/local/lib/python2.7/copy.py", line 174, in deepcopy y = copier(memo) File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 3511, in __deepcopy__ setattr(result, k, copy.deepcopy(v, memo)) File "/usr/local/lib/python2.7/copy.py", line 174, in deepcopy y = copier(memo) File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 3511, in __deepcopy__ setattr(result, k, copy.deepcopy(v, memo)) File "/usr/local/lib/python2.7/copy.py", line 163, in deepcopy y = copier(x, memo) File "/usr/local/lib/python2.7/copy.py", line 257, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/usr/local/lib/python2.7/copy.py", line 174, in deepcopy y = copier(memo) File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 2436, in __deepcopy__ setattr(result, k, copy.deepcopy(v, memo)) File "/usr/local/lib/python2.7/copy.py", line 190, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/usr/local/lib/python2.7/copy.py", line 334, in _reconstruct state = deepcopy(state, memo) File "/usr/local/lib/python2.7/copy.py", line 163, in deepcopy y = copier(x, memo) File "/usr/local/lib/python2.7/copy.py", line 257, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/usr/local/lib/python2.7/copy.py", line 190, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/usr/local/lib/python2.7/copy.py", line 334, in _reconstruct state = deepcopy(state, memo) File "/usr/local/lib/python2.7/copy.py", line 163, in deepcopy y = copier(x, memo) File "/usr/local/lib/python2.7/copy.py", line 257, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/usr/local/lib/python2.7/copy.py", line 190, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/usr/local/lib/python2.7/copy.py", line 334, in _reconstruct state = deepcopy(state, memo) File "/usr/local/lib/python2.7/copy.py", line 163, in deepcopy y = copier(x, memo) File "/usr/local/lib/python2.7/copy.py", line 257, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/usr/local/lib/python2.7/copy.py", line 190, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/usr/local/lib/python2.7/copy.py", line 334, in _reconstruct state = deepcopy(state, memo) File "/usr/local/lib/python2.7/copy.py", line 163, in deepcopy y = copier(x, memo) File "/usr/local/lib/python2.7/copy.py", line 257, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/usr/local/lib/python2.7/copy.py", line 163, in deepcopy y = copier(x, memo) File "/usr/local/lib/python2.7/copy.py", line 230, in _deepcopy_list y.append(deepcopy(a, memo)) File "/usr/local/lib/python2.7/copy.py", line 190, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/usr/local/lib/python2.7/copy.py", line 334, in _reconstruct state = deepcopy(state, memo) File "/usr/local/lib/python2.7/copy.py", line 163, in deepcopy y = copier(x, memo) File "/usr/local/lib/python2.7/copy.py", line 257, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/usr/local/lib/python2.7/copy.py", line 190, in deepcopy y = _reconstruct(x, rv, 1, memo) File "/usr/local/lib/python2.7/copy.py", line 334, in _reconstruct state = deepcopy(state, memo) File "/usr/local/lib/python2.7/copy.py", line 163, in deepcopy y = copier(x, memo) File "/usr/local/lib/python2.7/copy.py", line 257, in _deepcopy_dict y[deepcopy(key, memo)] = deepcopy(value, memo) File "/usr/local/lib/python2.7/copy.py", line 182, in deepcopy rv = reductor(2) TypeError: can't pickle thread.lock objects
seba...@iihnordic.com
Nov 28, 2018, 5:27:27 AM11/28/18
to cloud-composer-discuss
Hello Terrence! I know this is an older thread, but I encountered the exact same problem today, and found out that its most likely a UI bug.
I resolved the problem by going to 'Browse' -> 'Task Instances' -> check the instances that I wanted to clear -> With selected -> clear.
Hope this resolve it for others running into the issue
Dennis O'Brien
Mar 1, 2019, 4:01:52 PM3/1/19
to cloud-composer-discuss
I'm seeing the same issue. That tip to clear the tasks through the Task Instances menu saved my sanity. Thanks!
mike....@thekeysupport.com
Apr 11, 2019, 11:13:01 AM4/11/19
to cloud-composer-discuss
Annoyingly when I do that clear isn't an option under 'With selected'. Oh well. The insanity remains.
Wilson Lian
Apr 12, 2019, 8:34:04 PM4/12/19
to mike....@thekeysupport.com, cloud-composer-discuss
The clear option is not present in Airflow 1.9.0 but is present in 1.10.0 and higher. You could consider upgrading your environment to a later Airflow version.
--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/642a120f-02cd-4dfe-babd-f622b1c5b0bd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Cedrik Neumann
Jun 19, 2019, 11:49:04 AM6/19/19
to cloud-composer-discuss
Clearing a task in the UI is standard operation since the beginning or Airflow. This is clearly a bug!
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.
Paul Messinesi
Nov 28, 2019, 9:44:10 AM11/28/19
to cloud-composer-discuss
Still having this issue today, with the latest composer-1.8.1-airflow-1.10.3 ...
Konrad Komorowski
Nov 28, 2019, 10:18:09 AM11/28/19
to cloud-composer-discuss
I was looking into it recently as well...
These threads (no pun intended) should help:
Specific pointers for fixes:
- make sure your defaults arguments are of the correct type: https://stackoverflow.com/a/49650096/11229376
- only use BaseOperator parameters in default arguments: https://issues.apache.org/jira/browse/AIRFLOW-351?focusedCommentId=15536210&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-15536210 / https://issues.apache.org/jira/browse/AIRFLOW-351?focusedCommentId=16330590&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16330590
- make sure your Operators are serializable (pickle-able: https://issues.apache.org/jira/browse/AIRFLOW-351?focusedCommentId=16640103&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16640103
Some potential fixes on Airflow's side (it will take months until they make their way to Cloud Composer release cycle though, so don't get your hopes high):
Lastly, the /admin/taskinstance/ workaround can help as a last resort.
Best,
Konrad
Konrad Komorowski
Nov 28, 2019, 11:10:20 AM11/28/19
to cloud-composer-discuss
I'm also thinking...
Cloud Composer still runs Python 3.6! https://issuetracker.google.com/issues/144367868
Maybe there are pickling improvements in Python 3.7 & 3.8 that could help? Not sure how forward/backwards compatible the pickling protocol needs to be, so maybe this isn't the case.
Konrad
Paul M
Nov 28, 2019, 12:51:22 PM11/28/19
to cloud-composer-discuss
Thank you for the helpful reads. We do have our own set of custom python operators, I'll check if they are all serializable!
Boris Shilov
Jun 25, 2020, 6:16:42 AM6/25/20
to cloud-composer-discuss
We also observe this bug in the latest Cloud Composer running Python 3.
Benjamin GRENIER
Jun 25, 2020, 8:55:40 AM6/25/20
to cloud-composer-discuss
Check if you have :
- log into the constructor of a custom operator ?
- 2 or more task with same task_id ?
Daniel Gies
Sep 28, 2020, 3:36:33 PM9/28/20
to cloud-composer-discuss
Issue is still happening with composer-1.12.0-airflow.1.10.10 and seems to be a problem with any of the GCP Dataproc operators.
Reply all
Reply to author
Forward
0 new messages