PythonVirtualenvOperator givig PicklingError

[gaurav...@blackrock.com]

I am trying to run a PythonVirtualenvOperator within a dag in a gcloud composer environment but am getting an error while the virtual environment is created. A google search led me to https://github.com/uqfoundation/dill/issues/93 but I am not sure I understand where the problem exactly is.

Detailed StackOverflow Question: https://stackoverflow.com/questions/54073583/pickle-picklingerror-cant-pickle-class-jinja2-utils-missingtype-its-not

I get an error : _pickle.PicklingError: Can't pickle <class 'jinja2.utils.MissingType'>: it's not found as jinja2.utils.MissingType

Operator is defined as :

Start = PythonVirtualenvOperator(
                    task_id="start_it",
                    python_callable=dummy_func,
                    requirements=["pandas==0.23.4","numpy","dill"],
                    use_dill=True,
                    system_site_packages=False,
                    #op_kwargs={'config':config},
                    provide_context=True,
                    dag=DAG
                    )

[Wilson Lian]

Hi,

Can you please attach full task instance logs?

thanks,

Wilson

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-di...@googlegroups.com.
To post to this group, send email to cloud-compo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-composer-discuss/1a48aad1-13e2-4bc2-9397-9ceee6e17a7c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[gaurav...@blackrock.com]

Hi Wilson,

It resolved after I set provide_context as False. According to documentation :param provide_context: if set to true, Airflow will pass a set of keyword arguments that can be used in your function. This set of kwargs correspond exactly to what you can use in your jinja templates. For this to work, you need to define **kwargs in your function header. which causes the error.

The full Traceback Stack is below:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/pickle.py", line 269, in _getattribute
    obj = getattr(obj, subpath)
AttributeError: module 'jinja2.utils' has no attribute 'MissingType'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/pickle.py", line 918, in save_global
    obj2, parent = _getattribute(module, name)
  File "/usr/local/lib/python3.6/pickle.py", line 272, in _getattribute
    .format(name, obj))
AttributeError: Can't get attribute 'MissingType' on <module 'jinja2.utils' from '/usr/local/lib/python3.6/site-packages/jinja2/utils.py'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/usr/local/lib/airflow/airflow/bin/airflow", line 27, in <module>
    args.func(args)
  File "/usr/local/lib/airflow/airflow/bin/cli.py", line 528, in test
    ti.run(ignore_task_deps=True, ignore_ti_state=True, test_mode=True)
  File "/usr/local/lib/airflow/airflow/utils/db.py", line 50, in wrapper
    result = func(*args, **kwargs)
  File "/usr/local/lib/airflow/airflow/models.py", line 1583, in run
    session=session)
  File "/usr/local/lib/airflow/airflow/utils/db.py", line 50, in wrapper
    result = func(*args, **kwargs)
  File "/usr/local/lib/airflow/airflow/models.py", line 1492, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/usr/local/lib/airflow/airflow/operators/python_operator.py", line 89, in execute
    return_value = self.execute_callable()
  File "/usr/local/lib/airflow/airflow/operators/python_operator.py", line 245, in execute_callable
    self._write_args(input_filename)
  File "/usr/local/lib/airflow/airflow/operators/python_operator.py", line 283, in _write_args
    dill.dump(arg_dict, f)
  File "/usr/local/lib/python3.6/site-packages/dill/_dill.py", line 286, in dump
    pik.dump(obj)
  File "/usr/local/lib/python3.6/pickle.py", line 409, in dump
    self.save(obj)
  File "/usr/local/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python3.6/site-packages/dill/_dill.py", line 893, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/local/lib/python3.6/pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/lib/python3.6/pickle.py", line 847, in _batch_setitems
    save(v)
  File "/usr/local/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python3.6/site-packages/dill/_dill.py", line 893, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/local/lib/python3.6/pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/lib/python3.6/pickle.py", line 847, in _batch_setitems
    save(v)
  File "/usr/local/lib/python3.6/pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local/lib/python3.6/pickle.py", line 634, in save_reduce
    save(state)
  File "/usr/local/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python3.6/site-packages/dill/_dill.py", line 893, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/local/lib/python3.6/pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/lib/python3.6/pickle.py", line 847, in _batch_setitems
    save(v)
  File "/usr/local/lib/python3.6/pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local/lib/python3.6/pickle.py", line 634, in save_reduce
    save(state)
  File "/usr/local/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python3.6/site-packages/dill/_dill.py", line 893, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/local/lib/python3.6/pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/lib/python3.6/pickle.py", line 847, in _batch_setitems
    save(v)
  File "/usr/local/lib/python3.6/pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local/lib/python3.6/pickle.py", line 634, in save_reduce
    save(state)
  File "/usr/local/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python3.6/site-packages/dill/_dill.py", line 893, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/local/lib/python3.6/pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/lib/python3.6/pickle.py", line 847, in _batch_setitems
    save(v)
  File "/usr/local/lib/python3.6/pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local/lib/python3.6/pickle.py", line 634, in save_reduce
    save(state)
  File "/usr/local/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python3.6/site-packages/dill/_dill.py", line 893, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/local/lib/python3.6/pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/lib/python3.6/pickle.py", line 847, in _batch_setitems
    save(v)
  File "/usr/local/lib/python3.6/pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local/lib/python3.6/pickle.py", line 634, in save_reduce
    save(state)
  File "/usr/local/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python3.6/site-packages/dill/_dill.py", line 893, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/local/lib/python3.6/pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/lib/python3.6/pickle.py", line 847, in _batch_setitems
    save(v)
  File "/usr/local/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python3.6/site-packages/dill/_dill.py", line 893, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/local/lib/python3.6/pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/lib/python3.6/pickle.py", line 847, in _batch_setitems
    save(v)
  File "/usr/local/lib/python3.6/pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local/lib/python3.6/pickle.py", line 634, in save_reduce
    save(state)
  File "/usr/local/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python3.6/site-packages/dill/_dill.py", line 893, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/local/lib/python3.6/pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/lib/python3.6/pickle.py", line 847, in _batch_setitems
    save(v)
  File "/usr/local/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python3.6/pickle.py", line 781, in save_list
    self._batch_appends(obj)
  File "/usr/local/lib/python3.6/pickle.py", line 808, in _batch_appends
    save(tmp[0])
  File "/usr/local/lib/python3.6/pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local/lib/python3.6/pickle.py", line 634, in save_reduce
    save(state)
  File "/usr/local/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python3.6/site-packages/dill/_dill.py", line 893, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/local/lib/python3.6/pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/lib/python3.6/pickle.py", line 847, in _batch_setitems
    save(v)
  File "/usr/local/lib/python3.6/pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local/lib/python3.6/pickle.py", line 634, in save_reduce
    save(state)
  File "/usr/local/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python3.6/site-packages/dill/_dill.py", line 893, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/local/lib/python3.6/pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/lib/python3.6/pickle.py", line 847, in _batch_setitems
    save(v)
  File "/usr/local/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python3.6/site-packages/dill/_dill.py", line 1377, in save_function
    obj.__dict__), obj=obj)
  File "/usr/local/lib/python3.6/pickle.py", line 610, in save_reduce
    save(args)
  File "/usr/local/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python3.6/pickle.py", line 751, in save_tuple
    save(element)
  File "/usr/local/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python3.6/site-packages/dill/_dill.py", line 893, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/local/lib/python3.6/pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/lib/python3.6/pickle.py", line 847, in _batch_setitems
    save(v)
  File "/usr/local/lib/python3.6/pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local/lib/python3.6/pickle.py", line 605, in save_reduce
    save(cls)
  File "/usr/local/lib/python3.6/pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/lib/python3.6/site-packages/dill/_dill.py", line 1321, in save_type
    StockPickler.save_global(pickler, obj)
  File "/usr/local/lib/python3.6/pickle.py", line 922, in save_global
    (obj, module_name, name))
_pickle.PicklingError: Can't pickle <class 'jinja2.utils.MissingType'>: it's not found as jinja2.utils.MissingType

Also the code i ran

On Wednesday, January 9, 2019 at 2:23:13 AM UTC+5:30, Wilson Lian wrote:

Hi,

Can you please attach full task instance logs?

thanks,

Wilson

On Mon, Jan 7, 2019 at 9:30 PM <gaurav...@blackrock.com> wrote:

I am trying to run a PythonVirtualenvOperator within a dag in a gcloud composer environment but am getting an error while the virtual environment is created. A google search led me to https://github.com/uqfoundation/dill/issues/93 but I am not sure I understand where the problem exactly is.

Detailed StackOverflow Question: https://stackoverflow.com/questions/54073583/pickle-picklingerror-cant-pickle-class-jinja2-utils-missingtype-its-not

I get an error : _pickle.PicklingError: Can't pickle <class 'jinja2.utils.MissingType'>: it's not found as jinja2.utils.MissingType

Operator is defined as :

Start = PythonVirtualenvOperator(
                    task_id="start_it",
                    python_callable=dummy_func,
                    requirements=["pandas==0.23.4","numpy","dill"],
                    use_dill=True,
                    system_site_packages=False,
                    #op_kwargs={'config':config},
                    provide_context=True,
                    dag=DAG
                    )

--
You received this message because you are subscribed to the Google Groups "cloud-composer-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-composer-discuss+unsub...@googlegroups.com.

[Bob Muscovite]

Greetings,

I am hitting this issue today in the exact same circumstances. I want to be able to use the task_instance object to manipulate xcoms. I see this is also an open issue on Airflow tracker: https://issues.apache.org/jira/browse/AIRFLOW-2738

It seems this destroys a lot of the utility of the virtual env operator. Any workarounds to be able to pull and push xcoms from within them?

Best regards,

Boris

[Bob Muscovite]

It looks like one can get xcom pulls by using a ninja templating workaround, and for com pushes there is a diabolical solution involving printing to stdout.