Error encountered in migration from Zoo 0.11.0 to BigDL 2.1.0

[clar...@gmail.com]

Hi, I am trying to upgrade the version of BigDL used on the spark cluster from Zoo 0.11.0 to BigDL 2.1.0. I changed the old code:

import zoo.common.nncontext as nncon
nncon.init_engine()

to the new code below:

from bigdl.dllib.nncontext import init_nncontext
sc = init_nncontext()

Is that equivalent? I encounter error below when running the new code. May you tell me what's the  right way to translate the old code? Thanks!

Clare

Error message:

>       sc = init_nncontext()

test_dl_pipeline.py:65:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../../../../opt/anaconda3/envs/cos_env_upgrade/lib/python3.7/site-packages/bigdl/dllib/nncontext.py:536: in init_nncontext
    sc = init_internal_nncontext(conf, spark_log_level, redirect_spark_log)
../../../../../opt/anaconda3/envs/cos_env_upgrade/lib/python3.7/site-packages/bigdl/dllib/nncontext.py:665: in init_internal_nncontext
    redire_spark_logs()
../../../../../opt/anaconda3/envs/cos_env_upgrade/lib/python3.7/site-packages/bigdl/dllib/utils/common.py:476: in redire_spark_logs
    callBigDlFunc(bigdl_type, "redirectSparkLogs", log_path)
../../../../../opt/anaconda3/envs/cos_env_upgrade/lib/python3.7/site-packages/bigdl/dllib/utils/common.py:623: in callBigDlFunc
    for jinvoker in JavaCreator.instance(bigdl_type, gateway).value:
../../../../../opt/anaconda3/envs/cos_env_upgrade/lib/python3.7/site-packages/bigdl/dllib/utils/common.py:58: in instance
    cls._instance = cls(bigdl_type, *args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <bigdl.dllib.utils.common.JavaCreator object at 0x7fdbd2919e10>
bigdl_type = 'float'
gateway = <py4j.java_gateway.JavaGateway object at 0x7fdbc1cced50>

    def __init__(self, bigdl_type, gateway):
        self.value = []
        for creator_class in JavaCreator.get_creator_class():
            jclass = getattr(gateway.jvm, creator_class)
            if bigdl_type == "float":
>               self.value.append(getattr(jclass, "ofFloat")())
E               TypeError: 'JavaPackage' object is not callable

../../../../../opt/anaconda3/envs/cos_env_upgrade/lib/python3.7/site-packages/bigdl/dllib/utils/common.py:108: TypeError

[guoqiong song]

It is correct to start nncontext this way.

from bigdl.dllib.nncontext import init_nncontext
sc = init_nncontext()

The error is from jar file not accessible, can you update the environment if you run the code in a conda environment?

pip install bigdl-dllib

please refer here for further information.

https://bigdl.readthedocs.io/en/latest/doc/DLlib/Overview/install.html