py4j.protocol.Py4JNetworkError: Answer from Java side is empty

[yu jiahao]

Hi all, I am trying to run the SimpleTraining.py for pytorch on Colab and I follow the docs to prepare the environment with JAVA 8 and zoo ( the only exception is that I downgrade the version of tf to 1.14 otherwise it reports an error).  Outputs are given:

---------------------------------------------

Prepending /usr/local/lib/python3.6/dist-packages/bigdl/share/conf/spark-bigdl.conf to sys.path Adding /usr/local/lib/python3.6/dist-packages/zoo/share/lib/analytics-zoo-bigdl_0.10.0-spark_2.4.3-0.8.0-jar-with-dependencies.jar to BIGDL_JARS Prepending /usr/local/lib/python3.6/dist-packages/zoo/share/conf/spark-analytics-zoo.conf to sys.path

creating: createTorchNet creating: createTorchCriterion

creating: createSeqToTensor creating: createScalarToTensor creating: createFeatureLabelPreprocessing creating: createNNClassifier creating: createAdam

ERROR:root:Exception while sending command. Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/py4j/java_gateway.py", line 1159, in send_command raise Py4JNetworkError("Answer from Java side is empty") py4j.protocol.Py4JNetworkError: Answer from Java side is empty During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/py4j/java_gateway.py", line 985, in send_command response = connection.send_command(command) File "/usr/local/lib/python3.6/dist-packages/py4j/java_gateway.py", line 1164, in send_command "Error while receiving", e, proto.ERROR_ON_RECEIVE) py4j.protocol.Py4JNetworkError: Error while receiving

------------------------------------------

I have set 'export SPARK_DRIVER_MEMORY=20g' and it does not help. I am not familiar with JAVA and I have no idea which part is incorrect since I do not change codes of the given toy example.

And this is the link to my codes. Any idea is helpful.

Thx a lot in advance!

[Xin Qiu]

!pip install torch==1.1.0 torchvision==0.3.0

please add this, we need pytorch 1.1.0

Bests,

-Xin

[yu jiahao]

Thx for replying! And I have tried the specific version of torch and it reports another error so I still cannot run it successfully. The error is as follows:

----------------------------------------------------

/usr/local/lib/python3.6/dist-packages/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
    326                 raise Py4JJavaError(
    327                     "An error occurred while calling {0}{1}{2}.\n".
--> 328                     format(target_id, ".", name), value)
    329             else:
    330                 raise Py4JError(

Py4JJavaError: An error occurred while calling o177.fit.
: java.lang.reflect.InaccessibleObjectException: Unable to make field jdk.internal.ref.PhantomCleanable jdk.internal.ref.PhantomCleanable.prev accessible: module java.base does not "opens jdk.internal.ref" to unnamed module @2ca923bb
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:340)
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:280)
	at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:176)
	at java.base/java.lang.reflect.Field.setAccessible(Field.java:170)
	at org.apache.spark.util.SizeEstimator$$anonfun$getClassInfo$3.apply(SizeEstimator.scala:336)
	at org.apache.spark.util.SizeEstimator$$anonfun$getClassInfo$3.apply(SizeEstimator.scala:330)
	at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
	at org.apache.spark.util.SizeEstimator$.getClassInfo(SizeEstimator.scala:330)
	at org.apache.spark.util.SizeEstimator$.getClassInfo(SizeEstimator.scala:324)
	at org.apache.spark.util.SizeEstimator$.visitSingleObject(SizeEstimator.scala:222)
	at org.apache.spark.util.SizeEstimator$.org$apache$spark$util$SizeEstimator$$estimate(SizeEstimator.scala:201)
	at org.apache.spark.util.SizeEstimator$.estimate(SizeEstimator.scala:69)
	at org.apache.spark.util.collection.SizeTracker$class.takeSample(SizeTracker.scala:78)
	at org.apache.spark.util.collection.SizeTracker$class.afterUpdate(SizeTracker.scala:70)
	at org.apache.spark.util.collection.SizeTrackingVector.$plus$eq(SizeTrackingVector.scala:31)
	at org.apache.spark.storage.memory.DeserializedValuesHolder.storeValue(MemoryStore.scala:665)
	at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:222)
	at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:299)
	at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1165)
	at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1156)
	at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1091)
	at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1156)
	at org.apache.spark.storage.BlockManager.putIterator(BlockManager.scala:914)
	at org.apache.spark.storage.BlockManager.putSingle(BlockManager.scala:1481)
	at org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:123)
	at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:88)
	at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
	at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:62)
	at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1489)
	at com.intel.analytics.bigdl.models.utils.ModelBroadcastImp.broadcast(ModelBroadcast.scala:133)
	at com.intel.analytics.bigdl.models.utils.ModelBroadcastImp.broadcast(ModelBroadcast.scala:95)
	at com.intel.analytics.bigdl.optim.DistriOptimizer$.com$intel$analytics$bigdl$optim$DistriOptimizer$$initThreadModels(DistriOptimizer.scala:563)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.intel.analytics.zoo.pipeline.api.keras.layers.utils.KerasUtils$.invokeMethod(KerasUtils.scala:302)
	at com.intel.analytics.zoo.pipeline.api.keras.layers.utils.KerasUtils$.invokeMethodWithEv(KerasUtils.scala:329)
	at com.intel.analytics.zoo.pipeline.api.keras.models.InternalOptimizerUtil$.initThreadModels(Topology.scala:1010)
	at com.intel.analytics.zoo.pipeline.api.keras.models.InternalDistriOptimizer.train(Topology.scala:1159)
	at com.intel.analytics.zoo.pipeline.api.keras.models.InternalDistriOptimizer.train(Topology.scala:1382)
	at com.intel.analytics.zoo.pipeline.nnframes.NNEstimator.internalFit(NNEstimator.scala:470)
	at com.intel.analytics.zoo.pipeline.nnframes.NNEstimator.internalFit(NNEstimator.scala:198)
	at org.apache.spark.ml.DLEstimatorBase.fit(DLEstimatorBase.scala:58)
	at org.apache.spark.ml.DLEstimatorBase.fit(DLEstimatorBase.scala:50)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:238)
	at java.base/java.lang.Thread.run(Thread.java:834)

----------------------------------------------------

And it remains the same error when I switched to the latest tf version (I think it has nothing to do with tf).
在 2020年4月27日星期一 UTC+8下午4:32:02，Xin Qiu写道：

[Xin Qiu]

It works fine in my colab https://colab.research.google.com/drive/1Sfe9_3lxDr-I8zDzXkxpbMHGpO5aYHXp

[yu jiahao]

Solved, resetting the environment can help. Thanks for your advice!

在 2020年4月28日星期二 UTC+8上午11:06:26，Xin Qiu写道：