com.databricks.backend.daemon.driver.ExitSecurityException: Program attempted to exit with code -1
asgu...@eng.ucsd.edu
I have spark 1.6.0 cluster (11workers) in Databricks and pysparkling library 1.6.9
I am trying to run the pysparkling on Databricks and when run the below command
hc = H2OContext(sc).start()
I am getting the following error
com.databricks.backend.daemon.driver.ExitSecurityException: Program attempted to exit with code -1
---------------------------------------------------------------------------
Py4JJavaError Traceback (most recent call last)
<ipython-input-18-5b120b76a612> in <module>()
----> 1 hc = H2OContext(sc).start()
/local_disk0/spark-07b9e62d-48d4-416b-94fe-a81feaf3704b/userFiles-1af20c19-16ed-4b8e-8624-caba16f80ea5/addedFile7794027593878916846dbfs__FileStore_jars_e38c09f1_f099_4e93_b1a3_f93b89f6dde1_h2o_pysparkling_1_6_1_6_99999_py2_7_6fbf5-6582f.egg/pysparkling/context.py in start(self)
122 H2O python client.
123 """
--> 124 self._jhc.start()
125 self._client_ip = self._jhc.h2oLocalClientIp()
126 self._client_port = self._jhc.h2oLocalClientPort()
/databricks/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py in __call__(self, *args)
811 answer = self.gateway_client.send_command(command)
812 return_value = get_return_value(
--> 813 answer, self.gateway_client, self.target_id, self.name)
814
815 for temp_arg in temp_args:
/databricks/spark/python/pyspark/sql/utils.py in deco(*a, **kw)
43 def deco(*a, **kw):
44 try:
---> 45 return f(*a, **kw)
46 except py4j.protocol.Py4JJavaError as e:
47 s = e.java_exception.toString()
/databricks/spark/python/lib/py4j-0.9-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
306 raise Py4JJavaError(
307 "An error occurred while calling {0}{1}{2}.\n".
--> 308 format(target_id, ".", name), value)
309 else:
310 raise Py4JError(
Py4JJavaError: An error occurred while calling o314.start.
: com.databricks.backend.daemon.driver.ExitSecurityException: Program attempted to exit with code -1
at com.databricks.backend.daemon.driver.TrapExitSecurityManager.checkExit(DriverLocal.scala:379)
at java.lang.Runtime.exit(Runtime.java:107)
at java.lang.System.exit(System.java:971)
at water.H2O.exit(H2O.java:596)
at water.H2O.shutdown(H2O.java:603)
at water.H2O.fail(H2O.java:930)
at water.H2O.fail(H2O.java:946)
at water.H2O.registerExtensions(H2O.java:697)
at water.H2OStarter.start(H2OStarter.java:18)
at water.H2OStarter.start(H2OStarter.java:40)
at org.apache.spark.h2o.H2OContext.start(H2OContext.scala:220)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:381)
at py4j.Gateway.invoke(Gateway.java:259)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:209)
at java.lang.Thread.run(Thread.java:745)
can you please help
Thanks
-Abhijeet
mat...@0xdata.com
Have you by any chance already ran `H2OContext(sc).start()` on that Databricks cluster? The error you're getting should be thrown only if you call `start()` multiple times and there's already an H2O cloud running.
Easiest way to check it is to run this in your Python notebook:
from pysparkling import *
import h2o
h2o.cluster_info()
If it returns you cluster information then you shouldn't run start() anymore.
Regards,
Mateusz