Kylo-spark-shell is not starting automatically

[Ankita Thakkar]

Hi,

As per Kylo 0.9.1 documentation , While validating the Spark Shell step , When i click on Visual Query > Add Table > Continue to Step 2 , Transformation exception error occurs.

If i run manully run-spark-shell.sh and start the service manually then it works fine.

So , is there any property am i missing or do i need to add explicitly so that spark-shell service can start automatically without manual intervention.

Thanks in advance.

[Jagrut Sharma]

Anita - Try waiting for 3-4 minutes after Kylo starts to invoke the Visual Query. Sometimes, in the sandbox, Spark shell takes a while to start after Kylo services have started.

--

Jagrut

[Ankita Thakkar]

Hi Jagrut,

Even though after waiting for an half an hour, it never started automatically  and always had to do manual start.

Any other way?

Thanks in advance,

Ankita

[Greg Hart]

Hi Ankita,

Please ensure that your /opt/kylo/kylo-services/conf/spark.properties looks like the example below. You will need to have spark.shell.server.host and spark.shell.server.port commented out.

#spark.shell.server.host=localhost
#spark.shell.server.port=8450
spark.shell.master = local[1]

spark.shell.sparkArgs = --driver-memory 512m --executor-memory 512m --driver-class-path /opt/kylo/kylo-services/conf:/opt/kylo/kylo-services/lib/mariadb-java-client-1.5.7.jar --driver-java-options -Dlog4j.configuration=log4j-spark.properties

[Greg Hart]

Then restart kylo-services after making that change.

[Ankita Thakkar]

Hi Greg,

After setting spark.shell.sparkHome property to latest spark version (2 spark versions were installed) in /opt/kylo/kylo-services/conf/spark.properties file , spark shell service was able to start automatically.

Thank you for your help!

Thanks,

Ankita