Dr Elephant on AWS EMR

[Aditya Borde]

Hello,

I want to use Dr. Elephant for spark jobs on EMR cluster. I got success in installing it and I can see the Dr Elephant web UI (This is through SSH Tunnel). But UI does not show anything even after running some jobs.

One reason I suspect is, spark-defaults.conf has

spark.eventLog.dir               hdfs:///var/log/spark/apps
spark.history.fs.logDirectory    hdfs:///var/log/spark/apps


And Dr Elephant is looking at the path : /system/spark-history which is why I could not see the result on UI.As DR Elephant can't find anything in that directory.

Is that the reason correct? If yes, what change would help me to get this working and I can see the results on UI?
Or Am I doing something wrong?

Thanks,
Aditya Borde

[Akshay Rai]

Aditya, please check this issue on how to configure Dr. Elephant with Spark. Let me know if you still can't see Spark jobs being analyzed.

[Akhil Chalamalasetty]

Hi Aditya,

Did you use a bootstrap to do the install of Dr elephant ?

Could you please share the script you've used ?

Thank you,

Akhil