Getting build failed error from github master branch

[saurab...@zeotap.com]

Hi Dr Elephant team

I am getting build error while building dr-elephant with below settings:-

- github branch - master (commit ID: https://github.com/linkedin/dr-elephant/commit/860dbe6ad2830b8ffeb88fc97966fb33b010b5fd)

- compile.conf - hadoop_version=2.7.3

            spark_version=2.1.0

            play_opts="-Dsbt.repository.config=app-conf/resolver.conf"

Error: java.lang.IllegalStateException: impossible to get artifacts when data has not been loaded. IvyNode = com.fasterxml.jackson.core#jackson-databind;2.5.4

Please assist to resolve this error

Thanks

Saurabh

[saurab...@zeotap.com]

Found out that Spark 2.x is not supported upfront and require some setup in the SHS itself. (mentioned here: https://github.com/linkedin/dr-elephant/issues/327)

Is there a link / doc which lists the customization in SHS for successful monitoring spark 2.x jobs via dr-elephant?

Thanks

[Clemens Valiente]

Hi Saurabh,

Spark History Server runs independent of any Spark Applications - you just point it at the same log directory that your Spark Applications are logging to (spark.eventlog.dir for the spark applications, spark.history.fs.logDirectory for the history server). So you can run the "improved" spark history server from the Apache Spark vanilla version 2.3.1 alongside any older Spark application. Set SPARK_CONF_DIR to your spark configuration including the SHS, then you can run Dr. Elephant with the SparkFetcher!

Cheers

Clemens

--
You received this message because you are subscribed to the Google Groups "dr-elephant-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dr-elephant-us...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dr-elephant-users/3cd76056-c80e-45c7-971b-70a691aeeac1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[saurab...@zeotap.com]

Thanks Clemens for the info, it worked. 

Configured dr-elephant with below config for spark fetcher:-

  <fetcher>

    <applicationtype>spark</applicationtype>

    <classname>com.linkedin.drelephant.spark.fetchers.SparkFetcher</classname>

    <params>

    <event_log_size_limit_in_mb>500</event_log_size_limit_in_mb>

    <event_log_location_uri>webhdfs://NameNodeIP:50070/var/log/spark/apps</event_log_location_uri>

    </params>

  </fetcher>

The setup is working on AWS EMR v 5.6.0, Spark 2.1.0, Hadoop 2.7.3

Thanks