Spark jobs not being able read from the Spark history server

[nikhil bejugama]

Hi,

I have set the SPARK_HOME and SPARK_CONF_DIR but I couldn't Spark applications analysed by the Dr E.

02-14-2017 11:49:36 INFO  [Thread-6] com.linkedin.drelephant.ElephantRunner : Dr.elephant has started

02-14-2017 11:49:36 INFO  [Thread-6] com.linkedin.drelephant.security.HadoopSecurity : This cluster is Kerberos enabled.

02-14-2017 11:49:36 INFO  [Thread-6] com.linkedin.drelephant.security.HadoopSecurity : No login user. Creating login user

02-14-2017 11:49:36 INFO  [Thread-6] com.linkedin.drelephant.security.HadoopSecurity : Logging with xxxx/xxxx@xxxx and xxxx

02-14-2017 11:49:36 INFO  [Thread-6] com.linkedin.drelephant.security.HadoopSecurity : Logged in with user xxxx/xxxx@xxxx (auth:KERBEROS)

02-14-2017 11:49:36 INFO  [Thread-6] com.linkedin.drelephant.security.HadoopSecurity : Login is keytab based

02-14-2017 11:49:37 INFO  [Thread-6] com.linkedin.drelephant.analysis.HDFSContext : HDFS BLock size: 134217728

02-14-2017 11:49:37 INFO  [Thread-6] com.linkedin.drelephant.util.Utils : Loading configuration file AggregatorConf.xml

02-14-2017 11:49:37 INFO  [Thread-6] com.linkedin.drelephant.util.Utils : Configuation file loaded. File: AggregatorConf.xml

02-14-2017 11:49:37 INFO  [Thread-6] com.linkedin.drelephant.ElephantContext : Load Aggregator : com.linkedin.drelephant.mapreduce.MapReduceMetricsAggregator

02-14-2017 11:49:37 INFO  [Thread-6] com.linkedin.drelephant.ElephantContext : Load Aggregator : com.linkedin.drelephant.spark.SparkMetricsAggregator

02-14-2017 11:49:37 INFO  [Thread-6] com.linkedin.drelephant.util.Utils : Loading configuration file FetcherConf.xml

02-14-2017 11:49:37 INFO  [Thread-6] com.linkedin.drelephant.util.Utils : Configuation file loaded. File: FetcherConf.xml

02-14-2017 11:49:37 INFO  [Thread-6] com.linkedin.drelephant.mapreduce.fetchers.MapReduceFetcherHadoop2 : Connecting to the job history server at xxxx:19888...

02-14-2017 11:49:37 INFO  [Thread-6] com.linkedin.drelephant.mapreduce.fetchers.MapReduceFetcherHadoop2 : Connection success.

02-14-2017 11:49:37 INFO  [Thread-6] com.linkedin.drelephant.ElephantContext : Load Fetcher : com.linkedin.drelephant.mapreduce.fetchers.MapReduceFetcherHadoop2

02-14-2017 11:49:37 INFO  [Thread-6] com.linkedin.drelephant.ElephantContext : Load Fetcher : com.linkedin.drelephant.spark.fetchers.SparkFetcher

02-14-2017 11:49:37 INFO  [Thread-6] com.linkedin.drelephant.util.Utils : Loading configuration file HeuristicConf.xml

02-14-2017 11:49:37 INFO  [Thread-6] com.linkedin.drelephant.util.Utils : Configuation file loaded. File: HeuristicConf.xml

02-14-2017 11:49:37 INFO  [Thread-6] com.linkedin.drelephant.mapreduce.heuristics.GenericDataSkewHeuristic : Mapper Data Skew will use num_tasks_severity with the following threshold settings: [10.0, 50.0, 100.0, 200.0]

02-14-2017 11:49:37 INFO  [Thread-6] com.linkedin.drelephant.mapreduce.heuristics.GenericDataSkewHeuristic : Mapper Data Skew will use deviation_severity with the following threshold settings: [2.0, 4.0, 8.0, 16.0]

02-14-2017 11:49:37 INFO  [Thread-6] com.linkedin.drelephant.mapreduce.heuristics.GenericDataSkewHeuristic : Mapper Data Skew will use files_severity with the following threshold settings: [0.125, 0.25, 0.5, 1.0]

02-14-2017 11:49:37 INFO  [Thread-6] com.linkedin.drelephant.ElephantContext : Load Heuristic : com.linkedin.drelephant.mapreduce.heuristics.MapperDataSkewHeuristic

02-14-2017 11:49:37 INFO  [Thread-6] com.linkedin.drelephant.ElephantContext : Load View : views.html.help.mapreduce.helpMapperDataSkew

02-14-2017 11:49:37 INFO  [Thread-6] com.linkedin.drelephant.mapreduce.heuristics.GenericGCHeuristic : Mapper GC will use gc_ratio_severity with the following threshold settings: [0.01, 0.02, 0.03, 0.04]

02-14-2017 11:49:37 INFO  [Thread-6] com.linkedin.drelephant.mapreduce.heuristics.GenericGCHeuristic : Mapper GC will use runtime_severity_in_min with the following threshold settings: [5.0, 10.0, 12.0, 15.0]

02-14-2017 11:49:37 INFO  [Thread-6] com.linkedin.drelephant.ElephantContext : Load Heuristic : com.linkedin.drelephant.mapreduce.heuristics.MapperGCHeuristic

INFO  org.apache.spark.deploy.history.SparkFSFetcher$ : Looking for spark logs at logDir:xxxx

org.apache.spark.deploy.history.SparkFSFetcher$ : The event log limit of Spark application is set to 100.0 MB

I couldn't find the lines such as above in the dr_elephant.log. Could you please let me know if I'm missing anything.

Thanks

[Fawze Abujaber]

Hi,

Were you able to get this work?