Unable to compile Dr Elephant due to test failures

[AnilKumar B]

Hi,

I am unable to compile Dr Elephant due to test failures. May I know, what could be the issue with nn1.grid.example.com?

[info] Total for test com.linkedin.drelephant.spark.heuristics.ConfigurationHeuristicTest

[info] Finished in 0.002 seconds

[info] 29 tests, 0 failures, 0 errors

[info] SparkUtilsTest:

[info] SparkUtils

[info]   .fileSystemAndPathForEventLogDir

[info]   - returns a filesystem + path based on uri from fetcherConfg *** FAILED ***

[info]     java.lang.IllegalArgumentException: java.net.UnknownHostException: nn1.grid.example.com

[info]     at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:377)

[info]     at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:395)

[info]     at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.initialize(WebHdfsFileSystem.java:180)

[info]     at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2397)

[info]     at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89)

[info]     at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2431)

[info]     at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2413)

[info]     at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368)

[info]     at com.linkedin.drelephant.util.SparkUtils$class.fileSystemAndPathForEventLogDir(SparkUtils.scala:63)

[info]     at com.linkedin.drelephant.util.SparkUtilsTest$$anonfun$1$$anonfun$apply$mcV$sp$1$$anonfun$apply$mcV$sp$4$$anon$1.fileSystemAndPathForEventLogDir(SparkUtilsTest.scala:42)

[info]     ...

[info]     Cause: java.net.UnknownHostException: nn1.grid.example.com

[info]     at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:377)

[info]     at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:395)

[info]     at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.initialize(WebHdfsFileSystem.java:180)

[info]     at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2397)

[info]     at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89)

[info]     at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2431)

[info]     at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2413)

[info]     at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368)

[info]     at com.linkedin.drelephant.util.SparkUtils$class.fileSystemAndPathForEventLogDir(SparkUtils.scala:63)

[info]     at com.linkedin.drelephant.util.SparkUtilsTest$$anonfun$1$$anonfun$apply$mcV$sp$1$$anonfun$apply$mcV$sp$4$$anon$1.fileSystemAndPathForEventLogDir(SparkUtilsTest.scala:42)

[info]     ...

[info]   - returns a webhdfs filesystem + path based on spark.eventLog.dir when it is a webhdfs URL *** FAILED ***

[info]     java.lang.IllegalArgumentException: java.net.UnknownHostException: nn1.grid.example.com

[info]     at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:377)

[info]     at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:395)

[info]     at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.initialize(WebHdfsFileSystem.java:180)

[info]     at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2397)

[info]     at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89)

[info]     at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2431)

[info]     at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2413)

[info]     at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368)

[info]     at com.linkedin.drelephant.util.SparkUtils$class.fileSystemAndPathForEventLogDir(SparkUtils.scala:68)

[info]     at com.linkedin.drelephant.util.SparkUtilsTest$$anonfun$1$$anonfun$apply$mcV$sp$1$$anonfun$apply$mcV$sp$5$$anon$2.fileSystemAndPathForEventLogDir(SparkUtilsTest.scala:58)

[info]     ...

:

:

:

[info] *** 6 TESTS FAILED ***

[error] Failed: Total 337, Failed 6, Errors 0, Passed 330, Skipped 1

[error] Failed tests:

[error] com.linkedin.drelephant.util.SparkUtilsTest

[error] (test:test) sbt.TestsFailedException: Tests unsuccessful

[error] Total time: 57 s, completed Jun 14, 2017 11:49:31 PM

+ cd target/universal

./compile.sh: line 131: cd: target/universal: No such file or directory

++ /bin/ls '*.zip'

ls: *.zip: No such file or directory

+ ZIP_NAME=

+ unzip

UnZip 5.52 of 28 February 2005, by Info-ZIP.  Maintained by C. Spieler.  Send

bug reports using http://www.info-zip.org/zip-bug.html; see README for details.

Thanks,

B Anil Kumar.

[AnilKumar B]

Hi All,

Can anyone suggest, what could be wrong. I am trying to build it by using compile.sh as suggested in https://github.com/linkedin/dr-elephant/wiki/Quick-Setup-Instructions 

Or Is there any prebuilt distribution available?

Hadoop and Spark versions as below:

HADOOP_VERSION="2.4.0"

SPARK_VERSION="1.6.1"

Thanks,

B Anil Kumar.

[AnilKumar B]

Sorry for multiple posts. I just tried with as below and it went well. 

./compile.sh  ./compile.conf 

default compile.conf has hadoop_version=2.3.0 spark_version=1.4.0 versions. Which seems to be working.

So is there any issue with HADOOP_VERSION="2.4.0" and SPARK_VERSION="1.6.1" versions?

On Wednesday, 14 June 2017 23:53:33 UTC-7, AnilKumar B wrote: