Building Hue for "Vanilla" hadoop and spark

[linux...@gmail.com]

Hello everyone,

I'm trying to build Hue to work with "Vanilla" hadoop and spark. 

git clone https://github.com/cloudera/hue.git

git checkout release-3.9.0

vi maven/pom.xml

...

    <hadoop.version>2.7.1</hadoop.version>

    <spark.version>1.4.1</spark.version>

...

HADOOP_HOME=/opt/my/hadoop PREFIX=/opt/my make install

Finally I'm getting this error:

...

[INFO] Reactor Summary:

[INFO] 

[INFO] livy-main .......................................... SUCCESS [  1.260 s]

[INFO] livy-core_2.10 ..................................... SUCCESS [ 11.040 s]

[INFO] livy-repl_2.10 ..................................... FAILURE [  3.533 s]

[INFO] livy-yarn_2.10 ..................................... SKIPPED

[INFO] livy-server_2.10 ................................... SKIPPED

[INFO] Livy Project Assembly .............................. SKIPPED

[INFO] ------------------------------------------------------------------------

[INFO] BUILD FAILURE

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 16.095 s

[INFO] Finished at: 2015-09-01T14:27:02+02:00

[INFO] Final Memory: 27M/608M

[INFO] ------------------------------------------------------------------------

[ERROR] Failed to execute goal on project livy-repl_2.10: Could not resolve dependencies for project com.cloudera.hue.livy:livy-repl_2.10:jar:3.9.0-SNAPSHOT: Failed to collect dependencies at org.apache.spark:spark-assembly_2.10:jar:1.5.0: Failed to read artifact descriptor for org.apache.spark:spark-assembly_2.10:jar:1.5.0: Could not transfer artifact org.apache.spark:spark-assembly_2.10:pom:1.5.0 from/to cloudera.snapshots.repo (https://repository.cloudera.com/content/repositories/snapshots): Failed to transfer file: https://repository.cloudera.com/content/repositories/snapshots/org/apache/spark/spark-assembly_2.10/1.5.0/spark-assembly_2.10-1.5.0.pom. Return code is: 409 , ReasonPhrase:Conflict. -> [Help 1]

Please, could anyone help me resolve this.

[Erick Tryzelaar]

Hello,

That looks like it might just have been maven not properly downloading the artifact. Could you try "make apps" again and see if that clears this up?

To unsubscribe from this group and stop receiving emails from it, send an email to hue-user+u...@cloudera.org.

[Romain Rigaux]

I had the same error and the file is up now and 'make apps' just worked again:
http://repository.cloudera.com/cloudera/libs-snapshot-local/org/apache/spark/spark-assembly_2.10/1.5.0-cdh5.5.0-SNAPSHOT/spark-assembly_2.10-1.5.0-cdh5.5.0-20150901.043355-12.jar

[linux...@gmail.com]

I've tried to build again, but it still fails. 

I'm trying to build against "vanilla" hadoop and spark, not CDH ones. The build fails because this file is not there:

Downloading: https://repository.cloudera.com/content/groups/cdh-releases-rcs/org/apache/spark/spark-assembly_2.10/1.4.1/spark-assembly_2.10-1.4.1.pom

Downloading: https://repository.cloudera.com/content/repositories/snapshots/org/apache/spark/spark-assembly_2.10/1.4.1/spark-assembly_2.10-1.4.1.pom

Downloading: https://repo.maven.apache.org/maven2/org/apache/spark/spark-assembly_2.10/1.4.1/spark-assembly_2.10-1.4.1.pom

[Erick Tryzelaar]

Did you modify the maven/pom.xml to set the version to 1.4.1? If so, that explains the problem. We're using the spark-assembly for testing. Cloudera still distributes a spark-assembly jar, but the upstream Spark project doesn't.  This is a great excuse to get rid of it.

In the meantime, you should be able to skip building the tests by running `cd apps/spark/java && mvn package -DskipTests`. That should hopefully skip trying to download the spark assembly and get you on your way. Please let us know if you run into any more problems with it.

-Erick

[dmytro....@bbva.com]

I think, tests are skipped by default in this part. Here is a line from apps/spark/Makefile where the option -DskipTests is set:

cd $(LIVY_DIR) && mvn clean install -DskipTests $(MAVEN_OPTIONS)

Just in case I've also tried to run:

cd apps/spark/java && mvn package -DskipTests

but the result is the same: I get FAILURE for livy-repl_2.10 build

[ash]

Hi,

Did you find any solution to this issue. I am facing the same exact issue.