Issue running hourglass on CDH4

[Aaron Josephs]

I'm trying to get hourglass running on cloudera hadoop and I'm having issues on multiple approaches. Adding it as a dependency like

     <dependency>

            <groupId>com.linkedin.datafu</groupId>

            <artifactId>datafu-hourglass</artifactId>

            <version>0.1.3</version>

        </dependency>

does not work, I believe because hourglass was compiled from a different version of hadoop. I noticed Cloudera does have a fork of datafu (with hourglass in it) but I can't seem to get it to build, any suggestions. 

[Matthew Hayes]

Thanks for reporting this, can you mail the dev mailing list from this wiki?

http://datafu.incubator.apache.org/community/mailing-lists.html

Thanks,

Matt

--
You received this message because you are subscribed to the Google Groups "DataFu" group.
To unsubscribe from this group and stop receiving emails from it, send an email to datafu+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Matthew Hayes]

Actually I will just forward there.

[Aaron Josephs]

No problem, I tried several different approaches to get this to work, messing around with maven dependencies, switching the gradle dependency to cloudera hadoop and building from cloud era's cdh5 datafu branch (cdh4 doesn't seem to have it). Is there any approach in particular you think is the correct way. I can post more detailed error messages if needed.

[Matthew Hayes]

Sure, some error messages would help.  Also please include what version of hadoop you're using when you email the dev mailing list.  I forward the previous email and CC'd you.

[Aaron Josephs]

While attempting to make datafu master compile on cloudera I made these modifications to the code  

diff --git a/build.gradle b/build.gradle
index 9ed6eb1..658f73f 100644
--- a/build.gradle
+++ b/build.gradle
@@ -1,6 +1,13 @@
+
+configurations {
+    provided
+}
 buildscript {
   repositories {
     mavenCentral()
+    maven {
+        url "..." //blanked out for privacy,
+    }
   }


   apply from: file('gradle/buildscript.gradle'), to: buildscript
@@ -16,4 +23,8 @@ apply from: file("gradle/dependency-versions.gradle")


 allprojects {
   apply plugin: 'eclipse'
-}
\ No newline at end of file
+}
+
+dependencies {
+  provided  "org.apache.hadoop:hadoop-client:${hadoopVersion}"
+}
diff --git a/gradle/dependency-versions.gradle b/gradle/dependency-versions.gradle
index eb24e4a..43c679d 100644
--- a/gradle/dependency-versions.gradle
+++ b/gradle/dependency-versions.gradle
@@ -6,7 +6,7 @@ ext {
   commonsIoVersion="2.4"
   fastutilVersion="6.5.7"
   guavaVersion="17.0"
-  hadoopVersion="1.0.4"
+  hadoopVersion="2.0.0-mr1-cdh4.5.0"
   jodaTimeVersion="1.6"
   log4jVersion="1.2.17"
   mavenVersion="2.1.3"

the errors when running are  

Aarons-MacBook-Pro:incubator-datafu ajosephs$ ./gradlew assemble
The TaskContainer.add() method has been deprecated and is scheduled to be removed in Gradle 2.0. Please use the create() method instead.
Creating properties on demand (a.k.a. dynamic properties) has been deprecated and is scheduled to be removed in Gradle 2.0. Please read http://gradle.org/docs/current/dsl/org.gradle.api.plugins.ExtraPropertiesExtension.html for information on the replacement for dynamic properties.
Deprecated dynamic property: "outputFile" on "task ':datafu-pig:jarWithDependenciesNamespaced'", value: "/Users/ajosephs/Code/i...".
:build-plugin:compileJava UP-TO-DATE
:build-plugin:processResources UP-TO-DATE
:build-plugin:classes UP-TO-DATE
:build-plugin:jar UP-TO-DATE
:build-plugin:assemble UP-TO-DATE
:datafu-hourglass:compileJava


FAILURE: Build failed with an exception.


* What went wrong:
Could not resolve all dependencies for configuration ':datafu-hourglass:compile'.
> Could not find org.apache.hadoop:hadoop-core:2.0.0-mr1-cdh4.5.0.
  Required by:
      org.apache.datafu:datafu-hourglass:1.2.1
> Could not find org.apache.hadoop:hadoop-tools:2.0.0-mr1-cdh4.5.0.
  Required by:
      org.apache.datafu:datafu-hourglass:1.2.1


* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.


BUILD FAILED


Total time: 6.599 secs

I have tried multiple other setups of the gradle files (this is my first time using gradle) but this is the most current

[Matthew Hayes]

Can you email these details to the apache mailing list?