Problems building Scalding and Running REPL locally

137 views
Skip to first unread message

og...@spotify.com

unread,
May 6, 2016, 9:34:09 AM5/6/16
to Scalding Development
Hi,

I am not able to build Scalding properly as some tests fail:

[error] (scalding-hadoop-test/test:test) sbt.TestsFailedException: Tests unsuccessful
[error] (scalding-parquet/test:test) sbt.TestsFailedException: Tests unsuccessful
[error] (scalding-thrift-macros/test:test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 1235 s, completed May 5, 2016 3:33:07 PM

Also I could not run REPL:

./sbt "scalding-repl/run --local"
[info] Loading project definition from /Users/oguz/projects/training/scalding/project
[info] Set current project to scalding (in build file:/Users/oguz/projects/training/scalding/)
[info] Running com.twitter.scalding.ScaldingShell --local
log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
[error] (run-main-0) java.lang.NoClassDefFoundError: cascading/flow/FlowConnector
java.lang.NoClassDefFoundError: cascading/flow/FlowConnector
at com.twitter.scalding.Mode$.apply(Mode.scala:87)
at com.twitter.scalding.BaseScaldingShell$class.parseModeArgs(ScaldingShell.scala:117)
at com.twitter.scalding.ScaldingShell$.parseModeArgs(ScaldingShell.scala:196)
at com.twitter.scalding.BaseScaldingShell$class.process(ScaldingShell.scala:66)
at com.twitter.scalding.ScaldingShell$.process(ScaldingShell.scala:196)
at com.twitter.scalding.BaseScaldingShell$class.main(ScaldingShell.scala:127)
at com.twitter.scalding.ScaldingShell$.main(ScaldingShell.scala:196)
at com.twitter.scalding.ScaldingShell.main(ScaldingShell.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
Caused by: java.lang.ClassNotFoundException: cascading.flow.FlowConnector
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at com.twitter.scalding.Mode$.apply(Mode.scala:87)
at com.twitter.scalding.BaseScaldingShell$class.parseModeArgs(ScaldingShell.scala:117)
at com.twitter.scalding.ScaldingShell$.parseModeArgs(ScaldingShell.scala:196)
at com.twitter.scalding.BaseScaldingShell$class.process(ScaldingShell.scala:66)
at com.twitter.scalding.ScaldingShell$.process(ScaldingShell.scala:196)
at com.twitter.scalding.BaseScaldingShell$class.main(ScaldingShell.scala:127)
at com.twitter.scalding.ScaldingShell$.main(ScaldingShell.scala:196)
at com.twitter.scalding.ScaldingShell.main(ScaldingShell.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
[trace] Stack trace suppressed: run last scalding-repl/unprovided:run for the full output.
java.lang.RuntimeException: Nonzero exit code: 1
at scala.sys.package$.error(package.scala:27)
[trace] Stack trace suppressed: run last scalding-repl/*:run for the full output.
[error] (scalding-repl/*:run) Nonzero exit code: 1
[error] Total time: 2 s, completed May 6, 2016 9:26:19 AM


Can anyone help me with this? 

Marco Pennekamp

unread,
Oct 25, 2016, 8:04:55 PM10/25/16
to Scalding Development
You've probably fixed your problem already, but this is one of the first results when googling the problem, so I want to share my fix regardless.

I am running OS X Sierra. I also had the specific test suites fail. It took me some time to figure out the root cause, but my JAVA_HOME wasn't set correctly. I set it to (in .bash_profile):

export JAVA_HOME=$(/usr/libexec/java_home)

Now my tests pass at least. If this does not fix the problem, you could try to read the Hadoop cluster logs, which are scattered all over the place in the Scalding directory. These log files are called stderr and stdout though, so you might want to search for that. Alternatively, the cluster log folders are prefixed with "application_".

I haven't managed to run the REPL yet, which is still failing with "NoClassDefFoundError" for me.

David Getchell

unread,
Oct 26, 2016, 12:19:24 PM10/26/16
to Scalding Development
I'm getting the same issue.  My JAVA_HOME is set and points to jdk1.7.0_67-cloudera.  I'm running on hadoop 2.6.0-cdh5.7.1. Scala is version 2.10.4.

Any ideas what I should look at, or who I should ask?

P. Oscar Boykin

unread,
Oct 26, 2016, 1:20:27 PM10/26/16
to David Getchell, Scalding Development
I can repro this as well. Step 1, is raise an issue:


It seems like a build issue to me, but not obvious why it is not working. The repl depends on scalding code, and scalding core depends on cascading, which should provide this class.

--
You received this message because you are subscribed to the Google Groups "Scalding Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scalding-dev...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

P. Oscar Boykin

unread,
Oct 26, 2016, 1:22:29 PM10/26/16
to David Getchell, Scalding Development
Does running with:

`./sbt scalding-repl/console` work for you?

Marco Pennekamp

unread,
Oct 26, 2016, 1:38:03 PM10/26/16
to Scalding Development, da...@getchwood.com
Some additional info: I ran the command as root: `sudo ./sbt "scalding-repl/run --local"`, which downloaded the dependencies again and brought up the REPL. I had to chown the scalding folder to my user again, but now the REPL is also working without root privileges.

Executing `./sbt scalding-repl/console` works for me, but I can't say if it would have worked without the sudo "hack". 

David Getchell

unread,
Oct 26, 2016, 2:53:25 PM10/26/16
to Scalding Development, da...@getchwood.com
Why yes, yes it does.  And even though the system's Scala installation is 2.10.4, this is coming up as Scala 2.11.8.

Alex Levenson

unread,
Oct 26, 2016, 4:04:51 PM10/26/16
to David Getchell, Scalding Development
Invoking ./sbt (instead of `which sbt`) uses the version of sbt / scala / etc that scalding is setup to use, which is nice because it decouples us from what you might have installed on your system.

So that part at least is expected (that it uses scalding's version of scala) 

To unsubscribe from this group and stop receiving emails from it, send an email to scalding-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Alex Levenson
@THISWILLWORK
Reply all
Reply to author
Forward
0 new messages