Newbie problem about using scalding

[raas]

Sorry if this has been asked before. I am java developer trying scala for hadoop data processing. The scalding project looks really cool and easy. However, after I decided to run some test, I can't get it setup for days with my current project. Is there a way to use scalding like a jar library or include scalding in the dependency easily? Creating a new project just for scalding is not a good option here since we have many other packages already developed. 

[Oscar Boykin]

See the bottom here:

https://github.com/twitter/scalding#building

We publish jars you can use with maven, sbt, etc...:

http://search.maven.org/#search%7Cga%7C1%7Cscalding

In fact, at twitter we use those same jars (we don't rebuild internally).

There are many example projects out there:

https://github.com/twitter/scalding/wiki#how-tos

And also a few that are not on the wiki.

On Thu, May 23, 2013 at 4:07 PM, raas <yangs...@gmail.com> wrote:

Sorry if this has been asked before. I am java developer trying scala for hadoop data processing. The scalding project looks really cool and easy. However, after I decided to run some test, I can't get it setup for days with my current project. Is there a way to use scalding like a jar library or include scalding in the dependency easily? Creating a new project just for scalding is not a good option here since we have many other packages already developed. 

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.
To post to this group, send email to cascadi...@googlegroups.com.
Visit this group at http://groups.google.com/group/cascading-user?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Oscar Boykin :: @posco :: http://twitter.com/posco

[Dean Wampler]

Look at how the scald.rb script works. You would need to compile you scalding scripts, then reference them and the scalding jars in your classpath. The script also shows the JVM options you would need. dand command line arguments you would need to account for.

dean

On Thu, May 23, 2013 at 6:07 PM, raas <yangs...@gmail.com> wrote:

Sorry if this has been asked before. I am java developer trying scala for hadoop data processing. The scalding project looks really cool and easy. However, after I decided to run some test, I can't get it setup for days with my current project. Is there a way to use scalding like a jar library or include scalding in the dependency easily? Creating a new project just for scalding is not a good option here since we have many other packages already developed. 

--

You received this message because you are subscribed to the Google Groups "cascading-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.
To post to this group, send email to cascadi...@googlegroups.com.
Visit this group at http://groups.google.com/group/cascading-user?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Dean Wampler, Ph.D.
@deanwampler
http://polyglotprogramming.com

[raas]

Thanks Oscar. I finally got scalding running on my cluster. Those pages are very helpful. It would be better if you can update the content since some of them are outdated. 

[Oscar Boykin]

Great news.

If you could outline any steps you took, that would be helpful to the next person.

Also, any help updating the wiki is appreciated.

[raas]

The major issue for me is not familiar with sbt. Some of the examples, tutorials assume that readers are familiar with scala/sbt.

 

About the update, e.g., this line is referred to a few times in different places:

libraryDependencies ++= Seq(

"com.twitter" %% "scalding" % "0.8.3"

)




The latest scalding is now called scalding-core.