initial difficulties

[Ulises]

Hi,

I have been trying to get started using Alex Ott's clojure-hadoop lib
however I've ran into difficulties when trying to run the examples. I
added the wordcount examples 4 and 5 to the aot list and built the jar
with 'lein jar'. To run one of the examples I ran:

$ java -cp clojure-hadoop-1.3.1-SNAPSHOT.jar:lib/* clojure_hadoop.job
-job clojure-hadoop.examples.wordcount5/job -input README.txt -output
tmp

which results in

11/02/13 12:02:35 INFO jvm.JvmMetrics: Initializing JVM Metrics with
processName=JobTracker, sessionId=
11/02/13 12:02:35 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics
with processName=JobTracker, sessionId= - already initialized
Exception in thread "main" java.lang.IllegalArgumentException: Don't
know how to create ISeq from: org.apache.hadoop.mapreduce.Job
at clojure.lang.RT.seqFrom(RT.java:471)
at clojure.lang.RT.seq(RT.java:452)
at clojure.core$seq.invoke(core.clj:122)
at clojure_hadoop.config$fn__37.invoke(config.clj:68)
at clojure.lang.MultiFn.invoke(MultiFn.java:171)
at clojure_hadoop.job$run.invoke(job.clj:148)
at clojure_hadoop.job$run.invoke(job.clj:145)
at clojure_hadoop.job$tool_run.invoke(job.clj:161)
at clojure_hadoop.job.run(Unknown Source)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at clojure_hadoop.gen$gen_main_method$fn__117.doInvoke(gen.clj:65)
at clojure.lang.RestFn.applyTo(RestFn.java:138)
at clojure_hadoop.job.main(Unknown Source)
Ulises-Cervino-Beresis-MacBook:clojure-hadoop ulises$

Interestingly enough, lein test does not fail and even running the
examples from the repl with (run job) fails with low heap memory (so I
assume it'd work if I added more memory to the repl session).

I haven't dug into the code yet but I was wondering if there's
anything obvious that I'm missing? If so, then perhaps updating the
instructions on how to run the examples might be in order.

Cheers,

U

[Alex Ott]

Hmm, this is strange - we need to dig deeper, to find what causes this
to fail, because it worked in 1.2.0 without any problems - I had
similar code that I use in my work

--
With best wishes,                    Alex Ott, MBA
http://alexott.net/
Tiwtter: alexott_en (English), alexott (Russian)
Skype: alex.ott

[Ulises]

> Hmm, this is strange - we need to dig deeper, to find what causes this
> to fail, because it worked in 1.2.0 without any problems - I had
> similar code that I use in my work

I tried debugging it by adding lots of println and at some point conf
gets passed :job and a Job object and this is where it fails. I tried
patching things by adding a (map? ...) to cond and ignoring when :else
but that didn't quite work. I will try to dig deeper and also on 1.2.0
to see if this changes things. Thanks for looking into it.

U

[Ulises]

I can confirm that the examples run fine from the command line in
version 1.2.0 (stable in https://github.com/alexott/clojure-hadoop).
Command line used:

$ mvn assembly:assembly
$ java -Xmx1024M -cp target/clojure-hadoop-1.2.0-examples.jar

clojure_hadoop.job -job clojure-hadoop.examples.wordcount5/job -input
README.txt -output tmp

U

[Alex Ott]

I uploaded clojure-hadoop 1.2.0 to clojars, so you can use it in your
projects...

--

[Ulises]

> I uploaded clojure-hadoop 1.2.0 to clojars, so you can use it in your
> projects...

Thanks!

I'm wondering now what the main differences between 1.2.0 and 1.3.x are?

U

[Alex Ott]

As I remember, r0man had added ability to run code from repl, there
was a changes in setting configuration params, and some more changes.

--

[Ulises]

> As I remember, r0man had added ability to run code from repl, there
> was a changes in setting configuration params, and some more changes.

Running jobs from the repl would definitely be a plus, although I
guess one can always debug their mappers and reducers in the repl
anyway.

Thanks for the explanation.

U