Problems when running on a cluster

[cameron]

Hi,
I've been looking at using clojure-hadoop for some evolutionary
algorithm work
and have been seeing some odd differences when jobs are run from the
repl vs. being run from the command line or on a cluster

If I add (set! *warn-on-reflection* true) to any of the examples in
version 1.2 or the Alex Ott 1.3 branch the job runs from the repl but
fails when run on the cluster with the following error:
#<CompilerException java.lang.IllegalStateException: Can't change/
establish root binding of: *warn-on-reflection* with set (core.clj:0)>

I'm also having problems on the cluster but not locally when I eval
generated code, I get stack traces like the following:
java.lang.reflect.UndeclaredThrowableException: Unknown exception in
doAs
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:
1127)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.security.PrivilegedActionException:
java.lang.IllegalArgumentException: No matching method: gte
(NO_SOURCE_FILE:0)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:
1115)
... 1 more
Caused by: java.lang.IllegalArgumentException: No matching method: gte
(NO_SOURCE_FILE:0)
....

It seems like some of clojures more dynamic behaviour isn't available
for code that's called by the Hadoop Tool.run.
I'd appreciate any pointers, I'm a little lost on where to look next.
Is there some clojure initialisation I'm missing in the tool?

Thanks,
Cameron.

[RD]

Hi  Cameron,
     I think It makes more sense to use *warn-on-reflection* only when you are working with your code locally in your REPL.
Also, when you are using set! you are trying the change the root-binding of the var. This is not permitted. Have a look here

I'm not too sure about the "gte" exception. Is this defined by you? Maybe you can elaborate a litte more on that as to what you are doing?

rdsr

[cameron]

Hi,
I agree that the warn on relflection setting only makes sense at the
repl or during compilaton,
I was under the impression that the var would always have a root
binding with the default value of false and I was a little surprised
at the difference in behaviour.

The gte function is defined in one of my namespaces, I'm using the
java gramatical evolution library GEVA to generate a population of s-
expressions which
are then evaluated with (eval (read-string expr)). The gte symbol is
namespace qualified in the grammar, a sample generated expression
could look like this:

(fn [val]
(ge-test.experiments.symreg/gte
val
(geva/constant (int (+ 1 (mod (geva/to-integer 5 1 ) 599)))) ))

The same code seems to work for my initial implementation using pmap
when run from the command line and the hadoop version works when run
from the repl with the 1.3 changes.
I guess I'm confused as to why the code seems to behave differently
when called from the hadoop command line (via the clojure_hadoop.job/
tool-run method).

Cameron.