Strange exception intializing clojure.core using Spring-Hadoop

[Dave Kincaid]

I'm posting this here in hopes that someone might be able to steer us in the right direction. We have a Cascalog process that we're using Spring-Hadoop & Spring-Batch to send to a remote Hadoop cluster. It seems as though Spring-Hadoop is doing something funky with the classpath/classloader and we're getting the following exception when we run it:

java.lang.ExceptionInInitializerError
at clojure.core__init.__init0(Unknown Source)
at clojure.core__init.<clinit>(Unknown Source)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:266)
at clojure.lang.RT.loadClassForName(RT.java:2098)
at clojure.lang.RT.load(RT.java:430)
at clojure.lang.RT.load(RT.java:411)
at clojure.lang.RT.doInit(RT.java:447)
at clojure.lang.RT.<clinit>(RT.java:329)
at cascalog.Util.<clinit>(Util.java:29)
at jcascalog.Api.setApplicationConf(Api.java:99)
at com.test.DataShredder.run(DataShredder.java:113)

in trying to trace it we think that this is happening while clojure.lang.RT is scanning all the classes on the classpath. It seems to get to one class called StepExecution.class, which is part of the Spring Framework, and it throws this exception. We've got this posted over on the Spring forums too, since it's most likely something with Spring's manipulation of classpath and/or classloader while it's trying to get the MR jobs over to Hadoop.

If anyone has another idea, we'd love to hear it. We're kind of stuck right now and been working on it for a few days.

Thanks,

DAve

[Dave Kincaid]

A quick update on a little more progress troubleshooting this issue. We have gotten to the point where we are seeing this stacktrace:

java.lang.IllegalStateException: Attempting to call unbound fn: #'clojure.core/refer

    at clojure.lang.Var$Unbound.throwArity(Var.java:43)

    at clojure.lang.AFn.invoke(AFn.java:39)

    at clojure.lang.Var.invoke(Var.java:415)

    at clojure.lang.RT.doInit(RT.java:460)

    at clojure.lang.RT.<clinit>(RT.java:329)

does that give anyone an idea?

[atkaaz]

I've some idea, but it may not be right; I'm thinking that clojure needs its own classloader and if that spring thing overriden it somehow, it's not going to work initing clojure, just like in minecraft bukkit server with clojure-based plugins, ie. https://github.com/CmdrDats/clj-minecraft/

 I'm thinking maybe you need to do something similar with temporarily restoring clojure's classloader when clojure inits or something like that (I don't remember) but you may look at the code in clj-minecraft like here: https://github.com/CmdrDats/clj-minecraft/blob/master/javasrc/cljminecraft/BasePlugin.java#L56

What clojure version were you using when the above stacktrace happened?

--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[atkaaz]

looks like it didn't properly load clojure.core (possibly due to that classloader being "wrong"?) but I am not sure why it didn't fail sooner than on the line with refer

static void doInit() throws ClassNotFoundException, IOException{
    load("clojure/core"); //this wasn't loaded ok?!

    Var.pushThreadBindings(
            RT.mapUniqueKeys(CURRENT_NS, CURRENT_NS.deref(),
                   WARN_ON_REFLECTION, WARN_ON_REFLECTION.deref()
                    ,RT.UNCHECKED_MATH, RT.UNCHECKED_MATH.deref()));
    try {
        Symbol USER = Symbol.intern("user");
        Symbol CLOJURE = Symbol.intern("clojure.core");

        Var in_ns = var("clojure.core", "in-ns");
        Var refer = var("clojure.core", "refer");
        in_ns.invoke(USER); //wonder why it didn't fail here
        refer.invoke(CLOJURE); //fails here
        maybeLoadResourceScript("user.clj");
    }
    finally {
        Var.popThreadBindings();
    }
}

On Fri, May 17, 2013 at 7:38 PM, Dave Kincaid <kincai...@gmail.com> wrote:

--

[sudheer babu]

It was nice article it was very useful for me as well as useful for <a href="http://123trainings.com/it-hadoop-bigdata-online-training.html">online Hadoop training</a> learners.thanks for providing this valuable information.123trainings provides best <a href="https://www.youtube.com/watch?v=PqbYn5LXzRw">Hadoop online training</a>