Only variable scoped in the run() method could be seriliazed to map reduce function ?

[Jeff Zhang]

Hi 

I'd like to use variable scooped in the Job class in map reduce function, but it will through NullPointerException (the variable in map reduce phase is null).

And if I declare these local variable in the run() method, it could work. So my question is that does it mean only variable in the scope of run() method could be used seriliazed to map reduce function ?

BTW I am using 0.7.0-SNAPSHOT

// the following is the code snippet that will through NullPointerException

object FlowMetricsJob extends FPJob {

  val fieldDelimiter = ","

  val splitter = Splitter.on(fieldDelimiter)

  def run(){

     // use fieldDelimiter & splitter in map reduce function 

  }

}

[Alex Cozzi]

You need to declare them as "lazy val" to make it work. It is a bit of a pain but it is intrinsic to the way initializers and serialization interact.

[Eric Springer]

I know we had a lot of trouble trying to fix this sort of issue
directly, as its quite involved -- but I think detecting this and
throwing an error would be **immensely** useful, and would be be
pretty much as good as a compile error.

> --
> You received this message because you are subscribed to the Google Groups
> "scoobi-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to scoobi-users...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Eric Torreborre]

I'm wondering if we could try to do a first pass of serialisation with XML and parse to result to see if there's a way to reliably find those vals before submitting the job.

E.

[Ben Lever]

Another thought ...

I noticed that FPJob is mixed-in, which presumably mixes-in ScoobiApp. Do you see any differences if ScoobiApp is mixed in directly?