Scalding: Neither Java nor Kyro works for class

[Soumitra Kumar]

Hello,

I have following code:

class SrcSummary (args : Args) extends Job(args) {

  val keys = Set ("src") // FAILS

  val scan = new Scan
  val scanner = HBaseRawSource.convertScanToString (scan)

  val hpipe = new HBaseRawSource (
    tableName = "events",
    quorumNames = "localhost:2181",
    familyNames = Array ("cf"),
    base64Scan = scanner)
      .read
      .flatMapTo ('row -> 'data) {
        result : Result => {
          // val keys = Set ("src") // WORKS

As it is I get following errors:
https://github.com/twitter/scalding/wiki/Common-Exceptions-and-possible-reasons#cascadingflowplannerplannerexception
    at com.twitter.scalding.Tool$.main(Tool.scala:154)
    at com.twitter.scalding.Tool.main(Tool.scala)
Caused by: cascading.flow.planner.PlannerException: could not build flow from assembly: [Neither Java nor Kyro works for class: class SrcSummary$$anonfun$3 instance: <function1>
export CHILL_EXTERNALIZER_DEBUG=true to see both stack traces]
    at cascading.flow.planner.FlowPlanner.handleExceptionDuringPlanning(FlowPlanner.java:533)
    at cascading.flow.hadoop.planner.HadoopPlanner.buildFlow(HadoopPlanner.java:237)
...
Caused by: java.lang.RuntimeException: Neither Java nor Kyro works for class: class SrcSummary$$anonfun$3 instance: <function1>
export CHILL_EXTERNALIZER_DEBUG=true to see both stack traces

After setting 'export CHILL_EXTERNALIZER_DEBUG=true', I get 677040 lines of stack trace.

I set '-Dsun.io.serialization.extendedDebugInfo=true', but did not get any useful information.

If I move 'keys' within block, it works.

Thanks,
-Soumitra.

[Oscar Boykin]

Sounds like you solved your problem.

1) What version of scalding?

2) When you move keys inside the function, the closure that scala compiles is different and you don't capture the Job instance (and thus, it is not serialized by cascading).

3) making the vals lazy might also fix the issue.

Serialization of closures is a pain.

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.
To post to this group, send email to cascadi...@googlegroups.com.
Visit this group at http://groups.google.com/group/cascading-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/40ac57ff-8599-4445-aec8-cc1b5170f94e%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Oscar Boykin :: @posco :: http://twitter.com/posco

[Soumitra Kumar]

Thanks Oscar.

On Tuesday, November 12, 2013 4:10:23 PM UTC-8, Oscar Boykin wrote:

Sounds like you solved your problem.

1) What version of scalding?

0.9.0

2) When you move keys inside the function, the closure that scala compiles is different and you don't capture the Job instance (and thus, it is not serialized by cascading).

3) making the vals lazy might also fix the issue.

It did not.

Scalding is great, and am going to depend more on it.

I want to get inside Scalding code to debug/fix future issues, would appreciate any pointer.

[Oscar Boykin]

On Tue, Nov 12, 2013 at 4:22 PM, Soumitra Kumar <kumar.s...@gmail.com> wrote:

Thanks Oscar.

On Tuesday, November 12, 2013 4:10:23 PM UTC-8, Oscar Boykin wrote:

Sounds like you solved your problem.

1) What version of scalding?

0.9.0

Not released yet. rc1? rc4?

 

2) When you move keys inside the function, the closure that scala compiles is different and you don't capture the Job instance (and thus, it is not serialized by cascading).

3) making the vals lazy might also fix the issue.

It did not.

Scalding is great, and am going to depend more on it.

I want to get inside Scalding code to debug/fix future issues, would appreciate any pointer.

Wonderful!

A good starting point is to look at the issue tracker:

https://github.com/twitter/scalding/issues?state=open

Pick something that seems useful (or start at the top and work down), and I can collaborate with you to get a PR through.

 

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.
To post to this group, send email to cascadi...@googlegroups.com.
Visit this group at http://groups.google.com/group/cascading-user.

To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/8f4108a4-b901-45e0-8229-1bcf53f80bb1%40googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

[Jianbao Tao]

Hi Oscar,

Please allow me to reuse this thread. Basically, I am running into a very similar issue now. My code looks like this:

val threshold = new SimpleDateFormat("yyyy-MM-dd").parse(args("threshold")).getTime
val helper = new SomeHelperClass
inputTypedPipe.map(helper.doSomething(threshold)).write(...)

This doesn't work with error like this: "Caused by: java.lang.RuntimeException: Neither Java nor Kyro works for class: class com.twitter.scalding.typed.MapFn instance: <function1>"

However, if I change the signature of `helper.doSomething` so that it doesn't take the threshold parameter and instead use a hardcoded threshold inside the method body, then it will work, which is basically the same behavior as in the original post.

So, is there a good solution to solve this kind of issue? I mean, it is common to parameterize the mapping function used by TypedPipe. But right now it seems I have to hard code the parameters inside the mapping functions.

Many thanks.

Jianbao