packaging issue with nscala-time?

[Elliot Chow]

I am using nscala-time in my project and I seem to be having some packaging problems.  On my laptop, my job compiles and runs.  However, after packaging and putting it on the cluster, it fails with NoSuchMethodError.  I suspected libjars to be the culprit but I have been hardcoding "override lazy val upload = false" in my jobs and have also tried "nolibjars" in the command line so I do not see the libjars directory on hadoop.  Has anyone also run into this issue?  It also seems like it might be related to (my setup of) sbt or sbt-assembly since some of my coworkers have been successfully running jobs with maven.  Also, I have used the older version of the joda-time wrappers before, but I have not been able to attribute these problems necessarily to this library.

I have  "com.github.nscala-time" %% "nscala-time" % "0.2.0" as a library dependency and I am using the suggested merge strategy

mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
  {
    case x => {
      val oldstrat = old(x)
      if (oldstrat == MergeStrategy.deduplicate) MergeStrategy.first
      else oldstrat
    }
  }
                                                         }


This is the actual error I get on the mappers:

Error running child : java.lang.NoSuchMethodError: org.joda.time.LocalDate.compareTo(Lorg/joda/time/ReadablePartial;)I
    at com.ebay.mlr.dp.hadoop.jobs.Test$$anonfun$5.apply(Test.scala:69)
    at com.ebay.mlr.dp.hadoop.jobs.Test$$anonfun$5.apply(Test.scala:67)
    at com.nicta.scoobi.core.DList$$anonfun$filter$1.apply(DList.scala:134)
    at com.nicta.scoobi.core.DList$$anonfun$filter$1.apply(DList.scala:134)
    at com.nicta.scoobi.core.DList$$anon$1.process(DList.scala:99)
    at com.nicta.scoobi.core.DoFn$class.process(DoFn.scala:27)
    at com.nicta.scoobi.core.DList$$anon$1.process(DList.scala:97)
    at com.nicta.scoobi.core.DList$$anon$1.process(DList.scala:97)
    at com.nicta.scoobi.impl.plan.Smart$ParallelDo$$anonfun$optFuseParDos$2$$anon$8$$anon$9.emit(Smart.scala:280)
    at com.nicta.scoobi.core.DList$$anonfun$flatMap$1$$anonfun$apply$1.apply(DList.scala:123)
    at com.nicta.scoobi.core.DList$$anonfun$flatMap$1$$anonfun$apply$1.apply(DList.scala:123)
    at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
    at scala.collection.immutable.List.foreach(List.scala:76)
    at com.nicta.scoobi.core.DList$$anonfun$flatMap$1.apply(DList.scala:122)
    at com.nicta.scoobi.core.DList$$anonfun$flatMap$1.apply(DList.scala:122)
    at com.nicta.scoobi.core.DList$$anon$1.process(DList.scala:99)
    at com.nicta.scoobi.core.DoFn$class.process(DoFn.scala:27)
    at com.nicta.scoobi.core.DList$$anon$1.process(DList.scala:97)
    at com.nicta.scoobi.core.DList$$anon$1.process(DList.scala:97)
    at com.nicta.scoobi.impl.plan.Smart$ParallelDo$$anonfun$optFuseParDos$2$$anon$8$$anon$9.emit(Smart.scala:280)
    at com.nicta.scoobi.core.DList$$anonfun$map$1.apply(DList.scala:130)
    at com.nicta.scoobi.core.DList$$anonfun$map$1.apply(DList.scala:130)
    at com.nicta.scoobi.core.DList$$anon$1.process(DList.scala:99)
    at com.nicta.scoobi.core.DoFn$class.process(DoFn.scala:27)
    at com.nicta.scoobi.core.DList$$anon$1.process(DList.scala:97)
    at com.nicta.scoobi.core.DList$$anon$1.process(DList.scala:97)
    at com.nicta.scoobi.impl.plan.Smart$ParallelDo$$anonfun$optFuseParDos$2$$anon$8$$anon$9.emit(Smart.scala:280)
    at com.nicta.scoobi.impl.plan.DListImpl$$anon$1.process(DListImpl.scala:35)
    at com.nicta.scoobi.core.DoFn$class.process(DoFn.scala:27)
    at com.nicta.scoobi.impl.plan.DListImpl$$anon$1.process(DListImpl.scala:35)
    at com.nicta.scoobi.impl.plan.DListImpl$$anon$1.process(DListImpl.scala:35)
    at com.nicta.scoobi.impl.plan.Smart$ParallelDo$$anonfun$optFuseParDos$2$$anon$8.process(Smart.scala:280)
    at com.nicta.scoobi.impl.plan.Smart$ParallelDo$$anonfun$optFuseParDos$2$$anon$8.process(Smart.scala:273)
    at com.nicta.scoobi.impl.plan.Smart$ParallelDo$$anonfun$optFuseParDos$2$$anon$8.process(Smart.scala:280)
    at com.nicta.scoobi.impl.plan.Smart$ParallelDo$$anonfun$optFuseParDos$2$$anon$8.process(Smart.scala:273)
    at com.nicta.scoobi.impl.plan.Smart$ParallelDo$$anonfun$optFuseParDos$2$$anon$8.process(Smart.scala:280)
    at com.nicta.scoobi.impl.plan.Smart$ParallelDo$$anonfun$optFuseParDos$2$$anon$8.process(Smart.scala:273)
    at com.nicta.scoobi.impl.plan.Smart$ParallelDo$$anonfun$optFuseParDos$2$$anon$8.process(Smart.scala:280)
    at com.nicta.scoobi.impl.plan.Smart$ParallelDo$$anonfun$optFuseParDos$2$$anon$8.process(Smart.scala:273)
    at com.nicta.scoobi.impl.plan.AST$GbkMapper$$anon$7.map(AST.scala:94)
    at com.nicta.scoobi.impl.exec.MscrMapper$$anonfun$map$1.apply(MscrMapper.scala:73)
    at com.nicta.scoobi.impl.exec.MscrMapper$$anonfun$map$1.apply(MscrMapper.scala:63)
    at scala.collection.immutable.Set$Set1.foreach(Set.scala:86)
    at com.nicta.scoobi.impl.exec.MscrMapper.map(MscrMapper.scala:63)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1178)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)

[Eric Torreborre]

Hi Elliot,

Can you please check what ends up being on the mapred.classpath variable in the configuration on the cluster?

Thanks,

Eric.

[Elliot Chow]

Hi Eric,

It doesn't look like there is a value set explicitly.

Anything else I should check?

[Eric Torreborre]

That might be the issue, can you please post the DEBUG logs?

Thanks.

[Elliot Chow]

Here are the logs.  Thanks.

[Alex Cozzi]

To help reproduce the problem I took your wordcount and added a single call to joda time (see the attached files)

the program runs locally, but it thrown and exception when running on the cluster after being packaged with "assembly"

Here is the error

> hadoop jar MyApp-assembly-0.1.jar 

Exception in thread "main" java.lang.NoSuchMethodError: org.joda.time.LocalDate.now()Lorg/joda/time/LocalDate;

        at main.scala.WordCount$.run(WordCount.scala:23)

        at com.nicta.scoobi.application.ScoobiApp$$anonfun$main$1.apply$mcV$sp(ScoobiApp.scala:78)

        at com.nicta.scoobi.application.ScoobiApp$$anonfun$main$1.apply(ScoobiApp.scala:75)

        at com.nicta.scoobi.application.ScoobiApp$$anonfun$main$1.apply(ScoobiApp.scala:75)

        at com.nicta.scoobi.application.Hadoop$class.runOnCluster(Hadoop.scala:99)

        at main.scala.WordCount$.runOnCluster(WordCount.scala:21)

        at com.nicta.scoobi.application.Hadoop$class.executeOnCluster(Hadoop.scala:58)

        at main.scala.WordCount$.executeOnCluster(WordCount.scala:21)

        at com.nicta.scoobi.application.Hadoop$$anonfun$onCluster$1.apply(Hadoop.scala:44)

        at com.nicta.scoobi.application.InMemoryHadoop$class.withTimer(InMemory.scala:49)

        at main.scala.WordCount$.withTimer(WordCount.scala:21)

        at com.nicta.scoobi.application.InMemoryHadoop$class.showTime(InMemory.scala:57)

        at main.scala.WordCount$.showTime(WordCount.scala:21)

        at com.nicta.scoobi.application.Hadoop$class.onCluster(Hadoop.scala:44)

        at main.scala.WordCount$.onCluster(WordCount.scala:21)

        at com.nicta.scoobi.application.Hadoop$class.onHadoop(Hadoop.scala:50)

        at main.scala.WordCount$.onHadoop(WordCount.scala:21)

        at com.nicta.scoobi.application.ScoobiApp$class.main(ScoobiApp.scala:75)

        at main.scala.WordCount$.main(WordCount.scala:21)

        at main.scala.WordCount.main(WordCount.scala)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597)

        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

[Alex Cozzi]

here is build.sbt:

import AssemblyKeys._

assemblySettings

name := "MyApp"

version := "0.1"

scalaVersion := "2.9.2"

libraryDependencies += "com.nicta" %% "scoobi" % "0.6.1-cdh3"

libraryDependencies += "joda-time" % "joda-time" % "2.1"

libraryDependencies += "org.joda" % "joda-convert" % "1.2"

scalacOptions ++= Seq("-Ydependent-method-types", "-deprecation")

resolvers ++= Seq(

    "nicta's avro" at "http://nicta.github.com/scoobi/releases",

    "cloudera" at "https://repository.cloudera.com/content/repositories/releases",

    "Sonatype-snapshots" at "http://oss.sonatype.org/content/repositories/snapshots")

    

    

    

mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>

  {

    case x => {

      val oldstrat = old(x)

      if (oldstrat == MergeStrategy.deduplicate) MergeStrategy.first

      else oldstrat

    }

  }

}

=======================================

and WordCount.scala with the change in red

=======================================

/**

 * Copyright 2011,2012 National ICT Australia Limited

 *

 * Licensed under the Apache License, Version 2.0 (the "License");

 * you may not use this file except in compliance with the License.

 * You may obtain a copy of the License at

 *

 * http://www.apache.org/licenses/LICENSE-2.0

 *

 * Unless required by applicable law or agreed to in writing, software

 * distributed under the License is distributed on an "AS IS" BASIS,

 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

 * See the License for the specific language governing permissions and

 * limitations under the License.

 */

package main.scala

import com.nicta.scoobi.Scoobi._

import org.joda.time.LocalDate

object WordCount extends ScoobiApp {

  def run() {

    val now = LocalDate.now

    // Firstly we load up all the (new-line-separated) words into a DList

    val lines: DList[String] =

      if (args.length == 0)

        DList(generateWords(5000): _*)

      else if (args.length == 2)

        fromTextFile(args(0))

      else

        sys.error("Expecting input and output path, or no arguments at all.")

    // Now what we want to do, is record the frequency of words. So we'll convert it to a key-value

    // pairs where the key is the word, and the value the frequency (which to start with is 1)

    val keyValuePair: DList[(String, Int)] = lines flatMap { _.split(" ") } map { w => (w, 1) }

    // Now let's group all words that compare the same

    val grouped: DList[(String, Iterable[Int])] = keyValuePair.groupByKey

    // Now we have it in the form (Word, ['1', '1', '1', 1' etc.])

    // So what we want to do, is combine all the numbers into a single value (the frequency)

    val combined: DList[(String, Int)] = grouped.combine(_ + _)

    val outputDirectory: String = if (args.length == 0) "word-count-results" else args(1)

    // We can evaluate this, and write it to a text file

    persist(toTextFile(combined, outputDirectory, overwrite = true))

  }

  /* Generate 'count' random words with a high amount of collisions */

  private def generateWords(count: Int) = {

    val r = new scala.util.Random()

    // function to make a 5 letter random "word"

    def randomWord(): String = {

      val wordLength = 5

      val sb = new StringBuilder(wordLength)

      (1 to wordLength) foreach {

        _ => sb.append(('A' + r.nextInt('Z' - 'A')).asInstanceOf[Char])

      }

      sb toString

    }

    // we start off by generating count/10 different "words"

    var words: IndexedSeq[String] =

      for (i <- 1 to count / 10)

        yield randomWord()

    // and now we will pick 'count' of them to output

    for (i <- 1 to count)

      yield words(r.nextInt(words.length))

  }

}