isses with classpath in scoobi

Alex Cozzi

unread,

Feb 26, 2013, 4:10:07 PM2/26/13

to scoobi...@googlegroups.com

(changed title from https://groups.google.com/forum/?fromgroups=#!topic/scoobi-users/vBJixQSeIAI to better reflect the issue)

I was able to reproduce the issue with the latests scoobi snapshot as well (0.7-cdh3-SNAPSHOT)

Exception in thread "main" java.lang.NoSuchMethodError: org.joda.time.LocalDate.now()Lorg/joda/time/LocalDate;

at com.ebay.scoobi.examples.Listings$.run(Listings.scala:21)

at com.nicta.scoobi.application.ScoobiApp$$anonfun$main$1.apply$mcV$sp(ScoobiApp.scala:77)

at com.nicta.scoobi.application.ScoobiApp$$anonfun$main$1.apply(ScoobiApp.scala:74)

at com.nicta.scoobi.application.Hadoop$class.runOnCluster(Hadoop.scala:100)

at com.ebay.scoobi.examples.Listings$.runOnCluster(Listings.scala:15)

at com.nicta.scoobi.application.Hadoop$class.executeOnCluster(Hadoop.scala:59)

at com.ebay.scoobi.examples.Listings$.executeOnCluster(Listings.scala:15)

at com.nicta.scoobi.application.Hadoop$$anonfun$onCluster$1.apply(Hadoop.scala:45)

at com.nicta.scoobi.application.InMemoryHadoop$class.withTimer(InMemory.scala:64)

at com.ebay.scoobi.examples.Listings$.withTimer(Listings.scala:15)

at com.nicta.scoobi.application.InMemoryHadoop$class.showTime(InMemory.scala:72)

at com.ebay.scoobi.examples.Listings$.showTime(Listings.scala:15)

at com.nicta.scoobi.application.Hadoop$class.onCluster(Hadoop.scala:45)

at com.ebay.scoobi.examples.Listings$.onCluster(Listings.scala:15)

at com.nicta.scoobi.application.Hadoop$class.onHadoop(Hadoop.scala:51)

at com.ebay.scoobi.examples.Listings$.onHadoop(Listings.scala:15)

at com.nicta.scoobi.application.ScoobiApp$class.main(ScoobiApp.scala:74)

at com.ebay.scoobi.examples.Listings$.main(Listings.scala:15)

at com.ebay.scoobi.examples.Listings.main(Listings.scala)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

Eric Torreborre

unread,

Feb 27, 2013, 7:15:08 PM2/27/13

to scoobi...@googlegroups.com

This is puzzling.

I couldn't reproduce the issue but I'm on cdh4 so it might be related to that.

I had a look at the dependency graph and I don't see any joda time dependency < 2.1:

[info] +-com.github.nscala-time:nscala-time_2.9.2:0.2.0 [S]

[info] | +-joda-time:joda-time:2.1

[info] | +-org.joda:joda-convert:1.2

[info] |

[info] +-com.nicta:scoobi_2.9.2:0.7.0-cdh3-SNAPSHOT (evicted by: 0.7.0-cdh4-SNAPSHOT)

[info] +-com.nicta:scoobi_2.9.2:0.7.0-cdh4-SNAPSHOT [S]

[info] +-com.chuusai:shapeless_2.9.2:1.2.2 [S]

[info] +-com.github.mdr:ascii-graphs_2.9.2:0.0.2 [S]

[info] +-com.googlecode.kiama:kiama_2.9.2:1.4.0 [S]

[info] | +-jline:jline:1.0

[info] |

[info] +-com.thoughtworks.xstream:xstream:1.4.3

[info] +-javassist:javassist:3.12.1.GA

[info] +-org.apache.avro:avro-mapred:1.7.3.1

[info] | +-org.apache.avro:avro-ipc:1.7.3.1

[info] | | +-io.netty:netty:3.4.0.Final

[info] | | +-org.apache.avro:avro:1.7.3.1

[info] | | | +-com.thoughtworks.paranamer:paranamer:2.3

[info] | | | +-org.codehaus.jackson:jackson-core-asl:1.8.8

[info] | | | +-org.codehaus.jackson:jackson-mapper-asl:1.8.8

[info] | | | | +-org.codehaus.jackson:jackson-core-asl:1.8.8

[info] | | | |

[info] | | | +-org.slf4j:slf4j-api:1.6.4

[info] | | | +-org.xerial.snappy:snappy-java:1.0.4.1

[info] | | |

[info] | | +-org.apache.velocity:velocity:1.7

[info] | | | +-commons-collections:commons-collections:3.2.1

[info] | | | +-commons-lang:commons-lang:2.4 (evicted by: 2.5)

[info] | | |

[info] | | +-org.codehaus.jackson:jackson-core-asl:1.8.8

[info] | | +-org.codehaus.jackson:jackson-mapper-asl:1.8.8

[info] | | | +-org.codehaus.jackson:jackson-core-asl:1.8.8

[info] | | |

[info] | | +-org.mortbay.jetty:jetty-util:6.1.26

[info] | | +-org.mortbay.jetty:jetty:6.1.26

[info] | | | +-org.mortbay.jetty:jetty-util:6.1.26

[info] | | | +-org.mortbay.jetty:servlet-api:2.5-20081211

[info] | | |

[info] | | +-org.mortbay.jetty:servlet-api:2.5-20081211

[info] | | +-org.slf4j:slf4j-api:1.6.4

[info] | |

[info] | +-org.codehaus.jackson:jackson-core-asl:1.8.8

[info] | +-org.codehaus.jackson:jackson-mapper-asl:1.8.8

[info] | | +-org.codehaus.jackson:jackson-core-asl:1.8.8

[info] | |

[info] | +-org.slf4j:slf4j-api:1.6.4

[info] |

[info] +-org.apache.avro:avro:1.5.4 (evicted by: 1.7.3.1)

[info] +-org.apache.avro:avro:1.7.3.1

[info] | +-com.thoughtworks.paranamer:paranamer:2.3

[info] | +-org.codehaus.jackson:jackson-core-asl:1.8.8

[info] | +-org.codehaus.jackson:jackson-mapper-asl:1.8.8

[info] | | +-org.codehaus.jackson:jackson-core-asl:1.8.8

[info] | |

[info] | +-org.slf4j:slf4j-api:1.6.4

[info] | +-org.xerial.snappy:snappy-java:1.0.4.1

[info] |

[info] +-org.apache.hadoop:hadoop-client:2.0.0-mr1-cdh4.0.1

[info] | +-org.apache.hadoop:hadoop-common:2.0.0-cdh4.0.1

[info] | | +-asm:asm:3.2

[info] | | +-com.google.guava:guava:11.0.2

[info] | | | +-com.google.code.findbugs:jsr305:1.3.9

[info] | | |

[info] | | +-com.google.protobuf:protobuf-java:2.4.0a

[info] | | +-com.googlecode.json-simple:json-simple:1.1

[info] | | +-com.jcraft:jsch:0.1.42

[info] | | +-commons-codec:commons-codec:1.4

[info] | | +-commons-collections:commons-collections:3.2.1

[info] | | +-commons-configuration:commons-configuration:1.6

[info] | | | +-commons-beanutils:commons-beanutils-core:1.8.0

[info] | | | | +-commons-logging:commons-logging:1.1.1

[info] | | | |

[info] | | | +-commons-collections:commons-collections:3.2.1

[info] | | | +-commons-digester:commons-digester:1.8

[info] | | | | +-commons-beanutils:commons-beanutils:1.7.0

[info] | | | | +-commons-logging:commons-logging:1.1.1

[info] | | | |

[info] | | | +-commons-lang:commons-lang:2.5

[info] | | | +-commons-logging:commons-logging:1.1.1

[info] | | |

[info] | | +-commons-el:commons-el:1.0

[info] | | | +-commons-logging:commons-logging:1.1.1

[info] | | |

[info] | | +-commons-io:commons-io:2.1

[info] | | +-commons-lang:commons-lang:2.5

[info] | | +-commons-logging:commons-logging:1.1.1

[info] | | +-commons-net:commons-net:3.1

[info] | | +-log4j:log4j:1.2.16

[info] | | +-org.apache.avro:avro:1.5.4 (evicted by: 1.7.3.1)

[info] | | +-org.apache.commons:commons-math:2.1

[info] | | +-org.apache.hadoop:hadoop-auth:2.0.0-cdh4.0.1

[info] | | | +-commons-codec:commons-codec:1.4

[info] | | | +-log4j:log4j:1.2.16

[info] | | | +-org.slf4j:slf4j-api:1.6.1 (evicted by: 1.6.4)

[info] | | | +-org.slf4j:slf4j-log4j12:1.6.1

[info] | | | +-log4j:log4j:1.2.16

[info] | | | +-org.slf4j:slf4j-api:1.6.1 (evicted by: 1.6.4)

[info] | | |

[info] | | +-org.codehaus.jackson:jackson-mapper-asl:1.8.8

[info] | | | +-org.codehaus.jackson:jackson-core-asl:1.8.8

[info] | | |

[info] | | +-org.slf4j:slf4j-api:1.6.1 (evicted by: 1.6.4)

[info] | | +-org.slf4j:slf4j-log4j12:1.6.1

[info] | | | +-log4j:log4j:1.2.16

[info] | | | +-org.slf4j:slf4j-api:1.6.1 (evicted by: 1.6.4)

[info] | | |

[info] | | +-oro:oro:2.0.8

[info] | | +-xmlenc:xmlenc:0.52

[info] | |

[info] | +-org.apache.hadoop:hadoop-core:2.0.0-mr1-cdh4.0.1

[info] | | +-commons-cli:commons-cli:1.2

[info] | | +-commons-codec:commons-codec:1.4

[info] | | +-commons-el:commons-el:1.0

[info] | | | +-commons-logging:commons-logging:1.1.1

[info] | | |

[info] | | +-commons-httpclient:commons-httpclient:3.1

[info] | | | +-commons-codec:commons-codec:1.4

[info] | | | +-commons-logging:commons-logging:1.1.1

[info] | | |

[info] | | +-commons-net:commons-net:3.1

[info] | | +-hsqldb:hsqldb:1.8.0.10

[info] | | +-javax.servlet.jsp:jsp-api:2.1

[info] | | +-javax.servlet:servlet-api:2.5

[info] | | +-net.java.dev.jets3t:jets3t:0.6.1

[info] | | | +-commons-codec:commons-codec:1.4

[info] | | | +-commons-httpclient:commons-httpclient:3.1

[info] | | | | +-commons-codec:commons-codec:1.4

[info] | | | | +-commons-logging:commons-logging:1.1.1

[info] | | | |

[info] | | | +-commons-logging:commons-logging:1.1.1

[info] | | |

[info] | | +-org.codehaus.jackson:jackson-core-asl:1.8.8

[info] | | +-org.codehaus.jackson:jackson-mapper-asl:1.8.8

[info] | | | +-org.codehaus.jackson:jackson-core-asl:1.8.8

[info] | | |

[info] | | +-org.eclipse.jdt:core:3.1.1

[info] | | +-org.mortbay.jetty:jetty-util:6.1.26

[info] | | +-org.mortbay.jetty:jetty:6.1.26

[info] | | | +-org.mortbay.jetty:jetty-util:6.1.26

[info] | | | +-org.mortbay.jetty:servlet-api:2.5-20081211

[info] | | |

[info] | | +-oro:oro:2.0.8

[info] | | +-tomcat:jasper-compiler:5.5.23

[info] | | +-tomcat:jasper-runtime:5.5.23

[info] | | | +-commons-el:commons-el:1.0

[info] | | | | +-commons-logging:commons-logging:1.1.1

[info] | | | |

[info] | | | +-javax.servlet:servlet-api:2.5

[info] | | |

[info] | | +-xmlenc:xmlenc:0.52

[info] | |

[info] | +-org.apache.hadoop:hadoop-hdfs:2.0.0-cdh4.0.1

[info] | +-com.google.protobuf:protobuf-java:2.4.0a

[info] | +-commons-logging:commons-logging:1.1.1

[info] | +-log4j:log4j:1.2.16

[info] |

[info] +-org.apache.hadoop:hadoop-core:2.0.0-mr1-cdh4.0.1

[info] | +-commons-cli:commons-cli:1.2

[info] | +-commons-codec:commons-codec:1.4

[info] | +-commons-el:commons-el:1.0

[info] | | +-commons-logging:commons-logging:1.1.1

[info] | |

[info] | +-commons-httpclient:commons-httpclient:3.1

[info] | | +-commons-codec:commons-codec:1.4

[info] | | +-commons-logging:commons-logging:1.1.1

[info] | |

[info] | +-commons-net:commons-net:3.1

[info] | +-hsqldb:hsqldb:1.8.0.10

[info] | +-javax.servlet.jsp:jsp-api:2.1

[info] | +-javax.servlet:servlet-api:2.5

[info] | +-net.java.dev.jets3t:jets3t:0.6.1

[info] | | +-commons-codec:commons-codec:1.4

[info] | | +-commons-httpclient:commons-httpclient:3.1

[info] | | | +-commons-codec:commons-codec:1.4

[info] | | | +-commons-logging:commons-logging:1.1.1

[info] | | |

[info] | | +-commons-logging:commons-logging:1.1.1

[info] | |

[info] | +-org.codehaus.jackson:jackson-core-asl:1.8.8

[info] | +-org.codehaus.jackson:jackson-mapper-asl:1.8.8

[info] | | +-org.codehaus.jackson:jackson-core-asl:1.8.8

[info] | |

[info] | +-org.eclipse.jdt:core:3.1.1

[info] | +-org.mortbay.jetty:jetty-util:6.1.26

[info] | +-org.mortbay.jetty:jetty:6.1.26

[info] | | +-org.mortbay.jetty:jetty-util:6.1.26

[info] | | +-org.mortbay.jetty:servlet-api:2.5-20081211

[info] | |

[info] | +-oro:oro:2.0.8

[info] | +-tomcat:jasper-compiler:5.5.23

[info] | +-tomcat:jasper-runtime:5.5.23

[info] | | +-commons-el:commons-el:1.0

[info] | | | +-commons-logging:commons-logging:1.1.1

[info] | | |

[info] | | +-javax.servlet:servlet-api:2.5

[info] | |

[info] | +-xmlenc:xmlenc:0.52

[info] |

[info] +-org.scalaz:scalaz-concurrent_2.9.2:7.0.0-M7 [S]

[info] | +-org.scalaz:scalaz-core_2.9.2:7.0.0-M7 [S]

[info] | +-org.scalaz:scalaz-effect_2.9.2:7.0.0-M7 [S]

[info] | +-org.scalaz:scalaz-core_2.9.2:7.0.0-M7 [S]

[info] |

[info] +-org.scalaz:scalaz-core_2.9.2:7.0.0-M7 [S]

So my guess is that there must be a joda-time 2.0 jar somewhere on the cluster (especially if things work ok locally). Could you please check your mapred.classpath configuration variable for the failing job?

Thanks,

Eric.

Alex Cozzi

unread,

Feb 28, 2013, 12:43:34 AM2/28/13

to scoobi...@googlegroups.com

I run it on our other cluster on a different distribution, and I could also not get the error. On cdh3 the error happens before the job setup, so I can not get the mapred.classpath from hadoop. How do I enable more verbose logging? the only think I could think of is to use

"hadoop classpath"

which does not show any joda files.

Eric Torreborre

unread,

Feb 28, 2013, 6:06:36 AM2/28/13

to scoobi...@googlegroups.com

Did you inspect the content of the assembled jar? Does it contain "old" Joda time classes, and is Joda time even included?

Alex Cozzi

unread,

Feb 28, 2013, 12:13:45 PM2/28/13

to scoobi...@googlegroups.com

In the version that I compile with maven it very clearly includes joda-time under the lib directory (that is packaged using maven package task)

570478 Thu Jan 10 20:59:32 PST 2013 lib/joda-time-2.1.jar

38460 Thu Jan 10 20:59:26 PST 2013 lib/joda-convert-1.2.jar

the MyApp compiled with sbt includes it as well:

org/joda/time/LocalTime.class

and I assume that is the right one because of the two lines in the build.sbt:

libraryDependencies += "joda-time" % "joda-time" % "2.1"

libraryDependencies += "org.joda" % "joda-convert" % "1.2"

But I agree with you: there is an old version of joda somewhere in the classpath, and it must be in the classpath set up by the hadoop jar command. I think I will put a "echo" in that bash script to see whether I can find it.

Alex Cozzi

unread,

Feb 28, 2013, 8:35:16 PM2/28/13

to scoobi...@googlegroups.com

I tried to go over the entire classpath on the cluster but saw not trace of joda time anywhere. Very puzzling. Just for the record, the distribution is Hortonworks 1.1.

Alex Cozzi

unread,

Mar 1, 2013, 1:44:18 PM3/1/13

to scoobi...@googlegroups.com

Solved the mystery: apparently our classpath includes: /apache/hbase/lib/* which includes /apache/hbase/lib/jruby-complete-1.6.5.jar which includes:

3035 Tue Sep 14 00:05:04 GMT-07:00 2010 org/joda/time/LocalDate$Property.class

14393 Tue Sep 14 00:05:04 GMT-07:00 2010 org/joda/time/LocalDate.class

3158 Tue Sep 14 00:05:04 GMT-07:00 2010 org/joda/time/LocalDateTime$Property.class

15018 Tue Sep 14 00:05:04 GMT-07:00 2010 org/joda/time/LocalDateTime.class

from some old version of joda time.

Now at least I know why the problem is there, and it has nothing to do with scoobi. Now I just need to find out how to override it.

Eric Torreborre

unread,

Mar 2, 2013, 10:47:19 PM3/2/13

to scoobi...@googlegroups.com

I'm glad you found it! Maybe now you can tweak the mapred.classpath variable to reorder it and have the more recent joda-time coming first.

E.

Reply all

Reply to author

Forward