ClassNotFoundException: cascading.tap.hadoop.io.MultiInputSplit

[Christopher Severs]

Hi,

I'm getting a class not found exception when running a fairly simple job with the Scalding typesafe API. I can't see how this is related to Scalding or the typesafe API but I thought I would include the info just in case. Stack trace is below. I've verified that the jar contains cascading-core and cascading-hadoop in the lib directory and I made sure that the unpacked jar in /tmp on the machine I run from has the correct jars in lib. This happened to me on two versions of Hadoop, one of which is essentially identical to 0.20.203 and the other is version 0.22. Neither of them are cloudera distros and I've never had trouble running Cascading or Scalding jobs on either of them.

The job submits fine but the mappers error out.

Has anyone run into an issue like this previously?

Thanks,
Chris

Stack trace:

java.io.IOException: Split class cascading.tap.hadoop.io.MultiInputSplit not found
	at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:352)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:377)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:337)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:242)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
	at org.apache.hadoop.mapred.Child.main(Child.java:236)
Caused by: java.lang.ClassNotFoundException: cascading.tap.hadoop.io.MultiInputSplit
	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:247)
	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:818)
	at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:350)
	... 7 more

[Ken Krugler]

Hi Chris,

On Oct 2, 2012, at 6:32pm, Christopher Severs wrote:

Hi,

I'm getting a class not found exception when running a fairly simple job with the Scalding typesafe API. I can't see how this is related to Scalding or the typesafe API but I thought I would include the info just in case. Stack trace is below. I've verified that the jar contains cascading-core and cascading-hadoop in the lib directory and I made sure that the unpacked jar in /tmp on the machine I run from has the correct jars in lib. This happened to me on two versions of Hadoop, one of which is essentially identical to 0.20.203 and the other is version 0.22. Neither of them are cloudera distros and I've never had trouble running Cascading or Scalding jobs on either of them.

The job submits fine but the mappers error out.

Has anyone run into an issue like this previously?

I haven't, sorry.

There was an issue reported against CDH4 + Cascading that involved AvroReflectSerialization, in case this came about during your work on cascading.avro :)

https://issues.cloudera.org/browse/DISTRO-401?focusedCommentId=14655&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14655

-- Ken

Stack trace:

java.io.IOException: Split class cascading.tap.hadoop.io.MultiInputSplit not found
	at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:352)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:377)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:337)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:242)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
	at org.apache.hadoop.mapred.Child.main(Child.java:236)
Caused by: java.lang.ClassNotFoundException: cascading.tap.hadoop.io.MultiInputSplit
	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:247)
	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:818)
	at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:350)
	... 7 more

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To view this discussion on the web visit https://groups.google.com/d/msg/cascading-user/-/dudz9HJ-tWUJ.
To post to this group, send email to cascadi...@googlegroups.com.
To unsubscribe from this group, send email to cascading-use...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/cascading-user?hl=en.

--------------------------------------------

http://about.me/kkrugler

+1 530-210-6378

[Philippe Laflamme]

Hi,

I just had the exact same stacktrace using Scalding and I had to set the "cascading.app.appjar.path" configuration property to fix it. I haven't had time to investigate this issue, so I don't know why it's not set correctly in the first place (it was pointing the the scalding jar inside my own jar).

I'm not using scalding's ruby script to launch my job, but I am using com.twitter.scalding.Tool, so maybe it's confused. Let me know if you find anything!

Hope that helps,

Philippe

[Christopher Severs]

Hi Philippe,

Is that something you set in the job itself or something passed on the command line?

Hi Ken,

Unrelated to Avro and CDH4 at the moment. That pain will likely come later!

Thanks to both of you for the help,

Chris

[Philippe Laflamme]

Hi Chris,

It's a property in the job's configuration. I overrode config() in my Scalding Job like so:

override def config(implicit mode: Mode): Map[AnyRef, AnyRef] = {

  super.config(mode) ++ Map("cascading.app.appjar.path" -> "/absolute/path/to/jar.jar")

}

Philippe

To view this discussion on the web visit https://groups.google.com/d/msg/cascading-user/-/Pl1RSQJHRiUJ.

[Christopher Severs]

Hi Philippe,

That worked great, thank you much!

New problem: kryo buffer overflow.

I'll start a new thread for that issue.

Thanks again,
Chris

[Oscar Boykin]

Thanks for debugging this one.

Does anyone have any ideas why the cascading.app.appjar.path needs to be set?

I wonder if it is how your are building scalding? We make deploy jars
that have all of cascading inside. But I guess your jars must have
that, as Chris mentioned he did.

On Wed, Oct 3, 2012 at 10:09 AM, Christopher Severs

> https://groups.google.com/d/msg/cascading-user/-/bxcPS6cZsmAJ.

>
> To post to this group, send email to cascadi...@googlegroups.com.
> To unsubscribe from this group, send email to
> cascading-use...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/cascading-user?hl=en.



--

Oscar Boykin :: @posco :: https://twitter.com/intent/user?screen_name=posco

[Christopher Severs]

I'm not sure why the info about the jar needs to be passed now but not previously. I did find a slightly nicer way to do this, rather than hardcoding the absolute path of the jar you can instead set cascading.app.appjar.class to be the classOf whatever your job class is. That way at least you can put the jar wherever you want.

I recall that in Cascading 1.2 there was (sometimes?) a need to use the setJarByClass but it looks like that has been deprecated in favor of the newer AppProps version.

My current jars are getting built using Maven's assembly plugin. All the cascading jars get put into a lib/ directory inside that jar and are definitely getting unpacked correctly.

----
Chris

[neelamaje...@gmail.com]

Hi i am running to same issue . This is my first cascading program trying to run example found online can someone help me?

I am not sure how to include the properties that helped you to resolve this issue in my program

override def config(implicit mode: Mode): Map[AnyRef, AnyRef] = {

  super.config(mode) ++ Map("cascading.app.appjar.path" -> "/absolute/path/to/jar.jar")

}

package impatient;

import java.util.Properties;

import cascading.flow.FlowDef;

import cascading.flow.hadoop.HadoopFlowConnector;

import cascading.pipe.Pipe;

import cascading.property.AppProps;

import cascading.scheme.Scheme;

import cascading.scheme.hadoop.TextDelimited;

import cascading.tap.Tap;

import cascading.tap.hadoop.Hfs;

public class Main

  {

  public static void  main( String[] args )

    {

    String inPath = args[ 0 ];

    String outPath = args[ 1 ];

    Properties properties = new Properties();

    AppProps.setApplicationJarClass( properties, Main.class );

    HadoopFlowConnector flowConnector = new HadoopFlowConnector( properties );

    // create the source tap

    Scheme sourceScheme = new TextDelimited( true, "\t" );

    Tap inTap = new Hfs(sourceScheme, inPath );

    // create the sink tap

    Scheme sinkScheme = new TextDelimited( true, "\t" );

    Tap outTap = new Hfs( sinkScheme, outPath );

    // specify a pipe to connect the taps

    Pipe copyPipe = new Pipe( "copy" );

    // connect the taps, pipes, etc., into a flow

    FlowDef flowDef = FlowDef.flowDef()

     .addSource( copyPipe, inTap )

     .addTailSink( copyPipe, outTap );

    // run the flow

    flowConnector.connect( flowDef ).complete();

    }

  }

[David Shimon]

Hi,

Do you think we should add 

"cascading.app.appjar.class" -> getClass.getName
to com.twitter.scalding.Job?

Is this something that may interfere?

Thanks

This message may contain confidential and/or privileged information. 

If you are not the addressee or authorized to receive this on behalf of the addressee you must not use, copy, disclose or take action based on this message or any information herein. 

If you have received this message in error, please advise the sender immediately by reply email and delete this message. Thank you.

[Oscar Boykin]

I really don't know anything about that key, and I don't know why we don't need it, but some people seem to.

Perhaps cwensel or someone can explain why it is needed in some cases and not others.

To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.

To post to this group, send email to cascadi...@googlegroups.com.

Visit this group at http://groups.google.com/group/cascading-user?hl=en.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Oscar Boykin :: @posco :: http://twitter.com/posco

[Chris K Wensel]

Its not required if you set the Hadoop analog. If you don't set the Hadoop analog, it is required. Its a bit of future proofing against platforms that have different key values.

--

Chris K Wensel

ch...@concurrentinc.com

http://concurrentinc.com

[Tomer Ben David]

I had the same issue and my solution (at least temporal) was that my jar is not proper so I moved from a jar with this structure

/lib

/META-INF

to an jar which contains already extracted all the internal content with maven-shade-plugin