API wars considered harmful to newcomers

[Jim White]

Howdy gang!

I've just begun trying Cascading in order to use Hadoop in my work.  Naturally the first thing I do is run the wordcount example using fresh Cascading 2.1.6 and Hadoop 1.0.4 installs.  All goes well testing on my local machine, then off to the cluster.  Things don't go so well.  The first sign of trouble (initially the only sign of trouble because apparently output laying about from running locally meant the whole job completed w/o error in spite of this "warning"):

13/05/08 17:39:02 INFO flow.FlowStep: [export word] submitted hadoop job: job_local_0003

13/05/08 17:39:02 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@687ec028

13/05/08 17:39:02 INFO io.MultiInputSplit: current split input path: hdfs://diana.local:9000/user/jimwhite/cout/words/part-00000

13/05/08 17:39:02 WARN mapred.LocalJobRunner: job_local_0003

java.lang.ClassCastException: cascading.tap.hadoop.io.MultiInputSplit cannot be cast to org.apache.hadoop.mapred.FileSplit

at org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:51)

at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:197)

at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)

at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)

13/05/08 17:39:02 INFO flow.FlowStep: [export url] submitted hadoop job: job_local_0002

13/05/08 17:39:02 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@42bd93cd

13/05/08 17:39:02 INFO io.MultiInputSplit: current split input path: hdfs://diana.local:9000/user/jimwhite/cout/urls/part-00000

13/05/08 17:39:02 INFO hadoop.TupleSerialization: using hadoop serializations from the job conf: cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization 

13/05/08 17:39:02 INFO hadoop.TupleSerialization: adding serialization token: 127, for classname: org.apache.hadoop.io.BytesWritable

13/05/08 17:39:02 INFO mapred.MapTask: numReduceTasks: 0

13/05/08 17:39:02 INFO hadoop.FlowMapper: cascading version: Concurrent, Inc - Cascading 2.1.6

13/05/08 17:39:02 INFO hadoop.FlowMapper: child jvm opts: -Xmx200m

Naturally there's no one else using Hadoop here and so I'm left wondering what is the cause and any number of classpath issues are plausible, especially since the cluster has both Hadoop 0.20.203.0 and 1.0.4 installed.  After verifying that all the build stuff looks right and unpacking various jar files and generally wasting a couple hours I decide that the couple messages spread over the last four years about the "new API" and "old API" are indeed likely to be related to my problem which seems strange since how could an API change that happened so many years ago be a problem for current released and stable code?

The fixes I saw talked about a jobconf xml file and I was sure I didn't want anything to do with that but the mapred.mapper.new-api and mapred.reducer.new-api properties were worth a shot.  Taking a stab in the dark I added a couple "set false" calls to the Property in the main of wordcount and gave it whirl.  Success.  Yay.  Grrrr.

diff -ur src-old/java/wordcount/Main.java src/java/wordcount/Main.java

--- src-old/java/wordcount/Main.java 2013-05-06 03:58:28.000000000 -0700

+++ src/java/wordcount/Main.java 2013-05-08 17:25:07.000000000 -0700

@@ -99,6 +99,8 @@

     {

     // set the current job jar

     Properties properties = new Properties();

+    properties.setProperty("mapred.mapper.new-api", "false");

+    properties.setProperty("mapred.reducer.new-api", "false");

     AppProps.setApplicationJarClass( properties, Main.class );

     FlowConnector flowConnector = new HadoopFlowConnector( properties );

 

I think this should be handled within Cascading and shouldn't be a user-visible "feature" at all.  In any case the example code should at least have these settings in there so that it runs out-of-the-box and the user can be warned about the dragons that be here.

Jim

[Chris K Wensel]

What distribution of Hadoop?

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.
To post to this group, send email to cascadi...@googlegroups.com.
Visit this group at http://groups.google.com/group/cascading-user?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Chris K Wensel

ch...@concurrentinc.com

http://concurrentinc.com

[Jim White]

Locally (Mountain Lion - JDK 1.6) I used Apache of course, but I don't admin the cluster (CentOS - Linux 2.6.18, JDK 1.6) so I can't be sure but I can't see why he would use anything else (and I know we picked the version by looking at the Apache downloads page).  It was installed as a symlinked /opt/hadoop -> /opt/hadoop -> hadoop-1.0.4 which I don't think a CentOS RPM would do, but that's still a guess.  I will find out (but that may take a day or so).

Having gotten that little screed out I realize this may still be an issue with our cluster configuration, so I do apologize in advance if this turns out to be something really weird about my situation.

Jim

[Chris K Wensel]

per the Hadoop code, those values when using the Hadoop stable api methods and JobConf, are by default 'false'.

for example, from 1.0.4 Apache Hadoop:

  public boolean getUseNewReducer() {

    return getBoolean("mapred.reducer.new-api", false);

  }

So there is no need for Cascading to be redundant, but it _might_ be wise to force these values back to false.

That said, attempting to use the "mapreduce" experimental (MR2) apis in tandem with the stable ones will lead to troubles. the very same you are encountering I suspect, so forcing them back to false will continue to lead to troubles.

Further we and friends of the community run Cascading against various versions of Hadoop with no issues.

http://www.cascading.org/support/compatibility/

ckw

[Jim White]

How do I find out what's going wrong?  

I can see in the stack trace that some part of the logic knows the old API is being used (runOldMapper @ MapTask:418) but somewhere else there is this conflict.   Why is the exception not fatal to the job?

If I run with properties.setProperty("mapred.job.tracker", "local") (and not setting anything to the new-api properties) then the job runs fine.  So the problem is something about running on a slave.

Could a node be running the old version of Hadoop somehow?  How would I find out?  

I've spent a whole day chasing this now and examined many web pages talking about many sorts of issues with Hadoop.   There has been very little helpful in the way of how to get useful debug information.  

Jim

jimwhite@dryas:~$ hadoop classpath | groovy -a : -pe "split.sort().join('\n')" -       

/opt/hadoop-1.0.4/libexec/..

/opt/hadoop-1.0.4/libexec/../conf

/opt/hadoop-1.0.4/libexec/../hadoop-core-1.0.4.jar

/opt/hadoop-1.0.4/libexec/../lib/asm-3.2.jar

/opt/hadoop-1.0.4/libexec/../lib/aspectjrt-1.6.5.jar

/opt/hadoop-1.0.4/libexec/../lib/aspectjtools-1.6.5.jar

/opt/hadoop-1.0.4/libexec/../lib/commons-beanutils-1.7.0.jar

/opt/hadoop-1.0.4/libexec/../lib/commons-beanutils-core-1.8.0.jar

/opt/hadoop-1.0.4/libexec/../lib/commons-cli-1.2.jar

/opt/hadoop-1.0.4/libexec/../lib/commons-codec-1.4.jar

/opt/hadoop-1.0.4/libexec/../lib/commons-collections-3.2.1.jar

/opt/hadoop-1.0.4/libexec/../lib/commons-configuration-1.6.jar

/opt/hadoop-1.0.4/libexec/../lib/commons-daemon-1.0.1.jar

/opt/hadoop-1.0.4/libexec/../lib/commons-digester-1.8.jar

/opt/hadoop-1.0.4/libexec/../lib/commons-el-1.0.jar

/opt/hadoop-1.0.4/libexec/../lib/commons-httpclient-3.0.1.jar

/opt/hadoop-1.0.4/libexec/../lib/commons-io-2.1.jar

/opt/hadoop-1.0.4/libexec/../lib/commons-lang-2.4.jar

/opt/hadoop-1.0.4/libexec/../lib/commons-logging-1.1.1.jar

/opt/hadoop-1.0.4/libexec/../lib/commons-logging-api-1.0.4.jar

/opt/hadoop-1.0.4/libexec/../lib/commons-math-2.1.jar

/opt/hadoop-1.0.4/libexec/../lib/commons-net-1.4.1.jar

/opt/hadoop-1.0.4/libexec/../lib/core-3.1.1.jar

/opt/hadoop-1.0.4/libexec/../lib/hadoop-capacity-scheduler-1.0.4.jar

/opt/hadoop-1.0.4/libexec/../lib/hadoop-fairscheduler-1.0.4.jar

/opt/hadoop-1.0.4/libexec/../lib/hadoop-thriftfs-1.0.4.jar

/opt/hadoop-1.0.4/libexec/../lib/hsqldb-1.8.0.10.jar

/opt/hadoop-1.0.4/libexec/../lib/jackson-core-asl-1.8.8.jar

/opt/hadoop-1.0.4/libexec/../lib/jackson-mapper-asl-1.8.8.jar

/opt/hadoop-1.0.4/libexec/../lib/jasper-compiler-5.5.12.jar

/opt/hadoop-1.0.4/libexec/../lib/jasper-runtime-5.5.12.jar

/opt/hadoop-1.0.4/libexec/../lib/jdeb-0.8.jar

/opt/hadoop-1.0.4/libexec/../lib/jersey-core-1.8.jar

/opt/hadoop-1.0.4/libexec/../lib/jersey-json-1.8.jar

/opt/hadoop-1.0.4/libexec/../lib/jersey-server-1.8.jar

/opt/hadoop-1.0.4/libexec/../lib/jets3t-0.6.1.jar

/opt/hadoop-1.0.4/libexec/../lib/jetty-6.1.26.jar

/opt/hadoop-1.0.4/libexec/../lib/jetty-util-6.1.26.jar

/opt/hadoop-1.0.4/libexec/../lib/jsch-0.1.42.jar

/opt/hadoop-1.0.4/libexec/../lib/jsp-2.1/jsp-2.1.jar

/opt/hadoop-1.0.4/libexec/../lib/jsp-2.1/jsp-api-2.1.jar

/opt/hadoop-1.0.4/libexec/../lib/junit-4.5.jar

/opt/hadoop-1.0.4/libexec/../lib/kfs-0.2.2.jar

/opt/hadoop-1.0.4/libexec/../lib/log4j-1.2.15.jar

/opt/hadoop-1.0.4/libexec/../lib/mockito-all-1.8.5.jar

/opt/hadoop-1.0.4/libexec/../lib/oro-2.0.8.jar

/opt/hadoop-1.0.4/libexec/../lib/servlet-api-2.5-20081211.jar

/opt/hadoop-1.0.4/libexec/../lib/slf4j-api-1.4.3.jar

/opt/hadoop-1.0.4/libexec/../lib/slf4j-log4j12-1.4.3.jar

/opt/hadoop-1.0.4/libexec/../lib/xmlenc-0.52.jar

/usr/java/latest/lib/tools.jar

[Chris K Wensel]

Are you positive the distribution of Hadoop on your cluster is Apache?

[Paco Nathan]

Hi Jim,

Have you tried working with the sample apps in https://github.com/Cascading/Impatient ?

Those have a bit more explanation included on requirements, troubleshooting, etc.

Paco

[David Brodbeck]

Hi, I'm the admin of the cluster, and I'm trying to help sort this out.

On Thursday, May 9, 2013 9:30:38 AM UTC-7, Chris K Wensel wrote:

Are you positive the distribution of Hadoop on your cluster is Apache?

Yup, I installed from a tarball downloaded from Apache's site.

One complicating factor may be that I upgraded from 0.20.203, and essentially copied the config over to the new installation. However, the new install runs hadoop's built-in wordcount example just fine.

[Chris K Wensel]

As Paco notes, working through the Impatient series may help illuminate things.

You can also try running Load on your cluster.

http://www.cascading.org/load/

ckw

[Mason]

Not sure this is helpful, but I just open-sourced our Vagrant config for running a Hadoop 1.0 cluster on CentOS locally (e.g. within an OSX host): https://github.com/Verba/wandering-elephant. Might help you get closer to parity with your cluster.

-Mason

[Chris K Wensel]

Nice. Now if there was one of these for standard Apache as well. 

[Mason]

I'm not doin' it, but maybe someone will make that fork :)

[Jim White]

The problem is that Cascading 2.1.6 is not compatible with Hadoop 1.0.4.  The problem only shows up though when an actual split is serialized and deserialized, which is why it works locally and not when running on slaves.  As the log snippet I posted in my original message showed, the error that is occurring is:

13/05/08 17:39:02 WARN mapred.LocalJobRunner: job_local_0003

java.lang.ClassCastException: cascading.tap.hadoop.io.MultiInputSplit cannot be cast to org.apache.hadoop.mapred.FileSplit

at org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:51)

at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:197)

at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)

at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)

Line 51 of o.a.h.mapred.TextInputFormat is:

      return new LineRecordReader(job, (FileSplit) genericSplit);

But cascading.tap.hadoop.io.MultiInputSplit is defined thusly:

public class MultiInputSplit implements InputSplit, JobConfigurable

But o.a.h.mapred.FileSplit is a class that implements the InputSplit interface:

public class FileSplit extends org.apache.hadoop.mapreduce.InputSplit 

                       implements InputSplit {

I suspect that is a problem with Hadoop 1.0.4 (surely the interface is the thing to depend on, not some particular implementation), but after too many hours of yak shaving with these complex systems in trying to run a HelloWorld example I conclude that the problem is not in our cluster setup (or its distribution).

Jim

[David Brodbeck]

On Sat, May 11, 2013 at 4:13 PM, Jim White <james.pa...@gmail.com> wrote:

I suspect that is a problem with Hadoop 1.0.4 (surely the interface is the thing to depend on, not some particular implementation), but after too many hours of yak shaving with these complex systems in trying to run a HelloWorld example I conclude that the problem is not in our cluster setup (or its distribution).

Good sleuthing, Jim.  Since you're currently the only one using Hadoop, if there's a preferred version I'd be willing to switch to it.  We had a similar situation with Mahout, where I had to downgrade to an earlier release to get back some functionality that was broken in later versions.

[Chris K Wensel]

Cascading works fine with hadoop 1.0.4.

What it doesn't work with is the experimental API's introduced under the "mapreduce" package.

For whatever reason, your config is being told to use them instead of the stable apis under the "mapred" package.

My suggestion is to figure out why your cluster is defaulting to use the experimental apis. This usually happens when you try to use the stable and experimental apis simultaneously. Hadoop fails if you try to do so. 

Or force them back to stable per early discussion in this thread. If Cascading attempted that, it would cause different failures. 

ckw

[Jim White]

As I said, this has nothing to do with our cluster.  This analysis is based entirely on code from Apache.  To ensure absolutely no chance of error on my part I've repeated my analysis by examining the Apache source in their SVN.

org.apache.hadoop.mapred.FileSplit extends org.apache.hadoop.mapreduce.InputSplit and implements org.apache.hadoop.mapred.InputSplit.

http://svn.apache.org/viewvc/hadoop/common/tags/release-1.0.4/src/mapred/org/apache/hadoop/mapred/FileSplit.java?revision=1397701&view=markup

org.apache.hadoop.mapred.TextInputFormat tries to cast the split file serialized as cascading.tap.hadoop.io.MultiInputSplit (which implements org.apache.hadoop.mapred.InputSplit but does not extend org.apache.hadoop.mapred.FileSplit) to org.apache.hadoop.mapred.FileSplit.

http://svn.apache.org/viewvc/hadoop/common/tags/release-1.0.4/src/mapred/org/apache/hadoop/mapred/TextInputFormat.java?revision=1397701&view=markup

Jim

--
You received this message because you are subscribed to a topic in the Google Groups "cascading-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cascading-user/nDAI6pcIFqE/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to cascading-use...@googlegroups.com.

[Chris K Wensel]

I think we are crossing wires.

There are two parallel APIs in Hadoop in differing packages. The "stable" ones in "mapred", and the "experimental" ones in "mapreduce".

If you create a JobConf ( via new JobConf() ), it will use the stable ones. Until the point where you, the developer, touches an experimental API by adding an Input/OutputFormat from the "mapreduce" package to the JobConf instance, for example.

This tells the JobCont to use the "new" apis internally when submitting/executing the actual job. That is, you cannot mix stable and experimental in a single MR job via a JobConf.

Cascading only uses the "stable" apis.

If the defaults have changed (because a vendor changed them, or you are creating a custom Tap/Scheme using the "mapreduce" apis) you will get the exceptions you are seeing.

Many of our regression tests boot up a cluster using the default JobConf properties. So we should be able to detect changes in the defaults in the Apache distribution.

Cascading is never published without passing its battery of tests (which takes about 6-7 compute hours), and is currently being published against Apache Hadoop 1.0.4

http://conjars.org/repo/cascading/cascading-hadoop/2.1.6/cascading-hadoop-2.1.6.pom

ckw

[Sangjin Lee]

Hi Jim,

Somethings seems off to me. If my understanding is correct, MultiInputSplit is used only by MultiInputFormat. In other words, MultiInputSplits are generated only when the input format class is MultiInputFormat. The fact that your input format class is the plain TextInputFormat but the splits came from MultiInputFormat sounds strange.

Could you check if the input format is somehow set to TextInputFormat after your splits are generated?

Sangjin

[Jim White]

Hi Sangjin!

Yes, something is going wrong, but how can I find out what?  I'm expert in Java and have spent a lot of time digging around on the assumption something is wrong in our setup but I haven't been able to find out how to do basic things like show how the particular Java slave that is throwing the exception is configured when it launches.  Surely there are some debugging flags that will dump that kind of information to some logs but I've not found and information on how to actually do that for Hadoop (I find plenty of code and Javadocs for Hadoop and Cascading and a little on how to write apps but basically nothing on debugging the cluster configuration or faults).

Jim

[Ken Krugler]

Hi Jim,

Using the Hadoop GUI (browser-based), you should be able to get the job.xml file (serialized version of the JobConf) for the failing job.

Inside of this you'd find all of the properties for the job, including things like what input and output formats are being specified.

-- Ken

--------------------------

Ken Krugler

+1 530-210-6378

http://www.scaleunlimited.com

custom big data solutions & training

Hadoop, Cascading, Cassandra & Solr