Pangool and oozie in CDH3

[Alexei Perelighin]

Hi,

Ran into a problem with running Pangool jobs from oozie. 

When I generate a jar file that does not contain the dependent jars and run the job from the command line (hadoop jar ./pangool-plc.jar -Dmapred.reduce.tasks=3 -libjars ${LIBJARS}) where LIBJAR includes all of the dependent jars, it works fine. 

The problem begins when I try to use OOZIE.

When I generate a big fat jar that includes all of the dependent classes, put it into the HDFS, into folder specified by the oozie.libpath and invoke it from the java action, it works fine.

But when I generate a jar without dependencies, put it with all of it dependent jars into the HDFS folder to which oozie.libpath points and run the java action, it fails and the file in the OUTPUT FOLDER/_logs/history/ contains following errors:

ReduceAttempt TASK_TYPE="SETUP" TASKID="task_201303061629_60273_r_000031" TASK_ATTEMPT_ID="attempt_201303061629_60273_r_000031_0" TASK_STATUS="FAILED" FINISH_TIME="1367250329138"

 HOSTNAME="XXXX" ERROR="java\.lang\.RuntimeException: java\.lang\.ClassNotFoundException: com\.datasalt\.pangool\.tuplemr\.mapred\.lib\.output\.ProxyOutputFormat

at org\.apache\.hadoop\.conf\.Configuration\.getClass(Configuration\.java:1004)

at org\.apache\.hadoop\.mapreduce\.JobContext\.getOutputFormatClass(JobContext\.java:253)

at org\.apache\.hadoop\.mapred\.Task\.initialize(Task\.java:509)

at org\.apache\.hadoop\.mapred\.ReduceTask\.run(ReduceTask\.java:360)

at org\.apache\.hadoop\.mapred\.Child$4\.run(Child\.java:266)

at java\.security\.AccessController\.doPrivileged(Native Method)

at javax\.security\.auth\.Subject\.doAs(Subject\.java:396)

at org\.apache\.hadoop\.security\.UserGroupInformation\.doAs(UserGroupInformation\.java:1278)

at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:260)

Caused by: java\.lang\.ClassNotFoundException: com\.datasalt\.pangool\.tuplemr\.mapred\.lib\.output\.ProxyOutputFormat

at java\.net\.URLClassLoader$1\.run(URLClassLoader\.java:202)

at java\.security\.AccessController\.doPrivileged(Native Method)

at java\.net\.URLClassLoader\.findClass(URLClassLoader\.java:19" .

Has anybody run Pangool jobs using oozie when dependent jars are not packaged with the jar that has the class with main method?

Thanks,

Alexei

[Iván de Prado]

Hi Alexei, 

I imagine that the problem is related with that:

http://stackoverflow.com/questions/11042495/how-oozie-handle-dependencies

Libraries defined by the property oozie.libpath will be present on job execution, but not on main execution. And the pangool.jar library is also needed in the main, as the Job is built using the TupleMRBuilder. 

I think the solution is just adding the pangool.jar to the Oozie lib folder. Try it and tell us if this works. 

Thanks!

Iván

2013/4/29 Alexei Perelighin <alex...@googlemail.com>

--
Has recibido este mensaje porque estás suscrito al grupo "pangool-user" de Grupos de Google.
Para anular la suscripción a este grupo y dejar de recibir sus correos electrónicos, envía un correo electrónico a pangool-user...@googlegroups.com.
Para obtener más opciones, visita https://groups.google.com/groups/opt_out.
 
 

--
Iván de Prado

CEO & Co-founder

www.datasalt.com

[Alexei Perelighin]

Thanks, I can not try it today, but I tried to remove the pangool.jar from the libpath and from the shared path mentioned on the stackoverflow.

When the pangool.jar is not available to the main method, than I get:

Failing Oozie Launcher, Main class [MY CLASS], exception invoking main(), com/datasalt/pangool/tuplemr/TupleMapper
java.lang.NoClassDefFoundError: com/datasalt/pangool/tuplemr/TupleMapper
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:247)
	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:951)
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1002)
	at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:388)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
	at org.apache.hadoop.mapred.Child.main(Child.java:260)
Caused by: java.lang.ClassNotFoundException: com.datasalt.pangool.tuplemr.TupleMapper
	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
	... 13 more

But when pangool.jar is present in those directories, I get:

java.lang.RuntimeException: java.lang.ClassNotFoundException: com.datasalt.pangool.tuplemr.mapred.lib.output.ProxyOutputFormat

Also in this case there are not Exceptions thrown and Oozie java actions finish with SUCCESS status. But when I check for output, the directory is there but no _SUCCESS file.

[Iván de Prado]

Hi Alexei, 

I was able to execute Pangool jobs with Oozie. It is enough with adding the dependencies to the lib folder inside your workflow directory. 

In the same folder where you have your workflow.xml file, you have to create a lib folder. This folder should include all the libraries needed for the execution of your job, like the pangool.jar and all the other indirect dependencies. hadoop.jar is not needed there. For example, this is the tree of the workflow I executed:

oozie-app/

├── lib

│   ├── antlr-2.7.7.jar

│   ├── antlr-3.0.1.jar

│   ├── antlr-runtime-3.0.1.jar

[many more]

│   ├── pangool-core-0.60.3.jar

│   ├── pangool-examples-0.60.3-hadoop.jar

[many more]

│   ├── xml-apis-1.3.04.jar

│   ├── xz-1.0.jar

│   └── zookeeper-3.4.3.jar

└── workflow.xml

Then I just uploaded the oozie-app folder to the HDFS and run the job with the following command:

oozie job -oozie http://localhost:11000/oozie -config job.properties -run

The content of job.properties is:

nameNode=hdfs://localhost:9000

jobTracker=localhost:9001

queueName=default

oozie.wf.application.path=${nameNode}/user/${user.name}/oozie-app

And the content of worflow.xml is:

<workflow-app xmlns='uri:oozie:workflow:0.1' name='java-main-wf'>

    <start to='java1' />

    <action name='java1'>

        <java>

            <job-tracker>${jobTracker}</job-tracker>

            <name-node>${nameNode}</name-node>

            <configuration>

                <property>

                    <name>mapred.job.queue.name</name>

                    <value>default</value>

                </property>

            </configuration>

            <main-class>com.datasalt.pangool.examples.Driver</main-class>

            <arg>moving_average</arg>

            <arg>/user/ivan/movavg.txt</arg>

            <arg>/user/ivan/mov-avg-out</arg>

            <arg>2</arg>

        </java>

        <ok to="end" />

        <error to="fail" />

    </action>

    <kill name="fail">

        <message>Java failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>

    </kill>

    <end name='end' />

</workflow-app> 

Try using the lib folder and tell us if that worked. 

Regards!

Iván

[Alexei Perelighin]

Hi Ivan,

Did exactly that, still get the same exception.

I am getting the same exception when executing code from the command line, without OOZIE, by setting up the HADOOP_CLASSPATH but not adding -libjars to the command line.

Could you copy/paste your java.class.path from the oozie's task log?

Also could you post the result of  ls /usr/lib/hadoop-0.20/lib ?

Thanks,

Alexei

[Iván de Prado]

Hi Alexei, 

Finally I was able to reproduce the problem. You were right. The problem is the following: when running java actions, Oozie includes jars on lib folder in the classpath, but Oozie won't include them in the classpath of you job. In other words, jar on lib folders are not automatically included into the distributed cache to be available in Jobs when using the java action. 

I found a workaround that works under the following requirements:

1) Your main class must implement Tool

2) Your Job must be executed via the ToolRunner class or the Driver class. 

In this case, we can use the following trick for making jars available for a job run with the java action:

1) Include another folder inside the lib folder (for example joblibs) with all the dependendencies for your job. Tipically the folder lib will have the same jar files than the lib/joblibs folder. The reason for that is because all folders inside the lib folder will be available at the working directory when running any application with Oozie. But that it is important: the folder must be inside the lib folder. Folders inside lib folder are available at working directory: that means, that if you have a folder lib/joblibs you will have access to files just by accessing the path joblibs from your application. 

2) Invoke your program using the -libjars property. (http://grepalex.com/2013/02/25/hadoop-libjars/) 

Let's see that with an example that runs the Game of Life Pangool example (http://www.datasalt.com/2012/05/pangools-game-of-life/). You have to create a oozie-app with the following contents. The jars inside the lib and lib/joblibs folder are the dependencies of pangool-examples plus the pangool-examples.jar file. That is how the folder should looks like:

oozie-app/

├── lib

│   ├── antlr-2.7.7.jar

│   ├── antlr-3.0.1.jar

[many more]

│   ├── jetty-util-6.1.14.jar

│   ├── jline-0.9.94.jar

│   ├── joblibs

│       ├── antlr-2.7.7.jar

│       ├── antlr-3.0.1.jar

[many more]

│       ├── jetty-6.1.14.jar

│       ├── jetty-util-6.1.14.jar

│       ├── xml-apis-1.3.04.jar

│       ├── xz-1.0.jar

│       └── zookeeper-3.4.3.jar

└── workflow.xml

The file workflow.xml should looks like the following:

--------

<workflow-app xmlns='uri:oozie:workflow:0.1' name='java-main-wf'>

    <start to='java1' />

    <action name='java1'>

        <java>

            <job-tracker>${jobTracker}</job-tracker>

            <name-node>${nameNode}</name-node>

            <configuration>

                <property>

                    <name>mapred.job.queue.name</name>

                    <value>default</value>

                </property>

            </configuration>

            <main-class>com.datasalt.pangool.examples.Driver</main-class>

            <arg>game_of_life</arg>

            <arg>-libjars</arg>

            <arg>

joblibs/json-20090211.jar,joblibs/protostuff-api-1.0.1.jar,joblibs/avro-mapred-1.6.3.jar,joblibs/slf4j-api-1.6.4.jar,joblibs/jackson-mapper-lgpl-1.7.9.jar,joblibs/hive-service-0.10.0.jar,joblibs/lucene-highlighter-4.0.0-BETA.jar,joblibs/commons-digester-1.8.jar,joblibs/mockito-all-1.8.2.jar,joblibs/lucene-analyzers-common-4.0.0-BETA.jar,joblibs/velocity-1.7.jar,joblibs/commons-codec-1.6.jar,joblibs/servlet-api-2.5-20081211.jar,joblibs/lucene-queryparser-4.0.0-BETA.jar,joblibs/datanucleus-rdbms-2.0.3.jar,joblibs/stop.jar,joblibs/lucene-memory-4.0.0-BETA.jar,joblibs/datanucleus-core-2.0.3.jar,joblibs/wstx-asl-3.2.7.jar,joblibs/httpclient-4.1.3.jar,joblibs/commons-io-2.1.jar,joblibs/lucene-analyzers-morfologik-4.0.0-BETA.jar,joblibs/libfb303-0.9.0.jar,joblibs/lucene-misc-4.0.0-BETA.jar,joblibs/JavaEWAH-0.3.2.jar,joblibs/paranamer-2.3.jar,joblibs/protobuf-java-2.3.0.jar,joblibs/jetty-util-6.1.14.jar,joblibs/commons-compress-1.4.1.jar,joblibs/xercesImpl-2.9.1.jar,joblibs/stringtemplate-3.2.1.jar,joblibs/httpcore-4.1.4.jar,joblibs/lucene-analyzers-kuromoji-4.0.0-BETA.jar,joblibs/avro-ipc-1.6.3.jar,joblibs/commons-logging-api-1.0.4.jar,joblibs/xz-1.0.jar,joblibs/solr-core-4.0.0-BETA.jar,joblibs/libthrift-0.6.1.jar,joblibs/httpmime-4.1.3.jar,joblibs/commons-beanutils-1.7.0.jar,joblibs/hive-common-0.10.0.jar,joblibs/hcatalog-core-0.5.0-incubating.jar,joblibs/hive-cli-0.10.0.jar,joblibs/commons-beanutils-core-1.8.0.jar,joblibs/hive-builtins-0.10.0.jar,joblibs/pangool-examples-0.60.3.jar,joblibs/morfologik-stemming-1.5.3.jar,joblibs/commons-cli-1.2.jar,joblibs/jackson-mapper-asl-1.8.8.jar,joblibs/hive-metastore-0.10.0.jar,joblibs/commons-fileupload-1.2.1.jar,joblibs/protostuff-model-1.0.1.jar,joblibs/hive-shims-0.10.0.jar,joblibs/commons-logging-1.0.4.jar,joblibs/jline-0.9.94.jar,joblibs/jdo2-api-2.3-ec.jar,joblibs/protostuff-parser-1.0.1.jar,joblibs/jsr305-1.3.9.jar,joblibs/avro-1.6.3.jar,joblibs/lucene-queries-4.0.0-BETA.jar,joblibs/spatial4j-0.2.jar,joblibs/hive-pdk-0.10.0.jar,joblibs/lucene-spatial-4.0.0-BETA.jar,joblibs/commons-pool-1.5.4.jar,joblibs/morfologik-polish-1.5.3.jar,joblibs/hive-exec-0.10.0.jar,joblibs/javolution-5.5.1.jar,joblibs/hive-serde-0.10.0.jar,joblibs/jetty-6.1.14.jar,joblibs/slf4j-log4j12-1.5.8.jar,joblibs/antlr-runtime-3.0.1.jar,joblibs/lucene-analyzers-phonetic-4.0.0-BETA.jar,joblibs/jcsv-1.4.0.jar,joblibs/zookeeper-3.4.3.jar,joblibs/xml-apis-1.3.04.jar,joblibs/guava-11.0.2.jar,joblibs/asm-3.1.jar,joblibs/datanucleus-connectionpool-2.0.3.jar,joblibs/snappy-java-1.0.4.1.jar,joblibs/log4j-1.2.16.jar,joblibs/lucene-core-4.0.0-BETA.jar,joblibs/protostuff-core-1.0.1.jar,joblibs/jackson-core-lgpl-1.7.9.jar,joblibs/lucene-suggest-4.0.0-BETA.jar,joblibs/commons-io-1.3.2.jar,joblibs/commons-lang-2.5.jar,joblibs/antlr-2.7.7.jar,joblibs/opencsv-2.3.jar,joblibs/solr-solrj-4.0.0-BETA.jar,joblibs/commons-collections-3.2.jar,joblibs/jackson-core-asl-1.8.8.jar,joblibs/pangool-core-0.60.3.jar,joblibs/jackson-jaxrs-1.7.9.jar,joblibs/commons-configuration-1.6.jar,joblibs/derby-10.4.2.0.jar,joblibs/antlr-3.0.1.jar,joblibs/netty-3.2.7.Final.jar,joblibs/joda-time-2.0.jar,joblibs/morfologik-fsa-1.5.3.jar,joblibs/datanucleus-enhancer-2.0.3.jar,joblibs/servlet-api-2.5.jar,joblibs/commons-dbcp-1.4.jar,joblibs/lucene-grouping-4.0.0-BETA.jar,joblibs/protostuff-compiler-1.0.1.jar

            </arg>

            <arg>gameoflife-out</arg>

            <arg>2</arg>

            <arg>4</arg>

        </java>

        <ok to="end" />

        <error to="fail" />

    </action>

    <kill name="fail">

        <message>Java failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>

    </kill>

    <end name='end' />

</workflow-app>

-----------

As you can see, we have to include the list of jars your Job depends on. You can obtain this list with the following command: find joblibs|grep jar|sed -r 's/\.\///'|xargs|tr ' ' ','

Finally, you need a file job.properties with the following content:

----------

nameNode=hdfs://localhost:9000

jobTracker=localhost:9001

queueName=default

oozie.wf.application.path=${nameNode}/user/${user.name}/oozie-app

----------

Now you are ready to upload the oozie-app to the hdfs and run the Oozie workflow:

hadoop fs -put oozie-app oozie-app

oozie job -oozie http://localhost:11000/oozie -config oozie-app/job.properties -run

Tell me if that worked for you. 

Regards, 

Iván

2013/4/30 Alexei Perelighin <alex...@googlemail.com>

[Alexei Perelighin]

Hi Iavan,

Thanks, that has worked. But as my libraries must be in the HDFS I had to use the full hdfs URI in the list of libs. 

Example:

            <arg>-libjars</arg>

            <arg>hdfs://localhost:54310/tmp/alexei/workflow/lib/pangool-plc.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/ant-1.6.5.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/avro-1.7.3.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/commons-cli-1.2.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/commons-codec-1.4.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/commons-collections-3.2.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/commons-el-1.0.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/commons-httpclient-3.1.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/commons-io-1.3.2.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/commons-lang3-3.1.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/commons-logging-1.0.4.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/commons-net-1.4.1.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/core-3.1.1.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/guava-10.0.1.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/guava-r09-jarjar.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/hamcrest-core-1.3.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/hsqldb-1.8.0.7.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/jackson-core-asl-1.7.9.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/jackson-core-lgpl-1.7.9.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/jackson-jaxrs-1.7.9.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/jackson-mapper-asl-1.7.9.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/jackson-mapper-lgpl-1.7.9.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/jasper-compiler-5.5.23.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/jasper-runtime-5.5.23.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/jcsv-1.4.0.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/jets3t-0.6.1.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/jetty-6.1.26.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/jetty-util-6.1.26.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/joda-time-1.6.2.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/jsp-api-2.0.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/jsp-api-2.1.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/jsr305-1.3.9.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/junit-4.11.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/log4j-1.2.16.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/opencsv-2.3.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/oro-2.0.8.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/pangool-core-0.60.3.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/paranamer-2.3.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/servlet-api-2.5-20081211.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/servlet-api-2.5.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/slf4j-api-1.6.4.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/slf4j-log4j12-1.6.4.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/snappy-java-1.0.4.1.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/xercesImpl-2.9.1.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/xml-apis-1.3.04.jar,hdfs://localhost:54310/tmp/alexei/workflow/lib/xmlenc-0.52.jar</arg>

I think it will be nice to update Pangool documentation with how to run the code without maven and with oozie. As in some production and test environments maven might not be available and the installation directories (libs) might be write protected.

Thanks a lot,

Alexei

It could save a lot of time 

[Iván de Prado]

Done!

http://pangool.net/executing_examples.html

Thanks Alexei, 

Iván

2013/5/1 Alexei Perelighin <alex...@googlemail.com>

[Alexei Perelighin]

Hi Ivan,

To avoid long list of libs in the oozie workflow.xml I have created an utility method to do it for me. 

The idea is quite simple:

1. As oozie will create the "String[] args" and pass it to the main method, I will put just the path to the directory which contains the lib jars;

            <arg>-libjars</arg>

            <arg>hdfs://localhost:54310/tmp/alexei/workflow/lib</arg>

2. "GenericOptionsParser parser = new GenericOptionsParser(conf, args);" will actually process the -libjars. So before executing it, my code will find the -lbjars and replace the single directory with the list of full paths to the all of the jars in the directory, rest of the arguments stay the same. So "args" becomes "amendedArgs";

                        FileSystem fs = FileSystem.get(conf);

        if (dirPath.startsWith("hdfs") || dirPath.startsWith("HDFS")) {

        libDir = new Path(dirPath);

        } else {

        libDir = new Path(fs.getUri() + dirPath);        

        }

                        FileStatus[] inTheDir = fs.listStatus(libDir);

        for (FileStatus inDirFile : inTheDir) {

        paths.add(inDirFile.getPath().toString());

        }

                        finalArgs.add(StringUtils.join(paths, ','));

3. execute the "GenericOptionsParser parser = new GenericOptionsParser(conf, amendedArgs);"

Thanks,

Alexei

[Iván de Prado]

Thanks for sharing!

2013/5/20 Alexei Perelighin <alex...@googlemail.com>