Lipstick fails with Hadoop 2 and pig version 0.12.1.2.1.1.0-385
149 views
Skip to first unread message
Abhishek Samsung
Jul 11, 2014, 3:12:44 PM7/11/14
to lipsti...@googlegroups.com
when I try the following command:-
hadoop jar /usr/local/Lipstick/build/lipstick-console-0.6-SNAPSHOT-withPig.jar -Dlipstick.server.url=http://172.28.58.54:8080/lipstick-1.0/
normal commands work
grunt> ls
hdfs://gnosis-01-01-01.crl.samsung.com:8020/user/root/NOTICE.txt<r 3> 1962
hdfs://gnosis-01-01-01.crl.samsung.com:8020/user/root/student.txt<r 3> 28
grunt> data = LOAD 'student.txt' AS (user:CHARARRAY, score1:INT, score2:INT);
grunt> dump data;
2014-07-11 12:08:38,479 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: UNKNOWN
2014-07-11 12:08:38,512 [main] INFO org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, GroupByConstParallelSetter, LimitOptimizer, LoadTypeCastInserter, MergeFilter, MergeForEach, NewPartitionFilterOptimizer, PartitionFilterOptimizer, PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter], RULES_DISABLED=[FilterLogicExpressionSimplifier]}
2014-07-11 12:08:38,755 [main] INFO com.netflix.lipstick.pigstatus.RestfulPigStatusClient - Initializing class com.netflix.lipstick.pigstatus.RestfulPigStatusClient with serviceUrls: http://172.28.58.54:8080/lipstick-1.0
2014-07-11 12:08:38,756 [main] INFO com.netflix.lipstick.listeners.LipstickPPNL - UUID: 9ba32ebd-236a-44d4-a4e0-283f86801b1f
2014-07-11 12:08:38,756 [main] INFO com.netflix.lipstick.listeners.LipstickPPNL - [com.netflix.lipstick.pigtolipstick.BasicP2LClient@56f3b98c]
2014-07-11 12:08:38,773 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2014-07-11 12:08:38,799 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2014-07-11 12:08:38,799 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2014-07-11 12:08:38,864 [main] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at gnosis-01-01-06.crl.samsung.com/172.28.48.26:8050
2014-07-11 12:08:38,996 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2014-07-11 12:08:39,021 [main] WARN org.apache.pig.tools.pigstats.ScriptState - unable to read pigs manifest file
2014-07-11 12:08:39,434 [main] INFO com.netflix.lipstick.pigstatus.RestfulPigStatusClient - Trying Lipstick server http://172.28.58.54:8080/lipstick-1.0
2014-07-11 12:08:39,604 [main] ERROR com.netflix.lipstick.pigstatus.RestfulPigStatusClient - Error contacting Lipstick server. Received status code 500 (wget and curl to this url works fine)
2014-07-11 12:08:39,605 [main] ERROR com.netflix.lipstick.pigstatus.RestfulPigStatusClient - Error contacting Lipstick server.
2014-07-11 12:08:39,605 [main] ERROR com.netflix.lipstick.pigtolipstick.BasicP2LClient - Failed to properly create lipstick client and save plan. Lipstick will be disabled.
2014-07-11 12:08:39,609 [main] WARN org.apache.pig.backend.hadoop20.PigJobControl - falling back to default JobControl (not using hadoop 0.20 ?)
java.lang.NoSuchFieldException: runnerState
at java.lang.Class.getDeclaredField(Class.java:1948)
at org.apache.pig.backend.hadoop20.PigJobControl.<clinit>(PigJobControl.java:51)
at org.apache.pig.backend.hadoop.executionengine.shims.HadoopShims.newJobControl(HadoopShims.java:98)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:289)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:191)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1324)
at org.apache.pig.LipstickPigServer.launchPlan(LipstickPigServer.java:142)
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1309)
at org.apache.pig.PigServer.storeEx(PigServer.java:980)
at org.apache.pig.PigServer.store(PigServer.java:944)
at org.apache.pig.PigServer.openIterator(PigServer.java:857)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:774)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
at com.netflix.lipstick.Main.run(Main.java:524)
at com.netflix.lipstick.Main.main(Main.java:130)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
2014-07-11 12:08:39,612 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2014-07-11 12:08:39,613 [main] WARN org.apache.pig.tools.pigstats.ScriptState - unable to read pigs manifest file
2014-07-11 12:08:39,613 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.reduce.markreset.buffer.percent is deprecated. Instead, use mapreduce.reduce.markreset.buffer.percent
2014-07-11 12:08:39,613 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2014-07-11 12:08:39,614 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2014-07-11 12:08:39,616 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - creating jar file Job1716081551276715358.jar
2014-07-11 12:08:45,013 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - jar file Job1716081551276715358.jar created
2014-07-11 12:08:45,013 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.jar is deprecated. Instead, use mapreduce.job.jar
2014-07-11 12:08:45,039 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2014-07-11 12:08:45,045 [main] INFO org.apache.pig.data.SchemaTupleFrontend - Key [pig.schematuple] is false, will not generate code.
2014-07-11 12:08:45,045 [main] INFO org.apache.pig.data.SchemaTupleFrontend - Starting process to move generated code to distributed cache
2014-07-11 12:08:45,047 [main] INFO org.apache.pig.data.SchemaTupleFrontend - Setting key [pig.schematuple.classes] with classes to deserialize []
2014-07-11 12:08:45,086 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
2014-07-11 12:08:45,086 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker.http.address is deprecated. Instead, use mapreduce.jobtracker.http.address
2014-07-11 12:08:45,090 [JobControl] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at gnosis-01-01-06.crl.samsung.com/172.28.48.26:8050
2014-07-11 12:08:45,098 [JobControl] ERROR org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl - Error while trying to run jobs.
java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setupUdfEnvAndStores(PigOutputFormat.java:225)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.checkOutputSpecs(PigOutputFormat.java:186)
at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:458)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:343)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
at org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
at org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:240)
at org.apache.pig.backend.hadoop20.PigJobControl.run(PigJobControl.java:121)
at java.lang.Thread.run(Thread.java:744)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:271)
2014-07-11 12:08:45,098 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
2014-07-11 12:08:45,104 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to stop immediately on failure.
2014-07-11 12:08:45,105 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - job null has failed! Stop running all dependent jobs
2014-07-11 12:08:45,105 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2014-07-11 12:08:45,111 [main] ERROR org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to recreate exception from backend error: Unexpected System Error Occured: java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setupUdfEnvAndStores(PigOutputFormat.java:225)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.checkOutputSpecs(PigOutputFormat.java:186)
at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:458)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:343)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
at org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
at org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:240)
at org.apache.pig.backend.hadoop20.PigJobControl.run(PigJobControl.java:121)
at java.lang.Thread.run(Thread.java:744)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:271)
2014-07-11 12:08:45,111 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
2014-07-11 12:08:45,112 [main] WARN org.apache.pig.tools.pigstats.ScriptState - unable to read pigs manifest file
2014-07-11 12:08:45,113 [main] INFO org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:
HadoopVersion PigVersion UserId StartedAt FinishedAt Features
2.4.0.2.1.1.0-385 root 2014-07-11 12:08:38 2014-07-11 12:08:45 UNKNOWN
Failed!
Failed Jobs:
JobId Alias Feature Message Outputs
N/A data MAP_ONLY Message: Unexpected System Error Occured: java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setupUdfEnvAndStores(PigOutputFormat.java:225)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.checkOutputSpecs(PigOutputFormat.java:186)
at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:458)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:343)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1557)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
at org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
at org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:240)
at org.apache.pig.backend.hadoop20.PigJobControl.run(PigJobControl.java:121)
at java.lang.Thread.run(Thread.java:744)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:271)
Input(s):
Failed to read data from "hdfs://gnosis-01-01-01.crl.samsung.com:8020/user/root/student.txt"
Output(s):
Failed to produce result in "hdfs://gnosis-01-01-01.crl.samsung.com:8020/tmp/temp-2019597738/tmp-94243094"
Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0
Job DAG:
null
2014-07-11 12:08:45,113 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
2014-07-11 12:08:45,117 [main] ERROR org.apache.pig.tools.grunt.LipstickGrunt - ERROR 1066: Unable to open iterator for alias data
Details at logfile: /usr/local/Lipstick/pig_1405105705724.log
===========================================================================================================================================================================================================================
the following commands work fine. I can also see the page in the browser.
[root@gnosis-01-02-11 Lipstick]# curl http://172.28.58.54:8080/lipstick-1.0/
[root@gnosis-01-02-11 Lipstick]# wget http://172.28.58.54:8080/lipstick-1.0/
===========================================================================================================================================================================================================================
[root@gnosis-01-02-11 Lipstick]# hadoop jar /usr/local/Lipstick/build/lipstick-console-0.6-SNAPSHOT-full.jar -Dlipstick.server.url=http://172.28.58.54:8080/lipstick-1.0/
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/pig/tools/parameters/ParseException
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at org.apache.hadoop.util.RunJar.main(RunJar.java:205)
Caused by: java.lang.ClassNotFoundException: org.apache.pig.tools.parameters.ParseException
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 3 more
===========================================================================================================================================================================================================================
Jacob Perkins
Jul 11, 2014, 3:59:43 PM7/11/14
to lipsti...@googlegroups.com
Abhishek,
The master branch build of lipstick doesn't run, as is, against hadoop 2. You'll have to remove the pigInJar and hadoopInJar entries in your build.gradle and replace them with the following:
pigInJar('org.apache.pig:pig:0.12.1:h2') {
exclude module: 'hsqld'
exclude module: 'jetty'
exclude module: 'jsp-api-2.1'
exclude module: 'servlet-api-2.5'
}
hadoopInJar('org.apache.hadoop:hadoop-common:2.4.0') {
exclude module: 'jersey-api'
exclude module: 'jersey-json'
exclude module: 'jersey-guice'
exclude module: 'jersey-server'
exclude module: 'jersey-test-framework'
}
hadoopInJar('org.apache.hadoop:hadoop-mapreduce-client-jobclient:2.4.0') {
exclude module: 'jersey-api'
exclude module: 'jersey-json'
exclude module: 'jersey-guice'
exclude module: 'jersey-server'
exclude module: 'jersey-test-framework'
}
and rebuild.
--jacob
Reply all
Reply to author
Forward
0 new messages