Problem to use Faunus on HDP1.3
170 views
Skip to first unread message
Yanxin Zhang
Sep 30, 2013, 10:45:57 AM9/30/13
to aureliu...@googlegroups.com
Hello,
I installed Faunus 0.32 on HDP 1.3 When I follow the get-start test case inhttps://github.com/thinkaurelius/faunus/wiki/Getting-Started, I got following errors
Compression codec com.hadoop.compression.lzo.LzoCodec not found.
[root@hadoop121 faunus-0.3.2]# bin/gremlin.sh
\,,,/
(o o)
-----oOOo-(_)-oOOo-----
gremlin> g = FaunusFactory.open('bin/faunus.properties')
==>faunusgraph[graphsoninputformat->graphsonoutputformat]
gremlin> g.V
13/09/29 23:52:22 WARN mapreduce.FaunusCompiler: Using the distribution Faunus job jar: lib/faunus-0.3.2-job.jar
13/09/29 23:52:22 INFO mapreduce.FaunusCompiler: Compiled to 1 MapReduce job(s)
13/09/29 23:52:22 INFO mapreduce.FaunusCompiler: Executing job 1 out of 1: MapSequence[com.thinkaurelius.faunus.mapreduce.transform.VerticesMap.Map]
13/09/29 23:52:22 INFO mapreduce.FaunusCompiler: Job data location: output/job-0
13/09/29 23:52:22 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
13/09/29 23:52:23 INFO input.FileInputFormat: Total input paths to process : 1
13/09/29 23:52:23 INFO mapred.JobClient: Cleaning up the staging area hdfs://hadoop121.ctd.com:8020/user/root/.staging/job_201309292334_0004
Compression codec com.hadoop.compression.lzo.LzoCodec not found.
Display stack trace? [yN] y
java.lang.RuntimeException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.
at com.thinkaurelius.faunus.tinkerpop.gremlin.ResultHookClosure.call(ResultHookClosure.java:54)
at groovy.lang.Closure.call(Closure.java:428)
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSite.invoke(PogoMetaMethodSite.java:231)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:64)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at org.codehaus.groovy.tools.shell.Groovysh.setLastResult(Groovysh.groovy:324)
at org.codehaus.groovy.tools.shell.Groovysh.this$3$setLastResult(Groovysh.groovy)
at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.setProperty(MetaClassImpl.java:2416)
at groovy.lang.MetaClassImpl.setProperty(MetaClassImpl.java:3347)
at org.codehaus.groovy.tools.shell.Shell.setProperty(Shell.groovy)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.setGroovyObjectProperty(ScriptBytecodeAdapter.java:528)
at org.codehaus.groovy.tools.shell.Groovysh.execute(Groovysh.groovy:152)
at org.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:114)
at org.codehaus.groovy.tools.shell.Shell$leftShift$0.call(Unknown Source)
at org.codehaus.groovy.tools.shell.ShellRunner.work(ShellRunner.groovy:88)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$work(InteractiveShellRunner.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1079)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:128)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:148)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.work(InteractiveShellRunner.groovy:100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:272)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:52)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:137)
at org.codehaus.groovy.tools.shell.ShellRunner.run(ShellRunner.groovy:57)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$run(InteractiveShellRunner.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1079)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:128)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:148)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.run(InteractiveShellRunner.groovy:66)
at com.thinkaurelius.faunus.tinkerpop.gremlin.Console.<init>(Console.java:54)
at com.thinkaurelius.faunus.tinkerpop.gremlin.Console.<init>(Console.java:61)
at com.thinkaurelius.faunus.tinkerpop.gremlin.Console.main(Console.java:66)
Caused by: java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.
at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:96)
at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:134)
at com.thinkaurelius.faunus.formats.graphson.GraphSONInputFormat.isSplitable(GraphSONInputFormat.java:33)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:258)
at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:1024)
at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1041)
at org.apache.hadoop.mapred.JobClient.access$700(JobClient.java:179)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:959)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:912)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:912)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:500)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:530)
at com.thinkaurelius.faunus.mapreduce.FaunusCompiler.run(FaunusCompiler.java:322)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at com.thinkaurelius.faunus.FaunusPipeline.submit(FaunusPipeline.java:1075)
at com.thinkaurelius.faunus.FaunusPipeline.submit(FaunusPipeline.java:1058)
at com.thinkaurelius.faunus.tinkerpop.gremlin.ResultHookClosure.call(ResultHookClosure.java:38)
... 55 more
Caused by: java.lang.ClassNotFoundException: com.hadoop.compression.lzo.LzoCodec
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:802)
at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:89)
... 75 more
gremlin>
I tried several methods, but none of them are working
1. disable output compression mapred.output.compress=false;
2. put lzo library liblzo2.so in native directory /usr/lib/hadoop/lib/native/Linux-amd64-64;
3. use HDP default SnappyCodec in bin/faunus.properties, mapred.output.compression=org.apache.hadoop.io.compress.SnappyCodec
Please help me out on the wired problem.
Thanks in advance.
Marko Rodriguez
Sep 30, 2013, 11:47:35 AM9/30/13
to aureliu...@googlegroups.com
Hello,
The only thing I can think to do is make sure that the .jar that has the LZOCodec in it is on the classpath of each of your task tracker machines. Unfortunately, you can't set your own compression codec as Faunus will override it:
I just made a ticket so you will see a manual override in Faunus 0.4.0.
You could try and figure out why HDP is using LZO for its default codec…. ?.
Good luck. If you figure it out, please report back.
Marko.
--
You received this message because you are subscribed to the Google Groups "Aurelius" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aureliusgraph...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Marko Rodriguez
Sep 30, 2013, 12:15:56 PM9/30/13
to aureliu...@googlegroups.com
Hey Yanxin,
I updated Faunus 0.4.0-SNAPSHOT to respect user defined compression codecs.
With that, you can do:
It still holds that you need LZOCodec on your tasktracker class paths to get around your problem in Faunus 0.3.2.
Take care,
Marko.
Yanxin Zhang
Sep 30, 2013, 2:26:49 PM9/30/13
to aureliu...@googlegroups.com
Thanks Marko for such detailed help.
The default Codec after I install HDP1.3 is org.apache.hadoop.io.compress.SnappyCodec
Because of the error, I manually change it to com.hadoop.compression.lzo.LzoCodec
However, it still have the same error.
I am newbie to Java, could you let know how to check ".jar that has the LZOCodec in it is on the classpath of each of your task tracker machines" ?
Yanxin Zhang
Sep 30, 2013, 2:49:21 PM9/30/13
to aureliu...@googlegroups.com
Hey Marko,
I got the 4.0 SNAPSHOT from github.
Can you show me how to compile the faunus? I see there is no lib folder in the downloaded files.
How can I run the gremlin.sh ?
Sorry about stupid question. I am newbie to Java. I google how to compile the faunus, but I have no luck.
Yanxin
Marko Rodriguez
Sep 30, 2013, 2:56:29 PM9/30/13
to aureliu...@googlegroups.com
Hi Yanxin,
I got the 4.0 SNAPSHOT from github.Can you show me how to compile the faunus? I see there is no lib folder in the downloaded files.How can I run the gremlin.sh ? Sorry about stupid question. I am newbie to Java. I google how to compile the faunus, but I have no luck.
This is typically where I throw a hissy-fit because I lack patience. But today is your lucky day --- through some self-control, I have sequestered these emotions for one of compassion and service.
faunus$ git pullfaunus$ mvn clean installfaunus$ bin/gremlin.sh
You will need Maven3 as that is Faunus' build manager: http://maven.apache.org/download.cgi
Good luck,
Marko.
Yanxin Zhang
Sep 30, 2013, 6:03:29 PM9/30/13
to aureliu...@googlegroups.com
Thanks Marko for kindness and great help.
I have a question when using mvn
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:compile (default-compile) on project faunus: Compilation failure
[ERROR] Failure executing javac, but could not parse the error:
[ERROR] javac: invalid target release: 1.7
[ERROR] Usage: javac <options> <source files>
[ERROR] use -help for a list of possible options
[ERROR] -> [Help 1]
In the pom.xml, I found it is set to 1.7
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>2.5.1</version>
<configuration>
<source>1.7</source>
<target>1.7</target>
</configuration>
</plugin>
The default HDP1.3 is using JDK 1.6
[root@hadoop121 faunus-974a68afdad771d04b0c79ebbeb554e6d237000f]# mvn -version
Apache Maven 3.1.0 (893ca28a1da9d5f51ac03827af98bb730128f9f2; 2013-06-27 19:15:32-0700)
Maven home: /usr/local/bin/maven
Java version: 1.6.0_31, vendor: Sun Microsystems Inc.
Java home: /usr/jdk64/jdk1.6.0_31/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-358.18.1.el6.centos.plus.x86_64", arch: "amd64", family: "unix"
I tried to change the 1.7 to 1.6, but I got all test errors.
<configuration>
<source>1.6</source>
<target>1.6</target>
</configuration>
15:02:11 WARN snappy.LoadSnappy: Snappy native library not loaded
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.119 sec <<< FAILURE!
Running com.thinkaurelius.faunus.FaunusEdgeTest
Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.004 sec <<< FAILURE!
Running com.thinkaurelius.faunus.formats.graphson.GraphSONOutputFormatTest
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.003 sec <<< FAILURE!
Running com.thinkaurelius.faunus.formats.edgelist.rdf.RDFBlueprintsHandlerTest
testInGremlinImports(com.thinkaurelius.faunus.formats.script.ScriptOutputFormatTest)
testPathsFromVertices(com.thinkaurelius.faunus.mapreduce.transform.PathMapTest)
testPathsAndGetException(com.thinkaurelius.faunus.mapreduce.transform.PathMapTest)
testPathCount(com.thinkaurelius.faunus.mapreduce.util.ElementPickerTest)
testId(com.thinkaurelius.faunus.mapreduce.util.ElementPickerTest)
testLabel(com.thinkaurelius.faunus.mapreduce.util.ElementPickerTest)
Tests run: 180, Failures: 0, Errors: 145, Skipped: 0
Can you give me more instruction? :)
Thank you in advance.
I like Faunus and I want to run Faunus in my environment :)
Yanxin
Marko Rodriguez
Sep 30, 2013, 6:47:27 PM9/30/13
to aureliu...@googlegroups.com
Hi,
Titan 0.4.0 and Faunus 0.4.0 will require at least Java 1.7. You can use HDP 1.3 with Java 1.7 as well.
Marko.
Yanxin Zhang
Sep 30, 2013, 7:21:14 PM9/30/13
to aureliu...@googlegroups.com
Thanks Marko.
I install Java 1.7 in my env, and successfully built Faunus main branch.
Here is what I get. I still have the same error. It's too wired.
Any idea?
[root@hadoop121 faunus]# bin/gremlin.sh
\,,,/
(o o)
-----oOOo-(_)-oOOo-----
gremlin> g = FaunusFactory.open('bin/faunus.properties')
==>faunusgraph[graphsoninputformat->graphsonoutputformat]
gremlin> g.getConf('mapred.map.output.compression')
==>mapred.map.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec
gremlin> g.V.out.count()
16:17:08 WARN mapreduce.FaunusCompiler: Using the developer Faunus job jar: target/faunus-0.4.0-SNAPSHOT-job.jar
16:17:08 INFO mapreduce.FaunusCompiler: Compiled to 2 MapReduce job(s)
16:17:08 INFO mapreduce.FaunusCompiler: Executing job 1 out of 2: MapSequence[com.thinkaurelius.faunus.mapreduce.transform.VerticesMap.Map, com.thinkaurelius.faunus.mapreduce.transform.VerticesVerticesMapReduce.Map, com.thinkaurelius.faunus.mapreduce.transform.VerticesVerticesMapReduce.Reduce]
16:17:08 INFO mapreduce.FaunusCompiler: Job data location: output/job-0
16:17:08 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
16:17:09 INFO input.FileInputFormat: Total input paths to process : 1
16:17:09 INFO mapred.JobClient: Cleaning up the staging area hdfs://hadoop121.ctd.com:8020/user/root/.staging/job_201309301614_0002
Compression codec com.hadoop.compression.lzo.LzoCodec not found.
Display stack trace? [yN] g.getConf().setClass('mapred.map.output.compression.codec',org.apache.hadoop.io.compress.SnappyCodec,CompressionCodec.class)n
gremlin> g.getConf().setClass('mapred.map.output.compression.codec',org.apache.hadoop.io.compress.SnappyCodec,CompressionCodec.class)
16:17:38 WARN snappy.LoadSnappy: Snappy native library is available
16:17:38 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16:17:38 WARN snappy.LoadSnappy: Snappy native library not loaded
==>null
gremlin> g.getConf('mapred.map.output.compression')
==>mapred.map.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec
gremlin> g.V.out.count()
16:18:26 WARN mapreduce.FaunusCompiler: Using the developer Faunus job jar: target/faunus-0.4.0-SNAPSHOT-job.jar
16:18:26 INFO mapreduce.FaunusCompiler: Compiled to 2 MapReduce job(s)
16:18:26 INFO mapreduce.FaunusCompiler: Executing job 1 out of 2: MapSequence[com.thinkaurelius.faunus.mapreduce.transform.VerticesMap.Map, com.thinkaurelius.faunus.mapreduce.transform.VerticesVerticesMapReduce.Map, com.thinkaurelius.faunus.mapreduce.transform.VerticesVerticesMapReduce.Reduce]
16:18:26 INFO mapreduce.FaunusCompiler: Job data location: output/job-0
16:18:26 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
16:18:27 INFO input.FileInputFormat: Total input paths to process : 1
16:18:27 INFO mapred.JobClient: Cleaning up the staging area hdfs://hadoop121.ctd.com:8020/user/root/.staging/job_201309301614_0003
Compression codec com.hadoop.compression.lzo.LzoCodec not found.
Display stack trace? [yN]
Yanxin Zhang
Sep 30, 2013, 7:24:39 PM9/30/13
to aureliu...@googlegroups.com
I guess Snappy is not Loaded into Faunus.
It's still use LZO for compression.
16:17:38 WARN snappy.LoadSnappy: Snappy native library is available
16:17:38 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16:17:38 WARN snappy.LoadSnappy: Snappy native library not loaded
==>null
I think in your environment, the LZO is set up correctedly, so even the Sanppy is not loaded into Faunus, Faunus still use LZO by default.
Yanxin Zhang
Sep 30, 2013, 8:09:27 PM9/30/13
to aureliu...@googlegroups.com
Hi Marko,
I followed your commit, and try to disable compression output completely, but I get the same error
Caused by: java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.
job.getConfiguration().setBoolean(MAPRED_COMPRESS_MAP_OUTPUT, false);
//if (null == job.getConfiguration().get(MAPRED_COMPRESS_MAP_OUTPUT, null))
// job.getConfiguration().setBoolean(MAPRED_COMPRESS_MAP_OUTPUT, true);
//if (null == job.getConfiguration().get(MAPRED_MAP_OUTPUT_COMPRESSION_CODEC, null))
// job.getConfiguration().setClass(MAPRED_MAP_OUTPUT_COMPRESSION_CODEC, DefaultCodec.class, CompressionCodec.class);
Can I force do do NOT use any compression output at all?
I want to follow the traceback of error, but failed.
Caused by: java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.
at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:116)
at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:156)
at com.thinkaurelius.faunus.formats.graphson.GraphSONInputFormat.isSplitable(GraphSONInputFormat.java:33)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:258)
at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:1054)
at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1071)
at org.apache.hadoop.mapred.JobClient.access$700(JobClient.java:179)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:983)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:550)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:580)
at com.thinkaurelius.faunus.mapreduce.FaunusCompiler.run(FaunusCompiler.java:328)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at com.thinkaurelius.faunus.FaunusPipeline.submit(FaunusPipeline.java:1075)
at com.thinkaurelius.faunus.FaunusPipeline.submit(FaunusPipeline.java:1058)
at com.thinkaurelius.faunus.tinkerpop.gremlin.ResultHookClosure.call(ResultHookClosure.java:37)
Yanxin Zhang
Oct 1, 2013, 12:24:35 AM10/1/13
to aureliu...@googlegroups.com
Hi Marko,
I solve the fist problem, that Snappy Codec can not be loaded.
I hard coded the java.library.path in gremlin.sh file
JAVA_OPTIONS="-Xms32m -Xmx512m -Djava.library.path=/usr/lib/hadoop/lib/native/Linux-amd64-64 "
where /usr/lib/hadoop/lib/native/Linux-amd64-64 is my native JAVA library location.
However, I still have the issue of lzo not found.
[root@hadoop121 faunus]# bin/gremlin.sh
\,,,/
(o o)
-----oOOo-(_)-oOOo-----
gremlin> g = FaunusFactory.open('bin/faunus.properties')
==>faunusgraph[graphsoninputformat->graphsonoutputformat]
gremlin> g.getConf().setClass('mapred.map.output.compression.codec',org.apache.hadoop.io.compress.SnappyCodec,CompressionCodec.class)
21:20:05 WARN snappy.LoadSnappy: Snappy native library is available
21:20:05 INFO util.NativeCodeLoader: Loaded the native-hadoop library
21:20:05 INFO snappy.LoadSnappy: Snappy native library loaded
==>null
gremlin> g.V
21:22:45 WARN mapreduce.FaunusCompiler: Using the developer Faunus job jar: target/faunus-0.4.0-SNAPSHOT-job.jar
21:22:45 INFO mapreduce.FaunusCompiler: Compiled to 1 MapReduce job(s)
21:22:45 INFO mapreduce.FaunusCompiler: Executing job 1 out of 1: MapSequence[com.thinkaurelius.faunus.mapreduce.transform.VerticesMap.Map]
21:22:45 INFO mapreduce.FaunusCompiler: Job data location: output/job-0
21:22:45 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
21:22:46 INFO input.FileInputFormat: Total input paths to process : 1
21:22:46 INFO mapred.JobClient: Cleaning up the staging area hdfs://hadoop121.ctd.com:8020/user/root/.staging/job_201309302049_0006
Compression codec com.hadoop.compression.lzo.LzoCodec not found.
Display stack trace? [yN]
In /usr/lib/hadoop/lib/native/Linux-amd64-64 , I also have lzo lib files,
[root@hadoop121 Linux-amd64-64]# ls -lstr
total 144
4 -rw-r--r--. 1 root root 1248 Aug 13 2012 libgplcompression.la
16 -rwxr-xr-x. 1 root root 15768 Aug 13 2012 libgplcompression.so.0.0.0
24 -rw-r--r--. 1 root root 22918 Aug 13 2012 libgplcompression.a
4 -rw-r--r--. 1 root root 1004 Aug 19 18:40 libhadoop.la
32 -rwxr-xr-x. 1 root root 31992 Aug 19 18:40 libhadoop.so.1.0.0
64 -rw-r--r--. 1 root root 62582 Aug 19 18:40 libhadoop.a
0 lrwxrwxrwx. 1 root root 23 Sep 27 22:15 libsnappy.so -> /usr/lib64/libsnappy.so
0 lrwxrwxrwx. 1 root root 18 Sep 27 22:16 libhadoop.so.1 -> libhadoop.so.1.0.0
0 lrwxrwxrwx. 1 root root 18 Sep 27 22:16 libhadoop.so -> libhadoop.so.1.0.0
0 lrwxrwxrwx. 1 root root 26 Sep 27 22:16 libgplcompression.so.0 -> libgplcompression.so.0.0.0
0 lrwxrwxrwx. 1 root root 26 Sep 27 22:16 libgplcompression.so -> libgplcompression.so.0.0.0
0 lrwxrwxrwx. 1 root root 27 Sep 29 23:28 liblzo2.so.2.0.0 -> /usr/lib64/liblzo2.so.2.0.0
0 lrwxrwxrwx. 1 root root 23 Sep 29 23:29 liblzo2.so.2 -> /usr/lib64/liblzo2.so.2
0 lrwxrwxrwx. 1 root root 21 Sep 29 23:29 liblzo2.so -> /usr/lib64/liblzo2.so
liblzo2 is preinstalled by HDP1.3. How can I verify lzo2 is installed correctly?
Yanxin
Yanxin Zhang
Oct 1, 2013, 1:02:06 AM10/1/13
to aureliu...@googlegroups.com
Hi Marko,
I have figured out the temp solution. It's kind of HDP bug, which was reported by 2-3 persons.
hadoop-lzo-0.5.0.jar is in hadoop lib folder, /usr/lib/hadoop/lib/
And I do have CLASSPATH = /usr/lib/hadoop/lib/
However, it can not find the lzo library.
My temp fix is to copy hadoop-lzo-0.5.0.jar to faunus lib folder, which in my env it is /root/faunus-0.3.2/lib
Then it works.
Hopefully, my notes can help other ppl who encountered the same problem.
gremlin> g = FaunusFactory.open('bin/faunus.properties')
==>faunusgraph[graphsoninputformat->graphsonoutputformat]
gremlin> g.V
13/09/30 21:57:51 WARN mapreduce.FaunusCompiler: Using the distribution Faunus job jar: lib/faunus-0.3.2-job.jar
13/09/30 21:57:51 INFO mapreduce.FaunusCompiler: Compiled to 1 MapReduce job(s)
13/09/30 21:57:51 INFO mapreduce.FaunusCompiler: Executing job 1 out of 1: MapSequence[com.thinkaurelius.faunus.mapreduce.transform.VerticesMap.Map]
13/09/30 21:57:51 INFO mapreduce.FaunusCompiler: Job data location: output/job-0
13/09/30 21:57:51 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
13/09/30 21:57:52 INFO input.FileInputFormat: Total input paths to process : 1
13/09/30 21:57:52 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
13/09/30 21:57:52 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev cf4e7cbf8ed0f0622504d008101c2729dc0c9ff3]
13/09/30 21:57:52 WARN snappy.LoadSnappy: Snappy native library is available
13/09/30 21:57:52 INFO util.NativeCodeLoader: Loaded the native-hadoop library
13/09/30 21:57:52 INFO snappy.LoadSnappy: Snappy native library loaded
13/09/30 21:57:53 INFO mapred.JobClient: Running job: job_201309302049_0010
13/09/30 21:57:54 INFO mapred.JobClient: map 0% reduce 0%
Reply all
Reply to author
Forward
0 new messages