Google Groups

'native-lzo library not available' issue


unionx Jul 23, 2012 7:37 PM
Posted in group: elephantbird-dev
I have installed and configured lzo on my cluseter. I can use a script to compress my data on HDFS now.

I have configured codecs in core-site.xml and mapred-site.xml, and I have added JAVA_LIBRARY_PATH and HADOOP_CLASSPATH in hadoop-env.sh.

But there are some error. When I try to use Pig or Java to load lzo files, there is 'native-lzo library not available' error:

2012-07-24 10:22:13,113 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
2012-07-24 10:22:13,122 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2999: Unexpected internal error. native-lzo library not available
2012-07-24 10:22:13,122 [main] ERROR org.apache.pig.tools.grunt.GruntParser - java.lang.RuntimeException: native-lzo library not available
    at com.hadoop.compression.lzo.LzopCodec.createDecompressor(LzopCodec.java:104)
    at com.hadoop.compression.lzo.LzopCodec.createInputStream(LzopCodec.java:89)
    at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:70)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.initialize(PigRecordReader.java:176)
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:418)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:620)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)

But when the job is intializating, it can intialize lzo native library successfully:

2012-07-24 10:18:31,016 [Thread-12] INFO  com.hadoop.compression.lzo.GPLNativeCodeLoader - Loaded native gpl library
2012-07-24 10:18:31,021 [Thread-12] INFO  com.hadoop.compression.lzo.LzoCodec - Successfully loaded & initialized native-lzo library [hadoop-lzo rev 6bb1b7f8b9044d8df9b4d2b6641db7658aab3cf8]

Is there something I missed?

--
‘(hello world)