'native-lzo library not available' issue

[帝归]

I have installed and configured lzo on my cluseter. I can use a script to compress my data on HDFS now.

I have configured codecs in core-site.xml and mapred-site.xml, and I have added JAVA_LIBRARY_PATH and HADOOP_CLASSPATH in hadoop-env.sh.

But there are some error. When I try to use Pig or Java to load lzo files, there is 'native-lzo library not available' error:

2012-07-24 10:22:13,113 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
2012-07-24 10:22:13,122 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2999: Unexpected internal error. native-lzo library not available
2012-07-24 10:22:13,122 [main] ERROR org.apache.pig.tools.grunt.GruntParser - java.lang.RuntimeException: native-lzo library not available
    at com.hadoop.compression.lzo.LzopCodec.createDecompressor(LzopCodec.java:104)
    at com.hadoop.compression.lzo.LzopCodec.createInputStream(LzopCodec.java:89)
    at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:70)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.initialize(PigRecordReader.java:176)
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:418)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:620)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)

But when the job is intializating, it can intialize lzo native library successfully:

2012-07-24 10:18:31,016 [Thread-12] INFO  com.hadoop.compression.lzo.GPLNativeCodeLoader - Loaded native gpl library
2012-07-24 10:18:31,021 [Thread-12] INFO  com.hadoop.compression.lzo.LzoCodec - Successfully loaded & initialized native-lzo library [hadoop-lzo rev 6bb1b7f8b9044d8df9b4d2b6641db7658aab3cf8]

Is there something I missed?

--
‘(hello world)

[Dmitriy Ryaboy]

It sounds like the LZO library is set up on the machine you are
submitting from, but not available on the cluster machines.

Is it possible you did not install it on those machines, or did not
update the configs on the TaskTrackers to point at the newly installed
compression lib, or updated the configs but forgot to restart the
TaskTrackers?

Dmitriy.

> --
> You received this message because you are subscribed to the Google Groups
> "elephantbird-dev" group.
> To post to this group, send email to elephant...@googlegroups.com.
> To unsubscribe from this group, send email to
> elephantbird-d...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/elephantbird-dev?hl=en.



--
Dmitriy V Ryaboy
Twitter Analytics
http://twitter.com/squarecog

[unionx]

Yes!

I checked out my slaves, I forgot copying my native library to them. Thank you.

在 2012年7月24日星期二UTC+8上午11时13分39秒，Dmitriy Ryaboy写道：

> To post to this group, send email to elephantbird-dev@googlegroups.com.

> To unsubscribe from this group, send email to

> elephantbird-dev+unsubscribe@googlegroups.com.