FAILED_TO_UNCOMPRESS(5)

[Himanish Kushary]

Hi,

I was trying to read a file generated through Hadoop Snappy Compression using the snappy-java library.I keep getting the below error.

This is the code - SnappyInputStream in = new SnappyInputStream(new BufferedInputStream(new FileInputStream(new File(fileName))));

java.io.IOException: FAILED_TO_UNCOMPRESS(5)

at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78)

at org.xerial.snappy.SnappyNative.rawUncompress(Native Method)

at org.xerial.snappy.Snappy.rawUncompress(Snappy.java:395)

at org.xerial.snappy.Snappy.uncompress(Snappy.java:431)

at org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:127)

at org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:88)

at org.xerial.snappy.SnappyInputStream.<init>(SnappyInputStream.java:58)

Could somebody please give an idea why I could be getting this.

Thanks

Himanish

[Taro L. Saito]

Hi,

SnappyInputStream should be used for uncompressing the data compressed by SnappyOutputStream.

2012年10月2日火曜日 Himanish Kushary hima...@gmail.com:

--
You received this message because you are subscribed to the Google Groups "Xerial" group.
To view this discussion on the web visit https://groups.google.com/d/msg/xerial/-/QREkITxksZgJ.
To post to this group, send email to xer...@googlegroups.com.
To unsubscribe from this group, send email to xerial+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/xerial?hl=en.

--
--
Taro L. Saito
<l...@xerial.org>
University of Tokyo
http://www.xerial.org/leo
Tel. +81-47-136-4065 (64065)

[Taro L. Saito]

I guess Hadoopy Snappy compression uses a little bit different format from the one used in snappy-java.

2012年10月2日火曜日 Taro L. Saito tar...@gmail.com:

[Himanish Kushary]

Thanks Taro.Do you have any suggestion about how to read a snappy file that has been generated by Hadoop using Java.I would also like to change few csv file to snappy files for consumption by Hadoop M/R.

It would have been great of some java library was available to do that.

Thanks

Himanish

Thanks & Regards
Himanish

[Taro L. Saito]

Hi, 

It would be possible to create HadoopSnappyInput/OutputStream classes based on

snappy-java if the information of the data compression scheme used in Hadoop is available.

For example, snappy-java uses the following stream format to compress large data streams:

|header|chunk size| compressed data using snappy | cunk size | compressed data ...| ...

If no header is found in the compressed data, SnappyInputStream tries to decompress the whole data using snappy. 

If SnappyInputStream doesn't work for Hadoop result files, it might be using some special 

format tailored for Hadoop. 

We need to look at the code of Hadoop.

Regards,

--
Taro L. Saito
<l...@xerial.org>
University of Tokyo
http://www.xerial.org/leo
Tel. +81-47-136-4065 (64065)