Go bindings for LZ4

[hungys]

Hi All,

I have interoperability issue between the original C implementation and a third-party "native Go" implementation (https://github.com/pierrec/lz4), and also found that most of Go bindings on GitHub are out-of-date. So I create a Go bindings (using cgo) based on the latest C implementation.

Since Go's binary.Read() is really slow, I also provides a set of APIs for compressing from or decompressing to struct or slice. Feedback is welcomed :)

GitHub repo: https://github.com/hungys/go-lz4

[Cyan]

Thanks for notification Yu-Hsin,

> I have interoperability issue between the original C implementation and a third-party "native Go" implementation (https://github.com/pierrec/lz4)

Pierre Curto's Go version is advertised as compatible with lz4 cli.
All versions compatible with lz4 cli are labelled "interoperable",

because they read and generate the same frame format : 

https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md

I guess when you mention "original C implementation", you mean using directly LZ4_compress_default() ?
This function generates compressed blocks (a subdivision of a frame).
Block format predates frame format by a few years, hence it's more commonly used.
Your API seems to faithfully reproduces lz4.h API, with identical scope, which means it generates compressed blocks.

I believe it's fine, the scope just needs to be clear.

One important limitation of the block format is that it's just that, a compressed block.

One still needs to provide compressed/decompressed size information for decompression to proceed.

When everything is done inside the same program, it's not an issue : a few more variables are allocated, storing requested information.

So it's a good fit.

But for a file format, it's a more tricky topic.

In general, it translates into some additional metadata, shipped into the file alongside the compressed block.

And because everyone implements its metadata differently, these files are not "interoperable".

They only work with the implementation which generates them.

My understanding is that your binding is not meant to generate files.

It's expecting to compress and decompress blobs of bytes from the same program.

It's fine, it's just a question of selecting a scope, and clearly document it.

[hungys]

Thanks for clarification.

I found that there're functions called `CompressBlock` and `UncompressBlock` in Pierre's version (which were located in block.go). Maybe I should try these.

And yes, I'm writing client/server. The client written in C uses LZ4_compress_fast() for compression, the server written in Go will decompress them. What I'm manipulating is actually block.

Cyan於 2017年8月7日星期一 UTC+8上午11時39分19秒寫道：

[PierreC]

Hello,

Indeed you can use the CompressBlock and UncompressBlock functions as it seems to be within your scope.

If you run into incompatibilities with the C version, file an issue on github!

Pierre