Interoperability c++/Java

[Flo]

Hi,

For a given input byte array, i need that a c++ program and a Java program return the same output compressed byte array, so that eventually c++ side can decompressed what has been compressed on java side and vice versa.

I'm working on the java side and i don't get my compressed output to match with the compressed output sent by the c++ colleague. Some slight difference is observed.

Under what conditions this should work? Block format? xxhash?

On java side, i have a maven dependency to net.jpountz.lz4 1.3-0.

C++ side uses c implementation from lz4.c

Thanks

[Cyan]

Only implementations tagged "interoperable" are compatible with each other :

http://lz4.github.io/lz4/#interoperable-lz4

To this day, there is no Java implementation which is tagged "interoperable".
jpountz version uses a custom format, which is not supposed to be compatible with any other version (save by chance).

Writing an interoperable LZ4 version in Java is an achievement still to be unlocked.

It's a matter of respecting the Frame format specification .

When working on a binding to the reference C implementation, it's a matter of binding to lz4frame API ,

which automatically insert conformant metadata.

Alternatively, if your sole use case is linked to jpountz java implementation,

you can try to reverse-engineed its format. It may be documented somewhere.

It's quite likely a simple format, such as adding some size fields at the beginning.

[Flo]

Thanks for your input.

So i understand that it should work if c++ side uses the frame format and java side does JNI binding to the same frame format c implementation. Please correct if this is the wrong missunderstanding.

[Cyan]

Correct !

[stephen fogarasi]

Has there been any further work done in the interoperable Java <-> C++ versions ? I have the same issue (compress in C++, decompress in Java)  If there is way, can someone document the process ?

thx

[Cyan]

Well, as mentioned on LZ4 homepage,

Apache Commons offers a Java LZ4 library which uses the official frame format,
hence is interoperable with any other library using the same format, including C++ ones.

[stephen fogarasi]

thanks for the pointer to the apache.commons.compress.compressors.lz4 library

I was able to code a Java decompressor (and compressor) that was able to read lz4 files files created with the C++ code (and vice versa).

However the performance compared to the  net.jpountz.lz4 was .... poor, really poor. Compressing was really slow, but fortunately, I only really need decompression in Java for now.

Is there something I can do boost the performance of the decompression ? This is my code for decompression :

    public byte[] decompressUsingApacheCommonsCompressFramesStreams(byte[] data) {

        byte[] decompressed = null;

        ByteArrayOutputStream byteOutputStream = new ByteArrayOutputStream();

        try {

            ByteArrayInputStream byteInputStream = new ByteArrayInputStream(data, 0, data.length);

            BufferedInputStream in = new BufferedInputStream(byteInputStream);

            FramedLZ4CompressorInputStream lz4InputStream = new FramedLZ4CompressorInputStream(in, true);

            int count = 0;

            int totalCount = 0;

            final byte[] buffer = new byte[bufferSize];

            while ((count = lz4InputStream.read(buffer)) != -1) {

                totalCount += count;

                byteOutputStream.write(buffer, 0, count);

            }

            logger.info("total size decompressed : " + totalCount);

            logger.info("number of compressed bytes decompressed into stream : " + lz4InputStream.getCompressedCount());

            byteOutputStream.flush();

            decompressed = byteOutputStream.toByteArray();

            byteOutputStream.close();

            lz4InputStream.close();

            logger.info("input compressed length : " + data.length + ", output decompressed length : "

                    + decompressed.length);

        }

        catch (IOException e) {

            logger.error("Error ! IOException thrown by LZ4 buffer streams : " + e.getMessage());

        }

        catch (Exception e) {

            logger.error("Error ! General exception thrown : " + e.getMessage());

        }

        return decompressed;

    }

FYI, I used a bufferSize of 64K

[Cyan]

I can't help on this front.
Maybe some Java developer can step in to provide better insight on Java performance.