ARM Cortex M4 implementation

[Frank]

Hi,

I am trying to port this code to ARM Cortex M4. My application is a simple on the fly compression. I get 10Kb of data from a peripheral, I like to compress this data on the fly and send it wirelessly to another device. 

I have tried to HC version but it fails immediately probably the memory is not enough. 

I have fallen back to the normal version and I have some questions:

- I do a simple 10K buffer and fill it with a simple 8 bit counter (0-255) and repeat this for the whole buffer. I call the function, it returns 367. I repeat the same experiment on x86 and it returns 306. 

- I can decode the x86 file but not the ARM file. My ARM is STM32 it works in little endian mode. 

I am guessing there is a memory problem somewhere (I don't have enough of it). 

Any hints from the experts here?

[Yann Collet]

Hi Frank

> I have tried to HC version but it fails immediately probably the memory is not enough.

The HC version requires 256KB of memory for its tables.
In contrast, the fast version requires only 16KB.

> I can decode the x86 file but not the ARM file. My ARM is STM32 it works in little endian mode.

There might be a bug in the Endian detection macro.
It is this part in LZ4.c :

// Little Endian or Big Endian ?
// Note : overwrite the below #define if you know your architecture endianess
#if (defined(__BIG_ENDIAN__) || defined(__BIG_ENDIAN) || defined(_BIG_ENDIAN) || defined(_ARCH_PPC) || defined(__PPC__) || defined(__PPC) || defined(PPC) || defined(__powerpc__) || defined(__powerpc) || defined(powerpc) || ((defined(__BYTE_ORDER__)&&(__BYTE_ORDER__ == __ORDER_BIG_ENDIAN__))) )
#  define LZ4_BIG_ENDIAN 1
#else
// Little Endian assumed. PDP Endian and other very rare endian format are unsupported.
#endif

You can force Little endian mode if you know this is the one used by your CPU.

If it turns out to be the right reason for your problem, i'm interested in understanding what's wrong in the detection routine,
in order to correct it.


Best Regards

Yann

[Frank W.]

Hi Yann,

Thanks for the quick feedback. 

I have managed to get the fast version. However there are a few concerns:

- 16Kb is a big number but still acceptable. I observed, if I compress a 10K data (a simple 0 to 255 filled, repeating array) the stack usage is <10K, if I increase the data size to 20K, I see larger stack requirement. In my platform, I couldn't test larger values. I wonder if this is normal? 

- There are no return error codes for memory allocation, probably it is ok to assume this situation in server/pc space but in embedded systems this is deadly. Especially it is not clear if the memory requirements would change with the data. It would be great to have a non memory allocation option where every memory required is predefined and allocated in arrays and there is no malloc/free. (It would be awesome if you had this already built in). Since, we would know we never run out of memory and place locations etc such that OS never touches that type of area. Now I make a 20KB stack in my application where 16KB is used by the library. At least to me, this is not good practice. 

- Speed of the library is impressive, based on the cycle count the 20K data is compressed in 0.44msec (160MHz Cortex M4). This is with compiler optimization (level 3) turned on. 

[Yann Collet]

> - 16Kb is a big number but still acceptable.

This can be defined to basically any value you want.
This is in the "tuning parameters" :

// COMPRESSIONLEVEL :
// Increasing this value improves compression ratio
// Lowering this value reduces memory usage
// Reduced memory usage typically improves speed, due to cache effect (ex : L1 32KB for Intel, L1 64KB for AMD)
// Memory usage formula : N->2^(N+2) Bytes (examples : 12 -> 16KB ; 17 -> 512KB)
#define COMPRESSIONLEVEL 12

To decrease memory usage, decrease compression level.
For example, to reach a 4K memory usage, just decrease compression level to 10.

Note that compression performance will suffer from decreased memory usage.
The optimal trade-off is up to you.

If the above instruction is not clear enough, maybe i will have to change the naming and/or documentation associated.

> There are no return error codes for memory allocation

Since memory allocation is using standard allocation processes (stack or malloc()),
i have no idea how this could be done in a standard and portable way.
Please feel free to suggest,
and fork this version to support features you need.
Sometimes, it happens some specific features cannot be provided in the generic version.

> it is not clear if the memory requirements would change with the data

No, it doesn't. Table size if fixed.

> It would be great to have a non memory allocation option where every memory required is predefined and allocated in arrays and there is no malloc/free.

This is possible.
You can have a look at the function :

int LZ4_compress64kCtx(void** ctx, const char* source,  char* dest, int isize);

> Speed of the library is impressive

That's great, good to know !

Also, note that, if i do understand correctly, Cortex M4 supports unaligned access.
Sometimes, the macro in charge of detecting this feature is over-cautious, and trigger "safe memory access" code on ARM CPU.
You may improve your performance by manually enabling unaligned access, by forcing the following define :

// Unaligned memory access is automatically enabled for "common" CPU, such as x86.
// For others CPU, the compiler will be more cautious, and insert extra code to ensure aligned access is respected
// If you know your target CPU supports unaligned memory access, you may want to force this option manually to improve performance
#if defined(__ARM_FEATURE_UNALIGNED)
#  define LZ4_FORCE_UNALIGNED_ACCESS 1
#endif

Regards

Le vendredi 20 juillet 2012 14:26:50 UTC+2, Frank a écrit :

Hi Yann,

Thanks for the quick feedback. 

I have managed to get the fast version. However there are a few concerns:

- 16Kb is a big number but still acceptable. I observed, if I compress a 10K data (a simple 0 to 255 filled, repeating array) the stack usage is <10K, if I increase the data size to 20K, I see larger stack requirement. In my platform, I couldn't test larger values. I wonder if this is normal? 

- There are no return error codes for memory allocation, probably it is ok to assume this situation in server/pc space but in embedded systems this is deadly. Especially it is not clear if the memory requirements would change with the data. It would be great to have a non memory allocation option where every memory required is predefined and allocated in arrays and there is no malloc/free. (It would be awesome if you had this already built in). Since, we would know we never run out of memory and place locations etc such that OS never touches that type of area. Now I make a 20KB stack in my application where 16KB is used by the library. At least to me, this is not good practice. 

- Speed of the library is impressive, based on the cycle count the 20K data is compressed in 0.44msec (160MHz Cortex M4). This is with compiler optimization (level 3) turned on. 

[Frank W.]

Yan,

Sorry to ask a specific question, I am using LZ4 to compress a RAW data from a camera. Camera spits out 12 bit data, I make a 16 bit out of it and take 20K of this data and feed to the compressor. Resulting compression is really disappointing. I attach a binary file that contains my images in this case they are fully black (no lens at the camera just cap). 

Interestingly, if I use a 7z to compress, it is also disappointing. I am suspecting I am doing something wrong. Do you mind suggesting what am I doing wrong? Would increasing compression level help or any other trick?

Thanks

F. 

ps. Here is a snippet of the raw image file, I also attached the original file but not so sure it will make it to the group. 

E6 00 EA 00 EE 00 ED 00 EF 00 EE 00 E7 00 E5 00 EC 00 EC 00 E5 00 E7 00 E8 00 F0 00 EF 00 F2 00 EC 00 E1 00 F0 00 E7 00 EB 00 EC 00 F5 00 F5 00 F2 00 EE 00 F3 00 F0 00 F2 00 EF 00 E8 00 EB 00 F1 00 F2 00 F2 00 F1 00 F0 00 F4 00 F5 00 F9 00 F5 00 EF 00 F5 00 F0 00 F2 00 F0 00 EE 00 F1 00 F3 00 F3 00 F5 00 F3 00 EF 00 EC 00 EA 00 F2 00 F4 00 F2 00 EB 00 F3 00 F1 00 F5 00 F2 00 F4 00 EE 00 E6 00 F0 00 F1 00 F2 00 F4 00 ED 00 F6 00 F1 00 EE 00 F3 00 EF 00 F3 00 F4 00 EB 00 EC 00 F5 00 F1 00 EE 00 F3 00 F5 00 F2 00 ED 00 ED 00 F0 00 EE 00 F1 00 EF 00 F2 00 F8 00 F2 00 F5 00 F1 00 F1 00 EE 00 F3 00 F4 00 EF 00 FC 00 EE 00 F2 00 F1 00 EB 00 F4 00 F5 00 F0 00 F1 00 F0 00 EE 00 EF 00 F7 00 F2 00 F9 00 ED 00 F1 00 F3 00 ED 00 F5 00 EF 00 F7 00 F4 00 F8 00 F2 00 F1 00 F3 00 F3 00 F9 00 F2 00 F2 00 EE 00 F5 00 F5 00 F1 00 F3 00 ED 00 FD 00 F6 00 F2 00 F0 00 F9 00 F7 00 F2 00 FC 00 F0 00 F8 00 F1 00 F5 00 F5 00 F6 00 F0 00 F6 00 F5 00 FD 00 FE 00 FC 00 F9 00 F2 00 F5 00 EF 00 FA 00 F6 00 F3 00 F6 00 F3 00 F3 00 FA 00 F8 00 F6 00 EF 00 FA 00 F5 00 F7 00 FC 00 F8 00 FA 00 F8 00 F4 00 F6 00 F4 00 F3 00 F3 00 EF 00 F4 00 F2 00 F3 00 F8 00 FB 00 F0 00 FC 00 F7 00 FA 00 F9 00 F7 00 02 01 FC 00 F1 00 F4 00 F9 00 FF 00 F9 00 03 01 FB 00 F7 00 F4 00 F6 00 F2 00 F8 00 F4 00 F8 00 F5 00 F5 00 F8 00 F2 00 F3 00 FB 00 F1 00 F7 00 F2 00 FB 00 F6 00 F8 00 FA 00 FD 00 F3 00 F5 00 EC 00 F4 00 F6 00 F0 00 F9 00 F7 00 F6 00 FB 00 FB 00 FB 00 F6 00 F0 00 F7 00 F8 00 F4 00 FE 00 F8 00 FC 00 F4 00 FB 00 FA 00 FC 00 FD 00 F6 00 FB 00 F9 00 F4 00 F9 00 F1 00 FA 00 FC 00 F3 00 FB 00 F8 00 F3 00 F7 00 00 01 F4 00 F6 00 FC 00 F4 00 F3 00 F2 00 FA 00 F4 00 F8 00 FC 00 FE 00 00 01 FB 00 FC 00 FA 00 F8 00 F7 00 FB 00 F3 00 FA 00 03 01 00 01 FB 00 F7 00 01 01 F7 00 FF 00 00 01 F6 00 F7 00 F8 00 FE 00 F6 00 01 01 FB 00 FC 00 FB 00 F5 00 F6 00 FA 00 FB 00 FD 00 F8 00 FD 00 F1 00 FD 00 FD 00 F8 00 F9 00 F8 00 00 01 F5 00 FD 00 F9 00 F9 00 F4 00 FF 00 EF 00 FD 00 F5 00 F6 00 FE 00 F7 00 F9 00 FD 00 F9 00 F6 00 F9 00 F5 00 FA 00 FB 00 FC 00 F9 00 F6 00 FB 00 04 01 00 01 F8 00 F8 00 FD 00 F5 00 FD 00 F6 00 F7 00 FD 00 F9 00 FF 00 FA 00 FF 00 F6 00 FE 00 F9 00 FE 00 FB 00 FF 00 FB 00 FC 00 F9 00 FF 00 F2 00 FE 00 FB 00 F9 00 FC 00 F8 00 FC 00 FB 00 FE 00 F3 00 F8 00 FC 00 F9 00 FA 00 00 01 FF 00 F0 00 03 01 FD 00 FA 00 FC 00 FD 00 F8 00 FA 00 FA 00 FC 00 F7 00 FA 00 FB 00 00 01 FB 00 F8 00 FC 00 FD 00 FC 00 FB 00 F5 00 F9 00 F6 00 F7 00 F5 00 F8 00 FA 00 FD 00 F7 00 FD 00 05 01 FF 00 F9 00 05 01 FE 00 F4 00 FF 00 F5 00 FC 00 FF 00 03 01 FE 00 FE 00 FB 00 FA 00 F7 00 FA 00 F6 00 02 01 02 01 F6 00 F2 00 FC 00 FD 00 FC 00 FB 00 F4 00 FE 00 F2 00 FB 00 FC 00 FF 00 01 01 F7 00 F7 00 FB 00 FA 00 FC 00 F8 00 FE 00 FC 00 F5 00 FB 00 F8 00 FC 00 FA 00 FC 00 01 01 FB 00 F5 00 F9 00 02 01 FE 00 F5 00 FC 00 FA 00 00 01 F6 00 FC 00 FE 00 F7 00 FD 00 F5 00 FE 00 FF 00 FF 00 FB 00 01 01 FA 00 FC 00 F8 00 FF 00 FA 00 FD 00 FF 00 F8 00 FB 00 FA 00 01 01 F7 00 FB 00 00 01 00 01 F8 00 F4 00 FD 00 FD 00 03 01 F8 00 FB 00 FF 00 FD 00 FE 00 FF 00 F8 00 FE 00 FB 00 FA 00 F9 00 F4 00 F9 00 00 01 FF 00 FB 00 03 01 F6 00 F8 00 FD 00 00 01 F7 00 F9 00 FD 00 FB 00 FB 00 FA 00 F9 00 FB 00 FE 00 FC 00 FD 00 FD 00 F6 00 FB 00 FE 00 FC 00 F4 00 F2 00 FE 00 F9 00 FC 00 FD 00 03 01 FC 00 FB 00 FE 00 F9 00 F5 00 F3 00 FF 00 04 01 FA 00 FD 00 FE 00 F7 00 06 01 00 01 F4 00 00 01 FF 00 00 01 F5 00 F7 00 00 01 00 01 FC 00 F9 00 FD 00 F9 00 F6 00 FC 00 FE 00 00 01 FA 00 F6 00 FC 00 F8 00 F9 00 FA 00 FD 00 F5 00 F7 00 F5 00 F5 00 FC 00 00 01 FB 00 FB 00 F3 00 01 01 01 01 F6 00 FD 00 F4 00 FF 00 FC 00 F9 00 F2 00 FD 00 F7 00 01 01 F7 00 FC 00 F9 00 FD 00 FC 00 03 01 F7 00 F7 00 FC 00 FC 00 F9 00 01 01 FE 00 F8 00 05 01 FC 00 F8 00 03 01 F7 00 FC 00 FE 00 FF 00 FA 00 F4 00 F7 00 F9 00 FB 00 FF 00 F6 00 F8 00 FA 00 FB 00 FC 00 F9 00 FA 00 F7 00 F7 00 F8 00 01 01 FA 00 FA 00 FB 00 FA 00 FC 00 F6 00 FF 00 FA 00 F3 00 F6 00 FF 00 F7 00 FA 00 F8 00 FF 00 FE 00 F9 00 FB 00 F8 00 FD 00 F7 00 F9 00 FD 00 FF 00 00 01 F4 00 FB 00 F8 00 F5 00 00 01 FD 00 FD 00 05 01 FB 00 F9 00 F6 00 F7 00 FE 00 03 01 FA 00 F5 00 F7 00 02 01 F8 00 00 01 F4 00 03 01 F9 00 01 01 F7 00 F6 00 00 01 F7 00 F8 00 03 01 F1 00 F9 00 F8 00 FB 00 F9 00 FA 00 FE 00 F9 00 F8 00 FB 00 F8 00 FC 00 FB 00 F4 00 F8 00 F7 00 F5 00 FF 00 FB 00 F6 00 F8 00 F7 00 F9 00 FD 00 F9 00 FB 00 F3 00 F8 00 F9 00 F6 00 F7 00 F9 00 F6 00 F3 00 F4 00 F6 00 F7 00 FC 00 FA 00 F8 00 F8 00 F8 00 F3 00 F3 00 F5 00 F9 00 F2 00 FB 00 FD 00 F7 00 F6 00 FD 00 F9 00 F9 00 FB 00 F7 00 F6 00 E6 00 F5 00 EF 00 F0 00 EC 00 EE 00 F2 00 ED 00 EE 00 EC 00 F0 00 EF 00 E7 00 E7 00 EE 00 F3 00 F1 00 F4 00 F2 00 EF 00 E9 00 EF 00 F1 00 EF 00 EA 00 F6 00 F2 00 FD 00 EA 00 F8 00 F4 00 F3 00 EC 00 F6 00 F3 00 F0 00 F4 00 F6 00 F7 00 F6 00 FC 00 F3 00 EF 00 F2 00 F0 00 EF 00 F7 00 E9 00 F2 00 F3 00 F1 00 F3 00 F0 00 F8 00 F5 00 F9 00 F8 00 F8 00 F2 00 F8 00 F5 00 F1 00 EF 00 F6 00 F6 00 F2 00 F0 00 F2 00 F3 00 F4 00 F5 00 F6 00 F7 00 F8 00 F5 00 ED 00 F4 00 F3 00 F3 00 F1 00 F4 00 F6 00 F9 00 F3 00 EE 00 EF 00 FF 00 EE 00 F9 00 F6 00 F5 00 F6 00 F3 00 F2 00 F3 00 F6 00 F7 00 F1 00 F5 00 F8 00 FB 00 F6 00 EF 00 F7 00 F2 00 FA 00 F2 00 F4 00 F6 00 F7 00 FA 00 F7 00 F2 00 F1 00 F5 00 F5 00 F0 00 F5 00 F5 00 F5 00 F9 00 FC 00 FB 00 F8 00 F6 00 F8 00 F7 00 F0 00 FC 00 F5 00 F9 00 F8 00 F8 00 F1 00 FE 00 F5 00 FA 00 F6 00 FA 00 FE 00 F0 00 FB 00 FE 00 FA 00 F2 00 FA 00 FD 00 F4 00 F9 00 FC 00 FB 00 F8 00 F6 00 F5 00 F5 00 FA 00 FB 00 FB 00 FB 00 F8 00 F8 00 FC 00 FC 00 F4 00 F4 00 F1 00 FC 00 F6 00 F3 00 FC 00 F2 00 F3 00 FB 00 F8 00 FD 00 F6 00 F8 00 FC 00 F8 00 01 01 F6 00 FC 00 FA 00 F2 00 F7 00 FD 00 00 01 FB 00 FD 00 F8 00 F8 00 FB 00 F6 00 FF 00 FB 00 F6 00 F2 00 F6 00 FC 00 F8 00 F8 00 F4 00 FA 00 FC 00 F3 00 F9 00 FF 00 F6 00 FC 00 F4 00 FF 00 FD 00 FD 00 F6 00 F3 00 FB 00 FC 00 FD 00 F9 00 00 01 00 01 04 01 F9 00 F8 00 F7 00 FE 00 F9 00 F9 00 FC 00 F7 00 FF 00 FA 00 F8 00 F4 00 FB 00 F5 00 FB 00 FD 00 F7 00 FC 00 01 01 F8 00 FA 00 00 01 FA 00 03 01 FD 00 F9 00 02 01 03 01 FE 00 FE 00 04 01 00 01 F0 00 FC 00 FD 00 F5 00 01 01 FF 00 FA 00 FF 00 00 01 FC 00 FB 00 FB 00 FD 00 FB 00 FB 00 F9 00 FB 00 F4 00 01 01 00 01 00 01 00 01 FC 00 FD 00 04 01 F7 00 FE 00 00 01 FE 00 06 01 F9 00 F9 00 00 01 FE 00 04 01 F8 00 02 01 F4 00 FC 00 FD 00 00 01 FD 00 FD 00 FC 00 F4 00 F7 00 FE 00 FE 00 FB 00 FC 00 04 01 FC 00 FC 00 FB 00 FA 00 FD 00 FC 00 FF 00 F7 00 FF 00 FF 00 F9 00 FB 00 F9 00 FB 00 FC 00 FC 00 FE 00 FA 00 FE 00 FE 00 02 01 02 01 FB 00 FA 00 F8 00 F9 00 FE 00 FD 00 FC 00 FB 00 FE 00 FD 00 FF 00 FB 00 00 01 FC 00 FC 00 FF 00 FD 00 FA 00 01 01 03 01 FE 00 FD 00 FE 00 FF 00 FF 00 03 01 FD 00 FC 00 F9 00 03 01 04 01 FA 00 FC 00 FD 00 FE 00 01 01 FE 00 00 01 F8 00 FE 00 FC 00 FE 00 01 01 01 01 FD 00 F9 00 03 01 01 01 00 01 FC 00 02 01 FF 00 01 01 03 01 05 01 00 01 00 01 FD 00 FD 00 FC 00 FB 00 FC 00 FC 00 FC 00 02 01 06 01 06 01 FE 00 08 01 FF 00 06 01 FB 00 F8 00 FD 00 01 01 F9 00 00 01 01 01 FC 00 0A 01 FE 00 06 01 03 01 FD 00 01 01 FF 00 F3 00 F9 00 FE 00 FD 00 00 01 FA 00 05 01 FC 00 F7 00 FD 00 01 01 FF 00 01 01 02 01 00 01 FC 00 00 01 04 01 05 01 F5 00 FC 00 08 01 00 01 FF 00 FF 00 FA 00 F7 00 F9 00 FD 00 FD 00 08 01 FA 00 FA 00 FF 00 F9 00 02 01 06 01 FC 00 03 01 FD 00 FF 00 FC 00 01 01 00 01 00 01 F9 00 FA 00 05 01 FD 00 01 01 FE 00 FF 00 FE 00 FF 00 00 01 FD 00 FF 00 03 01 03 01 FD 00 F5 00 FA 00 F9 00 F8 00 F9 00 F8 00 FE 00 01 01 02 01 04 01 FB 00 03 01 FD 00 00 01 01 01 FD 00 FC 00 02 01 01 01 06 01 F9 00 F8 00 FA 00 01 01 01 01 00 01 01 01 FF 00 FF 00 05 01 FF 00 FE 00 01 01 02 01 07 01 FA 00 FF 00 FA 00 03 01 F4 00 04 01 FD 00 03 01 FA 00 02 01 08 01 FA 00 FE 00 FF 00 00 01 FF 00 04 01 FF 00 FE 00 02 01 01 01 FA 00 FD 00 FE 00 02 01 F6 00 01 01 FC 00 FC 00 FC 00 F6 00 FC 00 FE 00 07 01 00 01 FE 00 FD 00 01 01 FE 00 FD 00 04 01 02 01 FB 00 FC 00 FD 00 FB 00 02 01 0A 01 F8 00 F4 00 FD 00 FD 00 00 01 FA 00 02 01 FF 00 FC 00 FE 00 FC 00 FB 00 F9 00 02 01 00 01 FE 00 FF 00 02 01 FD 00 F9 00 F7 00 FF 00 04 01 03 01 05 01 02 01 00 01 FF 00 01 01 FA 00 FE 00 07 01 FD 00 FE 00 FE 00 FF 00 FC 00 FD 00 07 01 FE 00 FF 00 FD 00 FF 00 00 01 FE 00 FB 00 FE 00 FB 00 FF 00 07 01 FD 00 03 01 FF 00 02 01 07 01 00 01 FD 00 03 01 01 01 FA 00 FC 00 01 01 FD 00 F9 00 FD 00 FD 00 01 01 F7 00 FC 00 06 01 00 01 FF 00 00 01 FA 00 F9 00 FF 00 08 01 FA 00 FA 00 FA 00 F5 00 05 01 FD 00 FC 00 01 01 FB 00 F4 00 01 01 FD 00 04 01 FE 00 FF 00 FE 00 FD 00 FE 00 FE 00 01 01 04 01 01 01 03 01 F7 00 00 01 FC 00 04 01 F8 00 F8 00 FB 00 F8 00 05 01 FC 00 03 01 F8 00 03 01 FA 00 FF 00 00 01 04 01 FD 00 FC 00 01 01 00 01 05 01 FB 00 FC 00 01 01 FC 00 FB 00 05 01 FA 00 06 01 01 01 FB 00 01 01 F6 00 01 01 FA 00 FC 00 FB 00 02 01 FA 00 02 01 FA 00 FB 00 F7 00 FB 00 FE 00 FE 00 FB 00 00 01 FF 00 FD 00 F9 00 FE 00 00 01 FD 00 FE 00

[Yann Collet]

Not sure if there is any problem with your methodology (12b->16b).
At least, if you can regenerate the image on the other side of the transmission link, it means your transformation works as intended.

What's immediately apparent in your sample,
is that one byte out of 2 is almost always 0.
"Almost", because sometimes it is 1.

The other byte has a very high value, but it's changing constantly.
So if the original image is black, then it is not "completely black", like some small luminosity variations (probably some noise).

Unfortunately, such data pattern is not suitable for an LZ transformation.
LZ4, or zlib, or 7zip, or whatever LZ-based compressor, will give disappointing results.

For better results, I think you need another algorithm, more "image oriented".


Regards

Le samedi 21 juillet 2012 14:01:16 UTC+2, Frank a écrit :

Yan,

Sorry to ask a specific question, I am using LZ4 to compress a RAW data from a camera. Camera spits out 12 bit data, I make a 16 bit out of it and take 20K of this data and feed to the compressor. Resulting compression is really disappointing. I attach a binary file that contains my images in this case they are fully black (no lens at the camera just cap). 

Interestingly, if I use a 7z to compress, it is also disappointing. I am suspecting I am doing something wrong. Do you mind suggesting what am I doing wrong? Would increasing compression level help or any other trick?

Thanks

F. 

ps. Here is a snippet of the raw image file, I also attached the original file but not so sure it will make it to the group. 

E6 00 EA 00 EE 00 ED 00 EF 00 EE 00 E7 00 E5 00 EC 00 EC 00 E5 00 E7 00 E8 00 F0 00 EF 00 F2 00 EC 00 E1 00 F0 00 E7 00 EB 00 EC 00 F5 00 F5 00 F2 00 EE 00 F3 00 F0 00 F2 00 EF 00 E8 00 EB 00 F1 00 F2 00 F2 00 F1 00 F0 00 F4 00 F5 00 F9 00 F5 00 EF 00 F5 00 F0 00 F2 00 F0 00 EE 00 F1 00 F3 00 F3 00 F5 00 F3 00 EF 00 EC 00 EA 00 F2 00 F4 00 F2 00 EB 00 F3 00 F1 00 F5 00 F2 00 F4 00 EE 00 E6 00 F0 00 F1 00 F2 00 F4 00 ED 00 F6 00 F1 00 EE 00 F3 00 EF 00 F3 00 F4 00 EB 00 EC 00 F5 00 F1 00 EE 00 F3 00 F5 00 F2 00 ED 00 ED 00 F0 00 EE 00 F1 00 EF 00 F2 00 F8 00 F2 00 F5 00 F1 00 F1 00 EE 00 F3 00 F4 00 EF 00 FC 00 EE 00 F2 00 F1 00 EB 00 F4 00 F5 00 F0 00 F1 00 F0 00 EE 00 EF 00 F7 00 F2 00 F9 00 ED 00 F1 00 F3 00 ED 00 F5 00 EF 00 F7 00 F4 00 F8 00 F2 00 F1 00 F3 00 F3 00 F9 00 F2 00 F2 00 EE 00 F5 00 F5 00 F1 00 F3 00 ED 00 FD 00 F6 00 F2 00 F0 00 F9 00 F7 00 F2 00 FC 00 F0 00 F8 00 F1 00 F5 00 F5 00 F6 00 F0 00 F6 00 F5 00 FD 00 FE 00 FC 00 F9 00 F2 00 F5 00 EF 00 FA 00 F6 00 F3 00 F6 00 F3 00 F3 00 FA 00 F8 00 F6 00 EF 00 FA 00 F5 00 F7 00 FC 00 F8 00 FA 00 F8 00 F4 00 F6 00 F4 00 F3 00 F3 00 EF 00 F4 00 F2 00 F3 00 F8 00 FB 00 F0 00 FC 00 F7 00 FA 00 F9 00 F7 00 02 01 FC 00 F1 00 F4 00 F9 00 FF 00 F9 00 03 01 FB 00 F7 00 F4 00 F6 00 F2 00 F8 00 F4 00 F8 00 F5 00 F5 00 F8 00 F2 00 F3 00 FB 00 F1 00 F7 00 F2 00 FB 00 F6 00 F8 00 FA 00 FD 00 F3 00 F5 00 EC 00 F4 00 F6 00 F0 00 F9 00 F7 00 F6 00 FB 00 FB 00 FB 00 F6 00 F0 00 F7 00 F8 00 F4 00 FE 00 F8 00 FC 00 F4 00 FB 00 FA 00 FC 00 FD 00 F6 00 FB 00 F9 00 F4 00 F9 00 F1 00 FA 00 FC 00 F3 00 FB 00 F8 00 F3 00 F7 00 00 01 F4 00 F6 00 FC 00 F4 00 F3 00 F2 00 FA 00 F4 00 F8 00 FC 00 FE 00 00 01 FB 00 FC 00 FA 00 F8 00 F7 00 FB 00 F3 00 FA 00 03 01 00 01 FB 00 F7 00 01 01 F7 00 FF 00 00 01 F6 00 F7 00 F8 00 FE 00 F6 00 01 01 FB 00 FC 00 FB 00 F5 00 F6 00 FA 00 FB 00 FD 00 F8 00 FD 00 F1 00 FD 00 FD 00 F8 00 F9 00 F8 00 00 01 F5 00 FD 00 F9 00 F9 00 F4 00 FF 00 EF 00 FD 00 F5 00 F6 00 FE 00 F7 00 F9 00 FD 00 F9 00 F6 00 F9 00 F5 00 FA 00 FB 00 FC 00 F9 00 F6 00 FB 00 04 01 00 01 F8 00 F8 00 FD 00 F5 00 FD 00 F6 00 F7 00 FD 00 F9 00 FF 00 FA 00 FF 00 F6 00 FE 00 F9 00 FE 00 FB 00 FF 00 FB 00 FC 00 F9 00 FF 00 F2 00 FE 00 FB 00 F9 00 FC 00 F8 00 FC 00 FB 00 FE 00 F3 00 F8 00 FC 00 F9 00 FA 00 00 01 FF 00 F0 00 03 01 FD 00 FA 00 FC 00 FD 00 F8 00 FA 00 FA 00 FC 00 F7 00 FA 00 FB 00 00 01 FB 00 F8 00 FC 00 FD 00 FC 00 FB 00 F5 00 F9 00 F6 00 F7 00 F5 00 F8 00 FA 00 FD 00 F7 00 FD 00 05 01 FF 00 F9 00 05 01 FE 00 F4 00 FF 00 F5 00 FC 00 FF 00 03 01 FE 00 FE 00 FB 00 FA 00 F7 00 FA 00 F6 00 02 01 02 01 F6 00 F2 00 FC 00 FD 00 FC 00 FB 00 F4 00 FE 00 F2 00 FB 00 FC 00 FF 00 01 01 F7 00 F7 00 FB 00 FA 00 FC 00 F8 00 FE 00 FC 00 F5 00 FB 00 F8 00 FC 00 FA 00 FC 00 01 01 FB 00 F5 00 F9 00 02 01 FE 00 F5 00 FC 00 FA 00 00 01 F6 00 FC 00 FE 00 F7 00 FD 00 F5 00 FE 00 FF 00 FF 00 FB 00 01 01 FA 00 FC 00 F8 00 FF 00 FA 00 FD 00 FF 00 F8 00 FB 00 FA 00 01 01 F7 00 FB 00 00 01 00 01 F8 00 F4 00 FD 00 FD 00 03 01 F8 00 FB 00 FF 00 FD 00 FE 00 FF 00 F8 00 FE 00 FB 00 FA 00 F9 00 F4 00 F9 00 00 01 FF 00 FB 00 03 01 F6 00 F8 00 FD 00 00 01 F7 00 F9 00 FD 00 FB 00 FB 00 FA 00 F9 00 FB 00 FE 00 FC 00 FD 00 FD 00 F6 00 FB 00 FE 00 FC 00 F4 00 F2 00 FE 00 F9 00 FC 00 FD 00 03 01 FC 00 FB 00 FE 00 F9 00 F5 00 F3 00 FF 00 04 01 FA 00 FD 00 FE 00 F7 00 06 01 00 01 F4 00 00 01 FF 00 00 01 F5 00 F7 00 00 01 00 01 FC 00 F9 00 FD 00 F9 00 F6 00 FC 00 FE 00 00 01 FA 00 F6 00 FC 00 F8 00 F9 00 FA 00 FD 00 F5 00 F7 00 F5 00 F5 00 FC 00 00 01 FB 00 FB 00 F3 00 01 01 01 01 F6 00 FD 00 F4 00 FF 00 FC 00 F9 00 F2 00 FD 00 F7 00 01 01 F7 00 FC 00 F9 00 FD 00 FC 00 03 01 F7 00 F7 00 FC 00 FC 00 F9 00 01 01 FE 00 F8 00 05 01 FC 00 F8 00 03 01 F7 00 FC 00 FE 00 FF 00 FA 00 F4 00 F7 00 F9 00 FB 00 FF 00 F6 00 F8 00 FA 00 FB 00 FC 00 F9 00 FA 00 F7 00 F7 00 F8 00 01 01 FA 00 FA 00 FB 00 FA 00 FC 00 F6 00 FF 00 FA 00 F3 00 F6 00 FF 00 F7 00 FA 00 F8 00 FF 00 FE 00 F9 00 FB 00 F8 00 FD 00 F7 00 F9 00 FD 00 FF 00 00 01 F4 00 FB 00 F8 00 F5 00 00 01 FD 00 FD 00 05 01 FB 00 F9 00 F6 00 F7 00 FE 00 03 01 FA 00 F5 00 F7 00 02 01 F8 00 00 01 F4 00 03 01 F9 00 01 01 F7 00 F6 00 00 01 F7 00 F8 00 03 01 F1 00 F9 00 F8 00 FB 00 F9 00 FA 00 FE 00 F9 00 F8 00 FB 00 F8 00 FC 00 FB 00 F4 00 F8 00 F7 00 F5 00 FF 00 FB 00 F6 00 F8 00 F7 00 F9 00 FD 00 F9 00 FB 00 F3 00 F8 00 F9 00 F6 00 F7 00 F9 00 F6 00 F3 00 F4 00 F6 00 F7 00 FC 00 FA 00 F8 00 F8 00 F8 00 F3 00 F3 00 F5 00 F9 00 F2 00 FB 00 FD 00 F7 00 F6 00 FD 00 F9 00 F9 00 FB 00 F7 00 F6 00 E6 00 F5 00 EF 00 F0 00 EC 00 EE 00 F2 00 ED 00 EE 00 EC 00 F0 00 EF 00 E7 00 E7 00 EE 00 F3 00 F1 00 F4 00 F2 00 EF 00 E9 00 EF 00 F1 00 EF 00 EA 00 F6 00 F2 00 FD 00 EA 00 F8 00 F4 00 F3 00 EC 00 F6 00 F3 00 F0 00 F4 00 F6 00 F7 00 F6 00 FC 00 F3 00 EF 00 F2 00 F0 00 EF 00 F7 00 E9 00 F2 00 F3 00 F1 00 F3 00 F0 00 F8 00 F5 00 F9 00 F8 00 F8 00 F2 00 F8 00 F5 00 F1 00 EF 00 F6 00 F6 00 F2 00 F0 00 F2 00 F3 00 F4 00 F5 00 F6 00 F7 00 F8 00 F5 00 ED 00 F4 00 F3 00 F3 00 F1 00 F4 00 F6 00 F9 00 F3 00 EE 00 EF 00 FF 00 EE 00 F9 00 F6 00 F5 00 F6 00 F3 00 F2 00 F3 00 F6 00 F7 00 F1 00 F5 00 F8 00 FB 00 F6 00 EF 00 F7 00 F2 00 FA 00 F2 00 F4 00 F6 00 F7 00 FA 00 F7 00 F2 00 F1 00 F5 00 F5 00 F0 00 F5 00 F5 00 F5 00 F9 00 FC 00 FB 00 F8 00 F6 00 F8 00 F7 00 F0 00 FC 00 F5 00 F9 00 F8 00 F8 00 F1 00 FE 00 F5 00 FA 00 F6 00 FA 00 FE 00 F0 00 FB 00 FE 00 FA 00 F2 00 FA 00 FD 00 F4 00 F9 00 FC 00 FB 00 F8 00 F6 00 F5 00 F5 00 FA 00 FB 00 FB 00 FB 00 F8 00 F8 00 FC 00 FC 00 F4 00 F4 00 F1 00 FC 00 F6 00 F3 00 FC 00 F2 00 F3 00 FB 00 F8 00 FD 00 F6 00 F8 00 FC 00 F8 00 01 01 F6 00 FC 00 FA 00 F2 00 F7 00 FD 00 00 01 FB 00 FD 00 F8 00 F8 00 FB 00 F6 00 FF 00 FB 00 F6 00 F2 00 F6 00 FC 00 F8 00 F8 00 F4 00 FA 00 FC 00 F3 00 F9 00 FF 00 F6 00 FC 00 F4 00 FF 00 FD 00 FD 00 F6 00 F3 00 FB 00 FC 00 FD 00 F9 00 00 01 00 01 04 01 F9 00 F8 00 F7 00 FE 00 F9 00 F9 00 FC 00 F7 00 FF 00 FA 00 F8 00 F4 00 FB 00 F5 00 FB 00 FD 00 F7 00 FC 00 01 01 F8 00 FA 00 00 01 FA 00 03 01 FD 00 F9 00 02 01 03 01 FE 00 FE 00 04 01 00 01 F0 00 FC 00 FD 00 F5 00 01 01 FF 00 FA 00 FF 00 00 01 FC 00 FB 00 FB 00 FD 00 FB 00 FB 00 F9 00 FB 00 F4 00 01 01 00 01 00 01 00 01 FC 00 FD 00 04 01 F7 00 FE 00 00 01 FE 00 06 01 F9 00 F9 00 00 01 FE 00 04 01 F8 00 02 01 F4 00 FC 00 FD 00 00 01 FD 00 FD 00 FC 00 F4 00 F7 00 FE 00 FE 00 FB 00 FC 00 04 01 FC 00 FC 00 FB 00 FA 00 FD 00 FC 00 FF 00 F7 00 FF 00 FF 00 F9 00 FB 00 F9 00 FB 00 FC 00 FC 00 FE 00 FA 00 FE 00 FE 00 02 01 02 01 FB 00 FA 00 F8 00 F9 00 FE 00 FD 00 FC 00 FB 00 FE 00 FD 00 FF 00 FB 00 00 01 FC 00 FC 00 FF 00 FD 00 FA 00 01 01 03 01 FE 00 FD 00 FE 00 FF 00 FF 00 03 01 FD 00 FC 00 F9 00 03 01 04 01 FA 00 FC 00 FD 00 FE 00 01 01 FE 00 00 01 F8 00 FE 00 FC 00 FE 00 01 01 01 01 FD 00 F9 00 03 01 01 01 00 01 FC 00 02 01 FF 00 01 01 03 01 05 01 00 01 00 01 FD 00 FD 00 FC 00 FB 00 FC 00 FC 00 FC 00 02 01 06 01 06 01 FE 00 08 01 FF 00 06 01 FB 00 F8 00 FD 00 01 01 F9 00 00 01 01 01 FC 00 0A 01 FE 00 06 01 03 01 FD 00 01 01 FF 00 F3 00 F9 00 FE 00 FD 00 00 01 FA 00 05 01 FC 00 F7 00 FD 00 01 01 FF 00 01 01 02 01 00 01 FC 00 00 01 04 01 05 01 F5 00 FC 00 08 01 00 01 FF 00 FF 00 FA 00 F7 00 F9 00 FD 00 FD 00 08 01 FA 00 FA 00 FF 00 F9 00 02 01 06 01 FC 00 03 01 FD 00 FF 00 FC 00 01 01 00 01 00 01 F9 00 FA 00 05 01 FD 00 01 01 FE 00 FF 00 FE 00 FF 00 00 01 FD 00 FF 00 03 01 03 01 FD 00 F5 00 FA 00 F9 00 F8 00 F9 00 F8 00 FE 00 01 01 02 01 04 01 FB 00 03 01 FD 00 00 01 01 01 FD 00 FC 00 02 01 01 01 06 01 F9 00 F8 00 FA 00 01 01 01 01 00 01 01 01 FF 00 FF 00 05 01 FF 00 FE 00 01 01 02 01 07 01 FA 00 FF 00 FA 00 03 01 F4 00 04 01 FD 00 03 01 FA 00 02 01 08 01 FA 00 FE 00 FF 00 00 01 FF 00 04 01 FF 00 FE 00 02 01 01 01 FA 00 FD 00 FE 00 02 01 F6 00 01 01 FC 00 FC 00 FC 00 F6 00 FC 00 FE 00 07 01 00 01 FE 00 FD 00 01 01 FE 00 FD 00 04 01 02 01 FB 00 FC 00 FD 00 FB 00 02 01 0A 01 F8 00 F4 00 FD 00 FD 00 00 01 FA 00 02 01 FF 00 FC 00 FE 00 FC 00 FB 00 F9 00 02 01 00 01 FE 00 FF 00 02 01 FD 00 F9 00 F7 00 FF 00 04 01 03 01 05 01 02 01 00 01 FF 00 01 01 FA 00 FE 00 07 01 FD 00 FE 00 FE 00 FF 00 FC 00 FD 00 07 01 FE 00 FF 00 FD 00 FF 00 00 01 FE 00 FB 00 FE 00 FB 00 FF 00 07 01 FD 00 03 01 FF 00 02 01 07 01 00 01 FD 00 03 01 01 01 FA 00 FC 00 01 01 FD 00 F9 00 FD 00 FD 00 01 01 F7 00 FC 00 06 01 00 01 FF 00 00 01 FA 00 F9 00 FF 00 08 01 FA 00 FA 00 FA 00 F5 00 05 01 FD 00 FC 00 01 01 FB 00 F4 00 01 01 FD 00 04 01 FE 00 FF 00 FE 00 FD 00 FE 00 FE 00 01 01 04 01 01 01 03 01 F7 00 00 01 FC 00 04 01 F8 00 F8 00 FB 00 F8 00 05 01 FC 00 03 01 F8 00 03 01 FA 00 FF 00 00 01 04 01 FD 00 FC 00 01 01 00 01 05 01 FB 00 FC 00 01 01 FC 00 FB 00 05 01 FA 00 06 01 01 01 FB 00 01 01 F6 00 01 01 FA 00 FC 00 FB 00 02 01 FA 00 02 01 FA 00 FB 00 F7 00 FB 00 FE 00 FE 00 FB 00 00 01 FF 00 FD 00 F9 00 FE 00 00 01 FD 00 FE 00