LZTURBO 0.91 Parallel Compressor (Win32/Linux)

[Black_Fox]

I think there's some suboptimality in LZTurbo's optimal parsing. I have a tar archive, which is very compressible (7z compresses it from 63 MB to 426 kB)... it took 3 hours to compress with PAQ8o9 (using one core) and 2 hours for latest LZTurbo using both cores...

[Bulat Ziganshin]

i'm pretty sure that reason is the naive binary tree implementation which makes lzturbo extremely slow on any binary data. try cabarc on this file - it was also naive :) otoh, Hamid optimizes lzt for enwik9, so i don't expect that he will make any changes for other datatypes

[Black_Fox]

Good idea, but that's not it - cabarc finished in about a minute. Maybe the difference comes from dictionary size.

I made just a simple test - some results are here and the 7zipped file is here
saves.tar is original file, saves.rep is saves.tar processed with rep

[Bulat Ziganshin]

well, another idea: lzt uses hc or ht with a very deep search (say, 1000 elements) what makes it very slow when there are lot of 4-byte repetitions

[encode]

Quoting: Bulat Ziganshin

hc or ht with a very deep search (say, 1000 elements)

1000 is not so deep. All versions of BALZ uses HC with 4096 elements... ;)