Parallel archiver and scalability...

[aminer]

Hello,

I think i am happy now, please read again...

I have benchmarked parallel archiver using parallel LZMA using
5 threads on a quad core, so this have giving false results on the
timing...

So i have started parallel archiver with a single thread and this
has giving a more accurate results, here is my correction please read
again...

I have come to an interresting subject, so be smart and follow with me
please...

I have tried to do a worst scalability prediction with an HDD hardisk
for my parallel archiver(you will find my parallel archiver here:
http://pages.videotron.com/aminer/) with Parallel LZMA, and i think
it's worst than what i have thought..

there is four things in my Parallel LZMA algorithm:

First we have to copy serially a stream from the hardisk to the memory
and this will take in average 0.2 second and in the compression method
we have to copy a stream to the memory and this will take in average
0.05 second and in the compression method you have to compress a stream
to another stream in memory and this will take in average 13 seconds seconds
and in the compression method you have to copy a compressed stream to a
hardisk file and this will take in average 0.01 second.

So we have the serial part that is: 0.2 second + 0.01 second + 0.05
second = 0.26 second = 0.02%
and the parallel part will that is: 13 seconds = 0.98%


So the worst case scalability scenario using an HDD and using the Amdahl
equation will
give us: 1/0.02% + (0.98%/N) = 50X scalability (N: is the number of cores)

So this will scale up to: 50X , so as you have noticed with an HDD
hardisk this is a good scalability.

So what can we do to scale more parallel archiver using parallel LZMA ?

You can for example use a RAID 10 with a base configuration of 4 HDD
hardrives,
so this will cut in 4 the 0.2 second and the 0.01 second , so this will
give a scalability of 124X
and this is better.. but to speed more the things we can use SSD
hardrives that are 2X time faster than a HDD hardrives and
with a RAID 10 configuration and this will give: 434X worst case
scalability.

So as you have noticed if you are using only an HDD
with a multicore system you will get a 50X scalability with my
parallel archiver using parallel LZMA, and if you use RAID 10 with SSD
drives you will get 434X scalability.


And this is why i have talked about RAID 10 etc.



Thank you,
Amine Moulay Ramdane.

[aminer]

Hello again,

I have done those benchmarks with a Q6600 Intel quad core.