copy using multiple threads/queue depth

[jta...@gmail.com]

HDD/SSD nowadays are very fast, same with CPU. can Fast Copy have option to copy at higher than 1 Queue Depth for both read/write.

I copy lots of small files use fastcopy, ~70seconds, I use Richcopy a very old tool and copy at 4-6 threads/queue depth, it takes 18seconds.

pls add this as a new feature.

[Hiroaki SHIROUZU]

Did you clear filesystem(meta-data and data) cache before starting to test?

If you cleared, I will know more details.

FastCopy uses these threads.

 1. Main Thread (for GUI)

 2. Read Thread (for Read I/O and read source dir-entry)

 3. Write Thread (for Write I/O and read dest dir-entry)

 4. Calcurate Read Data hash(md5 or etc) for verify

 5. Calcurate Writed Data hash(md5 or etc) for verify

I think I/O threads(2. or 3.) must be processed sequentially, because parallel Read(or paralle write) for a HDD occurs many seek.

What kind of processing do you want to parallel through additional threads?

If it has very good effect for performance, I will consider to develop it.

[jta...@gmail.com]

thank you Hiroaki Shirouzu for replying.

yes I have checked with a CPU meter and seen the thread process you mentioned, only 2 threads are being used read/write under diff HDD mode.

I wish for multiple  

2. Read Thread (for Read I/O and read source dir-entry)

3. Write Thread (for Write I/O and read dest dir-entry)

heres some screenshot of fastcopy and robocopy thread usage, settings and speed, robocopy with multiple threads finish in 11 seconds.. fast copy is 51 seconds. from same drive source/destination and same files.

[Hiroaki SHIROUZU]

Your FastCopy v2.11 is very old, lastest version is v3.41.

I recommend to use lastest version.

But it is not the important point.

I tried some test, and almost result indicates FastCopy is 20%～50% faster than RichCopy v4.0. (I expand I/O thread to 6 in RichCopy, my CPU is 6core)

Only "4GB file test", FastCopy a little faster. but the result time of RichCopy is not real copy time.

This doesn't contain flush to device time, because RichCopy uses OS cache. (FastCopy uses Direct I/O, so FastCopy passes OS cache)

You can confirm it in this image. (But this image is XP explorer)

https://ipmsg.org/tools/fc_perfmon.png

In an easy way, you can confirm HDDs usage in performance tab in task manager.

Anyway, I tried test patterns are in the following and I tested 1-4 * a-d matrix.

Test file patterns (Your test file size average seems 1-2MB)

1. 1023KB files and total-size 4GB (To reduce 1KB means ... FastCopy's best pattern is 1024KB, and 1023KB brings ftruncating overhead to FastCopy)

2. 2048KB files and total-size 4GB

3. 4GB files

4. cygwin dir (5000files, 141MB)

Device patterns

a. HDD to HDDb. HDD to SSD

c. SSD to HDD

d. SSD to SSD

All tests indecates FastCopy is faster than RichCopy.

So, I can't believe your test results.

Did you clear OS cache, before starting test each times?

And I recommend to watch HDD usage even if RichCopy says "Finish". (Perhaps OS still continue to write to HDD for flushing OS cache)

I don't know why you think to increase I/O threads(more than the number of devices) affect good performance.

I think it brings many seek in a HDD, and bad effect for performance. 

If you know the principle/mechanism (have you learnt computer science?), please explain.

[jta...@gmail.com]

yes you are correct. RichCopy doesn't bypass OS cache (memory) but it only happens when device (HDD) is not fast enough to take in data.  when device is fast enough, OS cache clear instantly (at most only 1-2 seconds after copy finish, such as high end expensive PCIE SSD).  HDD seek time with multiple IO is very slow but SSD is very fast.

Please check screenshot, ATTO can not test QD of 1, so I test overlap I/O 2 threads vs 6 threads, all sequential read/write. (all tests not affected by OS cache because memory usage does not change, you can also download the test and try), also check CDM.

as you can see below, higher multi threads I/O is much faster than single thread I/O.  could you please add multiple I/O threads for both read and write?  if you add the option, default can be 1 thread for read/write different HDD mode, but users with only fast SSD can choose their settings to use maybe 4 thread for read/write different HDD mode.

 

[Hiroaki SHIROUZU]

You must understand "multi-threads is not queue-depth".

Since 2015, FastCopy v3.0 has been supported "Overlapped I/O".

I think "queue-depth" that you say means Overlapped I/O, but the formal name in Win32 API is Overlapped I/O.

https://msdn.microsoft.com/ja-jp/library/windows/desktop/ms686358(v=vs.85).aspx

You can specify the number of Overlapped I/O in FastCopy v3.x.

https://ipmsg.org/tools/fastcopy.html.en#settings

And to increase threads for I/O is meaningless.

[jta...@gmail.com]

ok I will try it out and let you know details, thank you!

[jta...@gmail.com]

hey Hiroaki, sorry got back so late.

so i look around Disk meter monitoring and you're correct, there is OS cache flushing for both richcopy and fastcopy, in fastcopy I can drastically reduce it by changing to 1KB max size for NTFS and have most of transferred files bigger than 1KB will not use OS cache at all, very good!

I just came across today in Richcopy guide they also can adjust cache size per thread which I didn't notice it before as it was a hidden setting and I wish to make a few changes and test and compare.  my Disk meter only shows graph so not very good then I saw one of your reply earlier "https://ipmsg.org/tools/fc_perfmon.png"

Do you know where I can download this monitor as it looks like have a lot of information and I wish to monitor difference between fastcopy and richcopy at different settings and get back to you with more info again.

thanks for your help!

[Hiroaki SHIROUZU]

perfmon.exe is pre-installed application in windows.