Copying large dataset, verify errors

[Mark Smith]

Hello!  So here's a strange one.  I just copied about 16TB of data from one RAID array to another using RapidCopy on my Mac.  During the verify phase, I received several errors like this:

Verify Error detected. srcdigest=6abb3b7b2e9627c7 dstdigest=be9bdc5c04195d8c
Check your system environment. Change Max I/O size set to 1.

Only thing is, I don't see where I can change the max I/O size in Settings.  I checked the files, and the date / filesize are identical, but it's true that the hashes do NOT match, which is troubling.

This is all I see in the I/O settings though:

[Kengo Sawatsu]

Hi Mark

Thank you for your question.
The IO size is currently not adjustable.
Since I am displaying an inappropriate message, the message content will be corrected in the next version.

Regards.

2023年9月12日火曜日 12:42:44 UTC+9 akra...@gmail.com:

[Mark Steven Smith, Jr.]

Hi Kengo,

Is there anything else to be done about the invalid data?  I ran another sync, but it reports everything copied.  If I run a verify, however, I get a long list of errors.  Is there a way to force it to copy anything that doesn't match the hash?

Thanks,

Mark

--
You received this message because you are subscribed to a topic in the Google Groups "RapidCopy(Pro) BBS" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rapidcopy_en/KRMKqqXaRDc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rapidcopy_en...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rapidcopy_en/a490aab9-e6f1-4786-af1d-35760458c532n%40googlegroups.com.

[Kengo Sawatsu]

Hi mark

Unfortunately there is no way to copy only the data that does not match the validation.
The only option is to copy each path displayed in the standard log or detailed log one by one.

If there are only a few errors, identify the file paths one by one and rerun the verify copy with overwrite.
If there are a large number of errors, we recommend re-running the verify copy in overwrite mode.
It takes a long time, but there are no mistakes.

However, if a large number of verification errors occur, there is already a problem with the copy source or destination disk device.
Also keep in mind that no matter how many times you copy something, there's a good chance it won't work.

RapidCopy is powerless against internal errors due to sector level errors or file system corruption.

RapidCopy can only be checked at the application level(layer).

I also notice that the behavior when a verification error occurs is unfriendly.
The original FastCopy has a setting to delete error files when a verification error occurs, so I plan to port this.

Regards.

2023年9月12日(火) 13:16 Mark Steven Smith, Jr. <marks...@smithjr.cc>:

--

----

Kengo Sawatsu(澤津健吾)
R&D / Software Engineer

saw...@lespace.co.jp
cell 080-2231-1423

Lespace Vision Co.ltd
〒151-0051 東京都渋谷区千駄ヶ谷4-26-15 秀和代々木駅前ビル4F
Tel 03-3478-0521
Fax 03-3478-0696

http://www.lespace.co.jp

[Mark Smith]

I thought I had replied already but I can't find my last message.  Just in case, I'll repeat what I said earlier.

I ran an "Rclone sync -c --links" to verify the checksums of every single file on both volumes, copying symlinks as symlinks.  That worked.  Since it's about 16TB of data, it took two days.  I then ran it again, just to be doubly sure - and it completed in another two days without copying anything the second time (as would be expected).

I realize RapidCopy wouldn't be able to do anything about hardware issues.  But I think this might be a bug.  I'm copying from a RAID-5 volume to a RAID-6 volume.  I've checked each drive's SMART data independently.  There are no errors, no reallocated sectors, etc.  Everything checks out.  

I checked filenames, and didn't see a pattern - I don't think it has to do with special symbols in the filename, path length, etc.  I also tried forcing a copy of a specific folder (that originally failed) over top of the original (Copy - Overwrite All) with verification on, and it seems to have copied just fine according to both RapidCopy and manually checking the hashes.  So I can't reproduce it without doing a full copy again.

This appears to only happen with large datasets (this one is 16TB).  I will say there are also a LOT of files, varying from small music files, to large video files.  It could be the sheer number of files, or the sheer amount of data.  I'm uncertain.  I can't get it to repeat without doing a whole copy all over again, and unfortunately I don't have another spare volume to test with right now.

If there's anything I can do to help, please let me know.  And I apologize for not getting back to you in my other thread - I just realized that I didn't respond.  Unfortunately I no longer have that dataset to test with, but you'll be happy to know it's been a long time since RapidCopy crashed on me.  I appreciate you and RapidCopy very much!  I have used it, and FastCopy, for many, many years.  :)

[Kengo Sawatsu]

Hi Mark,

Thank you for your detailed report and opinion.
I need detailed logs so that I can understand your environment and file system information.

The explanation is below.
https://app.lespace.co.jp/file_bl/rapidcopy/manual/index_en.html#Log

If you send me a standard log file, preferably a verbose log file, I might be able to improve the problem.
However, I will be on vacation from now on, so it will probably take more than a week for me to reply to you.

Thank you for your patience and love for RapidCopy.
I plan to start working on improving RapidCopy's shortcomings after my vacation.

Regards.

2023年9月15日(金) 23:54 Mark Smith <akra...@gmail.com>:

You received this message because you are subscribed to the Google Groups "RapidCopy(Pro) BBS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rapidcopy_en...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rapidcopy_en/1dbd2571-2208-4dc1-acef-1ba9876ec12an%40googlegroups.com.