New copy mode suggestion

[Soarer]

Hello,

Is it possible to allow for a new HDD copy mode that will fill the input buffer before starting to write to the destination, then begin write but also continue filling the buffer?

This would be different to Same HDD mode as the buffer would continue to fill during write, and not stop until write has been finished.

My reason is that I have a lot of RAM, a very fast destination (LTO tape) but sometimes my source HDD cannot keep up. If I can somehow have FastCopy fill the buffer first (eg. 50GB) then start writing to tape and continue to fill the buffer once writing starts that would be perfect.

[SHIROUZU Hiroaki]

If it takes a few hours to write, pre-filled buffer will be exhausted in few minutes? (It seems meaningless)

2021年6月18日金曜日 18:56:20 UTC+9 Soarer:

[Soarer]

The writes usually take about 1 hour for a full write, but if I can buffer 50GB before beginning to write, the tape doesn't stop and start every few minutes and wear the tape out prematurely.

50GB of buffer is enough to sustain over the entire write if it keeps reading/filling buffer during tape write. I have tested successfully with SSD as the source, but this is not available on the server with tape.

It's not easy to describe the situation in words, sorry... I wish I could be clearer with my intentions but I believe that filling the buffer first, then writing to tape and continuing to fill the buffer would work for this situation.

[SHIROUZU Hiroaki]

Could you explain this?

 - source device speed

 - dest device speed

 - total write size

2021年6月21日月曜日 21:50:17 UTC+9 Soarer:

[Soarer]

Sure thing:

- Source device speed: RAID0 w. 2x HDDs... approx 300-350MB/sec

- Destination: LTO8 tape - 300-350MB/sec

- Total write size: approx. 1TB

If I can give the buffer a head start and fill before write, I can avoid the buffer becoming empty during write and prevent shoe-shining the tape.

Thank you!

[Soarer]

I don't even mind if it's a command line only option. No need to modify UI for very small use case like this.

[SHIROUZU Hiroaki]

OK, I understood what you wrote.

But I think the best solution is improve source device speed in this case.

2021年6月25日金曜日 22:18:05 UTC+9 Soarer:

[Kengo Sawatsu]

Hi Soarer

I am running LTFS in my company.
I have developped LTFS mount tool on macOS.

If your LTO is using LTFS, then I think your request is make no sense:(

LTFS on Windows commits the tape once every few minutes to prevent data loss due to power failure.
At this time, the tape is stopped and rewind, so the data transfer is always interrupted.
Therefore, there is no point in Fastcopy preparing a large 50GB buffer.

But,

If your environment is "tar" or Symantec products, it would be a different story.

Translated with www.DeepL.com/Translator (free version)https://github.com/KengoSawa2

https://github.com/KengoSawa2

2021年6月25日金曜日 22:18:05 UTC+9 Soarer:

Sure thing: