Problem with the speed when the copy verification is enabled

[marc_al]

Hello,

I have a big speed problem when the option speed validation is enabled
and I copy a file from either an archive (zip) or a FTP server.
Without the option, the copy is done with a speed around 40 MB / sec.
When I do it from a ftp file (the server is local on the computer
doing the copy) = 40 MB / sec without the option, 500-600K / sec with
the option (§the same values apply with a file inside a zip archive.

Can you please confirm?
Thank you
Marc

[arbiter]

When verification enabled, Nomad does not use write caching to ensure that file is completely written to disk before verification. So yes, copying will be much slower with this option.

--
Liberavi animam meam!
arbiter

[marc_al]

Hello,

In this case, is it possible to choose to enable the verification only
for a copy and not for a copy from a FTP / archive file?
Because going from 40 MB/sec to 500 K/sec (80 times slower), It will
be a lot faster to copy (extract) twice (or even 3 times) the file
and after that compare the content :)

Thank you
Marc

On 4 jan, 16:54, arbiter <arbi...@nomad-net.info> wrote:
> When verification enabled, Nomad does not use write caching to ensure that
> file is completely written to disk before verification. So yes, copying
> will be much slower with this option.
>

[arbiter]

Of course it will be faster, but when caching enabled, there is no guarantee that at comparison time, data will not came from cache. Main slowdown here is not extracting, but disabled write cache (especially when you copying many small files) and, actually, I didn't see a way to improve speed without compromising validity check.

But you can disable (or enable) verification right inside copy dialog, so I think it is acceptable. This feature is not intended to be always enabled, only on some special cases.

[marc_al]

Hello,

I have seen on Ms web site that there is a possibility to flush the
cache with the streams
http://msdn.microsoft.com/en-us/library/kztecsys.aspx

In .NET4 there is also a flush method in the stream to force it to
flush the cache. Perhaps it would be interesting to do a test with one
of the methods?
(just to have an idea about the speed).

Thank you
Marc

[arbiter]

You confusing different levels of cache. There is:

1. .net write buffer

2. os file level write buffer (cache)

3. os volume level write buffer (cache)

4. hardware write buffer (cache) in the hdd

The sample you give, cover only first two levels of cache. To be completely sure that file is written before validation all 4 level must be covered, and there is only one way to ensure this - file must be opened for writing with FILE_FLAG_WRITE_THROUGH and FILE_FLAG_NO_BUFFERING.

I can say even more, right now Nomad does not using FILE_FLAG_NO_BUFFERING flag now, and so, it is possible that validation check can be false positive, because if  FILE_FLAG_WRITE_THROUGH flag is used, data is actually written to disk, but it is also written to volume cache, so it is possible, that a little later, when file is opened for checking, data from cache will be used. This will be changed in the next version of Nomad, it will require some additional coding, but I'll add support for FILE_FLAG_NO_BUFFERING flag too.

As you see, validation check is exact performance - integrity trade, and there is no way to speed up validation and be sure that validation is correct as much as possible at the same time. But as I said earlier in most of use-cases and situations you will not need this validation.

[marc_al]

Hello,

Thank you I didn't know that the os volume was not covered.

Marc