Mountainduck shortcomings (S3/minIO)

[Gavin Greenwalt]

I'm testing out solutions for our TrueNAS server and their S3 service (minIO) seems promising as a data back for WAN transfers. (We previously were using Synology Drive which is amazingly good for a free product). 

Cyberduck mostly works great.   The only problem I have with it is that when a file transfer hits 100% it just sits there for a long time.  I have checksums on Downloads and Uploads disabled in the settings, so I don't know what's happening.  But the speed is incredible.  I can see 3-4gbps which is far more than I ever expected or even need over our gigabit fiber.  The long pauses though on large uploads is a little annoying though.  Is that a minIO issue or is that a Cyberduck issue? 

The real problems though start with Mountainduck.  Does Mountainduck not smartly do multipart uploads over S3?   I'm getting 30MB/s which is literally 10x slower than Cyberduck.  It's also taking FOREVER to start like 50GB uploads during the "Creating Checksums process" Is there not a way to disable checksums in Mountainduck like in Cyberduck?   Or is Cyberduck still creating Checksums anyway and that's the long pause at 100%?  The pauses on Cyberduck are faster though than Mountainduck which is unacceptably slow. 

I tried out Expandrive and I hate its interface with the passion of 1,000 suns... but I get multi-gig uploads so it's definitely using multipart.  And it's also uploading immediately instead of pausing for a long time to generate checksums.  I would much prefer to deploy Mountainduck licenses to my coworkers and clients but only if the performance actually works.   I'm hoping there are some undocumented flags I can add and include in Bookmarks to distribute that make it far more performant and fully saturate our WAN. 

On a side note, Expansdrive also has a neat feature where you can mount S3 directly as an SMB share in Windows running as a service.   Is there a Mountainduck system level service for Servers and since it's mounting the s3 storage via the UNC path as a Network Drive, is it possible to similarly match Expansdrive Server Edition's ability to expose that UNC path to \\server\share to other clients as well?   I'd love to set aside say 4TB of NVME storage on a server, and then have it intelligently cache an S3 store over the Windows SMB service to other LAN clients as well. 

Thanks,
Gavin

[David Kocher]

On 27 Jan 2023, at 22:45, Gavin Greenwalt <im.tha...@gmail.com> wrote:
>
> I'm testing out solutions for our TrueNAS server and their S3 service (minIO) seems promising as a data back for WAN transfers. (We previously were using Synology Drive which is amazingly good for a free product).
>
> Cyberduck mostly works great. The only problem I have with it is that when a file transfer hits 100% it just sits there for a long time. I have checksums on Downloads and Uploads disabled in the settings, so I don't know what's happening. But the speed is incredible. I can see 3-4gbps which is far more than I ever expected or even need over our gigabit fiber. The long pauses though on large uploads is a little annoying though. Is that a minIO issue or is that a Cyberduck issue?

For Downloads, the issue could be the segmented file transfer option. See https://github.com/iterate-ch/cyberduck/issues/13610. For uploads this could be Minio requiring some time when completing the multipart upload. One would need to have at the HTTP log, see https://docs.cyberduck.io/cyberduck/support/#logging-output

> The real problems though start with Mountainduck. Does Mountainduck not smartly do multipart uploads over S3?

Mountain Duck uses multipart uploads for S3.

> I'm getting 30MB/s which is literally 10x slower than Cyberduck. It's also taking FOREVER to start like 50GB uploads during the "Creating Checksums process" Is there not a way to disable checksums in Mountainduck like in Cyberduck? Or is Cyberduck still creating Checksums anyway and that's the long pause at 100%? The pauses on Cyberduck are faster though than Mountainduck which is unacceptably slow.

If you connect in „Smart Synchronization“ mode, there should be no notable difference in transfer speeds.

>
> I tried out Expandrive and I hate its interface with the passion of 1,000 suns... but I get multi-gig uploads so it's definitely using multipart. And it's also uploading immediately instead of pausing for a long time to generate checksums. I would much prefer to deploy Mountainduck licenses to my coworkers and clients but only if the performance actually works. I'm hoping there are some undocumented flags I can add and include in Bookmarks to distribute that make it far more performant and fully saturate our WAN.

Mountain Duck does compute a checksum over the entire file prior upload. This cannot currently be disabled. We have opened an internal ticket to discuss possible remedies for this use case.

>
> On a side note, Expansdrive also has a neat feature where you can mount S3 directly as an SMB share in Windows running as a service. Is there a Mountainduck system level service for Servers and since it's mounting the s3 storage via the UNC path as a Network Drive, is it possible to similarly match Expansdrive Server Edition's ability to expose that UNC path to \\server\share to other clients as well? I'd love to set aside say 4TB of NVME storage on a server, and then have it intelligently cache an S3 store over the Windows SMB service to other LAN clients as well.

We are looking into the options we have to support running Mountain Duck as a service.

>
> Thanks,
> Gavin