Very slow rclone from DNS323 - can it be improved?

[Danny French]

First up, thank you Mr Cardoso for your miraculous work on this old little box... ALT-F is just fantastic and it's entirely down to you that I've managed to keep this box going for so many years.

To my problem - which I suspect is entirely attributable to the DNS323 just being slow and old, but I just thought I'd check first.

I am running my DNS323 as RAID1 but would like to back up to my unlimited Google Drive too. I installed rclone from Endware-ng and did some test transfers - I can confirm that they do work, which seems almost miraculous in itself, but torturously slow, even slower than I would have expected.

I'm getting about 12-16MB/s from the box to Windows, and about 10-12MB/s from Windows to the box - which I accept is as good as we can expect from this hardware.

However, my rclone jobs to Google Drive are topping out at 320-380kb/s - and while they're running the CPU is at 99% and the fans sound like a Concorde engine. (I should mention I'm on 1gb FTTP and these weren't tiny files - I tried to transfer a single file of 2gb.)

That 2gb file took almost three hours... the 323 has about 4tb data... so, yeah, but no...

Would this be what you would expect?

Anything you can think of that I could do, or is this an ask too far for the venerable D-Link?

Cheers and thank you!

D

[Joao Cardoso]

On Wednesday, June 3, 2020 at 8:29:57 PM UTC+1, Danny French wrote:

First up, thank you Mr Cardoso for your miraculous work on this old little box... ALT-F is just fantastic and it's entirely down to you that I've managed to keep this box going for so many years.

To my problem - which I suspect is entirely attributable to the DNS323 just being slow and old, but I just thought I'd check first.

I am running my DNS323 as RAID1 but would like to back up to my unlimited Google Drive too. I installed rclone from Endware-ng and did some test transfers - I can confirm that they do work, which seems almost miraculous in itself, but torturously slow, even slower than I would have expected.

Are you using encryption? That will slow down things.

To see what the problem is you might try to use an ftp remote without encryption, and not using ftps, using a local ftp server on a PC. If that is also slow, there is no hope.

 

I'm getting about 12-16MB/s from the box to Windows, and about 10-12MB/s from Windows to the box - which I accept is as good as we can expect from this hardware.

Those are the figures I also get. Plain ftp is slightly faster.

But if you try using ssh, you will see 1-2 MB/s. https servers top at about 3MB/s. Encryption with strong ciphers on a non crypto-enable cpu is hopeless. I remember using arc4 as the cipher with ssh, and it was several times faster then standard aes, but arc4 is now considered weak and retired.

 

However, my rclone jobs to Google Drive are topping out at 320-380kb/s - and while they're running the CPU is at 99% and the fans sound like a Concorde engine. (I should mention I'm on 1gb FTTP and these weren't tiny files - I tried to transfer a single file of 2gb.)

That 2gb file took almost three hours... the 323 has about 4tb data... so, yeah, but no...

yeah, no.

I also tried duplicity, disabling compression, and it was somewhat faster, but yet slow. However duplicity from Alt-F is currently "broken". Perhaps installing it from Entware-ng, but don't have many hope. Duplicity allows disabling encryption and compression.

Would this be what you would expect?

yes, sorry

[Danny French]

Thanks, Joao!

On Thursday, 4 June 2020 03:49:04 UTC+1, Joao Cardoso wrote:

Are you using encryption? That will slow down things.

No crypto, I wouldn't even dare. ;)

 

To see what the problem is you might try to use an ftp remote without encryption, and not using ftps, using a local ftp server on a PC. If that is also slow, there is no hope.

Innnteresting... DNS323 -> Local FTP on another PC... ~~~14MB/s - the same 2gb file transferred in a little over two minutes.

What a pain that you can't talk to them with FTP.

I'm not sure that I have the willpower to investigate this any further. If the backup goes at 300kb/s I'm looking at ~six months to backup the drive, and it would probably have melted within a week anyway. Even if I could push it up to say, 1MB/s I'm still looking at a month of running the DNS323 at 100% CPU... and that's not even counting new data going on the drive. Not practical.

This box has been running continuously since Christmas 2006. It's served its time. Time for it to retire gracefully.

Thanks again, for the software and for your advice here.

Be well :)
D

[João Cardoso]

On Thursday, 4 June 2020 18:35:04 UTC+1, Danny French wrote:

Thanks, Joao!

On Thursday, 4 June 2020 03:49:04 UTC+1, Joao Cardoso wrote:

Are you using encryption? That will slow down things.

No crypto, I wouldn't even dare. ;)

 

To see what the problem is you might try to use an ftp remote without encryption, and not using ftps, using a local ftp server on a PC. If that is also slow, there is no hope.

Innnteresting... DNS323 -> Local FTP on another PC... ~~~14MB/s - the same 2gb file transferred in a little over two minutes.

What a pain that you can't talk to them with FTP.

Google Drive implements rate limiting, at least on free accounts. It would be an interesting experiement to build a rclone partial but big 'remote' using a local PC and ftp, and try to rclone it to Google, to see how better performance is. But you don't have the willpower to do that :-)

I'm not sure that I have the willpower to investigate this any further. If the backup goes at 300kb/s I'm looking at ~six months to backup the drive, and it would probably have melted within a week anyway. Even if I could push it up to say, 1MB/s I'm still looking at a month of running the DNS323 at 100% CPU... and that's not even counting new data going on the drive. Not practical.

This box has been running continuously since Christmas 2006. It's served its time. Time for it to retire gracefully.

I hope to made some gardening or bricolage when that time comes :-) ... not a reason to bury the box or put it in a nursing home :-) You can still use it as a secondary weekly backup in your garage, or in a trusted neighbor or family member.

[Danny French]

Dear Joao,

Thank you - maybe I'll revisit this at some point just to satisfy my curiosity... but in the mean time I came to a solution which is a bit dirty but I learned a lot about Linux (I'm a Windows guy) was pretty educational getting it set up so it was worthwhile.

I have a RPI I'm already using as a torrent box, so I installed rclone on that, mounted the SMB share on the DNS323 and ran the job on the RPI - with crypto it did ~14MB/s without really breaking into a sweat so I got the whole backup done in five days.

It's a bit of a bodge but it works, I did it without having to install any new equipment, and the 323 now lives to see another day, which surprisingly I find quite heartwarming.

Cheers again

D

[danny...@gmail.com]

Final followup for now... I managed to optimise the rclone speed directly from the DNS323 edging up close to 600kBytes/s...

...though it does fluctuate - by increasing the drive chunk size from 8mb to 32mb - going any higher is impossible due to the 323 only having 64mb of RAM, but upgrading the RAM and pushing the chunk size up to 64mb or even 96mb may help a little? I dunno... The CPU is still getting hammered, the fan is going like a jet engine, and swap kicks in too. Take the chunk size too high, and you're out of RAM so the transfer goes slowly. Take it too low and the CPU can't provide the chunks quickly  enough so the transfer goes slowly. Perhaps there's no way to win. It's fun to test the limits of this little box but in the end, there's a limit on what can be done and this might just be it.

Thanks all :)
D