Copying a lot of files slows down the transfer after while

[Jarek Pe]

Hi,

I have DNS-323 on Alt-F 1.0. I have recently mounted two new 2TB hard drives set to RAID 1. And now I'm finishing data migration process. As I'm copying a lot of files I have discovered strange thing. Conditions:

1. DNS-323 samba share.

2. Windows 10 and mounted mentioned share.

3. Connected through WiFi (but on cable having the same issue).

4. Copying thousands of files. Mostly JPGs and RAWs but also some video files.

5. I'm using total commander for copying from my laptop to DNS-323 NAS.

When I'm starting the copying process I am having transfer about 8 MB/s. Doesn't matter if it's small, few MB file or hundreds one. Transfer systematically drops down. After the hour I have about 1-2 MB/s. When I cancel copying process and start it over it's still on 1-2 MB/s level. But when I reboot DNS-323 and start again copying the same group of files transfer is again about 8 MB/s and slowly drops down. Why is it happening? It looks like some buffers ends or something. 

I think it's not network issue. I have switched my WiFi on unused channel some time ago to ensure clear signal. Signal strength is maximum. NAS is connected to router via ethernet cable. But as I said on when I connect laptop via Ethernet directly to router to eliminate Wifi, there is the same problem.

--

Regards,

Jarek

[Jarek Pe]

I thing I have narrow the issue. After restarting samba service, instead of entire NAS, transfer is on about 8 MB/s level and is continuously dropping down.

[Joao Cardoso]

On Thursday, February 13, 2020 at 12:54:19 AM UTC, Jarek Pe wrote:

I thing I have narrow the issue. After restarting samba service, instead of entire NAS, transfer is on about 8 MB/s level and is continuously dropping down.

It would be interesting to know if Entware samba (install samba36-server 3.6.25-9 after installing Entware itself, Packages->Entware-ng) has the same issues in the same exact circumstances.

Both Alt-F and Entware samba have the same versions and can coexist, although only one can be active at a time. From the command line, Alt-F samba binaries have precedence over Enwtare binaries, so some care is needed when starting or stopping each of them. You can use your current samba configuration files on Enware, just copy /etc/samba/smb.conf to /opt/etc/samba/smb.conf (possibly smbpasswd and smbusers also need to be copied).

 If you want to try and need further assistance please ask.

[ro ri]

Hello Jarek,

I can confirm this behaviour on my DNS-323-A1, RAID1 using 2x SAMSUNG HD154UI (1,5TB each, ext4 formatting).

The box is currently  running Alt-F 1.0 with kernel 4.4.86, and is
flashed with "Alt-F-1.0, initrd" and kernel "Alt-F-1.0, kernel 4.4.86".

As I am in a Linux-only environment, I "solved" this by switching to NFS (instead of SAMBA).

NFS is faster for me, however I can confirm that I had the same behaviour as decribed by you using SAMBA.

SAMBA: First 8 MB/s, then dropping to 2 MB/s when I copy to the DNS-323.

NFS: I see around 6MB/s (constantly) in my rsync job. Not stellar, but 3x faster compared to SAMBA.

@Joao: This makes me think that RAID1 could be the common denominator, as all other things seem different?

BR

RoRi

[ro ri]

Another observation: CPU load seems to be at 100% on this small machine using RAID1 , so maybe that is why transfer speed gets limited? Do you see such high CPU loads as well?

[Jarek Pe]

Hi,

Thanks for idea for using NFS. I have some concerns about using it. It's very hard to secure access to NFS share. Samba user/password security policy is obvious and easy to implement in home environment. I will try to tune samba settings first :)

--

Regards,

Jarek

[Akki K]

Copying multiple small files over network is a very slow operation - that's why they suggest tar them all in the source, sftp the tar (maybe zipped) and explode it at the destination. 

On Sun, Feb 16, 2020, 2:56 AM 'ro ri' via Alt-F <al...@googlegroups.com> wrote:

Hello Jarek,

I can confirm this behaviour on my DNS-323-A1, RAID1 using 2x SAMSUNG HD154UI (1,5TB each).

The box is currently  running Alt-F 1.0 with kernel 4.4.86, and is
flashed with "Alt-F-1.0, initrd" and kernel "Alt-F-1.0, kernel 4.4.86".

As I am in a Linux-only environment, I solved this by using NFS (instead of SAMBA).

NFS is faster for me, however I can confirm that I had the same behaviour as decribed by you using SAMBA.

==> First 8 MB/s, then dropping to 2 MB/s when I copy to the DNS-323.

@Joao: This makes me think that RAID1 could be the common denominator, as all other things seem different?

BR

RoRi

--
You received this message because you are subscribed to the Google Groups "Alt-F" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alt-f+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/alt-f/1633bb92-0f64-42fa-a209-5a277527dbbb%40googlegroups.com.

[Jarek Pe]

Thanks Joao for an idea. I'll try to do it in a few days. In the meanwhile I have explore some internet about this issue and it looks like this problem is common in samba. People says to try tune samba config. Do you have any idea which option will be fine to tune without killing this little CPU of DNS-323? :D

--

Regards,

Jarek

[Jarek Pe]

So... I have installed Entware samba as you said. I have copied smb.conf file and smbpasswd and smbusers to /opt/etc/samba/. But I think it is not necessary as in smb.conf file there is a referrer to smbpasswd and smbusers as shown:

        smb passwd file = /etc/samba/smbpasswd
        username map = /etc/samba/smbusers

Before launching entware samba I'm stopping alt-f samba. But I can't start Entware samba via web interface or /opt/etc/init.d/S08samba. When I do it, instead of starting entware samba, standard Alt-f samba launches. "Help me Obiwan Kenobi" :) 

--

Regards,

Jarek

W dniu piątek, 14 lutego 2020 19:56:01 UTC+1 użytkownik Joao Cardoso napisał:

[João Cardoso]

On Wednesday, 19 February 2020 13:12:48 UTC, Jarek Pe wrote:

So... I have installed Entware samba as you said. I have copied smb.conf file and smbpasswd and smbusers to /opt/etc/samba/. But I think it is not necessary as in smb.conf file there is a referrer to smbpasswd and smbusers as shown:

        smb passwd file = /etc/samba/smbpasswd
        username map = /etc/samba/smbusers

Before launching entware samba I'm stopping alt-f samba. But I can't start Entware samba via web interface or /opt/etc/init.d/S08samba. When I do it, instead of starting entware samba, standard Alt-f samba launches.

Why do you say that are Alt-F smbd/nmbd processes that are started?

Are you starting Entware Samba by using Services->User, entware,Configure, samba, StartNow? Or from the command line '/opt/etc/init.d/S08samba 

start'? In any case, Entware initscript have absolute paths names to /opt/sbin/smbd and /opt/sbin/nmbd, so those are the ones that are started and running.

Right, Services->Network,samba says that it is running (and it is), but it is not the Alt-F one.

You can check it by examining /var/run/samba and noticing that no smbd.pid file exists there (the Alt-F ones), but /opt/var/run has smbd.pid (the Entware ones). Why does Alt-F samba status says that smbd is running? because it is, there is a smbd process running in the system...

Anyway, Entware S08samba initscript is crippled, it does not even respects Entware own conventions (namely the 'check' argument).

You can replace it with the following code (perhaps saving the original with a name not starting with 'S')

#!/bin/sh
#
# startfile for entware-samba, written by linux.tinkerer at gmail.com
# For Ily's FW of Asus Routers
# store this script in /opt/etc/init.d if you've installed entware

# Prgmname=/full_path/Prgmname
prgmname1="/opt/sbin/nmbd"
prgmname2="/opt/sbin/smbd"

# configfile=/full_path/configfile
configfile="/opt/etc/samba/smb.conf"

#location of pid-file
nmbdpid="/opt/var/run/nmbd.pid"
smbdpid="/opt/var/run/smbd.pid"

# Changes from here till the end, use Entware rc.funcs

ENABLED=yes
PROCS="nmbd smbd"
ARGS="-D -s $configfile"
PREARGS=""
DESC=$PROCS
PATH=/opt/sbin:/opt/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

# check and remove stale pid
if ! pidof $PROCS 1>/dev/null 2>&1; then
   rm -f /opt/var/run/$PROCS.pid
fi

. /opt/etc/init.d/rc.func

Notice that for Services->User displays Enware as running it is necessary that all Entware boot-enabled services are running.

I noticed an almost 20% throughput increase in read speed when using Netware's Samba, not sure if it will solve your issues.

By the way, how many "thousands" small files are you transferring? 2000? 20000? 200000? On "wider" or "deep" organized folders? With tens, hundreds or thousands files per folder? Is your's issue transfer a one-time need, that could be solved by some other means, or a recurrent and frequent need or use-case?

 

"Help me Obiwan Kenobi" :) 

Let the Force be with you, grasshopper? :-)

[Jarek Pe]

Thanks João!

Now I see where I was wrong. I didn't check  /var/run/samba or /opt/var/run in search for smbd.pid file... I was only relying on www interface status. That was confusing to me when I launched entware samba and saw smb status "running" in Services->Network Services but "stopped" status on Services->User->entware->[configure button]->samba. After modification of S08samba as you said I have entware samba service status properly "running". Great! 

I must say that entware samba is more efficient on my DNS-323 than Alt-f's samba. Upload speed (PC to NAS transfer) is more less 1,5-2 MB/s faster. I have also noticed better CPU utilization. When I was copying files on Alt-f samba I had problems with listing directories and files on remote share. It was very slow process to access file list or I had timeout. Doing something on ssh session was also terribly slow. Entware samba is much faster in the same conditions. I don't have timeouts when listing directories/files when copying to samba share.

Despite of this improvement I still have the same slowing copying problem :) At the beginning of copying a bunch of files I have transfers about 10 MB/s (vs 8 MB/s on Alt-f samba). After a hour it drops to more less 1 MB/s. ). Samba restart repair this problem. But as I said - NAS is more responsible now. It is crucial to me because Windows 10 backup service works smoothly now (i'm still testing but first tests are promising.

The tests were made on folder that has 5140 files displaced in 106 folders - 31 GB total. Earlier tests were made on greater amount of files. There were photos (jpgs and raws) an also bigger ones (some video files for example). But I have copied them to NAS already :) so It would be time consuming to copy it again to PC and again to NAS. I think it's not relevant here. Anyway, current test file and folder structure is:

Photo:

-folder1 containing more less 10-100 *.CR2 (raw photo files, 11-15 MB each CR2 file)

--jpg1 folder contenting *.jpg files in the same amount as in folder1 CR2 files (3-4,5 MB each jpg file)

-folder2 containing more less 10-100 *.CR2 (raw photo files, 11-15 MB each CR2 file)

--jpg2 folder contenting *.jpg files in the same amount as in folder2 CR2 files (3-4,5 MB each jpg file)

-folder3 containing more less 10-100 *.CR2 (raw photo files, 11-15 MB each CR2 file)

--jpg3 folder contenting *.jpg files in the same amount as in folder3 CR2 files (3-4,5 MB each jpg file)

.

.

.

-folderN 

It's much of improvement but do you have any idea how to improve it more? :D

--

Regards,

Jarek

:D 

[João Cardoso]

On Sunday, 23 February 2020 22:46:46 UTC, Jarek Pe wrote:

Thanks João!

Now I see where I was wrong. I didn't check  /var/run/samba or /opt/var/run in search for smbd.pid file...

Actually I should have told you to use the command

ls -l /proc/$(pidof smbd)/exe

that unequivocally shows the path of the binary being executed, /mnt/sdc3/opt/sbin/samba_multicall in my case.

I was only relying on www interface status. That was confusing to me when I launched entware samba and saw smb status "running" in Services->Network Services but "stopped" status on Services->User->entware->[configure button]->samba. After modification of S08samba as you said I have entware samba service status properly "running". Great! 

I must say that entware samba is more efficient on my DNS-323 than Alt-f's samba.

Yes, I know, they don't  need to compile it for (flash-memory) space saving, they compile it for execution speed. They also apply patches to remove some generally not needed samba functionalities; otherwise it is exactly the same version that Alt-F uses.

 

Upload speed (PC to NAS transfer) is more less 1,5-2 MB/s faster. I have also noticed better CPU utilization. When I was copying files on Alt-f samba I had problems with listing directories and files on remote share. It was very slow process to access file list or I had timeout. Doing something on ssh session was also terribly slow.

Because CPU/memory was actively being used by samba? That's the only explanation.

 

Entware samba is much faster in the same conditions. I don't have timeouts when listing directories/files when copying to samba share.

It would be interesting to diagnose why that happens.

Despite of this improvement I still have the same slowing copying problem :) At the beginning of copying a bunch of files I have transfers about 10 MB/s (vs 8 MB/s on Alt-f samba). After a hour it drops to more less 1 MB/s. ). Samba restart repair this problem.

Have you tried to restart samba without canceling the current copy operation? Will windows recover from the temporary server fault? I'm afraid that I'm not really a windows user

 

But as I said - NAS is more responsible now. It is crucial to me because Windows 10 backup service works smoothly now (i'm still testing but first tests are promising.

 

 I use a DNS-325 with Alt-F smbd to make windows-10 backups (using Windows File History and windows-7 kind backups) without any issues.

The tests were made on folder that has 5140 files displaced in 106 folders - 31 GB total. Earlier tests were made on greater amount of files. There were photos (jpgs and raws) an also bigger ones (some video files for example).

 

Thanks for the details, they are necessary if I need to reproduce the situation.

 

But I have copied them to NAS already :) so It would be time consuming to copy it again to PC and again to NAS. I think it's not relevant here. Anyway, current test file and folder structure is:

Photo:

-folder1 containing more less 10-100 *.CR2 (raw photo files, 11-15 MB each CR2 file)

--jpg1 folder contenting *.jpg files in the same amount as in folder1 CR2 files (3-4,5 MB each jpg file)

-folder2 containing more less 10-100 *.CR2 (raw photo files, 11-15 MB each CR2 file)

--jpg2 folder contenting *.jpg files in the same amount as in folder2 CR2 files (3-4,5 MB each jpg file)

-folder3 containing more less 10-100 *.CR2 (raw photo files, 11-15 MB each CR2 file)

--jpg3 folder contenting *.jpg files in the same amount as in folder3 CR2 files (3-4,5 MB each jpg file)

.

.

.

-folderN 

It's much of improvement but do you have any idea how to improve it more? :D

I'm afraid I don't. I believe that the DNS-323 64MB of memory is the major limitation.

You could try to install Enware manually and install samba4. Notice that the Alt-F webUI to install and manage Entware is based on Entware-ng, which is not updated anymore -- it was merged with several other Entware forks to become (again) Entware ;-)

[Jarek Pe]

W dniu poniedziałek, 24 lutego 2020 20:45:03 UTC+1 użytkownik João Cardoso napisał:

On Sunday, 23 February 2020 22:46:46 UTC, Jarek Pe wrote:

Thanks João!

Now I see where I was wrong. I didn't check  /var/run/samba or /opt/var/run in search for smbd.pid file...

Actually I should have told you to use the command

ls -l /proc/$(pidof smbd)/exe

that unequivocally shows the path of the binary being executed, /mnt/sdc3/opt/sbin/samba_multicall in my case.

I was only relying on www interface status. That was confusing to me when I launched entware samba and saw smb status "running" in Services->Network Services but "stopped" status on Services->User->entware->[configure button]->samba. After modification of S08samba as you said I have entware samba service status properly "running". Great! 

I must say that entware samba is more efficient on my DNS-323 than Alt-f's samba.

Yes, I know, they don't  need to compile it for (flash-memory) space saving, they compile it for execution speed. They also apply patches to remove some generally not needed samba functionalities; otherwise it is exactly the same version that Alt-F uses.

 

Upload speed (PC to NAS transfer) is more less 1,5-2 MB/s faster. I have also noticed better CPU utilization. When I was copying files on Alt-f samba I had problems with listing directories and files on remote share. It was very slow process to access file list or I had timeout. Doing something on ssh session was also terribly slow.

Because CPU/memory was actively being used by samba? That's the only explanation.

 

Entware samba is much faster in the same conditions. I don't have timeouts when listing directories/files when copying to samba share.

It would be interesting to diagnose why that happens.

"No worries"... on Entware samba I have reached the same problem after 24hrs of windows 10 backup process - unfinished. I had to cancel it because my NAS was fried :) Here is webUI status:

Temperature
45.5°C/113°F

Fan speed
3510

Load
17.81

CPU
100%

Memory
83% of 64MB

Swap
53% of 1023MB

After clicking on any other tab of webUI it became unresponsive. So I have launched ssh session and this is what top said:

Mem: 58048K used, 2608K free, 0K shrd, 4048K buff, 723484K cached
CPU:   3% usr  26% sys   0% nic   0% idle  65% io   0% irq   4% sirq
Load average: 15.12 15.06 15.79 1/94 2386
  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
 2380   663 root     D     2640   4%   5% stunnel /etc/stunnel/stunnel-https.conf
 1966   663 root     S     1480   2%   3% dropbear -i
 1319  2102 root     D    44152  73%   2% smbd -D -s /opt/etc/samba/smb.conf
  843  2102 root     D    77036 127%   2% smbd -D -s /opt/etc/samba/smb.conf
 2271  2102 root     D    55044  91%   2% smbd -D -s /opt/etc/samba/smb.conf
 4015  2102 root     D    45940  76%   2% smbd -D -s /opt/etc/samba/smb.conf
 2659  2102 root     D     165m 279%   2% smbd -D -s /opt/etc/samba/smb.conf
 2798  2102 root     D    45788  75%   2% smbd -D -s /opt/etc/samba/smb.conf
 2850  2102 root     D    36468  60%   2% smbd -D -s /opt/etc/samba/smb.conf
 2341  2010 root     R     1204   2%   2% top
   74     2 root     SW<      0   0%   2% [kworker/0:1H]
 3642  2102 root     D     100m 170%   2% smbd -D -s /opt/etc/samba/smb.conf
    3     2 root     SW       0   0%   2% [ksoftirqd/0]
 2508  2102 root     D    53112  88%   1% smbd -D -s /opt/etc/samba/smb.conf
 1937     2 root     SW       0   0%   1% [kworker/0:0]
   14     2 root     DW       0   0%   1% [kswapd0]
 3384  2102 root     D    35000  58%   1% smbd -D -s /opt/etc/samba/smb.conf
 2016  2015 root     D     1288   2%   1% {status.cgi} /bin/sh status.cgi
 1070  1069 twonky   S    13364  22%   1% /opt/twonky/twonkyserver -appdata /var/lib/twonky -inifile /etc/twonky/twonky.conf -logfile /var/log/twonky/twonky.log
 1086  1069 twonky   S    13364  22%   0% /opt/twonky/twonkyserver -appdata /var/lib/twonky -inifile /etc/twonky/twonky.conf -logfile /var/log/twonky/twonky.log
  574     1 root     S      592   1%   0% sysctrl
 1083  1069 twonky   S    13364  22%   0% /opt/twonky/twonkyserver -appdata /var/lib/twonky -inifile /etc/twonky/twonky.conf -logfile /var/log/twonky/twonky.log
 1069  1068 twonky   S    13364  22%   0% /opt/twonky/twonkyserver -appdata /var/lib/twonky -inifile /etc/twonky/twonky.conf -logfile /var/log/twonky/twonky.log
 1081  1069 twonky   S    13364  22%   0% /opt/twonky/twonkyserver -appdata /var/lib/twonky -inifile /etc/twonky/twonky.conf -logfile /var/log/twonky/twonky.log
 1096  1069 twonky   S    13364  22%   0% /opt/twonky/twonkyserver -appdata /var/lib/twonky -inifile /etc/twonky/twonky.conf -logfile /var/log/twonky/twonky.log
 1071  1069 twonky   S    13364  22%   0% /opt/twonky/twonkyserver -appdata /var/lib/twonky -inifile /etc/twonky/twonky.conf -logfile /var/log/twonky/twonky.log
 1079  1069 twonky   S    13364  22%   0% /opt/twonky/twonkyserver -appdata /var/lib/twonky -inifile /etc/twonky/twonky.conf -logfile /var/log/twonky/twonky.log
 1077  1069 twonky   S    13364  22%   0% /opt/twonky/twonkyserver -appdata /var/lib/twonky -inifile /etc/twonky/twonky.conf -logfile /var/log/twonky/twonky.log
 1068     1 twonky   S    13364  22%   0% /opt/twonky/twonkyserver -appdata /var/lib/twonky -inifile /etc/twonky/twonky.conf -logfile /var/log/twonky/twonky.log
 1082  1069 twonky   S    13364  22%   0% /opt/twonky/twonkyserver -appdata /var/lib/twonky -inifile /etc/twonky/twonky.conf -logfile /var/log/twonky/twonky.log
 1080  1069 twonky   S    13364  22%   0% /opt/twonky/twonkyserver -appdata /var/lib/twonky -inifile /etc/twonky/twonky.conf -logfile /var/log/twonky/twonky.log
 1088  1069 twonky   S    13364  22%   0% /opt/twonky/twonkyserver -appdata /var/lib/twonky -inifile /etc/twonky/twonky.conf -logfile /var/log/twonky/twonky.log
 1078  1069 twonky   S    13364  22%   0% /opt/twonky/twonkyserver -appdata /var/lib/twonky -inifile /etc/twonky/twonky.conf -logfile /var/log/twonky/twonky.log
 2102     1 root     S     5560   9%   0% smbd -D -s /opt/etc/samba/smb.conf
 2068     1 root     S     5264   9%   0% nmbd -D -s /opt/etc/samba/smb.conf
 2008   663 root     S     2640   4%   0% stunnel /etc/stunnel/stunnel-https.conf
 2042  2036 root     S     2312   4%   0% /usr/lib/sftp-server
  603     1 root     S     1936   3%   0% smartd -i 1800
 1993   663 root     S     1224   2%   0% dropbear -i
  485     1 root     S     1224   2%   0% syslogd -C -m 0 -D
  617     1 root     S     1216   2%   0% crond
 2015  2008 root     S     1212   2%   0% httpd -ifh /usr/www
  663     1 root     S     1208   2%   0% inetd
 2010  1966 root     S     1208   2%   0% -sh
    1     0 root     S     1204   2%   0% init
  974     1 root     S     1204   2%   0% /bin/sh --
  975     1 root     S     1200   2%   0% {watch-inetd.sh} /bin/sh /usr/sbin/watch-inetd.sh
 2036  1993 root     S     1200   2%   0% sh -c /usr/lib/sftp-server
  488     1 root     S     1192   2%   0% klogd
 2376   975 root     S     1188   2%   0% sleep 180
   72     2 root     SW<      0   0%   0% [loop0]
  359     2 root     SW       0   0%   0% [md0_raid1]
  462     2 root     SW       0   0%   0% [jbd2/md0-8]
  754     2 root     SW       0   0%   0% [kworker/0:1]
 1356     2 root     SW       0   0%   0% [kworker/0:3]
  570     2 root     SW       0   0%   0% [kworker/u2:1]
    7     2 root     SW<      0   0%   0% [writeback]
    2     0 root     SW       0   0%   0% [kthreadd]
   39     2 root     SW       0   0%   0% [scsi_eh_1]
   37     2 root     SW       0   0%   0% [scsi_eh_0]
   50     2 root     SW<      0   0%   0% [bioset]
   32     2 root     SW<      0   0%   0% [bioset]
    9     2 root     SW<      0   0%   0% [bioset]
   30     2 root     SW<      0   0%   0% [bioset]
   35     2 root     SW<      0   0%   0% [bioset]
   36     2 root     SW<      0   0%   0% [bioset]

NAS is so busy that it can't even handle reboot command... That was few minutes ago and still nothing :)

 

Despite of this improvement I still have the same slowing copying problem :) At the beginning of copying a bunch of files I have transfers about 10 MB/s (vs 8 MB/s on Alt-f samba). After a hour it drops to more less 1 MB/s. ). Samba restart repair this problem.

Have you tried to restart samba without canceling the current copy operation? Will windows recover from the temporary server fault? I'm afraid that I'm not really a windows user

 

But as I said - NAS is more responsible now. It is crucial to me because Windows 10 backup service works smoothly now (i'm still testing but first tests are promising.

 

 I use a DNS-325 with Alt-F smbd to make windows-10 backups (using Windows File History and windows-7 kind backups) without any issues.

The tests were made on folder that has 5140 files displaced in 106 folders - 31 GB total. Earlier tests were made on greater amount of files. There were photos (jpgs and raws) an also bigger ones (some video files for example).

 

Thanks for the details, they are necessary if I need to reproduce the situation.

I'll do some more test for you :D I'm very motivated to solve this riddle. It's just driving me nuts :P 

[Joao Cardoso]

On Monday, February 24, 2020 at 10:04:36 PM UTC, Jarek Pe wrote:

W dniu poniedziałek, 24 lutego 2020 20:45:03 UTC+1 użytkownik João Cardoso napisał:

On Sunday, 23 February 2020 22:46:46 UTC, Jarek Pe wrote:

...

 

"No worries"... on Entware samba I have reached the same problem after 24hrs of windows 10 backup process - unfinished. I had to cancel it because my NAS was fried :) Here is webUI status:

Temperature
45.5°C/113°F

Fan speed
3510

Temperature is not high, neither the fan speed, it can go up to 6000RPM.

If your disks are hotter and you are concerned about that, you can set the fan speed higher, see Services->System,sysctrl,Configure. Read its online help (blue (?) icon)

 

 

Load
17.81

Now, that is something! you have 18 processes ready to run ASAP. That is not good.

 

CPU
100%

Memory
83% of 64MB

Swap
53% of 1023MB

neither that, swapping! For the current running processes to run comfortably the box should have ~600MB of memory. Linux is doing its best to emulate that amount of memory using slow disk. 

After clicking on any other tab of webUI it became unresponsive.

The contention for CPU/memory only increased from 18 processes to 19 or 20. Give it time :-)

 

So I have launched ssh session and this is what top said:

Mem: 58048K used, 2608K free, 0K shrd, 4048K buff, 723484K cached
CPU:   3% usr  26% sys   0% nic   0% idle  65% io   0% irq   4% sirq

65% of IO. It's paging/swapping in action

 

Load average: 15.12 15.06 15.79 1/94 2386
  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND

... 

 1319  2102 root     D    44152  73%   2% smbd -D -s /opt/etc/samba/smb.conf
  843  2102 root     D    77036 127%   2% smbd -D -s /opt/etc/samba/smb.conf
 2271  2102 root     D    55044  91%   2% smbd -D -s /opt/etc/samba/smb.conf
 4015  2102 root     D    45940  76%   2% smbd -D -s /opt/etc/samba/smb.conf
 2659  2102 root     D     165m 279%   2% smbd -D -s /opt/etc/samba/smb.conf
 2798  2102 root     D    45788  75%   2% smbd -D -s /opt/etc/samba/smb.conf
 2850  2102 root     D    36468  60%   2% smbd -D -s /opt/etc/samba/smb.conf 

... 

 3642  2102 root     D     100m 170%   2% smbd -D -s /opt/etc/samba/smb.

conf
 ...

 2508  2102 root     D    53112  88%   1% smbd -D -s /opt/etc/samba/smb.conf

... 

 3384  2102 root     D    35000  58%   1% smbd -D -s /opt/etc/samba/smb.

conf
 ... 

1070  1069 twonky   S    13364  22%   1% /opt/twonky/twonkyserver -appdata /var/lib/twonky -inifile /etc/twonky/twonky.conf -logfile /var/log/twonky/twonky.log

hmmm having twonky running is not good. But it is not currently consuming resources.

 

 2102     1 root     S     5560   9%   0% smbd -D -s /opt/etc/samba/smb.conf
 2068     1 root     S     5264   9%   0% nmbd -D -s /opt/etc/samba/smb.conf

...

So Windows-10 thinks it's talking to a 1GB quad-core box on a Gbps network. Fair, the year is 2020, not 2000 ;-)

That is the source of the problem, too many smbd processes, using hundreds of MB, and all them waiting for disk (the D in the top STAT column)

You need to convince win-10 to use less box resources.

You have to edit (/opt)/etc/samba/smb.conf and add options to try to limit resources. How will win-10 react to such lack of resources I don't know, you have to experiment. Try to add  in the global section, right after the line  [global]:

max smbd processes = 2

max connections = 2

And restart samba. Does win-10 complains? Does backup still works? BTW, what utility are you using for doing the backup?

The default values above are 0, meaning that there are no limits, which is not true. Thus win-10 bad manners are justifiable.

The value 2 above should allow another user to still use the box for trivial operations, but you can try restricting that yet more.

Remember, I'm not a windows user and I have no samba expertise, you might have to search for other ways to use samba on memory and CPU restricted computers.

Another possibility for win-10 "bad manners" is that given the lack of a timely response from the box, due to swapping/paging, win-10 will open another connection. "async smb echo handler = yes" or "deadtime = 5" (minutes) might help in this case. Don't know, just reading the manual page.

BTW, what filesystem are you using?

...

[Jarek Pe]

On Tuesday, February 25, 2020 at 2:54:40 AM UTC+1, Joao Cardoso wrote:

On Monday, February 24, 2020 at 10:04:36 PM UTC, Jarek Pe wrote:

W dniu poniedziałek, 24 lutego 2020 20:45:03 UTC+1 użytkownik João Cardoso napisał:

On Sunday, 23 February 2020 22:46:46 UTC, Jarek Pe wrote:

...

 

"No worries"... on Entware samba I have reached the same problem after 24hrs of windows 10 backup process - unfinished. I had to cancel it because my NAS was fried :) Here is webUI status:

Temperature
45.5°C/113°F

Fan speed
3510

Temperature is not high, neither the fan speed, it can go up to 6000RPM.

If your disks are hotter and you are concerned about that, you can set the fan speed higher, see Services->System,sysctrl,Configure. Read its online help (blue (?) icon)

Yes, I know :) I meant that my NAS was almost not responsive :D

 

 

 

Load
17.81

Now, that is something! you have 18 processes ready to run ASAP. That is not good.

 

CPU
100%

Memory
83% of 64MB

Swap
53% of 1023MB

neither that, swapping! For the current running processes to run comfortably the box should have ~600MB of memory. Linux is doing its best to emulate that amount of memory using slow disk. 

It's swapping only if I'm coping a lot of files. Usually it don't use swap. For that 31 GB testing sample or windows backup I have recorded that swap usage increase for about 6% per hour of coping process.

 

After clicking on any other tab of webUI it became unresponsive.

The contention for CPU/memory only increased from 18 processes to 19 or 20. Give it time :-)

I was testing my patient beyond any measure :D

Brilliant :) Thanks. It took me one step further. After deploying this settings I still can run my backup process without killing NAS after few hours. 

I'm using standard build in Windows 10 backup feature. It does the job really nice.

But as the NAS survive, samba don't. Here is what happening after 8 hours of backup (it should backup about 56 GB of data from: my documents, 31 GB of those test files and some video materials - transfer drops down as during normal copying process):

Temperature
44.5°C/112°F
Fan speed
3212
Load
3.37
CPU
100%
Memory
71% of 64MB
Swap
38% of 1023MB

Mem: 52204K used, 8452K free, 0K shrd, 3696K buff, 722972K cached
CPU:   0% usr   3% sys   0% nic   0% idle  95% io   0% irq   0% sirq
Load average: 3.33 3.25 2.83 1/86 2745

  PID  PPID USER     STAT   VSZ %VSZ %

CPU COMMAND
 3740  1984 root     R     1204   2%   1% top
 3200  3110 root     D     262m 443%   1% smbd -D -s /opt/etc/samba/smb.conf
 1392  3110 root     D     144m 244%   1% smbd -D -s /opt/etc/samba/smb.conf
 1768   584 root     S     1804   3%   1% dropbear -i
 1015  1014 twonky   S    13284  22%   0% /opt/twonky/twonkyserver -appdata /var/lib/twonky -inifile /etc/twonky/twonky.conf -logfile /var/log/twonky/twonky.log
 1046  1014 twonky   S    13284  22%   0% /opt/twonky/twonkyserver -appdata /var/lib/twonky -inifile /etc/twonky/twonky.conf -logfile /var/log/twonky/twonky.log
 1060  1014 twonky   S    13284  22%   0% /opt/twonky/twonkyserver -appdata /var/lib/twonky -inifile /etc/twonky/twonky.conf -logfile /var/log/twonky/twonky.log
  506     1 root     S      592   1%   0% sysctrl
   14     2 root     SW       0   0%   0% [kswapd0]
   74     2 root     SW<      0   0%   0% [kworker/0:1H]
    3     2 root     SW       0   0%   0% [ksoftirqd/0]
 2311     2 root     SW       0   0%   0% [kworker/0:2]
 2365     2 root     SW       0   0%   0% [kworker/u2:2]
 1014  1006 twonky   S    13284  22%   0% /opt/twonky/twonkyserver -appdata /var/lib/twonky -inifile /etc/twonky/twonky.conf -logfile /var/log/twonky/twonky.log
 1049  1014 twonky   S    13284  22%   0% /opt/twonky/twonkyserver -appdata /var/lib/twonky -inifile /etc/twonky/twonky.conf -logfile /var/log/twonky/twonky.log
 1044  1014 twonky   S    13284  22%   0% /opt/twonky/twonkyserver -appdata /var/lib/twonky -inifile /etc/twonky/twonky.conf -logfile /var/log/twonky/twonky.log
 1006     1 twonky   S    13284  22%   0% /opt/twonky/twonkyserver -appdata /var/lib/twonky -inifile /etc/twonky/twonky.conf -logfile /var/log/twonky/twonky.log
 1034  1014 twonky   S    13284  22%   0% /opt/twonky/twonkyserver -appdata /var/lib/twonky -inifile /etc/twonky/twonky.conf -logfile /var/log/twonky/twonky.log
 1040  1014 twonky   S    13284  22%   0% /opt/twonky/twonkyserver -appdata /var/lib/twonky -inifile /etc/twonky/twonky.conf -logfile /var/log/twonky/twonky.log
 1045  1014 twonky   S    13284  22%   0% /opt/twonky/twonkyserver -appdata /var/lib/twonky -inifile /etc/twonky/twonky.conf -logfile /var/log/twonky/twonky.log
 1041  1014 twonky   S    13284  22%   0% /opt/twonky/twonkyserver -appdata /var/lib/twonky -inifile /etc/twonky/twonky.conf -logfile /var/log/twonky/twonky.log
 1043  1014 twonky   S    13284  22%   0% /opt/twonky/twonkyserver -appdata /var/lib/twonky -inifile /etc/twonky/twonky.conf -logfile /var/log/twonky/twonky.log
 1050  1014 twonky   S    13284  22%   0% /opt/twonky/twonkyserver -appdata /var/lib/twonky -inifile /etc/twonky/twonky.conf -logfile /var/log/twonky/twonky.log
 1042  1014 twonky   S    13284  22%   0% /opt/twonky/twonkyserver -appdata /var/lib/twonky -inifile /etc/twonky/twonky.conf -logfile /var/log/twonky/twonky.log
 3110     1 root     S     5560   9%   0% smbd -D -s /opt/etc/samba/smb.conf
 3105     1 root     S     5264   9%   0% nmbd -D -s /opt/etc/samba/smb.conf
 3806  3803 SeSim    S     2320   4%   0% /usr/lib/sftp-server
 2027  2026 root     S     2308   4%   0% /usr/lib/sftp-server
  528     1 root     S     1936   3%   0% smartd -i 1800
 3792   584 root     S     1804   3%   0% dropbear -i
 3801  3792 SeSim    S     1224   2%   0% -sh
 3799   584 root     S     1224   2%   0% dropbear -i
 1915   584 root     S     1224   2%   0% dropbear -i
  466     1 root     S     1224   2%   0% syslogd -C -m 0 -D
  540     1 root     S     1216   2%   0% crond
 1984  1768 root     S     1216   2%   0% -sh
  584     1 root     S     1208   2%   0% inetd
    1     0 root     S     1204   2%   0% init
  849     1 root     S     1204   2%   0% /bin/sh --
  850     1 root     S     1200   2%   0% {watch-inetd.sh} /bin/sh /usr/sbin/watch-inetd.sh
 3803  3799 SeSim    S     1200   2%   0% sh -c /usr/lib/sftp-server
 2026  1915 root     S     1200   2%   0% sh -c /usr/lib/sftp-server
  469     1 root     S     1192   2%   0% klogd
 2473   850 root     S     1188   2%   0% sleep 180

   72     2 root     SW<      0   0%   0% [loop0]
  359     2 root     SW       0   0%   0% [md0_raid1]

  868     2 root     SW       0   0%   0% [jbd2/md0-8]
 2017     2 root     SW       0   0%   0% [kworker/0:1]
 1878     2 root     SW       0   0%   0% [kworker/u2:0]
 2239     2 root     SW       0   0%   0% [kworker/0:0]

    2     0 root     SW       0   0%   0% [kthreadd]
   39     2 root     SW       0   0%   0% [scsi_eh_1]
   37     2 root     SW       0   0%   0% [scsi_eh_0]

   11     2 root     SW<      0   0%   0% [ata_sff]

    9     2 root     SW<      0   0%   0% [bioset]

   12     2 root     SW<      0   0%   0% [md]
   27     2 root     SW<      0   0%   0% [bioset]

   32     2 root     SW<      0   0%   0% [bioset]

   33     2 root     SW<      0   0%   0% [bioset]
   10     2 root     SW<      0   0%   0% [kblockd]

   35     2 root     SW<      0   0%   0% [bioset]
   36     2 root     SW<      0   0%   0% [bioset]

  869     2 root     SW<      0   0%   0% [ext4-rsv-conver]
   34     2 root     SW<      0   0%   0% [bioset]
   15     2 root     SW       0   0%   0% [fsnotify_mark]
   40     2 root     SW<      0   0%   0% [scsi_tmf_1]

I have very high io usage but load is quite nice. I can work with webUI, it's slower but acceptable.

What does it mean when VSZ = 262m  and %VSZ = 443% ?

As I understand this 2 smbd process are swapping because the have eaten all available ram to them? 

If it is so. I'm wondering... what If I connect pendrive (yes I know the limitation of silicon in standard usb stick ;) - just for testing) or SSD drive to NAS USB and switch swap to it? Maybe it will boost for a little swapping?

The default values above are 0, meaning that there are no limits, which is not true. Thus win-10 bad manners are justifiable.

The value 2 above should allow another user to still use the box for trivial operations, but you can try restricting that yet more.

Remember, I'm not a windows user and I have no samba expertise, you might have to search for other ways to use samba on memory and CPU restricted computers.

Another possibility for win-10 "bad manners" is that given the lack of a timely response from the box, due to swapping/paging, win-10 will open another connection. "async smb echo handler = yes" or "deadtime = 5" (minutes) might help in this case. Don't know, just reading the manual page.

 

I'll do some further testing with different configs and let you know.

BTW, what filesystem are you using?

ext4

...

[Paul Wilkie]

[Joao Cardoso]

On Thursday, February 27, 2020 at 9:52:01 PM UTC, Jarek Pe wrote:

On Tuesday, February 25, 2020 at 2:54:40 AM UTC+1, Joao Cardoso wrote:

On Monday, February 24, 2020 at 10:04:36 PM UTC, Jarek Pe wrote:

W dniu poniedziałek, 24 lutego 2020 20:45:03 UTC+1 użytkownik João Cardoso napisał:

On Sunday, 23 February 2020 22:46:46 UTC, Jarek Pe wrote:

...

... 

Mem: 52204K used, 8452K free, 0K shrd, 3696K buff, 722972K cached
CPU:   0% usr   3% sys   0% nic   0% idle  95% io   0% irq   0% sirq
Load average: 3.33 3.25 2.83 1/86 2745
  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
 3740  1984 root     R     1204   2%   1% top
 3200  3110 root     D     262m 443%   1% smbd -D -s /opt/etc/samba/smb.conf
 1392  3110 root     D     144m 244%   1% smbd -D -s /opt/etc/samba/smb.conf
 1768   584 root     S     1804   3%   1% dropbear -

I have very high io usage but load is quite nice. I can work with webUI, it's slower but acceptable.

What does it mean when VSZ = 262m  and %VSZ = 443% ?

virtual memory used by the process. As there are only 64MB of physical memory on the DNS-323/321, the missing part is emulated using disk, i.e., swap.

You only aggravate things by running many programs at the same time, e.g. twonky is eating 24MB. Not much, but it aggravates the problem.

The D under STAT means that the process is waiting in an uninterruptible state, i.e, it is not responsible as it is waiting for disk operations to complete.

Having swap in use is not by itself very worrying, if the process does not needs to be copied from swap to main memory for doing some processing. If that happens, parts of another process that is in physical memory has to be moved to swap, and when that happens frequently it's calling paging, and that is troublesome and has to be avoided.

You idea of using a USB pen for swap will make paging work a little faster, it is USB-2, but it is not the solution. There is no solution other than adding more physical memory or restricting the usage to give to the box. Swapping to USB is disabled by default, see Disk->utilities. 

It seems that samba-3.6 has issues with windows-10 backup for you. The fact that memory usage slowly increases over time seems to indicate that there is a memory leak in samba, but even if we discover that it is indeed he case, it would take a considerable amount of effort to fix it, if possible, and samba-3.6 has reach EOL five years ago and nobody in the samba team would be interested in digging into it now. Nor me.

So, your only option is to identify the problematic folder(s) and exclude it(them) from the backup, or use some other method or protocol to do the backup. Perhaps using windows 7 backups (Backup and Restore (Windows 7))? I think that it creates several zip files and, as Akki says, it is more efficient. I don't know, as I'm not a windows user. But my wife uses win-10 File History backups and also win-7 weekly backups without issues (using Alt-F on a DNS-325, 256MB of RAM).

[Jarek Pe]

Sorry Joao I didn't know that it is impossible to solve this issue and I did it ;) OK, I'm cheating a little bit but who cares! It works :P

I was thinking about the limitation of hardware - slow CPU, very limited RAM etc. General issue is when samba reserve/addresses space on file system for new files. If there is a lot of them queue arise and we encounter this slowing problem and memory leaking and the end. So how to rid of numerous files during transfer? Make one BIG file :D Yes, I know, I could use tar on source machine and copy it to NAS and then untar it. But not in this case. I need to configure differential file backup not full backup on every backup launch. I needed file container. So I took Vera Crypt and make 200 GB encrypted container and uploaded it to NAS and mounted it under my Windows 10 as drive. Then in Windows 10 backup feature I have pointed Vera Crypt mounted drive as target of backup and that's it :) Backup works smoothly in max reachable speed 10 MB/s - all the time, without slowing. Normal load. Very low IO's during transfer. No swapping. And the end I have nice file history in Windows 10 backup tool - that was the target of this exercise :)

I only need to figure how to auto mount Vera Crypt file container on Windows logon. Right now it is manual process. 

Although I have encountered one little problem with this solution. I need to test it more to isolate exact conditions. In general, sometimes if I restart Windows after using file container I can't access any samba share. I need to restart samba on NAS and everything works fine. If I don't find any solution I will schedule in cron samba restart somewhere in the night. It's not elegant solution but should do the job :)

Thanks for supporting me with this issue! :)

[João Cardoso]

On Sunday, 1 March 2020 16:24:45 UTC, Jarek Pe wrote:

Sorry Joao I didn't know that it is impossible to solve this issue

Using available resources ;) 

and I did it ;) 

OK, I'm cheating a little bit but who cares! It works :P

I was thinking about the limitation of hardware - slow CPU, very limited RAM etc. General issue is when samba reserve/addresses space on file system for new files. If there is a lot of them queue arise and we encounter this slowing problem and memory leaking and the end. So how to rid of numerous files during transfer? Make one BIG file :D Yes, I know, I could use tar on source machine and copy it to NAS and then untar it. But not in this case. I need to configure differential file backup not full backup on every backup launch. I needed file container. So I took Vera Crypt and make 200 GB encrypted container and uploaded it to NAS and mounted it under my Windows 10 as drive.

So, using an existing samba share? You had no issues copying a 200 GB file using Alt-F samba?

Couldn't you just create a windows virtual drive (vdx) instead? You can even browse and specify a network location during its creation. I created and used a 200MB one without issues.

 

Then in Windows 10 backup feature I have pointed Vera Crypt mounted drive as target of backup and that's it :) Backup works smoothly in max reachable speed 10 MB/s - all the time, without slowing. Normal load. Very low IO's during transfer. No swapping. And the end I have nice file history in Windows 10 backup tool - that was the target of this exercise :)

Excellent.

Of course the encryption/decryption is done in the windows client, otherwise you would have problems -- smb is being used just as a raw transport protocol. iSCSI, when it worked in Alt-F, could provide a similar solution.

[Jarek Pe]

On Monday, March 2, 2020 at 9:06:57 PM UTC+1, João Cardoso wrote:

On Sunday, 1 March 2020 16:24:45 UTC, Jarek Pe wrote:

Sorry Joao I didn't know that it is impossible to solve this issue

Using available resources ;) 

and I did it ;) 

OK, I'm cheating a little bit but who cares! It works :P

I was thinking about the limitation of hardware - slow CPU, very limited RAM etc. General issue is when samba reserve/addresses space on file system for new files. If there is a lot of them queue arise and we encounter this slowing problem and memory leaking and the end. So how to rid of numerous files during transfer? Make one BIG file :D Yes, I know, I could use tar on source machine and copy it to NAS and then untar it. But not in this case. I need to configure differential file backup not full backup on every backup launch. I needed file container. So I took Vera Crypt and make 200 GB encrypted container and uploaded it to NAS and mounted it under my Windows 10 as drive.

So, using an existing samba share? You had no issues copying a 200 GB file using Alt-F samba?

Yes I have used existing samba share. It looks like coping only one big file is not the problem or this problem not revels with full power in this conditions. I didn't try to copy for example 500 GB file :D I don't have so much space on source machine right now. 

 

Couldn't you just create a windows virtual drive (vdx) instead? You can even browse and specify a network location during its creation. I created and used a 200MB one without issues.

 

I didn't know that this feature is available on Desktop Windows ... I though that it is reserved only for Windows Server :D Frankly I used first idea that came to my mind and that was Vera Crypt. I have just applied VHDX... It runs like hell! I must test dynamic vhdx drive later. You are genius Joao :) Thanks again!

 

Then in Windows 10 backup feature I have pointed Vera Crypt mounted drive as target of backup and that's it :) Backup works smoothly in max reachable speed 10 MB/s - all the time, without slowing. Normal load. Very low IO's during transfer. No swapping. And the end I have nice file history in Windows 10 backup tool - that was the target of this exercise :)

Excellent.

Of course the encryption/decryption is done in the windows client, otherwise you would have problems -- smb is being used just as a raw transport protocol. iSCSI, when it worked in Alt-F, could provide a similar solution.

 

I only need to figure how to auto mount Vera Crypt file container on Windows logon. Right now it is manual process. 

Although I have encountered one little problem with this solution. I need to test it more to isolate exact conditions. In general, sometimes if I restart Windows after using file container I can't access any samba share. I need to restart samba on NAS and everything works fine. If I don't find any solution I will schedule in cron samba restart somewhere in the night. It's not elegant solution but should do the job :)

If anyone find this useful - I think I know why I can't access samba share after Windows restart. Probably it's happening because of applied session limitation in samba config. Probably samba is keeping connection of active mounted Vera Crypt container and after restart it drops another session that exceeds the limit. When I dismount container before restart everything works fine. It is even possible to auto mount this container on Windows logon.

[KenK]

I'm experiencing the same issue on my dsn-320.  So I tried Joao's suggestion to set "max smbd process = 2".  However, that setting limits the number of connections only and has no effect to the issue.  But since I'm at the configuration page, so I started to check out other settings.  Anyway, I came across a setting "max mus" which default at 50.  I changed to 20 and run a windows backup.  Initially the speed is about 10MB/s.  After 2 hours into the backup, the speed is still around 7.5MB/s.  Could this be the answer?  Anyone want to give it a try?

[Jarek Pe]

I have tried to set max mus. Different values. Started from 20, ended on 5. Slowing transfer and big load no matter the value. On DNS-323 it's better to use some file container :)