Alt-F DNS-323 to Alt-F DNS-323 slow (5-7 MB/s) transfer times (0.1 RC3)

[Richard Trout]

Have just upgraded 2x DNS-323 to Alt-F, to allow a migration from RAID 1 2x 1.5GB to 2x 3TB drives (WD Red). When it comes time to transfer data from one DNS-323 to the other, I seem to have slower transfer times than I would expect. All on Gb Ethernet.

I've tried samba (via a Windows machine), rsync and ftp (via wget) (both DNS-to-DNS) and transfer rates are typically 5-7 MB/s (rsync slower of course). On stock firmware, PC to DNS-323 via SAMBA, I would get 12-20MB/s. Read times are certainly faster.

I haven't performed exhaustive tests, and don't really want to revert to stock firmware to perform additional A-B tests. Based on other forum posts, these statistics seem to be at the slow end of expectations. In particular I thought wget/ftp would be quicker than SAMBA, especially as the transfer is direct rather than through a host Windows machine.

I have 500GB+ to transfer, and can wait the days it will take to complete. However for the benefit of others who may migrate to Alt-F and face similar problem, is there anything I should review? Are these rates typical?

Thanks, Richard.

PS: Awesome work on Alt-F. Can't believe how much care has been taken for the basic functionality to 'just work', and how complete the functionality is.

[TJ]

I can contribute my dns-to-dns performance, and pc-to-dns performance but I think we should decide on specifically what software we should use. Can poll threads be started on here?

[João Cardoso]

On Friday, May 31, 2013 3:30:42 AM UTC+1, TJ wrote:

I can contribute my dns-to-dns performance, and pc-to-dns performance but I think we should decide on specifically what software we should use.

That is a very important prerequisite.

Unfortunately I'm a very picky person, as you will see below.

If you search the net for "samba low speed", or similar, you will find lots of miraculous solutions.

But you will often find followup-posts of users that applied the solution and it didn't work for them!

The reason is the same as when Dr. House starts prescribing drugs to their patients without having first correctly diagnosed their disease :-)

To apply a cure to a problem one has to first know the source of the problem.

I'm not an expert on none of these, but all the following can affect performance.

One has to first identify which one is the culprit, as applying cures to the wrong one will often causes issues on the others, specially on our low power cpu and memory box.

1-physical network status or setup

2-network setup on the box

3-protocol tune

4-disk "layout"

5-filesystem type

6-test files

1-physical network status or setup

Gbps network, routers, switches, defective network cables, excessive traffic, collisions...

programs such as 'netcat'/'netperf' can diagnose most of these (as they are mostly independent of the other factors), by showing the raw network transfer speed http://dns323.kood.org/information:benchmarks

2-network setup on the box

frame size (MTU), send or receive buffers, other box activity; 'ifconfig', '/proc/sys/net/ shows most of these. All services must be stopped in the box.

You will find in the net many setups for these, the first one I got was http://www.cyberciti.biz/faq/linux-tcp-tuning/, but remember that most of those are for desktop computers with Giga bytes of memory, and if you use one of those settings in our box it will strongly degrade other system functions.

3-protocol tune

ftp is the simplest, and has almost no tunable parameters (don't use SSL!).

samba has some 386, yes, at least 386, tunable parameters!

Read my last post in the NEWS topic regarding samba setup (the samba package). You will find thousands of references in the net regarding those settings, but read first the samba author comment on those. Samba authors also say that samba performance should be equal to ftp, but I strongly doubt that!

Also, the operation system version matters, winXP is different from Vista /Win7/8 or linux, so always state which one you are using.

4-disk "layout"

RAID requires most of the box, so there must be two sets of tests, for standard and RAID1 setups. Always state which one you are using.

5-filesystem type

ext2 is the fastest, it is used by default in the D-Link fw, ext4 is the next fastest. So always state which one is in use.

But this is not all -- is the disk in use a newly formated one, or has data for several months or years? Is it near full? It makes a difference!

Converting filesystem type from ext2 to ext3 or ext4 will not change the files by itself (read about 'chattr'), and only new files will benefice.

Also, what mount options are in use? Some options make the data to be more secure, but they also slow down access time ( see 'barrier' or 'data' mount options)

I opted for the most secure mount options by default, as Alt-F was motivated by the poor security that exists in the stock firmware.

As a rule of thumb, *all* packages are shipped with its tunable settings as the software author think that's is best -- after all he must be an expert!

6-test files

Are files fragmented? the 'filefrags' program will show this. If the disk head has to move forth and back often, you will see irregular transmission speed, with many speed drops, if you use a network graphic tool. A few tens or hundreds of extends are of no concern.

Are you transmitting a single large file, or hundreds of small files? It is different, as the filesystem starts to play a big role here. Always use a single large (GB) file, and be aware of cache effects on the desktop computer.

To establish a baseline, I propose to use the default network settings, and use ftp and samba to measure the read throughput of a single GB file with most services stopped.

And repeat the transfer a couple of times.

-Using Samba from a linux computer, Gbps network, standard MTU, standard disk, standard ext4 mount options (rw,relatime,barrier=1,data=ordered), reading a 700MB/9 extents file from the box

Reading:

Samba:

# smbclient -N //nas/sda4 -c "cd foo; get Immortel.avi"

getting file \foo\Immortel.avi of size 733151232 as Immortel.avi (18358.6 KiloBytes/sec) (average 18358.6 KiloBytes/sec)

Same file, a few minutes latter:

# smbclient -N //nas/sda4 -c "cd foo; get Immortel.avi"

getting file \foo\Immortel.avi of size 733151232 as Immortel.avi (16864.6 KiloBytes/sec) (average 16864.6 KiloBytes/sec)

ftp:

-Same, but on a (degraded) RAID1, standard ext4 mount options, reading a 700MB/182 extents file from the box

# wget ftp://ftp@nas:RO/Videos/Immortel.avi

...

2013-05-31 18:51:04 (21.2 MB/s) - `Immortel.avi.1' saved [733151232]

Same results a few minutes latter

Writing:

samba:

# smbclient -N //nas/sda4 -c "cd foo; put Immortel.avi"

putting file Immortel.avi as \foo\Immortel.avi (12891.5 kb/s) (average 12891.5 kb/s)

a few minutes latter:

# smbclient -N //nas/sda4 -c "cd foo; put Immortel.avi"

putting file Immortel.avi as \foo\Immortel.avi (12837.4 kb/s) (average 12837.4 kb/s)

samba "tuned":

Same, but adding SO_SNDBUF=87380 SO_RCVBUF=87380 to socket options in smb.conf and restarting samba:

# smbclient -N //nas/sda4 -c "cd foo; put Immortel.avi"

putting file Immortel.avi as \foo\Immortel.avi (17291.8 kb/s) (average 17291.8 kb/s)

ftp: (lukemftp client)

writing to the degraded RAID1:

# ftp -u ftp@nas:RW/foo Immortel.avi 

...

733151232 bytes sent in 00:45 (15.37 MB/s)

writing to the standard fs:

# ftp -u ftp@nas/mnt/sda4/foo/ Immortel.avi 

...

733151232 bytes sent in 00:43 (16.00 MB/s)

I don't yet have netcat/netperf to remove the disk and filesystem variables from the equation (someone please fill-in a package request for this, so I will not forget)

And I don't have a second DNS-323 to perform this topic's tests, but making write tests should provide the same info (and a clear separation of issues -- is it a read or a write problem?)

As a final! consideration, remember that Alt-F is based on free (as in beer and speech) software.

The DNS-323 support was contributed to the linux kernel by Marvell, the maker of our box SoC/CPU, but some details was left out from the contribution -- perhaps those missing parts can explain why the D-Link fw is in some aspects faster then the free software, I don't know.

> Can poll threads be started on here?

I'm afraid not.

But you could create a new Topic and maintain data on its first post, editing it when users contribute new data

[Richard]

Joao thank you very much for your detailed reply on benchmarking. Did you see my OP and whether you thought that was a reasonable speed? Some details I had missed earlier. Filesystem is ext2 and the files being transferred are 1-45GB in size (yes, 45GB).

My speeds are about half of yours, although DNS-323 to DNS-323 rather than DNS-323 to fast host.

[João Cardoso]

On Friday, May 31, 2013 10:36:53 PM UTC+1, Richard wrote:

Joao thank you very much for your detailed reply on benchmarking. Did you see my OP and whether you thought that was a reasonable speed? Some details I had missed earlier. Filesystem is ext2 and the files being transferred are 1-45GB in size (yes, 45GB).

> I've tried samba (via a Windows machine), rsync and ftp (via wget) (both DNS-to-DNS) and transfer rates are typically 5-7 MB/s

We need to see if the problem is on samba or whatever, and establish what exactly the 5-7MB/s means -- samba or ftp?

How are you measuring the box-to-box transfer speed? Using what method to transfer the file? ftp? samba? smbclient? smgbet? wget?

-What is the reading speed when:

 -using ftp/wget to read from box-to-box?

 -using  smbclient/smbget to read from box-to-box? (in my linux PC smbget gives lower throughput than smbclient -- why?)

 -using a MS-Win computer to read from box-to-PC using samba? and when using ftp?

 

smbget/smbclient are included in the samba-extra Alt-F package

What does the Status page says about network speed and duplex?

My speeds are about half of yours, although DNS-323 to DNS-323 rather than DNS-323 to fast host.

Both DNS-323 have Alt-F? 

[Richard]

Hi Joao

Thanks very much for your reply. I've answered your questions below.

> I've tried samba (via a Windows machine), rsync and ftp (via wget) (both DNS-to-DNS) and transfer rates are typically 5-7 MB/s 

We need to see if the problem is on samba or whatever, and establish what exactly the 5-7MB/s means -- samba or ftp?

The rates were very similar for samba (via Windows host) and ftp/wget.

How are you measuring the box-to-box transfer speed? Using what method to transfer the file? ftp? samba? smbclient? smgbet? wget?

wget ftp. using the statistics reported by wget and rsync

samba was via windows host using teracopy, and the speeds reported by it.

-What is the reading speed when:

 -using ftp/wget to read from box-to-box?

5-7 MB/s

 -using  smbclient/smbget to read from box-to-box? (in my linux PC smbget gives lower throughput than smbclient -- why?)

Didn't try this.

 

 -using a MS-Win computer to read from box-to-PC using samba? and when using ftp?

Will try this soon.

What does the Status page says about network speed and duplex?

Network

Speed: 1000Mbps Duplex: full MTU: 1500

(on both DNS-323)

 

Both DNS-323 have Alt-F? 

Yes, flashed to 0.1 RC3. At the time both units were freshly flashed with no additional packages loaded.

Regards, Richard.

 

[João Cardoso]

When I increase Alt-F samba loglevel to 5 and use it from my linux computer, I can see in the box samba logs that the auto-tuned receive buffers are set to about 16BK (or was it 32KB?) and the send buffers are set to about 87KB.

This explains why the box read speed is OK but the write speed is low.

When you connect two boxes together using samba, then probably both the send and receive buffer will be limited to the above 16KB (or was it 32KB?), as one box offers a low value of the receive buffer, and that will be used by the other box as its send buffer. Got it?

If I increase the box receive buffer to the same value as the send buffer, my write speed increase significantly.

So, what I propose you to experiment with is to stop samba using the command 

rcsmb stop

edit the /etc/samba/smb.conf file and change the line

socket options = IPTOS_LOWDELAY TCP_NODELAY

to 

socket options = IPTOS_LOWDELAY TCP_NODELAY SO_SNDBUF=65535 SO_RCVBUF=65535

and restart samba with the command

rcsmb start

Desktop computers often use 131071 instead of 65535. Setting a much high value will deprive other OS components from memory, so the best is for you to experiment several values and use the lower possible value.

Usual values are powers of 2, i.e., 8192, 16384, 32768, 65536 and 103764. Sums of powers of two are also OK.

Of course you have execute those commands after telnet or ssh the box and logging in as the 'root' user (same passwd as the webUI password)

Did it works? If yes, what was the lower value that you found reasonable? Please report back.

 

[Richard]

Thanks for your informative reply Joao. I'll take some time to review the samba logs and experiment with the buffers, as well as running before & after tests, and post the results back to here.

What I still don't understand is why wget/ftp should have been so low. Would you say that in most cases FTP would provide better performance than samba?

Thanks, Richard.

[João Cardoso]

On Tuesday, June 4, 2013 1:46:54 AM UTC+1, Richard wrote:

Thanks for your informative reply Joao. I'll take some time to review the samba logs and experiment with the buffers

Remember that performance drops a lot when you increase the 'log level' -- I used it only to find what values was being used for the buffers. And the amount of log (5) is enormous, better do a 'grep BUF'

 

, as well as running before & after tests, and post the results back to here.

What I still don't understand is why wget/ftp should have been so low. Would you say that in most cases FTP would provide better performance than samba?

Yes, because it is "lighter" then smb.

if you don't specify the buffer size (the "socket" buffer size, there are others, don't play with more than one at a time, and be aware of cache effects for small files), samba will use what the linux kernel thinks is more appropriate (and this strongly depends on available memory). This is called "auto-tuning".

But the same way as samba can override that kernel "advise", by using the socket option, other applications can also do it -- or not.

So, some clients, be them ftp/wget/smbget/smgclient, can choose to use their own values or the kernel defaults -- or not, and then throughput will be higher or lower.

This is all "theoretical", that's why you have to experiment with.

My symptoms (from my linux box) was good read speed and low write speed to the box -- at first I though that a low write speed was because of the disk writing, but it turns out to not be that the reason.

From a MS-Vista PC those symptoms did not occur, and both the BUF values was "high" and similar. This seems to indicate that some form of negotiation occurred between the MS-Vista PC and the box samba server that did not occur between my linux PC and the box samba server.

As having two nas talking to each other is equivalent to my linux-PC<->nas setup, I'm guessing (extrapolating) that the reason is the same.

Thanks, Richard.

[Jim Nielsen]