solaris downloads much slower than linux
Doug
some HP and Dell servers runing Debian Linux. All the servers have
gigabit ethernet connections into the same network switch.
I used "dladm show-dev" to confirm the ethernet links are set to full
duplex and 1000Mbps on the Sun servers.
I recently noticed when downloading large files from FTP and web sites
using wget that the Solaris servers get download speeds 2-10x slower
than the linux servers. For instance, there are about 85G of genomic
sequence data here: ftp://ftp.ncbi.nih.gov/blast/db/. Today, I get
between 10-20MB/s download rate on the linux systems, but only 1-2MB/s
downloading on the solaris systems. I tried downloading some linux
ISO files from various mirrors (ftp and http) and saw in every case
the solaris systems were significantly slower than the linux servers.
At first, I thought this was a general Solaris network problem or
misconfiguration (the Sun servers all have Intel e1000g network
interfaces.) But, the Solaris systems can download files at >100MB/s
when I use wget or curl from my linux servers (sharing files using
http or nfs from the linux servers.)
Has anybody else seen this sort of performance problem?
Thanks
Dan McDonald
Doug <dy2...@gmail.com> wrote:
<SNIP!>
>
>At first, I thought this was a general Solaris network problem or
>misconfiguration (the Sun servers all have Intel e1000g network
>interfaces.) But, the Solaris systems can download files at >100MB/s
>when I use wget or curl from my linux servers (sharing files using
>http or nfs from the linux servers.)
>
>Has anybody else seen this sort of performance problem?
So local-network transfers are fine, but long-distance ones are a problem?
I suspect it's your TCP window sizes. I'm still not sure why they aren't
fixed by default yet, but utter these with appropriate privilege:
ndd -set /dev/tcp tcp_xmit_hiwat 1048576
ndd -set /dev/tcp tcp_recv_hiwat 1048576
and try some of those long-distance transfers again. Please report your
findings here.
Thanks,
--
Daniel L. McDonald - Solaris Security & Networking Engineering
Mail: dan...@sun.com | * MY OPINIONS ARE NOT NECESSARILY SUN'S! *
35 Network Drive Burlington, MA |"rising falling at force ten
http://blogs.sun.com/danmcd/ | we twist the world and ride the wind" - Rush
Doug
> So local-network transfers are fine, but long-distance ones are a problem?
>
> I suspect it's your TCP window sizes. I'm still not sure why they aren't
> fixed by default yet, but utter these with appropriate privilege:
>
> ndd -set /dev/tcp tcp_xmit_hiwat 1048576
> ndd -set /dev/tcp tcp_recv_hiwat 1048576
>
> and try some of those long-distance transfers again. Please report your
> findings here.
Hi Dan,
Thanks for the tip. Both the tcp_xmit_hiwat and tcp_recv_hiwat
parameters were set at 49152. I raised them to what you wrote. Now
the download speed on the Solaris system is approx the same as the
Linux ones.
Also, the value of tcp_max_buf is also set at 1048576. A thread from
2007 says that it should be increased to more than the TCP transmit or
recv high water mark values. Ref:
http://groups.google.com/group/comp.unix.solaris/browse_frm/thread/196d18389f85842e
These Solaris servers are running Solaris 10 update 4 (8/07) but I
have been patching them regularly since then (running kernel
141415-10)
Thanks for giving me a clue about what was going on.
Doug
Dan McDonald
Doug <dy2...@gmail.com> wrote:
>
>Hi Dan,
Hello!
>Thanks for the tip. Both the tcp_xmit_hiwat and tcp_recv_hiwat
>parameters were set at 49152.
Those defaults are low given:
- Increases in network bandwidth.
- Increases in network latency thanks to brain-damage like NAT,
and ISP deep-packet inspection.
> I raised them to what you wrote. Now the download speed on the Solaris
>system is approx the same as the Linux ones.
As it should be (I'd hope even faster in some cases).
>Also, the value of tcp_max_buf is also set at 1048576. A thread from
>2007 says that it should be increased to more than the TCP transmit or
>recv high water mark values. Ref:
>http://groups.google.com/group/comp.unix.solaris/browse_frm/thread/196d18389f85842e
Yeah. That thread explains the basics about bandwidth * delay products, too.
>These Solaris servers are running Solaris 10 update 4 (8/07) but I
>have been patching them regularly since then (running kernel
>141415-10)
>
>Thanks for giving me a clue about what was going on.
No problem. I like happy customers! :)
solx
> In article <f87fc405-aa31-4c34...@g23g2000yqh.googlegroups.com>,
> Doug <dy2...@gmail.com> wrote:
>> Hi Dan,
>
> Hello!
>
>> Thanks for the tip. Both the tcp_xmit_hiwat and tcp_recv_hiwat
>> parameters were set at 49152.
>
> Those defaults are low given:
>
> - Increases in network bandwidth.
>
> - Increases in network latency thanks to brain-damage like NAT,
> and ISP deep-packet inspection.
>
>> I raised them to what you wrote. Now the download speed on the Solaris
>> system is approx the same as the Linux ones.
>
> As it should be (I'd hope even faster in some cases).
>
>> Also, the value of tcp_max_buf is also set at 1048576. A thread from
>> 2007 says that it should be increased to more than the TCP transmit or
>> recv high water mark values. Ref:
>> http://groups.google.com/group/comp.unix.solaris/browse_frm/thread/196d18389f85842e
>
> Yeah. That thread explains the basics about bandwidth * delay products, too.
>
>> These Solaris servers are running Solaris 10 update 4 (8/07) but I
>> have been patching them regularly since then (running kernel
>> 141415-10)
>>
>> Thanks for giving me a clue about what was going on.
>
> No problem. I like happy customers! :)
>
Should these values be set as the defaults in OpenSolaris/Solaris?
ndd -set /dev/tcp tcp_xmit_hiwat 1048576
ndd -set /dev/tcp tcp_recv_hiwat 1048576
My system is currently running Solaris 10 u5 and checking I found that
# ndd -get /dev/tcp tcp_max_buf
1048576
was already set to 1048576