Question about performance issues: Latency problem?
Michael Russell
I've run across a problem with performance using a database app across a
certain part of our network. In the office, we're running 100BaseT; it's a
small office, only 8 stations. The server is a single processor 350MHz P3,
1GB memory, RAID 5 disk array. The main app being used is Great Plains
Dynamics, using Pervasive SQL 2000i as the database. Performance in the
office is great.
The problem is at our remote office, which is about 1 mile away, connected
to the main office via fiber. Dynamics (and a FoxPro app) take about 3
times as long to complete large tasks from the remote office as they do
locally. The topology between the locations looks like this:
Main office/Server-->Hub-->Switch-->Fiber-->Switch-->Hub-->remote office
computer
Watching traffic, the network is only hitting about 10-15% utilization,
almost no collisions. We ran the NT Performance Monitor on the server, CPU
stayed around 30-40% (except a couple brief spikes), Memory was constant at
around 400 MB used, disk queue length was low.
My first reaction to help the situation is to add a second NIC to the server
and remove the hubs between the server and the remote computer:
Server-->Switch-->Fiber-->Switch-->remote computer.
It seems that the hub between the server and the office switch removes any
benefit a switch might provide.
The fiber is supposed to be a 100Mb link. My main question is: how much
latency does this distance bring into the picture? My assumption is that
it's significant, in that it will be noticeably slower to run network apps
from the remote computer than from the office.
Thanks,
Michael
Bill Somerville
packet sizes (e.g. > MTU)? Latency can be increased by retransmits, for
instance.
-- Bill
"Michael Russell" <mr_b1...@hotmail.com> wrote in message
news:fBbl8.1583$0_3.1...@newsfeed.slurp.net...
Manfred Kwiatkowski
"Michael Russell" <mr_b1...@hotmail.com> writes:
>Hi,
>I've run across a problem with performance using a database app across a
>certain part of our network. In the office, we're running 100BaseT; it's a
>small office, only 8 stations. The server is a single processor 350MHz P3,
>1GB memory, RAID 5 disk array. The main app being used is Great Plains
>Dynamics, using Pervasive SQL 2000i as the database. Performance in the
>office is great.
>
>The problem is at our remote office, which is about 1 mile away, connected
>to the main office via fiber. Dynamics (and a FoxPro app) take about 3
>times as long to complete large tasks from the remote office as they do
>locally. The topology between the locations looks like this:
>
>Main office/Server-->Hub-->Switch-->Fiber-->Switch-->Hub-->remote office
>computer
>
>Watching traffic, the network is only hitting about 10-15% utilization,
>almost no collisions. We ran the NT Performance Monitor on the server, CPU
You cannot run 100BaseFX half duplex over 1 mile. If the distance is
greater than 412 m (1300 ft) you will have to set both ends to full
duplex manually.
--
Manfred Kwiatkowski kwiat...@zrz.tu-berlin.de
John Rowe
> The fiber is supposed to be a 100Mb link. My main question is: how much
> latency does this distance bring into the picture?
Short answer: none worth talking about.
Long answer: one mile is 1.6 km, ie a 3000m round trip. The speed of
light is 300 metres per microsecond so that equates to around ten
microseconds. I'm not a fibre optics person but I have a gut feeling
that the speed of signals in fibre may be a bit less than that so
guess a total of 10-20 microseconds. For comparison, two servers in
the same room give me a ping time of around 160 microseconds.
John
wall...@kmsi.net
Do you know whether those switches are running store-and-forward or
cut-through? S&F can significantly increase propagation delay on large
packets due to extra serialization delays.
You might try some ping tests at varying packet sizes. And some raw bulk
file transfers might provide a sanity check on the link throughput.
But if you really need maximum data access performance at a remote
location, you need to quite moving the data. Look at a thin client
solution. Terminal server or a custom-written web app, say.
/kenw
Ken Wallewein
Calgary, Alberta
ke...@kmsi.net
www.kmsi.net
Donald McLachlan
> "Michael Russell" <mr_b1...@hotmail.com> wrote:
>
> >Hi,
> >I've run across a problem with performance using a database app across a
> >certain part of our network. In the office, we're running 100BaseT; it's a
> >small office, only 8 stations. The server is a single processor 350MHz P3,
> >1GB memory, RAID 5 disk array. The main app being used is Great Plains
> >Dynamics, using Pervasive SQL 2000i as the database. Performance in the
> >office is great.
> >
> >The problem is at our remote office, which is about 1 mile away, connected
> >to the main office via fiber. Dynamics (and a FoxPro app) take about 3
> >times as long to complete large tasks from the remote office as they do
> >locally. The topology between the locations looks like this:
> >
> >Main office/Server-->Hub-->Switch-->Fiber-->Switch-->Hub-->remote office
> >computer
> Do you know whether those switches are running store-and-forward or
> cut-through? S&F can significantly increase propagation delay on large
> packets due to extra serialization delays.
>
> You might try some ping tests at varying packet sizes. And some raw bulk
> file transfers might provide a sanity check on the link throughput.
>
> But if you really need maximum data access performance at a remote
> location, you need to quite moving the data. Look at a thin client
> solution. Terminal server or a custom-written web app, say.
>
> /kenw
>
> Ken Wallewein
> Calgary, Alberta
> ke...@kmsi.net
> www.kmsi.net
Seems to me this is mostly a transaction processing (latency) problem, not a
throughput problem. In order to maintain database consistency, you must perform
a single transaction at a time, so you probably cannot stream a whole bunch
of data across the link. More likely request the record, receive the record,
modify the record, send the record, ack. Then proceed to the next record.
That would appear to add 2 RTT's to every transaction.
Even if you have a thin client, the requests/acks must traverse the link and
will slow things down.
What is the RTT from the remote clients to the server (ping)?
--
Donald McLachlan E-mail Donald.M...@crc.ca
Communications Research Centre / RNS Tel (613) 998-2845
3701 Carling Ave., Fax (613) 998-9648
Ottawa, Ontario
K2H 8S2
Canada
Geo
Your server should definately be moved to the main office switfh and
confirmed that it has negotiated for Fast Ethernet using Full Duplex. You
didn't mention if the remote net was still operating at 10Mbit shared, but I
would move the critical users to the remote office switch also, even it it's
only 10T on the desktop switch.
As long as your fiber link has negotiated Fast Ethernet using Full
Duplex, you can go to 2Kmeters without an appreciably noticeable delay due
to distance.
Assuming all devices are behaving properly, simply placing the server
on its own switch port you should increase the percievable performance of
your distant systems. By moving the heavy demand users (or all) to the
remote switch ports, you should be able to "raise the bar" over what
performance the remote users are experiencing in the current setup.
Best of luck,
George
"Michael Russell" <mr_b1...@hotmail.com> wrote in message
news:fBbl8.1583$0_3.1...@newsfeed.slurp.net...
p...@icke-reklam.ipsec.nu
> Hi,
> I've run across a problem with performance using a database app across a
> certain part of our network. In the office, we're running 100BaseT; it's a
> small office, only 8 stations. The server is a single processor 350MHz P3,
> 1GB memory, RAID 5 disk array. The main app being used is Great Plains
> Dynamics, using Pervasive SQL 2000i as the database. Performance in the
> office is great.
> The problem is at our remote office, which is about 1 mile away, connected
> to the main office via fiber. Dynamics (and a FoxPro app) take about 3
> times as long to complete large tasks from the remote office as they do
> locally. The topology between the locations looks like this:
> Main office/Server-->Hub-->Switch-->Fiber-->Switch-->Hub-->remote office
> computer
> Watching traffic, the network is only hitting about 10-15% utilization,
> almost no collisions. We ran the NT Performance Monitor on the server, CPU
> stayed around 30-40% (except a couple brief spikes), Memory was constant at
> around 400 MB used, disk queue length was low.
> My first reaction to help the situation is to add a second NIC to the server
> and remove the hubs between the server and the remote computer:
> Server-->Switch-->Fiber-->Switch-->remote computer.
> It seems that the hub between the server and the office switch removes any
> benefit a switch might provide.
The _real question_ is : how many interactions does this application needs
for a transaction. Easily answered by a measurement.
Apps using 1000 question-answer sequences will take 1000 times your
ping time to get an answer.
This is all to common, and yes, it's the sign of a broken ( or badly
designed application)
There is no cure except for :
a/ rewite
b/ use X-windows , or citrix / terminal server technology.
> The fiber is supposed to be a 100Mb link. My main question is: how much
> latency does this distance bring into the picture? My assumption is that
> it's significant, in that it will be noticeably slower to run network apps
> from the remote computer than from the office.
> Thanks,
> Michael
--
Peter Håkanson
IPSec Sverige (At the Riverside of Gothenburg, home of Volvo)
Sorry about my e-mail address, but i'm trying to keep spam out.
Remove "icke-reklam" and it works.
wall...@kmsi.net
>In article <n63c9u0jgfprhpqa4...@4ax.com>, wall...@kmsi.net writes:
...
>> But if you really need maximum data access performance at a remote
>> location, you need to quite moving the data. Look at a thin client
>> solution. Terminal server or a custom-written web app, say.
>>
>> /kenw
...
>Seems to me this is mostly a transaction processing (latency) problem, not a
>throughput problem. In order to maintain database consistency, you must perform
>a single transaction at a time, so you probably cannot stream a whole bunch
>of data across the link. More likely request the record, receive the record,
>modify the record, send the record, ack. Then proceed to the next record.
>That would appear to add 2 RTT's to every transaction.
>
>Even if you have a thin client, the requests/acks must traverse the link and
>will slow things down.
No, they wouldn't. That's the whole point. If they do, it's not a thin
client. The data should stay on the LAN between the data server and the
thin client server.
>...
Michael Russell
some sort of remote control/terminal server setup.
Michael
<wall...@kmsi.net> wrote in message
news:n63c9u0jgfprhpqa4...@4ax.com...
Arie Goldberg
"Michael Russell" <mr_b1...@hotmail.com> wrote in message
news:fBbl8.1583$0_3.1...@newsfeed.slurp.net...
Most software sould not be sensitive to the fiber delay, its in a range that
most software would not see.
Two cases could cause packet loss and result in slow performance and
timeouts:
1. When both sides use half-duplex and the distance is over 400 meters. You
must be in full duplex (and if using a switch, it must be set to full duplex
on its fiber port) in order to avoid late undetected collisions and
undetected loss of packets.
2. When the two ends are set to different duplex modes, one to half-duplex
and the other to full-duplex. This causes undetected collisions since the
full-duplex side does not behave.
In both case retries are performed by the software. This causes slowdowns,
sometimes actual timeouts (depends on how busy the net is and the software
behaviour).
My suggestion, check the two sides for duplex modes.
One easy way to test this theory is using Ping (with -t parameter) from A to
B and B to A and watch for ping timeouts, if you see any, its probably
because some get lost in transit (due to undetected collisions).
--
Arie Goldberg
Omnitron Systems Technology, Inc.
Manufacturer of Fiber Media Converters
agol...@omnitron-systems.com
http://www.omnitron-systems.com
Phone: (949) 250-6510 / Fax: (949) 250-6514