internet slowdown
prad
our cable company (shaw cable) connecting our internal network to the
outside. the servers do websites, email and nameserve (bind94)
sometimes one or both become sluggish letting us connect to websites.
the ping times become 10+ times as long.
pinging the servers from the outside also slows down.
whenever we reboot the servers, everything is fine again for several
hours and then the speed starts fluctuating and then eventually
everything slows down again. (however the several hours is not certain
either because i just rebooted and everything slowed down after a few
minutes this time).
i tried /etc/netstart and that speeded things up for a few pings only.
doing a systat shows the cpu to be idle more than 90% of the time.
we are presently trying to figure out netstat.
we never experienced this issue with freebsd 6.3 which we ran for
several months. we've run freebsd7 since the beginning of june and this
problem seemed to start showing up within the first week.
i am not sure where to look in order to solve this problem -
specifically which log files might provide some clues regarding the net.
suggestions please?
--
In friendship,
prad
... with you on your journey
Towards Freedom
http://www.towardsfreedom.com (website)
Information, Inspiration, Imagination - truly a site for soaring I's
_______________________________________________
freebsd-...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questi...@freebsd.org"
Bill Moran
> we have 2 freebsd7 servers using pf with static ip addresses through
> our cable company (shaw cable) connecting our internal network to the
> outside. the servers do websites, email and nameserve (bind94)
>
> sometimes one or both become sluggish letting us connect to websites.
> the ping times become 10+ times as long.
> pinging the servers from the outside also slows down.
>
> whenever we reboot the servers, everything is fine again for several
> hours and then the speed starts fluctuating and then eventually
> everything slows down again. (however the several hours is not certain
> either because i just rebooted and everything slowed down after a few
> minutes this time).
>
> i tried /etc/netstart and that speeded things up for a few pings only.
> doing a systat shows the cpu to be idle more than 90% of the time.
> we are presently trying to figure out netstat.
>
> we never experienced this issue with freebsd 6.3 which we ran for
> several months. we've run freebsd7 since the beginning of june and this
> problem seemed to start showing up within the first week.
>
> i am not sure where to look in order to solve this problem -
> specifically which log files might provide some clues regarding the net.
>
> suggestions please?
Is the console responsive when this happens? It doesn't sound like a
CPU issue to me -- ICMP messages are low enough CPU overhead that the
system would have to be pretty badly bogged for CPU starvation to cause
problems.
What do your network traffic graphs show during the slowdowns? Perhaps
some sort of network flood occurring?
--
Bill Moran
http://www.potentialtech.com
prad
Bill Moran <wmo...@potentialtech.com> wrote:
> Is the console responsive when this happens?
>
when it bogs it takes ages to even ssh into the server.
> It doesn't sound like a
> CPU issue to me -- ICMP messages are low enough CPU overhead that the
> system would have to be pretty badly bogged for CPU starvation to
> cause problems.
>
agreed - otherwise why would the cpu be idle.
> What do your network traffic graphs show during the slowdowns?
> Perhaps some sort of network flood occurring?
>
we are looking at flow-tools unless you have any other suggestions.
there has been no problems for 2+ days, but we are going to follow
through on this anyways and be prepared.
thank you for your ideas bill and i'll let you know if we learn
anything.
--
In friendship,
prad
... with you on your journey
Towards Freedom
http://www.towardsfreedom.com (website)
Information, Inspiration, Imagination - truly a site for soaring I's
Jonathan Chen
> On Mon, 16 Jun 2008 08:23:32 -0400
> Bill Moran <wmo...@potentialtech.com> wrote:
>
> > Is the console responsive when this happens?
> >
> when it bogs it takes ages to even ssh into the server.
You do realise that this particular symptom usually points to reverse-lookup
errors. I would examine your DNS server logs.
--
Jonathan Chen <jo...@chen.org.nz>
----------------------------------------------------------------------
"Lots of folks confuse bad management with destiny"
- Kin Hubbard
Bill Moran
> On Tue, Jun 17, 2008 at 10:05:14PM -0700, prad wrote:
> > On Mon, 16 Jun 2008 08:23:32 -0400
> > Bill Moran <wmo...@potentialtech.com> wrote:
> >
> > > Is the console responsive when this happens?
> > >
> > when it bogs it takes ages to even ssh into the server.
>
> You do realise that this particular symptom usually points to reverse-lookup
> errors. I would examine your DNS server logs.
It could be, but it depends. I've also seen this problem as a result
of high load and sshd taking a long time to spawn a child process
because of the load.
Prad, once you _are_ logged in, is the server responsive? Or does the
sluggishness persist?
--
Bill Moran
http://www.potentialtech.com
prad
Jonathan Chen <jo...@chen.org.nz> wrote:
> You do realise that this particular symptom usually points to
> reverse-lookup errors. I would examine your DNS server logs.
>
i will look, but we are using the same named.conf and zone files as we
did when we had 6.3 where we had no problems like this. also, would
there be such sporadic issues? shouldn't a reverse dns problem be
consistently problematic?
--
In friendship,
prad
... with you on your journey
Towards Freedom
http://www.towardsfreedom.com (website)
Information, Inspiration, Imagination - truly a site for soaring I's
prad
Bill Moran <wmo...@potentialtech.com> wrote:
> Prad, once you _are_ logged in, is the server responsive? Or does the
> sluggishness persist?
>
there is no problem once we are in - the cpu is idle 95% and you can
move around and do stuff without any slowdown at all.
right now, at 11:30am pst we were experiencing very high activity due to
a press release done in toronto on canadian horse slaughter from this
site: http://defendhorsescanada.org
the slowdown is upon us too - but i don't know if it is related or not
because the ping times are varying from 100ms to 1000+ms to the same
site(s). it's all over the place!
what is really weird is that the machine that hosts that site isn't the
only one which has slowed down. the other machine which only serves
email experiences the same thing (slow ssh connection, long ping times).
we are going to bypass the servers by using one of our dynamic ip
addresses and see what it is like then. if that computer experiences a
slowdown, i think it may suggest that the problem is due to network
activity within the cable company. if there is no slowdown, then that
seems to point the problem exclusively to our servers.
--
In friendship,
prad
... with you on your journey
Towards Freedom
http://www.towardsfreedom.com (website)
Information, Inspiration, Imagination - truly a site for soaring I's
Bill Moran
> On Wed, 18 Jun 2008 09:16:09 -0400
> Bill Moran <wmo...@potentialtech.com> wrote:
>
> > Prad, once you _are_ logged in, is the server responsive? Or does the
> > sluggishness persist?
> >
> there is no problem once we are in - the cpu is idle 95% and you can
> move around and do stuff without any slowdown at all.
Sounds like DNS, either that or a forking issue. Run ssh with -v to see
details of where the process pauses. You can also run sshd in foreground
mode on the server to see lots of debugging information.
> right now, at 11:30am pst we were experiencing very high activity due to
> a press release done in toronto on canadian horse slaughter from this
> site: http://defendhorsescanada.org
>
> the slowdown is upon us too - but i don't know if it is related or not
> because the ping times are varying from 100ms to 1000+ms to the same
> site(s). it's all over the place!
Sounds to me like your network is overwhelmed. You need to get some
management stuff online -- what is the rate of traffic through each of
the interfaces involved?
> what is really weird is that the machine that hosts that site isn't the
> only one which has slowed down. the other machine which only serves
> email experiences the same thing (slow ssh connection, long ping times).
I'm confused as to why you think this is related to the machine when there
are multiple machines involved? It sure sounds like a network issue, from
the description of the symptoms.
If you're _absolutely_ sure the problem started occurring with the 7
upgrade, I'd look at the possibility that the NICs you're using aren't
as well supported in 7 as they were in 6. Have you verified all the
speed/duplex settings are matched?
> we are going to bypass the servers by using one of our dynamic ip
> addresses and see what it is like then. if that computer experiences a
> slowdown, i think it may suggest that the problem is due to network
> activity within the cable company. if there is no slowdown, then that
> seems to point the problem exclusively to our servers.
That's also a good diagnostic step.
--
Bill Moran
http://www.potentialtech.com
prad
Bill Moran <wmo...@potentialtech.com> wrote:
> > what is really weird is that the machine that hosts that site isn't
> > the only one which has slowed down. the other machine which only
> > serves email experiences the same thing (slow ssh connection, long
> > ping times).
>
> I'm confused as to why you think this is related to the machine when
> there are multiple machines involved?
>
i may have stated the case poorly. i thought is was a 7 issue not a
machine issue.
> If you're _absolutely_ sure the problem started occurring with the 7
> upgrade, I'd look at the possibility that the NICs you're using aren't
> as well supported in 7 as they were in 6. Have you verified all the
> speed/duplex settings are matched?
>
we don't know how to do this, but will try to find out. however, see
below, since i no longer think the problem is 7.
> > we are going to bypass the servers by using one of our dynamic ip
> > addresses and see what it is like then. if that computer
> > experiences a slowdown, i think it may suggest that the problem is
> > due to network activity within the cable company. if there is no
> > slowdown, then that seems to point the problem exclusively to our
> > servers.
>
> That's also a good diagnostic step.
>
ok i am 99% convinced now that this has nothing to do with freebsd 7 at
all. here's why:
1. there are 2 servers involved and they both get affected so as you
say, bill "It sure sounds like a network issue, from the description of
the symptoms." if it were a 7 issue, then there is no reason for them
to be affected simultaneously - but if it is a network issue, they
would experience the slowdowns together which they do.
2. by-passing the servers produced mirror results. the machine on the
dynamic address displayed an identical ping pattern to the same sites
as those going through the servers. this would suggest that the problem
has nothing to do with our servers.
3. we just found that on more than one instance when there was heavy
activity on the website server, the pinging rate was low. this suggests
that our servers are more than capable of handling the load and are not
slowing our access down at all.
4. though i said we didn't have the problem with 6.3 initially which
was true, this doesn't mean that the problem lies with 7 - all it
means is that we didn't notice anything wrong with 6.3 while we used
it. if this is a network issue from the outside, it may have started
recently and merely coincides with our upgrade to 7. also, my son
recalls that on rare occasions 6.3 may have acted somewhat slower than
usual (though nothing like what we are experiencing now).
5. i don't see why there would be a dns problem since we are sshing
from within our local network which the servers are part of. in fact,
it is slow even when you use the ip address directly.
so my present conclusion is that bottlenecks may be developing in the
'vicinity' of our assigned ip addresses (static and dynamic). these may
not even be the fault of our cable company possibly, but i'll check
with them anyway again.
the only nagging matter though is why sshing in sometimes becomes slow.
i will do as you suggest though bill (and jonathan) and produce some
network graphs and look at the dns. it will be good to become familiar
with these things since part of the reason for setting up our home
servers was to gain an education about this stuff.
i really appreciate the interest you've shown in our little problem and
will follow through on your earlier suggestions as well as any others
you may have.
--
In friendship,
prad
... with you on your journey
Towards Freedom
http://www.towardsfreedom.com (website)
Information, Inspiration, Imagination - truly a site for soaring I's
Bill Moran
> from within our local network which the servers are part of. in fact,
> it is slow even when you use the ip address directly.
Because the ssh server always does DNS lookups on connecting IPs in
order to have hostnames to put in the logs. If DNS is sluggish,
unresponsive, or configured poorly, it will cause long delays during
login.
--
Bill Moran
http://www.potentialtech.com
John Almberg
> say, bill "It sure sounds like a network issue, from the
> description of
> the symptoms." if it were a 7 issue, then there is no reason for them
> to be affected simultaneously - but if it is a network issue, they
> would experience the slowdowns together which they do.
Do you have PF installed? pftop is a quick way to see what's
happening on the network interface.
-- John
prad
John Almberg <jalm...@identry.com> wrote:
> Do you have PF installed? pftop is a quick way to see what's
> happening on the network interface.
>
thanks john that's a good utility!
also, thanks bill for the explanation of the ssh problem.
we've found the problem we think after a lot of testing and working with
the fellow from the cable company.
it was our old 10T hubs and nics. we'd put together our server with bits
and pieces given to us or bought from the salvation army or value
village. it was a homeschooling project for my son and me. it's worked
just fine for us until this media blitz that started recently.
normally, our most active sites might get 10000 hits a month, but
what's been happening is we're exceeding that per day. june 11 was
35000+ and yesterday was 44500+
that's what seemed to be bogging everything down.
so we bought some 100T switches and nics. we also went to cat6 cabling
from cat5 and it looks like we're keeping up with the demand nicely now.
now we're thinking about replacing our old 700MHs servers (one has
192M the other 128M), with something more powerful. but that's going
to be a subject for another thread.
all the assistance here is very much appreciated!
--
In friendship,
prad
... with you on your journey
Towards Freedom
http://www.towardsfreedom.com (website)
Information, Inspiration, Imagination - truly a site for soaring I's