Bay Area backbone lag
Philip J. Koenig
>Philip J. Koenig wrote:
>> Makes me wonder if there was some sort of backbone problem
>> in the Bay Area, i.e. at the Pacbell NAP, MAE-WEST, Sprint,
>> MCI, or ?? This was between 4-7 pm or so.--
>And here I kept telling my boss that www.sun.com really wasn't the
>problem, (Yesterday). It seemed I was successful 1 out of 7 retries.
>I ran some router traces which returned acknowledgement that the
>target hosts were up. If someone can confirm what the problem was
>I would like to know.
Maybe all the RBOCs and backbone providers were using "NetDay 96" as a cover for
a coordinated router downtime to install NSA/CIA sniffers to monitor traffic as
per the telecom/fcc legislation.
(Hehe, I'm not paranoid... :-)
Phil
Edgar Nielsen
Most of the day thursday, internetMCI had problems at their
SF(hayward) POP that caused extremely high packet loss between
some sites. Here is some of the explanation they gave once they had
resolved the problem late in the evening:
****
There was an IS-IS problem between borderx1.hay (San Francisco) and cpe1.hay
(The Barrnet CPE router). The problem caused cpe1.hay to continuously loss
routing to the border1.hay and border3.hay. Anyone who peered with us (MCI)
on either of those routers would have been unable to reach the cpe1.hay router
or anything behind it in the Barrnet cloud. The problem appears to have been
fixed when on of our 3rd level engineers cleared the CLNS neighbors on
Borderx1.hay.
****
edgar nielsen
TLGnet staff
Marcos Della
>
> Philip J. Koenig wrote:
>
> > Makes me wonder if there was some sort of backbone problem
> > in the Bay Area, i.e. at the Pacbell NAP, MAE-WEST, Sprint,
> > MCI, or ?? This was between 4-7 pm or so.--
> And here I kept telling my boss that www.sun.com really wasn't the
> problem, (Yesterday). It seemed I was successful 1 out of 7 retries.
> I ran some router traces which returned acknowledgement that the
> target hosts were up. If someone can confirm what the problem was
> I would like to know.
The problem actually stemed out of the 2300 WALSH complex in Santa
Clara. A Backhoe managed to dig up a Natural Gas main right in front of
our building. The fire department evacuated the area and had everyone
shut down power, etc and wouldn't let either WilTel or us turn on
generators to power our equipment. Once the batteries died, it was all
over not only for us, but for PSI, CIS, AGIS, NetCom SC Node, InterNex,
and the WilTel POP (30-50% of all phone traffic routes through there).
I always was worried about fiber cuts or loss of power. This is the
first time I had ever heard of the fire department forcing an outage. I
know what I lost in monetary worth, but I'm even afraid to estimate what
WilTel lost. According to the engineers miling around, this was the
first time that this node had been offline since it was built several
years ago.
Anyway, the mail did spark and had a blazing fire for over 2 hours while
the fire department was trying to prevent it from melting the place
down. PG&E finally ended up redigging down and up stream of the cut and
crushing the pipe. We were back up within minutes and WilTel completed
their poweron tests in a little over 30 minutes.
The big problem is that both CIX, AGIS, and us route a significant
amount of traffic out of Mae-West. Once these were down, it all
defaulted over to Sprint and MCI. Since MCI is riding on the edge as it
is, adding more fuel to the fire didn't help...
Marcos
William Coney
: >
: The big problem is that both CIX, AGIS, and us route a significant
: amount of traffic out of Mae-West. Once these were down, it all
: defaulted over to Sprint and MCI. Since MCI is riding on the edge as it
: is, adding more fuel to the fire didn't help...
Could you elaborate on the MCI comment? I'm considering using their frame
relay services to link 5 of my corporate sites in a WAN.
Thanks and regards.
Jonathan Heiliger
: Could you elaborate on the MCI comment? I'm considering using their frame
internetMCI's capacity issues, and MCI's frame relay service are un-related.
Not the same hardware by any means.
-jh-
Sean Donelan
> generators to power our equipment. Once the batteries died, it was all
> over not only for us, but for PSI, CIS, AGIS, NetCom SC Node, InterNex,
> and the WilTel POP (30-50% of all phone traffic routes through there).
I believe only AC power was effected. DC power continued through the
entire incident.
--
Sean Donelan, Data Research Associates, Inc, St. Louis, MO
Affiliation given for identification not representation
Jim Dixon
> In article <314AF8...@InterNex.Net>, Marcos Della <MDe...@InterNex.Net> writes:
> > generators to power our equipment. Once the batteries died, it was all
> > over not only for us, but for PSI, CIS, AGIS, NetCom SC Node, InterNex,
> > and the WilTel POP (30-50% of all phone traffic routes through there).
>
> I believe only AC power was effected. DC power continued through the
> entire incident.
Customers like Agis supply their own battery backup. Agis's batteries
went down after an hour or so.
An interesting minor point is that many small components (CSU/DSUs, for
example) were casually plugged into commercial power: they had no backup
at all; everyone trusted that Wiltel's backup generator would kick in.
This meant that many circuits failed immediately, although routers kept
on functioning.
--
Jim Dixon VBCnet GB Ltd +44 117 929 1316 fax +44 117 927 2015
http://www.uk.vbc.net VBCnet West +1 408 971 2682 fax +1 408 971 2684
Sean Donelan
> Customers like Agis supply their own battery backup. Agis's batteries
> went down after an hour or so.
We went with WorldComm battery backup even though its only available for
DC powered equipment. I believe it's rated for 10+ hours. This time our
bet paid off for us. We didn't lose power at any time through the entire
incident. Maybe next time we won't be so lucky.
> An interesting minor point is that many small components (CSU/DSUs, for
> example) were casually plugged into commercial power: they had no backup
> at all; everyone trusted that Wiltel's backup generator would kick in.
There are tremendous variations in how people wire their POP's. It is a
pain to follow CO quality standards, but there is a reason. There are a
lot of providers in the Santa Clara POP. An interesting minor point is
which providers are you not hearing about. Some providers aren't as
casual where they plug things in.