POWER: San Mateo/San Francisco power outage report
Sean Donelan
San Francisco/San Mateo power outage on December 8, 1998.
At 8:15am workers making changes to 115-kilovolt lines at the San Mateo
substation failed to properly ground the system. When they turned on
the power, it caused a low frequency condition and tripped protective
breakers on the substation. It also caused a cascade effect, tripping
the Hunters Point and Potrero power plants. This isolated San Mateo
County and northward to parts of San Francisco City from the electrical
grid primary power sources. Aside from the human error, the system worked
"as designed to protect the rest of the grid."
Approximately 375,000 customers were without power, including San
Francisco Airport (SFO), the Pacific Stock Exchange, traffic lights,
and cable cars. BART (subway) used emergency power to bring trains
to the next station, and then shutdown. Since the electrical grid
was isolated, PG&E had to "blackstart" the system on a circuit by
circuit basis.
The SFO airport power was restored at 9:10AM. The San Francisco Office
of Emergency Services was activated at 9:22AM according to EDIS.
At 1:00PM about 200,000 customers were still without power. PG&E
estimated that full restoration would be completed by 2:15PM, but
some customers were still without power through the afternoon.
No major injuries have been reported directly attributable to the
power failure. No major Internet provider reported any network
problems directly attributable to the power failure.
Since this outage was caused by a PG&E error, customers may make
claims for some types of damages. Other damages may be covered
by private insurance.
http://www.pge.com/resources/claim_form/claim_form.html
In other news:
On December 7, 1998 PG&E donated $87,000 to San Fransico's
Emergency Services Communications Center to install extra back up
electrical facilities for the center. The new facility will work
with PG&E during major service interruptions to establish restoration
priorities.
--
Sean Donelan, Data Research Associates, Inc, St. Louis, MO
Affiliation given for identification not representation
Jerry Scharf
just a minor note ont he initial cause. They grounded the system just fine,
then just forgot to pull the safety grounds before they repowered the line.
As for "power grid outages shouldn't spread", there's a rule in that biz, it's
always easier/faster to reset 10 circuits that replace something. If you leave
a turbine generator up in a situation where output load is whacked, you could
be looking at a several week outage. The reason it took them so long to get
the plants back on line is they have to do a complete system inspection to
make sure the breakers tripped in time.
One other thing to remember, there isn't a whole lot of risk working on a
router outage. When you're restoring electric service, you have to go slower
so you keep the possibility of frying a lineman to a minimum. I spent a few
weeks installing power line electronic equipment in a 115kv substation and got
a whole new level of respect. You couldn't pay me enough to do that work (but
I like their trucks.)
jerry
Chris A. Icide
What you know about building a network does not apply to building a power
generation and distribution system. Imagine what you might design if a
large BGP flap could physically destroy a POP.
A small variance in voltage or frequency can cause significant damage not
only to the transmission and generation facilities, but also to end user
equipment. When you lose or gain significant load on a distribution
system, it causes the frequency to rise or fall. When the frequency varies
significantly from 60Hz, you begin to catastrphically destroy generator
sets and sensitive transmission gear. So, trip circuits are set to
identify a frequency change that is significant and to trip the gen sets
off-line. At this point if you don't isolate the problem, it becomes a
domino efffect. The load generating capacity keeps dropping as the demand
remains the same. The only option is to isolate the failed grid area, and
then slowly bring the grid back on-line bit by bit.
-Chris
At 09:04 AM 12/9/98 -0500, Jon Zeeff wrote:
>
>> breakers on the substation. It also caused a cascade effect, tripping
>> the Hunters Point and Potrero power plants.
>
>> grid primary power sources. Aside from the human error, the system worked
>> "as designed to protect the rest of the grid."
>
>I suggest that causing a cascade effect and increasing the area of an
>outage isn't a good way to design a system.
>
Chris A. Icide / ch...@skycache.com
VP Engineering/Operations
SkyCache / www.skycache.com
(v) 301-598-0500 x2235
Karl Denninger
> One other thing to remember, there isn't a whole lot of risk working on a
> router outage. When you're restoring electric service, you have to go slower
> so you keep the possibility of frying a lineman to a minimum. I spent a few
>weeks installing power line electronic equipment in a 115kv substation and got
> a whole new level of respect. You couldn't pay me enough to do that work (but
> I like their trucks.)
>
> jerry
Yeah, your first mistake around that kind of energy is almost always your last.
I used to a lot of work with embedded control systems in very-high-power
microwave transmitter gear (ie: 25KW stuff). Those things run around
with several tens of KV at ~2A (yes, that's AMPS) in the HV drawer. Even
"pedestrian" HPA hardware (3kw Cband stuff) has a couple hundred ma in the
HV drawer, which at 20kv is more than enough to fry you very dead.
If you manage to route that kind of energy through yourself (and it WILL jump
air gaps at that voltage) they don't even bother with paramedics - just call
the coroner. That ignores the risk from the emitted microwave energy
itself, which is substantial as well (directly in the front of the antenna
the ERP of these things can be in the range of a couple of MW - more than
enough to cause you all kinds of physiological trouble (like death))
And that's NOTHING compared to the energy levels running around in circuits
at 115kv substations!
--
--
Karl Denninger (ka...@denninger.net) http://www.mcs.net/~karl
I ain't even *authorized* to speak for anyone other than myself, so give
up now on trying to associate my words with any particular organization.
CON...@pjm.com
> -----Original Message-----
> From: jze...@verio.net [SMTP:jze...@verio.net]
> Sent: Wednesday, December 09, 1998 9:04 AM
> To: SE...@SDG.DRA.COM
> Cc: na...@merit.edu
> Subject: Re: POWER: San Mateo/San Francisco power outage report
>
>
> > breakers on the substation. It also caused a cascade effect, tripping
> > the Hunters Point and Potrero power plants.
>
> > grid primary power sources. Aside from the human error, the system worked
> > "as designed to protect the rest of the grid."
>
> I suggest that causing a cascade effect and increasing the area of an
> outage isn't a good way to design a system.
>
>
> If you're interested in finding our more about how the power grid is operated,
> my company has a lot of operational info documented on the web. It's scope
> is the operation of the interconnected grid in the mid-Atlantic region of the
> US (PA-NJ-DE-MD-DC-VA). The concepts and procedures documented there are
> probably very similar to the ones used on the west coast (YMMV, I'm a
> networking tech not a power systems engineer).
>
> All of our Operations Manuals are available at: http://pubs.pjm.com
>
> Of specific interest might be: Emergency Operations
> http://pubs.pjm.com:80/dynaweb/PJMpubs/m13/@Generic__BookView and
> Transmission Operations
> http://pubs.pjm.com:80/dynaweb/PJMpubs/m03/@Generic__BookView
>
>
> Chuck Conway
> Network Management
> PJM Interconnection, LLC
> con...@pjm.com
Jay R. Ashworth
> > grid primary power sources. Aside from the human error, the system worked
> > "as designed to protect the rest of the grid."
>
> I suggest that causing a cascade effect and increasing the area of an
> outage isn't a good way to design a system.
I suggest that you don't understand power distribution and transmission
very well. Just like any other generator, if you load a power station
too heavily, the voltage and the frequency will both be pulled down.
Since the tolerances on public electricity are something like +-5v and
+- .5 Hz., they really have no choice but to break ties when a plant or
segment goes out. They really shouldn't cross-tie power on a running
basis as much as they probably do... but since the Green folks won't
always let them build plants...
Cheers,
-- jr 'Damn I'm glad I don't run a power company...' a
--
Jay R. Ashworth j...@baylink.com
Member of the Technical Staff Buy copies of The New Hackers Dictionary.
The Suncoast Freenet Give them to all your friends.
Tampa Bay, Florida http://www.ccil.org/jargon/ +1 813 790 7592