Muni Computer Fouled Up From Start

3 views
Skip to first unread message

Bill Hough

unread,
Oct 6, 1998, 3:00:00 AM10/6/98
to
Muni Computer Fouled Up From Start
Audits show trouble with Canadian firm
for 6 years
Bill Wallace, Chronicle Staff Writer

Tuesday, October 6, 1998

The Muni Metro's new train control system was
plagued with defective parts and software
throughout the six-year project, according to
Municipal Railway documents obtained by The
Chronicle.

At one point last year, an audit of the construction
being done by Alcatel Transport Automation
Systems, the Canadian company that the Muni paid
$51 million to automate the underground portion of
its rail system, showed problems in 85 percent of
the equipment and software being tested.

More problems were identified this year, and some
have persisted even since the automated train
control system went online August 22. A Muni
listing prepared last month cites uncorrected
defects with emergency braking on entering stations
and train shutdowns and slow-ups in certain track
sections.

The documents give insight into the reason that the
Muni project was running more than two years late
by the time it began operation, to disastrous results.
For weeks after the switch,

repeated breakdowns, mismatched equipment and
driver confusion made for maddening delays for
Metro riders.

Hundreds of pages of contract materials and quality
assurance memorandums released by the Muni in
response to a Chronicle request under the state
Public Records Act show that Muni officials have
repeatedly expressed concern about the quality of
Alcatel's work over the past six years and have
found numerous defects in newly installed control
system parts built by the company or its
subcontractors.

Most of the problems have been corrected, and the
system is being debugged. However, the system's
history of technical problems probably will be a
major factor in the legal battle over any penalties
that Alcatel has to pay to the city as a result of the
delays and disruptions in Muni service.

The amounts could be enormous, as the Muni can
charge the Canadian firm as much as $5.1 million
as compensation for damages caused by technical
problems.

Alcatel officials did not respond to numerous
telephone calls and e-mail messages requesting an
interview for this report. But they already have
made public statements blaming Muni for the delays
and disruptions in bringing the new system on line.

In addition to its public statements, Alcatel has
filed
dozens of legal claims against the city that appear to
be directed at shifting responsibility for the new
system's problems to the Muni, arguing, among
other things, that the Muni did not give Alcatel
sufficient time to test the new system.

Although Alcatel's legal claims are still being
evaluated, Deputy City Attorney Marc Slavin, a
spokesman for City Attorney Louise Renne, said a
preliminary review indicated that none of them has
merit.

John Thomas, who was brought in by the Muni to
run the Alcatel train control project two years ago,
said some of the company's arguments were
``disingenuous.''

``They claim they did not have enough time to test
the trains before revenue service began,'' Thomas
said. ``But because of delays, they have had
months of extra time to complete their testing.''

Thomas said Alcatel's basic train control equipment
had to be extensively modified to meet the Muni's
technical requirements. Many of the most serious
problems did not surface until late in the game, he
said, after most of the hardware had been installed
and the software that runs the cars was being
tested.

``Most of the current fixes are software-related,''
he said. ``Until all the equipment is in place, you
really have nothing for the software to control.''

But as early as 1992, the year the contract to build
the Muni's new train control system was awarded
to Alcatel, Muni officials were dissatisfied with the
firm's quality control program -- a critical element
in
the overall project because it dictates how Alcatel
will find and correct defects in its equipment and
software.

Muni staff member Robert Stacy said in a memo
that year that Alcatel's plan ``is composed of
ambiguous statements that are supposed to be
impressive.'' He said the firm's plan did not
establish any clear system for finding and fixing
problems.

Despite their misgivings about Alcatel, the Muni
was stuck. Alcatel was the only one of the three
companies that submitted bids for the project that
could satisfy all the requirements the Muni had
imposed.

After the project began, Muni officials continued to
question Alcatel's quality control procedures. A
1995 memo written by Barbara Moy, Thomas'
predecessor as the Muni's Alcatel project manager,
refers to continuing problems in this area.

``We are also very concerned about the apparent
lack of oversight of your installation
subcontractor,''
Moy wrote in the memo to Alcatel manager
Duncan Lewis, noting that some items that needed
correction had gone unresolved for months after
they had been discovered.

In 1996, Rhodes Gardner, a quality control
engineer the Muni had hired to monitor the project,
wrote a memo complaining that Alcatel did not
seem to be sticking to the monitoring plan it had
outlined.

``Alcatel needs to address this issue, particularly at
a time when adequate quality control has been a
major Muni concern,'' Gardner wrote.

The concern Muni officials expressed about the
laxity of Alcatel's quality assurance program
seemed to be well placed, because defective
equipment and workmanship began to crop up as
the control system was installed.

During an inspection of eight of the light-rail
vehicles that had been improved for the new train
control system in late 1996, Muni engineers
discovered 25 defects that they attributed to
Alcatel subcontractors that made components for
the project.

An audit by the Muni as recently as September 14
uncovered such problems as trains flashing
incorrect destination signs, station train controllers
going off line for no reason, power glitches in the
Breda light-rail vehicles, system lockups in the
Muni Metro turnback near the Embarcadero
Station, and unexplained car slowdowns.

Several of the entries in the audit involved problems
docking light-rail cars at passenger stations,
including Embarcadero and Forest Hill. Others
showed difficulties in routing trains from the
Duboce Station onto the Market Street line, and in
turning back automatic trains at one Muni terminal.

Some of these problems had first been identified as
long ago as February but were still listed as ``open''
-- uncorrected -- in the audit three weeks ago.
--

Visit the Unofficial JFK Airport 50th Anniversary Page:
http://members.tripod.com/~psa188/index.html

CAVU Images:
http://www.geocities.com/CapeCanaveral/Hangar/3964/

Attend the Newark Airport Airline Collectible Show & Sale:
http://www.freeyellow.com/members/psa188/page1.html
http://www.freeyellow.com/members2/mrpanam/ewr.html

Steve Geller

unread,
Oct 7, 1998, 3:00:00 AM10/7/98
to
[regarding Alcatel]

> ``Most of the current fixes are software-related,''
> he said. ``Until all the equipment is in place, you
> really have nothing for the software to control.''
>
> But as early as 1992, the year the contract to build
> the Muni's new train control system was awarded
> to Alcatel, Muni officials were dissatisfied with the
> firm's quality control program -- a critical element in
> the overall project because it dictates how Alcatel
> will find and correct defects in its equipment and
> software.

I've never done train control software, but I have worked on a number
of software projects which went thru rough spots similar to Alcatel's.

Reading between the lines, I have some guesses about what might be
behind the current MUNI train control problem:

1. Failure to define the requirements. MUNI may be so demoralized and
politically driven that nobody did much more that rubber stamp
Alcatel's design.

2. No organized test plan. Any complicated software-driven system
needs extensive testing. There is no way to get it right the first
time. If no one at MUNI is in charge of quality control, the
contractor isn't going to do it for them. It may be that there was a
political push to "get it going" ready or not.

3. Design Malfunction. Nearly all of Alcatel's installed systems are
closed, like BART, not with routes running in traffic. It may be that
Alcatel tried to jam-fit their other software to fit MUNI.

4. Inadequate Training. The MUNI bosses may have failed to understand
how much of a learning curve there would be for the operators, and did
not budget enough training time. It is very difficult, of course, to
train people when the system itself is not working properly. The
natural reaction of the operators will be to defeat the automatic
stuff and go back to the old reliable manual control.


sge...@dsp.net

David A. Villa

unread,
Oct 7, 1998, 3:00:00 AM10/7/98
to
brhoug...@ibm.net posted:

> Muni Computer Fouled Up From Start

> Bill Wallace, Chronicle Staff Writer
> Tuesday, October 6, 1998

> Many of the most serious problems did not surface until
> late in the game, [John Thomas] said, after most of the


> hardware had been installed and the software that runs
> the cars was being tested.

<snip>

> Some of these problems had first been identified [by Muni]


> as long ago as February but were still listed as ``open''
> -- uncorrected -- in the audit three weeks ago.

The article says "the system is being debugged". The industry term for
this is fire-fighting. It helps to write the software well from the
beginning, or otherwise you end up with a bunch of patchwork that seems to
work but nobody can really verify.

Steve Geller wrote:

> 3. Design Malfunction. Nearly all of Alcatel's installed
> systems are closed, like BART, not with routes running in
> traffic. It may be that Alcatel tried to jam-fit their
> other software to fit MUNI.

I have no doubt they tried to do that. Under their $70 million budget
there's no way they could have started from the foundation (see: Other
agencies rave about new Muni control system). I don't know if you want to
blame Muni or Alcatel for a subcontractor's substandard work, but to me
this is quite definately the part that made it substandard. If I were to
have to guess I would say that operation will never be smooth, at least
not consistently, until the software is rewritten from the top.

> 4. Inadequate Training. The MUNI bosses may have failed
> to understand how much of a learning curve there would be
> for the operators, and did not budget enough training time.

This may have been partly true, I can't say for certain, but with the
effort they did make to train operators I don't think it's necessarily the
case.

> It is very difficult, of course, to train people when the

> system itself is not working properly. [...]

Excellent point, as are the others.

-David Villa

Richard Mlynarik

unread,
Oct 7, 1998, 3:00:00 AM10/7/98
to
From: sge...@dsp.net (Steve Geller)
Newsgroups: misc.transport.rail.americas,ba.transportation,misc.transport.urban-transit
Date: Wed, 07 Oct 1998 05:19:33 GMT

[For non ba.transportation background: Muni operates five different
"light rail" lines, all of which converge on a central two-track
subway under Market Street and terminate at stub-end terminal. Trains
can be formed from up to four vehicles. Recent changes have been the
installation of a moving-block signalling system ($75 million),
progressive vehicle replacement (circa $400 million), construction of
a new "turn back facility" (MMT) beyond the former stops of the
terminal station ($210 million), and extension of the line (MMX)
through the MMT to serve a new real estate development owned by a very
politically powerful corporation ($50 million). During peak periods a
total of approximately 25-30 trains per hour or approximately 45
vehicles per hour used the subway and terminal before the
"improvements".]

[See http://www.nycsubway.org/transitmaps/munimap.gif
and http://www.petrich.com/transit/Muni_Tracks/Muni_Tracks.html
for a rough diagram of the system layout. The former doesn't include
the MMT and MMX, they can be found in
http://www.petrich.com/transit/Muni_Tracks/E_Embarcadero.gif]

[regarding Alcatel]


> ``Most of the current fixes are software-related,''
> he said. ``Until all the equipment is in place, you
> really have nothing for the software to control.''
>
> But as early as 1992, the year the contract to build
> the Muni's new train control system was awarded
> to Alcatel, Muni officials were dissatisfied with the
> firm's quality control program -- a critical element in
> the overall project because it dictates how Alcatel
> will find and correct defects in its equipment and
> software.

I've never done train control software, but I have worked on a number
of software projects which went thru rough spots similar to Alcatel's.

Reading between the lines, I have some guesses about what might be
behind the current MUNI train control problem:

1. Failure to define the requirements. [...]

2. No organized test plan. [...]

3. Design Malfunction. [...]

4. Inadequate Training. [...]

While I agree with most of the points you made, there are three things
missing from this finger-pointing:

Most fundamental is the question of WHY Muni was installing a
multi-million dollar, newish-technology, moving-block signalling
system in a small-scale, medium-traffic, medium-headway railway system
in the first place.

The second is the basic engineering management failure to allow for
failure and arrange for fallbacks: Muni not only unwisely undertook a
whole slew of poorly-planned, capital-intensive, poorly-justified,
poorly-procured and poorly-managed projects simultaneously, but made
them all almost complete interdependent. The inevitable failure(s) of
any element of the capital program lead to system-wide meltdown with
no chance for graceful degradation to Plan B.

The third is the question of why an engineering fiasco continues to be
inflicted upon the Muni customers and tax-payers and residents of San
Francisco when decommissioning the system would _increase_ the level
of public transportation service in the city.


For the first part, the delivery of rail service in Muni's Market
Street subway has been constrained by the following (in rough order):
* Unreliability of equipment
++ primarily, Muni's inability to provide anywhere NEAR the number of
rail vehicles required to provide advertised and timetabled
service to customers
++ secondarily, the effects of rail vehicle failure while in service.
* Lack of line supervision; any delay on any part of the rail system
has been unmonitored and uncorrected, leading to huge service gaps
which individual train operators are unable to unilaterally act to fill
* Unreliability of operator staffing
Muni hasn't been employing enough train operators to provide the
timetabled service, nor rigorously ensured that those on the payroll
work when customer demand demands it
* Traffic congestion at the terminal station
Trains on circa two-minute headways converging and terminating at a
single pair of terminal platforms and reversing back via a single
scissors crossover (see <URL:news:8686951...@dejanews.com>
aka http://www.dejanews.com/getdoc.xp?AN=256359236 for a longer
description.) can lead to contention for paths across the cross-over
and backups for incoming trains.
* Counter-productive work rules.
Until recently it was the case that a two-car train required two
operators, one of whom spent half his or her time with no
responsibilities other than fare collection.
It is still the case that reassignment of a vehicle and its operator
from its scheduled run to any other service (ie reallocating service
to where they are most needed because of delays or loadings)
requires a premium to be paid to the operator.

Note that, due to an inability to provide trains to provide service,
line capacity in the Market Street subway has never been an issue, and
will not be an issue for many years, if ever. The new signalling
system, advertised as expanding the line capacity of the Market Street
subway by reducing headways between trains, is SIMPLY UNNECESSARY.

Note also that line capacity measured in _trains_ per hour is a quite
different matter from what most affects customers of the system, which
is reliable headways on each of the lines combined with adequate
passenger capacity measured in_vehicles_ per hour.

The moving-block signalling system and automatic train control is not
only overkill for a small-scale operation with few trains on
undemanding headways, but it COMPLETELY FAILS to address the real
problems.

Now to the second matter, that of the unnecessary inter-connectedness
of capital projects.

Muni has recently spent over $700 million on capital projects of
extraordinarily dubious efficacy.

* Discussed above it the ATCS signalling system. It does not solve any
operational problem, was budgeted below $30 million, was contracted to
Alcatel at around $50 million, and presently represents a waste of
nearly $80 million (including nearly $30 million in "facilitiation"
fees paid to the Booz-Hamilton consultancy.)

Functioning of the ATCS system is entirely dependent upon installation
of on-board equipment on each of the vehicles which will use it. Muni
"planned" that it would have replaced its existing fleet of unreliable
and aging Boeing LRV streetcars with a new fleet of vehicles before
ATCS was introduced, and so would not need to require backward
compatability of ATCS with existing cars. So ATCS functioning at all
depended upon successful delivery of a complete fleet of...

* New streetcars. Muni specified a custom, globally-unique design and
was only able to attract a single bid, from Breda Ferroviarie, to
build it. The resulting cars are grossly over-weight, are grossly
over-cost by world metrics (just under $3 million per 25m vehicle!)
and have proven extremely unreliable. It is fashionable within Muni
and within San Francisco politics to lay all blame for the complete
disintegration of the rail system at the feet of the (admittedly)
"obsolete", "unmaintainable" "aging" and "mis-designed" Boeing LRV
cars; what is conveniently not mentioned is that the fleet
availability of the Breda LRV2's is only "approaching" that of the
much-maligned Boeings, according to testimony from Muni's own
politically-appointed director. It is quite extraordinary that such
huge sums of money could be spent on such not addressing the problem.

When it became apparent that the years-late ATCS system would be
placed in operation years before the more-years-late fleet replacement
program, it was decided to outfit a portion of the Boeing LRV fleet
with ATCS equipment. The present-day result is a fleet of 52
ATCS-compliant Bredas, 58 ATCS-compliant Boeings, and 20 non-ATCS
Boeings. (Note that, in another spectacular example of mis-planning,
Breda and Boeing cars cannot couple together to form trains.)
This is down from a peak fleet of nearly 130 Boeings. The
formerly-scheduled peak hour demand was for 99 cars, more recently
reduced (though un-announced service cutbacks) to 91. Before ATCS
introduction, Muni was consistently fielding fewer than 65 vehicles;
this figure has in fact _dropped_ since ATCS introduction. Nobody
appears to bat an eye when it is reported that Muni is running far
less than 70% of its advertised service.

* Muni Metro Turnback. The addition of this track, which forms a
small subterranean yard beyond the former end of the system, is the
only part of Muni's planning which made operational sense.

<out---\-/----------------------\-------/----\--------from MMX<
X PLATFORM >----\/-----/\----
<in----/-\----------------------/------\----/-----------to MMX>

By adding a new track (combining storage and reversing functions) and
adding new cross-overs, this facility allows for Muni to increase the
predicability of service in the subway, to add some redundancy to one
of the more problematic of the multiple single points of system-wide
failure (the scissors crossing denoted "X" above) and to add some
operational flexibility (by providing locations to store reserve or
disabled vehicles.)

Unfortunately, and professionally inexcusably, operation of the MMT
was made entire dependent upon ATCS, so this useful, on-time,
under-budget project was held hostage to a wasteful and unnecessary
signalling system which in turn depended upon a wasteful and
poorly-procured complete vehicle fleet replacement program.


* Muni Metro Extension. This line serves no transportation use, but
came about for a combination of reasons.

The first is that a large and extraordinarily lucrative property
development (Mission Bay, owned by Catellus Development, bankrolled by
the City of San Francisco) is about to get under way once again. A
gift of a $50+ million dollar rail line (the eventual cost will be
closer to $200 million as MMX is extended further into Mission Bay
under the guise of Muni's "Third Street Light Rail Line"
http://thecity.sfsu.edu/~thirds -- a Catellus subsidy and
gentrification engine masquerading as transportation to Bayview and
Visitacion Valley) to a politically powerful developer when cheaper,
faster bus service could have provided better service is about par for
politics in San Francisco.

To add insult to injury, the design of the development was turned on
its head (apparently to make it more palatable to the taxpayers who
are footing the bill, by asking them to pay for services to a
university annex as well as those for Catellus residential, office and
retail development) after MMX had been planned and nearly constructed,
and so the present terminal station of the constructed system will be
isolated and useless when the Third Street project starts extending
further along a new and different alignment.

To add a final slap in the face, the _entire_ MMX alignment will be
largely redundant if the planned Third-Stockton Street subway -- a
$450 million gift to Catellus masquerading as transportation to
Chinatown -- is built to bypass it in a decade or so.

The second is that Muni's staff are itching -- as the ATCS procurement
fiasco amply demonstrates -- to get their hands on the wads of money
and transit-professional prestige which appears to come with rail
extensions in this county. Building MMX gets them a toe-hold in the
federal New Rail Starts pork-barrel stakes. Improving the chronic
reliability problems of the existing rail system, improving the
chronic reliability problems of the much larger and under-capitalized
bus and trolley-bus system, planning for and making needed service
changes to respond to evolving demand, advocating for exclusive
vehicle rights of way on existing streets, improving efficiency and
safety, providing a more pleasant experience for riders and so forth
and so on are extremely unglamourous compared to planning and building
and attending the opening of spanky new, multi-hundred-million-dollar
rail extensions... even when, or especially when, the same
transportation need could have been filled less than one quarter of
the cost by significantly improving bus service (trolley bus
electrification, exclusive right of way, signal preemption, low-floor
vehicles.)

Note that Operation of the MMX was dependent upon completion of the
MMT (through which it runs) and hence on ATCS and Breda procurement.

The oveall third question is: "Why this fiasco is being inflicted on
San Francisco?"

Inception of the ATCS system was an unmitigated disaster. Despite
over three years of evening and weekend closures of the subway to
revenue service, borne by Muni's riders through unreliable and
inadequate "replacement" bus shuttle service, the ATCS system
completely failed. Trains were delayed for hours. Weeks later, the
problems are not fully resolved.

Rather than reconsider whether operation of this expensive boondoggle --
which solves no pressing or even medium-term problem of the system --
should be suspended, Muni's management, out of some combination of
political directive and professional embarrassment, has persisted in
operation of the system. Unfortunately, this has happened to the
direct cost of the customers of the Muni system.

* In order to improve "reliability" of ATCS, the 20 available non-ATCS
Boeing cars have been removed from service. Muni's already
broken-down, under-sized, chronically-unreliable streetcar fleet was
thus immediately reduced by 15%.

* In order to prove some sort of political point, the MMX line --
which carries only around 3000 riders PER DAY, hardly enough to
justify a bus line -- it is being served by through service from what
was Muni's most heavily-used and what is now Muni's most
chronically-unreliable rail line: the N-Judah. Since the additional
round-trip made by N trains from Embarcadero to the MMX terminus on
King Street and back again takes approximately an additional 20
minutes on top of the former N line's approximately 80-minute
schedule, this represents a 20% DECREASE in the level of service along
the entire N line, and approximately a 5% decrease in the level of
service in the congested Embarcadero-Civic Center section of the
rail system.

* Partly in order to appear to be using the ATCS "investment", and
partly because chronic and partly-ATCS-induced vehicle shortages offer
no alterantive, Muni has cut down the number of multi-car trains in
the subway and dramatically increased the number of uncoupled single
cars which are run. This enables their director to grand-stand and
boast of how many "trains per hour" are being run in the new regime
without mentioning either that overall system passenger-carrying
_capacity_ has been grossly reduced or that those trains are _still_
not dispatched in any rational, demand-reflecting manner. Lots of
"trains per hour" does one little good if the headways on a particular
line are not maintained and if, when a car for the desired line
finally appears, it is so crowded that boarding is physically
impossible.

* Presumably in order to justify the MMT, _no_ trains are being turned
at the Embarcadero scissors crossing. (In a rational world one would
have expected the MMT and the existing crossover to function together
as a system to improve overall system throughput.) Running trains out
through Embarcadero, into MMT and back is an operation which requires
from three to nine minutes more than reversing at the platform. This
represents approximately a further 5% degradation in overall revenue
service provision.


So, the result is:

Muni planning started a large number of expensive projects.
The cost-effectiveness, and indeed the usefullness at any price, of a
number of those projects is extraordinarily poor.

Muni planning designed those projects so that they success of any one
of them depends on most of the others.
ATCS: Breda
MMT: ATCS
MMX: MMT
MMX: ATCS

There were no contingency plans for what to do if any component
failed; say, late Breda delivery or non-functioning ATCS. Individual
failures are seldom predictable in any engineering undertaking, but
the possibility must always be allowed for. Failure to do so was an
engineering management and agency management disaster of the first
rank, and no amount of finger-pointing at other agencies or at
contractors or at past administrations can obscure this fact.

The result of this lack of fallback is a $700 million "investment"
which has to be "made" to "work" because it has become a matter of all
or nothing.

The result of that is that service on the street to actual tax-paying,
fare-paying, San Francisco residents, workers and visitors has gone to
hell so that Muni officials and their political bosses can boast about
increasing the number of "trains per hour" and can have their photos
taken at ceremonies inaugurating service on new rail lines.

Another result is that the appalling state of the much larger and much
more patronized remainder of the Muni system -- the diesel and
electric buses lines -- has been been further swept under the carpet.
Muni Metro rail service -- even in its present disasterous and reduced
state -- is in many ways superior to that on many bus lines, and is
certainly receiving a disproportionate amount of attention, and will
continue to receive a disproportionate amount of investment.

Reply all
Reply to author
Forward
0 new messages