---------- Original Message ----------
From: Ed Finnell <Efinn...@AOL.COM>
To: IBM-...@bama.ua.edu
Subject: Check out Computer glitch to cause flight delays across U.S. - MarketWatch
Date: Thu, 19 Nov 2009 09:53:16 EST
_Computer glitch to cause flight delays across U.S. - MarketWatch_
(http://www.marketwatch.com/story/computer-glitch-to-cause-flight-delays-across-us-2
009-11-19)
Probably some 4th grader in lower Slobovia playing a slightly modified
version of 'Blaster'....
----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to list...@bama.ua.edu with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html
____________________________________________________________
Hotel
Hotel pics, info and virtual tours. Click here to book a hotel online.
http://thirdpartyoffers.juno.com/TGL2141/c?cp=_IgLn09dr_BJVyAT1G2KEQAAJ1DMCvI8C_EKS4yGKHO7DleMAAYAAAAAAAAAAAAAAAAAAADNAAAAAAAAAAAAAAAAAAATRAAAAAA=
----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to list...@bama.ua.edu with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html
"The FAA is implementing the Stratus servers, which use Intel Xeon 2.8 MHz large cache MP processors and support the Microsoft Windows operating system, at control centers in Atlanta and Salt Lake City.
Uninterrupted availability of the NADIN 1 is important to all aspects of the aviation industry, as well as the nation's economy and, increasingly, as a tool to help protect national security. After thorough technology and product evaluations, Stratus was chosen as best able to provide an open platform with 99.999 percent uptime reliability - which is mandatory for running an application as important as NADIN 1 - together with the required caliber of maintenance, logistical support, and long service life."
http://findarticles.com/p/articles/mi_m0EIN/is_2005_March_14/ai_n12936121/
Silly me: I had thought that Windows and 99.999 uptime was an oxymoron.
-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-...@bama.ua.edu] On Behalf Of ess...@juno.com
Sent: Thursday, November 19, 2009 7:02 AM
To: IBM-...@bama.ua.edu
Subject: Re: Check out Computer glitch to cause flight delays across U.S. - Mar ketWatch
They probably moverd off of ACP/TPF to a smaller platform.
Typical for a non mainframe Platform
---------- Original Message ----------
From: Ed Finnell <Efinn...@AOL.COM>
To: IBM-...@bama.ua.edu
Subject: Check out Computer glitch to cause flight delays across U.S. - MarketWatch
Date: Thu, 19 Nov 2009 09:53:16 EST
_Computer glitch to cause flight delays across U.S. - MarketWatch_
(http://www.marketwatch.com/story/computer-glitch-to-cause-flight-delays-across-us-2
009-11-19)
Probably some 4th grader in lower Slobovia playing a slightly modified
version of 'Blaster'....
NOTICE: This electronic mail message and any files transmitted with it are intended
exclusively for the individual or entity to which it is addressed. The message,
together with any attachment, may contain confidential and/or privileged information.
Any unauthorized review, use, printing, saving, copying, disclosure or distribution
is strictly prohibited. If you have received this message in error, please
immediately advise the sender by reply email and delete all copies.
<snip>
After thorough technology and product evaluations, Stratus was chosen as
best able to provide an open platform with 99.999 percent uptime reliability
</snip>
Guess they got their .001.
Dennis Roach
GHG Corporation
Lockheed Martin Mission Services
Facilities Design and Operations Contract Strategic Technical Engineering NASA/JSC
Address:
2100 Space Park Drive
LM-15-4BH
Houston, Texas 77058
Mail:
P.O. Box 58487
Mail Code H4C
Houston, Texas 77258
Phone:
Voice: (281)336-5027
Cell: (713)591-1059
Fax: (281)336-5410
E-Mail: Dennis...@LMCo.COM
All opinions expressed by me are mine and may not agree with my employer or any person, company, or thing, living or dead, on or near this or any other planet, moon, asteroid, or other spatial object, natural or manufactured, since the beginning of time.
> -----Original Message-----
> From: IBM Mainframe Discussion List [mailto:IBM-...@bama.ua.edu] On
> Behalf Of Hal Merritt
> Sent: Thursday, November 19, 2009 10:49 AM
> To: IBM-...@bama.ua.edu
> Subject: Re: Check out Computer glitch to cause flight delays across
> U.S. - Mar ketWatch
>
>The news item mentioned a 'NADIN' failure. A chain of Googles lead to a
document dated March, 2005: "FAA modernizing National Airspace Data
Interchange Network with Stratus ftServer systems; Stratus to provide 10-
year logistical and service support"
>
>"The FAA is implementing the Stratus servers, which use Intel Xeon 2.8 MHz
large cache MP processors and support the Microsoft Windows operating
system, at control centers in Atlanta and Salt Lake City.
The problem seems to me to be that they are using Xeon processors operating
in the 2.8 MHz (megahertz) speed range instead of their normal 2.8 GHz
(gigahertz) speed. (Or else the article got it wrong -- but that can't be since
those are "professional journalists".)
Seriously UNDERclocked processors?
--
cheers,
Alex
about that period, we marketed ha/cmp for 1-800 against stratus. at the
time stratus took down system for software maintenance (would have
outage that precluded even five-nines). ha/cmp could have fall-over
between multiple servers as part of maint. strategy ... limiting outage
to few seconds. the response was that then stratus could install
multiple servers in (software fall-over) HA configuration ... to
eliminate maint. outage window. the response was then why spend for the
hardware redundancy
as an aside their box was also being relogo'ed and marketed as the S/88.
this recent post
http://www.garlic.com/~lynn/2009p.html#85 Anyone going to Supercomputers '09 in Portland
has some old "marketplace news" abstracts (29jun92) mentioning
remarketing branded stratus box ... the article might have a
misstatement ... the amount was what was paid for the program ... not
that much was actually sold.
as an aside ... i believe FAA has had some number of Flex-ES
installations (running mainframe software on intel platforms) ... old
post mentioning Flex-ES also available on stratus boxes
http://www.garlic.com/~lynn/2003d.html#10 Low-end processors (again)
for random other drift ... recent posts about old Jim Gray paper that
by early 80s, majority of outages had shifted from hardware faults to
other things
http://www.garlic.com/~lynn/2009.html#5 Is SUN going to become x86'ed ??
http://www.garlic.com/~lynn/2009.html#39 repeat after me: RAID != backup
http://www.garlic.com/~lynn/2009.html#47 repeat after me: RAID != backu
including scan of old copy of a version of presentation that I had
laying around ... and put up on the web:
http://www.garlic.com/~lynn/grayft84.pdf
I've mentioned before Jim and I had something of dust-up at acm sigops
in '91 about whether commodity components could be used for HA operation
... he was still at DEC and their party-line was vax/cluster. later when
he moved to m'soft ... he was up on stage for the announcement of m'soft
HA fallover product.
--
40+yrs virtualization experience (since Jan68), online at home since Mar1970
>Well, 99.999% reliability means that 1/10000 of the time the thing is not
>reliable = about 8 hours per year.
Well, 99.999% reliability means that 1/100000 of the time the thing is not
reliable = about 0.1 hours per year.
>Not to confuse with z/VM reliability that was 99.999999 if I recall it correctly
>back in 1991 or so for VM/ESA 1.0.
-- gil
>>Well, 99.999% reliability means that 1/10000 of the time the thing is not
>>reliable = about 8 hours per year.
>
>Well, 99.999% reliability means that 1/100000 of the time the thing is not
>reliable = about 0.1 hours per year.
Of course, we also have to agree on what "reliable" means.
And Windows is about as "open" as was the former Soviet Union.
-jc-
--
Mark Pace
Mainline Information Systems
1700 Summit Lake Drive
Tallahassee, FL. 32317
-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-...@bama.ua.edu] On Behalf Of Nemo
Sent: Thursday, November 19, 2009 9:08 AM
To: IBM-...@bama.ua.edu
Subject: Re: Check out Computer glitch to cause flight delays across U.S. - Mar ketWatch
NOTICE: This electronic mail message and any files transmitted with it are intended
exclusively for the individual or entity to which it is addressed. The message,
together with any attachment, may contain confidential and/or privileged information.
Any unauthorized review, use, printing, saving, copying, disclosure or distribution
is strictly prohibited. If you have received this message in error, please
immediately advise the sender by reply email and delete all copies.
----------------------------------------------------------------------
Well, 99.999% reliability means that 1/10000 of the time the thing is
not
reliable = about 8 hours per year.
Not to confuse with z/VM reliability that was 99.999999 if I recall it
correctly
back in 1991 or so for VM/ESA 1.0.
<SNIP>
Let me try to put this into some perspective. This is the flight plan
system. Its failure means that flight plans must be entered by FAA
personnel manually.
Let us look at ATC reliability. If ATC were to have a 99.9% reliability
at Chicago O'Hare -- 1 fatal accident per hour. 99.999 was 1 per month.
I attended an FAA presentation where they were putting the percentages
of reliability into terms that had meaning for the average person (in
that presentation, the average person was a licensed pilot).
Regards,
Steve Thompson
-- Opinions expressed are those of the poster and may not reflect those
of poster's employer --
And finally, why are we spending so much time on obviously incomplete and inaccurate articles?
re:
http://www.garlic.com/~lynn/2009q.html#26 Check out Computer glitch to cause flight delays across U.S. - MarketWatch
when I was out marketing ha/cmp,
http://www.garlic.com/~lynn/subtopic.html#hacmp
I coined the terms "disaster survivability" and "geographic surviability"
http://www.garlic.com/~lynn/submain.html#available
and I got asked to write a section in the corporate continuous
availability strategy document ... but it got pulled because both
rochester & pok objected (because at the time, they couldn't meet the
requirements).
after we left, we spent some time with large national financial
network. they claimed 100% availability for extended number of years was
because of
1) ims hot-standby
2) automated operator
... the ims hot-standby involved machines separated by geographic
distances (failures had shifted primarily to human mistakes and local
environmental conditions).
some of this was back to when my wife had been con'ed into doing a stint
in POK responsible for loosely-coupled architecture ... where she
created "peer-coupled shared data" architecture
http://www.garlic.com/~lynn/submain.html#shareddata
... which, except for ims hot-standby ... saw very little uptake until
sysplex. that and the battles with the communication group oper whether
SNA had to be used for loosely-coupled coordination, resulted in her not
staying long in the position.
Because we take a perverse pleasure in seeing squatty box failures when
the reliability (FSVO reliability) of a Mainframe is called for?
Or to keep it on topic.
We wish to learn from others mistakes. Unfortunately that would require
fairly accurate info so the various theories abound..
-----Original Message-----
Schwarz, Barry A
And finally, why are we spending so much time on obviously incomplete
and inaccurate articles?
----------------------------------------------------------------------
http://gcn.com/articles/2009/11/19/faa-software-hackers-delays.aspx?s=FAAnewsalert
Dennis Roach
GHG Corporation
Lockheed Martin Mission Services
Facilities Design and Operations Contract Strategic Technical Engineering NASA/JSC
Address:
2100 Space Park Drive
LM-15-4BH
Houston, Texas 77058
Mail:
P.O. Box 58487
Mail Code H4C
Houston, Texas 77258
Phone:
Voice: (281)336-5027
Cell: (713)591-1059
Fax: (281)336-5410
E-Mail: Dennis...@LMCo.COM
All opinions expressed by me are mine and may not agree with my employer or any person, company, or thing, living or dead, on or near this or any other planet, moon, asteroid, or other spatial object, natural or manufactured, since the beginning of time.
> -----Original Message-----
> From: IBM Mainframe Discussion List [mailto:IBM-...@bama.ua.edu] On
> Behalf Of Howard Brazee
> Sent: Thursday, November 19, 2009 2:11 PM
> To: IBM-...@bama.ua.edu
> Subject: Re: Check out Computer glitch to cause flight delays across
> U.S. - Mar ketWatch
>
>Looks like a router failure
>
>http://gcn.com/articles/2009/11/19/faa-software-hackers-delays.aspx?
s=FAAnewsalert
>
4 hours to fix a router? Good Grief.
>On Thu, 19 Nov 2009 10:48:49 -0600, Hal Merritt wrote:
>
>
>
>>The news item mentioned a 'NADIN' failure. A chain of Googles lead to a
>>
>>
>document dated March, 2005: "FAA modernizing National Airspace Data
>Interchange Network with Stratus ftServer systems; Stratus to provide 10-
>year logistical and service support"
>
>
>>"The FAA is implementing the Stratus servers, which use Intel Xeon 2.8 MHz
>>
>>
>large cache MP processors and support the Microsoft Windows operating
>system, at control centers in Atlanta and Salt Lake City.
>
>
>The problem seems to me to be that they are using Xeon processors operating
>in the 2.8 MHz (megahertz) speed range instead of their normal 2.8 GHz
>(gigahertz) speed. (Or else the article got it wrong -- but that can't be since
>those are "professional journalists".)
>
>Seriously UNDERclocked processors?
>
>
>
--------------------------------------------------------------------------
How about "seriously under-brained journalists"??? Would be far more
typical.
Rick
>> <>Looks like a router failure
>>
>> http://gcn.com/articles/2009/11/19/faa-software-hackers-delays.aspx?s=FAAnewsalert
>
>
>4 hours to fix a router? Good Grief.
>
>
---------------------------------<unsnip>-------------------------------
That's why the squatty boxen have "Administrators", rather than
"Programmers" or "Systems Programmers". Can we speak of IQ levels here?
Or maybe "training and experience" ??
Rick
>---------------------------------<snip>---------------------------------
>
>>> <>Looks like a router failure
>>>
>>> http://gcn.com/articles/2009/11/19/faa-software-hackers-delays.aspx?s=FAAnewsalert
>>
>>
>>4 hours to fix a router? Good Grief.
>>
>>
>---------------------------------<unsnip>-------------------------------
>That's why the squatty boxen have "Administrators", rather than
>"Programmers" or "Systems Programmers". Can we speak of IQ levels here?
>Or maybe "training and experience" ??
>
>Rick
>
Would a router used with system z be under the control of the system z
staff or a general communications staff? If the latter, why the
communications staff in a z environment be any more or less responsive
than the staff in other environments? If I understood the story, the
router that fouled things was in Salt Lake City, not either of the
facilities with servers.
An aside, is Stratus still either owned or marketed by IBM? Does it
use a special version of Windows designed for fall over and take over?
Tom Kelman
Enterprise Capacity Planner
Commerce Bank of Kansas City
(816) 760-7632
> -----Original Message-----
> From: IBM Mainframe Discussion List [mailto:IBM-...@bama.ua.edu] On
> Behalf Of scott
> Sent: Thursday, November 19, 2009 7:12 PM
> To: IBM-...@bama.ua.edu
> Subject: Re: Check out Computer glitch to cause flight delays across
U.S.
> - Mar ketWatch
>
*****************************************************************************
If you wish to communicate securely with Commerce Bank and its
affiliates, you must log into your account under Online Services at
http://www.commercebank.com or use the Commerce Bank Secure
Email Message Center at https://securemail.commercebank.com
NOTICE: This electronic mail message and any attached files are
confidential. The information is exclusively for the use of the
individual or entity intended as the recipient. If you are not
the intended recipient, any use, copying, printing, reviewing,
retention, disclosure, distribution or forwarding of the message
or any attached file is not authorized and is strictly prohibited.
If you have received this electronic mail message in error, please
advise the sender by reply electronic mail immediately and
permanently delete the original transmission, any attachments
and any copies of this message from your computer system.
*****************************************************************************
Eric Bielefeld
Sr. Systems Programmer
IBM Global Services Division
Dubuque, Iowa
414-477-7259
----- Original Message -----
From: "Kelman, Tom" <Thomas...@COMMERCEBANK.COM>
Newsgroups: bit.listserv.ibm-main
To: <IBM-...@bama.ua.edu>
Sent: Friday, November 20, 2009 9:23 AM
Subject: Re: Check out Computer glitch to cause flight delays across U.S. -
Mar ketWatch
> That's what I was thinking. A system this critical and there is no
> backup and no failover for 7*24 uptime. Just think - this is one of the
> systems that controls our airline flights. I was talking with someone
> that I work with who is actually an ex-employee of one of the major
> airlines. The airlines have to electronically file their flight plans
> into this system a certain number of hours before the pilots are allowed
> to leave the ground. That's what caused all the delays and
> cancellatioins - no flight plan filed - no flight. I'm about to take a
> flight in a couple of weeks, and the fact that the system seems to have
> no backup/failover process is very scary to me. I sure hope that there
> are better failover facilities for the actual flight controllers' system
> that's used once the plane is in the air.
>
> Tom Kelman
> Enterprise Capacity Planner
> Commerce Bank of Kansas City
> (816) 760-7632
----------------------------------------------------------------------
It's unfortunate that there isn't a technical journal devoted to the
topic of systems failures on all platforms. It would be fascinating
to learn about the many ways that systems can fail.
Instead, each failure is kept as a closely guarded secret, and as a
profession, we learn nothing.
A wise systems architect once told me, in another context: "Don't
tell me how it works, tell me how it *fails*"
That's what I was thinking. A system this critical and there is no
backup and no failover for 7*24 uptime. Just think - this is one of the
systems that controls our airline flights. I was talking with someone
that I work with who is actually an ex-employee of one of the major
airlines. The airlines have to electronically file their flight plans
into this system a certain number of hours before the pilots are allowed
to leave the ground. That's what caused all the delays and
>>
There's software that will do network analysis or
you can roll your own with cut-set theory. My eyes rolled over the
first few chapters of Balabanian and Bickert's Electrical Network Theory,
but once the lights went on umm that's good stuff.
The ACM used to have a mailing list that did just that, named RISKS. I haven't been subscribed for years and years, so I don't know if it still exists or not. And when you say "all platforms" they meant it. ATMs, gasoline pumps, and so on were also discussed.
Mark Post
> >>> On 11/20/2009 at 10:38 AM, Don Leahy <don....@LEACOM.CA> wrote:
> -snip-
> > It's unfortunate that there isn't a technical journal devoted to the
> > topic of systems failures on all platforms. It would be fascinating
> > to learn about the many ways that systems can fail.
>
> The ACM used to have a mailing list that did just that, named RISKS. I
> haven't been subscribed for years and years, so I don't know if it still
> exists or not. And when you say "all platforms" they meant it. ATMs,
> gasoline pumps, and so on were also discussed.
>
http://catless.ncl.ac.uk/Risks
re:
http://www.garlic.com/~lynn/2009q.html#26 Check out Computer glitch to cause flight delays across U.S. - MarketWatch
http://www.garlic.com/~lynn/2009q.html#28 Check out Computer glitch to cause flight delays across U.S. - MarketWatch
we had project a decade ago with some of the people that had done the
1960s implementation ... and later left to form their own company to do
various things.
these were modified 360/50s that ran in triple configuration.
this comes up in the virtual machine folklore. the cambridge science
center was trying to get a 360/50 to build their hardware modifications
for supporting virtual memory ... but because all the available 360/50s
were going to air traffic control ... the science center had to settle
for 360/40. the first virtual machine system then was "cp/40" (instead
of "cp/50") ... morphing into cp67 (when they got a standard 360/67 that
came with hardware virtual memory support) ... which later morphed into
vm370. misc. past posts mentioning science center
http://www.garlic.com/~lynn/subtopic.html#545tech
some of that early 360 FAA software was moved along to various platforms
... including some of it eventually running in Flex-ES virtual machines
on intel platforms (possibly even stratus intel ... as mentioned in
previous post).
in the late 80s, we started ha/cmp product ... misc. past posts
http://www.garlic.com/~lynn/subtopic.html#hacmp
and part of that effort, we did detailed availability studies of tcp/ip
and various tcp/ip environments.
there have also been some number of (failed) FAA "modernization"
efforts. About the time we were doing ha/cmp ... there was such an
effort using triple-RS6000s as basic component. Because of our work on
high availability ... we were periodically asked to participate in some
of the discussions/reviews. There were some number of interesting
failure-modes that they overlooked ... and it was one of the
"modernization" efforts that ran into difficulty.
After we left in the 90s, we were invited in to consult with small
client/server startup that wanted to payment transactions ... the
startup had also invented this technology called "SSL" they wanted
to use.
Two of the people that we worked on ha/cmp for parallel Oracle
... referenced in jan92 meeting mentioned in this post
http://www.garlic.com/~lynn/95.html#13
were at this startup in charge of something called the "commerce
server". As part of doing what is now frequently called "electronic
commerce", we deployed something called the "payment gateway" ... misc.
past posts
http://www.garlic.com/~lynn/subnetwork.html#gateway
which acts as gateway for payment transactions between webservers on the
internet and payment infrastructure. An early prototype of this had a
situation where a merchant called up not being able to do transactions.
Normally, payment trouble-call desk has 5mins elapsed time to do first
level problem determination. In this case, three hours later, the
trouble ticket was closed with no-trouble-found.
In order to try and approach the non-internet availability ... we
deployed payment gateway on HA/CMP configuration with multiple diverse
routes ("telco provisioning") into different parts of the internet. we
also had to deploy/invent some number of compensating procedures to
compensate for vaguries of the internet ... as well as to compensate for
large number of identified security vulnerabilities.
One of the issues ... was that I had planned on also
broadcast/advertising the different routes ... but in the period of the
deployment; the internet backbone transitioned to hierarchical routing.
As a result, the remaining alternate path mechanism had to rely on
multiple-A record support in DNS (where the client is provided multiple
ip-address records in response to request domain lookup). The client
then cycles through each of the addresses until it finds one that makes
a succesful connection.
In any case, part of that deployment (for "electronic commerce")
included compensating for large number of different kinds of failures
that might happen anywhere in the internet infrastructure.
If they were running on 3081s, that would be a big innovation.
Last I heard they were still on 2050s (stripped down 360/50s).
Gerhard Postpischil
Bradford, VT
Eric Bielefeld
Sr. Systems Programmer
IBM Global Services Division
Dubuque, Iowa
414-477-7259
re:
http://www.garlic.com/~lynn/2009q.html#29 Check out Computer glitch to cause flight delays across U.S. - MarketWatch
a little ATC history:
http://www.centennialofflight.gov/essay/Government_Role/Air_traffic_control/POL15.htm
from above:
In December 1993, the FAA reviewed its order for the planned AAS. IBM
was far behind schedule and had major cost overruns. In 1994 the FAA
simplified its needs and picked new contractors. The revised
modernization program continued under various project names. Some
elements met further delays. In 1999, controllers began their first use
of an early version of the Standard Terminal Automation Replacement
System, which included new displays and capabilities for approach
control facilities. During the following year, FAA completed deployment
of the Display System Replacement, providing more efficient workstations
for en route controllers.
... snip ...
A little more historical (some of the documents are PDF)
http://www.faa.gov/about/history/
http://www.faa.gov/about/history/photo_album/air_traffic_control/index.cfm?cid=automation
I had some interaction with one of the FAA's Air Route Traffic Control Centers in May, 1978. The IBM mainframes then being used by the FAA were specialty versions of the S/360 models 50 and 65 that could do multiprocessing, meaning two or more CPUs sharing central storage. Around 1982 I heard of an effort to upgrade the equipment to whatever was the latest and greatest CPUs at the moment when the FAA first started researching what to ask for in their next major upgrade. It was probably 3081s. Since it is now 27 years since that conversion effort started, I suspect that it is about time for the FAA to upgrade again to computer equipment that became obsolete two years ago.
A. The various non-intelligence-community parts of the federal government do their computer acquisitions and upgrades like this (with only a modicum of exaggeration):
1. Top management (an oxymoron already) decides to do an upgrade. This might not be precipitated until there is intense political pressure to do so, such as too many airplanes colliding in mid-air, 100 million tax returns processed incorrectly and very late, Al Qaeda hacks into the FBI's most secret system, the Pentagon admits they have lost track of the last 25 trillion dollars worth of budget information going back to 1866, Sean Hannity reveals that the Obama administration has detailed dossiers on all 7 billion people on earth and 2 billion space aliens disguised as Rottweilers, etc.
2. Top management produces some kind of amorphous framework document, gives it to a low-level employee who can just barely spell MVS, and he is tasked to write the RFP (Request for Proposal).
3. He labors for at least 6 months lashing together mass quantities of boiler plate sections of the previous RFP, which happened 25 years earlier. He is transferred just before finishing, and his replacement can't find the working document, so the new guy starts over from scratch. One whole year is spent producing the RFP.
4. The existence of the RFP, which by now weighs 34 pounds and has about 5,000 pages of "technical" descriptions of what each component of the system must do, is published in the Commerce Business Daily, which is the feds' official organ for publishing RFPs (http://cbdnet.gpo.gov/). All big vendors have a large, full-time staff scouring each day's new issue of the CBD. These people are always wearing drool buckets in case they see a particularly juicy contract up for grabs.
5. Vendors are allowed X number of months to prepare the proposals on why they must be chosen to implement this project because they have the lowest price per whatever, they write up their latest and greatest equipment in their official bid for the work, slightly more intelligent bipeds working for the vendors lash together thousands of pages of boiler-plate documents from previous bids, charts and graphs are generated, beautiful color photos of the sexy-looking equipment are inserted into the huge and growing document, then finally the vendor delivers their 25-pound document to the agency that published the RFP.
6. Some of the vendors file protest law suits against some of the wording in the RFP. Hearings are held in federal court to resolve these issues. This takes two or three months.
7. Several maroons (according to William Blair, a bipedal species significantly lower on the evolution tree than morons) in the agency begin perusing the document. They compare the vendor's technical specifications with the technical specs laid out in the RFP. They do not notice that instead of "DS8000" the document mentions a "2314" because the vendor maroon didn't change that one occurrence of "2314" in the 35-year-old boiler plate he lashed together. Any time they find a discrepancy they make a note of it to help the next higher level of maroons choose which vendor to bless with the juicy contract. An example of such a discrepancy might be that the RFP "requires" the vendor's disk drives to have a maximum seek time of 7.5 milliseconds, but the vendor's bid says their latest and greatest disk technology has a maximum seek time of 7.51 milliseconds. This failure to comply with the RFP will later be the cause of law suits galore.
8. Finally the higher up maroons decide on the vendor. They are very careful not to be photographed when going out to fancy restaurants on K Street in Washington, DC with any vendor sales people.
9. The choice is published, along with a deadline for filing protests.
10. On the last day of the protest deadline, the four vendors that were not chosen all file law suits against the one vendor that was awarded the contract. They may also try to sue the XYZ Agency of the government. They are seeking to have the award thrown out because of all the reasons set forth in their 85-pound document that they filed with the legal form announcing the protest. At the very least, they ask for a court-ordered temporary injunction against Agency XYZ to prevent the awardee from starting to work on the contract.
11. Expert witnesses are sought for by all the parties to all the law suits. Depositions are given, hearings are held in the United States Court of Federal Claims ("... the Court has jurisdiction to hear both pre-award and post-award bid protest suits by disappointed bidders on government contracts." [see http://www.uscfc.uscourts.gov/], click on Jurisdiction, then Court, and scroll to the bottom of page 5]
[I had interactions with some of these vendors as an expert witness and appeared in said court in the early 1990s.]
12. The Court renders its decisions. Assume the very best case, that the original awardee is allowed to proceed.
13. The winning vendor's stock price soars and all the VPs have a huge celebration party with hundreds of bottles of Dom Perignon, waiters wearing white gloves serving canap�s to everyone, a violin trio providing background music, etc. The vendor begins implementing their proposal. New people must be hired, managers must be selected, managers must hire hundreds of new individual contributors, blah blah. More months go by.
14. The vendor finishes the contracted work 2.5 years later. This was 1.5 years too late and $500 million over the proposed total expense, but they have been diligent in filing all the required contract budget overrun explanatory forms every few weeks for the extra 1.5 years. In spite of the losing vendors' whinings and possibly new law suits, the original vendor is allowed to complete the work.
15. The agency accepts delivery of the upgraded equipment and begins training its people in how to use the newer technology. The training is supplied, of course, by the winning vendor. The training instructors are being charged to the government agency at the rate of $350 per hour. Four months later the equipment starts being gradually phased in. In the case of the FAA, there are about 20 places around the country known as Air Route Traffic Control Centers, each with its own computers, disk storage units, radar units, radar-processing computers, etc. The FAA decides to phase in this new equipment one ARTCC at a time, and allows 2 months to go by before attempting to begin using the new equipment at each different ARTCC. 2 months multiplied by 20 ARTCCs equals 40 months.
16. Finally after 40 months of phasing in, all the ARTCCs are using the new gear.
17. The total time elapsed so far from when the agency's top management (same oxymoron) had their vision that an upgrade was necessary has been 7 years. None of the top management of the agency are still with the agency, as a new president has been inaugurated and all presidential appointees lose their jobs overnight to be replaced by people supposedly capable of managing the work of each agency but who are also loyal supporters of the new president and his political party.
18. The equipment being used now to keep airplanes from colliding into each other in mid-air is 7 years obsolete when it starts being used everywhere.
19. Agency XYZ continues to try to use this same equipment for the next TWENTY years. After about 15 years the mean-time-between failure of some critical part of the equipment is only a few hours. Finally the vendor of this equipment gives Agency XYZ an official letter from their Vice President of Technical Whatever that the vendor will no longer be able to supply spare parts for the equipment they proposed 22 years earlier. Six months later the vendor stops supplying spare parts.
20. Some catastrophic event occurs with the aged equipment which causes top agency management to embrace the horror that another upgrade is necessary.
As one person connected with Project Apollo quipped, we sent people to the moon with everything supplied by the lowest bidders.
B. The intelligence-community agencies do not operate like this. They have an infinitely deep budget, the amount of which is a closely guarded national secret. They always get the very latest, greatest, biggest, and fastest of everything. Then they set their own expert employees loose on the equipment to try to make it even faster or better in some way. They have acres and acres of such equipment in underground data centers that almost no one knows about. All the top management of all the big vendors know all this, and have internal skunkworks departments that handle all the classified RFPs, bids, etc. Nothing is ever divulged publicly about anything, let alone published in the CBD. No law suits are allowed, as this would cause the LUGA [Large Unnamed Government Agency] to suffer a delay in using the newest latest and greatest stuff, and if there is any delay, then the terrorists win.
Bill Fairchild
Software Developer
Rocket Software
275 Grove Street � Newton, MA 02466-2272 � USA
Tel: +1.617.614.4503 � Mobile: +1.508.341.1715
Email: bi...@mainstar.com
That was actually Alan Sheppard, when he was asked by a reporter what he was thinking when he was waiting during count down.
That was 6 or 7 years earlier.
-
Too busy driving to stop for gas!
certain (federal) division of the company was well known for some of
those tactics. in the late 80s when I was doing high-speed datatransport
project ... the division claimed they had to be briefed on the project.
misc. past posts
http://www.garlic.com/~lynn/subnetwork.html#hsdt
I scheduled an all day presentation ... and something like 20-30 of
their people showup. A few weeks later, I get a bill ... they wanted to
charge at the going rate for listening to my presentation. I wouldn't
pay ... may counter-offer was I would bill them for the presentation
(that they had requested).
In the early 90s ... doing ha/cmp
http://www.garlic.com/~lynn/subtopic.html#hacmp
we would periodically drop into the division president's office ... who
we were acquated ... but also knew his technical assistant reasonably
well ... old email reference about division decided to make ha/cmp
cluster scaleup their strategic direction
http://www.garlic.com/~lynn/2006w.html#email911119
... of course this was before the cluster scaleup effort was transferred
and we were told we couldn't work on anything with more than four
processors. This time they didn't charge me for their time (in the
above referenced meeting).
In anycase, at the time, the president's technical assistant was
spending almost all their time trying to spearhead getting the
FAA effort back on track. related history note
http://www.garlic.com/~lynn/200q.html#31 Check out Computer glitch to cause flight delays across U.S. - MarketWatch
as to the cluster scaleup ... some old newsclipping from '92
http://www.garlic.com/~lynn/2001n.html#6000clusters1
http://www.garlic.com/~lynn/2001n.html#6000clusters2
other old email
http://www.garlic.com/~lynn/lhwemail.html#medusa
I am reminded of doing support in the late 80s/early 90s. LUGA calls and
has an abend in TPX. Guy has got the psw/registers declassified and is
faxing them to me. He will get any page of the dump declassified I need
or he can answer questions over the phone.
He asks why we don't have an "official representative". They get full
clearance. When dump occurs they fly to wherever, read the dump and fly
home.
I asked, but management wouldn't bite.
p.s.don't remember if new fix or not but registers were enough to identify
fix. now squatty box apps produce 30+ line module trace and obscure
message. but the message box is very - oh so - pretty.
IBM Mainframe Discussion List <IBM-...@bama.ua.edu> wrote on 11/20/2009
01:13:00 PM:
...
B. The intelligence-community agencies do not operate like this. They
have an infinitely deep budget, the amount of which is a closely guarded
national secret. They always get the very latest, greatest, biggest, and
fastest of everything. Then they set their own expert employees loose on
the equipment to try to make it even faster or better in some way. They
have acres and acres of such equipment in underground data centers that
almost no one knows about. All the top management of all the big vendors
know all this, and have internal skunkworks departments that handle all
the classified RFPs, bids, etc. Nothing is ever divulged publicly about
anything, let alone published in the CBD. No law suits are allowed, as
this would cause the LUGA [Large Unnamed Government Agency] to suffer a
delay in using the newest latest and greatest stuff, and if there is any
delay, then the terrorists win.
Bill Fairchild
-----------------------------------------
The information contained in this communication (including any
attachments hereto) is confidential and is intended solely for the
personal and confidential use of the individual or entity to whom
it is addressed. If the reader of this message is not the intended
recipient or an agent responsible for delivering it to the intended
recipient, you are hereby notified that you have received this
communication in error and that any review, dissemination, copying,
or unauthorized use of this information, or the taking of any
action in reliance on the contents of this information is strictly
prohibited. If you have received this communication in error,
please notify us immediately by e-mail, and delete the original
message. Thank you
>If they were running on 3081s, that would be a big innovation.
>Last I heard they were still on 2050s (stripped down 360/50s).
>
>Gerhard Postpischil
>Bradford, VT
>
The 9020 (not 2050) was made up of modified 360-65 and 360-50 systems. A
complete 9020E was made up of three or four Model 65锟絪, three Model 50锟絪,
storage units, and peripheral adapters for things like displays and
incoming radar data
These were replaced by 3083s (might have been 3081s). They still existed
into this decade and IBM supplied the LIC source to the FAA so they could
make them Y2K ready. I have been told they were running VM/SP HPO 5 and the
FAA also made that software Y2K ready (after all VM ships as mostly source).
I thought all of those had finally been retired, but as you all know, old
computer systems never seem to die. I know of a customer who still has
MP2000s running OS/390 1.4 with loads of Series 1 machines!
Jim
> On Fri, 20 Nov 2009 11:15:59 -0500, Gerhard Postpischil <
> ger...@VALLEY.NET>
> wrote:
>
> >If they were running on 3081s, that would be a big innovation.
> >Last I heard they were still on 2050s (stripped down 360/50s).
> >
> >Gerhard Postpischil
> >Bradford, VT
> >
> The 9020 (not 2050) was made up of modified 360-65 and 360-50 systems. A
> complete 9020E was made up of three or four Model 65𠏋, three Model 50𠏋,
> storage units, and peripheral adapters for things like displays and
> incoming radar data
>
> These were replaced by 3083s (might have been 3081s). They still existed
> into this decade and IBM supplied the LIC source to the FAA so they could
> make them Y2K ready. I have been told they were running VM/SP HPO 5 and the
> FAA also made that software Y2K ready (after all VM ships as mostly
> source).
>
> I thought all of those had finally been retired, but as you all know, old
> computer systems never seem to die. I know of a customer who still has
> MP2000s running OS/390 1.4 with loads of Series 1 machines!
>
Jim, what were the weird characters after "65" and "50" supposed to be? Just
curious.
>Jim, what were the weird characters after "65" and "50" supposed to be? Just
>curious.
>
Weird, they are apostrophes.
four Model 65's, three Model 50's
Jim
Thank you for the clarification. I'm not sure how I missed that,
but you learn something new every millennium <g>
Gerhard Postpischil
Bradford, VT
Of course, those apostrophes should simply be removed. This is plural
not possessive usage.
--
Edward E Jaffe
Phoenix Software International, Inc
5200 W Century Blvd, Suite 800
Los Angeles, CA 90045
310-338-0400 x318
edj...@phoenixsoftware.com
http://www.phoenixsoftware.com/
>On Sat, 21 Nov 2009 10:18:42 -0500, P S <zos...@GMAIL.COM> wrote:
>
>>Jim, what were the weird characters after "65" and "50" supposed to be? Just
>>curious.
>>
>Weird, they are apostrophes.
>
>four Model 65's, three Model 50's
>
>Jim
The problem is that they are the curly quotes in the Windows-1252
x80-9f code point range that are not valid as ISO-8859-1 and thus do
not display correctly.
>Weird, they are apostrophes.
No. Your e-mail software probably has a bug in it, but apostrophe in
ISO-8859-1 is '27'x, not '92'x.
Aso, where does that '0B'x come from?
--
Shmuel (Seymour J.) Metz, SysProg and JOAT
ISO position; see <http://patriot.net/~shmuel/resume/brief.html>
We don't care. We don't have to care, we're Congress.
(S877: The Shut up and Eat Your spam act of 2003)
>Jim, what were the weird characters after "65" and "50" supposed to be?
Interesting. He sent his message as ISO-8859-1 and you sent your reply as
UTF-8 without translating those characters. The ones you asked about are
92's; there are also some 0B's that you didn't mention. My guess is that
the 92 was supposed to be 27, an apostrophe, but I have no idea where the
0B came from.
--
Shmuel (Seymour J.) Metz, SysProg and JOAT
ISO position; see <http://patriot.net/~shmuel/resume/brief.html>
We don't care. We don't have to care, we're Congress.
(S877: The Shut up and Eat Your spam act of 2003)
----------------------------------------------------------------------
>The 9020 (not 2050) was made up of modified 360-65 and 360-50 systems.
A modified 2050 or 2075 was one of the compnents. But don't forget the
modified 2040 (360/40) used for an I/O processing element.
--
Shmuel (Seymour J.) Metz, SysProg and JOAT
ISO position; see <http://patriot.net/~shmuel/resume/brief.html>
We don't care. We don't have to care, we're Congress.
(S877: The Shut up and Eat Your spam act of 2003)
----------------------------------------------------------------------
>I had some interaction with one of the FAA's Air Route Traffic Control
>Centers in May, 1978. The IBM mainframes then being used by the FAA
>were specialty versions of the S/360 models 50 and 65 that could do
>multiprocessing, meaning two or more CPUs sharing central storage.
9020.
>7. Several maroons (according to William Blair, a bipedal species
>significantly lower on the evolution tree than morons)
Are you sure that he didn't give a citation for the cartoon origin of the
term? Google for "What a maroon".
--
Shmuel (Seymour J.) Metz, SysProg and JOAT
ISO position; see <http://patriot.net/~shmuel/resume/brief.html>
We don't care. We don't have to care, we're Congress.
(S877: The Shut up and Eat Your spam act of 2003)
----------------------------------------------------------------------
>This is kind of off the topic, but related. Wasn't there a discussion on
> IBM-Main a couple years ago about the air traffic control system being
>run on old IBM 3081s?
Stock 3081's or special models that look like 9020's?
--
Shmuel (Seymour J.) Metz, SysProg and JOAT
ISO position; see <http://patriot.net/~shmuel/resume/brief.html>
We don't care. We don't have to care, we're Congress.
(S877: The Shut up and Eat Your spam act of 2003)
----------------------------------------------------------------------
I would speculate that configuring such routers/firewalls would be one or more orders of magnitude more complex than the already overly complex configurations. I have not seen any numbers, but it seems like such configuration issues are very high on the top ten failure causes.
-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-...@bama.ua.edu] On Behalf Of scott
Sent: Thursday, November 19, 2009 5:12 PM
To: IBM-...@bama.ua.edu
Subject: Re: Check out Computer glitch to cause flight delays across U.S. - Mar ketWatch
Patrick Lyon wrote:
> On Thu, 19 Nov 2009 13:56:25 -0700, Roach, Dennis (N-GHG)
> <dennis...@LMCO.COM> wrote:
>
>
>> Looks like a router failure
>>
>> http://gcn.com/articles/2009/11/19/faa-software-hackers-delays.aspx?
>>
> s=FAAnewsalert
>
>
> 4 hours to fix a router? Good Grief.
And no backup? Sounds like a single point of failure to me. Makes
one wonder what other single points of failure there are.
----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to list...@bama.ua.edu with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html
NOTICE: This electronic mail message and any files transmitted with it are intended
exclusively for the individual or entity to which it is addressed. The message,
together with any attachment, may contain confidential and/or privileged information.
Any unauthorized review, use, printing, saving, copying, disclosure or distribution
is strictly prohibited. If you have received this message in error, please
immediately advise the sender by reply email and delete all copies.
>Of course, those apostrophes should simply be removed.
That depends on which style guide you are using, alas.
>This is plural not possessive usage.
The apostrophe is used for more than possessives.
--
Shmuel (Seymour J.) Metz, SysProg and JOAT
ISO position; see <http://patriot.net/~shmuel/resume/brief.html>
We don't care. We don't have to care, we're Congress.
(S877: The Shut up and Eat Your spam act of 2003)
----------------------------------------------------------------------
>The problem is that they are the curly quotes in the Windows-1252
Partially that, and partially that the messages header specified
Content-Type: text/plain; charset="iso-8859-1" instead of the character
set that the message was actually using.
--
Shmuel (Seymour J.) Metz, SysProg and JOAT
ISO position; see <http://patriot.net/~shmuel/resume/brief.html>
We don't care. We don't have to care, we're Congress.
(S877: The Shut up and Eat Your spam act of 2003)
----------------------------------------------------------------------