Standards Update, The Death of the NIST Computer Systems Lab

[Nicholas M. Stoughton]

Submitted-by: ni...@usenix.org (Nicholas M. Stoughton)

Nicholas M. Stoughton <ni...@usenix.org>, Report Editor

The Death of the NIST Computer Systems Lab

Stephen R. Walli <ste...@softway.com> reports on The Death
of the NIST Computer Systems Lab

Events have occurred recently in the NIST Computer Systems
Lab that are very disturbing, and could have a number of
implications for those of us that purchase systems as part
of our jobs. NIST, the National Institute for Standards and
Technology, is part of the Department of Commerce. (It was
formerly known as the National Bureau of Standards.) The
Computer Systems Lab (CSL) is the group that is responsible
for producing Federal Information Processing Standards
(FIPS) to aid in government procurement, certification
programmes to support the FIPS that are widely recognised in
the private sector as well as the U.S. federal government,
and represents one of the primary centers for computer and
information processing standards and specifications
expertise in the U.S. federal government. Instead of
sticking to this core competency, however, there is a new
world order that not only ignores but destroys the good work
done to date, and further ignoring history, boldly wanders
off into the weeds.

What's a FIPS?

A FIPS is a simple document that acts as a government
procurement reference to another standard. Its usefulness
is probably best understood by example. The IEEE POSIX.1
standard defines the programmatic interface to operating
system services, based on the historical UNIX system. In
its conformance section, the standard claims that
implementations must provide all the services described in
the standard, that they must behave as described, and
describes a set of documentation requirements that become a
system's POSIX.1 Conformance Document. The standard has a
number of implementation options, limits, and there are a
number of places that are labelled implementation defined or
unspecified. Therefore two different systems could both
implement different options (for example one supports job
control, and the other doesn't), but still claim conformance
to the standard.

NIST FIPS Pub 151-2 is the POSIX.1 FIPS. It incorporates
the text of the IEEE POSIX.1 standard by reference, then
further identifies what optional functionality is required,
what limits need to be set, and so on. Thus there is better
consistency with respect to the POSIX.1 standard across

- 2 -

systems that meet the FIPS 151-2 criteria, and therefore the
source-code portability model across the machines is more
consistent.

There are FIPS for ANSI C (FIPS 160), the POSIX.2 Shell and
Utilities (FIPS 189), SQL, Fortran, and so on.

Testing and Certification

To ensure systems claiming conformance to a FIPS actually
meet some minimum requirements, NIST CSL puts testing and
certification processes in place wherever possible. Using
the POSIX.1 FIPS as an example again, a system vendor takes
their implementation to an accredited testing lab, who runs
the NIST POSIX.1 Conformance Test Suite (PCTS), and those
results along with the conformance document are submitted to
NIST. The results are validated, and if all is well, a
certificate is issued acknowledging that the system has run
the test suite, and the test results have been inspected.

The POSIX.1 test suite has its roots in the original System
V Verification Suite and has been evolving since 1988 when
the original suite was built. It is a large and robust test
suite. The process and policies have also been continuously
evolving and improving over time.

While the FIPS 151-2 document is essential to government
procurements requiring POSIX-like services, the FIPS 151-2
certificate has broader recognition and visibility in the
commercial world as well as a reasonable ``Good Housekeeping
Seal of Approval'' that the system of interest meets certain
fundamental criteria. It is a reasonable hurdle that
supports a level playing field approach to systems
procurement regardless of whether the purchaser is a
government or commercial organization.

There has been a long history of success with the testing
programmes. (A USENIX conference speaker once made
reference to the original NIST FORTRAN compiler test suite
with awe and respect, figuring that it had been written by a
group of pedantic grandmothers sucking lemons.) As there
has been a need for further test suites faster to support
the rapid progress of technology, and with the recognition
that good test suites cost money, NIST has adapted to the
changing times. As it has become too expensive and slow to
develop the test technology in-house, NIST personnel worked
with industry to adapt and adopt appropriate test suites.

Standards Expertise and Participation

To build a reasonable and representative FIPS and its
attendant certification program requires experience and
understanding. Government organizations that mandate things
without understanding them tend to create white elephants

- 3 -

that finally collapse under their own weight. Sticking to
the POSIX FIPS example, the NIST CSL has actively
participated in the community, developing the standard, such
that FIPS 151-2 and the PCTS were well balanced and
accomplished their goals. Indeed, NIST CSL's participation
dates back to the /usr/group 1984 standard.

There has been exactly once that NIST has thrown its weight
around with respect to POSIX, and that was the introduction
of the original FIPS 151 in 1988 just prior to the final
close of the IEEE POSIX.1 standard itself. Again, this
decision was likely based on experience with the standards
development body, and NIST's understanding of the stableness
of the standard itself. (Both IEEEix and the IEEE Trial Use
POSIX.1 standard were ``cooked''). NIST CSL personnel have
apparently always been sensitive to the fact that they are
part of the Department of Commerce, and that while they may
want to mandate certain things for U.S. federal
procurements, they also need to support an economy full of
companies and employees that work and pay taxes.

While this work lacked sex appeal, it was essential and
responsible. And one would expect the government to do the
essential and responsible, supporting the economic needs of
the government procurement agencies, while indirectly
contributing to the industry as a whole, rather than looking
for sex appeal.

So What's Happened?

In the blood-sport of re-organization at the Department of
Commerce, mandates have been changed and personnel re-
organized to new missions within the NIST CSL. Somewhere,
someone has decided that all the essential work that has
gone on to date is no longer necessary or relevant.

The personnel in the core of the FIPS program and
responsible for certifications have been re-assigned to new
programmes. There is apparently a single person left with
the authority to sign FIPS certificates, but that person has
no idea how few weeks or months they may be allowed continue
in this role, as they are essentially doing this of their
own volition. The new direction (or lack thereof) feels
that this work is no longer needed and that the INDUSTRY
WILL POLICE ITSELF! Such naivete is stunning!

It has even been suggested that Manufacturer Declarations
would suffice. Systems vendors will be able to stand-up and
say I conform to that FIPS -- trust me. There have been a
number of occasions (some recent) where I have spoken to
vendors that claim to be POSIX conforming, but have been
unable to show even a conformance document, let alone a FIPS
certificate. And being saddled with a situation where the
user claims something is broken in an implementation with

- 4 -

respect to the standard is a terrible case of closing the
gate after the horses have escaped. At best, responsive
vendors guided by the size of the customer may fix it for
the next release.

It will be interesting to see how long it takes for history
to repeat itself, as it's being ignored, and the demand for
these certification programs to be established occurs again.
It will be interesting to see how long it takes for CSL
personnel to come up to speed on the specifications and
standards and programmes the next time around, as presumably
the current expertise will have been lost through re-
assignment and attrition by the time someone realises things
are broken and need fixing.

The New World Order

One might be able to understand this ugly amputation if it
was naively done for cost savings and budgetary reasons.
The worst part of this is that this is more of a mandate
change, walking away from a role with a proven successful
track record, towards a role with a proven failed track
record. People just have not read their history books as
they created this bold new mandate.

Essentially, the resources of the NIST CSL are being turned
towards developing and delivering new fundamental
technologies to industry. (No, really, that's what they
want to do.) One acronym example of the government success
rate here: PCTE (the Portable Common Tool Environment).
Here the government attempted to ``encourage'' a technology
that they believed was fundamental, and that the industry
was just not supporting. They built and delivered a
reference implementation. There was even work on an ISO
standard for this, (but then there's also an ISO standard
for BASIC.) PCTE still hasn't taken the industry by storm,
and still isn't accepted as a particularly viable technology
in the real-world. What tools exist are expensive. By
fiddling around with the technology, supporting something
they believe is goodness and light but has no industry
support, the government created a false economy as there is
always someone willing to build expensive tools for
government demand. In the new world order of the CSL, there
may not even be a government market for new ``fundamental''
technologies, and it will be a complete waste of effort.

Forcing a technological model without the economic support
of a real market place has failed on other occasions. There
was supposedly a point where the installed base of TCP/IP
was growing faster than the demand for OSI. Economic
reality finally settled in after who knows HOW much money
was sunk into the OSI white elephant.

Requiem

- 5 -

So hat's off to the team of people that have done so much
useful work in our community. They and their efforts will
be missed. Let us hope not too much damage is done before
their efforts are appreciated at least in hind-sight by
their employer.

R.I.P. NIST Computer Systems Lab.