[GAO-02-41 ] Information Technology: State Department Led Overseas Modernization Program Faces Management Challenges , Part 1/2

[in...@www.gao.gov]

Message-ID: <GAORPTd...@us.govnews.org>
MIME-Version: 1.0

Information Technology: State Department Led Overseas
Modernization Program Faces Management Challenges (16-NOV-01,
GAO-02-41).

To promote U.S. interests in the face of rapid economic,
political, technological, and environmental change, 24 federal
agencies are engaged in foreign affairs activities at 255
overseas locations in 162 countries. The Department of State is
responsible for coordinating and supporting federal agencies'
international activities and providing a means for effective
interagency information sharing. State is leading a multiagency
program to modernize the information technology (IT) environment
supporting federal agencies' overseas operations. State is in the
early, formative stage of a long-term plan to acquire and deploy
a common knowledge management system for overseas-based agencies.
This system is to provide basic Internet access and e-mail to
mission-critical policy formulation and crisis management
support. In the near-term, State is employing informal management
controls, which are adequate given the department's stated
purposes and scope of these activities. However, acquiring and
deploying system capabilities for operational use, particularly a
system that involves multiple agencies and performs
mission-critical functions, requires a much greater level of
management discipline than that needed for system prototyping and
pilot testing. To date, it is appropriate that State has not yet
established these rigorous management controls because its focus
has been on evaluating system prototypes. However, without these
more rigorous controls, it is unlikely that State and its agency
partners will deliver needed operational system capabilities on
time and within budget.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-02-41
ACCNO: A02470
TITLE: Information Technology: State Department Led Overseas
Modernization Program Faces Management Challenges
DATE: 11/16/2001
SUBJECT: Foreign policies
Information technology
Interagency relations
Internal controls
International relations
Management information systems

******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Testimony. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-02-41

Report to the Chairman, Committee on International Relations, House of
Representatives

United States General Accounting Office

GAO

November 2001 INFORMATION TECHNOLOGY

State Department Led Overseas Modernization Program Faces Management
Challenges

GAO- 02- 41

Page i GAO- 02- 41 State- Led Modernization Initiative Letter 1

Results in Brief 2 Background 4 State?s Near- Term Plans Are Focused on Risk
Reduction Activities 13 Much Remains to be Done Before Acquiring the
Operational

System 15 Conclusions 25 Recommendations for Executive Action 25 Agency
Comments and Our Evaluation 27

Appendix I Objectives, Scope, and Methodology 30

Appendix II Comments From the Department of State 33

Appendix III GAO Contact and Staff Acknowledgment 47

Tables

Table 1: Summary of Management Responsibilities and Functions of the
Overseas Presence Committee and its Entities 7 Table 2: Planned System
Functionality by Increment 9 Table 3: Summary of Needed Process Controls for
Managing the

Acquisition of the Operational Version of the System 24

Figures

Figure 1: Management Structure Established to Address Several Panel
Recommendations 6 Figure 2: Simplified Diagram of Target Overseas Knowledge

Management System 11 Contents

Page ii GAO- 02- 41 State- Led Modernization Initiative Abbreviations

CIO chief information officer DTS Department of State Telecommunications
Service FASI Foreign Affairs Systems Integration IT information technology
ITS Interagency Technology Subcommittee MSP Managing State Projects OMB
Office of Management and Budget SEI Software Engineering Institute

Page 1 GAO- 02- 41 State- Led Modernization Initiative

November 16, 2001 The Honorable Henry J. Hyde Chairman Committee on
International Relations House of Representatives

Dear Mr. Chairman: Dramatic changes continue to occur in the world?s
economic, political, technological, and environmental landscapes. Increases
in global trade, transnational terrorism and organized crime, and
international health concerns, for example, are combining to create a more
complex, more vulnerable world scene. To promote U. S. interests in the face
of such rapid change, 24 federal agencies are collectively engaged in
foreign affairs activities at 255 overseas locations in 162 countries. 1

As our nation?s lead federal agency for foreign affairs, the Department of
State (State), among other things, is responsible for coordinating and
supporting federal agencies? international activities, such as providing a
means for effective interagency information sharing. Currently, this means
does not exist, resulting in agencies not getting the right information to
the right people at the right time. To address this situation, State is
leading a multiagency program to modernize the information technology (IT)
environment supporting federal agencies? overseas operations. In this role,
State is acting on behalf of the other foreign affairs agencies to acquire
and test a common knowledge management prototype and pilot system. 2

1 These agencies include the Departments of Agriculture, Commerce, Defense,
Energy, Health and Human Services, Interior, Justice, Labor, Transportation,
Treasury, and Veterans Affairs; the Environmental Protection Agency, the
Federal Emergency Management Agency, the General Services Administration,
the National Aeronautics and Space Administration, the Library of Congress,
the Social Security Administration, the American Battle Monuments
Commission, the Agency for International Development, U. S. Trade
Representative, the Overseas Private Investment Corporation, the Trade and
Development Agency, the National Science Foundation, and the Peace Corps.
These agencies are represented overseas in 162 countries at 255 posts (162
embassies, 76 consulates, 17 other missions and offices).

2 Knowledge management involves the use of business processes and
intellectual and technological assets to promote and provide for
collaboration and information exchange for the purpose of accomplishing
mission goals and objectives.

United States General Accounting Office Washington, DC 20548

Page 2 GAO- 02- 41 State- Led Modernization Initiative

This report provides our assessment of State?s efforts to lead this program
to modernize the IT environment supporting federal agencies? overseas
operations. As agreed with your office, our objectives were to determine (1)
what State?s plans are for acquiring a multiagency, common- platform
knowledge management system and (2) whether State has effective controls in
place for acquiring this system, that is, defined roles for State and its
agency partners 3 and effective controls over its investment, enterprise
architecture, and system acquisition management processes. In conducting
this review, we compared State?s current and planned activities against
legislative requirements, federal guidance, and best practices relevant to
systems modernization. Our objectives, scope, and methodology are presented
in more detail in appendix I.

State is in the early, formative phase of a long- term plan to acquire and
deploy a common knowledge management system for overseas- based agencies
engaged in foreign affairs activities. This system is to provide
functionality ranging from basic Internet access and e- mail to
missioncritical policy formulation and crisis management support. While
detailed long- term modernization plans understandably do not yet exist,
near- term plans do. These plans show that State will focus first on
prototyping and pilot testing a system to better understand user
requirements and alternative design options. The plans also show that State
has assumed responsibility and accountability for funding and leading the
management and administration of these near- term activities. System
prototyping and pilot testing, which involve investing a relatively small
amount of time and resources in building and evaluating a much simpler
version of the operational system solution, can be effective risk- reduction
measures for large system modernization programs. State currently plans to
complete prototype evaluation by May 2002 and pilot testing by September
2002.

To lead the near- term activities, State is employing informal management
controls, which are adequate given the department?s stated purpose and scope
of these activities. However, acquiring and deploying system capabilities
that are intended for operational use, particularly a system like this that
involves multiple agencies and performs mission- critical

3 State?s primary agency partners include the U. S. Agency for International
Development, the Peace Corps, and the Departments of Defense, Justice,
Treasury, Agriculture, Transportation, and Commerce. Together, State and
these eight agencies represent 99 percent of our overseas presence. State
and the Department of Defense represent about 80 percent of our foreign
overseas presence. Results in Brief

Page 3 GAO- 02- 41 State- Led Modernization Initiative

functions, requires a much greater level of management discipline than that
needed for system prototyping and pilot testing. Specifically, the more
rigorous management controls that are needed include (1) explicitly defining
a multiagency governance and funding structure for acquiring and deploying
the operational system; (2) ensuring that time and resources are invested in
economically justified, architecturally compliant increments of the
operational system and verifying that increments meet return- oninvestment
expectations before investing heavily in later system increments; (3) having
and using an overseas- presence enterprise architecture as a blueprint to
guide and constrain system investments; and (4) having and following mature
and disciplined software acquisition management capabilities. To date, it is
appropriate that State has not yet established these controls because its
focus has been on evaluating system prototypes. However, without these more
rigorous controls, it is unlikely that State and its agency partners will
deliver needed operational system capabilities on time and within budget.

During the latter stages of our review, State initiated steps to establish
some of these management controls and committed to establishing others.
However, much remains to be accomplished, and plans do not yet exist for
establishing some controls. Accordingly, we are making recommendations to
the Secretary of State, as the designated lead official for the
overseaspresence system modernization program, that are intended to ensure
that each of these control areas is fully addressed before acquisition of
the operational system solution begins.

In commenting on a draft of this report, State agreed with our findings and
conclusions that its management controls over near- term prototype and
pilot- testing activities are adequate, but disagreed that the department?s
controls for managing the next phase of the program are not adequate.
Additionally, State commented that our recommendations should be deleted
because (1) we reported that management controls over near- term activities
are adequate and thus corrective actions are not needed; (2) except for
defining agency roles and responsibilities, State?s existing management
controls are adequate for the next phase of the program; and (3) the lead
agency for the next phase of the program is uncertain and thus the
recommendations are premature.

We disagree with State?s comments. First, the level of management rigor and
discipline needed to effectively manage prototype and pilot- testing
activities, as discussed in the report, is not as demanding as that needed
for acquisition and deployment of a system that is to be used operationally.
Thus, for example, while State?s lack of an enterprise

Page 4 GAO- 02- 41 State- Led Modernization Initiative

architecture to guide and constrain prototype and pilot- testing activities
does not introduce significant program risk, it would introduce considerable
risk for State to begin the next phase of the program without this
architecture. Second, our recommendations are appropriate and warranted
because (1) they are not intended to correct problems with management of the
prototype and pilot activities, but rather to proactively provide a
recommended road map for how to successfully manage the next phase of the
program; (2) the existing controls, as explained above, are not adequate for
the next phase of the program; and (3) even though the possibility exists
that State might not be the lead agency for the next phase of the program,
prudent planning dictates that the agencies? roles and responsibilities for
the next phase be clearly defined and agreed to sooner rather than later to
provide continuity to the program and avoid unnecessary delays.

State?s written comments, along with our responses, are reproduced in
appendix II.

In response to the tragic 1998 bombings in Kenya and Tanzania, the former
Secretary of State established an independent panel, the Overseas Presence
Advisory Panel (Panel), to analyze our overseas presence and develop
recommendations for organizing and managing our embassies and consulates
overseas. After visiting several overseas posts, the Panel concluded that
the condition of U. S. posts and missions abroad was unacceptable and
recommended eight major types of changes, which, according to its report,
would improve the efficiency and effectiveness of our overseas presence. 4
Specifically, the panel directed the President, the Secretary of State, and
congressional leaders to form a partnership to implement the following eight
recommendations:

improve security and establish accountability for security at our overseas
facilities;

4 America?s Overseas Presence in the 21st Century: The Report of the
Overseas Presence Advisory Panel, U. S. Department of State (November 1999).
Three other studies of the condition of our overseas posts have been
performed during the last 3 years. These studies are Road Map for National
Security: Imperative for Change, The United States Commission on National
Security/ 21st Century (February 15, 2001); Independent Task Force Report:
State Department Reform, Council on Foreign Relations (February 2001); and
Equipped for the Future: Managing U. S. Foreign Affairs in the 21st Century,
The Henry L. Stimson Center (October 1998). All these studies cited the need
to upgrade information technology at U. S. overseas locations. Background

Page 5 GAO- 02- 41 State- Led Modernization Initiative

create the right size and shape for our overseas presence, ensuring that
posts are not over- or under- staffed;

ensure that U. S. facilities are quickly and efficiently renovated and are
properly maintained and managed on an ongoing basis;

adopt best practices for human resources management, such as improving
recruitment and expanding training and promotion opportunities;

invest in up- to- date information technology, including providing
Internet access and e- mail to all staff;

improve and reinforce consular services, such as same- day processing of
visas;

enact cost- and time- saving administrative reforms, such as consolidating
some functions (e. g., accounting practices) and automating others; and

reinforce the responsibilities and authorities of Ambassadors, who
represent the interests of the President of the United States.

Because State is the lead agency for foreign affairs, 5 President Clinton
directed the former Secretary of State, on February 10, 2000, to lead a
cabinet- level committee to implement the Panel?s recommendations. As part
of the former Secretary?s response, the Under Secretary for Management
formed an interagency Overseas Presence Committee (the Committee) in
February 2000 to address three of the eight recommendations. The Committee
then established three subcommittees to implement the Panel?s
recommendations: Overseas Facilities, Interagency Rightsizing, and
Interagency Technology. 6 (See fig. 1 for the Committee?s structure and
table 1 for allocation of responsibilities.)

5 As the lead agency representing U. S. interests overseas, State is vested
with a wide range of responsibilities, including formulating U. S. policy on
diverse international issues, coordinating and implementing U. S. government
programs and activities overseas, and influencing other countries to adopt
policies and practices consistent with U. S. interests.

6 According to State?s Director of the Office of Management, Policy, and
Planning, this organizational arrangement is currently under review for
potential restructuring at a later date.

Page 6 GAO- 02- 41 State- Led Modernization Initiative

Figure 1: Management Structure Established to Address Several Panel
Recommendations

a FASI = Foreign Affairs Systems Integration. Source: State.

Page 7 GAO- 02- 41 State- Led Modernization Initiative

Table 1: Summary of Management Responsibilities and Functions of the
Overseas Presence Committee and its Entities Entity Responsibility/ Function

Overseas Presence Committee Established to implement the Panel?s
recommendations (in its 1999 report) regarding overseas facilities,
interagency rightsizing, and information technology. Chaired by State?s
Under Secretary for Management. Overseas Facilities Subcommittee Tasked with
seeking new ways to manage and finance overseas facilities, such as

evaluating the option of charging rent to the occupants of overseas
diplomatic facilities. Chaired by the Deputy Assistant to the Secretary of
State. Interagency Rightsizing Subcommittee Directed to address issues such
as the size and shape of overseas posts, the streamlining

of every mission, the reallocation of all personnel, and budget savings from
reducing the size of overstaffed posts. Chaired by the U. S. Deputy
Representative to the United Nations. Interagency Technology Subcommittee
(ITS) a Charged with defining the high- level functional requirements,
selecting the specific

enabling strategies, and identifying the needed funding for the common
overseas system through its two permanent working groups- knowledge
management and information technology. Chaired by State?s Chief Information
Officer (CIO) and includes the CIOs of State?s agency partners for this
initiative. Knowledge Management Working Group b Established within the ITS
to facilitate information sharing to support a collaborative work

environment. Charged with defining the operational requirements for
knowledge management, recommending enabling strategies, identifying
resources to implement those strategies, providing oversight and guidance,
and identifying needed funding. Led by State?s Deputy CIO, who is also
State?s Chief Knowledge Officer, and includes representatives from State and
its agency partners. Information Technology Working Group b Established
within the ITS to provide the foreign affairs community with a common,

interoperable technology and infrastructure. Intended to identify
commercial- and government- off- the- shelf applications and Internet and
Internet- like technologies to support interagency collaboration. This
effort is to include analyzing requirements for a common overseas platform
as well as related planning tasks such as design, architecture, and
acquisition. Led by State?s Deputy CIO/ Chief Knowledge Officer and
comprises representatives from State and its agency partners. Interagency
Capital Financing Working Group c Established within the ITS to develop a
capital financing strategy for funding the

implementation of the Panel?s IT recommendations. Led by State?s Deputy CIO/
Chief Knowledge Officer and comprises representatives from State and six of
its eight agency partners. FASI Architectural Working Group Established
within the ITS to develop and implement the enterprise architecture for the

common overseas system, ensuring compliance with State and its agency
partners? existing enterprise architectures. Led by State?s Director for
Standards, Policy, Planning, and Architecture. FASI Program Office Created
to plan and implement the common overseas platform. Using the user and
system

requirements devised and prioritized by the Knowledge Management and
Information Technology working groups, this team is to design and deploy an
unclassified, interoperable information technology infrastructure and
knowledge management system that is intended to address the recommendations
in the Panel?s report. Led by State?s Deputy CIO/ Chief Knowledge Officer. a
The efforts of this subcommittee are the primary focus of this report.

b These working groups were combined in April 2001. c According to State,
membership for this working group currently consists of representatives from
the U. S. Agency for International Development, the Peace Corps, and the
Departments of Defense, Treasury, Justice, and Agriculture.

Page 8 GAO- 02- 41 State- Led Modernization Initiative

In October 2000, the then Secretary of State reported on the progress made
in addressing the Panel?s recommendations for seven of the eight
recommendations. 7 In response to the recommendation to invest in up- todate
information technology, State reported that it had

established an acquisition and deployment strategy, preliminary system
requirements, a high- level implementation plan, and an architectural
concept for a common overseas knowledge management system infrastructure;

established September 2000 as the goal for completing a formal program
management plan, with a goal of revising the plan by April 2001 after
completing evaluation of a prototype system; and

established December 2001 as the goal for completing system pilot testing
at two posts.

On August 8, 2001, State issued its final status report on its efforts to
address the Panel?s recommended changes to the Chairman of the Senate
Committee on Foreign Relations. With regard to the recommendation to upgrade
information technology, State reported that the interagency group has
developed and is testing a prototype of the common overseas system and that
a field test of the system?s infrastructure is being conducted at pilot
posts in Mexico and India. In addition, State officials told us that the
date for completing system prototyping has been delayed by 13 months to May
2002, and the date for completing pilot testing at two posts has been
delayed by 9 months to September 2002. Also, the three vendors selected to
develop system prototypes are not scheduled to deliver these systems to
State for evaluation until November 2001.

The modernization program is intended to put in place a common overseas
knowledge management system to facilitate unclassified information/
knowledge sharing among foreign affairs agencies. Key system capabilities
are to include, among other things, interagency e- mail, post- specific news
and information web- based links, crisis coordination support, policy
formulation support, and various administrative functions (e. g.,
accounting, contract management, training, and travel). According to State?s
draft program management plan, the department will acquire and deploy system
functionality in each of these areas in three increments, or three
prioritized groupings of user requirements, which State calls pilot,

7 The report did not address the Panel?s recommendations for enacting cost-
and timesaving administrative reforms. Common Overseas System

Modernization: A Brief Description

Page 9 GAO- 02- 41 State- Led Modernization Initiative

full- scale implementation, and deferred. (See table 2 for planned system
functionality by increment.)

Table 2: Planned System Functionality by Increment Increment 1: Pilot

Type: General user requirements -- accommodate Sensitive But Unclassified
and Unclassified information and

processing -- ensure access to system during working hours at all locations
-- support mobile/ traveling users -- provide a powerful ?smart? search
capability -- support the ability to identify experts in foreign policy,
technical, and administrative disciplines (i. e., a knowledge management
locator service) -- enable ?self service? processing (e. g., the ability for
users to update records and

initiate transactions) Type: Interagency e- mail

-- provide antivirus processing -- enable delivery of e- mail to the user?s
existing agency e- mail system -- provide access to e- mail addresses and
other basic information on participants in

the Overseas Presence Advisory Panel system -- enable end users to create
customized directories and group addresses -- provide e- mail capability
between post and non- U. S. government entities -- support private and
public mailing lists and list management services Type: News and
information: provide access to

-- administrative information (e. g., financial bulletins, per- diem rates,
travel regulations) -- post or regional newsletters and bulletin boards --?
news gathering and analysis? activities -- news ?clipping? services --
issue- oriented databases Type: Crisis coordination: provide support for

-- identifying and locating employees at a given post -- collaborating and
conferencing to support work groups and information sharing -- coordinating
activities (e. g., transportation, medical treatment) Type: Policy
formulation: provide support for

-- researching issues related to a given policy -- publishing of current
policies and decisions -- collaboratively developing policy documents,
guidelines, processes, and decisions -- identifying responsible departments
and persons for a given policy area Type: Administration: provide support
for

-- international Cooperative Administrative Support Services System --
contacting management through a world- wide, on- line staff directory --
coordinating and managing calendars and schedules

Page 10 GAO- 02- 41 State- Led Modernization Initiative

Increment 2: Full- scale implementation

Type: General user requirements -- provide support for teams with non- U. S.
government members Type: Interagency e- mail: provide

-- e- mail forwarding (inside or outside the collaboration zone) --
integration with workflow tool( s) -- embedded URLs (Internet and Intranet)
Type: News and information: provide access to

-- local/ regional translation services -- specialized subscription services
-- commercial telephone directories -- geographic databases -- mission
performance plan data -- telegrams (current and archival) Type: Crisis
coordination

-- provide support for notifying and disseminating information, orders, and
instructions to government and U. S. citizens Type: Policy formulation

-- provide support for bureau and mission performance plans Type:
Administration: provide support for

-- training and human resources -- procurement -- travel planning and
management -- budgeting and financial management

Increment 3: Deferred

Type: General user requirements -- support access from electronic
organizers, cellular telephones, personal digital assistants, etc. --
provide access to agency (noncommon- overseas) systems Type: Administration:
provide support for

-- motor- pool scheduling -- inventory and equipment maintenance -- property
management -- applications development

Source: State.

The system is to use commercially available, Internet- based technology to
enable users to access the system from any location using their respective
agencies? existing networks and operating platforms. The desktop Internet
browser is intended to be the connection to the common overseas system and
its knowledge management tools. A common website is to serve as the portal
to the customized database applications, files, and communication tools to
be shared across the foreign affairs community?s diverse, distributed
environment. (See fig. 2 for a simplified diagram of the system.)

Page 11 GAO- 02- 41 State- Led Modernization Initiative

Figure 2: Simplified Diagram of Target Overseas Knowledge Management System

Source: State.

Page 12 GAO- 02- 41 State- Led Modernization Initiative

In 1999, the Panel estimated the cost to acquire and deploy this
unclassified system at about $200 million, assuming the use of commercially
available technology. About a year later, State estimated that the
investment in the system through fiscal year 2004 would be about $271
million. State plans to update this cost estimate. According to State?s
draft program management plan, the estimated cost to perform system
prototyping and pilot testing is $18.2 million.

The conference report accompanying the Departments of Commerce, Justice, and
State, the Judiciary, and Related Agencies Appropriations Act, 2001, stated
the conference committee?s expectation that State would spend $17 million to
carry the program through pilot testing. 8 State has met with subcommittee
staff to keep them informed of its plans and progress, and State has agreed
to submit a spending plan for the pilot project, as directed by the
conference report. Prototyping activities are currently scheduled for
completion in May 2002.

In June 2000, we testified that State was in the early stages of planning
for the overseas- presence system modernization- establishing preliminary
milestones, developing rough cost estimates, and formulating a program
management plan. 9 At that time, we expressed concern regarding (1) State?s
ability to obtain agreement internally as well as from the foreign affairs
partner agencies in devising and deploying a common technology solution and
(2) State?s lack of an approved architecture to guide and constrain its own
IT investments. In our testimony, we stated, among other things, that it was
important for State to (1) carefully plan the initiative establishing
realistic goals and milestones, (2) install the needed management and
oversight accountability to properly guide the acquisition of the system,
and (3) anticipate the steps needed to gain the full cooperation of its
agency partners.

8 H. R. Conf. Rep. No. 106- 1005, at 294 (2000) states, in part, that $17
million shall be for a pilot project to establish a common technology
platform at overseas posts pursuant to the recommendations of the Overseas
Presence Advisory Panel. The conference report also directs that State
submit a spending plan for this pilot project.

9 Foreign Affairs: Effort to Upgrade Information Technology Overseas Faces
Formidable Challenges (GAO/ T- AIMD/ NSIAD- 00- 214, June 22, 2000). Recent
Review of Common

Overseas System Modernization Program Noted Management Challenges

Page 13 GAO- 02- 41 State- Led Modernization Initiative

Office of Management and Budget (OMB) guidance outlines risk reduction
principles that are consistent with best commercial practices and, if
effectively implemented, can increase the probability that system
modernization programs will succeed. 10 Among other things, this guidance
supports (1) prototyping system components before acquiring or developing
the operational system, (2) using commercially available technology, and (3)
securing system stakeholders? involvement and agreement on products such as
management plans and system requirements.

State?s near- term plans generally provide for each of these areas of
guidance. In particular, State is currently focused on acquiring and
evaluating system prototypes that are to employ either commercially
available and/ or government- used hardware and software products. On July
5, 2001, State selected three vendors, each of which was to develop a
prototype system by October 2001. However, according to a State official,
because of security and infrastructure concerns identified during testing
that the vendors must address, the systems are now scheduled for delivery in
November 2001. In addition, State planned to evaluate the delivered
prototype systems using evaluation criteria that were to be developed by a
private firm by September 28, 2001. 11 While an initial draft of the
criteria was provided on that date, according to a State official, revisions
were needed, and the deadline for finalizing the evaluation criteria was
changed to mid- November 2001. Although these events will delay the deadline
for completing the prototype evaluation one month to May 2002, piloting
activities will not be affected and will begin as scheduled. Once the
prototype system solutions have been evaluated, State and its agency
partners plan to use the best capabilities from each prototype to refine
user requirements and develop a specification for the foundational system
capabilities that will be pilot tested. State plans to pilot test these
foundational capabilities in India and Mexico between April and September
2002. According to State, no commitments have been made to proceed beyond
this pilot test, and any decisions about how best to proceed will have to
involve agency partners, OMB, and relevant congressional committees.

10 Office of Management and Budget (OMB) Circular A- 130, Management of
Federal Information Resources (November 30, 2000) and Capital Programming
Guide, Version 1.0, OMB Circular A- 11, part 3 (July 2000).

11 State officials told us that the deadline for the criteria was changed
from September 28, 2001, to mid- November 2001 because of the events of
September 11, 2001. State?s Near- Term

Plans Are Focused on Risk Reduction Activities

Page 14 GAO- 02- 41 State- Led Modernization Initiative

In addition to prototyping the system and using commercially available
products, State has also taken steps to engage other foreign affairs
agencies in its prototyping activities, including securing stakeholder
involvement and buy- in. Specifically, responses from the seven
modernization program partner agencies that responded to our survey showed
that they were generally satisfied with State?s efforts to engage them in
near- term program activities. 12 Additionally, four of these agencies
stated that State had informally obtained their concurrence on such items as
program plans and system requirements. As for the other three agencies, one
stated that while it had reviewed some of the program documentation, it had
not participated in an approval process; another stated that it had reviewed
program documentation for informational purposes only and was unaware of any
approval process; and the third stated that it had not reviewed any of the
program documentation.

Nevertheless, we found that these and other interagency meetings, such as
program working group meetings, 13 have occurred and provided forums for
informally engaging agency partners. In general, agency partners attended
these meetings and participated in the program. For example, two agency
partners have assigned staff to the FASI Program Office. 14

Available documentation and State CIO officials? statements confirm that
agency partners? buy- in to program decisions, such as agreement on the
requirements for the prototype and approval of plans, is being accomplished
informally. According to State, representatives for the partner agencies
agreed that their lack of opposition or comments equated to concurrence on
actions, plans, and/ or documents, thereby eliminating the need to obtain
actual signatures to indicate approval. Although State could not provide us
with any written agreements to this effect, revised minutes of an
Interagency Technology Subcommittee meeting reflected this informal
agreement.

12 As of November 9, 2001, the U. S. Agency for International Development,
the Peace Corps, and the Departments of Agriculture, Justice,
Transportation, Commerce, and Treasury had responded to our survey; the
Department of Defense had not.

13 These meetings include the Knowledge Management Working Group, the
Information Technology Working Group, the Architecture Working Group, and
the Interagency Capital Financing Working Group.

14 Treasury has assigned an employee to State?s Knowledge Management Team,
and the Justice Department has detailed an individual to assist State in
developing the overseas presence enterprise architecture.

Page 15 GAO- 02- 41 State- Led Modernization Initiative

The Clinger- Cohen Act of 1996, in combination with OMB guidance
implementing the act, provide federal agencies with a framework for
effectively managing modernization efforts that are consistent with best
commercial practices. 15 Among other things, OMB requires (1) a clear
definition of program management roles, including an explicit definition of
responsibilities and accountabilities for such things as funding, investment
decisionmaking, requirements management, architecture definition and
integration with legacy systems, and testing; (2) use of incremental
investment management principles to minimize the enormous risk inherent in
large modernization programs that involve many things to be done over many
years; (3) the development, maintenance, and implementation of an enterprise
architecture, which is an institutional blueprint for guiding operational
and technological modernization; and (4) the use of effective IT project
management processes, such as processes for managing software- intensive
system acquisition projects.

For its overseas knowledge management system modernization program, State
program officials told us that they plan to establish and implement each of
these management controls. However, much remains to be accomplished because
thus far State has devoted its attention to system prototyping activities,
and, according to State officials, they first had to brief congressional
staff earlier in fiscal year 2001 on the use of appropriated funds before
establishing these management controls. 16

It is essential that these controls be effectively implemented on the
overseas- presence system modernization program before State proceeds with
its acquisition of the operational system. To do less would unnecessarily
increase the risk that needed system capabilities will not be delivered on
time and within budget.

15 Clinger- Cohen Act of 1996, Public Law 104- 106 and OMB Circular A- 130,
Management of Federal Information Resources (November 30, 2000). 16
Consistent with congressional direction to submit a spending plan for the
pilot project, State briefed congressional staff on its plans for the pilot
project and its proposed use of the funds before it began spending the $17
million. Much Remains to be

Done Before Acquiring the Operational System

Page 16 GAO- 02- 41 State- Led Modernization Initiative

OMB guidance recognizes the importance of fully and explicitly defining and
ensuring understanding of program stakeholders? areas of responsibility and
accountability in managing and funding a program. Similarly, the Software
Engineering Institute (SEI), recognized for its expertise in managing
software- intensive systems, emphasizes the importance of clearly
designating roles and responsibilities on system acquisition programs. While
doing so is essential on any program, it is particularly critical for
programs like the overseas- presence knowledge management modernization
program because its success depends on the effective interplay of multiple
federal agencies. Without well- defined, understood, and agreed- to roles in
such areas as funding and acquisition management (e. g., investment
decisionmaking, requirements management, architecture definition and
integration with legacy systems, and testing), State cannot, for example, be
assured that necessary resources will be provided (e. g., financial, human
capital, and technological) and critical tasks executed.

According to State, the roles, responsibilities, and accountabilities of
foreign affairs agencies have been formally defined. However, State did not
provide us with documentation to support this position, as we requested.
Furthermore, agency partners that responded to our survey stated that their
respective roles, responsibilities, and accountabilities for this initiative
have not yet been formally defined.

According to State, as the principal foreign affairs agency, its role is to
provide administrative and management leadership and support for nearterm
prototype and pilot- testing activities, and it is the agent for bringing
the foreign affairs agencies together. However, State said that it is not
responsible for engaging agency partners as part of the near- term prototype
and pilot- testing activities.

As noted earlier, given the stated purpose and scope of these near- term
activities and State?s leadership role, this definition of roles is not
currently a significant issue. However, it is essential that State and its
agency partners fully define agreed- to roles, responsibilities, and
accountabilities for program funding and management before moving beyond
planned near- term activities. To do less would unnecessarily put the
modernization program at risk. Clear Definition of

Stakeholders? Accountability Is Critical to Proceeding Beyond Pilot Testing

Page 17 GAO- 02- 41 State- Led Modernization Initiative

Incremental investment management involves three fundamental components: (1)
developing/ acquiring a large system in a series of smaller projects or
system increments; (2) individually justifying investment in each increment
on the basis of costs, benefits, and risks; and (3) monitoring actual
benefits achieved and costs incurred on ongoing increments and modifying
subsequent increments/ investments to reflect lessons learned. Using this
approach, agencies can avoid discovering too late that their systems are not
cost beneficial and reduce the risks associated with large, expensive
projects. This approach allows overlap and smooth transition among
increments because lessons learned from actual results of ongoing increments
would be monitored and evaluated continuously so that these results will be
available for use in defining and justifying future increments.

The Clinger- Cohen Act of 1996 requires agencies to follow, to the maximum
extent practical, an incremental approach to investing in IT projects. Also,
OMB policy requires that investments in major systems be implemented in
increments, with each increment delivering measurable benefits. 17 More
specifically, OMB?s Capital Programming Guide 18 describes the use of
modular contracting, or incremental investment, including its application
and benefits. In particular, OMB states that project increments should
provide for the following:

Separability: Each increment should be fully funded, have substantial
programmatic use that is not dependent on any subsequent increment, and be
capable of performing its principal functions, even if no subsequent
increments are acquired.

Interoperability: Each increment should comply with a common architecture
or commercially acceptable technology standards and should be compatible
with and capable of being integrated with other increments.

Performance requirements: Each increment?s performance requirement should
be consistent with the performance requirements of the completed overall
system and should address interface requirements with other increments.

17 OMB Circular A- 130 (November 30, 2000); Capital Programming Guide,
Version 1. 0,

OMB Circular A- 11, part 3 (July 2000), pp. 545- 572. 18 Capital Programming
Guide, Version 1. 0, OMB Circular A- 11, part 3, Supplement,

Planning, Budgeting, and Acquisition of Capital Assets (July 1997), pp. 35-
37. Incremental Investment

Management Could Reduce Overall Program Risks

Page 18 GAO- 02- 41 State- Led Modernization Initiative

In short, incremental investment helps to mitigate the risks inherent in
acquiring/ developing large systems by dividing a single, large program into
smaller, independently useful components with known and defined
relationships and dependencies. It is well understood that making investment
decisions up front on large- scale, long- term programs is risky because
their economic justification is based on costs, benefits, and risks that are
difficult to forecast reliably, partially because later increments are not
well understood or defined and partially because they are subject to change
in light of experience on nearer term increments and changing business
needs. Through incremental investment management, organizations can

reduce the level of program risk and complexity faced at any one time by
spreading this risk and complexity across a series of smaller investments;

permit the delivery of some part of the expected business value earlier
rather than waiting until later for the total, but more uncertain, business
value;

continuously monitor and evaluate the delivery of cost and benefit
expectations on ongoing increments and use this information to better define
and economically justify these increments; and

permit later increments to exploit technology advances or accommodate
evolving business needs.

According to modernization officials, preliminary plans provide for
acquiring and investing in the overseas knowledge management system in three
increments: pilot, full- scale implementation, and deferred. However, State
has not yet established specific plans and management processes for
acquiring the operational system because, according to State?s CIO, it does
not have congressional authorization to do so. We were unable to corroborate
State?s position as to why it has not established these plans and processes.

In addition, modernization officials stated that the ongoing system
prototyping and planned pilot- testing activities were not economically
justified as a separate increment. However, these officials noted that all
future incremental investments in the system will be economically justified,
and the results and lessons learned from prior increments will be used to
define and make investment decisions about future increment( s).
Additionally, they stated that future increments will also be evaluated to
ensure that return- on- investment commitments are met.

It is critical for State to follow through on its stated commitments to
implement incremental investment practices. Otherwise, it risks making a

Page 19 GAO- 02- 41 State- Led Modernization Initiative

single, monolithic investment decision for the entire program. This ?all or
nothing? approach to investing in IT has historically resulted in agencies
investing huge sums of money in systems that do not provide commensurate
benefits, and thus has been abandoned by successful organizations. The need
to avoid this pitfall was a major impetus for the investment management
reforms of the Clinger- Cohen Act.

Enterprise architectures are essential tools for effectively and efficiently
reengineering business processes and for implementing and evolving their
supporting systems. Enterprise architectures systematically capture- in
useful models, diagrams, and narrative- the full breadth and depth of the
mission- based mode of operations for a given enterprise, which can be (1) a
single organization or (2) a functional or mission area that transcends more
than one organizational boundary (e. g., financial management, acquisition
management, or overseas- presence foreign affairs activities).

An architecture describes the enterprise?s operations in both (1) logical
terms, such as interrelated functions, information needs and flows, work
locations, and system applications, and (2) technical terms, such as
hardware, software, data, communications, security attributes, and
performance standards. It provides these perspectives for both the
enterprise?s current, or ?as is,? environment and its target, or ?to be,?
environment; it also provides an IT capital investment road map for moving
between the two environments.

The development, implementation, and maintenance of enterprise architectures
are recognized hallmarks of successful public and private sector
organizations. Managed properly, an enterprise architecture can clarify and
help optimize the interdependencies and interrelationships among an
organization?s business operations and the underlying IT infrastructure and
applications that support these operations. Our experience with federal
agencies has shown that attempting to define and build major IT systems
without first completing an enterprise architecture often results in systems
that are duplicative, not well integrated, unnecessarily costly to maintain
and interface, and not effectively optimizing mission performance.

Federal CIO Council guidance defines a set of recognized key practices
(management structures and processes) for developing and implementing
Enterprise Architectures

Are Essential Tools for Guiding and Constraining Investment Decisions

Page 20 GAO- 02- 41 State- Led Modernization Initiative

an enterprise architecture that are employed by successful public and
private sector organizations. 19 Among other things, these practices include
the following:

Because the architecture is a corporate asset for systematically managing
institutional change, the head or leader of the enterprise should support
and sponsor the architecture, giving it a clear mandate in the form of an
enterprise policy statement. Such support is crucial to gaining the
commitment of all organizational components of the enterprise, all of whom
should participate in developing and implementing the enterprise
architecture.

The enterprise architecture effort should be directed and overseen by an
executive body, empowered by the head( s) of the organization( s), with
members who represent all stakeholder organizations and have the authority
to commit resources and to make and enforce decisions for their respective
organizations.

The enterprise architecture effort should be led by a Chief Architect who
reports to the enterprise CIO, and it should be managed as a formal program.
A formal program entails creating a program office; committing core staff;
implementing a program management plan that details work breakdown structure
and schedules; allocating resources and tools; performing basic program
management functions (e. g., risk management, change control, quality
assurance, configuration management); and tracking and reporting progress
against measurable goals.

The enterprise architecture should conform to a specified framework.
Although State modernization officials have acknowledged the importance of
having an overseas- presence enterprise architecture and using this
architecture to guide and constrain investment in a common overseas
knowledge management system, to date substantive progress in this area has
not occurred. Currently, State and its agency partners have yet to either
develop the architecture or establish effective management structures and
processes for developing, maintaining, and implementing one, such as those
specified in federal CIO Council guidance. More specifically, although State
established an Enterprise Architecture Study Group in December 2000, the
group met only twice and progressed no further than drafting a charter and
setting a goal of completing the architecture by November 2001.

19 Chief Information Officer Council, A Practical Guide to Federal
Enterprise Architecture, Version 1. 0 (February 2001).

Page 21 GAO- 02- 41 State- Led Modernization Initiative

On June 8, 2001, State designated a Chief Architect to lead this effort. In
July 2001, State reconvened the group, renaming it the Foreign Affairs
Systems Integration Architectural Working Group, and set February 15, 2002,
as the new goal for completing the architecture. Since reconvening, the
working group has held three meetings to begin planning for development of
the overseas- presence enterprise architecture and developed a draft
charter. The working group has also begun surveying its agency partners to
obtain general information on architectural requirements of overseas
locations. State?s ongoing prototyping and planned piloting efforts, which
are intended to provide information to better understand business and
technology requirements and alternate approaches, should facilitate
development of the architecture.

It is essential that State and its partner agencies effectively develop,
maintain, and implement an enterprise architecture for its overseas
modernization program. The federal CIO Council architecture management guide
provides a road map for doing so. Our experience with system modernization
programs in other federal agencies has shown that attempting to define and
build major systems without first completing an enterprise systems
architecture often results in systems that are duplicative, not well
integrated, unnecessarily costly to maintain and interface, and not
effectively optimizing mission performance. 20

The Clinger- Cohen Act requires agency CIOs to establish effective IT
management processes, such as processes for acquiring software. 21 The SEI,
recognized for its expertise in software process controls, has developed a
model approach for software acquisition. This model outlines the key
acquisition process management controls that, if implemented effectively,
can greatly increase the chances of acquiring softwareintensive systems that
provide promised capabilities on time and within budget. The key processes
are as follows:

20 See, for example, Air Traffic Control: Complete and Enforced Architecture
Needed for FAA Systems Modernization (GAO/ AIMD- 97- 30, February 3, 1997)
and Customs Service Modernization: Architecture Must Be Complete and
Enforced to Effectively Build and Maintain Systems (GAO/ AIMD- 98- 70, May
5, 1998).

21 Clinger- Cohen Act of 1996, Public Law 104- 106. Disciplined Acquisition

Management Processes Can Further Reduce Program Risks

Page 22 GAO- 02- 41 State- Led Modernization Initiative

Software Acquisition Planning: Ensures that reasonable planning for the
software acquisition is conducted and that all aspects of the total software
acquisition effort are included in these plans.

Solicitation: Ensures that a request for proposal that delineates a
project?s software- related requirements is prepared, and, consistent with
relevant solicitation laws and regulations, that a contractor that can most
costeffectively satisfy these requirements is selected.

Requirements Development and Management: Establishes and maintains a
common and unambiguous definition of software requirements among the
acquisition team, the system users, and the software development contractor.

Project Office Management: Provides for management of the activities
within the project office and supporting contractors to ensure a timely,
efficient, and effective software acquisition.

Contract Tracking and Oversight: Ensures that the software development
contractor performs according to the terms of the contract; needed contract
changes are identified, negotiated, and incorporated into the contract; and
contractor performance issues are identified early, when they are easier to
address.

Evaluation: Determines whether the acquired software products and services
satisfy contract requirements prior to acceptance.

Transition and Support: Provides for the effective and efficient ?hand-
off? of the acquired software products to the support organization
responsible for software maintenance.

Acquisition Risk Management: Identifies risks as early as possible and
adjusts the acquisition to mitigate those risks.

Within these key processes, SEI identifies key practices that are needed to
effectively execute each key process. Among others, these key practices
include: (1) designating responsibility for activities and assigning
adequate resources; (2) having a written policy; (3) developing,
documenting, and adhering to a plan; (4) performing management review
activities; and (5) measuring the status of key activities, and using these
measurements to make decisions.

According to State program officials, the modernization program will be
managed in accordance with State?s methodology, Managing State Projects
(MSP). MSP specifies major project management activities to be performed,
the products to be generated, and the control gates (i. e., decision points)
to be used to ensure that projects are ready for the next

Page 23 GAO- 02- 41 State- Led Modernization Initiative

phase in their life cycle. We analyzed State?s MSP and found that it is
consistent with best practice models, such as SEI?s acquisition model. 22

For its near- term prototyping and pilot- testing activities, which do not
demand the level of management process control rigor and discipline
associated with acquiring a system solution intended for operational use,
State is employing adequate acquisition management process controls. For
example, State has defined a work breakdown structure and schedule to guide
near- term activities. Further, State CIO officials told us that management
plans relating to key process areas have been drafted, although they did not
provide us with copies of the draft plans, as we requested. According to
State, these plans will be available when finalized and approved. These
officials also told us that program management is using two automated tools
to track the status of program activities. We verified that State has been
using one of these tools to track programrelated tasks. 23

According to State, it has not yet established acquisition management
controls for the future operational version of the overseas- presence
knowledge management system because it does not have congressional
authorization to plan beyond the pilot project for this initiative. Beyond
intentions to employ those controls already in place for key processes for
the prototype and pilot efforts (i. e., designating a responsible official
and having a written policy), State has important practices that will need
to be established before it is ready to acquire the operational system.
Table 3 summarizes the key process areas for managing acquisition of the
operational version of the system that already exist and those that will
need to be established.

22 Based on a comparison of the MSP?s major process areas with SEI?s key
processes. 23 We did not pursue verification of State?s use of the other
tool.

Page 24 GAO- 02- 41 State- Led Modernization Initiative

Table 3: Summary of Needed Process Controls for Managing the Acquisition of
the Operational Version of the System Process area

Designate responsible official

Assign adequate resources

Have a written policy a

Document a plan outlining activities

Perform management review of activities

Measure and report on status of activities

Software acquisition planning

Solicitation

Requirements development and management

Project office management

Contract tracking and oversight

Evaluation

Transition and support

Acquisition risk management

-Satisfied

- Not satisfied a Based on State?s plans to adhere to its Foreign Affairs
Manual, which requires implementation of its

MSP methodology. Source: State.

It is imperative that State ensures that it fully implements key acquisition
management process controls, as defined in its MSP and the SEI model, on its
overseas knowledge- management modernization program before acquiring the
operational system. Our reviews of other agency modernization efforts have
shown that failure to implement rigorous and disciplined acquisition
processes on a given program or project can lead to systems that do not
perform as intended, cost more than expected, and are not delivered on time.
24

24 See, for example, Customs Service Modernization: Serious Management and
Technical Weaknesses Must Be Corrected (GAO/ AIMD- 99- 41, February 26,
1999) and Land Management Systems: BLM?s Actions to Improve Information
Technology Management

(GAO- 01- 282, February 27, 2001).

Page 25 GAO- 02- 41 State- Led Modernization Initiative

[in...@www.gao.gov]

The scope of the near- term activities for State?s overseas- presence
knowledge management modernization program, as described by State CIO
officials and described in draft plans, is appropriately limited to system
prototyping and associated pilot testing of a version of the operational
system. In light of this scope, as well as State?s leadership role in
managing and funding these near- term efforts and its efforts to date to
bring its agency partners together, there is currently an adequate level of
(1) explicit definition and understanding surrounding the overseaspresence
agencies? respective responsibilities and accountabilities, (2) agency
involvement in the program, and (3) State?s management control discipline.

However, before State can effectively acquire and deploy the operational
knowledge management system, it needs to more explicitly define stakeholder
agencies? program management and funding roles, as well as establish more
rigorous and disciplined management controls. Accordingly, it is critical
for State to also move aggressively in the near term to (1) clearly define,
through written agreements, the respective management and funding
responsibilities and accountabilities of the program?s agency partners and
(2) institute key modernization management controls, including incremental
investment management practices, enterprise architecture governance, and
software acquisition management rigor and discipline. While State has
verbally committed to addressing these areas, much remains to be done before
State and its agency partners will be ready to proceed beyond prototyping
and pilot testing and begin acquiring the system solution intended for
operational deployment and use. Attempting to acquire the operational system
solution without these controls risks not delivering needed system

capabilities on time and within budget.

To provide for an explicit definition and understanding of the respective
roles of all overseas- presence modernization agency stakeholders, we
recommend that the Secretary of State develop and submit a proposal for OMB
and/ or congressional action, whichever the Secretary deems appropriate,
within 60 days of the date of this report that clearly defines and assigns
responsibilities and accountabilities for each overseaspresence agency
regarding the knowledge management system modernization program. We further
recommend that the Secretary direct the Under Secretary for Management to
ensure that this proposal (1) sets expectations for each overseas- presence
agency, including the State Department, relative to modernization management
and funding and (2) is Conclusions

Recommendations for Executive Action

Page 26 GAO- 02- 41 State- Led Modernization Initiative

coordinated with and agreed to by the Under Secretary?s counterparts in each
of the partner agencies.

To ensure that key system modernization management controls are implemented
before the operational system is acquired, we recommend that the Under
Secretary of State for Management direct State?s CIO, as chair of the
Interagency Technology Subcommittee, to ensure that the subcommittee allows
only architecturally compliant knowledge management system investments to be
approved and funded, unless the subcommittee issues a written waiver in
response to a written justification. Furthermore, we recommend that the
Under Secretary direct the CIO to ensure that architecturally compliant
investments in the system be made incrementally. To this end, the CIO should
ensure that processes and structures are established for (1) acquiring the
system in a series of smaller system increments; (2) individually justifying

investment in each increment on the basis of costs, benefits, and risks; and
(3) monitoring actual benefits achieved and costs incurred on ongoing

increments and modifying subsequent increments to reflect lessons learned.

To further ensure that key management controls are implemented on the
overseas- presence knowledge management system modernization, we recommend
that the Secretary of State designate as a modernization program priority
the development, implementation, and maintenance of an overseas foreign-
affairs presence enterprise architecture. To this end, we recommend that the
Secretary direct the Under Secretary for Management to establish an
enterprise architecture steering committee, chaired by the Under Secretary
and composed of the Under Secretary?s counterparts from the other overseas-
presence agencies. We also recommend that this steering committee be
assigned responsibility and accountability for ensuring that a complete and
approved enterprise architecture is developed and available to guide and
constrain overseas- presence system modernization efforts. To assist the
steering committee, we recommend that the Under Secretary direct the State
CIO to establish an enterprise architecture project office, headed by the
recently named Chief Architect, to develop and maintain the enterprise
architecture and submit appropriate versions of this architecture to the
steering committee for approval. We further recommend that the State CIO, at
a minimum, ensure that the project office (1) include staff from each of the
overseas- presence agencies; (2) integrate the architecture with these
agencies? respective operational and systems environments; and (3) adhere,
as appropriate, with the federal CIO Council?s published guidance on
managing enterprise architectures.

Page 27 GAO- 02- 41 State- Led Modernization Initiative

To further address State?s need to implement modernization management
controls, we recommend that the Under Secretary for Management direct the
State CIO to ensure that the overseas- presence system modernization office,
consistent with State?s project management requirements and recognized
models for effective software acquisition, establishes and implements
rigorous and disciplined acquisition processes, including processes for
acquisition planning, contractor solicitation, requirements development and
management, contractor tracking and oversight, testing and evaluation,
transition to support the acquired system, and risk management.

Until these recommended modernization management controls are in place, we
also recommend that the Secretary limit further investment in the overseas-
presence knowledge management modernization program to (1) conducting
ongoing prototyping and limited pilot testing and (2) establishing needed
modernization management controls.

In written comments on a draft of this report (reprinted in appendix II),
State?s Acting Chief Financial Officer agreed with our findings and

conclusions that its management controls over near- term prototype and

pilot- testing activities are adequate, but disagreed with our finding and
conclusion that State?s existing controls are not adequate for effectively
managing the more demanding next phase of the program- acquisition and
deployment of the operational system. Furthermore, State disagreed with our
recommendations for instituting stronger management controls before
proceeding to this next phase. According to State, our recommendations
should be deleted from the report because (1) we reported that management

controls over near- term activities are adequate and thus corrective actions

are not needed; (2) except for clearly defining and agreeing to all
agencies? roles and responsibilities, State?s existing management controls
are adequate for acquiring and deploying the operational system; and (3) the

lead agency for the next phase of the program is uncertain and thus the

recommendations are premature. At the same time, State said that most of the
recommendations might more reasonably be submitted as suggestions for
consideration to the lead agency for the next phase of the program.

We do not agree with State?s comment that its existing management controls
are adequate for the next phase of the program. As stated in our report, the
nature and purpose of system prototyping and pilot testing are fundamentally
different; thus, the associated risks are less significant than those
related to acquiring a system that is to be operationally deployed. As
Agency Comments

and Our Evaluation

Page 28 GAO- 02- 41 State- Led Modernization Initiative

a result, it is widely recognized that the level of management control to
effectively put in place an operational system, particularly a system like
the overseas knowledge- management system that involves multiple agencies,
is greater than is needed to prototype or pilot test the system. Thus, for
example, while State?s lack of an enterprise architecture to guide and
constrain near- term prototype and pilot- testing activities does not

introduce significant program risk, it would introduce considerable risk for

State to begin the next program phase without this architecture. Similarly,
while State?s approach to managing its investment in near- term activities
is adequate, effectively acquiring and deploying the operational system will
require rigorous management processes that, for example, provide for
incremental economic justification and measurement of system return- on-
investment, and provide for fully implementing disciplined acquisition
management practices in such areas as managing system requirements.

We also do not agree with State?s rationale for deleting the report?s
recommendations because these recommendations (1) are not intended to
correct problems with management of the prototype and pilot activities, as
State asserts, but rather to proactively provide a recommended road map for
how to successfully manage the next phase of the program; (2) are consistent
with our conclusions that existing management controls are not adequate for
the next phase of the program; and (3) recognize that prudent planning
requires that agency roles and responsibilities for the next phase of the
program, including designation of the lead agency, be clearly defined and
agreed to during this phase to better ensure program continuity and to avoid
unnecessary delays. Accordingly, the recommendations are both appropriate
and warranted.

State?s written comments, along with our responses, are reproduced in
appendix II.

We are sending copies of this report to the Chairman and Ranking Minority
Members of the Senate Committee on Governmental Affairs; House Committee on
Government Reform; Senate Committee on Foreign Relations; House Committee on
International Relations; the Subcommittee on Commerce, Justice, State, and
the Judiciary, Senate Committee on Appropriations; and the Subcommittee on
Commerce, Justice, State, the Judiciary and Related Agencies, House
Committee on Appropriations. We are also sending copies to the Director,
Office of Management and Budget; the Secretary of State; the Under Secretary
for Management; and the Chief

Page 29 GAO- 02- 41 State- Led Modernization Initiative

Information Officer, the Department of State. Copies will be made available
to others upon request.

If you have any questions on matters discussed in this report, please call
me at (202) 512- 3439 or Cynthia Jackson, Assistant Director, at (202)
5125086. We can also be reached by e- mail at hiter@ gao. gov and jacksonc@
gao. gov, respectively. Key contributors to this assignment are listed in
appendix III.

Sincerely yours, Randolph C. Hite Director, Information Technology Systems
Issues

Appendix I: Objectives, Scope, and Methodology

Page 30 GAO- 02- 41 State- Led Modernization Initiative

Our objectives were to determine (1) what State?s plans are for a
multiagency, common platform, knowledge management system, and (2) whether

State has effective controls in place for acquiring this system, that is,

defined roles for State and its agency partners and effective controls over

its investment, enterprise architecture, and system acquisition management
processes.

To determine State?s plans for the system, we reviewed State budget
submissions, relevant congressional direction (hearing testimony,
transcript, and related legislation), the results of our prior review, 25
the Overseas Presence Advisory Panel report and recommendations, 26 and
available State modernization program management plans and documents.
Additionally, we interviewed State modernization program officials,
including the Deputy Chief Information Officer (CIO)/ Chief Knowledge
Officer, Deputy Chief Knowledge Officer, and the Director for Standards,
Policy, Planning, and Architecture, to discuss State?s near- and long- term
plans, as well as representatives of the Overseas Presence Committee and the
Interagency Technology Subcommittee to discuss the status of these plans and
activities, and inter- and intra- agency coordination concerns. To confirm
the extent to which State had engaged its agency partners, we surveyed
State?s eight agency partners to determine whether they (1) were satisfied
with State?s efforts to engage them in the program, (2) had reviewed and
approved the program plans and related documents, and (3) believed that
State was adequately addressing their requirements and/ or information
needs. In surveying State?s agency partners, we contacted and received
responses from those individuals designated by their respective agencies as
the authorized representatives for this initiative.

To determine whether State has effective controls in place for acquiring the
system, we did the following.

25 U. S. Department of State: The Budget in Brief- Fiscal Year 2002; U. S.
Department of State: Congressional Presentation Document- Fiscal Year 2001
Budget Volume I;

Written Testimony of Fernando Burbano, Department of State Chief Information
Officer, presented to The House Committee on International Relations (June
22, 2000); Oversight of the State Department: Technology Modernization and
Computer Security, Hearing before the Committee on International Relations,
House of Representatives, Serial No. 106- 171, June 22, 2000; Office of

Management and Budget (OMB) Circular A- 130, Management of Federal

Information Resources (November 30, 2000); and Foreign Affairs: Effort to

Upgrade Information Technology Overseas Faces Formidable Challenges

(GAO/ T- AIMD/ NSIAD- 00- 214, June 22, 2000). 26 America?s Overseas

Presence in the 21st Century: The Report of the Overseas Presence Advisory

Panel, U. S. Department of State (November 1999). Appendix I: Objectives,
Scope, and

Methodology

Appendix I: Objectives, Scope, and Methodology

Page 31 GAO- 02- 41 State- Led Modernization Initiative

To determine whether State and its agency partners? roles,
responsibilities, and accountabilities had been formally defined for each
phase of the program (e. g., prototype, pilot), we reviewed documentation
provided by State (e. g., briefing slides, meeting minutes, working group
charters), interviewed State officials, and surveyed State?s eight agency
partners, as described above.

To determine State and its agency partners? plans for managing investment
in the system, we reviewed relevant legislative requirements and associated
OMB guidance, 27 as well as applicable State policy and guidance, on
incremental investment management. 28 We then obtained and analyzed
documentation outlining State?s investment review process for selecting,
controlling, and evaluating projects since plans are currently for State to
fully fund the project (prototype and pilot). Because documentation on
State?s investment management activities to date and future plans for this
system were not available, we interviewed officials from the modernization
program office [Foreign Affairs Systems Integration (FASI) Office] and
Deputy CIO for Architecture, Planning, and Regulations Office to determine
if State and its partners planned to use an incremental investment
management strategy. We then compared available information on State?s
investment management plans against relevant federal requirements and
guidance to identify if any variances existed.

To determine whether an enterprise architecture exists for the system to
guide and constrain investment, we reviewed relevant legislative
requirements and associated OMB guidance, 29 as well as federal CIO Council
published guidance 30 and applicable State policy and guidance, 31 on
developing, maintaining, and implementing an enterprise architecture.

27 Clinger- Cohen Act of 1996, Public Law 104- 106; OMB Circular A- 130
(November 30, 2000);

Capital Programming Guide, Version 1. 0, OMB Circular A- 11, part 3 (July

1997), pp. 545- 572; and Capital Programming Guide, Version 1. 0, OMB

Circular A- 11, part 3, Supplement, Planning, Budgeting, and Acquisition of

Capital Assets (July 2000), pp. 3537.

28 U. S. Department of State Foreign Affairs Manual, Section 5 FAM 626. 2:
Incremental

(February 4, 2000, update). 29 Clinger- Cohen Act of 1996, Public Law 104-
106 and OMB Circular A- 130 (November 30, 2000). 30 Chief Information

Officer Council, A Practical Guide to Federal Enterprise Architecture,

Version 1. 0 (February 2001). 31 U. S. Department of State Foreign Affairs
Manual, Section 5 FAM 115: IT Management Oversight Objectives (January 23,
2001, update).

Appendix I: Objectives, Scope, and Methodology

Page 32 GAO- 02- 41 State- Led Modernization Initiative

We then reviewed available documentation, including plans and management
structures, a charter, process controls, and architectural artifacts, as
well as interviewing FASI Program Office officials, including the Director
for Standards, Policy, Planning, and Architecture, to determine the state of
any enterprise architecture development and implementation efforts. We then
compared the available information to relevant federal requirements and
guidance to identify if any variances existed.

To determine whether State and its agency partners have the needed
capabilities to effectively acquire the system, we obtained and analyzed
documentation on agencywide policies and procedures governing system
acquisition efforts, which State refers to as its Managing State Projects
methodology. 32 We then reviewed available best practice models on system
and software acquisition management (e. g., the Software Engineering
Institute (SEI) model 33 ) and compared the primary components of the
Managing State Projects methodology against the primary components of SEI?s
model to determine if any variances existed. SEI?s primary components
include software acquisition planning, solicitation, requirements
development and management, project office management, contractor tracking
and oversight, evaluation, transition and support, and risk management.
Because we found that the primary components of State?s methodology were
consistent with best practices, we obtained and reviewed available program
management and system documentation and interviewed FASI Program Office
officials to determine the extent to which the methodology was being
implemented on the program.

We performed our work at State Department headquarters in Washington, D. C.,
from October 2000 through November 2001 in accordance with generally
accepted government auditing standards.

32 Managing State Projects, U. S. Department of State (January 19, 1999). 33
Software Acquisition Capability Maturity Model SM (SA- CMM). Capability
Maturity Model SM is a service mark of Carnegie Mellon University, and CMM
is registered in the U. S. Patent and Trademark Office.

Appendix II: Comments From the Department of State

Page 33 GAO- 02- 41 State- Led Modernization Initiative

Appendix II: Comments From the Department of State

Note: GAO comments supplementing those in the report text appear at the end
of this appendix.

See comment 1.

Appendix II: Comments From the Department of State

Page 34 GAO- 02- 41 State- Led Modernization Initiative

See comment 3. See comment 2.

Appendix II: Comments From the Department of State

Page 35 GAO- 02- 41 State- Led Modernization Initiative

Appendix II: Comments From the Department of State

Page 36 GAO- 02- 41 State- Led Modernization Initiative

See comment 8. See comment 7. See comment 6.

See comment 5. See comment 4.

Appendix II: Comments From the Department of State

Page 37 GAO- 02- 41 State- Led Modernization Initiative

See comment 8.

Appendix II: Comments From the Department of State

Page 38 GAO- 02- 41 State- Led Modernization Initiative

See comment 10. See comment 9.

Appendix II: Comments From the Department of State

Page 39 GAO- 02- 41 State- Led Modernization Initiative

See comment 11.

Appendix II: Comments From the Department of State

Page 40 GAO- 02- 41 State- Led Modernization Initiative

See comment 12.

Appendix II: Comments From the Department of State

Page 41 GAO- 02- 41 State- Led Modernization Initiative

Appendix II: Comments From the Department of State

Page 42 GAO- 02- 41 State- Led Modernization Initiative

Appendix II: Comments From the Department of State

Page 43 GAO- 02- 41 State- Led Modernization Initiative

Appendix II: Comments From the Department of State

Page 44 GAO- 02- 41 State- Led Modernization Initiative

1. See comments 2 through 12. 2. We did not address the adequacy of funding
for near- term activities as part of this review.

3. We disagree. Our report clearly links our findings, conclusions, and
recommendations. Specifically, the report states that the level of

management rigor and discipline needed to effectively manage prototype and

pilot- testing activities is not as demanding as that needed for acquisition
and deployment of a system that is to be used operationally. For example,
while State?s lack of an enterprise architecture to guide and constrain
prototype and pilot- testing activities does not introduce significant risk,

it would introduce considerable risk for State to begin the next phase of

the program without this architecture. Accordingly, we concluded that
State?s existing management controls were not adequate for the next phase of
the program, and we made recommendations that are intended to proactively
provide a framework for effectively managing the more demanding next phase
of the program. We made no recommendations intended to correct problems with
State?s management of its near- term activities because we concluded that
controls over the less demanding prototype and pilot- testing activities
were adequate.

4. We disagree. Our report clearly describes how State?s existing management
controls are not adequate for the next phase of the program. For example,
State does not yet have an enterprise architecture to effectively guide and
constrain investment in the system. Our experience with federal agencies has
shown that attempting to define and build major systems without first
completing and using an enterprise architecture often results in systems

that are duplicative, not well integrated, unnecessarily costly to maintain
and interface, and not effectively optimizing mission performance.

Similarly, State has not yet implemented an effective process for managing
requirements for an operational system, as evidenced in the report by the
fact that agency partners have not formally approved requirements for the
prototype and pilot systems. Although not having formally approved
requirements for building prototypes and conducting pilot tests does not
introduce significant risks, it would introduce considerable risk for State
to acquire a system that is to be operationally deployed without having
formally approved requirements.

5. We agree that State has not proposed program and management plans beyond
the prototype and pilot- testing activities. GAO Comments

Appendix II: Comments From the Department of State

Page 45 GAO- 02- 41 State- Led Modernization Initiative

6. We agree. State?s plans provide for preparing a prototype and pilot
testing evaluation report. Also, such a report will be needed by not only
the lead agency for acquiring and deploying the operational system, but also
all participating agencies that will need to integrate the operational
system with their respective legacy systems.

7. See comments 3, 4, and 5. 8. We agree. As stated in our report, the
Secretary of State was directed to lead a cabinet- level committee to
implement the recommendations of the Overseas Presence Advisory Panel, which
included acquiring and deploying a multiagency common platform knowledge
management system. While we recognize the possibility that State might not
be the lead agency for the next phase of the program, we strongly believe
that, to allow for program continuity and avoid unnecessary delays, these
issues must be addressed now. Since State is currently the lead agency for
the program, the recommendations are being directed to the Secretary of
State.

9. We agree with State?s comment, and note in our report, that the roles,
responsibilities, and accountabilities of State and its agency partners
would need to be fully and formally defined before moving beyond planned
near- term activities. Although State asserted during the course of our
review, and in its comments, that its present informal structure for
managing agency relationships is working and its comments assert that it
provided documents to us attesting to this on numerous occasions, this
documentation, while voluminous and repetitive, did not demonstrate what the
current process is and whether it was working. As a result, we were forced
to expand the scope of work to include a survey of State?s agency partners
to determine whether State had engaged them in programrelated activities.

From our survey, we were able to conclude that State had taken adequate
action to engage the agencies. Nevertheless, some agencies raised concerns
that must be addressed before beginning the next phase of the program. For
example, one agency partner stated that it had not participated in an
approval process for program planning documentation, another partner stated
that it was unaware of any approval process for planning- related
activities, and a third stated that it had not reviewed any of the program
planning documentation. Moreover, one agency partner stated that a more
formal program governance structure is needed, especially for reviewing and
approving such documents as the program?s concept of operations and the
system?s functional requirements. A similar

Appendix II: Comments From the Department of State

Page 46 GAO- 02- 41 State- Led Modernization Initiative

statement was made in a meeting of the State- led Foreign Affairs Systems
Integration Architectural Working Group. Specifically, it was stated that
protocols needed to be established, such as officially listing the documents
that would be shared so that all participants would be aware of these
documents and have the opportunity to review them before making any
decisions about how to proceed. While these issues are less significant for
program near- term activities, they should be resolved before moving beyond
system prototyping and piloting activities. Accordingly, our report contains
proactive recommendations for doing so.

10. We agree with State?s comment that the need to formally define agency
roles, responsibilities, and accountabilities should be focused on the next
phase of the program. We disagree that there is a need to eliminate a
reference to insufficiency in defining existing roles, responsibilities, and
accountabilities regarding near- term program prototype and pilot- testing
activities because no such reference exists. In fact, our report concludes,
and State acknowledges in its comments, that management controls over near-
term activities are adequate.

11. See comments 3 and 4. 12. See comments 3, 4, and 5.

Appendix III: GAO Contact and Staff Acknowledgment

Page 47 GAO- 02- 41 State- Led Modernization Initiative

Cynthia Jackson, (202) 512- 5086 In addition to the person named above,
Nabajyoti Barkakati, Katherine I. Chu- Hickman, Lester P. Diamond, Patrick
R. Dugan, Sophia Harrison, and Kenneth A. Johnson made key contributions to
this report. Appendix III: GAO Contact and Staff

Acknowledgment GAO Contact Acknowledgments

(310204)

The General Accounting Office, the investigative arm of Congress, exists to
support Congress in meeting its constitutional responsibilities and to help
improve the performance and accountability of the federal government for the
American people. GAO examines the use of public funds; evaluates federal
programs and policies; and provides analyses, recommendations, and other
assistance to help Congress make informed oversight, policy, and funding
decisions. GAO?s commitment to good government is reflected in its core
values of accountability, integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents is through the
Internet. GAO?s Web site (www. gao. gov) contains abstracts and full- text
files of current reports and testimony and an expanding archive of older
products. The Web site features a search engine to help you locate documents
using key words and phrases. You can print these documents in their
entirety, including charts and other graphics.

Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as ?Today?s Reports,? on its Web
site daily. The list contains links to the full- text document files. To
have GAO E- mail this list to you every afternoon, go to our home page and
complete the easy- to- use electronic order form found under ?To Order GAO
Products.?

The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent of
Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more
copies mailed to a single address are discounted 25 percent. Orders should
be sent to:

U. S. General Accounting Office P. O. Box 37050 Washington, D. C. 20013

To order by phone: Voice: (202) 512- 6000 TDD: (301) 413- 0006 Fax: (202)
258- 4066

GAO Building Room 1100, 700 4th Street, NW (corner of 4th and G Streets, NW)
Washington, D. C. 20013

Contact: Web site: www. gao. gov/ fraudnet/ fraudnet. htm, E- mail:
fraudnet@ gao. gov, or 1- 800- 424- 5454 (automated answering system).

Jeff Nelligan, Managing Director, NelliganJ@ gao. gov (202) 512- 4800 U. S.
General Accounting Office, 441 G. Street NW, Room 7149, Washington, D. C.
20548 GAO?s Mission

Obtaining Copies of GAO Reports and Testimony

Order by Mail or Phone Visit GAO?s Document Distribution Center

To Report Fraud, Waste, and Abuse in Federal Programs

Public Affairs
*** End of document. ***