ASTEWG TEMPO

Autonomous System Test and Evaluation Working Group

The ASTEWG Charter: to develop and promote principles and practices that produce confident assessment and diagnosis of system(s) capability, whenever and wherever needed.

Motivation: Autonomy is a growing presence due to continued exponential growths of computing and network technologies and to evolving theories of higher-order systems. This results in ubiquitous and persistent trends towards a spectrum of autonomous systems. Further, autonomy applies not only to how a system interacts with its context but also how its components can autonomously adapt internal content, interrelationships and gradients on the relationships.

Advancements in autonomy have created new challenges and opportunities in systems praxis, both in new ways of formulating models and ensuring responsive realization of systems (new modes of systems engineering) and in new ways of determining and diagnosing the viability of a system (new horizons in test and evaluation) as a system evolves from initial concept to deployment to N^th year in operation.

System engineering must transition from:

creating a prescient scenario of how a system must operate, how the enabling functions and features must be arranged and specifying the components to be engineered to
creating a set of interoperable capabilities and the rules by which they can be manifest and be organized or may self-organize for specified purpose.

In parallel, system engineering must nominate experiments, tests and exercises to be applied at the various stages of system instantiation in order to ensure a deliverable and reproducible system configuration that sustains Fit for Purpose even while engagement scenarios and measures of effectiveness change.

System viability assessment and diagnostics must cope with systems far greater in extent, variety and ambiguity than has been the case and must accomplish its purpose in approximately one tenth the time and cost than has been the case. An integration of design reviews, verification (system right), validation (right system), simulation and test and evaluation will not suffice. Instead we must system engineer, develop, deploy and continuously adapt a new enterprise that is exemplary in system viability assessment and diagnosis.

It must generate viability and diagnostic systems that discover and convey useful knowledge regarding not only

whether the system of interest meets requirements but also
whether the system of interest is fit for a specified purpose and beyond to
also discover the dynamic and integrity limits of any envisioned or manifest system of interest.

Knowledge discovery, exchange and vetting: We presume that although some guidance exists in available literature the major effort toward an assessment and diagnostic enterprise must be conducted by current and future members of this group. We shall become an autocatalytic system that produces a model of an autonomous system assessment and diagnosis system and encourages realization of such systems wherever and whenever needed.

Various members will author and share essays regarding principles and practices relevant to:

PS, the Problematic Situation (of assessing and diagnosing viability of systems that are large in extent, variety and ambiguity) that identifies stakeholders or subsets thereof and identifies their tensions or angst with respect to the panoply of autonomous systems being developed.
Underlying Problem System that generates the tensions or angst.
Stakeholder preferences, both positive and negative, for mediating, resolving or dissolving the tensions and angst.
Strategy for intervention, including identifying Objectives, impediments, applicable resources and formulation of resource allocation and scheduling scenario.
Designs and architectures for an exemplary assessment and diagnosis system that can be manifest with the requisite variety, saliency, responsiveness and accuracy befitting the problematic situation.
Designs and architectures for an exemplary enterprise that can carry on from WG essays to realizations of exemplary assessment and diagnosis systems.
Recommendations for determining fallibility (dynamic and integrity limits) in any proposed enterprise.
Recommendations for determining fallibility (dynamic and integrity limits) in any proposed assessment and diagnosis system model.
Recommendations for determining fallibility (dynamic and integrity limits) in strategy, stakeholder preferences, problem system and problematic situation.

Essays will range from a few hundred word ‘talking points’ memos to a couple of thousand word (two pictures?) working papers to conference posters, papers, and tutorials, to journal articles and book chapters.

Sharing and dialogue about essay content will be enabled by use of a knowledge exchange site such as xxxxxxxx and twice monthly teleconferences/webinars. Convergence sessions will be held at IW and IS as well as at other relevant workshops and conferences.

AS->PS: Autonomous Systems addressing Problematic Situations
SE->AS: Systems Engineering addressing Autonomous Systems
SE=DD+V&V+T&E: Systems Engineering as a function of Design, Development, Verification, Validation, Test & Evaluation (independent & objective)

Possible Histories:

The value of conventional T&E is waning
The advantage of fielding systems with limited T&E outweighs the risk
The complexity of new systems (with expanding composability) outstrips T&E ability to keep pace

Belief (possibility): the what-why-how

A form of T&E exists that can lead to requisite AS capable of addressing targeted Problematic Situations

Desire (preferences):

AST&E must address autonomous components
AST&E must address autonomous systems
AST&E must address composability

Intentions (commitments): the when-where-how

ASTEWG intends to develop ASTE principles and practices
ASTEWG intends to promote ASTE principles and practices
ASTEWG intends to produce confident assessment and diagnosis of system(s) capability, whenever and wherever needed

Possible Futures:

ASTE fails to materialize and users remain defacto TE group in the value stream
ASTE imposed by regulation fails to contribute in a meaningful way leading to waste of resources in design and development and more A’s for effort instead of A’s for accomplishment in addressing problematic situation
ASTE orchestrated as a synchronized OODA process with design and development leads to requisite capability in addressing problematic situations that can only be solved with autonomy

21 Principles from Asilomar "Beneficial AI Conference"

Jack Ring