Architecture Frameworks in System Design: Motivation, Theory, and ...

Architecture Frameworks in System Design: Motivation, Theory, and Implementation Matthew G. Richards, Nirav B. Shah, Daniel E. Hastings, and Donna H. Rhodes Massachusetts Institute of Technology 77 Massachusetts Ave., NE20-343 Cambridge, MA 02139 Copyright © 2007 by Richards, Shah, Hastings, and Rhodes. Published and used by INCOSE with permission.

Abstract. This research examines the underlying rationale of architecture frameworks and their use in acquisition with the goal of identifying best practices to improve system development. Three key questions are addressed: (1) What is the role of artifacts in system design? (2) What are the benefits provided by architecture frameworks? And (3) what are the best measures-ofeffectiveness for assessing the value of architecture frameworks? To address the first question, we review relevant literature on collaborative design and propose a rationale for using artifacts as communicative and illustrative tools. Second, we enumerate the motivations for their use in system design and acquisition. Third, eight measures-of-effectiveness of architecture frameworks are derived from the literature and our experience with the Department of Defense Architecture Framework.

Introduction Architecture frameworks are tools for managing system complexity by structuring data into views with a common language and format. By characterizing the form, function, and rules governing systems, architecture frameworks (1) serve as a communication tool to stakeholder communities with different views of the system and (2) facilitate comparative evaluation across architectures. Recent evidence collected by the US Department of Defense suggests that there is a gap between the intended use of architecture frameworks in acquisition and their deployment in industry (Office of the Assistant Secretary of Defense for Networks and Information Integration 2005). The authors’ experience using the Department of Defense Architecture Framework (DoDAF) to model the Hubble Space Telescope and associated servicing missions (Richards, Shah, Hastings, and Rhodes 2006) reinforce these recent findings. In particular, the static work products as promulgated by the DoDAF were insufficient to capture the dynamics of the spacecraft necessary to design a servicing architecture.1 This research examines both the underlying rationale of architecture frameworks and their use in acquisition with the goal of identifying best practices to improve system development. In order to understand how architecture frameworks are used, one must first consider the role of artifacts in system design. An architecture framework specifies particular artifacts that, if constructed in a coherent fashion, simultaneously manage complexity and enable communication. In addition, architecture frameworks provide a means to manage the entire system design process being executed by a diverse group of stakeholders. Therefore, the effectiveness of architecture 1

This companion paper was presented at the 2006 Conference on Systems Engineering Research (Los Angeles, CA) and may be downloaded from the MIT Engineering System Division’s website [http://esd.mit.edu/WPS/2007/esdwp-2007-09.pdf].

frameworks requires consideration of the artifacts produced and how they form a coherent whole.

Role of Artifacts in System Design Design artifacts may be defined as physical or virtual objects produced during the design process with the exception of the final product or system itself. In designing complex systems, typical engineering artifacts include physical objects, such as drawings and documents, as well as computational objects, such as computer-aided design (CAD) models and finite element solutions. The particular list of artifacts produced will vary from engineer to engineer, but all will agree that the production of these artifacts is a key step in the process from problem definition to engineered solution. Each of the artifacts produced plays a specific role in design. Broadly speaking, these roles may be seen as serving three key underlying functions: 1. Enabling effective communication 2. Ensuring that knowledge gained through the design is preserved 3. Managing complexity The first and most obvious role of artifacts is communication. Rarely does design occur in a vacuum. Requirements and specification documents are used to aid in communicating pertinent information between stakeholders involved in the design. The artifacts serve as boundary objects in space between stakeholder communities. Such information needs to survive beyond the current design effort to enable institutional learning and the passing on of experience. This need leads directly to the second role listed above, that of artifacts as boundary object in time— connecting one design effort to the next. The final role of artifacts, managing complexity, is more subtle. Ever since humans first began to design, they have used everything from clay tablets to CAD to reduce the instantaneous cognitive load during the design process. Even when working in isolation, the engineer naturally partitions the design problem and manages movement from one step to the next using artifacts. The use of artifacts as boundary objects between design roles and tasks is termed ‘managing complexity.’ While most designers produce artifacts to fulfill these roles, all artifacts may not necessarily be well-suited for their use in the design process. In practice, artifacts frequently serve multiple roles and, therefore, may consist of compromised representations. For example, the representation of a part that is best suited for aerodynamic analysis may not include all the detail needed to create manufacturing tooling. Here a compromise is made between a communicative role and a complexity management role. By removing details irrelevant to the aerodynamic analysis, the aeronautical engineer may reduce the complexity required to model the part. However, this reduction in complexity also reduces the usefulness of the part representation as a communication tool to the manufacturing engineer. This example of the compromise between local artifact optimization and global applicability is representative of a broader class of tradeoffs, which exists for other design artifacts such as requirements specifications. Guidance for achieving the communicative roles in both space and time can be found in the literature on boundary objects and organizational knowledge literatures. The communicative role of using artifacts as boundary object between groups is thoughtfully explained in Carlile (2002) and the references cited therein. Carlile develops three characteristics on which to judge

the effectiveness or quality of a boundary object. First, a boundary object should establish a shared syntax or language for individuals to represent their knowledge. Common language establishes a baseline that is necessary for meaningful communication to take place. Second, a boundary object has a semantic role to reveal “differences and dependencies” between the communicating groups. Carlile uses the example of physical prototypes as a means of revealing dependencies between designers responsible for different functions within a product. He then observers, however, that revealing “differences and dependencies” is alone itself insufficient and that such information must also be actionable. Carlile’s third characteristic is that a boundary object should facilitate a joint transformation of a communicator’s knowledge of the system. For an artifact to be an effective boundary object, it must improve the design process as a whole. While Carlile’s discussion gives good guidance on the communicative role of artifacts, the knowledge management literature speaks more directly to the role of knowledge preservation. While a full discussion of issues related to artifacts as knowledge stores is beyond the scope of this paper,2 an overview of Nonaka, et al. (2000) provides an interesting framework for how knowledge3 may be transferred from one project to the next. Nonaka’s knowledge preservation framework consists of three elements: (1) SECI, a process conversion between tacit and explicit knowledge, (2) a shared context and knowledge creating interactions, and (3) the leadership actions that drive the SECI process. SECI is a four-fold process by which tacit knowledge (e.g., from a design effort) is transferred to another person. Socialization (the “S” in SECI) refers to the transfer of tacit knowledge through shared experience (e.g., apprenticeship). Externalization (the “E” in SECI) refers to a broader transfer of knowledge at the organizational level. Here, tacit knowledge is converted to explicit knowledge that can more readily be shared. Artifacts play a key role in capturing explicit knowledge. For example, requirements documents and prototypes serve to transform tacit ideas to explicit objectives and implementation strategies. Combination (the “C” in SECI) is the human action of assembling and recombining explicit knowledge into a form that is useful within the context at hand (e.g., a financial statement that aggregates and transforms individual transactions into an overall assessment). Finally, internalization (the “I” in SECI) is the process by which explicit knowledge is incorporated into the tacit knowledge base that governs actions taken by the individual. Taken as a whole, the SECI process provides an abstract framework through which to analyze knowledge transfer and preservation between individuals and organizations. There is less literature available on the final role of artifacts—managing complexity. Latour (1986) traces the history and development of engineering drawing as a means of maintaining consistency between design tasks. Henderson (1991) examines how engineers use sketches to enable rapid idea generation and communication to stakeholders. Her case study reveals one of the key tensions between artifacts as a means to manage complexity and artifacts as communicative/knowledge transfer vehicles. The latter roles of artifacts concern interaction between individuals and the former benefits to the individual. In trying to perform well with 2

See Alavi and Leidner (2001). Nonaka et al. (2000) define knowledge as ‘justified true belief’. ‘Belief’ means that knowledge is an idea that exists in the mind as opposed to an object or process. ‘True’ means that the knowledge represents reality. ‘Justified’ means that the mind that holds the idea subscribes to its truth. 3

respect to communication, providing complexity management may be compromised. For example, Henderson describes how the implementation of a company-wide CAD system to tie together and better coordinate processes between different functional units within the firm failed to create a set of effective boundary objects. Although intended to provide better consistency and visibility, the CAD system did not map well to the traditional processes by which design was accomplished. As a result, engineers used CAD as they had previously used sketches, defeating the intended purpose of the CAD process. This tension—between creating artifacts as contextspecific tools for reducing complexity and as tools for transferring knowledge across multiple contexts—underlies much of difficulty in creating architecture frameworks as is discussed in the following two sections.

Role of Architecture Frameworks in System Design The overall objective of architecture frameworks is to improve the ability of the system acquirer, builder, and user to make technical decisions. As the complexity of systems has increased, there has been a corresponding interest in developing descriptive standards. The primary purpose of such standards is to enhance the ability of engineers to communicate system architectural and design information in a manner which enables technical decisions by removing ambiguity, providing clarity, and dealing with complexity through the use of multiple views and viewpoints. Development of formal architectural frameworks is a relatively recent initiative, but efforts to find robust ways to describe system architectural and design information have been ongoing for decades. Karas and Rhodes (1987) discuss an early descriptive methodology for capturing high level system information using three views: functional, physical and operational (behavioral). The motivation for development of this method was the desire to capture systems information in a manner that enriched the overall system description. An additional driving factor was the advent of collaboration between multiple corporations in the development of a single system. This necessitated more formalism and rigor in system descriptions to be shared across geographically distributed teams. Architectural frameworks are further evolutions of this essential need to develop system descriptions that are complete, correct, coherent, and usable for multiple purposes and by multiple stakeholders in the development of systems. Modern architectural frameworks have resulted from the collaboration of many experts to develop a comprehensive and coherent set of views where each view reflects a unique stakeholder perspective. Maier and Rechtin (2000) identify five goals of architecture frameworks: (1) institutionalize best practices for architectural description, (2) ensure system sponsors receive information in the format they desire, (3) facilitate comparative evaluation of architectures, (4) improve the productivity of development teams, and (5) improve interoperability of information systems by requiring that critical interfaces are described. In addition to these overarching goals, we highlight four important enabling roles that architecture frameworks play in the system design process: 1. Mechanism to leverage expert knowledge regarding the complete and comprehensive description of the system from multiple stakeholder perspectives. 2. Means to provide technical information ownership and configuration control to give teams access to best and current information.

3. Construct for encapsulating information in a manner that can enable effective use of modelbased systems engineering approaches and toolsets. 4. Approach that reconciles the systems engineer’s drive to provide a complete system description with the pragmatic reality that any one engineer can effectively specify only partial information. Architectural frameworks leverage expert knowledge. Current architectural frameworks have been developed through collaborative work and consensus building on what views and descriptive mechanisms are most effective in capturing system information. Without these frameworks, an individual architect or architectural team would likely develop a much less complete architectural description than would be enabled through use of a standard framework as a guiding reference. Architectural frameworks provide a mechanism for ownership and control authority. The development of complex systems involves a steady stream of information flow about the system. It can be difficult to discern when this information is from a “trusted source” and is authorized by the appropriate technical decision authority. Additionally, sharing information in highly complex endeavors can often mean that the specific information needed by a particular user is embedded in a more complex information exchange. Architectural frameworks provide formal mechanisms where ownership and configuration control for view descriptions can be granted to the most appropriate technical authority. Thus, the frameworks, when appropriately used and controlled, establish a sense of trust in regard to the current architectural information set. Architectural frameworks encapsulate information in a usable form. Model-based engineering toolsets are offering powerful new capability to support executable architectures. However, this is only possible if the system description is developed in a manner where information is encapsulated so that if can be made executable. Standardizing on architectural descriptive information is a necessary prerequisite for realizing the potential of model-based systems engineering. Frameworks play to the systems engineering mindset. It is the nature of the systems engineer to seek to describe the total system. As such, without putting tight bounds on the engineering task, the engineer would develop descriptive information that would keep expanding toward the total system description. The architectural views of a framework set well-defined boundaries for a given aspect of the system, yet at the same reassure the engineering team that the full architectural description will serve to describe the system in full. The use of frameworks provides a comfortable environment for the systems team to “divide and conquer” the complex system. Any individual can provide partial information “in good conscience,” realizing that the specification of all system views will in the end result in a complete system picture. Architecture frameworks have the potential for significant positive impact on the “goodness” of the system design. The actual impact relates to whether a framework is used to provide simple descriptions in a superficial manner or if it is embraced by the team and fully used as a descriptive and communicative mechanism.

Evaluating Architecture Framework Effectiveness With many system architecture frameworks, such as DoDAF, emphasis is placed on final architecture products rather than process. Work products are frequently too complex to present to senior leadership without modification. Most fundamentally, weaknesses in the DoDAF have been identified as it undergoes transition from a static, descriptive tool to a framework attempting to characterize dynamic system properties. Little guidance is provided on how to translate requirements into the design of the work products. As promulgated, the DoDAF does not have a companion architecture development process to take advantage of its interconnected views. As a result, many developers of DoDAF have treated it as a contract deliverable as opposed to a central communications tool in the design process. While it is not the business of DoD to stipulate how contractors conduct system design, it is in the interest of DoD to require architectures that are internally consistent and support dynamic performance analysis. Architecture development software—with DoDAF extensions and integrated modeling and simulation capabilities—is available to fill this void. In practice, however, 70% of DoDAF developers are not building executable architectures (Office of the Assistant Secretary of Defense for Networks and Information Integration 2005). Development of effective architecture frameworks involves many challenges: achieving internal and external consistency, proper selection of the work products of concern to the client, and performing verification and validation activities. While the descriptive nature of system architecture frameworks does not require architects to adhere to formal modeling methods or standardized development processes, there are fundamental objectives of architecture frameworks which are universally applicable. Previous work on architecture framework quality was completed by Blevins (2005) in which he outlined “7 Habits of Highly Effective Architecture Efforts” for information systems: (1) link to end user capabilities in the operational environment, (2) link doctrine, organization, training, materiel, leadership, personnel, and facilities, (3) support gap and overlap analysis, (4) decouple the architect from the architecture (and ensure institutional learning), (5) flow architecture information throughout the lifecycle, (6) clearly state work tasks, (7) publish architecture successes and lessons learned. The authors have applied the DoDAF to characterize the Hubble Space Telescope and develop an executable model of potential Hubble servicing missions (Richards et al. 2006). The remainder of this section synthesizes lessons learned from these modeling activities by further enumerating measures of architecture framework quality. Eight measures-of-effectives are proposed to help guide the development of value-added4 work products. 1. 2. 3. 4. 5. 6. 7. 8. 4

Purposefulness Applicability Internal consistency External consistency Clarity Scalability “Execute-ability” Analytic extensibility

Value is determined from the perspective of the sponsor or client of the architecture framework development.

Purposefulness. The existence of a clear purpose for building the high-level Hubble architecture framework (i.e., serviceability assessment) was a critical element in the construction of views that were both compliant with the static DoDAF taxonomy and useful for understanding dynamic system properties. For the value of the DoDAF to be fully realized, its construction must be problem-oriented, focused on providing information that supports decision-making processes. Applicability. The degree to which the views address the attributes of concern to the decision maker responsible for system development drive architecture framework effectiveness. Use of the DoDAF by System Program Offices (SPOs) in acquisition is illuminating. While the primary motivation for creating the DoDAF was to standardize how C4ISR are represented, a secondary motivation was to satisfy Congressional requirements in the acquisition of information technology. However, many components critical for SPO activities—performance models, cost models, program management activities—are not captured in the standard work products (Maier and Rechtin 2000). Given that application of the DoDAF has grown beyond evaluation of interoperability to include architecture description in early system design (Maier, Emery, and Hilliard 2004), a metric of architecture framework effectiveness is the inclusion of a complete set of decision maker concerns. Internal Consistency. The development of architecture frameworks may be a major undertaking involving the talents of multiple experts. For example, the U.S. Army Future Combat System’s architecture requires interoperability of 1,540 systems, 10,000 DoDAF work products, and 800,000 information exchanges (Jain and Dickerson 2005). Given distributed development efforts, a governance structure to ensure configuration management is absolutely critical. This governance structure must permit the flexibility of the architecture to be used and accessed by other developers and the client while maintaining data integrity and preventing information use outside of the intended context (Troche, Eiden, and Potts 2004). External Consistency. One of the principal goals of system architecture frameworks is to standardize descriptions to aid in comparative evaluation and interoperability analysis. To enable comparative evaluation, developers must ensure that work products are consistent with the set of architecture frameworks in use by the client. To enable interoperability analysis, views must be consistent and adequately represent information, power, and other transactional interfaces. Iaquinto (2003) has discussed the use of standards (e.g., MIL-STD-490, IEEE 1233) for proving interoperability. Clarity. Are the work products understandable to the client? One of the lessons learned from modeling servicing missions for Hubble is the challenge in presenting the internal structure of select DoDAF work products at an appropriate level of abstraction. A similar conclusion was reached by an internal MITRE study which found that architecture tool outputs can be difficult to present and may lack necessary contextual information to be understandable by the uninitiated (Troche, Eiden, and Potts 2004). Scalability. May the views be presented at varying levels of detail to the client? Effective architecture frameworks are scalable such that the developer may easily tailor presentations at

different levels of abstraction. It is important that the client both understands the results (simple top-level view) and gains an appreciation for the complexity of the underlying models. “Execute-ability.” User experience with the DoDAF has shown that communicating with clients using the static work products (e.g., pictures, diagrams, textual descriptions) is not adequate despite the fact that these representations may contain all the necessary data (Levis and Wagenhals 2000). A critical role of the architecture framework developer is to capture, through simulation, dynamic system properties such as logic, behavior, and performance. Analytic Extensibility. Although not required by architecture frameworks such as the DoDAF, the ability of executable models to accept data from the static work products as an input is a key indicator of whether the architecture efficiently communicates dynamic system properties. While “execute-ability” may be achieved by independently packaging models and simulations in separate work products, analytic extensibility is achieved by automating the sharing of data from static representations to executable models, lowering the cost to the client for design iterations. Levis and Wagenhals (2000) have proposed the presence of an executable (dynamic) model that traces all information to the static representations of the system as a litmus test for determining the sufficiency of the work products. Given that the components of system architecture frameworks are loosely related and in the absence of methodological guidance, one of the principal challenges facing developers is constructing internally-consistent, value-added work products. It is hoped that these measuresof-effectiveness stimulate a discussion on “best practices” for architecture framework development.

Summary Artifacts play three key roles in system design: communication, knowledge transfer, and managing complexity. In order to accomplish these three roles in a coherent manner, architecture frameworks are created to specify a particular set of artifacts and associated artifact creation processes that balance these tasks. Architecture frameworks (1) leverage expert knowledge regarding the system as a whole, (2) provide information ownership and appropriately distribute process control among designers, (3) enable model-based design at the system level, and (4) reconcile the engineer’s desire for a comprehensive representation with the pragmatic reality that one engineer cannot fully understand all system complexities. Evaluating success of architecture framework development efforts requires consideration of eight criteria: purposefulness, applicability, internal consistency, external consistency, clarity, scalability, executability, and analytic extensibility.

References Alavi, M., and Leidner, D., “Review: Knowledge Management and Knowledge Management Systems: Conceptual Foundations and Research Issues,” MIS Quarterly, Vol. 25, No. 1, 2001. Blevins, T., “Architecture Support for Large Projects: 7 Habits of Highly Effective Architecture Efforts,” MITRE Presentation, 2005. Carlile, P., “A Pragmatic View of Knowledge and Boundaries: Boundary Objects in New Product Development,” Organization Science, Vol. 13, No. 4, 2002.

Henderson, K., “Flexible Sketches and Inflexible Data Bases: Visual Communication, Conscription Devices, and Boundary Objects in Design Engineering,” Science, Technology, and Human Values, Vol. 16, No. 4, 1991. Iaquinto, J., “The C4ISR Architecture Framework V 2.0: Considerations for Usage,” INCOSE Proceedings, June 2003. Jain, P., and Dickerson, C., “Family-of-Systems Architecture Analysis Technologies,” INCOSE Proceedings, July 2005. Karas, L., and Rhodes, D., “Systems Engineering Technique,” AGARD, The Design, Development and Testing of Complex Avionics Systems, 1987. Latour, B., “Visualization and Cognition: Thinking with Eyes and Hands,” Knowledge and Society: Studies in the Sociology of Culture Past and Present, Vol. 6, 1986 Levis, A. and Wagenhals, L., "C4ISR Architectures: I. Developing a Process for C4ISR Architecture Design," Systems Engineering, Vol. 3, No. 4, 2000. Maier, M., Emery, D., and Hilliard, R., "ANSI/IEEE 1471 and Systems Engineering," Systems Engineering, Vol. 7, No. 3, 2004. Maier, M. and Rechtin, E., The Art of Systems Architecting. Boca Raton: CRC Press, 2000. Nonaka, I., Toyama, R., and Konno, N., “SECI, Ba and Leadership: a Unified Model of Dynamic Knowledge Creation,” Long Range Planning, Vol. 33, 2000. Office of the Assistant Secretary of Defense for Networks and Information Integration, Architecture Development and Analysis Survey: The State of DoD Architecting, December 2005. Richards, M., Shah, N., Hastings, D., and Rhodes, D., “Managing Complexity with the Department of Defense Architecture Framework: Development of a Dynamic System Architecture Model,” Conference on Systems Engineering Research, April 2006. [http://esd.mit.edu/WPS/2007/esd-wp-2007-09.pdf] Troche, C., Eiden, G., and Potts, F., Architecture Development Lessons-Learned: A Three-Year Retrospective, MITRE Corporation Technical Report, January 2004.

Biographies Matthew Richards is a graduate student at MIT pursuing a Ph.D. in Engineering Systems. As a research assistant for MIT’s Systems Engineering Advancement Research Initiative (SEARI), Matt’s research focuses on system architecture and design for survivability, space systems engineering, and innovation management. His work experience includes Mars rover mission design at the Jet Propulsion Laboratory and systems engineering support on two autonomous vehicle programs for the Defense Advanced Research Projects Agency. Matt has B.S. and M.S. degrees in Aeronautics and Astronautics (MIT 2004, 2006) and a M.S. degree in Technology and Policy (MIT 2006). Nirav Shah is a graduate student at MIT pursuing a Ph.D. in Aeronautics and Astronautics. Supported by SEARI and affiliated with MIT's Lean Aerospace Initiative, his research focuses on architecture of systems of systems. In particular, he is studying the impact of coupling in the physical, functional, and organizational spaces on system evolution. His work experiences include positions at Los Alamos National Laboratory and with Booz Allen Hamilton. Nirav has B.S. and M.S. degrees in Aeronautics and Astronautics (MIT 2001, 2004).

Daniel Hastings is a Professor of Aeronautics and Astronautics and Engineering Systems at MIT. Dr. Hastings has taught courses and seminars in plasma physics, rocket propulsion, advanced space power and propulsion systems, aerospace policy, technology and policy, and space systems engineering. He served as chief scientist to the U.S. Air Force from 1997 to 1999 and as director of MIT’s Engineering Systems Division from 2004 to 2005. He is a member of the National Science Board, the International Academy of Astronautics, the Applied Physics Lab Science and Technology Advisory Panel, and the Air Force Scientific Advisory Board. Donna Rhodes is the director of SEARI and a Senior Lecturer in Engineering Systems at MIT. Her research interests are focused on systems engineering, systems management, and enterprise architecting. Dr. Rhodes has 20 years of experience in the aerospace, defense systems, systems integration, and commercial product industries. Prior to joining MIT, she held senior level management positions at IBM Federal Systems, Lockheed Martin, and Lucent Technologies in the areas of systems engineering and enterprise transformation. Dr. Rhodes is a Past-President and Fellow of the International Council on Systems Engineering (INCOSE).