System Development Planning via System Maturity ... - CiteSeerX

IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT, VOL. 56, NO. 3, AUGUST 2009

533

System Development Planning via System Maturity Optimization Jose Emmanuel Ramirez-Marquez and Brian J. Sauser, Member, IEEE

Abstract—Many U.S. government agencies and their contractors have subscribed to using the prescriptive metric of technology readiness level (TRL) as a measure of maturity of an individual technology, with a view toward operational use in a system context. A comprehensive set of concerns becomes relevant when this metric is abstracted from an individual technology to a system context, which may involve interplay among multiple technologies that are integrated through a system development life cycle. This paper proposes a system-focused approach for managing system development and making effective and efficient decisions during this life cycle. A system readiness level (SRL) index that incorporates both the current TRL scale and the concept of an integration readiness level is presented with methods for determining current and future readiness of a system. Using techniques in evolutionary algorithms, the SRL index is optimized based on resource allocation to provide a decision support approach that enhances managerial capabilities in the system’s development life cycle. The optimization model for the SRL is then executed with a case example and resource constraints of 75%, 60%, 45%, 30%, and 15% to demonstrate how it can be used to make strategic planning decisions in the system’s development life cycle. Index Terms—Integration readiness level, project monitoring and control, project planning, system readiness level, technology management, technology readiness level.

I. INTRODUCTION

I

N A system development life cycle, there are factors that may strategically alter the decision to develop one system over another; adopt a new, more functional system over another; determine whether a system or technology has become inadequate due to changes in other systems or technologies; or invest in the development of a new system or maintain existing systems. For addressing some of these issues in engineering design and development, it is a prescribed practice for project and engineering managers to use qualitative decision methods [1], but there is a continuous challenge with finding methods or approaches for justifying the optimal allocation of any available resource to minimize development uncertainties [2]. In addition, the allocation of resources becomes that much more challenging when a project or engineering manager is faced with resource constraints that have been influenced by political, economic, or technology market changes. In project management, the allocation of resources is frequently done with the purpose of creating individual tasks to

Manuscript received November 13, 2007; revised April 21, 2008, July 20, 2008, August 24, 2008, and September 19, 2008. First published March 27, 2009; current version published July 17, 2009. Review of this manuscript was arranged by Department Editor A. E. Marucheck. The authors are with the School of Systems and Enterprises, Stevens Institute of Technology, Hoboken, NJ 07030 USA (e-mail: [email protected]; [email protected]). Digital Object Identifier 10.1109/TEM.2009.2013830

maintain schedule and budget that can lead to an assignmenttype project scheduling problem [3] when the ultimate object of any project is to develop a product (or system) to satisfy a customer. Consequently, disconnects often emerge between the priorities of the project manager and the engineering manager with respect to the optimization of the design during the development life cycle of a system. The greater challenge is that the allocation of resources in medium to large-scale systems is a complex problem [4]. Salewiski et al. [3] expressed this concern for complex interaction to minimize cost and solve a time-resource-cost tradeoff problem, but the focus was still on tasks/activities and not necessarily optimization of systems’ developmental maturity. While there has been significant research in project management to find analytical techniques for simplifying this challenge of resource allocation [5], they have often overlooked the interconnection between resources and development as it impacts the system development life cycle and a complete and applicable model for resource allocation is still elusive. However, some have looked at this problem from a design solution perspective. For example, Belhe and Kusiak [6] used heuristic algorithms to propose a solution to scheduling of design activities within the systems’ developmental life cycle, Sachon and Pat´e-Cornell [7] determined the optimal budget allocation strategy for the development of new technologies in safety-critical systems through multiple decision periods, and Dillon et al. [8], [9] developed a decision-support framework for guiding the design while quantitatively demonstrating the implications that constrained resources can have on the engineering of systems. Unfortunately, much of this work has been focused on budget allocations made once at the onset of a project, and new models are needed that allow for informed decisions to be made concerning the allocation of resource at key milestones throughout the life cycle [2]. A fundamental conflict to this problem, as stated by Pich et al. [10], is that in project management tasks are interdependent and coordinated in parallel; therefore, engineers cannot afford to wait for complete information and will often continue through the project life cycle while coordinating design activities with preliminary, ambiguous, or subjective information. This creates a tension between overview and detail between the project and engineering managers [11]. Unfortunately, subjective assessment techniques are human-intensive and error- prone and assessments should be based on system attributes that can be quantitatively measured using system metrics [12]. Therefore, an approach that can combine the analytic rigor of resource allocation with a subjective assessment of system development could provide a potential solution to this challenge.

0018-9391/$25.00 © 2009 IEEE Authorized licensed use limited to: Stevens Institute of Technology. Downloaded on July 30, 2009 at 11:11 from IEEE Xplore. Restrictions apply.

534

IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT, VOL. 56, NO. 3, AUGUST 2009

For this reason, the objective of this paper is to combine a subjective assessment technique for determining system maturity with a quantitative method of resource allocation focused on progressing system development. Using techniques in evolutionary algorithms, this paper proposes to optimize a system maturity index based on resource allocation to the maturation of technologies and their integrations to provide a better decision support approach in the system development life cycle. This approach will be implemented in a case example with resource constraint scenarios of 75%, 60%, 45%, 30%, and 15% of the original budget to demonstrate its capabilities for strategic planning decision making in systems development. The paper concludes with a discussion of the implications of this approach to the engineering management of systems.

II. SYSTEM MATURITY ASSESSMENT The tension between detail and subjectivity is rationalized through prescriptive techniques, which allow people to make better decisions by using normative models, but with knowledge of the limitations and descriptive realities of human judgment [13]. In project and engineering management one of the prescriptive tools used is soft metrics, which are measured through subjective judgment and are relatively easy to derive, but require a complementary rationale that explains the assessment [14]. Within agencies in the U.S. Government, the prescriptive tool and soft metric of technology readiness level (TRL) has been used as an assessment of the maturity of evolving technologies prior to incorporating them into a system or subsystem. TRL’s origin is a by-product of the National Aeronautics and Space Administration’s (NASA) post-Apollo era and a revision in the National Space Policy stating that NASA’s technology program shares the mantle of responsibility for shaping the agency’s future; therefore, NASA had to make strategic decisions on which technologies to invest in for future use. In response to this growing pressure, Sadin et al. [15] developed what originally was a seven-level metric for describing the maturity of a technology. This seven-level metric later became the ninelevel TRL metric that it is today. In the last five to seven years, other government agencies and their contractors have adopted the TRL scale with specific variations to satisfy their needs (e.g., Department of Defense (DoD), Department of Energy (DoE), National Air and Space Intelligence Center). There have been many attempts to identify alternative readiness/maturity levels that will complement TRL (e.g., design readiness level, manufacturing readiness level, software readiness level, operational readiness level, human readiness levels, habitation readiness level, capability readiness levels [16]–[18]). Although each has faltered in addressing the core issue with TRL as identified in the literature (i.e., technology integration), many of the legacy constraints with TRL’s abstraction have remained. These constraints include: 1) distorting many aspects of technology readiness into one metric, the most problematic being integration [19]; 2) not assessing uncertainty involved in maturing and integrating a technology into a system [18]–[22];

3) not considering obsolescence and the ability of a less mature technology to meet system requirements [19], [20], [22]; 4) an inability to meet the need for a common platform for system development and technology insertion evaluation [14], [23]. To compound this, the U.S. Government Accountability Office (GAO) has performed two independent studies of the DoD and the DoE that indicate faults in making developmental decisions with immature information. According to the GAO, from 1997 to 2007, the estimated acquisition costs remaining for many major DoD programs increased nearly 120% while the annual funding provided for these programs merely increased 57% [24]. In a review of 54 DoD programs, those that started a phase of systems design review with immature technologies averaged 41% developmental cost growth, a 13-month schedule delay, and a 21% acquisition unit cost growth [25]. In the DoE, the GAO reported that 67% of the projects investigated experienced cost increases ranging from $79 million to $7.9 billion [26]. GAO also stated that there are few metrics used to gauge the impact of investments or the effectiveness of processes used to develop and transition technologies, and additional metrics in technology transition are needed. In the design of systems, component-level concerns involving integration, interoperability, and sustainment (e.g., producibility, supportability, and evolvability) become equally or more important, from a systems perspective, when used in an operational environment [27], although many of the factors that may determine the implementation of a system into its operational environment are not always effectively implemented in the interim of the developmental life cycle [28]. Fundamentally, any system under development is composed of core technology components and their linkages (i.e., architecture). Henderson and Clark [29] showed that the distinction between the relationships of the components and the system architecture requires two types of knowledge: component knowledge and architectural knowledge (i.e., knowledge of how the components are integrated). Henderson and Clark [29] emphasized that systems often fail because attention is given to the technology, and knowledge of the linkages/integrations is overlooked. This has an impact on the projects’ technical evolution, organizational experience, recurrent task, and technical knowledge as they relate to the component linkages in addition to the product architecture, communication channels, and problem-solving strategies. While TRL provides the metric for describing component knowledge, based on Henderson and Clark [29], one would still be interested in a metric that provides a description of architectural knowledge or integration. This dynamic has also been shown to be of strategic relevance and an unresolved issue in the coupling and maturation of multiple technologies [30], [31]. While there have been some efforts to develop metrics that can be used to evaluate integration (e.g., [32]–[35]), there is a need for a metric that can be understood by all the relevant stakeholders, can evaluate integration maturity, and can be used with TRL to effectively determine a system maturity. Gove et al. [36], [37] address these issues with an evaluation of

Authorized licensed use limited to: Stevens Institute of Technology. Downloaded on July 30, 2009 at 11:11 from IEEE Xplore. Restrictions apply.

RAMIREZ-MARQUEZ AND SAUSER: SYSTEM DEVELOPMENT PLANNING VIA SYSTEM MATURITY OPTIMIZATION

535

TABLE I INTEGRATION READINESS LEVELS [36], [37]

documented integration maturity metrics (IMM) that resulted in a refined IMM entitled integration readiness level (IRL). IRL is a systematic analysis of the interfacing of compatible interactions for various technologies and the consistent comparison of the maturity between integration points (i.e., TRLs). IRLs describe the integration maturity of a developing technology with another technology, developing or mature. The addition of IRLs not only provides a check to where the technology is on an integration readiness scale but also a direction for improving integration with other technologies. As TRL has been used to assess the risk associated with developing technologies, IRL is designed to assess the risk of integration. Table I presents the nine levels of IRL described

by Gove et al. and gives a definition and description of each level. Although component TRLs and IRLs are necessary for evaluating system risk and potential, they are generally insufficient when used independently and not within the context of overall system development. To address this insufficiency, Sauser et al. [22], [38] previously described a systems readiness level (SRL) metric, an approach that incorporated the TRL and IRL to determine the maturity of a system and its status within a developmental life cycle. That is, if every technology is assessed using TRL and the system architecture is used to build an integrated representation of the system (e.g., physical architecture, context diagram) in which integrations are assessed using IRL,

Authorized licensed use limited to: Stevens Institute of Technology. Downloaded on July 30, 2009 at 11:11 from IEEE Xplore. Restrictions apply.

536

IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT, VOL. 56, NO. 3, AUGUST 2009

a metric that provides an assessment of a system’s maturity can be considered. The rationale behind the SRL developed by Sauser et al. [38] is that in the development life cycle, one would be interested in addressing the following considerations. 1) Quantifying how a specific technology is being integrated with every other technology to develop the system. Note that this quantifier should be a function of both the integration of a technology with every other technology with which it has to be integrated (as dictated by the system architecture) and the maturity of the different technologies. That is, for each technology, this metric should be a function of both TRLs and IRLs. Thus, for technology i, one can view this metric (SRLi ), as a “subsystem” measurement of this technology as it integrates within the system. In a mathematical representation: SRLi = f (TRLj , IRLij ). 2) Based on such a metric (SRLi ), SRL should provide a system-level measurement of readiness. Note that this new metric should be a function of the different SRLs of each technology or in mathematical representation: SRL = f (SRL1 , SRL2 , . . . , SRLn ) under the assumption that the system contains n technologies. For this reason, the computation of SRL has been considered as a normalized matrix of pairwise comparisons of TRL and IRL. The SRL matrix consists of one element for each of the constituent technologies and as discussed, from an integration perspective, quantifies the readiness level of a specific technology with respect to every other technology in the system while also accounting for the development state of each technology through TRL. It should be mentioned that although the original (1,9) scale for both TRL and IRL could be used, the use of normalized values allows for a more accurate assessment when comparing the use of competing technologies. Therefore, the values used in the matrices [TRL] and [IRL] are normalized (0,1) from the original (1,9) levels. In addition, when no integration is present between two technologies, an IRL value of 0 is assigned, and for integrations unto itself, an IRL value of 9 is used (or 1 when normalized). Mathematically, for a system with n technologies, [SRL] is given by   SRL1  SRL  2   [SRL] =    ...  SRLn



IRL11 TRL1 +IRL12 TRL2 + · · · +IRL1n TRLn  IRL21 TRL1 +IRL22 TRL2 + · · · +IRL2n TRLn  =  ...

IRLn 1 TRL1 +IRLn 2 TRL2 + · · · +IRLn n TRLn



  . 

(1)

where IRLij = IRLj i . The representation of each of the SRL values obtained in (1) addresses the first consideration previously discussed. Note that

Fig. 1.

System with three technologies (A, B, and C) with TRLs and IRLs.

these values would fall within the interval (0, ni ) so, for consistency, for each technology, say i, its corresponding SRLi is divided by ni (ni being the number of integrations of technology i with itself and every other technology as dictated by the system architecture) to obtain its normalized value between (0,1). To address the second consideration, the SRL for the complete system is the average of all such normalized technologies’ SRL values, which is given by SRL =

(SRL1 /n1 + SRL2 /n2 + · · · + SRLn /nn ) . n

(2)

Therefore, the SRL can be understood as an index of maturity between 0 and 1 applied at the system level with the objective of correlating this indexing to appropriate systems engineering management principles that has previously been demonstrated in [38], [39]. To illustrate the SRL calculation, the following example will use a simple system of three technologies and two integrations (see Fig. 1) to show the steps involved in calculating the SRL value. For this system, the following matrices can be created for TRL and IRL as per the definitions presented earlier: 



TRLA



IRLAA

IRLAB

IRLAC

 [IRL] =  IRLBA IRLCA

IRLBB



9

1

0

IRLBC  

1.00

0.11

0.00

 = 1 0

9

 7  =  0.11

1.00

 0.78  .

    9 1.00       [TRL] =  TRLB  =  6  =  0.67  TRLC 6 0.67

7

IRLCB

9





IRLCC

0.00



0.78

1.00



Note that there is no integration between technologies A and C in this system, and hence the integration IRLAC = 0 and the IRL for self-integrations are 9 as per definition. The SRL would

Authorized licensed use limited to: Stevens Institute of Technology. Downloaded on July 30, 2009 at 11:11 from IEEE Xplore. Restrictions apply.

RAMIREZ-MARQUEZ AND SAUSER: SYSTEM DEVELOPMENT PLANNING VIA SYSTEM MATURITY OPTIMIZATION

be 







SRLA 1.07  SRLB   1.30  [SRL] = [IRL] × [TRL] =  =  SRLC 1.19

0.54 + 0.43 + 0.60 1.07/2 + 1.30/3 + 1.19/2 = 3 3 SRL = 0.52.

SRL =

Despite the utility of calculating a SRL, without an articulated correlation to qualitative systems engineering practices, it becomes difficult to determine the added value in understanding its implication on the developmental life cycle. Sauser et al. [38] were able to use documented qualitative data to calculate the SRL for four systems in development and correlate how the SRL of these systems can be described using any of four standard systems engineering life cycles (i.e., Typical High-Technology System, ISO 15288, DoD, and NASA). These cases represented systems development successes and failures, levels of abstraction, and views in retrospect. Finally, the authors of this paper have subsequently performed further verification and validation of this approach to other cases in conjunction with system developers from DoD, Lockheed–Martin, NASA, and Northrop Grumman. Thus, to provide a subjective assessment of system maturity, the SRL approach is used. III. RESOURCE ALLOCATION METHOD While the SRL has been demonstrated as a method for addressing some of the previously described challenges and providing insight into system developmental maturity, optimizing the future value of this index based on the allocation of resources to the technologies and integrations can enhance a project or engineering manager’s understanding of a system’s potential development. For example, evaluating the SRL of a system can give an estimate of its impending readiness as a function of its state of maturity. As previously described, Sauser et al. [22], [38] correlated the SRL index to a system’s engineering life cycle and demonstrated, for example, that a SRL of 0.58 could indicate the system is still in a development phase, where its capabilities are being incremented, its system requirements are being refined, and there is an ongoing effort to reduce integration and manufacturing risk. The system would also be ensuring operational supportability with demonstrated system integration, interoperability, safety, and utility. While this is important knowledge, optimizing a future SRL based on resource allocation to the technologies and integrations that compose this system can give a perspective on the potential state of the system under development and the needs and processes to be accomplished in its design. The optimization of SRL based on resource allocation can also allow for decisions to be made regarding the tradeoffs among: 1) system attributes such as availability, performance, efficiency, and total ownership cost and 2) the components necessary for producing affordable system operational effectiveness [40], [41]. These attributes have objectives and ranges for components such as capability, reliability, maintainability,

537

supportability, and producibility, and it is the interplay among them that drives the different levels for both IRL and TRL of the elements in a system. The optimal selection of which technologies and integrations to enhance, so as to improve the system SRL, becomes an optimal system design development problem. The optimal design of systems is a classical optimization problem in the area of systems engineering and engineering management (e.g., [42]–[50]). In general, the objective of these problems is to optimize a function of merit of the system design (reliability, cost, mean time to failure, supportability, etc.) subject to known constraints on resources (cost, weight, volume, etc.), and/or system performance requirements (reliability, availability, mean time to failure, etc.). To optimize this specific function, it is generally assumed that the system can be functionally decomposed into a known number of subsystems or elements (i.e., a system architecture) and, for each of these elements, a known set of functionally equivalent component types (with different performance specifications) can be used in the design. From a system engineering design perspective, an optimization approach that balances needs (i.e., the enhancement of the SRL) with resources (i.e., cost of technologies, cost of technology development, schedule, etc.), can be an effective and efficient method for system planning, i.e., the development of an SRL index correlated with the system’s development life cycle can be used as an optimization framework for the systems engineer or project manager to design-in enhanced system reliability, maintainability, and supportability to achieve the desired reductions in the necessary logistics footprint and the associated life cycle cost. A practical example of this SRL application will be provided in Section IV. It is important to point out that eventually all IRL/TRL levels will have to achieve their maximum levels. If one were to have infinite resources, this would not be a problem. Unfortunately, too often systems must be developed within a tight schedule and budget while trading off performance objectives. In this respect, an optimization model that helps guide the system and its development can allow: 1) the customer to keep track of how and when resources are invested as the system maturity evolves at each milestone review and 2) the project manager to understand how the resources available can be distributed to guide the maturity of the system throughout these reviews. Moreover, the solution to the optimization model can be used as a management tool to identify potentially infeasible expectations (regarding the SRL) based on the resources available both by the project manager or the customer. In summary, SRL optimization becomes crucial when trying to decide among competing system design alternatives or when trying to decide which individual TRL or IRL to improve. To make the best decision, this paper proposes an optimization model that will assist management to choose SRL improvement opportunities. It is reasonable to assume that improvements will result in costs associated with the purchase of new technology, reworking of existing equipment, training of employees, hiring new employees, and enhancements to information technology infrastructure. Similarly, these improvements consume a

Authorized licensed use limited to: Stevens Institute of Technology. Downloaded on July 30, 2009 at 11:11 from IEEE Xplore. Restrictions apply.

538

IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT, VOL. 56, NO. 3, AUGUST 2009

reasonable amount of time and, under tight schedules, it is imperative to allocate these resources efficiently. A. SRL Optimization Model and Solution Algorithm To address these concerns, the model SRLm ax illustrates an optimization model with the objective to maximize the SRL (a function of TRL and IRL) under constraints associated with resources. The purpose of the optimization model is to ultimately find a solution that provides the highest maturity of the system (SRL) based on the availability of resources (budget and schedule for the example in Section IV). The purpose is not to minimize development cost or schedule. This model recognizes that the technologies compete for resources and that benefits can result in an improved SRL via the optimal allocation of such resources. The general mathematical form of the model SRLm ax is as follows: Model SRLm ax Max SRL (TRL, IRL) s.t. R1 (TRL, IRL) ≤ r1 .. . RH (TRL, IRL) ≤ rH . The matrices IRL and TRL in SRLm ax contain the decision variables. Each of these variables are integer valued and bounded by (IRLi , 9) and (TRLi , 9) (i.e., before normalization), respectively. That is, the TRL/IRL for the ith component cannot be less than its current level or more than perfect technology development/integration. As described in Sauser et al. [38], the IRL matrix is obtained as a symmetric square matrix (of size n × n) of all possible integrations between any two technologies in the system. On the other hand, the vector TRL defines the readiness level of each of the technologies in the system. As previously stated, the objective function of the model SRLm ax of the system is a function of the decision variables, which dictate the values for both TRL and IRL and drive the optimization of SRL. With respect to the constraint set, the left-hand side of the set of inequalities defined by functions R1 to RH represents the consumption of the different resources (i.e., budget, time, etc.) as a function of the improvement of the technologies’ TRL and IRL. These functions allow the project manager to incorporate how resources (budget, schedule, etc.) are consumed as the different IRL/TRL levels for the different system components improve. Moreover, the right-hand side indicates the total amount available for a particular resource. To further clarify the optimization problem, Appendix A illustrates a complete characterization of the model for the example presented in Section IV. It is important to note that for the optimization model, the accurate definition of how resources are consumed is an important task to be undertaken by the project manager. Failure to obtain realistic estimates of resource consumption will lead to suboptimal allocation of resources. Although in practice it may be very difficult to achieve an exact value for these inputs, usually realistic estimates (approximation, average, median, percentiles, etc.)

can suffice, when used with the necessary caution. Parameter estimation for operations research is an area of intense research in its own right [51]–[53] so, at this stage, the actual forecasting of these values is outside the scope of this paper, but the authors plan to devote further research to the development of this area. To completely characterize the decision variables in the model SRLm ax , it is necessary to introduce the following transformation: ' 1, if TRLi = k yik = 0, otherwise and

xkij =

'

1,

if IRLij = k

0,

otherwise

,

for k = 1, . . . , 9

Note that based on these binary variables, each of the possible TRL and IRL in the system can be obtained as (9 ky k TRLi = k =1 i 9 and (9 k k =1 kxij IRLij = . 9 Based on these binary variables, the normalized SRLi is transformed to * )( * )( 9 9 k k k =1 kxi1 k =1 ky1 SRLi = 81 * )( * )( 9 9 k k k =1 kxi2 k =1 ky2 + 81 * )( * )( 9 9 k k k =1 kxij k =1 kyj + ··· + 81 * )( * )( 9 9 k k k =1 kxin k =1 kyn + ··· + 81 * )( * (n )(9 9 k k kx ky ij j j =1 k =1 k =1 . (3) = 81 Therefore, based on the computation of the SRL with these decision variables, SRLm ax belongs to the class of integer-valued nonlinear problems. For a system with n technologies containing m (m ≤ (n – 1)n/2) distinct integrations and assuming all technologies and integrations are at its lowest level, there are 9n +m possible solutions to the model SRLm ax . To generate an optimal allocation strategy, an evolutionary optimization approach, known as the probabilistic solution discovery algorithm (PSDA) [54], [55], has been used. B. Algorithm Description In general, PSDA is an evolutionary algorithm that, based on the analysis of initial solutions and in a probabilistic manner, iteratively explores regions of an optimization problem solution space with the intent of identifying an optimal solution.

Authorized licensed use limited to: Stevens Institute of Technology. Downloaded on July 30, 2009 at 11:11 from IEEE Xplore. Restrictions apply.

RAMIREZ-MARQUEZ AND SAUSER: SYSTEM DEVELOPMENT PLANNING VIA SYSTEM MATURITY OPTIMIZATION

PSDA has been proven to provide high-quality solutions for optimization problems as diverse as network interdiction [56], reliability allocation [57], container inspection [58], and ad hoc wireless networking [59] to name a few. For finding solutions to the nonlinear integer optimization problem presented in the model SRLm ax , PSDA generates solutions to the model SRLm ax through the probabilistic generation of potential TRL and IRL values (via Monte Carlo (MC) simulation), the analysis of these solutions (via the computation of its associated SRL, cost, and schedule), and the selection of new, potentially optimal TRL and IRL values (via an evolutionary technique). A detailed description of the steps involved in PSDA follows while the pseudocode of the algorithm is described in Appendix B. Initialization: The initialization step for PSDA requires the user to input: 1) the number of solutions, V , to be generated in each cycle, u, of the algorithm; 2) the total number of cycles, U , over which the algorithm will run; 3) the initial vectors of probabilities for γ iu and γ ij u , and 4) the subsample size, S, used to guide the evolution of PSDA. A discussion regarding the tuning of these parameters follows the description of the algorithm. Strategy development: In this step, based on the probabilities defined by vectors γ iu and γ ij u , MC simulation is used to generate a specified number (defined by V ) of potential designs. That is, for the current cycle u, the values for matrices TRLvu and IRLvu [where the superscript v (v = 1, . . . , V ) labels the solution number and the subscript u (u = 1, . . . , U ) the current algorithm cycle] are assigned randomly as dictated k by vectors γ iu and γ ij u . Note that for each technology i, γiu (the kth element of vector γ iu ) defines the probability that at cycle u, the TRL of such a technology will + increase , its current k = P yik = 1 ]. Similarly, readiness level to level k [i.e., γiu k γij u defines the probability that at cycle u, the IRL between the ith and jth technologies will + kincrease , its actual readiness level, k = P x = 1 ]. Moreover, the elements to level k [i.e., γij u ij in these two vectors describe the both evolution of the algorithm at each cycle u and the region of the solution space being analyzed. This step also contains the stopping rules of the algorithm. In essence, the first rule dictates the algorithm to be stopped once both vectors γ iu and γ ij u can no longer be updated (i.e., all initial “appearance” probabilities are either 0 or 1) and thus the algorithm has converged. The second rule allows the user to set a specific number of cycles U over which the algorithm will run. In the context of this algorithm, a cycle is understood as every time the value u is updated. Design analysis: In the third step, for each of the potential system designs previously generated, TRLvu and IRLvu , (2) and (3) are implemented to obtain their corresponding SRLvu value. Similarly, the resources consumed by these designs are computed via Rh (TRLvu , IRLvu ) (where h = 1, . . . , H). Note that these resource functions should relate how program resources (budget, schedule, etc.) are consumed by the effort to improve IRL and TRL levels. Usually, these functions are in the form of a step function that describes the effort (in terms of cost or time) that will be expected to bridge a technology/integration from one level to the next one.

539

Fig. 2. System concept diagram. Tech 1: remote manipulator system (RMS); Tech 2: special purpose dexterous manipulator (SPDM); Tech 3: electronic control unit (ECU); Tech 4: autonomous grappling (AG); Tech 5: autonomous proximity operations (APO); and Tech 6: laser image detection and radar (LIDAR).

Solution discovery: The fourth step of the algorithm starts by penalizing the value of SRLvu for each of the potential designs generated and evaluated in the strategy development and design analysis steps, respectively. In the context of this algorithm, penalizing a solution is understood as the process of reducing the value of SRLvu whenever any of the values yielded by Rh (TRLvu , IRLvu ) violate the resource consumption requirements. To characterize the penalization scheme function, ∆h (TRLvu , IRLvu ) has been used and is described in the appendix. In essence, the mathematical description of each function ∆h confines its values to the range (0,1) and thus, effectively reduces SRLvu whenever Rh (TRLvu , IRLvu ) violates the resource consumption requirements. After obtaining ∆h (TRLvu , IRLvu ) and computing the pev nalized SRL for each solution v in the current cycle u, SRLu , the solutions are ranked in decreasing order of magnitude with respect to this penalized value. Based on this ranking, the probabilities defined by the vectors γ iu and γ ij u are updated via the analysis of the values of the matrices TRLvu and IRLvu assov ciated with the S highest ranking SRLu values. In addition, the best five of these ranked solutions are stored in set K to keep track of the best solutions generated per cycle. Finally, the new vectors γ iu and γ ij u are sent to the strategy development step to check for termination or for further solution discovery. Termination: Finally, once either of the stopping rules has been met, the best solution in set K is chosen as the optimal system design. 1) Description of Parameters of Tuning: Initial vectors of probabilities (γ iu and γ ij u ): These vectors drive the evolution of the algorithm as the cycles progress. However, these parameters have to be input for the first cycle of the algorithm. The rationale behind assigning initial values to these vectors is based upon the lack of knowledge regarding the levels that both TRL and IRL, for every technology and integration, should attain so that the model SRLm ax is optimally solved. This lack of knowledge, also known as “Laplace principle of insufficient reason,” translates into initially providing the same

Authorized licensed use limited to: Stevens Institute of Technology. Downloaded on July 30, 2009 at 11:11 from IEEE Xplore. Restrictions apply.

540

IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT, VOL. 56, NO. 3, AUGUST 2009

TABLE II ESTIMATED CONSUMPTION OF RESOURCES FOR EACH TECHNOLOGY EFFORT

appearance probability in the final solution to each and every TRL and IRL level. Moreover, this same lack of information forces the choice of these levels to be equally likely. Subsample size (S): The subsample size S is used to update the probabilities defined by the vectors γ iu and γ ij u . The rationale for setting this value is based on work developed by B¨ack and Schwefel [60] and Schwefel and B¨ack [61] in the area of evolutionary strategies and its application to optimization algorithms that imitate certain principles of nature. In this respect, a population of individuals (a possible system design for this paper) collectively evolves toward better solutions by means of a parent’s selection process, a reproduction strategy, and a substitution strategy. These studies [60], [61] define a type of strategy, denoted by function Γ(µ, λ) = µ/λ, where µ initial individuals generate λ offspring, and selection takes place only among these λ offspring. A ratio of µ/λ ≈ 7 is suggested as a good setting for the relation between parent and offspring population size. The algorithm described in Section III. A works in a similar fashion since from a sample of size V of potential design strategies, the algorithm selects S among the best solutions, according to the penalized cost. This means that, for example, from 100 potential designs, the best 14 are selected. If V = 200 then S should be approximately equal to 28. However, preliminary tests indicate that excellent results are obtained by using a fixed value of S = 50. Generation size (V ): The number of potential design strategies (V ) has been set considering that the maximum allowable number of solutions searched (U × V ) would correspond to at most 10% of the solutions searched.

tional and attitude control is lost, certain instruments aboard could be permanently damaged either due to low temperatures or direct exposure to sunlight. To repair the HST, NASA will have to perform a service mission (SM-4) to keep the HST operating into the future. The problem is that the HST has not been serviced since before the Columbia disaster. In addition, the Columbia Accident Investigation Board requirements for shuttle operations would impact a HST SM-4 since time and fuel considerations may not allow the crew to safely dock to the International Space Station if damage is found on the Shuttle that could delay its safe return. To address this concern, a robotic servicing mission (RSM) has been suggested thereby reducing cost and the risk to human life [62]–[64]. Before presenting the SRL assessment, it is fair to state that a NASA-appointed independent committee for the RSM has considered more than simply technology readiness in their assessment of the options. In fact, they specifically describe the integrated performance of all the components involved. Furthermore, some of the TRLs of the components may be matured in other space-bound systems, such as the U.S. Air Force’s XSS11, which, for example, will move the TRL of the subsystem Laser Image Detection and Radar (LIDAR) past a TRL 6, but its specific integration maturity will be unchanged [63]. This raises the value of an integration maturity metric such as IRL for identifying any integration maturity risks associated with the coupling of technologies. Fig. 2 represents a subsystem concept diagram of the RSM. We have used qualitative data as described by [62]–[64] to assess the TRLs and IRLs for the RSM.

IV. ILLUSTRATIVE EXAMPLE

By evaluating the SRL of this system, an estimate of its actual readiness can be obtained before it is deployed. When reviewing the SRL for this system in its current state, an SRL of 0.50 has been obtained. The following section will explain further the practical meaning of this system maturity assessment value, but at this point the purpose is to demonstrate the SRL optimization model and its solution algorithm. To demonstrate the optimization of the model SRLm ax , Tables II and III present notional data that represents estimates of the resources (in terms of dollars (in 1000 s) and man-hours)

A. Case Overview After many years of exceptional service, NASA has been considering a technical solution to repairing the Hubble Space Telescope (HST) as it has far surpassed its expected lifetime. Its gyroscopes are approaching the end of their life cycle, its batteries are nearing their lower levels of acceptable performance, and the fine-guidance sensors have already begun failing. If the HST is not serviced soon and the batteries run out or naviga-

B. Calculation of SRL, Optimization Model, and Solution Algorithm

Authorized licensed use limited to: Stevens Institute of Technology. Downloaded on July 30, 2009 at 11:11 from IEEE Xplore. Restrictions apply.

RAMIREZ-MARQUEZ AND SAUSER: SYSTEM DEVELOPMENT PLANNING VIA SYSTEM MATURITY OPTIMIZATION

541

TABLE III ESTIMATED CONSUMPTION OF RESOURCES FOR EACH INTEGRATION EFFORT

TABLE IV SOLUTION CONVERGENCE PER PSD CYCLE, 30% BUDGET

TABLE V BEST SOLUTIONS FOR RESOURCE AVAILABILITY

consumed by the “development” program when attempting to increase the TRL and IRL of the different technologies, respectively. For this example, the estimated values represent average consumption for similar efforts. To clearly illustrate the objective function and resource consumption constraints used for this example, a complete characterization of its associated model is presented in Appendix A using a scenario where 30% of the resources are available.

Based on the resource consumption, the estimated total cost and time necessary to have a perfect SRL equals $26,574,000 and 19,122 man-hours, respectively. As previously discussed, the program manager must obtain these estimates. In practice, the model may be solved for each the following scenarios: worst, expected, and best. This would allow the manager flexibility regarding the solution. However, for more flexibility, a multiobjective optimization model perspective may be necessary. The authors of this paper are currently devoting research efforts to this type of modeling. To illustrate the approach, it has been assumed the program has been awarded 30% of these totals to improve the SRL of the system. We chose not to use a 100% resource allocation for our example because reductions in resources are often a reality as a result of political, economic, or technology market trends. Also, the proper allocation of resources under constraints is a more challenging problem for a project or engineering manager. To implement PSDA, it was assumed that U = 10, V = 1000, and S = 50. That is, of the possible 6.88 × 107 solutions, the algorithm will make a search among at most 0.013% of these solutions. The best three values of the solutions obtained by the algorithm at each cycle are illustrated in Table IV. This table illustrates that the algorithm converged to a solution at cycle 7. Note that during the first three cycles, the algorithm generates solutions of roughly the same quality. However, each of the subsequent cycles improves the best solution of the previous cycle. The best solution as illustrated in Table IV, cycle 7, has an SRL of 0.73 for a total cost of $7,724,000 and 5081 man-hours. To further illustrate the approach, Table V represents the best five solution values obtained considering 75%, 60%, 45%, 30%,

Authorized licensed use limited to: Stevens Institute of Technology. Downloaded on July 30, 2009 at 11:11 from IEEE Xplore. Restrictions apply.

542

IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT, VOL. 56, NO. 3, AUGUST 2009

TABLE VI SRL TO SYSTEMS DEVELOPMENT LIFE CYCLE

and 15% availability of the total resources necessary to claim a perfectly integrated and developed system. C. Implications to Project and Engineering Management As stated earlier, one of the advantages of the SRL index is to be able to associate resource constraints to developmental maturity. To understand developmental maturity, a framework for describing the maturity of a system comparative to its life cycle is needed. As discussed, Sauser et al. [38] performed a mapping of the SRL index to four system’s life cycles. We have further refined their results with the analysis of additional systems from DoD, Lockheed–Martin, NASA, and Northrop Grumman for which random maturity assessments were created to assess the sensitivity of the SRL index to a system’s life cycle. This analysis has created a recalibration of the mapping initially introduced in [22] with respects to the sensitivity of the ranges (see Table VI). This new mapping does not yet represent an ideal correlation between the SRL index and the system’s life cycle (the authors are currently undertaking extensive verification and validation that is beyond the scope of this paper). However, such mapping serves as a tool for the project and engineering managers in understanding the developmental maturity of their system. Based on the resource allocation and potential constraints for our case example, HST RSM, it can be claimed that there is significant variability regarding how far the system under development can mature. Table VI indicates that at the lowest level of allocated resources (15%), the system cannot be matured beyond a SRL of 0.64. This value corresponds to a delta of 0.36 when assuming 100% of the resources can be secured. Moreover, to progress to an operations and production phase, a SRL value in the range (0.90,1.00), a resource allocation of greater than 60% is necessary. These values are significant for a mission such as RSM that is linked to a launch vehicle

(i.e., Space Shuttle) that is being decommissioned and has limited launch opportunities in the next 5 years. Many science and engineering projects that were dependent on the launch schedule of the Space Shuttle over the next 5 years have been canceled because of missed opportunities in canceled or postponed missions. In addition, there has been significant debate on the value of servicing the HST, so RSM will always be a project that will be under continued budget and schedule scrutiny. Likewise, Pat´e-Cornell and Dillon [65] demonstrated in a study of NASA projects that what is essential for project management in general becomes important as resources become limited. Accordingly, the proposed analysis and optimization approach can provide the project and engineering manager an assessment that may allow them to make some strategic decisions on resource allocation or developmental planning to guarantee system delivery. For example, the project and engineering manager can use the solution obtained from the model SRLm ax as justification for acquiring additional resources or redesigning the system for use with another launch vehicle so as not to be dependent on the Space Shuttle. While the SRL index can have value for overall planning, one can assess the developmental maturity of each technology and corresponding integrations based on the resource constraints using the model SRLm ax . Table VII illustrates the associated TRL and IRL levels obtained from the optimal solution for each of the cases considered. This is very important when understanding how the optimization approach can influence the developmental maturity of the individual technologies and integrations. That is, the optimal TRL and IRL levels obtained from the model become a guidance tool for the project and engineering manager to better understand how the allocation of resources are impacting maturity at the lower levels of development. Table VII also indicates that for RSM, a 75% allocation of resources still would not produce a fully mature system as only three out of the seven integration links have reached complete maturity. Furthermore,

Authorized licensed use limited to: Stevens Institute of Technology. Downloaded on July 30, 2009 at 11:11 from IEEE Xplore. Restrictions apply.

RAMIREZ-MARQUEZ AND SAUSER: SYSTEM DEVELOPMENT PLANNING VIA SYSTEM MATURITY OPTIMIZATION

543

TABLE VII DESIGN SOLUTIONS

one of the integrations has not passed an IRL of 6 (i.e., IRL4,5 ). We also stated earlier that a fundamental construct of the SRL was that it uses more of a systems approach to assessing maturity than TRL alone. For instance, if from Table VII we only evaluated the maturation of TRL after consuming 75% of the resources, we would deploy a system with significant developmental risk. If we refer back to Table V and observe the SRLi values, we can see that SRL4 and SRL5 are one developmental phase behind SRL1,2,3, and 6 that cannot be observed in the TRLs alone. We do contend that because of the difficulty of fully validating space missions in their operational environment (i.e., space), it is not uncommon for these systems to be deployed with some level of immaturity [66], [67], but this assessment can indicate to a project and engineering manager the developmental maturity risk of what could be key components in the system even when the overall SRL index appears adequate. V. CONCLUSION The use of prescriptive metrics in project and engineering management has been a proven and successful practice for many organizations. As a benchmark for making decisions, metrics can perform an integral part of management activities for indicating performance or effectiveness of risk, quality, and maturity; identifying critical parameters; establishing milestones to assess progress; providing direction for risk management/mitigation; and sustaining entry and exit criteria for major milestones. With their widespread use, it becomes imperative that one finds more rigor in how these metrics are used to make strategic planning decision in systems development. This paper presented two prescriptive metrics, TRL and IRL, and showed how they can be used to quantitatively formulate a decision metric in SRL for assessing a system’s developmental maturity and life cycle position. We then presented a model (i.e., SRLm ax ) for making informed and strategic decisions on the allocation of resources to optimize the maturity with respect to the proposed prescriptive metrics. To further describe the capabilities of this research, Table VIII indicates how the methods presented in this paper can be utilized by project and engineering managers at progressive phases of the system’s life cycle based on eleven roles and activities of a systems engineering manager as described by Shenhar and Sauser [68]. Associating the application of the proposed methods with activities throughout the system’s life cycle emphasizes

the effectiveness and advantage of these metrics and the model SRLm ax . Table VIII also stresses that the ultimate functionality of any system’s development effort cannot be defined at the beginning of the program. The roles and activities (R& A) and implications in Table VIII underscore major elements of project planning such as assessing the decomposition of deliverables into a hierarchical architecture; determining the developmental strategy of incremental or evolutionary deliveries; identifying opportunities and associated risks for preventive, causative, and contingent plans; refining the project’s overall critical path; establishing resources needed to accomplish development; and committing the necessary resources for realizing system development [69]. R&A 1, 4, 6, and 7 distinctly complement the model SRLm ax . For example, at program launch (R&A 1), a match must be made between customer needs and available resources to assure that the proposed system is conceivable; midway through development (R&A 4), the product’s design must demonstrate its ability to meet performance requirements; and before production (R&A 6,7), the product has to be shown to be developed within cost, schedule, and quality parameters. SRLm ax not only allows the ability to aid the R& As of 1, 4, 6, and 7 for any current state but any projected state as well. In summary, only by the maturation of the technologies and integrations, matched with the evolving needs of the user, can the user be provided with the needed system solution. In most projects, the two most limiting resources are personnel and funds, with less attention given to the physical resources of systems development. Too often, projects are improperly structured and allow requirements to outstrip resources. A process followed by leading firms is effectively matching the customer’s need and the available resources for creating a system (product). In this paper, we presented an approach and model for evaluating the maturity of a system (i.e., customer needs) with available resources to determine the most effective allocation of these resources to optimize the system’s developmental maturity. While it is not impossible to subjectively assess what stage of development a simple system should be in, without the use of any quantitative model, the difficulty increases with the size and complexity of the system. In addition, the metrics and model described can be used as a tool for project and engineering managers to develop maturation plans, identify developmental milestone criteria, and prepare for future development strategies that will illustrate the timely implementation of

Authorized licensed use limited to: Stevens Institute of Technology. Downloaded on July 30, 2009 at 11:11 from IEEE Xplore. Restrictions apply.

544

IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT, VOL. 56, NO. 3, AUGUST 2009

TABLE VIII IMPLICATIONS TO THE ROLES AND ACTIVITIES OF A SYSTEMS ENGINEERING MANAGER

capability increments. For example, SRL and SRLm ax could be integrated with a project management maturity model [70] to assess the developmental investment and capability impacts of project management in an organization. Finally, while we believe the approach and methodology presented in this paper is a step in the right direction toward a more “systems-focused approach” to assessing system developmental maturity and the effective management of resources to realize a future maturity, it is not without its limitations. We conclude with some of these limitations and opportunities for future research. 1) Cognitive bias in the assessment: Self-assessments for the determining of any metric (e.g., TRL, IRL, resource

allocation) without rigorous criteria result in inaccurate analysis. The existing approaches we presented need further development and rigor in determining and assessing the input data to adjust for this bias. 2) Assessment completeness and oversimplification: While this is not the intent, those who apply readiness levels on programs have tended to focus on the number versus some of the underlying issues. Technologies, their integrations, and overall cost and schedule resources are not the only factors (e.g., reliability/availability/maintainability, software, interoperability, quality and producibility, safety, security, and supportability) that can determine a system’s operational effectiveness [40], [41]. A model of systems

Authorized licensed use limited to: Stevens Institute of Technology. Downloaded on July 30, 2009 at 11:11 from IEEE Xplore. Restrictions apply.

RAMIREZ-MARQUEZ AND SAUSER: SYSTEM DEVELOPMENT PLANNING VIA SYSTEM MATURITY OPTIMIZATION

development maturity will ultimately be more dynamic and complex. 3) System development is nonlinear with multiple objectives: The approach and methods presented made a fundamental assumption, that all technologies and their supporting integrations are equally important to the performance of the system with single objectives. Technologies will vary in their criticality to system performance and often have multiple objectives and thus an additional weight or penalty could be allocated to the technologies and their integrations. In addition, for more flexibility, a multiobjective optimization model perspective may be necessary to achieve a more realistic analysis. 4) TRLs and IRLs, in practice, are more probabilistic: Although, at this point, an inherent assumption is that TRLs and IRLs are being calculated deterministically, it is more rational to assume that the evaluation of TRLs and IRLs follow a probabilistic form. Despite this, the deterministic approach we used is based on the current practice of these maturity metrics. However, the authors are currently conducting further needed research for an approach that can estimate a confidence level for any calculated SRL based on a probabilistic TRL and IRL evaluation. 5) Resource determination: To demonstrate the model SRLm ax , resource expenditures for the maturation of the various TRL and IRL levels were not based on real case data. The determination of these values is a complex and challenging problem that lies beyond the scope of this paper, but a problem that is equally if not more important for any project or engineering manager. While we were able to demonstrate where resources should be allocated with respects to maturing the TRLs and IRLs, the value of the model SRLm ax (and any model or analysis) relies upon the quality and accuracy of the data. The success of implementing the model SRLm ax depends on consistent and continuous definition of needed capabilities (i.e., requirements), and the maturation of technologies that lead to disciplined development and production of systems that provide increasing capability toward a realized concept.

subject to cost: 0y18 + 900y19 + 0y28 + 765y29 + 0y37 + 689y38 + 1423y39 + 0y46 + 876y47 + 1297y48 + 2150y49 + 0y56 + 467y57 + 998y58 + 1187y59 + 0y66 + 780y67 + 803y68 + 1192y69 + 0x512 + 100x612 + 275x712 + 675x812 + 1275x912 + 0x613 + 200x713 + 600x813 + 1250x913 + 0x623 + 50x723 + 500x823 + 1050x923 + 0x524 + 275x624 + 815x724 + 1447x824 + 2192x924 + 0x635 + 345x735 + 802x835 + 1480x935 + 0x245 + 453x345 + 1034x445 + 1755x545 + 2655x645 + 3855x745 + 5287x845 + 7052x945 + 0x256 + 123x356 + 342x456 + 947x556 + 1647x656 + 2455x756 + 3458x856 + 4568x956 ≤ 7972 schedule: 0y18 + 349y19 + 0y28 + 432y29 + 0y37 + 476y38 + 775y39 + 0y46 + 127y47 + 468y48 + 1036y49 + 0y56 + 280y57 + 516y58 + 564y59 + 0y66 + 450y67 + 471y68 + 371y69 + 0x512 + 140x612 + 320x712 + 620x812 + 1120x912 + 0x613 + 93x713 + 258x813 + 647x913 + 0x623 + 25x723 + 345x823 + 810x923 + 0x524 + 164x624 + 484x724 + 916x824 + 1606x924 + 0x635 + 324x735 + 724x835 + 1224x935 + 0x245 + 200x345 + 600x445 + 1258x545 + 1958x645 + 2912x745 + 3933x845 + 5171x945 + 0x256 + 80x356 + 460x456 + 992x556 + 1613x656 + 2475x756 + 3473x856 + 4628x956 ≤ 5737 binary decision variables: yik =

ACKNOWLEDGMENT The authors wish to thank the Editor-in-Chief G. Farris, the Department Editor A. Marucheck, and the reviewers for their insightful comments. The support of the Naval Postgraduate School (contract # N00244–08-000), Northrop Grumman Integrated Systems, and the U.S. Army Armament Research, Development and Engineering Center. Finally, they would like to acknowledge the support of their research assistants, R. Magnaye and W. Tan, for being their checks and balance. APPENDIX A Max =

-

SRL

SRL1 /3+SRL2 /4+SRL3 /4+SRL4 /3+SRL5 /4+SRL6 /2 6

.

545

xkij =

' '

1,

if TRLi = k

0,

otherwise

1,

if IRLij = k

0,

otherwise

; for k = 1, . . . , 9

where the matrix (*) is as shown at the bottom of next page. APPENDIX B PROBABILISTIC SOLUTION DISCOVERY ALGORITHM FOR SYSTEM MATURITY OPTIMIZATION Initialize: V , S, U , γ iu , γij u , u = 1, K = Ø Step 1: (Design Development) For v = 1,. . .,V , generate a potential system design TRLvu and IRLvu via MC simulation as dictated by γ iu and γij u . /That is, for each element in γ iu , generate a random bi ∈ (0, 1)

Authorized licensed use limited to: Stevens Institute of Technology. Downloaded on July 30, 2009 at 11:11 from IEEE Xplore. Restrictions apply.

546

IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT, VOL. 56, NO. 3, AUGUST 2009

such that: if

γ 1iu

else

For i, j update vectors γ iu and γ ij u as follows:

≥ bi →

k −1 / t= 1

yi1

=1

t γiu < bi ≤

k / t= 1

k γiu =

k γiu → yik = 1,

(S

s=1

)

/(same procedure for γij u )/v → v + 1;} k k k k ∧ γij = 1 ∨ γiu ∧ γij if (γiu u u = 0 ∀ i, j) ∨ u = U , then ∗ SRL = arg max {SRL (TRL, IRL)}. Stop.

Go to Step 1. REFERENCES

IRL,TRL∈K

Else Go to step 2. Step 2: (Design Analysis) For v = 1, . . . , V obtain:

SRL (TRLvu , IRLvu ), R1 (TRLvu , IRLvu ) , . . . , RH (TRLvu , IRLvu ) Go to Step 3. Step 3: (Solution Discovery) For v = 1,. . .,V { ∆h (TRLvu , IRLvu ) 0 11 0 Rh (TRLvu , IRLvu ) − rh  1 − max {R (TRLv , IRLv ) − r } h h  u u v =  Rh (TRLvu , IRLvu ) > rh 1 o.w. SRL (TRLvu , IRLvu ) H 2

h= 1

∆h (TRLvu , IRLvu ) ; }

List SRL (TRLvu , IRLvu ) by decreasing order of magnitude: * * ) ) (1) (2) ≥ SRL TRL(2) SRL TRL(1) u , IRLu u , IRLu * ) ) (V ) ≥ · · · ≥ SRL TRL(V , IRL u u * ) (1) ; K → K ∪ SRL TRL(1) u , IRLu u → u + 1;

 (9 9 k = 8 ky1k   (  9 k    9 k = 8 ky2 SRL1    SRL   (9 2 k      9 k = 7 ky3  SRL3       = (  SRL4   9 9k = 6 ky4k     SRL5      (  9 9 ky k  5 SRL6 k=6    (9 k 9 k = 6 ky6

+ + + + + +

(9

k=8

(9

ky2k

k k = 8 ky1

(9

k=8

ky1k

(9

k k = 8 ky2

(9

k k = 7 ky3

(9

k k = 6 ky5

81

*

, S - V. S /(same procedure for γij u )/

i = 1, . . . , n; k = 2, . . . , 9

= SRL (TRLvu , IRLvu )

yik |yik ∈ TRL(s)

(9

k =5

(9

81

kxk12 +

k k =5 kx12

(9

k =6

(9

+

81

k =6

(9

+

kxk13 +

k k =5 kx24

(9

[1] D. M. Buede, “Engineering design using decision analysis,” in Proc. 1994 IEEE Int. Conf. Syst., Man, Cybern., 1994 ’Humans, Inf. Technol., pp. 1868–1873. [2] R. L. Dillon, M. E. Pat´e-Cornell, and S. D. Guikema, “Optimal use of budget reserves to minimize technical and management failure risks during complex project development,” IEEE Trans. Eng. Manage., vol. 52, no. 3, pp. 382–395, Aug. 2005. [3] F. Salewski, A. Schirmer, and A. Drexl, “Project scheduling under resource and mode identity constraints: Model, complexity, methods, and application,” Eur. J. Oper. Res., vol. 102, pp. 88–110, 1997. [4] C. K. Chang, M. J. Christensen, and T. Zhang, “Genetic algorithms for project management,” Ann. Softw. Eng., vol. 11, pp. 107–139, 2001. [5] C.-C. Wei, P.-H. Liu, and Y.-C. Tsai, “Resource-constrained project management using enhanced theory of constraint,” Int. J. Proj. Manage., vol. 20, pp. 561–567, 2002. [6] U. D. Belhe and A. Kusiak, “Scheduling design activities with a pull system approach,” IEEE Trans. Robot. Autom., vol. 12, no. 1, pp. 15–21, Feb. 1996. [7] M. Sachon and M. E. Pat´e-Cornell, “Managing technology development for safety-critical systems,” IEEE Trans. Eng. Manage., vol. 51, no. 4, pp. 451–461, Nov. 2004. [8] R. L. Dillon and M. E. Pat´e-Cornell, “APRAM: An advanced programmatic risk analysis method,” Int. J. Technol., Policy, Manage., vol. 1, pp. 47–65, 2001. [9] R. L. Dillon, M. E. Pat´e-Cornell, and S. D. Guikema, “Programmatic risk analysis for critical engineering systems under tight resource constraints,” Oper. Res., vol. 51, pp. 354–370, 2003. [10] M. T. Pich, C. H. Loch, and A. D. Meyer, “On uncertainty, ambiguity, and complexity in project management,” Manage. Sci., vol. 48, pp. 1008– 1023, 2002. [11] H. A. U. de Haes, “Life-cycle assessment and the use of broad indicators,” J. Ind. Ecol., vol. 10, pp. 5–7, 2006. [12] S. M. Yacoub and H. H. Ammar, “A methodology for architecture-level reliability risk analysis,” IEEE Trans. Softw. Eng., vol. 28, no. 6, pp. 529– 547, Jun. 2002. [13] J. E. Smith and D. von Winterfeldt, “Decision analysis in management science,” Manage. Sci., vol. 50, pp. 561–574, 2004. [14] T. Dowling and T. Pardoe, TIMPA—Technology Insertion Metrics, vol. 1. London, U.K.: Ministry of Defence, 2005.

kxk35 +

k k =2 kx56

(9

k =7

(9

ky3k

k k =7 ky3

81 (9

k =8

ky2k

k =6

ky5k

81 (9 (9

k =6

81

ky4k

(9

k =6

(9

kxk13

k k =6 kx23

(9

k =6

(9

k =2

(9

k =2

+

kxk23 + kxk45 kxk45 +

(9

k k =6 ky4

(9

k =6

(9

k =6

ky5k

ky6k

Authorized licensed use limited to: Stevens Institute of Technology. Downloaded on July 30, 2009 at 11:11 from IEEE Xplore. Restrictions apply.

(9

k k =5 kx24

(9

k =6

(9

k =2

kxk35

kxk56



           .          

(∗)

RAMIREZ-MARQUEZ AND SAUSER: SYSTEM DEVELOPMENT PLANNING VIA SYSTEM MATURITY OPTIMIZATION

[15] S. R. Sadin, F. P. Povinelli, and R. Rosen, “The NASA technology push towards future space mission systems,” Acta Astronaut., vol. 20, pp. 73– 77, 1989. [16] J. W. Bilbro, “A suite of tools for technology assessment,” in Technology Maturity Conference: Multi-Dimensional Assessment of Technology Maturity. Virginia Beach, VA: AFRL, 2007. [17] J. Connelly, K. Daues, R. K. Howard, and L. Toups, “Definition and development of habitation readiness level (HRL) for planetary surface habitats,” in Proc. 10th Biennial Int. Conf. Eng., Construction, Operations Challenging Environ., 2006, p. 81. [18] D. Cundiff, “Manufacturing readiness levels (MRL),” Unpublished White Paper, 2003. [19] J. Smith, “An alternative to technology readiness levels for nondevelopmental item (NDI) software,” Carnegie Mellon, Pittsburgh, PA, Tech. Rep., CMU/SEI-2004-TR-013, Apr. 2004. [20] R. Valerdi and R. J. Kohl, “An approach to technology risk management,” presented at the Eng. Syst. Division Symp., Cambridge, MA, 2004. [21] R. Shishko, D. H. Ebbeler, and G. Fox, “NASA technology assessment using real options valuation,” Syst. Eng., vol. 7, pp. 1–12, 2003. [22] B. Sauser, D. Verma, J. Ramirez-Marquez, and R. Gove, “From TRL to SRL: The concept of systems readiness levels,” presented at the Conf. Syst. Eng. Res., Los Angeles, CA, 2006. [23] DoD, Technology Readiness Assessment (TRA) Deskbook, DUSD(S&T), Ed. Washington, DC: Dept. of Defense, 2005. [24] GAO, Better Weapon Program Outcomes Require Discipline, Accountability, and Fundamental Changes in the Acquisition Environment, vol. GAO-08-782 T, GAO, Ed. Washington, DC: U.S. Government Accountability Office, 2008. [25] GAO, Defense Acquisitions: Assessments of Selected Major Weapon Programs, vol. GAO-05-301, GAO, Ed. Washington, DC: U.S. Government Accountability Office, 2005. [26] GAO Department of Energy: Major Construction Projects Need a Consistent Approach for Assessing Technology Readiness to Help Avoid Cost Increases and Delays. vol. GAO-07–336, GAO, Ed., Washington, DC: U.S. Government Accountability Office, 2007. [27] P. A. Sandborn, T. E. Hearld, J. Houston, and P. Singh, “Optimum technology insertion into systems based on the assessment of viability,” IEEE Trans. Compon. Packag. Technol., vol. 26, no. 4, pp. 734–738, Dec. 2003. [28] V. S. Parsons, “Project performance: How to assess the early stages,” Eng. Manage. J., vol. 18, pp. 11–16, 2006. [29] R. M. Henderson and K. Clark, “Architectural innovation: The reconfiguration of existing product technologies and the failure of established firms,” Adm. Sci. Quart., vol. 35, pp. 9–30, 1990. [30] S. Nambisan, “Complementary product integration by high-technology new ventures: The role of initial technology strategy,” Manage. Sci., vol. 48, pp. 382–398, 2002. [31] R. J. Watts and A. L. Porter, “R&D cluster quality measures and technology maturity,” Technol. Forecast. Social Change, vol. 70, pp. 735–758, 2003. [32] DoD, Levels of Information Systems Interoperability. Washington, DC: Dept. of Defense, Mar. 30, 1998. [33] J. C. Mankins, “Approaches to strategic research and technology (R&T) analysis and road mapping,” Acta Astronaut., vol. 51, pp. 3–21, Mar. 21, 2002. [34] J. Fang, S. Hu, and Y. Han, “A service interoperability assessment model for service composition,” in Proc. IEEE Int. Conf. Serv. Comput. (SCC’04), Shanghai, China, pp. 153–158. [35] E. G. Nilsson, E. K. Nordhagen, and G. Oftedal, “Aspects of systems integration,” in Proc. 1st Int. Syst. Integr., Morristown, NJ, 1990, pp. 434– 443. [36] R. Gove, “Development of an integration ontology for systems operational effectiveness,” M.Sc. thesis, Stevens Inst. Technol, Hoboken, NJ, 2007. [37] R. Gove, B. Sauser, and J. Ramirez-Marquez, “Integration maturity metrics: Development of an integration readiness level,” School of Systems and Enterprises, Stevens Inst. Technol., Hoboken, NJ, SSE_S& EM_004_2007, 2007. [38] B. Sauser, J. Ramirez-Marquez, R. Magnaye, and W. Tan, “A Systems approach to expanding the technology readiness level within defense acquisition,” Int. J. Defense Acquisition Manage., vol. 1, pp. 39–58, 2008. [39] B. Sauser, J. Ramirez-Marquez, D. Henry, D. DiMarzio, and J. Rosen, “Methods for estimating system readiness levels,” School of Systems and Enterprises White Paper, 2007. [40] D. Verma, J. Beck, and T. Parry, “Designing and assessing supportability in DoD weapon systems: A guide to increased reliability and reduced logistics footprint,” OSD, Ed. Washington, DC: DoD, 2003.

547

[41] D. Verma, J. Farr, and L. H. Johannesen, “System training metrics and measures: A key operational effectiveness imperative,” Syst. Eng., vol. 6, pp. 238–248, 2003. [42] J. E. Ramirez-Marquez and D. Coit, “A heuristic for solving the redundancy allocation problem for multistate series-parallel systems,” Reliabil. Eng. Syst. Saf., vol. 83, pp. 341–349, 2004. [43] J. E. Ramirez-Marquez, D. Coit, and A. Konak, “Reliability optimization of series-parallel systems using a max-min approach,” IIE Trans., vol. 36, pp. 891–898, 2004. [44] D. Kumar, D. Nowicki, J. E. Ramirez-Marquez, and D. Verma, “On the optimal selection of process alternatives in a six sigma project,” Int. J. Prod. Econ., vol. 111, pp. 456–467, 2008. [45] J. E. Ramirez-Marquez and D. Coit, “Multistate component criticality analysis and reliability prioritization in multistate systems,” Reliabil. Eng. Syst. Saf., vol. 92, pp. 1608–1619, 2007. [46] J. E. Ramirez-Marquez and D. Coit, “Optimization of system reliability in the presence of common cause failures,” Reliabil. Eng. Syst. Saf., vol. 92, pp. 1421–1434, 2007. [47] H. Taboada, F. Baheranwala, D. Coit, and N. Wattanapongsakorn, “Practical solutions for multi-objective optimization: An application to system reliability design problems,” Reliabil. Eng. Syst. Saf., vol. 92, pp. 314– 322, 2007. [48] H. Taboada, J. Espiritu, and D. Coit, “MOMS-GA: A multi-objective multi-state genetic algorithm for system reliability optimization design problems,” IEEE Trans. Reliabil., vol. 57, no. 1, pp. 182–191, Mar. 2006. [49] P. Pardalos and M. Resende, Handbook of Applied Optimization. Oxford, U.K.: Oxford Univ. Press, 2002. [50] A. Ravindran, Operations Research and Management Science Handbook. Boca Raton, FL: CRC Press, 2007. [51] M. Corstjens and P. Doyle, “A model for optimizing retail space allocations,” Manage. Sci., vol. 27, pp. 822–833, 1981. [52] M. Ferris, J. Lim, and D. Shepard, “An optimization approach for radiosurgery treatment planning,” SIAM J. Optim., vol. 13, pp. 921–937, 2002. [53] J. M. Mulvey, D. P. Rosenbaum, and B. Shetty, “Parameter estimation in stochastic scenario generation systems,” Eur. J. Oper. Res., vol. 118, pp. 563–577, 1999. [54] J. E. Ramirez-Marquez and C. Rocco, “All-terminal network reliability optimization via probabilistic solution discovery,” Reliabil. Eng. Syst. Saf., vol. 93, pp. 1689–1697, 2007. [55] J. E. Ramirez-Marquez and C. Rocco, “Probabilistic knowledge discovery algorithm in network reliability optimization,” presented at the ESREL Annu. Conf., Stavanger, Norway, 2007. [56] C. Rocco and J. E. Ramirez-Marquez, “Deterministic network interdiction optimization via an evolutionary approach,” Reliabil. Eng. Syst. Saf., vol. 94, no. 2, pp. 568–576, 2009. [57] J. E. Ramirez-Marquez, “New approaches for reliability design in multistate systems,” in Handbook on Performability Engineering, ch. 30, New York: Springer-Verlag, 2008. [58] J. E. Ramirez-Marquez, “Port-of-entry safety via the reliability optimization of container inspection strategy through an evolutionary approach,” Reliabil. Eng. Syst. Saf., vol. 93, pp. 1698–1709, 2008. [59] J. L. Cook and J. E. Ramirez-Marquez, “Optimal design of cluster based ad-hoc networks using probabilistic solution discovery,” Reliabil. Eng. Syst. Saf., vol. 94, no. 2, pp. 218–228, 2009. [60] T. B¨ack and H. P. Schwefel, “Evolutionary computation: An overview,” in Proc. 1996 IEEE Int. Conf. Evol. Comput., Nagoya, Japan, pp. 20–29. [61] H. P. Schwefel and T. B¨ack, “Evolution strategies i: Variants and their computational implementation,” in Genetic Algorithm in Engineering and Computer Science, J. Periaux and G. Winter, Eds. Hoboken, NJ: Wiley, 1995. [62] D. King, “Saving hubble,” in Proc. 55th Int. Astronaut. Congr., 2004, pp. 3223–3230. [63] L. J. Lanzerotti, “Assessment of options for extending the life of the hubble space telescope: Final report,” Washington DC: National Academies Press, 2005. [64] J. J. Mattice, “Hubble space telescope: Systems engineering case study,” C.f.S. Engineering, Ed. Hobson Way, OH: Air Force Inst. Technol. 2005. [65] M. E. Pat´e-Cornell and R. L. Dillon, “Success factors and future challenges in the management of faster-better-cheaper projects: Lessons learned from NASA,” IEEE Trans. Eng. Manage., vol. 48, no. 1, pp. 25–35, Feb. 2001. [66] B. Sauser, “Toward mission assurance: A framework for systems engineering management,” J. Syst. Eng., vol. 9, pp. 213–227, 2006. [67] B. J. Sauser, A. J. Shenhar, and E. J. Hoffman, “Identifying differences in space programs,” in Technology Management: A Unifying Discipline

Authorized licensed use limited to: Stevens Institute of Technology. Downloaded on July 30, 2009 at 11:11 from IEEE Xplore. Restrictions apply.

548

IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT, VOL. 56, NO. 3, AUGUST 2009

for Melting the Boundaries, T. R. Anderson, T. U. Daim, D. F. Kocaoglu, D. Z. Milosevic, and C. M. Weber, Eds. Piscataway, NJ: IEEE Press, 2005, pp. 392–402. [68] A. Shenhar and B. Sauser, “Systems engineering management: The multidisciplinary discipline,” in Handbook of Systems Engineering and Management, 2nd ed. A. P. Sage and W. R. Rouse, Eds. Hoboken, NJ: Wiley, 2008. [69] K. Forsberg, H. Mooz, and H. Cotterman, Visualizing Project Management. Hoboken, NJ: Wiley, 2000. [70] K. P. Grant and J. S. Pennypacker, “Project management maturity: An assessment of project management capabilities among and between selected industries,” IEEE Trans. Eng. Manage., vol. 53, no. 1, pp. 59–68, Feb. 2006.

Jose Emmanuel Ramirez-Marquez received the M.Sc. degree in statistics from the Universidad Nacional Autonoma de Mexico, Mexico city, Mexico, and the M.Sc. and Ph.D. degrees in industrial engineering from Rutgers University, New Brunswick, NJ. He is currently an Assistant Professor in the School of Systems and Enterprises, Stevens Institute of Technology, Hoboken, NJ. He was a Fulbright Scholar, His current research interests include the reliability analysis and optimization of complex systems, the development of mathematical models for sensor network operational effectiveness, and the development of evolutionary optimization algorithms. He has conducted funded research for both private industry and government. He is the author or coauthor of more than 50 refereed manuscripts related to these areas in technical journals, book chapters, conference proceedings, and industry reports. Ramirez-Marquez has presented his research findings both nationally and internationally in conferences such as Institute for Operations Research and Management Science (INFORMS), Industrial Engineering Research Conference (IERC), Applied Reliability Symposium (ARSym), and European Safety and Reliability (ESREL). He is the Director of the Quality Control and Reliability (QCRE) Division Board of the Institute of Industrial Engineers and is a member of the Technical Committee on System Reliability for the ESRA.

Brian J. Sauser (M’04) received the B.S. degree in agricultural development with an emphasis in horticulture technology from Texas A&M University, College Station, TX, the M.S. degree in bioresource engineering from Rutgers, The State University of New Jersey, New Brunswick, and the Ph.D. degree in project management from Stevens Institute of Technology, Hoboken, NJ. He spent more than 12 years working in government, industry, and academia both as a Researcher/Engineer and the Director of programs. He is currently an Assistant Professor in the School of Systems and Enterprises, Stevens Institute of Technology. His current research interests include theories, tools, and methods for bridging the gap between systems engineering and project management for managing complex systems. He is also involved in the advancement of systems theory in the pursuit of a biology of systems, system and enterprise maturity assessment for system and enterprise management, and systems engineering capability assessment. To learn more about the work of Drs. Sauser and Ramirez-Marquez and the Systems Development & Maturity Laboratory on systems maturity assessment see http://www.SystemReadinessLevel.com

Authorized licensed use limited to: Stevens Institute of Technology. Downloaded on July 30, 2009 at 11:11 from IEEE Xplore. Restrictions apply.