Leveraging Design Principles to Optimize Technology Portfolio Prioritization Brett Depenbrock Booz Allen Hamilton 901 15th Street Washington, D.C. 20005
[email protected]
Tibor Balint Royal College of Art, School of Design Kensington Gore, London, SW7 2EU United Kingdom
[email protected]
Abstract—Research and development organizations frequently encounter challenges when attempting to identify an investment strategy and to accurately forecast the cost and schedule performance of selected projects. Fast moving and complex environments require managers to analyze quickly and to diagnose the value of returns on innovation investments versus allocated resources. Our Project Assessment Framework Through Design (PAFTD) tool facilitates decision making for NASA senior leadership to enable more strategic and consistent technology development investment analysis, beginning at implementation and continuing through the project life cycle. The framework leverages design principles of usability, feasibility, and viability and aligns them with methods employed by NASA’s Independent Program Assessment Office for project performance assessment. The need exists to periodically revisit the justification and prioritization of technology development investments as changes occur over project life cycles. The framework informs management rapidly and comprehensively about diagnosed internal and external root causes of project performance.
Jeffrey Sheehy Space Technology Mission Directorate Mail Code OA NASA Headquarters Washington, D.C. 20546
[email protected]
4. CONCLUSIONS ................................................... 8 REFERENCES.........................................................9 BIOGRAPHY ........................................................ 10 1. INTRODUCTION Managers in technology development organizations are faced with a multitude of challenges regarding the selection of an optimized strategy that drives content planning and execution. In the government sector, organizational inertia and resource requirements discourage riskier investment choices and restrict the ability of managers to diagnose root causes of performance. In the private sector, companies often strive to increase the levels of profit from quarter to quarter in lieu of making investments for longer-term returns. Choosing a strategy to increase an organization’s potential for sustained success is often clouded by comforts of strategic planning and cost-based thinking [1]. Good strategy involves focus that propels organizations to confront challenges, to make logic of strategy explicit, and to force managers to see if desired outcomes are being achieved [1]. For choices to make sense, managers need to document the logic of beliefs about customers, industry, competition, and capabilities [1]. If an organization is always comfortable with its choices, it may miss ongoing advancements in its environment and fail to provide a defined value proposition.
PAFTD offers a means to measure and quantify key aspects of different projects to enable consistent comparisons between projects in a loosely coupled investment portfolio. The model allows senior leadership to quickly diagnose project performance strengths and weaknesses to further improve their investment decisions. The framework has been employed to assess system level technology development investments across the high technology readiness levels of the NASA Space Technology Mission Directorate. It has been robust enough for assessment of investment efforts at NASA centers, federally funded research and development centers (FFRDCs), and industry.
Given the complex array of drivers that can affect technology development in the government framework, the model of wicked problems could be used to characterize circumstances surrounding the diagnosis of root causes for project performance [2]. Unlike business, where maximizing profit is the primary metric, government programs set their own goals. When goals are created, managers decide on what key variables are needed to achieve them [3]. Feedback from measurement can be used to make adjustments to which tactics work and which do not [3]. There is a need for a simplified and integrated approach that forces managers, especially within the government sector, to confront thorny issues and to ask difficult questions to arrive at measured outcomes. Measurement over time has allowed inventors to make incremental design changes that have led to improvements. Without feedback
PAFTD can also be tailored depending on the strategy and the success parameters of the assessed entity. Specifically, it can be customized to any organization that invests in technology development projects and which could benefit from an easy to use, integrated means of analyzing the relevance of investments with technical and cost performance. In this paper we provide an overview of the framework, its capabilities, and its utility as demonstrated through example cases at NASA.
TABLE OF CONTENTS 1. INTRODUCTION ................................................. 1 2. METHODOLOGY ................................................ 2 3. DISCUSSIONS ..................................................... 7 978-1-4799-5380-6/15/$31.00 ©2015 IEEE
1
moving all four limbs, and two if its heart rate was over a hundred. Published in 1953, the score changed a clinical concept and the condition of new babies into numbers that people could collect and compare. Nearly every child born in a hospital came to have an Apgar score recorded at one minute after birth and at five minutes after birth. A baby with a terrible Apgar score at one minute could be resuscitated with measures like oxygen and warming, leading to an excellent score in five minutes. The Apgar score was practical and easy to calculate, and it provided clinicians at the bedside immediate and objective feedback on how effective their care was [9]. Much like the Apgar score, PAFTD was developed through thinking that a need for a quick, integrated and objective read on the health of a technology investment is necessary to diagnose effective management techniques and performance characteristics, and to begin comparing the strengths and weaknesses of the portfolio elements of uncoupled system level technologies.
from measurement, invention is doomed to be rare and erratic [3]. Design thinking may provide an approach to expand the evaluation of technology efforts outside of traditional linear models that simply measure cost and schedule adherence [4]. Design thinking can introduce flexibility to current approaches while harmonizing constraints affecting usability, feasibility, and viability. This approach is being employed within the Space Technology Mission Directorate (STMD) at NASA, where the Project Assessment Framework through Design (PAFTD) tool provides an integrated level of independent project assessment and a quick and efficient means of measuring key factors surrounding the strategic performance of selected investments. Collected data is essential for use in scientific and strategic decision making, and PAFTD is a qualitative modeling tool that allows for creative means to test different hypotheses that STMD management have about particular types of system level technologies [5]. PAFTD is a point design assessment tool comprised of a number of strategic questions, and selected reply differentiators provide senior management with quick and diverse insight into performance. The PAFTD evaluation process pushes leadership to contemplate comprehensive and thoughtful scenarios and to drive out deeper and more obscure root causes related to the health of the projects within the portfolio. At NASA, insight gained from applying PAFTD to projects can be leveraged for more intelligent investment selections, but most importantly it can facilitate essential discussions about the identification and implementation of the effectiveness of STMD’s strategy. PAFTD helps leadership design a way of coordinating and focusing actions to deal with varying challenges, in line the discussions provided in [6].
Framework Development In August of 2013 NASA commissioned Booz Allen Hamilton Inc. (Booz Allen) to investigate the cost, schedule, and technical performance of projects with high technology readiness levels (TRLs). High TRL projects are those that develop, demonstrate, and then attempt to infuse elements into other mission directorate efforts. High TRL investments also compose the most expensive and most highly visible parts of the STMD portfolio. A constrained budget for NASA, and further for STMD, creates additional pressure for selected efforts to hold to early estimates for cost and schedule. In technology development this is especially difficult considering that these projects possess wider bands of uncertainty; hence, possess more risk and larger potential for cost overruns. STMD management expressed a need for additional technology assessment insight to improve strategic decision making and to exert more control over future outcomes. Challenges abound for senior leaders at NASA given the variety of influences that may affect a project’s performance. External factors to NASA, external factors to the project, and internal project planning and executing all play a role in influencing project outcomes. Consequently, PAFTD was developed to ensure that strategic discussions would continue after its initial assessment efforts.
In this paper we will introduce the methodology used to create PAFTD and discuss potential uses of this assessment tool in support of strategic considerations by the leadership of a governing organization.
2. METHODOLOGY The Need for Models
STMD often classifies high TRL investments as crosscutting, which means advancing technologies for the benefit of multiple stakeholders at NASA, the broader aerospace industry, Other Government Agencies (OGAs), and academia, to address national needs. [10] The crosscutting nature of these investments exposes them to a number of different stakeholders with varying interests. NASA’s investment choices are generally framed by the Strategic Space Technology Investment Plan [11] and the National Research Council’s Space Technology Roadmaps [12]. PAFTD was developed to respond to outlined assumptions in these documents, and to help STMD
In general, models are created through the reduction of complex systems to simple ones [7]. Identifying and capturing the most important drivers while neglecting others define the fidelity of a model and at the same time introduce errors or deviations from the observed system. As George E.P. Box [8] pointed out, “essentially all models are wrong, but some are useful.” Atul Gawande mentions the usefulness of a particular model in the medical world [9]. The Apgar score was developed to allow a nurse to rate the condition of a baby at birth on a scale of zero to ten. An infant received two points if it was pink all over, two for crying, two for taking good, vigorous breaths, two for 2
Design Thinking
m management prioritize and direct d focus to o identifying and a ddirecting resourrces that align with organizattional goals.
D’s underlyinng foundation and strengthh are the PAFTD formulaated questionss that comprisse the model. PAFTD’s questioons were derivved from those presented as part of intervieews and questiionnaires. The finalized set of questions was baased on the exxtensive reseaarch and intervviews with NASA STMD senioor technical aadvisors and high TRL program m executives. T The questions w were then sorteed into one of the N NASA (IPAO) six review ccriteria, and theen mapped into a rrelated design thinking categgory, namely: ((a) Agency Strateggic Goals to Usability; (bb) Technical Approach, Resourrces other thhan Budget tto Feasibility; and (c) Manageement Approaach, Budget aand Schedule, and Risk Manageement to Viabiility (see Figurre 2).
S Senior leadersship identified d a diverse set of projects inncluding efforrts based out of NASA Ceenters, Federaally F Funded Research and Develo opment Centerss (FFRDCs), and a ccontractor sitess. To initiate the investigation, Booz Alllen ddeveloped surrveys to collect informaation from the t aappropriate Prrogram Officee. Booz Allen n used NASA A’s IIndependent Prrogram Assesssment Office (IPAO) standaard ooperating proceedure instruction (SOPI) [13] and the NAS SA S Space Flight Prrogram and Pro oject Managem ment Handbook k’s [14] six review w criteria to cap pture (a) cost, (b) schedule, (c) technical data,, and (d) inteerview respon nses. The IPA AO rreview criteria areas identified for assessmeents are shown n in F Figure 1. IPAO O is the agency y’s independentt body within the t O Office of Evaluation E th hat provides impartial and a ccomprehensive assessments free from the management or aadvocacy chain n of programs and a projects. IP PAO’s purposee is too determine whether the program’s/prroject’s plann ned bbudget and schedule s are adequate to accomplish the t pproposed techn nical work. IP PAO integratess cost, schedu ule, aand risk analyses [13]. A Agency Strateg gy Goals
Managemen nt Approach
T Technical Apprroach
Budget and Schedule
R Resources B Budget
Other O
than
Usabiliity addresses tthe issues of “why invest nnow,” user needs, infusion poiints, technoloogy push, and mission pull/neeeds. It introduuces strategic ddiscussion as it seeks to consideer customer nneeds, technollogy differentiiation, and obsolesscence. Feasiibility is relaated to the technical, engineeering, and sccience soundnness of the innovative conceppt. It also conncerns the team m and its relevant core knowleedge, and the aavailability of ssuitable facilitiies to carry out the development. Viability relattes to cost, othher funding sourcess, and schedulee, risk, and mannagement technniques.
Risk Manag gement
Figu ure 1: IPAO reeview criteria areas
Usaability
Feasibilitty
Q Questions werre initially deeveloped and interviews weere cconducted witth Program Office perso onnel includin ng: pprogram manag gers, cost and schedule integ gration leads, and a pproject mission n managers. Responses R weere then collatted innto the above listed l six areass.
Ageency Strrategy Goaals
Technical Approach
Maanagement Ap pproach
Resources Other than n Budget
Bu udget and Schedule
O Over the follow wing six month hs additional data was collectted ffrom not only the t Program Office, but also from Project and a H Headquarters (HQ) levels. Some additio onal sources of rreference and in nvestigation in ncluded STMD D’s Portfolio Pllan [10], NASA's Cost Estimaating Handboo ok [15], Projeect P Plans, and NA ASA Proceduraal Requirements NPR 7120.5E ((for NASA Spaace Flight Program and Projject Managemeent R Requirements) [16], and NPR R 7120.8 (for NASA Researrch aand Technolo ogy Program and Projecct Managemeent R Requirements) [16]. Booz Allen A then deeveloped projeect sspecific questtionnaires and d conducted site visits and a aadditional interrviews with prroject and HQ personnel. Bo ooz A Allen then begaan to define ST TMD high TRL L project succeess ccriteria by sy ynthesizing an nalysis from collected daata, qquestionnaire responses, r and d stakeholder interviews with w L Level 3 (Project), Level 2 (P Program Officce), Level 1 (H HQ ppersonnel), and d project perso onnel external to NASA. Theese ssuccess criteriaa include a series of key diifferentiators and a ccorresponding questions thatt could be app plied to measu ure vvarying levels of success acrross the STMD D projects in the t pportfolio of hig gh TRL efforts..
Viaability
Risk Maanagement
Figure 2: D Design Thinkiing Categoriess Designn thinking cann help senior managers to create and ment the most suitable strateegic frameworrk for their implem organizzation. By com mbining inform mation relatedd to future opportuunities and threeats with data ooriented decisiion making from pproject perfo rmance assesssments, mannagers can optimizze project connstraints withhin their portffolio. This balancee between inttuition and annalytics recoggnizes that understtanding and setting the problem spacce are as importaant as attemptss to solve it [177]. At the strateegic level a need exxists to examiine and priorittize a handful of success 3
“Agency Strategic Goals” was mapped to Usability, and described as follows:
factors that matter most. PAFTD aims to provide a robust design evaluation method that measures what is technologically feasible and what makes business sense for NASA.
“Agency Strategic Goals” 1.
Design thinking [18] provides a new approach to assessment. As stated in [2], “…design accounts for more than simple ergonomics and packaging. Therefore, we should focus less on the resulting artifacts and more on the approach to achieve the desired goal”. The design and creation of project architectures can benefit from design thinking and systems thinking, from a combination of observational research, brainstorming for new solutions, and rapid iterative prototyping. Design thinking looks at a broad range of considerations, including understanding of the culture, aspirations, motivations and context. Multiple stakeholders have diverse sets of drivers and expectations. Design thinking encourages us to take a divergent approach and to create new options, explore new alternatives, and in a subsequent convergence phase find new solutions and ideas that didn’t exist before. PAFTD addresses usability, feasibility, and viability related questions to enable management to roll information into higher level thinking. It is structured in a way that provides information in line with design thinking approaches. Using PAFTD provides systematically collected information, packaged in a way that is usable if management wants to use design thinking principles to find new solutions.
2.
3.
4.
To what extent is the technology development effort still programmatically relevant? Have there been any recent agency policy changes that might have affected the relevancy of the technology? To what extent are concurrent investments within industry (FFRDCs, e.g., the Jet Propulsion Laboratory/Caltech, MIT Lincoln Laboratory) or other organizations in the given technology not a threat to the relevancy of the technology investment? To what extent has the project succeeded in avoiding delays that might drive future interested/infusion technology partners away from this effort? To what extent is there a differentiating element for NASA's investment in this technology? Is there uniqueness about the value that NASA gets from advancing this technology?
The initial question concerning programmatic relevancy is aimed at asking if any recent agency policy changes might have affected the relevancy of the technology. The high TRL portfolio infuses technologies into more developed mission architectures in other Mission Directorates, and it is necessary to measure the strength of an STMD investment as it relates to agency policy that could change. NASA often shifts priorities from one administration to another, and it is necessary to measure how closely tied STMD investments are with other NASA Mission Directorates and closely tied political priorities that could shift.
Implementing PAFTD PAFTD serves as a guide to both evaluate the technical and cost performance for high TRL projects, but also to incorporate important considerations that may affect the decision to continue to invest in these efforts. The design thinking categories of Usability, Feasibility, and Viability include questions and key differentiators that can be weighted according to the evaluation priorities of the assessor. An independent assessor is the ideal navigator of the framework, and the exercise of going through the questions and making determinations about success criteria is most useful as a continuous exercise with senior leadership. PAFTD is chiefly designed to measure projects that are past the proposal, evaluation and selection stages. The success criteria are framed around the expectation that the maturity for projects under evaluation reflects efforts either approaching or already in implementation. These assumptions are present in identifying the appropriate level of success for the ratings. To use this tool, an evaluator selects a key differentiator from each question via the dropdown menus in the 'Success Level' column to characterize the success of the project across the three design thinking categories.
The second question examines the relevancy of investment from a competition perspective, where competing industry and other government authority investment may pose a significant risk to the relevancy of certain technology investments because of industry’s matching or better performance and faster completion potential. NASA often selects investments in which industry does not compete because of high risk or unforeseen benefit at the time of investment. In some cases NASA invests in projects that possess technical similarities and development goals to investments by industry or other foreign private or government organizations. In these cases NASA must be mindful of quick moving technology developments across aerospace and aviation fields that may render STMD’s contribution to a high TRL investment obsolete. STMD managers ought to consider that resource requirements and a cultural reluctance to cancel projects typically constrain flexibility, and the external competing technology environment must continually be evaluated [19].
Usability— Four questions comprise the Usability category. They include differentiators concerning programmatic relevancy, competing industry and other government authority related investment risks, infusion path timeliness, and investment differentiation. The IPAO category titled
The third question evaluates the timeliness of project scope 4
“Resources Other than Budget”
execution as it relates to interested customers. To maximize STMD’s ability to create options for continued value, a lean and fluid portfolio of high TRL efforts is essential. Infusion path timeliness is concerned with the timeline of identified partners and STMD’s ability to execute and deliver with fluidity and quickness. In technology development technical limits are challenges, and cost risk is encumbered to a point when mitigating risk finally involves descoping previously outlined requirements. This question aims at identifying the strength of infusion partnerships, and if delays within STMD occur, its answer would yield insight as to the continued interest of previously outlined infusion partners.
1.
2.
Questions in the Feasibility category serve to gather more project specific insight, and attempt to identify primary areas of technical concern for HQ. STMD needs to be cognizant of the risk that the appropriated budget from Congress may differ and often is less than what is asked in the President’s Budget Request [2]. A reduced STMD budget portfolio, but one that includes the same number of high TRL investment commitments requires greater emphasis on cost and schedule and less willingness to take risk that may result in overruns. If STMD wishes to push boundaries, it must develop expectations of the types of outcomes it may be willing to encumber should it select certain types of technologies to invest in.
The final question in the Usability category asks about the extent to which there a uniqueness about the value that NASA and STMD gets from advancing this technology. Some technologies possess more difficulty to overcome science and technology challenges. The question of uniqueness as it relates to NASA is meant as well to ask STMD managers how unique offerings from the organization are from other mission directorates. Does STMD possess a unique differentiation platform? Do other customers – both current and future – recognize areas where STMD invests and will invest going forward? The uniqueness of STMD’s investments now will determine the future pipeline of customers who will recognize STMD’s competitive advantage. With a defined differentiation platform NASA STMD is able to execute a recognizable slate of valued investments and inherit future opportunities.
The first question in the Feasibility “Technical Approach” section addresses the effectiveness of project management and execution tailoring. In developing a lean and agile set of investments STMD tailors project requirements to reduce process, reporting, and test requirements that do not contribute to the effectiveness of technology development. This is particularly challenging given the high level of operational and engineering complexity that STMD Project Managers encounter. In addition, a mission-oriented agency geared toward human exploration tends to reward proven technologies with lower risk postures. Tailoring system development efforts implies an increased risk tolerance. As such, reaching a consensus on what constitute appropriate levels of tailoring can be difficult because mission failures within the agency are not acceptable.
Feasibility— Seven questions comprise the Feasibility category: five of them are mapped to IPAO’s “Technical Approach”, and two are mapped to IPAO’s “Resources Other than Budget.” “Technical Approach” 1. 2.
3.
4.
5.
To what extent was the workforce planning and skill mix appropriate given the outlined technical content to be accomplished? To what extent do participating NASA Centers and/or lead contractors possess adequate manufacturing and test facilities to complete the outlined scope?
To what extent have project requirements been appropriately tailored? To what extent were the component based technology readiness levels (TRLs) for this project initiated at the appropriate levels to advance the system level TRL against Level 1 requirements? To what extent have heritage hardware/complexity assumptions been built into the bases of estimates, and how accurate were the assumptions surrounding work with hardware/software? To what extent have project descopes either affected Level 1 requirements or altered the technology value capability associated with interests from stakeholders? To what extent has the project made technical progress and how effective have developed technical solutions dealt with challenges that have occurred?
The second and third questions related to “Technical Approach” probe into technical assumptions. Component based pieces of subsystem technology at lower than anticipated TRLs as well as heritage must be investigated as part of system developments for high TRL efforts. The optimistic culture at NASA may lead managers to overestimate their ability to overcome risks inherent in delivering projects within available funding constraints [19]. The additional rework that comes from having to advance previously unforeseen technology as well as fixing inaccurate assumptions about uniqueness and the design of the hardware provide clear challenges to project managers within STMD. The fourth and fifth questions examine the origin and necessity of project descopes. When funding constraints are too tight, project managers face difficult decisions regarding descopes and continuing previously outlined technical progress. Project managers are able to adjust the dates of different reviews to mitigate risk, but the experience of a project manager is vital in the ability to diagnose the balance between technical progress and either 5
Viability questions are oriented toward identifying levels of accountability and corresponding management strategies. Hierarchical organizations like NASA need to continue to make roles and responsibilities clear, and to acknowledge the difficulties of communication gaps that inevitably occur.
cost or descopes mitigation practices. In “Resources Other than Budget” the questions concern the design of the business. It is a challenge for NASA to attract and to distribute top tier talent. Given the complexity of the technology issues within NASA, a continued evaluation of resources and facilities contributes to an intelligent design of the organization.
“Management Approach” identified questions that probe effective cost estimation and financial management techniques of the project and the corresponding program office. To some extent, it is necessary to acknowledge that for budget and schedule related issues, projects with the most knowns would tend to have the best cost-performance [20] [21] [22]. STMD is an organization with ambitious goals, where a project with excellent cost performance may simply be able to produce a reliable outcome. In an effort to create reliable outcomes, two qualities are identified within the “Budget and Schedule” category in relation to successful high-TRL projects, namely a committed and willing partner and a clear buy in from senior leadership. The ability for NASA HQ to allocate funding to a project is often determined by fluctuating resource levels, driven by external influences (e.g., from Congress for budget appropriation, and OMB, OSTP for budget requests) [2]. It is important to consider these factors as they play a vital role for enabling stability. “Risk Management” questions were constructed around outcomes that might involve clarity for HQ. Because of different factors incentivizing performance across NASA and its Centers, HQ sometimes may encounter challenges with extracting clear traces of cost and schedule growth. “Risk Management” identifies the effectiveness with which project applied risk, how adequately they were reported, and if they could easily be traced to encumbered cost and schedule growth.
Viability— Twelve questions comprise Viability, from which: five are related to IPAO’s “Management Approach,” four are related to IPAO’s “Budget and Schedule,” and three are related to “Risk Management.” “Management Approach” 1.
2.
3. 4.
5.
How well do primary calculations, rationales, and sources of data exist to explicitly identify how work breakdown structure (WBS) elements as part of project bases of estimates (BOEs) were created? To what extent has the project employed financial management best practices to assess results and performance? To what extent has the project effectively coupled and coordinated technical and financial efforts? To what extent has the reporting/communication process from the project to the program office included the appropriate level of detail of planning, tracking, and analysis results? To what extent have external influences ((Office of Management and Budget (OMB), Government and Accountability Office (GAO), Office of Science and Technology Policy (OSTP), Congress)) played a role in creating budget and schedule stability?
Using PAFTD
“Budget and Schedule” 1.
2. 3.
4.
The STMD questions, differentiators, and question-specific analysis include extensive intelligence gathered from research, meetings, and stakeholder interviews. At STMD the basic framework is formed and ready for immediate use to aid organizational decision makers. An STMD independent assessor is able to quickly evaluate and make judgment recommendations about varying areas of projects, but continued data collection and interviews would provide the most robust and defensible answers for given grades by the assessor.
To what extent has the project delivered robust cost performance, i.e., quality phasing plans, justified variances between obligations and cost, timely budget execution, risk mitigation strategies? To what extent is the project funded with the resources of a willing and vested partner(s)? To what extent has HQ coordinated the budget with the schedule plan to ensure that funds were available when needed for the project? To what extent is the current resource level sufficient to complete the work by the target date in the current NASA budget environment?
The independent assessor should first characterize the level of assessment as Successful, Partially Successful, or Not Successful across the three design thinking categories. To capture the level of success of a given project inside the PAFTD tool, the independent assessor clicks on the dropdown menus of the 'Success Level' column of the Process Assessment Rubric and selects Successful, Partially Successful, or Not Successful.
“Risk Management” 1. 2.
3.
To what extent were project reserves applied based on risk analysis findings? To what extent have encumbrances, liens, and threats been adequately and clearly reported and explained by the project? To what extent have project risk diagnoses accurately forecast future encumbered cost and schedule growth?
Once a selection is made, a description of the key differentiators for the selected ranking will populate the adjacent cell, so that the assessor can check to ensure that 6
are reached; (3) Solutions to wicked problems are not right or wrong; (4) Every wicked problem is essentially novel and unique; (5) Every solution to a wicked problem is a “one shot operation”; (6) Wicked problems have no given alternative solutions. Detailed discussion on these characteristics and their relevance to NASA is given in [2].
the appropriate ranking was selected. The key differentiators will identify characteristics that comprise the most accurate representation of what rating is most applicable to the project at a given point in time. Each ranking level is also linked to specific point values, namely: 3 for Successful, 2 for Partially Successful, and 1 for Not Successful. Weighting is the distribution of the category scores by percentage of the total score. Weighting impacts the overall assessment score of a project. This tool has been flexibly developed to allow the assessor to assign specific weight distributions based on project phase or assessor expertise. The weighting employed below is representative of a general assessment. Because weighting can be changed depending on the project phase or assessor subjectivity, the insight gleaned from the tool is most available in the categories and the questions that comprise them. As the framework evolves, and a number of projects are evaluated using it, the larger sample size will enable a fairer comparison of total score significance. Senior leadership must think critically when determining the significance of overall scores as desired outcomes that comprise PAFTD may not be the optimized outcomes. The usefulness of the questions is to facilitate revised exercises for increasing imaginative strength for furthering enduring organizational success.
Usability:
Feasibility:
Viability:
Weighted Possible Percentage = 5%
Weighted Possible Percentage= 25%
Weighted Possible Percentage = 70%
Total Possible Score – “Agency Strategic Goals”
Total Possible Score “Technical Approach”
Total Possible Score – “Management Approach”
= 5
= 10
Total Possible Score – “Resources Other Budget” = 15
= 20
Total Possible Score “Budget and Schedule” = 30
Total Possible Score – “Risk Management”
All of the scores roll up to present a final project assessment score for each design thinking category. The total score for each category is then calculated by dividing the actual score by the possible score, then multiplied by the designated weight. The total project assessment score is calculated by aggregating the individual category scores. For example, when looking at a project well into its implementation phase, the weighting and total score assignments by categories may be as follows (see Figure 3).
= 20
Figure 3: Weighting and total score assignment example
These weighting and score assignments can be adjusted based on the phase of the project and the assessment need of the users. For example, during the formulation phase usability and technical feasibility may play more dominant roles than in later project phases.
Wicked problems are not simply too hard or complex. They can’t be solved by additional considerations or have more stakeholders. The initial problem definitions and the outcomes are bi-directionally linked, and the various stakeholders may have radically different perspectives, motivations, and drivers related to the issues. Therefore, the assessment of an optimal outcome is dependent on the perspective of the stakeholder, instead of considering it universally correct. (For example, the perspectives of senior leadership at the strategic level during the project performance assessment may differ from the perspectives of project managers.) Because the initial problems and the related resource requirements are often ill defined, they are typically over-constrained. Often they can’t be solved definitively through analytical thinking, and may require innovative solutions. Decisions related to individual projects within a portfolio are integral parts of the strategic framework of an organization. Roberts identified three general strategies to tackle wicked problems [25], such as authoritative, competitive, and collaborative. Their implementation is influenced by management styles and institutional approaches.
3. DISCUSSIONS Wicked Problems at NASA The phrase “wicked problem” was introduced by Rittel and Webber [23] in connection with social planning. The term relates problems with no obvious solutions, driven by changing requirements, and incomplete or contradictory bounding conditions. The interdependencies between the various issues are complex, and solutions could result in subsequent new problems. Rittel and Webber’s ten general rules to describe wicked problems were reduced to 6 general characteristics by Conklin [24], which are as follows: (1) The problems are not understood until after the formulation of a solution; (2) Wicked problems have no stopping rules, difficult to know when the problems are solved or solutions 7
systematic way. PAFTD begins to facilitate an organizational examination that involves thinking through second and third order consequences. To maximize success, managers must be willing to confront and to challenge hinge assumptions in a dynamic technology environment in order to improve performance.
Complexities associated with NASA’s operational and budgetary framework within the government environment is driven by a broad range of factors and constraints at various levels of the organizational hierarchy, from the Government’s Executive and Legislative Branches, through NASA HQ, to the program and project levels (also including external contractors and influences). These connections can be described through the model of wicked problems. In [2] the wicked problems model was extended by including NASA relevant cyclical temporality and spatiality. The success and even the existence of projects and programs are often dependent on a large number of influencing factors and considerations beyond need (usability), technical feasibility, and fiscal viability. Understanding this extended framework beyond the bounds of NASA can impact strategic decision-making, especially in the problem diagnosis phase. It may also provide insights for project and program managers, who are typically involved with only a linear subset of this framework. Independent assessment results of a project using PAFTD may provide valuable information to senior leadership at the strategic level of an organization to support strategic decision-making and aid the dialogue with the project to better communicate the findings, constraints, and resulting actions to remedy the issues.
4. CONCLUSIONS The PAFTD tool can be used to capture and identify strategic knowledge related to the health and performance of a given project. It can also help to formulate Mission Directorate level strategies and communicate key aspects about the project to stakeholders at strategic levels (i.e., locally within the Mission Directorate; regionally to the Agency, and globally to the Government [1]). In addition, the captured knowledge could be used to communicate key findings to lower linear discipline levels (i.e., to the Level 2 Program Office and to the Level 3 Project). Due to its systematic and multi-disciplinary structure, addressing usability, feasibility and viability with gained knowledge is less biased than findings focusing on just a few aspects, such as project management and resource use history. Consequently, decisions based on the gained knowledge through the use of PAFTD could drive strategic understanding and the subsequent actions to respond with the appropriate approach. Thus, risks to both the project and to the overall Mission Directorate portfolio could be minimized.
What Makes Strategic Questions Different? The PAFTD framework is a comprehensive platform for evaluating strategic areas of project performance, and is not intended to replicate or replace current reporting criteria. Reporting tends to be reactionary and to create linear tracks for management thinking. Given the complexity of managing technology development at a strategic level, STMD ought to rely on multiple models to enable the creation of options for creative investment selections. STMD managers, especially at the strategic HQ level, delegate execution to a project, and consequently exercise less control over the day-to-day cost and schedule management. Once requirements are laid out, a project is selected and moves into an implementation phase. At times reporting may overburden projects with limited resources (e.g., related to workforce and budget).
The findings are currently incorporated into an Excel based tool, but future versions can incorporate more detailed reporting inputs that may capture the gained knowledge in possibly a graphic way or may extract a relevant narrative in a summary report format. The fidelity of the assessments using PAFTD provides a top level of detail that allows managers to drill down on areas of interest or concern. For example, viability questions considering the cost assessment are low compared to parametric and grass roots estimates, but sufficient to capture knowledge about the health of the project and to aid decision making. It may be used to flag issues that require detailed assessment. The PAFTD tool has been deployed across a number of projects that comprise the high TRL portfolio within NASA STMD. Specifically, Booz Allen recommendations that leveraged PAFTD insights have enabled a broader set of senior managers to collaborate and work off a defined and systematic set of logic for making decisions. PAFTD has fostered a greater awareness of the need for STMD senior leadership to examine the external technology environment and to develop a focus on a high TRL technology pipeline that would harness increased flexibility and broader impact now and into the future. The comprehensive nature of PAFTD has facilitated more stringent technology readiness assessments, established more defined thresholds for enforcing reviews of poor project performance, and instituted clearer and more consistent processes for traceability. PAFTD informed one senior manager of the
The most important and most difficult decisions are the strategic decisions with consequences for performance. PAFTD focuses on questions where STMD management can examine and define success criteria and attempt to influence outcomes. Strategic decisions are fundamentally different from the routine choices and judgments captured in project laboratory settings during execution. Sources of careful and dispassionate analysis foster an honest dialogue between the senior leadership and project management that creates perceptual acuity. Perceptual acuity is the ability to sense what is coming before the fog clears [26]. Decision makers need to develop skills to discern the nature of the decision at hand and then to respond with the appropriate approach [27]. PAFTD enables STMD managers to look at a handful of key variables to focus their decisions on, in a 8
[8] Box, G.E.P., Draper, N.R., 1987. “Empirical ModelBuilding and Response Surfaces”, Wiley Series in Probability and Statistics, ISBN-10: 0471810339
need to have constraints set into a center led project that would not allow it to be continually pushed out (i.e., a hard launch date). In addition, PAFTD also informed another senior manager of a competing and similar technology development effort being conducted by ESA that may have an effect on STMD’s current value proposition regarding a current and similar investment in the portfolio.
[9] Gawande, A., 2008. “Better: A Surgeon's Notes on Performance”, Picador, ISBN-13: 978-0312427658 [10] NASA, 2014a. Space Technology Mission Directorate Portfolio Plan (Internal Document), Space Technology Mission Directorate NASA Headquarters, Office of Strategic Integration & Analysis, May 2014
Ultimately, PAFTD aims to harness the multitude of options that managers can employ to influence optimized outcomes. Central to the model’s backbone is the ability of an organization to diagnose its success criteria [28]. Given the tendency for technology development leaders to set ambitious goals, PAFTD is a useful mechanism to foster increased accountability, communication, and realistic expectations of organizational capabilities and competition. It serves to provide a simplified means of performance measurement to create urgency for innovation and to begin the diagnosis of appropriate strategy to achieve lasting success.
[11] NASA, 2013a. “NASA Strategic Space Technology Investment Plan”, NASA Office of the Chief Technologist, Viewed: August 8, 2014, Website: http://www.nasa.gov/offices/oct/home/sstip.html#.UhFPf RbYP5Z [12] NRC, 2012. “NASA Space Technology Roadmaps and Priorities: Restoring NASA’s Technological Edge and Paving the Way for a New Era in Space”. Washington, DC: The National Academies Press. ISBN 978-0-30925362-8
REFERENCES [1] Martin, R.L., 2014. “The Big Lie of Strategic Planning”, Harvard Business Review January 2014: pp. 78-84
[13] NASA, 2013b. Standard Operating Procedure Instruction SOPI 5.02 Baseline: Programmatic Assessment Process Revision B (Internal Document), NASA Independent Program Assessment Office, January 2013
[2] Balint, T., Stevens, J., 2014. “Wicked Problems in Space Technology Development at NASA”, 65th International Astronautical Congress, Toronto, Canada, Paper number: IAC-14-D1.3.6, September 29 – October 3
[14] NASA, 2014b. Space Flight Program and Project Management Handbook, Office of the Chief Engineer, Viewed: October 13, 2014, Website: http://nodis3.gsfc.nasa.gov/oce_docs/oce_13.pdf
[3] Gates, B. 2013. “Annual Letter from Bill Gates,” 8 January 2013. Letter 5 of Annual Letters from Bill Gates, Bill and Melinda Gates Foundation. Seattle, WA: pp. 1-22
[15] NASA, 2008. Cost Estimating Handbook, NASA Cost Analysis Division, Viewed: October 13, 2014, Website: http://www.nasa.gov/pdf/263676main_2008-NASA-CostHandbook-FINAL_v6.pdf
[4] Balint, T., 2013. “Disruptive Innovation: a comparison between government and commercial space”, 64th International Astronautical Congress, Beijing, China, Paper number: IAC–13–D1.3.3–17306, September 23-27
[16] NODIS, 2014. “NODIS Library - NASA Online Directives Information System”, Viewed: October 10, 2014, Website: http://nodis3.gsfc.nasa.gov/lib_docs.cfm?range=7
[5] Watts, D., 2014. “Scientific Thinking in Business,” MIT Technology Review January 2014: pp. 4-5 [6] Rumelt, R.P., 2011. “Good Strategy, Bad Strategy: The Difference and Why It Matters”, Crown Publishing Group, Random House, New York, eISBN: 978-0- 30788625-5
[17] Martin, R.L., 2009. “Design of Business: Why Design Thinking is the Next Competitive Advantage”, Harvard Business Review Press, ISBN: 978-1-422-17780-8, iBooks. https://itun.es/us/Uxjrz.l
[7] Weinberg, G.M., 1991.”The Simplification of Science and the Science of Simplification”, in G.J. Klir (ed) “Facets of Systems Science”, International Federation for Systems Research International Series on Systems Science and Engineering Vol.7, pp 501-5, Springer US, doi: 10.1007/978-1-4899-0718-9_35
[18] Brown, T., 2009. “Change by Design; how design thinking transforms organizations and inspires innovation”, Harper Collins, ISBN 978-0-06-176608-4 [19] NASA, 2012. “NASA’s Challenges to Meeting Cost, Schedule, and Performance Goals”, Office of Inspector General; Viewed: October 13, 2014, Website: http://oig.nasa.gov/audits/reports/FY12/
9
[20] Balint, T., 2009. “Rapid Cost Assessm ment Methodolo ogy for Planetaary Program Planning”, 60 0th Internation nal Astronautical Congress, Daejeon, Rep public of Korea, Paper: IAC--09.D1.3.1, Octtober 12-16
BIOGRAPH HY B Brett Depenbroock is a management coonsultant for Booz Allen H Hamilton cuurrently suppoorting an enggagement w within NASA Space Tecchnology M Mission Direectorate. H He has prreviously adviised senior leaadership w within federal aerospace, defense, annd emergeency management organizzations on bussiness strategyy design and opptimized technollogy managem ment. He is a 2010 graduatte of the McDonnough Schoool of Busineess at Geoorgetown Univers rsity with a fo focus in Finannce and Interrnational Businesss.
[21] Peterson, C., C Cutts, J., Balint, T., Hall, J.L., Senske, D., D Kolawa, E., Bullock, M., 2008. 2 “Rapid cost assessmentt of space missio on concepts thrrough applicatiion of complex xity indices”, IEEE 2008 Aerosspace Conferen nce, March 1-8 [22] Peterson, C., Balint, T., Cutts, J., Kwo ok, J., Hall, J.L., 2009. “Ev valuating Low w Concept Maturity M Missiion Elements and a Architecttures for a Venus Flagsh hip Mission”, AIAA A Space 20 009 Conferencee and Expositio on, September 14-17 1 [23] Rittel, H.W W.J., Webber, M.M., 1973. “Dilemmas in n a General Th heory of Plan nning”, Policy Sciences, 4((2), pp.155-69, Elsevier Sciientific Publisshing Compan ny, Amsterdam
T Tibor Balint is currently a PhD caandidate at thee Royal Collegge of Art, Sc School of Desiggn, UK. His rresearch inn Innovation Design Enggineering foocuses on thee role of deesign to addvance technoological innovvation in thhe governmentt framework. H He spent 8 yearrs at the Jet Propulsion L Laboratory, Caalifornia Institutte of Technoloogy, in Pasadeena, CA, as a systems engineeer and missionn architect, annd the past 4 yyears at NASA Space Technnology Missioon Directoratte as a Prograam Executive fo for the Game C Changing Deveelopment Prograam and subssequently as a Senior T Technical Advisorr. Prior to his work at N NASA, for 9 yyears he conduccted nuclear saafety analysis,, as a nuclearr design engineeer, at Ontario H Hydro, Torontto, Canada. Hee holds a PhD inn Engineering from Warwicck University, UK; an MSc inn Mechanicall Engineeringg from the T Technical Univers rsity of Budapeest, Hungary; aan MPhil in C Chemical Engineeering from thee University of Exeter, UK;; and an MSS ((Master of SSpace Studiess) degree frrom the Internaational Space University, Strrasbourg, France. Dr. Balint is a full membber of the Inteernational Acaademy of Astronaautics.
[24] Conklin, J., J 2006. “Dialogue mapping: building sharred understandin ng of wicked problems”, p Chichester, Englan nd: Wiley Publiishing. ISBN:04 470017686 [25] Roberts, N., 2000. “W Wicked Problem ms and Netwo ork Approachess to Resolu ution,” Intern national Pub blic Managemen nt Review., 1(1), pp.1-19 [26] Merino, M., M Charan, R.., 2013. “You Can’t be Wim mp Make the Tough Calls,,” Harvard Business B Revieew November 2013: 2 pp. 72-78 8 [27] Rosenzweiig, P., 2013. “W What Makes Sttrategic Decisio ons Different,” Harvard Busin ness Review November N 201 13: pp. 88-93 [28] Courtney, H., Lovallo, D., D Clarke, C., 2013. “Decidiing How to Deecide,” Harvard Business Reeview Novemb ber 2013: pp. 62 2-70
Jefffrey Sheehy iis currently the Senior Technical Officcer at NASA A Space Technology Misssion Directorrate. He hass nearly 25 yyears of experrience in aerrospace tecchnology rresearch, devvelopment, annd demonstraation at NA ASA and the Air Force R Research Laboraatory (AFRL). Sheehy earneed a PhD in cchemical physicss from Indiaana Universiity and receeived a prestigi gious postdoctooral associateeship award ffrom the Nationaal Research Council. He has authoredd or coauthoreed 40 journal articles in peeer-reviewed sscientific literatuure, as well as five book chappters and 30 ttechnical reportss.
10
