C OV ER F E AT U RE
Achieving Simplicity with the Three-Layer Product Model Jan Bosch, Chalmers University of Technology, Sweden
Increased system complexity has historically been treated as an inevitable consequence of architecture evolution over time. The three-layer product model offers an innovative framework for managing system growth that encourages greater efficiency, nimbler responsiveness, and more opportunities for innovation during all stages of the software development life cycle.
P
rior research, particularly the work of David Parnas1 and of Dewayne Perry and Alexander Wolf,2 has clearly established that architectures erode as they age over time. Systems become less and less amenable to necessary adaptation and change, often resulting in significantly higher maintenance costs. Although precise definitions differ, both researchers and practitioners point to increased complexity as the primary challenge of dealing with long-lived software systems. This complexity has two main facets: that of the system’s original design (the problem level) and further complexity at the solution level, where architects and engineers add new structures, design rules, and design constraints to ensure that, over time, products continue to satisfy customer needs and preferences. As systems evolve, complexity at the problem level remains relatively constant, but complexity at the solution level inevitably grows; added functionalities fail to match original structures
34
COMPUTER
while, in the absence of refactoring, older rules and constraints remain in place, despite the fact that they are no longer relevant. In recent years two other sources of complexity have emerged: the broad adoption of software platforms to run applications on top of common shared software and the developing concept of software ecosystems as a way of articulating the relationships among system components.3 Although both separate platforms and clean interface design theoretically provide a powerful decoupling mechanism, in practice they more often have an opposite result: previously unrelated development teams become more dependent on one another to coordinate their activities, the software artifacts these individual teams inherit become correspondingly interdependent, and interfaces become more complex as teams struggle to maintain backward compatibility. Analysts and practitioners concerned with software system complexity generally treat complexity as an unavoidable rather than a manageable system quality. This is partly because existing approaches fail to distinguish three distinct layers of system functionality: • commoditized functionality that over time has become so integral to a system it no longer adds real value; differentiating functionality that offers newer, more specialized advantages and clearly has customer value (functionality at the commoditized layer most often starts as differentiating); and • innovative and experimental functionality that is under various stages of development and thus does not currently add value but has potential to do so.
Published by the IEEE Computer Society
0018-9162/13/$31.00 © 2013 IEEE
My own research and work with others,4,5 based on first-hand observation of dozens of companies, point to a fundamental problem: companies treat these three competing layers of functionality as though they are equal, ignoring the fact that each changes at a different rate and that their intermingling contributes to unwanted system complexity. Moreover, unequal resource allocation means that commoditized functionality increasingly receives more R&D than it warrants, leading to costly delays in updating and testing differentiating functionality and in the timely release of new products. Managing complexity thus requires separating these levels of functionality. The three-layer product model (3LPM) offers a novel solution to this problem, as demonstrated by in-depth, onsite analysis of the strategies implemented by three real-world companies outlined here.
RESEARCH METHODOLOGY Empirically, the 3LPM is derived from an intensive, six-year study of three global organizations using wellestablished case study methods 6 and data collection techniques,7 including interviews, workshops, and participant observations. Direct, personal interaction with these organizations—identified here as Company A, Company B, and Company C—took place over three years as part of a broader engagement with the company and included 10 to 15 formal interviews lasting from one-and-a-half to two hours per company with staff ranging from front-line engineers and architects to R&D managers and executives; multiple workshops (four each in Company A and Company C, two in Company B); and observations of numerous meetings, both formal and informal, involving all stages of the product-development and management decisionmaking processes.
Company A Company A is a Fortune 500 company that develops software products and services primarily for personal computers in both the consumer and business markets. It releases dozens of products each year, mostly updates of existing products but also some completely new ones. Average product size ranges from millions to tens of millions of lines of code. Products commonly require highly complex components to conform to national and international regulations. Although significant opportunities exist for sharing across different business units (BUs), the company has a BU-based structure. Software products developed within each BU typically form part of a specific product line. There is no interdependency between new product development and established product teams; teams are fully local, and both team types are co-located at sites in North America and Asia. The company combines an agile teams development approach with the team software process/
personal software process (TSP/PSP) model; its goal is 50 percent development based on each model.
Company B Company B is a Fortune 100 company that builds a wide variety of embedded systems for a diverse range of markets; this study focuses mainly on the division that develops products for the global consumer market. The company’s business strategy is to offer a rich set of consumer products applying software product line principles to minimize development expenditure. Software size runs to several million lines of code. Product development teams span three continents, so careful coordination is essential.
Competing layers of functionality change at different rates, and their intermingling contributes to system complexity.
Although individual products are built from a standard platform, platform development itself is not centralized: dedicated teams own specific platform components, even as product teams in other locations can use, extend, and modify them. The company does not employ the agile development team model as such but divides task responsibility between subsystem teams, who build components, and product teams, who build products based on subsystem components.
Company C Company C is a Fortune 100 company that releases embedded products with mechanical, hardware, and software components. It develops between 30 and 50 new offerings each year using a product line approach to decrease per-product R&D expenditure; consequently, more than half of software R&D is centralized at the platform level. Software size runs between 7 and 15 million lines of code. Platform organization is distributed worldwide, with development sites in Europe, the Americas, and India. The company is transitioning to an agile teams model— currently 30 percent of development is agile—and has plans to assign full component responsibility locally. Each team leader and lead architect coordinates over geographical and architectural boundaries but is located at corporate headquarters.
COMMON PROBLEMS IN MANAGING COMPLEXITY In terms of managing system complexity, the three companies each face, to varying degrees, four common problems as summarized in Table 1.
NOVEMBER 2013
35
C OV ER F E AT U RE Table 1. Common problems in managing complexity faced by three representative companies. Problem
Company A
Company B
Company C
Proprietary/commoditized solutions not replaced by alternatives
++
++
+
Low percentage of R&D resources invested in differentiating functionality
+
+
++
Eroding architectures causing commoditized and differentiating functionality to intertwine
++
o
o
Products increasingly unattractive to customers
+
+
++
Severity and relevance for each company are rated on a scale in which “o” indicates the problem has little or no bearing, “+” indicates the problem is of some concern, and “++” indicates the problem is a major concern.
Proprietary solutions are not replaced with other available solutions. In each of the three companies, engineering personnel tend to maintain existing/historic levels of investment in commoditized software components, even though over time commercial off-the-shelf and open source alternatives have become available. Often these legacy components once provided the basis for the system’s success, and even though their functionality is no longer differentiating, investment in those elements has never been adjusted downward accordingly. Consequently, adapting products for new functionalities necessitates significant additions to code, resulting in a much higher degree of system complexity. Percentage of R&D investment in differentiating functionality shrinks. The level of resources required to maintain increasingly complex systems that have been commoditized and so no longer represent differentiating functionality means that fewer R&D resources can be devoted to enhancing system competitiveness. This effectively hobbles the productivity of staff tasked with creating differentiating functionality. With the lion’s share of resources devoted to simply maintaining “table stakes” functionality, ageing architectures inevitably become more complex, exacerbating the difficulty of achieving simplicity. Eroding architectures result as older and newer functionalities intertwine. When R&D in an organization does not distinguish between commoditized and differentiating functionalities, they tend to become intertwined. No one recognizes the need for more efficient incorporation of differentiating functionality in the future, so there is little incentive for investment in architecture refactoring8 to reduce complexity in the present. As older and newer functionalities become mixed, software architectures erode over time; any change to one component of the system results in overall structural changes that ultimately affect multiple components. Teams adding new functionality to a platform do not always recognize how its structure has eroded through accretion, and so the changes they make have unintended and unpredictable consequences. The resulting errors surface late in the new development process and are thus difficult to identify and repair. Products become increasingly unattractive to customers. As development staff spend more time maintaining
36
COMPUTER
commoditized layers of the stack, fixing bugs in differentiating layers, and preparing update releases, the product’s actual differentiating functionality becomes negligible— consequently, the product becomes less appealing to both internal and external customers.
MANAGING FOR SIMPLICITY: THE 3LPM Faced with the implications of unmanaged complexity, companies need alternative approaches to help maintain the elegance and simplicity of product architecture as it evolves. The 3LPM, shown in Figure 1, offers software developers a potent solution by separating commoditized, differentiating, and innovation and experimentation functionality into three layers. (Most companies recognize the distinction between commoditized and differentiating functionality, although for practical purposes, separating them has proved difficult. Conceptualizing and demarcating an explicit experimentation functionality layer—and its relation to the other two layers—occurs much more rarely.) The 3LPM also includes two interfaces between layers. The commoditizing transition interface between the commoditized and differentiating functionality layers links shared software assets and the components that make up the product line, with the goal of sharing functionalities in the commoditized layer among as many products as possible to achieve high R&D efficiency. The new-product transition interface between the differentiating functionality and innovation and experimentation functionality layers is akin to the platform a company offers to external developers to build applications, solutions, or features on top of the existing functionality. In practice, software companies also use this interface internally to build experimental features for customer evaluation.
Functionality layers Each of the three 3LPM layers has specific purposes and means of achieving those purposes. Commoditized functionality. This underlying layer comprises functionality necessary for system operation but that customers take for granted. It often combines proprietary software, built in-house over time, with commercial and open source software solutions. The development team associated with this layer should focus on
Productize
Ecosystem partners
Challenges • Over time, products lose competitiveness • Platform becomes competitive disadvantage
New-product transition interface
Differentiating functionality layer (optimize for maximum customer value) Commoditize
Architecture refactoring process
Innovation and experimentation layer (optimize for maximum number of experiments)
Commoditizing transition interface
Commoditized functionality layer (optimize for minimizing total cost of ownership)
Characteristics • Each layer releases independently • Each layer optimizes different metrics • R&D efforts focus on highly differentiating functionality
Figure 1. Three-layer product model. The 3LPM organizes product architecture into a commoditized functionality layer, a differentiating functionality layer, and an innovation and experimentation functionality layer, with two interfaces.
minimizing the total cost of ownership, constantly analyzing both existing and new functionality to provide cheaper and more efficient operation by • incorporating as much functionality from the differentiating layer into the commoditized layer as feasible; • replacing proprietary software with commercial software components when possible; • replacing commercially licensed software components with available open source alternatives; and • incorporating new functionality that is necessary, but not differentiating from the customers’ perspective, so that it is available to users system-wide at the same time. Differentiating functionality. The middle layer of the stack comprises functionality that differentiates the product from its competitors and also appeals to customers. A product’s market success or failure results from the functionality of this layer. The development team responsible for the differentiating functionality layer should continually add new value to the system by • identifying and incorporating new functionality as it develops in the innovation and experimentation functionality layer; • recognizing functionality that has commoditized and appropriately transitioning it to the lower layer; and • developing and implementing new, differentiating functionality in response to customer demand or changes in business strategy.
Innovation and experimentation functionality. The focus of this layer is to develop future functionality, either in collaboration with external customers or internally, that can provide significant value and ultimately differentiate a system or product from its competitors. Such activities could involve creating new product features requested by a lead customer, conducting trials for extending existing products into new markets, or finding ways to improve already implemented functionalities. The innovation and experimentation development team should be responsible for • hypothesizing future differentiating functionality through customer interaction and analysis of competing products, as well as brainstorming and other ideation techniques; • identifying pertinent new or evolving technologies; • experimenting with likely candidates for innovation to evaluate their customer relevance and marketability and the feasibility of their implementation within the system; and • imagining forward-thinking innovations not necessarily relevant for a specific product but perhaps suitable for deployment within an entire software ecosystem with the potential for significant returns.
Commoditizing and new-product transition interfaces The purpose of the two 3LPM interfaces is to minimize complexity in the transition of functionality among the three layers.
NOVEMBER 2013
37
C OV ER F E AT U RE Table 2. 3LPM implementation strategies by three representative companies. Strategy
Explicit innovation and experimentation functionality layer
Company A
Company B
Company C
++
+
o
Separate commoditized and differentiating functionality layers
+
o
++
Architecture refactoring efforts
+
+
+
New-product transition interface
o
++
++
Commoditizing transition interface Discrete management of three layers Specified metrics for each layer
o
+
++
++
o
++
+
o
+
++: institutionalized practice; +: ad hoc practice; o: practice not implemented
Commoditizing transition interface. This interface allows products and functionalities no longer relevant or differentiating in themselves to move from the differentiating layer to the commoditized layer. This simplifies the evolution of products that remain in the differentiating layer by decoupling them from functionalities in the commoditized layer, encouraging more efficient and nimble development in the middle level, as well as facilitating more “trickle down” innovation from the upper level. Once differentiating solutions have transitioned to the commoditized level, these once proprietary functionalities can more easily be replaced with suitable commercial or open source solutions— another gain for simplicity. New-product transition interface. This interface allows functionality to move from the innovation and experimentation level to the differentiating layer. Once new features and products have been tested and refined in the upper layer, either in-house or by software ecosystem partners, and are ready to move out of the innovation and experimentation stage, they are ready for market distribution or widespread use within an organization. Ideally, this transition should occur with minimal customer disruption, usually by allowing the new product or version to coexist with one already offered within the differentiating functionality layer. Customers have a chance to get accustomed to the new functionality, while still having access to the original, until the new product’s stability and scalability are firmly established.
Architecture refactoring In addition to facilitating the flow of functionality from the innovation and experimentation layer to the differentiating layer and from the differentiating layer to the commoditized layer, the 3LPM requires a third process to ensure that a system’s structure does not deteriorate over time: proactive refactoring of the system’s architecture.8 This requires reorganizing code layers and structures to minimize the cost of evolution over time, manage interfaces, and maintain end-to-end quality requirements.
38
COMPUTER
THE 3LPM IN PRACTICE With the goal of finding solutions to the four common problems of managing complexity summarized in Table 1, the three companies described earlier each adopted practices defined by the 3LPM to a greater or lesser degree, depending on corporate needs, strategies, and culture. Table 2 charts this implementation scheme across the companies. As the table suggests, implementing new strategies in some cases became institutionalized (that is, formally defined as an operating mechanism practiced by all members of the organization) and in others remained ad hoc (that is, not established as an explicit policy but still recognized implicitly and practiced by members throughout the organization).
W
hat holds true for these three companies, and others studied for this analysis, suggests applications for the industry more broadly. As systems evolve, commoditized, differentiating, and innovative and experimentatal functionalities—which necessarily change at different rates—become intermixed, leading to increased complexity, decreased efficiency, and the stifling of product innovation in response to changing markets and customer needs. The goal, therefore, should be to make a clear distinction between the layers and thus to allocate resources appropriately to encourage development at the upper layer. The 3LPM can provide a blueprint for increased simplicity, resulting in higher productivity, enhanced product quality, and a longer life for systems overall. Although its implementation, whether completely or only in part, poses challenges—as does any process of change—companies should recognize that elements of the model already exist internally, often informally or in an ad hoc fashion. Adopting a more structured, explicitly managed approach can significantly benefit an organization’s profitability and continued viability in the market.
References 1. D.L. Parnas, “Software Aging,” Proc. 16th Int’l Conf. Software Eng. (ICSE 16), IEEE CS, 1994, pp. 279-287.
2. D.E. Perry and A.L. Wolf, “Foundations for the Study of Software Architecture,” ACM SIGSOFT Software Eng. Notes, vol. 17, no. 4, 1992, pp. 40-52. 3. J. Bosch, “From Software Product Lines to Software Ecosystems,” Proc. 13th Int’l Software Product Line Conf. (SPLC 09), ACM, 2009, pp. 111-119. 4. P. Bengtsson et al., “Architecture-Level Modifiability Analysis (ALMA),” J. Systems and Software, vol. 69, nos. 1-2, 2004, pp. 129-147. 5. J. Bosch, Design and Use of Software Architectures: Adopting and Evolving a Product Line Approach, Pearson Educational/Addison-Wesley, 2000. 6. K. M. Eisenhardt, “Building Theories from Case Study Research,” Academy of Management Rev., vol. 14, no. 4, 1989, pp. 532-550. 7. P. Reason and H. Bradbury-Huan, eds., SAGE Handbook of Action Research, 2nd ed., Sage, 2007.
8. M. Fowler et al., Refactoring: Improving the Design of Existing Code, Addison-Wesley, 1999.
Jan Bosch is a professor of software engineering and director of the Software Center at Chalmers University of Technology, Sweden. He also consults on issues of innovation and R&D efficiency for many companies worldwide. His research interests include software architecture assessment, design, and representation; software product lines, including variability management, organizational approaches, and product family architecture design; design erosion; and component-oriented software engineering. Bosch received a PhD in computer science from Lund University, Sweden. Contact him at
[email protected]. Selected CS articles and columns are available for free at http://ComputingNow.computer.org.
CONFERENCES
in the Palm of Your Hand
IEEE Computer Society’s Conference Publishing Services (CPS) is now offering conference program mobile apps! Let your attendees have their conference schedule, conference information, and paper listings in the palm of their hands. The conference program mobile app works for Android devices, iPhone, iPad, and the Kindle Fire.
For more information please contact
[email protected]
NOVEMBER 2013
39
