Automating Speculative Queries through Event-based Requirements Traceability Jane Cleland-Huang*, Carl K. Chang+, Gaurav Sethi, Kumar Javvaji, Haijian Hu, Jinchun Xia+ International Center for Software Engineering, University of Illinois at Chicago * Now at DePaul University, +Now at Iowa State University {
[email protected],
[email protected]} Abstract Posing speculative questions about a software system is an important yet often unsupported activity. Current impact analysis techniques tend to focus upon the functionality of the system, whilst the effects of change upon performance requirements are largely ignored until after implementation. This tendency can lead to costly and time-consuming mistakes. Event-Based traceability provides a robust method for handling both long-term evolutionary change as well as the short-term speculative change needed to support performance related impact analysis. By establishing dynamic links, capable of propagating data values and commands between requirements and performance models, it becomes possible to automate a wide range of speculative queries and to enhance the overall ability to predict the impact of change upon the performance of the system.
1. Introduction In today’s constantly changing environment, studies have shown that 40-90% of the cost of developing software is incurred following implementation, indicating that change is a key aspect in any long-term software project. The change process itself is managed through impact analysis techniques that identify the parts of the software system that will be affected by a change, and by estimating the cost and effort required to make those changes [1]. Unfortunately, most of the effort of impact analysis has focused on the functional aspects of change whilst non-functional performance issues are largely ignored until after implementation. This tendency can have a detrimental effect on the quality of the software system. Software engineers continually struggle with this issue and often introduce functional changes to a system without adequately considering how those changes will affect its performance. This situation was graphically illustrated through a story recounted firsthand to the authors concerning a leading telecommunication corporation’s experience in an
Asian country. The government introduced a new regulation that replaced the existing dialing pattern with 5-digit carrier access codes and instituted a new international dialing prefix. As this impacted the functionality of the mobile switching center the project leaders focused upon the functional changes that were needed and paid little attention to performance impacts. The enhancement worked as expected during internal unit and feature testing but when it was executed in the customer test bed several key processors crashed resulting in much time and effort spent resolving the problem and a decline in customer satisfaction. A postmortem analysis determined that future product enhancements should consider performance impacts on both hardware and software. Unfortunately the current software environment did not provide the means to link requirements to related performance indicators. This paper addresses the problem by proposing a method for establishing and utilizing traceability links between requirements and performance models. The approach not only identifies where relationships exist between requirements and performance models, but also supports the process of analyzing the impact of a proposed change upon the performance of the system through dynamic re-execution of requirement dependent models. Section 2 of this paper categorizes change according to the ways in which it impacts performance and proposes a method for performance related impact analysis. Section 3 briefly discusses current traceability practices and introduces an event-based traceability (EBT) method capable of supporting the proposed analysis. Sections 4 and 5 then demonstrate EBT’s support for performance related impact analysis, and section 6 provides additional examples of performance models that could be integrated into the proposed scheme. Section 7 describes the prototype we developed and gives an example of a speculative query and the results obtained by executing dependent performance models. Finally, section 8 concludes with a discussion of the approach and some suggestions for future work.
Proceedings of the IEEE Joint International Conference on Requirements Engineering (RE’02) 1090-705X/02 $17.00 © 2002 IEEE
2. Change The changes affecting a system can be categorized either as contextual or functional. In contextual change, the basic functionality of the system remains the same, but some quantitative value in the system is altered. For example, the number of users is doubled, a different workload pattern is anticipated, or a key piece of equipment is replaced with a unit that works at a different rate. When contextual changes occur, impact analysis must determine how well the system will perform within the new context. Contextual change is significant because in today’s economy many customers choose to accommodate new requirements through stretching existing systems rather than through developing new ones. Software engineers therefore need the ability to assess the degree to which a contextual change will affect the performance of a system. In functional change, the behavior of the system is changed in a significant way, such as the introduction of a new set of features, or modification of existing features. The performance impact of functional change is more difficult to predict than that of contextual change because it involves redesign followed by impact analysis to determine how the changes will affect both the functionality and the performance of the system. Although there has traditionally been a tendency within the Object-Oriented community to defer performance related issues until after implementation, recently emphasis has been placed on performance engineering throughout the development process in order to control the risks inherent to constructing large systems [2,3,4,5]. Smith proposed several performanceengineering techniques such as the use of execution graphs to determine whether critical scenario paths meet predefined response times. Dimitrov et al demonstrated how UML could be extended to model performance related characteristics such as timing, probabilities, and transfer rates, whilst Carriere et al proposed that multiple models should be constructed to represent the various high-risk attributes of a system. Applying performanceengineering techniques in these ways produces a varied set of performance models, each of which represents a key characteristic of the system. Just as importantly as constructing performance related characteristics into a system during its initial development, is the need to consider those characteristics during its maintenance phase. This is possible if the relevant models can be identified when a change is proposed and re-executed in a way that reflects those changes. Ideally, when changes are proposed, the impacted performance models should be identified and automatically re-executed using values representative of the proposed changes. To achieve this, traceability links
are established between performance models and the requirements specification through the use of a dynamic traceability scheme capable of speculatively driving the impacted models whenever a quantitative requirement is changed. Key values from within the individual performance models representing probabilities, rates, counts, and sizes etc are placed in the central requirements repository. Finely tuned links are then established between the data-values in the models and those in the repository. Performance related impact analysis is accomplished through changing the values of certain requirements in the central repository, propagating those changed values into related performance models, and observing the results.
3. Event-based traceability Traceability links define the relationships that exist between requirements and other software engineering entities [6,7,8]. To effectively support performance related impact analysis, traceability links must be strongly typed [9] and must be maintained in an accurate and current state throughout the system’s lifetime. The links should also be capable of propagating speculative values from requirements right into the heart of their related performance models [5], and of triggering the reexecution of those models. Most current traceability schemes use matrices, spreadsheets, graphs, or requirements management systems such as DOORS or Rational Requisite Pro [10], which do not inherently support this type of behavior. EBT, which is based upon the event-notifier design pattern [11] offers a solution to some of the difficulties inherent to maintaining accurate traceability by establishing loosely coupled traceability links through the use of publish-subscribe relationships between dependent objects [12]. This type of traceability scheme is designed to handle change robustly, and is therefore supportive of both long-term evolutionary change as well as the shortterm speculative change needed to drive performance related impact analysis. As depicted in Figure 1, objects use an event server to subscribe to the requirements on which they are dependent, and when a significant change event, such as a requirement merge, replacement, refinement, or abandonment occurs involving one of those requirements, an event notification message is published. The message is received by the event server and forwarded to all dependent objects. A subscriber manager handles all incoming event notifications for a set of similarly typed objects and follows predefined protocols to deal with specific types of event messages for that object. In some cases, the event manager may automatically update the managed object, whilst in other cases the event message must be stored in an event log to await manual intervention. In addition to solving many
Proceedings of the IEEE Joint International Conference on Requirements Engineering (RE’02) 1090-705X/02 $17.00 © 2002 IEEE
Event Server
Subscription / Events
Requirements Manager
Subscription Manager
Event Retriever (Pull Model)
Event Listener / Notifier
Event Listener (Push (Push Model) Model)
Event Trigger
Event Resolution Handler
Require ments
Local Transaction Log
Concrete Subscriber
Reqs API
Subscriber Manager
Figure 1. Event based traceability framework. problems related to evolutionary change, EBT also provides the mechanism needed to support the performance related impact analysis described in this paper. The complete EBT scheme is fully described in [12]. A tool survey was conducted to determine which RE tools support this type of traceability. SLATE [10], which was initially designed from a systems point of view, integrates requirements and design models, and supports many of the features described in this paper. It includes a requirements management tool, a scheduling engine that handles events and actions, and a simulation tool that utilizes the TCL scripting language and interfaces with a wide variety of simulation models. The SLATE engine however requires the use of its own proprietary requirements management system. In contrast, the EBT approach proposed in this paper can be integrated into almost any requirements management system, by utilizing published APIs and other features. For example, DOORS has a trigger feature that can generate event notifications to the external EBT event server, and an API that supports call- backs for further information. Other tool vendors such as RequisitePro do not support triggers, but if speculative events are initially triggered using an add-on tool, their APIs support the remainder of the process. Further vendor related information about implementing these techniques in specific RE tools is available from our SABRE Consortium website [13].
4. Support for speculative change In EBT, the requirements (or other change instigators) are seen as publishers of events, and dependent objects as the subscribers. One or more values in the extended requirements repository are speculatively changed and wrapped into a single speculate message that is sent to the event server. The event server queries its database to identify all entities that are subscribed to the changed requirements, filters the selected entities to identify only performance models that are compatible with the current speculative event, and forwards the message to their subscriber managers. To support performance related impact analysis, the EBT scheme is extended by enhancing the interface of the performance related subscriber managers. On receipt of a speculate message, the manager extracts data values from the message, injects them temporarily into the performance model, re-executes the model, reports back the results, and restores the model to its original state. As an example, consider the execution graph depicted in Figure 2, which models the activity of broadcasting and redrawing the components of a webbased whiteboard when a refresh request is issued [2]. As the graph illustrates, all of the shapes are broadcast first, followed by all of the text blocks. As it takes longer to broadcast an object than to display it, all of the objects are displayed during the broadcast of the next object. Only the display of the final text block, which does not occur
Proceedings of the IEEE Joint International Conference on Requirements Engineering (RE’02) 1090-705X/02 $17.00 © 2002 IEEE
concurrently with any broadcast and therefore contributes to the overall response time, is shown on the graph. The execution of the graph is driven by the values AvgShapeSize, Bandwidth, NoShapes, AvgTextSize, NoTextItems, WriteTime, setup, and pulldown. An additional variable TargetResponseTime represents the target time in which the scenario should be able to complete execution. To drive the model from requirements, any of these variables that might change are linked to a value in requirements. For example, AvgShapeSize links to requirement WB12, and Bandwidth to WB13. These related requirements are listed in Figure 3.
Setup AvgShapeSize Bandwidth Pulldown
AvgTextSize
If a proposal were made to extend the whiteboard to support clipart shapes in addition to standard geometric shapes, it would be necessary to first predict how this change would affect the overall performance of the whiteboard. Obviously, the clipart feature would enhance the expressiveness of the whiteboard, but the pertinent question would be whether the performance related requirements WB19 and WB20 could still be met. A speculative query should be posed to answer this question. Through observation, WB12 is identified as the primary requirement impacted by the proposed change. If it were determined that by introducing clipart the average shape size would increase from 30Bytes to 2K and that requirements WB19 and WB20 might be relaxed to 12
Time to Broadcast = setup + (AvgShapeSize / Bandwidth ) + pulldown.
NoShapes
Setup
4.1 Change proposal
Broadcast shape
Bandwidth
Broadcast text
Pulldown
Time to Broadcast = setup + (AvgTextSize / Bandwidth ) + pulldown.
NoTextItems
WriteTime
Write final text
Time to write text = WriteTime Target Response Time
Figure 2. Execution graph representing the response time to refresh a whiteboard.
Figure 3. Performance related requirements. Proceedings of the IEEE Joint International Conference on Requirements Engineering (RE’02) 1090-705X/02 $17.00 © 2002 IEEE
and 3 seconds respectively, then the following speculative query would be issued: “Speculate | 3 | WB12 | The average size of a shape is 25 Bytes | Value, 25, 2000 | WB19 | A screen request from a meeting member shall result in an average screen being redrawn within 10 seconds | Value, 10,12 | WB20 | When the floorholder adds text or a shape to the whiteboard, all member’s screens shall be updated within 2 seconds | Value, 2,3 |” Following the EBT protocol, illustrated in Figure 4, the speculate event is published to the event service, which then queries its internal database to identify subscribers. To support data injection into the performance model, the traceability link must be a “deep link” extending from the data item in the subscriber to the specific data value in the requirement. The event message is therefore customized for each subscriber according to its subscription by filtering out non-relevant parts of the message, and annotating each change with the linked variable. The customized message sent to the “refresh” scenario shown in Figure 2 is: “Speculate | 2 | WB12 | The average size of a shape is 25 Bytes | Value, 25, 2000 | AvgShapeSize | WB19 | A screen request from a meeting member shall result in an average screen being redrawn within 10 seconds | Value, 10, 12 | TargetResponseTime |” The message is received by the scenario’s manager, which temporarily replaces the existing values of 25 and 10 in variables AvgShapeSize and TargetResponseTime
respectively. The model is then re-executed using these values and results are reported back to the event server.
5. Context vs. Functional Change Despite the fact that the introduction of the clipart shapes to the whiteboard would in fact require some changes to the whiteboard’s functionality, the proposed change was treated primarily as a contextual change in which risk was assessed through changing only quantitative values in existing requirements. Abstracting change to a contextual level provides a useful predictive tool without incurring high costs in time and effort. Unfortunately, not every proposed change can be represented in this way. Functional changes that require creation of new classes or modules, or change the pattern of interactions between existing components provide a greater challenge to the impact analysis of the system’s performance, and existing performance models may be insufficient for representing the proposed changes. As illustrated by the story of the telecommunications system in section 1, the typical approach to functional change is to implement new functionality without considering its effect on performance. However, functional changes do affect performance, and so performance related impact analysis must be integrated into the more traditional functional impact analysis. Many traceability schemes provide little support for identifying the impact of entirely new requirements. Performance Model
WB19 A screen request .... shall result in an avg screen redrawn in 10 secs
(1)
(6 te ula ec Sp
Other Managers Other Managers Other Managers
Performance Model Performance Model
) cs se
(2) Event Service
By te s)
WB20 When text or Speculate shapes are added, (3 secs) all screens shall be (1) updated within 2 secs
(4) (5)
Execution Graph Manager
(3)
(2 ,0 00
Sp ec ul at e
WB12 The average size of a shape is 25Bytes
(5)
Subscription Data
(1)
(3)
Whiteboard Execution Graph AvgShapeSize
AvgTextSize
DrawTime
WriteTime
NoShapes
NoTextItems
Transaction Log
Bandwidth TargetResponse
Events (1) (2) (3) (4) (5)
Publish the composite speculative event specifying that WB19 = 6 secs, WB18 = 3 secs, and WB12 = 2K Event service identifies dependent objects, and forwards the speculate message to the relevant subscriber manager(s). In this case the message is forwarded to the Execution graph manager plus "other managers and their related performance models for which subscriptions were placed. The subscriber manager injects changed values as data items into relevant performance model(s). The subscriber manager sends an "Execute" command. The Execution graph and all other subscribed models execute and returns results.
Figure 4. Speculate event to determine if the whiteboard can be refreshed in less than 3 seconds. Proceedings of the IEEE Joint International Conference on Requirements Engineering (RE’02) 1090-705X/02 $17.00 © 2002 IEEE
Fortunately in EBT, entities can place subscriptions according to categories of requirements in which they might be interested. This subscription scheme supports dependencies on existing requirements as well as those that have not yet been created. For example, the Whiteboard execution graph depicted in Figure 2, might have placed subscriptions for requirements WB12,13,14,15,21,22,23, and 19 shown in Figure 3, as well as for a more generic subscription based upon the thesaurus-enhanced keywords “Redraw” and “Whiteboard”. If a new requirement related to redrawing the whiteboard were introduced, the execution graph would be automatically established as a subscriber. Impact analysis is therefore performed to identify the dependent entities, and to determine the cost and effort of introducing the change. In addition, existing performance models are identified and additional ones are constructed as necessary in order to model each of the major risks associated with the functional change. A checklist of attributes such as response time, throughput, security, portability, maintainability, cross-referenced against standard modeling techniques used for each attribute should be used to support the process of risk identification. Existing models must be adjusted as necessary to accommodate the proposed change, and then stored as variants of the original model. New models may be created to represent previously un-modeled risk factors. Links are established from all new and variant models to the new requirements, and a speculate event is executed to determine how performance attributes of the system will be affected by the proposed requirement change. If the proposed change is not implemented, the proposed requirements can be moved to a special folder and all related performance models and variant models would remain linked to the non-implemented requirements. This
Floor Neither Held Nor Requested Prob Assign Floor
Floor released by user
Floor removed by chair
Prob Floor Released
ProbRequestFloor
Request floor
Remove request for floor
Floor assigned by chair
Prob Remove Request
Prob Floor Removed
Floor Requested Floor granted by chair
Floor Held TimeToWrite * NoOfObjects
Figure 5. Floor control represented as a state transition diagram.
increases the pool of performance models available for testing future change proposals.
6. Examples Performance related impact analysis can be performed on a wide variety of models. All that is needed is an executable model and an API that supports injection of values into the model and the ability to trigger the execution of the model. In this section two additional examples of simple models are provided to demonstrate the generality of the approach to multiple modeling techniques and environments. The second example is deliberately taken from an entirely different engineering environment in order to demonstrate the breadth of this approach.
6.1 UML Models Dimitrov et al provided numerous examples of how existing UML diagrams can be augmented to represent performance related attributes such as timing and execution rates [3]. However, in order to support speculation, the UML modeling environment must be executable and must provide some output that can be compared to a specified performance requirement. Examples of performance enhanced UML models include a load- and time-weighted use case diagram in which actors represent the system’s workload, interaction diagrams in which an explicit time axis is introduced, implementation diagrams that specify resource requirements, and state and activity diagrams that relate the events and states within a class. In each of these cases, constant values that define times, probabilities, and resource usage can be replaced with variables and linked to requirements. As an example, consider the state diagram, depicted in Figure 5 that models the behavior of meeting members requesting the “floor” to gain access to the whiteboard. In this example there are three states representing FloorHeld, FloorRequested, and FloorNeitherHeldNorRequested. Transitions are represented as probability variables, which enables close integration with Markov modeling. A delay is introduced into the FloorHeld state so that once a person enters the state they will remain in that state for a time period equal to TimeToWrite * NoOfObjects, unless the floor is prematurely removed by the chair. These transitions are all represented as variables and tied into a set of related performance requirements that can be used to support speculative queries. An example of a speculative query might be to pose the question of what would happen to the waiting queue length if an arbitrary time limit were placed on the use of the whiteboard. Although this is a simple example, state transition diagrams can be used to solve much less
Proceedings of the IEEE Joint International Conference on Requirements Engineering (RE’02) 1090-705X/02 $17.00 © 2002 IEEE
trivial problems related to the performance of queueing models.
6.2 Hybrid Automaton A hybrid automaton is a useful modeling tool for representing the behavior of an embedded system in which specific transitions from one state to another occur discretely and deterministically and the evolution of values within a given state can be prescribed using differential equations. Using an example introduced by Henzinger et al [14] and shown in Figure 6, a hybrid automaton represents the behavior of a heater in a manufacturing plant. The temperature of the heater must be maintained between 1 and 3 degrees. When the heater is turned on, the temperature rises at a rate of –x + 5 degrees and when the heater is turned off the temperature drops at a rate of –x. The heater is turned on when x = 3, and turned off when x = 1. The automaton is used in several tools such as Hytech to perform reachability analysis to ensure that unsafe states are not reachable and to support parametric evaluation. By replacing the constant values of the original model with variables Temp, Temp_Rise, Temp_Fall, Turn_On, and Turn_Off, and by linking those variables to requirements it is possible to speculatively manipulate the model through changing values in the requirements. A variety of speculative queries could be posed such as “What would be the effect of replacing this heater with a different type of heater that heats at a different rate?” or “Is the same heater capable of maintaining the tank between 2 and 4 degrees?”
6.3 Other performance models Numerous other types of performance models can be linked to requirements in a similar manner. Remote terminal emulators that are used frequently to simulate workloads contain values that could be replaced with linked variables. Scenario sequence diagrams used to predict dataflow in distributed applications can likewise be linked to requirements. The biggest limitation is the need for models to provide an API that supports injection of changed values into the variables, to be executable, and to output some type of reportable result as opposed to observable behavior.
7. Prototype and related case studies The existing EBT prototype was extended to support performance related impact analysis. The prototype consisted of three major components – a requirements based tool to support the triggering of speculative queries, an event server, and a subscriber manager. The
on
x=2
.x = -x + 5
1
