Safety-Focused Security Requirements Elicitation for Medical Device Software Mikael Lindvall1, Madeline Diep1, Michele Klein1 1
Fraunhofer CESE College Park, MD, USA {mlindvall, mdiep, mklein}@fc-md.umd.edu
Paul Jones2, Yi Zhang2
2 U.S. Food and Drug Administration Silver Spring, MD, USA {paul.jones, yi.zhang2}@fda.hhs.gov
Abstract— Security attacks on medical devices have been shown to have potential safety concerns. Because of this, stakeholders (device makers, regulators, users, etc.) have increasing interest in enhancing security in medical devices. An effective means to approach this objective is to integrate systematic security requirements elicitation and analysis into the design and evaluation of medical device software. This paper extends the sequence-based enumeration approach, a systematic approach for defining the behavior of embedded software, to analyze the requirement documents of a medical device for the purpose of eliciting security requirements. As a proof of concept, we apply our approach on a concrete case study, which shows that the extended approach is useful for identifying sequences of medical device events that might be harmful to the patient, for example because the events are initiated by an active adversary trying to use the device in a malicious way. We then show how security requirements may be formulated based on the identified threats. By exploring these sequences systematically, the developers can reliably assess what, where, and how the security threats may manifest in their system, what the safety implications are, and finally they can evaluate the resulting requirements and mitigations. Index Terms— Medical device safety and security, requirements elicitation, sequence based enumeration.
I. INTRODUCTION As we move into a world with increasingly connected intelligent devices (such as mobile and cloud interfaces), the degree of connectivity among medical devices and their supporting infrastructures is also rapidly increasing. This increasing connectivity opens up opportunities for innovative therapies, treatments, and diagnostics; provides convenience to users (patients and caregivers); and reduces overall health care costs due to integrations and non-duplication of effort. However, it also introduces new security challenges [1, 2, 3]. Interfaces that allow for bi-directional flow of control and data between devices and infrastructure (e.g. remote controls, electronic health records, interoperable middleware, cloud and mobile infrastructures) lead to new security vulnerabilities that previously did not exist in stand-alone, non-networked devices. What makes medical devices particularly challenging is that unlike traditional information technology (IT) systems, they are almost exclusively safety-critical systems [4, 5]. Connected to (and sometimes implanted in) human beings, the consequence of illicit control of these devices can be severe, sometimes fatal, and often irremediable. Post-facto detection and remedia-
Eugene Vasserman3 3 Kansas State University Manhattan, KS, USA
[email protected]
tion, as is often the standard practice in IT security breaches, is often not possible, since lives cannot be restored and injuries may cripple the patient while they heal. The ability to directly affect human safety not only provides greater motivations for malicious agents to attack medical device systems, but also makes several traditional IT security countermeasures infeasible. Mitigating controls cannot be passive, and must actively block detected attacks before they cause any harmful effect. Additionally, patient data is valuable on the black market, orders of magnitude higher than credit-card numbers, which banks can easily invalidate in case they are compromised (unlike patient data), further increasing the attractiveness of medical devices for malicious agents [7]. The need for “designed-in security” thus becomes even more imperative to ensure that security controls are properly implemented [6]. Medical devices must perform safely and effectively [13, 14], and assuring acceptable security properties in them is an increasingly crucial part of it. Incorporating security from the very beginning of the design process starts from understanding the threats or potential attacks that the devices face, the implication of such attacks, and defining appropriate responses and mitigations for the attacks, which then are turned into security requirements. Other approaches for eliciting security requirements, such as threat modeling[18] and misuse cases [17], have been widely used in traditional IT system contexts. These approaches have been shown useful, but a more systematic approach that focuses on the safety implication is needed for addressing the criticality required in medical devices. We look instead at the application of the Sequence-Based Enumeration (SBE) approach, a systematic approach for defining complete system behavior commonly used in formal software development approaches [13], and previously used for developing medical device requirements [12, 15]. SBE is typically used to specify “nominal” functional behavior of a system under design. It is supported by the RealSBS tool to perform its requirement elicitation and analysis steps [10]. In this paper, we describe how we extended the SBE approach and its supporting tooling to enable the analyst to identify what possible malicious behaviors (as captured in the form of off-nominal behavior) are detected andtheir safety impact, and to specify what mechanism is needed to detect, avoid, or mitigate the danger to patients. We also report on a
concrete use-case – the application of this extended approach to a medical device to assess its safety against attacks. Our approach is focused on the first aspect of the requirement elicitation process, i.e., understanding the security needs of the system by enumerating the threats or potential attacks that the devices face, and defining appropriate responses and mitigations. The two main contributions are: 1. Aiding device developers/analysts to better qualify the safety implications their system faces given the identified security threats. 2. Relatedly, assisting the developers to identify which components in their system have the most safety risk due to security threats. This information enables developers to identify where security mechanisms are needed, and to prioritize their attention depending on the components which have the highest safety consequences. II. SEQUENCE-BASED ENUMERATION Given the potentially significant safety impact of security vulnerabilities in medical devices, the U.S. Food and Drug Administration (FDA) encourages manufacturers to implement mechanisms (e.g., detect, respond, and recover) in their devices to protect them from security attacks [3]. This is only possible if developers perform comprehensive identification of security threats faced by the devices, as well as systematic elicitation and analysis of security requirements to prescribe the security properties needed for safety. Techniques (and supporting tools) for comprehensive requirement elicitation and analysis for developing software in safetycritical systems such as medical devices exist, especially in the context of formal methods. One example is the SBE method as used in the Cleanroom Software Engineering (CRSE) process [13, 14]. An example device developed using CRSE is CARA (Computer Assisted Resuscitation Algorithm), which is a control-loop software system that infuses resuscitation fluids to maintain a patient’s blood pressure within safe levels [12]. The CRSE approach, like any formal software development approach, is intended to generate highly reliable products. The SBE method integrated in the CRSE process, developed by Prowell and Swain [10] and further defined in [11], allows developers to enumerate all possible behaviors of a system and systematically produces requirements that are complete and disjoint for that system [10]. “All possible behaviors” refers to those behaviors that are within the parameters specified by the developer. SBE can be applied from scratch (e.g., no requirements are yet defined), but can also be applied to an existing set of requirements to asses their completeness and to identify the missing requirements.. SBE is especially useful for highly scenario-specific embedded systems that make decisions based on events, such as the control systems in medical devices. Leveraging SBE for eliciting security requirements offers developers a formal way to investigate security threats faced by the devices, and craft mitigating security requirements accordingly. Therefore, SBE has the potential to improve the correctness, completeness, and consistency of safety requirements.
Below we discuss the SBE method and the RealSBS tool, and how we extended them to assist with security requirement elicitation. A. The SBE method The followings are the steps for performing SBE: Establish the system’s boundary by defining all of the system’s interfaces. Note that interfaces represent any actor that interacts with the system, through either stimuli (i.e input events) or responses (i.e. output). Interfaces could be human, software, or hardware. Enumerate all of the system’s stimuli. Stimuli are any signals or inputs that the system receives from the interfaces. The system is modeled as a reactive system that only acts if there is a stimulus. Active components (ones that can generate stimuli, such as timers) are treated as “external” system components and modeled as interfaces. A stimulus can be coupled to a boolean condition called predicate that allows the modeler to create conditional expressions such as if event A occurs and the temperature is above critical then…. else…. An action is a combination of a stimulus and its predicate. For the remainder of this paper, the use of the words “action” and “stimulus” will be interchangeable. Enumerate all of the system’s responses. These are the outputs that the system generates for the external components in response to stimuli or sequences of stimuli. Not every stimulus or sequence of stimuli has a response. Enumerate all possible sequences of actions and/or stimuli, and assign responses. A sequence can be legal or illegal. A sequence is illegal when it cannot occur, for example an alarm cannot go off when system power has been turned off. Otherwise, the sequence is legal. A legal sequence can be equivalent to another sequence. The enumeration starts with sequences of length 1. Only legal sequences that are not equivalent to other previously defined sequences carry on. Based on the legal and unique sequences, the enumeration continues by systematically enumerating all possible sequences of length 2 and so on. It ends when no unique legal sequences are enumerated. Link each sequence and its corresponding response to the system’s requirements. Generate a set of canonical sequences (i.e., unique and legal sequences). Each canonical sequence corresponds to a unique state of the system. Assign state variables to the canonical sequences so that one unique combination of the state variables corresponds to one unique system state (i.e. canonical sequence). The system is completely specified when all state variable combinations are covered and all canon-
ical sequences have been linked to a certain state variable combination. Note that with the SBE approach, it is possible that state explosion can occur, for example with a system that has a large number of stimuli and/or system variables. The SBE approach partially addresses this issue by using the notion of “equivalence” between sequences. The state explosion can be further mitigated by abstracting the stimuli and/or system’s variables. For example, in a system that interfaces with a keypad to provide numerical inputs, the stimuli can be abstracted by considering only the length of inputs or whether the inputs represent a valid or invalid passcode, instead of the actual input values. In the case of CARA, initially 25 stimuli were identified. Abstraction reduced the number of stimuli to 16. In addition,
there were 44 responses. The sequencing activity was complete at sequences of length 8, resulting in a total of 2992 sequences. Abstracting the stimulus or variable values is non-trivial and requires domain experience. B. The RealSBS tool The RealSBS tool [10], developed in Java using the Eclipse Modeling Framework, embodies the SBE method, and supports the user in carrying out the tasks described above. The tool provides automation support when enumerating all the possible sequences. Figure 1 shows a screenshot of the RealSBS being used to establish mappings between sequences, responses, and requirements.
Fig. 1. The GUI for the RealSBS tool, showing the mappings between the sequences, the requirements, the responses, and other information.
After the user enters all possible stimuli, the tool systematically and iteratively generates sequences of stimuli of increasing length, starting with the length of one. The user will then identify and mark each of the generated sequences as either legal or illegal. When a sequence is identified as a legal sequence, the user will then identify if it is equivalent to another legal sequence. A sequence is equivalent to another sequence if both sequences reach the same system state, e.g., tripping an alarm when the alarm is already going off. When a sequence of length n is identified as legal and unique, the tool will automatically generate sequences of length n+1 by appending each possible stimulus to the original sequence. Consequently, if a legal sequence is marked as equivalent to another, the tool will not generate a longer sequence
from it. The sequence generation process is repeated until no more sequences can be generated. In Figure 2, we show an example of the sequence enumeration (displaying some, but not all, sequences up to 3) for a simple alarm system that takes input from a sensor (e.g., temperature, motion, etc.). The alarm will go off when the input it receives from the sensor exceeds a threshold or falls outside a range (e.g., high temperature, moving objects instead of stationary objects). Initially the alarm generates only visual warnings, but if the alarm persists for a certain period of time it will turn into an audible alarm. A silent alarm can be turned off. An audio alarm can be turned off or silenced. Dotted lines in Figure 2 represent illegal sequences while the solid lines represent legal ones. “Lambda” is the toolprovided name of the starting state of the system. There are five stimuli for this sample system: PushClear is the action for
clearing the alarm, PushSilence is the action for silencing the alarm, TimerExpired is the action triggered by the alarm after a specified duration, ValueInRange is the action from the sensor signifying a value within the predefined threshold or range, and ValueOutRange is the action from the sensor signifying a value exceeding the predefined threshold or range.
Fig. 2. Visualization of sequence enumeration for sequences up to length=3 for an alarm system. The dotted lines in the graph represent illegal sequences while the solid lines represent legal sequences. We can see that at startup (Lambda) the only legal stimuli are ValueInRange and ValueOutRange. The reason is, e.g., because TimerExpired can only occur if the timer first has been initiated, which happens when ValueOutRange occurs.
For each legal sequence that the user identified, the user also needs to specify the system’s expected response by mapping the sequence to one of the enumerated responses. Conversely, an illegal sequence does not require the user to specify the system’s response, since it is expected to be infeasible and thus ignored. In addition to providing support to systematically generating the sequences, RealSBS also performs completeness analysis. For example, when defining the system’s states, RealSBS ensures that the states are complete (i.e., every possible value of any variable appears in at least one state) but are also disjoint (i.e., no overlapping states). For example, the state overlaps with the state , where the value “ANY” means any value of the variable. The tool can also check whether every interface is associated with at least one stimulus and/or an output; and whether stimuli and outputs are used in the mapping. We can automatically generate natural language requirements from the mapping produced from performing the SBE analysis. The mapping can also be used to generate a state diagram. C. SBE+Security – SBE Extended for Security During requirements elicitation, SBE is typically used to specify “nominal” functional behaviors, while “off-nominal” behaviors are typically denoted as illegal and thus not further analyzed. We observe that:
Certain security attacks can be viewed as illegal behavior. Therefore, it is important to further analyze illegal behavior and determine how the system should behave in the case such illegal behavior occurs. Attacks also tend to take advantage of acceptable or valid interactions between the system and its interfaces to introduce unexpected and undesired behavior. We make several adaptations to SBE to make it more useful for security analysis (which we call SBE+Security). The adaptations are as follows: Introduce a malicious agent as a dedicated interface and define its possible actions. Introduce an additional sequence criteria called “harmful” to denote a sequence that can cause harm to the device (and the user or operator of the device). Any legal or illegal sequence can be marked as “harmful”. A legal sequence is harmful if it contains an action or stimulus that comes from the malicious agent interface. Introduce an additional evaluation step, where the developer examines each illegal sequence to check whether that sequence is truly infeasible or the designers did not expect it to happen but do not actively prevent it from happening. If an illegal sequence is unexpected but can actually happen ( e.g., due to incorrect implementation), the developer should further examine if it could cause harm to the system or its users. If yes, then mark that sequence as harmful. If an illegal sequence is marked as harmful, then SBE+Security treats it the same as a legal sequence. That means that the user needs to specify what the system’s response should be. (In contrast, the original SBE method does not require a response for an illegal sequence, as discussed earlier.) The tool will also further enumerate longer sequences from a harmful illegal sequence by appending all possible stimulus to it. Note that, for each legal harmful sequence, the original SBE method requires the user to define a response for it. We have extended the RealSBS tool to incorporate this notion of harmful sequences. For all sequences marked harmful, we then create a requirement to either mitigate its potential harm or prevent it from ocurring. Specifically, for a legal harmful sequence, there should be a requirement representing functionalities to check if the stimulus causing the sequence comes from a credible interface or the malicious one. For an illegal harmful sequence, there should be a requirement representing an appropriate response to the sequence (e.g., block the sequence or put the system to a safe state), as defined by the application of the SBE approach. III. CASE STUDY – A PROPRIETARY BLOOD INFUSION PUMP In the following sections, we will describe a case study that we performed to evaluate the feasiability and usefulness of SBE+Security.
A. Blood Infusion Pump Overview We applied the SBE+Security approach to an infusion pump, which has CARA-like functionality (a control-loop software system that infuses resuscitation fluids), but infuses blood instead of saline solution to maintain the blood pressure (BP) of a patient at a safe level. Figure 3 shows an overview of the pump and its interfaces, where the orange and black boxes show the entry and exit points of the system. The device initially administers a certain amount of blood to the patient. To maintain the BP, the device then delivers blood continuously. If the device detects that the BP is at a dangerously low level, additional blood will be administered (i.e., at a higher infusion rate) as needed. If dangerously high BP is detected, the device will sound an alarm and pause the infusion. The patient’s BP is monitored using various sensors such as Continuous Noninvasive Arterial Pressure (CNAP). An alarm will be raised if any adverse physiology is detected and the blood infusion rate will be adjusted accordinly. The alarm functionality of this device is similar to the simple alarm example described earlier. The device also checks the “health” of the monitoring and pump subsystems, and raises an alarm when malfucntions are detected. Note that for this case study, we purposely establish the system boundary to only include the core part of the system, even though the device itself includes the CNAP, speaker, and pump component shown in Figure 3. Therefore, for our analysis, we do not make assumptions on what the “external” parts (i.e., entities outside of the boundary) can or cannot do, but simply deem them as black boxes that provide stimuli and accept responses through an interface.
Fig. 3. Blood infusion pump system
The design document of this device lists the requirements related to its functionalities. Using the design document as an input, we applied the SBE+Security method specifically to the BP alert/alarm functionality. A simplified set of the relevant requirements are provided below to give a sense of how the requirements were formulated. R1: If the system received a BP value that is above the warning threshold, then the system shall trigger a silent warning alarm and stop blood infusion. R1.1: If BP improved within 10 seconds of the silent warning alarm, then the system will clear the alarm, and resume blood infusion.
R1.2: If the silent warning alarm persists for 10 seconds or more, then the system shall trigger a audible warning alarm. R1.3: If BP improves, then the system shall resume blood infusion and clear the warning alarm. R1.4: If BP worsens from silent warning to the high critical range, then the system shall trigger a silent critical alarm and stop blood infusion. R1.5: If BP worsens from true warning to the low critical range, then the system shall trigger a silent critical alarm, stop blood infusion, continue the audible warning alarm while displaying the silent critical alarm. R1.6: The system shall disable the ‘Start Infusion’ button during the warning alarm. R2: If the system received a BP value that is above the critical level, then the system shall trigger a silent critical alarm and stop blood infusion. R2.1: If BP improves within 10 seconds of the silent critical alarm, then the system will clear the critical alarm and resume blood infusion. R2.2: If the silent critical alarm clears within 10 seconds and there was a true warning alarm before, then the system shall resume blood infusion and clear the warning alarm. R2.3: If BP improves from silent critical condition to silent warning condition, then the system shall trigger a silent warning alarm. R2.4: If the silent critical alarm continues for 10 seconds or more, then the system shall trigger a audible critical alarm. R2.5: If BP improves from true critical condition to true warning condition, then the system shall trigger a true warning alarm. R2.6: The system shall suggest the user with the new blood infusion rate when the true critical alarm clears to normal condition or to warning condition. R2.7: The system will not automatically re-establish the blood infusion rate when a true critical alarm condition is cleared to normal condition. R2.8: The system shall disable the “Start infusion” button whenever blood infusion is stopped due to critical alarm. R3: If the system received out-of-range BP values (less than 10 or greater than 180), the system shall discard the first out of range data received. R3.1: If the system received two consecutive out-ofrange BP low values (less than 10) or high values (greater than 180), then the system shall initiate system shutdown. Based on these requirements, we enumerate the following interfaces, stimuli, outputs, system states, and the mapping of leagal sequences and responses to facilitate the SBE+Security analysis:
Interfaces: Alarm, Alarm speaker, “Start Infusion” button, CNAP, pump, and log Stimuli: HighValue, BPImproved, BPWorsened, SilentCriticalAlarm10, SilentCriticalAlarm
