(This is a sample cover image for this issue. The actual cover is not yet available at this time.)
This article appeared in a journal published by Elsevier. The attached copy is furnished to the author for internal non-commercial research and education use, including for instruction at the authors institution and sharing with colleagues. Other uses, including reproduction and distribution, or selling or licensing copies, or posting to personal, institutional or third party websites are prohibited. In most cases authors are permitted to post their version of the article (e.g. in Word or Tex form) to their personal website or institutional repository. Authors requiring further information regarding Elsevier’s archiving and manuscript policies are encouraged to visit: http://www.elsevier.com/copyright
Author's personal copy
Advances in Engineering Software 58 (2013) 18–34
Contents lists available at SciVerse ScienceDirect
Advances in Engineering Software journal homepage: www.elsevier.com/locate/advengsoft
Software simulation and verification to increase the reliability of Intelligent Environments Juan Carlos Augusto a,⇑, Miguel J. Hornos b a b
Department of Computer Science, Middlesex University, UK Software Engineering Department, University of Granada, Spain
a r t i c l e
i n f o
Article history: Received 24 October 2012 Received in revised form 7 December 2012 Accepted 30 December 2012
Keywords: Reliability Intelligent Environments Simulation Verification Model checking Spin
a b s t r a c t This paper explains how the Spin model checker can be used to guide and inform the development of more reliable Intelligent Environments. The idea is to present a methodological guide which provides strategies and suggestions on how to model, simulate and verify these types of systems, as well as to facilitate the use of well-known tools like Spin in the development of Intelligent Environments. These tools, which have been developed by the Software Engineering community, have proven their usefulness for improving the quality of complex software systems, even in the industry field. However, researchers and developers in the area of Intelligent Environments do not usually make use of these tools. Our aim is therefore to encourage their use by colleagues working in this area to increase the reliability of these complex systems, which integrate aspects and elements of networks, sensors/actuators, ubiquitous/pervasive computing, human–computer interaction and artificial intelligence, among other related areas. Ó 2013 Elsevier Ltd. All rights reserved.
1. Introduction: importance of reliability in Intelligent Environments An Intelligent Environment (IE) [5,32] is a very complex system which is made up of a blend of components coming from different areas, such as networks of sensors and actuators, communication protocols, ubiquitous/pervasive computing, human–computer interaction and artificial intelligence, among others. The software that controls this type of systems is difficult to develop and error prone, since it is usually made up of diverse distributed processes which run concurrently. Given their mix of hardware and software components and their potentially rich interaction with humans, there are many opportunities for failure. These systems can be deployed in a variety of domains, for example: in home environments, to monitor and assist people; in educational organizations, to support teaching and learning activities; in smart cars, to make cars more efficient and make driving safer; in agriculture, to make it more productive; and in line production, to manufacture more efficiently. Fig. 1 shows a typical architecture for an IE. The data collected through sensors and other devices are stored in databases and pre-processed through the middleware layer before being passed to the reasoning layer. The reasoning modules will try to make sense of the data at a higher level. For example, if the system represents a smart home being used to control the temperature of the ⇑ Corresponding author. Tel.: +44 7426999393. E-mail addresses:
[email protected] (J.C. Augusto),
[email protected] (M.J. Hornos). 0965-9978/$ - see front matter Ó 2013 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.advengsoft.2012.12.004
house automatically, the reasoning module may take into account the time of the day, temperature outside the house, house owner preferences on levels of warmth, whether the house is empty or not, time when members of the family arrive everyday, and other elements of decision-making to infer what should be done. The result of this inference can be stored in a knowledge repository which keeps statistical information about the data itself as well as on the higher level decisions taken by the system. This can be mined by machine learning algorithms which can discover longterm usage of electricity patterns. The system may routinely be able to control the temperature of the house autonomously through the actuators of a system (in this simple example, the thermostat). Sometimes the intelligent system may decide that it is needed to request advice on ambiguous situations or that it is useful to notify the house owner on specific trends of temperature usage, or other information the user may consider useful. This is a very simple example and it illustrates the diversity of factors which may be taken into account. Some smart homes are designed to keep track of human behaviour; for example, in order to help if an elderly or vulnerable person living independently has fallen when nobody is at home. In all these systems, it is very challenging for the development team to make sure all the important scenarios have been considered and whether the intelligent system understands and reacts appropriately for all those important scenarios. IEs are specific examples of reactive systems [37,38], which are those that react to any stimulus (or event) that occurs in the environment, maintaining a continuous interaction with it. Their complexity lies more in the control than in the data; therefore,
Author's personal copy
J.C. Augusto, M.J. Hornos / Advances in Engineering Software 58 (2013) 18–34
19
Fig. 1. Generic architecture of an Intelligent Environment.
their behaviour is described by sequences of events, in which there are generally explicit temporal constraints. The informal reasoning is not appropriate for the development of such systems, since their executions are usually infinite and many of all their possible executions could not easily be considered, due to the interleaving of the many concurrent events that may occur. It is therefore necessary to apply techniques and methods that allow us to design and get correct IEs [7]. Software testing [29,9,34,31], which is by far the most used technique to improve software quality, is useful, but only to some extent, since testing alone is not suitable for exhaustively checking the correctness of complex systems in general, and of IEs in particular, because it consists on executing only a sampling of all possible runs of the system to be checked (the execution of all of them is usually unfeasible and not practical). Consequently, testing cannot prove neither the absence of errors in a software system nor the properly fulfilment of its goals. Moreover, it cannot guarantee that all errors (or even any of them) are found. Thus, the application of formal methods is essential to analyze and establish the correctness of such systems, especially in applications in which safety is a critical issue, where a small error in their design can even put human lives at risk. Think for example on the development of an IE to support independent living for vulnerable (e.g. elderly, disabled or chronic ill) people. This type of systems usually includes tasks like detecting whether the occupant of the house has fallen or is unwell in some way. It is therefore necessary to apply formal specification and verification to the development of these systems, operations which have the following meaning: a specification is the description (usually by means of a temporal logic formula) of the property (or temporal behaviour) to be fulfilled by the system under study, while the verification consists on (exhaustively) proving that the system in question is correct, i.e. all its executions meet its specification. We will focus on the verification technique called model checking [42,14], which consists on verifying whether any execution of the transition system that represents the system to be checked violates (i.e. does not meet) a certain property, specified by a temporal logic formula. In other words, it consists on checking that any execution of such transition system is a model of (i.e. it satisfies) that formula or specification. Hence the name given to this technique, which is also known as automatic verification, since it only requires minimal intervention of the user (verifier), who must create an abstract (simpler) model of the system to be verified and/or choose the crucial parts of the system to be checked, such as the underly-
ing basic algorithm, the communication protocol used or a restricted version of a system (where usually small limits are set to the values that variables can take, to the sizes of the message queues, etc.), instead of trying to verify the whole system. Thus, what must be verified is bounded and simplified, while we ensure that the essence of the software system is correct. The Software Engineering (SE) community often uses these methods [34], techniques and tools for verifying systems and software [12] at the earliest stages of their development. However, amongst IE developers these formal methods and tools are rarely used, though there are some exceptions which have started to appear recently [6,2,16,17,18,15,8,11,3]. Most of these works are isolated efforts which are based on very specific and specialized methods and tools, reason why they are rarely known in practice by the IE community. As a result, the corresponding proposals are only adopted by their developers. The present work aims to take a first step to build a bridge of understanding and collaboration between both SE and IE communities, by encouraging the use of methods and tools which are well-known by software engineers among researchers and developers of IEs, as well as their convenient adaptation to this type of systems, to make the resulting IEs more reliable. To do this, we provide a methodology with examples based on our experience in particular projects of this area which try to clarify how these methods and tools can be applied to similar projects or systems. The rest of the paper is organized as follows: Section 2 presents an overview of the methodology we follow to design more reliable software for IEs. Section 3 introduces the specific means we will use in this article to exemplify the application of our methodology. In this case, we decided to use Promela as modelling language, Linear Temporal Logic (LTL) as the language to specify behavioural properties, and Spin as the tool to check correctness. Section 4 proposes the application of the methodology to model sensor-based IEs by using model examples of increasing complexity, which can be found in Appendix A. This methodology tries to give strategies and suggestions on how IE developers can increase the reliability of their designs at the earliest development stages. Finally, Section 5 presents some concluding remarks.
2. Introducing the methodology Our proposal provides a simple but effective way to focus the mind of the developer on the important aspects of building a system for an IE, which at the same time: (a) Uses existing tools
Author's personal copy
20
J.C. Augusto, M.J. Hornos / Advances in Engineering Software 58 (2013) 18–34
developed by the SE community and (b) captures the essential behaviour of the IE components. We achieve these objectives through our methodology, which we call Methodology for Improving the Reliability of Intelligent Environments (MIRIE). The necessity of this methodology, which arises from the experience of the authors developing IEs, is clear from observing that most colleagues in the IE field do not follow a standard process to the development of their software systems. To increase its adoption we need a simple and flexible strategy, which consists in modelling the system to be developed using a very simple and abstract first model, and then refining it through successive steps by constructing models of increasing complexity. Our proposal is supported on the idea that an IE is basically a system which uses processing units, sensors and actuators distributed in the environment to support humans in their daily life [1]. Fig. 2 illustrates graphically that processing units are the core of the system, while sensors and actuator are the artefacts used to respectively capture the events from and act on the environment where the humans are living, working, driving, learning, etc., to cite only a few examples of activities carried out by humans in the corresponding environment. These activities will depend on the concrete use case or type of IE that we try to develop. The elements shown in Fig. 2 constitute the main entities to be modelled in an IE. These are the pillar concepts we will use in the remaining of this article and we will illustrate through examples how by focusing on those concepts we can think on the essential services we want the system to produce and how these services can be achieved. We will explain in other sections further down how we can model these concepts in more detail. Processing Units refer to the information processing elements of the system, this includes the software layers capable to retrieve and make sense of the sensor data (e.g. middleware and artificial intelligence software capable to learn, reason and decide on the behaviour of the system). The environment is the place where the system to be develop will be inserted and might be very diverse (e.g. a home, a factory, an office, a classroom, a vehicle, or a museum). Sensors refer to the devices which collect information from the environment, while actuators are those devices which can be used either to provide information to the environment or to perform some action on it. Some devices have both capabilities and are able to collect and to provide information (e.g. a touch screen). By humans we understand the people involved somehow with the system. Some humans may be the main expected recipient of the services, whilst others become involved with the system in a secondary and casual way. For example, consider the case of a smart home where an elderly person lives independently supported by the services provided by the system installed in the home, which understands the activities taking place thanks to the light switches, door and window sensors, pressure pads in sofas and beds, PIR (Passive
Fig. 2. Main elements in an IE.
Infra-Red) motion detectors in rooms and RFID (Radio Frequency IDentification) tags that are placed on objects of daily use and read by RFID readers. The house can take actions to increase comfort, safety and efficiency by regulating the temperature and lights of the house and can issue reminders and advices through a tablet PC, for example. The main user is the elderly person being cared for but occasionally family members, friends and social workers visit the occupant. Once we have identified some possible use cases as target systems and the main entities to be considered in them, it is time to mention the activities or phases that IE developers have to perform using our proposal. Similarly to the methodology described in [16], which presents a rigorous and very structured process for designing ambient intelligence applications consisting of four sequential workflows (with 12 tasks identified inside them) and the necessary iterations over them, ours also considers four design phases (which is compatible with the workflows of the mentioned work, although ours is not so structured, since we do not distinguish tasks inside each phase and we put more emphasis on giving examples of the iterative process, which is required to understand different aspects of a system): 1. Informal modelling. This phase provides informal descriptions of both the application domain and the correctness properties to be checked using natural language. All the stakeholders have to participate in this phase. 2. Structural modelling. This phase identifies the entities to be considered. Although a first approximation is given in Fig. 2, designers have to successively refine it and decide which the final entities to be modelled are. For example, in the example models we provide in Appendix A, a process (or proctype, using the specific keyword of the modelling language) is defined for each entity identified and selected to be modelled. 3. Behavioural modelling. The contents of each process created in the previous phase has to be defined or refined in this phase by modelling the dynamic behaviour (movements, activities, operations,. . .) of the corresponding entities involved. 4. Simulation and verification. Both techniques are used to increase the reliability of the models created. Using them, designers will discover faults, scenarios and situations that they would have hardly considered. This will require moving back to the previous phases to redefine the models. Since a simulation is equivalent to an execution of the system to be developed, it only gives some insight on how the model created behaves, while the formal verification of a correctness property using the model checking technique will determine that either all the executions of the model satisfy such property or any of them violates it; in this last case, a counterexample is given, which can be used to redefined the model and/or the property specified. Although the notations used in [16], which are ambient calculus, ambient logic and real time temporal logic, are closer in terminology to the one we use in this paper, so far the methodology presented in the mentioned work does not provide formal verification through model checking. Consequently, both methodologies can be considered in some sense as complementary. There are several well-known strategies from Software Engineering which can be followed in using the methodology here proposed, ours is more akin to the Spiral [13] or the Prototyping [33] Software Development Methods. As mentioned above, we start with small and basic models and we refine them into successively more complex versions.
Author's personal copy
J.C. Augusto, M.J. Hornos / Advances in Engineering Software 58 (2013) 18–34
21
Fig. 3. Interpretation of temporal operators on a sequence of atomic propositions.
3. Applying the methodology The previous section introduced MIRIE, a generic methodology which can be used to inform the development of any IE. MIRIE can be applied with the help of different developing methods and tools. To exemplify this we have selected a tool that is wellknown, efficient and provide us with most of what we need. The selected tool is a model checker which has a modelling language, plus simulation and formal verification capabilities. This choice is well founded but is not essential. The reader can replace our choice for this article for any similar tool or collection of tools which facilitates the deployment of the methodology. To illustrate the application of MIRIE we will use in this article the Spin model checker ([25,10], which is one of the most powerful and widely used systems for software verification. Its underlying algorithms (some of them can be found in [41,22]) are highly efficient and stable, its technical support is very good and its latest graphical user interface, called iSpin, is very nice and user-friendly. Moreover, it is freely available for educational and research purposes. As Spin is focused on the concept of models, users have to create them using its modelling language, called Promela (acronym for PROcess MEta-LAnguage) [21], which is intended to easily make good abstractions of the system to be modelled, putting emphasis on the description of processes and their interactions through message channels and variables. While processes specify a given behaviour, variables and synchronous (rendez-vous) or asynchronous (buffered) message channels are used by processes as communication and coordination means, and define the environment in which the processes run. Although its syntax is C-like, Promela is not a programming language, reason why it lacks pointers, functions and some data types such as floating point numbers. This non-deterministic specification language, which allows the dynamic creation of concurrent processes, has an interleaving semantics, i.e. the processes are interleaved, in such a way that the scheduler executes only one process at a time. Once a model has been created, the simulator included in Spin can be used to get some insights on the scenarios described in the model, since this is executable in certain sense. This tool can generate a simulation run randomly or interactively. In the former all non-deterministic choices in the model are resolved calling to the random number generator, while in the latter they are determined by the user direct selection. It is also possible to provide a trail file to perform a guided simulation; this file is generated in the verification mode and contains a counterexample or execution sequence showing that the model violates the property checked. Although simulation is useful for a quick first assessment of the model, it is not adequate to find subtle errors, since it is generally infeasible (and even impossible) to simulate all the possible runs of a model. Thus, simulation can only help to debug the models, but no amount of simulation runs can prove that a model satisfies a given property.
If our goal is this last case, we have to perform a formal verification (by applying model checking), which consists on ascertaining the correctness of a model (with respect to a certain property) by exhaustively checking whether all the executions implied by the model satisfy such property usually expressed as a LTL (Linear Temporal Logic) [36,20] formula. We must therefore provide both things to run the model checking algorithm included in Spin: the Promela model and the LTL specification. As a result, we can get one of the following three possible outcomes: (1) The model satisfies the property. (2) The model violates the property. In this case, Spin provides a counterexample, which is an execution path showing why that property is not true for that particular model. This counterexample is stored in a trail file that we can use as input to perform a guided simulation; thus, we will be able to visualize and study more conveniently the counterexample. (3) There are insufficient resources to solve the problem raised. This is due to the well-known state explosion problem [40]. In this case, we will have to construct a more abstract model and/or modify the (LTL) correctness property to be checked, and repeat the verification process for the new model and/or property. Another alternative would consist in running the verification on other computer with more computational resources (e.g. with a greater number of processors and available memory size), as well as extending the time given to perform the verification. Interested readers can find information on how to install Spin as well as an explanation of its main functionalities using its graphical user interface in [28]. As mentioned above, to specify the properties to be verified, Spin users can make use of Linear Temporal Logic (LTL) [36,20]. This logic extends Propositional Logic1 with some temporal operators for the future, being the most usual the following ones: }, h, s and U, which respectively are read as eventually, always, next and until. The three first are unary operators (i.e. they are followed by only one formula) whereas the last one is binary (i.e. it is inserted between two formulas). Here we need to clarify that these operators are adapted in Spin so that they can be expressed in ASCII notation; as a result, they are respectively represented as , [], X and U. Likewise, the classical logical symbols are also adapted to this notation; thus, ^ becomes &&, _ becomes ||, ? becomes ->, and becomes !. The meaning of the simplest formulas using such temporal operators is as follows: p: Eventually (in the future) p holds, i.e. p: is fulfilled at some future state (from the current state forward).
1
We will assume the reader has a basic understanding of this logic.
Author's personal copy
22
J.C. Augusto, M.J. Hornos / Advances in Engineering Software 58 (2013) 18–34
[]p: Always (in the future) p holds, i.e. from the current state forward p is true. Xp: At the next state p holds. pUq: p: is true (from the current state) until q is true. The interpretation of these temporal operators will be informally explained with the help of Fig. 3, which shows the states that satisfy each of the previous temporal formulas on a specific sequence of atomic propositions (p and q). sq (or Xq in Spin notation) is fulfilled at the state s4, since q holds at the next state (s5). The formula }q (or q) is satisfied if and only if there is at least a state in the future where q holds; this formula is therefore fulfilled at states from s0 to s5, since q holds at this last state. The formula hp (or []p) is satisfied at a state if p permanently holds from that state; thus, since p remains invariantly true forever from state s6, this formula meets in all these states. Finally, p[q (or pUq) means that p has to hold from the current state until the one where q holds (this can occur at the current or a future state); consequently, this formula is fulfilled from state s2 to state s5. Let r be an infinite sequence of states si and let f and g be any LTL formulas. The formal semantics of the temporal operators presented is given by the following expressions, taking into account that the symbol means ‘‘satisfies’’ and iff means ‘‘if and only if’’:
r f iff s0 f si f
iff
siþ1 f
si f
iff
8j; j >¼ i :
sj f
si }f
iff
9j; j >¼ i :
sj f
si f [ g
iff
9j; j >¼ i :
sj g
and 8k; i q) Persistence: []p Recurrence: []p
The first three constitutes common specification patterns which are frequently used. Application examples of these three patterns to the context of an IE can be the following ones: []!(fridgeOpen && fridgeClosed), which means ‘‘it is not possible that the fridge is open and closed at the same time’’. lightOn, which means ‘‘the light will be on at some instant in the future’’. [](smokeAlarmTriggered -> sendAlertSMS), which means ‘‘whenever the smoke alarm is triggered, the system responds sending an alert SMS (e.g. to the mobile phone of a caregiver)’’. In principle, we may be tempted to see verification as an oracle which can provide answers to all sort of important questions about the system we are building. However, there are some practical limitations to consider. On one hand, we may not be able to express all possible situations the system can go through with the tools we use. For example, it will be very hard to model very faithfully all the possibilities entailed by some sensors or devices (e.g. acceler-
ometers, cameras, and microphones) if the range or combinations of values they produce is too big to be modelled in Promela (a language mainly prepared for discrete systems). We can consider using other tools which allow the use of real numbers and more complex model checking techniques, e.g. UPPAAL [30] (examples of the application of this model checker can be consulted in [6,39]), but in most cases we can only aspire to work with a simplified (e.g. discretized) version of the original problem. On the other hand, even in the case we can express faithfully a specific situation; it may pose such a difficult computational challenge that it is not practical to try it. For example, the verification of some properties may take a long time to be completed or it may require more physical memory than available in the machine we are using. Despite all this, the range of interesting situations we can explore and the benefit of putting to the test the intended development strategies and being objectively informed by the available tools largely justifies the exercise and our next section attempts to provide some examples of such situations of interest.
4. Example on the application of MIRIE Promela models follow the traditional programming template, where each conceptual computing unit consists of a declaration section followed by a section where the behaviour of the system to external inputs is described. In this section, we apply MIRIE to model, simulate and verify sensor-based systems and we will try to show its usefulness with the help of a series of models (which can be found in Appendix A). To give examples on how to use Promela to model IEs we will focus on smart home applications and we will unfold models of increasing complexity. These models however will share the following common structure:
/ Definition of global data structures (e.g. sensors) / / ———————————————————————————————————————————/ active [X] proctype HumanActivity () {/ Definition of local data structures / / Behaviour modelling / } / ———————————————————————————————————————————/ active [Y] proctype SensorActuator () {/ Definition of local data structures / / Behaviour modelling / } / ———————————————————————————————————————————/ active [Z] proctype Processing_Unit () {/ Definition of local data structures / / Behaviour modelling / }
That is, we will assume we want to model mainly a home equipped with sensors and actuators, which is inhabited and will receive visitors. The effect that the actions of occupants and visitors (i.e. human activity) will have on the sensors and actuators will be collected by a series of processing units. Each of these concepts can be represented by a proctype (i.e. the declaration of a process) in Promela, where X, Y and Z should be replaced by the corresponding integer numbers representing the number of processes of each type which will run simultaneously in the system modelled. These can be represented in many different ways and degrees of detail or abstraction, we do not want to be prescriptive and each developing group will find the level of usage that is useful for their goals. We only provide examples which we hope will
Author's personal copy
J.C. Augusto, M.J. Hornos / Advances in Engineering Software 58 (2013) 18–34
motivate colleagues to use Spin and other similar tools to reason about their development and in doing so they will develop more carefully engineered, and hence safer, systems. Although the basic model structure presented is very simple in terms of the level of detail and insight that can offer on what happens inside the environment being modelled, it is useful to explain some basic principles before we add more information (and complexity). There are several design choices which can be different; for example, instead of using specific channels to model the communication between each pair of processes (proctypes) that have interaction, a common single channel could have been set up with a more complex message structure complementing the message contents with both sender and receiver IDs. Other models can be created to allow more than one occupant and/or visitor, identifying them, distinguish them amongst several possible actions, consider a wide range of specific sensors, etc. Observe that the generic model previously shown has a process named SensorActuator. This is because both sensors and actuators can be treated in that process. Alternatively, we can define two processes, one for sensors and another for actuators. Some of the subsequent models only have sensors, because that is what we want to model. Further below, other models will also have a process for actuators. Throughout the rest of the article we will illustrate the usefulness and relevance of tools like Spin to the development of IEs by showing how core concepts of a typical application can be addressed. To this end we will make use of one of the most popular application domains within IEs, smart homes. This concept refers to the deployment of technology in a house with the purpose of providing practical benefits to its inhabitants. One possible application is to implement services of domotics (house automation) whilst in the latest decade the most compelling application has become Ambient Assisted Living, where the expected benefits are mainly focused on increasing safety, supporting healthcare services and encouraging well-being [4]. Imagine we want to use our methodology to model such a smart home and the potential consequences of adding some sensors to it to understand how the people using the house can benefit
23
from these services. Initially, we will model the essential components of the system (shown in Fig. 2) as well as their interactions, at least to the extent we need for the planned services. The most basic infrastructure is provided in the first model (A, included in Appendix A) and the effects of its execution (i.e. simulation) are visible in Fig. 4. Subsequent models and figures reflect the work on interactions between the essential concepts of the IE modelled. More specifically, we focused on the interactions between humans and the system, which has processing units, sensors and actuators, and we try to assess if the system can successfully reflect the situations where human behaviour is reflected by technology in a discernible and accurate way. Model A (see Appendix A) specifies a bit more the previous generic model by modelling (via processes) one occupant, one visitor and one sensor, as well as one processing unit, reason why we named this last process as Central_Unit in this model (and in the subsequent ones). Fig. 4 illustrates the use of the simulation facility in Spin, which shows in the left hand side the Promela model (in this case, the model A shown in Appendix A), on the right the Message Sequence Chart (MSC) with the messages sent and received by each of the processes through channels, and at the bottom the sequence of executed lines in the Promela model plus their effect on data (left, in this case empty as we do not use variables) and channels (right). The model is very simple and the MSC on the right hand side shows the messages going from both the occupant and visitor processes to the sensing device (these denoting that they have performed some actions which have been perceived by a sensor) and this causing a message sent to the processing unit (representing that the sensor activation is being sent to a program or control box which will process it). Note that we used activities?_ as part of the contents of proctype Sensor, since at this stage we only want to detect activity regardless of whether it is from the occupant (occ_act) or from the visitor (vis_act). In other applications, it may be required to capture the message in such a way that it can be processed after that, for example through activities?M, where M is a variable of mtype which will retain the content of the message received.
Fig. 4. A simulation run of the model representing the most basic infrastructure.
Author's personal copy
24
J.C. Augusto, M.J. Hornos / Advances in Engineering Software 58 (2013) 18–34
Fig. 5. Safety verification with no errors found.
A simple safety verification shows that there are a priori no errors of this category, as shown at the black right-bottom panel in Fig. 5. If we want to conduct a more thorough investigation of the behaviour represented by this model, we can use properties expressed in Linear Temporal Logic (LTL). That can be achieved in iSpin by embedding an inline property specification which we can then refer to in the verification console. Say we want to check that whenever each time a human performs a detectable action, then eventually that will be notified to the processing unit, where it will be processed. One way to achieve that (not necessarily the only one or the best one) is to modify the model as shown in model B (see Appendix A). In this last model we introduce a way to detect when we have reached the two meaningful conditions for this case: ‘‘a human performs a detectable action’’ and ‘‘notified to the processing unit’’. Human actions can be either coming from the house occupant or from a visitor. So we can use a Boolean variable (act) to detect when an activity has been performed and other Boolean variable (received) to detect when the processing unit is notified that a sensor has been activated. The model B has new code (highlighted in red and underlined, see Appendix A) with respect to model A: the declaration of the auxiliary variables, the formal statement of the behavioural property to be checked, and the insertions in the code of the proctypes to detect when the system has reached the meaningful states we are interested to correlate. Fig. 6 shows the successful check of the property we described above. In the centre of the image, it can be seen we have asked iSpin to check the consistency of property named as p1 against our model, and on the right-bottom hand side, the system diagnosis is that there are no errors. Of course, the above model does not distinguish which one of the potential users was responsible for the sensor activation. If we want to check for each one separately we need to distinguish them further, as model C does. This again leads to the same result, shown in Fig. 7. Imagine that for some of those activities we want the system to do something specific, this could be recording the activity in a log or database, sending a notification message to a mobile phone or
interacting with the humans in the house (for example to issue a greeting message or a recommendation). Let us call this action from the system actuation. We may not want the system to react to every single instance of sensor triggering detected with an actuation so we can insert code which allow Spin to non-deterministically choose when to actuate and when not to. Model D (see Appendix A) incorporates all these changes and also includes a property to attempt to check whether it is true that whenever a human (occupant or visitor) performs a detectable action, this will result on an actuation. We will explain more about this property further down. Fig. 8 shows the simulation of this model. The MSC window shows that actuation is not always performed after a sensor activation, as we expected, since we inserted code (in proctype Central_Unit) to allow Spin to choose over this behaviour. Notice that when we check the property p1 included in model D (see Fig. 9), the result does not reflect the behaviour observed in the simulation. The check performed in Fig. 9 says the property is correct for model D. However, we have seen in the simulation that not all human activity entails an actuation. Where is the problem? The semantic gap is in the interpretation of the LTL formula checked, i.e. []((oact || vact) -> acted), which actually is satisfied when acted is true in after either oact or vact have occurred, but without taking into account frequency or correspondence relationships between sensing and actuation events. Thus, as the system executes infinite loops, it sees that there is always an opportunity to have either oact or vact followed by acted (it does not matter how close in time they are, since temporal distance is not measured by the operator, which means ‘sometimes in the future’). In other words, the above formula does not necessarily mean that each time a human (occupant or visitor) performs a detectable action, this will be strictly followed by an actuation. To explain this more clearly, imagine the following two sequences, r1 and r2, of sensing (si) and actuation (aj) events:
r1 : . . . s1 ; a1 ; s2 ; a2 ; s3 ; a3 ; s4 ; a4 . . . r2 : . . . s1 ; a1 ; s2 ; s3 ; s4 ; a4 . . .
Author's personal copy
J.C. Augusto, M.J. Hornos / Advances in Engineering Software 58 (2013) 18–34
25
Fig. 6. Checking that the model satisfies the property p1.
Fig. 7. Verifying again the property p1 against the modified model.
If all possible achievable sequences produced by our model were as in r1 then we could say that it is true that ‘‘whenever a human (occupant or visitor) performs a detectable action, this will result on an actuation’’. However, the system behaves more like in r2 (and we have seen that through a simulation). Thus, the formula []((oact || vact) -> acted) was found valid by iSpin, because for each si there is an aj in the future (it does not matter how close or far away they are in the sequence). Going back to our model, if we want to be more precise and be able to track the consequence of specific actions, we need to invest
a bit more in the model. Model E (see Appendix A) provides a variant which allows us to be more precise in the follow up of user actions. The following simulation (displayed in Fig. 10) shows how Spin non-deterministically interleaves the different processes, since before reaching actuator for the first sensed action it allows the detection of another visitor’s action. Eventually both are communicated. Then an action from the occupant is detected but not communicated through the channel actuation, as we encoded the central processing unit not to do so (using skip, i.e. ignore).
Author's personal copy
26
J.C. Augusto, M.J. Hornos / Advances in Engineering Software 58 (2013) 18–34
Fig. 8. Simulation of the new model.
Fig. 9. The model satisfies the property p1.
A verification of the LTL formula [](Source==unknown U (Source==vis U Source==unknown)), which is shown in Fig. 11, states that only visitors are identified. The idea behind this property is that the detection of unknown and visitor alternates. Model E includes the declaration of the variable Source, which is initialized to unknown. Making use of it, the model only allows identifying the source of sensor triggering when it has been labelled as coming from a visitor. Once a visitor triggering has been detected and identified, the system is set to unknown again.
If we for example would like to check if the system is actually capable to identify visitors as the source of an event triggering, we can check whether it is always true the formula [](Source==unknown) (note that this property have been inserted and named as p2 in the model). Fig. 12 shows us that this is not the case and that Spin generates a counterexample (which is saved in a file with name finished in ‘‘.trail’’). We can see the counterexample generated by visiting the simulation panel (see Fig. 13). The counterexample shows that Spin
Author's personal copy
J.C. Augusto, M.J. Hornos / Advances in Engineering Software 58 (2013) 18–34
27
Fig. 10. Simulation showing that some requests to the processing unit are not processed.
Fig. 11. Verifying the alternated detection of unknowns and visitors.
has a way to reach situations where visitors can be detected, hence contradicting the claim p2. So far we have focused on illustrating how the system can be used and we have kept models simple to explain the use of Spin and its various components. The following contents are focused on providing hints on how to represent classical features of IEs. This will come at the cost of increasing the complexity of the models and the computational complexity needed to verify them.
Model F (included in Appendix A) assumes one occupant and one visitor, but it expands on the possible locations. Let us assume the location humans are in can be tracked using a combination of PIR (Passive InfraRed) sensors and/or RFID (Radio Frequency IDentification) tags. The next room a human can visit is determined by the internal structure of the house, for example all rooms are communicated through the reception hall, which also allows going outside (front door). At any stage Spin can choose to simulate that a human has choose either to stay in
Author's personal copy
28
J.C. Augusto, M.J. Hornos / Advances in Engineering Software 58 (2013) 18–34
Fig. 12. Verification indicating that the system can identify events triggered by visitors.
Fig. 13. Guided simulation showing the counterexample.
the same room or to go to an adjacent room communicated by a door. Outside is considered as another room or area. The resulting models are rather long, but we opted to repeat code and make it easier to understand instead of more compact and efficient but cryptic. A simulation of model F (as the one shown in Fig. 14) gives a glimpse on how the system can behave. Following the sequence of choices in the middle lower panel, we can see the choices Spin has made for this simulation. Initially, the occupant was in the reception hall, while the visitor was outside, then the visitor enters the house whilst the occupant goes to the living room. The impor-
tant side of this exercise is that it should encourage us to think about these scenarios. Is this scenario feasible? What else can happen? For example, if a simulation shows us that the visitor enters the house without the occupant being at the reception hall it may indicate the possibility of a burglar or the visit of someone who is very familiar with the house, maybe a relative who has a copy of the key. Thinking of these possibilities will push the developing team into thinking about security and authentication. The most important process should occur in the developers’ minds, the model is only an excuse to trigger thoughts which otherwise they may have not thought about.
Author's personal copy
J.C. Augusto, M.J. Hornos / Advances in Engineering Software 58 (2013) 18–34
29
Fig. 14. Simulation run of the model which indicates the occupant’s and visitor’s location.
Fig. 15. Simulation capturing the triggering of a warning message.
Say that now we want to be more specific about the safety of the occupant and either an accelerometer or a camera is installed which allows the system to infer the position of the occupant’s body, e.g. whether s/he is sitting, standing, walking or lying (on the floor). If the system detects the person is lying on the floor, we would like the system to raise an alarm; to that end we will have to modify the behaviour of the proctype Central_Unit. Model G (see Appendix A) includes such changes, as well as modifications on the data types and on the rest of proctypes to allow this level of subtlety.
The simulation snapshot displayed in Fig. 15 captures an instance when the occupant is believed to be lying in the reception hall, triggering a warning message. We can investigate a wide range of conditions, for example: Can both occupant and visitor be outside simulatenously? (occ_loc==outside && vis_loc==outside)
Author's personal copy
30
J.C. Augusto, M.J. Hornos / Advances in Engineering Software 58 (2013) 18–34
Fig. 16. Verification process returning an error on search depth.
Fig. 17. Increasing the length of the search depth to verify a property.
Can the occupant be outside and visitor inside? (occ_loc==outside && (vis_loc==receptionhall || vis_loc==kitchen || vis_loc==livingroom || vis_loc==bedroom || vis_loc==bathroom))
Will the system issue an alarm when the occupant is lying on the floor? [](occ_activity==lying -> alarm) Will the system issue an alarm when the visitor is lying on the floor? [](vis_activity==lying -> alarm)
Author's personal copy
J.C. Augusto, M.J. Hornos / Advances in Engineering Software 58 (2013) 18–34
31
Fig. 18. Verification of the property p2.
Fig. 19. Verification of the property p3.
Let us assume we label these properties as p1, p2, p3, p4 respectively, then the results of their verification are as follows. Initially we are notified that ‘‘max search depth is too small’’ for this problem (see black panel in Fig. 16), that is, the model has a search space tree which is bigger than the limits set up in the Advanced Parameter Settings section (upper right hand side of the verification panel).
After increasing the length of the search depth, we try again and Spin declares property p1 is valid, as shown in Fig. 17. Notice from the statistics appearing in the lower right hand side of the screen (in Fig. 17) that the maximum depth Spin was forced to explore in the tree search was of length 330,785 steps. If we try to check p2 (see Fig. 18) Spin finds the property is also valid.
Author's personal copy
32
J.C. Augusto, M.J. Hornos / Advances in Engineering Software 58 (2013) 18–34
Fig. 20. Verification of the property p4.
The same occurs with the verification of property p3, as shown in Fig. 19. All this was as expected. Now, what do you think the result will be for property p4? The answer is given in Fig. 20. That is, Spin has found that property p4 is valid. That may surprise you because we did not intentionally set up that alarms should be raised when the visitor is lying on the floor. The issue here is again a mismatch between our beliefs and what the model entails. Spin checks that it is always possible that instances when visitor is lying can be followed in the future by the variable alarm been set to true (because of the occupant is lying on the floor!).
5. Validation of the methodology This methodology was tried in a project funded by the UK government in collaboration with government-funded healthcare organizations and a company focused on the provision of services
Fig. 21. Multi-agent architecture for the Nocturnal project.
for elderly people. In the project, called Nocturnal (Night Optimized Care Technology for UseRs Needing Assisted Lifestyles), Spin was used in the way explained in this article to inform the design of a multi-agent system at the core of the reasoning system. Some complementary details of this experience are reported in [8]. The goal of the project was to produce a commercially viable technological infrastructure which based in sensors can increase safety of elderly people at night time. Movement sensors were deployed in each room of the house and pressure pads were used in the main bed to allow the system to know where the person being cared for was. Lights and a bedside unit which was capable to provide calming music and images were used as actuators. The reasoning system at the core of the system was composed of a team of intelligent agents organized in a hierarchy where different agents had different responsibilities (see Fig. 21). One agent was given the task to detect restlessness, another one to detect when the user was out of bed and a third one whether the user was wandering. A coordinating agent was overseeing the three basic monitoring agents and, if the situation required it, then it will decide the best course of action, which sometimes included actuation through the bedside unit or the lights to guide the user back to bed and to provide a calming environment where the user is more likely to settle and rest. These interventions are responsibility of the Therapeutic Intervention Agent, which deploys the plan and monitors progress. Its effectiveness is measured by the Coordinating Agent based on how much the behaviour of the user approaches the ideal. If the Coordinating Agent perceives the actions taken have not ameliorated the problem, it can contact the specific user’s carers. There is a diversity of possible scenarios to consider and the use of models of increasing complexity was very useful to develop the system in a bottom-up manner. The use of simulations at early stages provided interesting situations which the developing team did not consider. Verification was applied to check that specific properties were expected to emerge as a result of specific implementation strategies. Examples of these were determining: How the system performed in the absence of malfunctioning components.
Author's personal copy
J.C. Augusto, M.J. Hornos / Advances in Engineering Software 58 (2013) 18–34
What happens when there is no user activity? How to detect it? How to proceed? Whether the different specialized agents were able to detect the situations they were responsible for (e.g. how the system was supposed to detect episodes of ‘restlessness’ and how that concept was defined). Whether the specialized agents will be able to effectively raise the attention of the Coordinating Agent when they detected an undesirable situation, e.g. will always the Wandering Agent be able to reach the Coordinating Agent when needed (and only when strictly needed)? How the system monitored the progression of a situation when the Therapeutic Intervention Agent was applying a strategy to bring a benefit to the user. The failure of some of the properties led to the redesign of the way the team of agents interacted with each other. The methodology was applied approximately along half the time of the project (one year and a half). A global and generic model of the whole system was complemented with more specific models for each of the agents in Fig. 21. Given the overall complexity and the expected state space explosion problem, it was not possible to have a model which considered all the agents together at a good level of detail. The best we were able to explore the implementation strategy was by analyzing the global interaction of the agents with the generic view model and by using the individual models for each agent to explore their internal logic in greater level of detail. On each independent dimension, a list of main situations of interest was considered. For example, 21 different situations were identified with regards to the processes of getting to bed, sleeping and getting up, such as ‘‘Going to bed – Abnormal – goes to bedroom but does not get into bed,. . ., Sleep – Restless – no exit of bedroom,. . ., Getting up – Normal – bathroom visit and return to bedroom without going to living area,. . .’’. These were used as inspiration to query the models and observe how different parts of the system will react on dealing with specific situations. As a result, we were able to identify several unexpected situations which were not initially considered, some of them generic (e.g. no sensors working, ambiguous situations arising from multiple occupancy) and some others more specific and subtle (e.g. how to model in different ways poor sleep as a phenomenon which can manifest either as peaks of activities in a few minutes or an accumulation along several hours). These explorations of our ideas led us to important insights ranging from decisions on when the system was useful and usable to how to implement specific trigger functions of the system. More details on the application of this methodology to the design and development of the Nocturnal project can be obtained from [8]. In summary, the overall experience was very positive. Some of the errors found were considered significant by the developer team and unlikely to have been detected by other means. Only taking into account this, we think that the application of the methodology to this project was worth using. At another level of evaluation, we have the intention of encouraging its use by colleagues working in this area and gathering their feedback in this experience by asking them to fill out a survey, with the aim of having a better evaluation, not only qualitative but also quantitative, of MIRIE.
6. Concluding remarks We have presented strategies and suggestions on how to model different features of IEs by using a generic methodology, MIRIE, which can be effectively applied to the design and development of any IE. For the sake of illustrating the application of MIRIE in this article, we have used a well-established tool, Spin, which already
33
brings modelling capabilities, through a language called Promela, as well as simulation and verification capabilities available at the push of a button. There are many ways to achieve the same; we do not claim the one given in this paper is the optimal one under all criteria. However, we believe ours is a valid choice and it will hopefully pave the way for more people developing IEs to take advantage of tools which have been developed by the SE community and used in industry. We also understand the use of tools like Spin has limitations in modelling all the complexity of IE systems, but ‘‘the good should not be enemy of the perfect’’, that is, not because we do not have a perfect tool we should preclude ourselves from enjoying the benefits we can get through what is available. One typical problem developers who have not previously used Spin (or other similar tools) will experience is what we all tend to initially do, i.e. programming, usually ending up with complex, detailed and cumbersome chunks of code. Modelling is about creating the simplest abstraction which allows us to check the correctness of an idea. Hence the idea is to apply our methodology to generate different conceptualizations of the system to be developed and explore both positive and negative consequences of different design choices. Any user that applies MIRIE should keep in mind that the most important benefit is on the stimulation of her/his mind in different directions than those s/he would have explored without following this process. To benefit the most out of this exercise, it is important to create the habit of reflexion associated with the results of a simulation or a verification run. When a simulation or a verification run gives us an unexpected result, it is important we ask ourselves: Can this happen in the real system if we apply the implementation strategy of this model? ‘Fixing’ the model until it behaves as we want is, in certain way, secondary. The capacity of both the random simulation and the verification results to surprise us is an important expected benefit, because in our quest to solve a given problem we are usually forced to focus our mind on the main issues and we often forget about less likely but still feasible and important scenarios. Our proposal does not want to be prescriptive but rather inspirational. It aims to build bridges between IE and SE communities. We have explained how important concepts of IEs can be mapped into existing SE tools with the added benefits of an increased understanding of the system under construction which facilitates the development of a more reliable final product. Another interesting perspective of work in this area is pointed out by the need for a more focused effort from the research community into the development of methodologies and tools which can facilitate the task of connecting the worlds of IE design and the one of existing tools. This task is not so much about discovering new ground breaking computational algorithms, but rather supporting IE development teams to more easily translate the technical infrastructures they aim to deploy into the logical models that can be then checked for soundness. In this sense, one interesting path to be explored for IEs consists in trying to apply the technique known as software model checking [27], which is based on an automated extraction of verification models for Spin directly from program source code [24]. Thus, instead of constructing manually abstract models, requiring a considerable investment of time and expertise, the effort now will shift to the construction of an adequate model extractor, which will act as a pre-processor for the Spin model checker and will be responsible for deciding which portions of the source code will be converted into model fragments, and how these fragments will have to be combined in the system model to be verified. As this technique has been successfully applied to several very large and complex software systems, such as the call processing module of a telephone switch [26] and the legacy flight software from NASA’s Deep Space One mission [23], we consider it could be also applied to IEs. Consequently, there is a wide range of
Author's personal copy
34
J.C. Augusto, M.J. Hornos / Advances in Engineering Software 58 (2013) 18–34
exciting opportunities lying ahead of us in this area, and hopefully this article will encourage our target scientific community to uncover and leverage them. Appendix A. Supplementary material Supplementary data associated with this article can be found, in the online version, at http://dx.doi.org/10.1016/j.advengsoft. 2012.12.004. References [1] Augusto JC. Ambient intelligence: the confluence of ubiquitous/pervasive computing and artificial intelligence. In: Schuster A, editor. Intelligent computing everywhere. Springer; 2007. p. 213–34. [2] Augusto JC. Increasing reliability in the development of intelligent environments. In: Proceedings of the fifth international conference on Intelligent Environments (IE’09), Barcelona, Spain. IOS Press; 2009. p. 134–41. [3] Augusto JC, Hornos MJ. Designing more reliable MAS-based ambient intelligence systems. In: Bosse T, editor. Agents and ambient intelligence – achievements and challenges in the intersection of agent technology and ambient intelligence. IOS Press; 2012. p. 65–90. [4] Augusto JC, Huch M, Kameas A, Maitland J, McCullagh PJ, Roberts J, et al., editors. Handbook on ambient assisted living – technology for healthcare, rehabilitation and well-being. AISE book series, vol. 11. Amsterdam: IOS Press; 2012. [5] Augusto JC, Callaghan V, Zamudio V, Egerton S, editors. Proceedings of the eight international conference on Intelligent Environments (IE 12), 26–29 June 2012, Guanajuato, Mexico. IEEE Press; 2012. [6] Augusto JC, McCullagh PJ. Ambient intelligence: concepts and applications. Int J Comput Sci Inf Syst 2007;4(1):1–28. [7] Augusto JC, McCullagh PJ, Augusto-Walkden J-A. Living without a safety net in an intelligent environment. ICST Trans Ambient Syst 2011;11(10–12):e6. http://dx.doi.org/10.4108/trans.amsys.2011.e6. [8] Augusto JC, Zheng H, Mulvenna M, Wang H, Carswell W, Jeffers P. Design and modelling of the nocturnal AAL care system. In: Proceedings of the second international symposium on ambient intelligence (ISAmI’11), Salamanca, Spain. Springer-Verlag; 2011. p. 109–16. [9] Beizer B. Software testing techniques. 2nd ed. New York: Van Nostrand Reinhold; 1990. [10] Ben-Ari M. Principles of the spin model checker. New York: Springer-Verlag; 2008. [11] Benghazi K, Hurtado MV, Hornos MJ, Rodríguez ML, Rodríguez-Domínguez C, Pelegrina AB, et al. Enabling correct design and formal analysis of ambient assisted living systems. J Syst Softw 2012;85(3):498–510. [12] Bérard B, Bidoit M, Finkel A, Laroussinie F, Petit A, Petrucci L, et al. Systems and software verification: model-checking techniques and tools. Berlin: Springer; 2001. [13] Boehm BW. A spiral model of software development and enhancement. Computer 1988;21(5):61–72. [14] Clarke EM, Grumberg O, Peled DA. Model checking. Cambridge: MIT Press; 1999. [15] Corno F, Sanaullah M. Design time methodology for the formal verification of intelligent domotic environments. In: Proceedings of the second international symposium on ambient intelligence (ISAmI’11), Salamanca, Spain. Berlin: Springer-Verlag; 2011. p. 9–16. [16] Coronato A, De Pietro G. Formal design of ambient intelligence applications. Computer 2010;43(12):60–8. [17] Coronato A, De Pietro G. Tools for the rapid prototyping of provably correct ambient intelligence applications. IEEE Trans Softw Eng 2011. http:// dx.doi.org/10.1109/TSE.2011.67.
[18] Coronato A, De Pietro G. Formal specification and verification of ubiquitous and pervasive systems. ACM Trans Auton Adapt Syst 2011;6(1):9:1–6. doi:1921641.1921650. [19] Dwyer MB, Avrunin GS, Corbett JC. Patterns in property specifications for finite-state verification. In: Proceedings of the twenty-first international conference on software engineering (ICSE’99). Los Angeles (CA): ACM; 1999. p. 411–20. [20] Emerson EA. Temporal and modal logic. In: van Leeuwen J, editor. Handbook of theoretical computer science. Formal models and semantics, vol. B. Amsterdam: Elsevier Science Publishers; 1990. p. 995–1072. [21] Gerth R. Concise Promela reference, 1997. (accessed 20.02.12). [22] Gerth R, Peled D, Vardi MY, Wolper P. Simple on-the-fly automatic verification of linear temporal logic. In: Proceedings of the fifteenth workshop on protocol specification testing and verification (PSTV’95). Warsaw, Poland: Chapman & Hall; 1995. p. 3–18. [23] Gluck PR, Holzmann GJ. Using spin model checking for flight software verification. In: Proceedings of the aerospace conference, big sky. MT (USA): IEEE; 2002. p. 1-105–13. [24] Holzmann GJ. 2001. From code to models. In: Proceedings of the international conference on applications of concurrency to system design (ACSD’01), Newcastle upon Tyne (UK): IEEE. p. 3–12. [25] Holzmann GJ. The SPIN model checker: primer and reference manual. Boston: Addision-Wesley Professional; 2003. [26] Holzmann GJ, Smith MH. Automating software feature verification. Bell Labs Tech J 2000;5(2):72–87. [27] Holzmann GJ, Smith MH. Software model checking: extracting verification models from source code. Softw Test Verif Reliab 2001;11:65–79. [28] Hornos MJ, Augusto J.C., 2012. Installation process and main functionalities of the Spin model checker. (accessed 22.02.12). [29] Kaner C, Falk J, Nguyen HQ. Testing computer software. 2nd ed. New York: John Wiley & Sons; 1999. [30] Larsen KG, Pettersson P, Yi W. UPPAAL in a nutshell. Int J Softw Tools Technol Transfer 1997;1(1–2):134–52. [31] Myers GJ, Sandler C, Badgett T. The art of software testing. 3rd ed. Hoboken: John Wiley & Sons; 2011. [32] Nakashima H, Aghajan H, Augusto JC, editors. Handbook on ambient intelligence and smart environments. New York: Springer; 2009. [33] Naumann JD, Jenkins AM. Prototyping: the new paradigm for systems development. MIS Quart 1982;6(3):29–44. [34] Patton R. Software testing. 2nd ed. Ontario: Sams; 2005. [35] Peled D. Software reliability methods. New York: Springer-Verlag; 2001. [36] Pnueli A. The temporal logic of programs. In: Proceedings of eighteenth IEEE symposium on foundations of computer science (FOCS’77). Providence, RI. IEEE; 1977. p. 46–57. [37] Pnueli A. Specification and development of reactive systems. In: Proc of the conference on information processing (IFIP’86). Dublin (Ireland): NorthHolland; 1986. p. 845–58. [38] Schneider K. Verification of reactive systems: formal methods and algorithms. Berlin: Springer-Verlag; 2004. [39] Silva-Anjos JM, Kisseloff-Coracini G, Villani E. A proposal and verification of a software architecture based on LabVIEW for a multifunctional robotic endeffector. Adv Eng Softw 2013;55:32–44. [40] Valmari A. The state explosion problem. Lect Notes Comput Sci 1998;1491:429–37. [41] Vardi MY, Wolper P. An automata-theoretic approach to automatic program verification (preliminary report). In: Proceedings of the first symposium on logic in computer science (LICS’86), Cambridge, MA. IEEE; 1986. p. 322–31. [42] Wolper P. An introduction to model checking. In: Proceedings of the software quality week, San Francisco, CA, 1995. (accessed 21.02.12).
