Visualizing Common Operating Picture of Critical Infrastructure Lauri Rummukainen*, Lauri Oksama, Jussi Timonen, Jouko Vankka Dept. of Military Technology, the Finnish National Defence University, Helsinki, Finland ABSTRACT This paper presents a solution for visualizing the common operating picture (COP) of the critical infrastructure (CI). The purpose is to improve the situational awareness (SA) of the strategic-level actor and the source system operator in order to support decision making. The information is obtained through the Situational Awareness of Critical Infrastructure and Networks (SACIN) framework. The system consists of an agent-based solution for gathering, storing, and analyzing the information, and a user interface (UI) is presented in this paper. The UI consists of multiple views visualizing information from the CI in different ways. Different CI actors are categorized in 11 separate sectors, and events are used to present meaningful incidents. Past and current states, together with geographical distribution and logical dependencies, are presented to the user. The current states are visualized as segmented circles to represent event categories. Geographical distribution of assets is displayed with a well-known map tool. Logical dependencies are presented in a simple directed graph, and users also have a timeline to review past events. The objective of the UI is to provide an easily understandable overview of the CI status. Therefore, testing methods, such as a walkthrough, an informal walkthrough, and the Situation Awareness Global Assessment Technique (SAGAT), were used in the evaluation of the UI. Results showed that users were able to obtain an understanding of the current state of CI, and the usability of the UI was rated as good. In particular, the designated display for the CI overview and the timeline were found to be efficient. Keywords: situational awareness, critical infrastructure, user interface design, common operating picture
1. INTRODUCTION In modern society, services like electricity, clean water, and Internet access might be taken for granted. An absence of any of these services reminds us just how important the continuity of and accessibility to some amenities are. Thus, industry actors, such as power plants, water supply companies, and telecommunication providers, form what we understand to be critical infrastructure (CI). These vital services face numerous threats over time, but our society is not defenseless. Individual industry actors acquire security systems of their own, and research is conducted to improve CI protection on national level. However, we cannot protect our CI if we do not know its current state and its potential threats. This raises a question: How do we improve our situational awareness (SA) of CI? This paper focuses on the question and proposes a solution to support human operators in getting to know the past and current condition of the most vital industry actors. Increased awareness of the surrounding environment helps human actors to make better decisions regarding their tasks. The main contribution of this research is the creation of a visualization environment—a user interface (UI)—that displays the state of CI. The UI was evaluated on how well it supports SA. The first steps taken to construct an initial UI were to search for similar implementations through literature review, as will be described in the Related Work section, and to identify a user group by interviewing a focus group. After the first draft, the UI was tested using visual and informal walkthroughs, as will be discussed in the Methods section, to get qualitative data on the attractiveness and usability of its implementation. A usability survey was also conducted to get a simple measure of the usability of the UI. Finally, after a series of improvements, the SA achieved with the UI was tested. The results derived from these user tests are discussed in the Results and Discussion section. The final UI that was implemented based on user test data is introduced in the SACIN User Interface section.
*
[email protected]; phone +358 400 670749
Next-Generation Analyst II, edited by Barbara D. Broome, David L. Hall, James Llinas, Proc. of SPIE Vol. 9122, 912208 · © 2014 SPIE · CCC code: 0277-786X/14/$18 doi: 10.1117/12.2050231 Proc. of SPIE Vol. 9122 912208-1 Downloaded From: http://proceedings.spiedigitallibrary.org/ on 01/15/2015 Terms of Use: http://spiedl.org/terms
The research described in this paper is part of a more extensive research project, the Digital Security of Critical Infrastructures (DiSCI), which aims to estimate and minimize threats facing the CI of a society. During the DiSCI project, the Situational Awareness of Critical Infrastructure and Networks (SACIN) framework was developed to evaluate the concept of CI monitoring1. The core idea of the SACIN framework is illustrated in figure 1: It is used to gather information from various industry source systems through a specific agent interface; it then processes and analyzes the gathered data and finally displays the result as a form of common operating picture that is the UI of the system1. Thus, any decision maker working with the source system can then monitor and gain an understanding of how well the system works and can affect its condition with any actions necessary1. In our research, we used the definition of Endsley2 for SA: “Situation awareness is the perception of the elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future.”2 This definition of SA involves three levels: perception of the relevant elements (level 1 SA), comprehension of their meaning (level 2 SA), and anticipation of what will happen in the near future (level 3 SA). The three-leveled definition fits well in the SACIN context as new events are processed through an analysis component with three similar levels.
Figure 1. SACIN1.
2. RELATED WORK In this study, the context used is a situation monitoring environment. A system based on the SACIN framework is intended to be used, for example, in military situation rooms or industry monitoring rooms1. A survey conducted by Pederson et al.3 in 2006 gives good insight into the state of CI modeling and visualization in different environments. The survey features three implementations that are intended for monitoring the entire CI similarly to SACIN: Athena, CIP/DSS4, and Fort Future5. Athena, which is implemented by On Target Technologies Inc., features a system quite similar to the SACIN framework as it is intended for users with varying backgrounds 3. The output of the Athena system is displayed in a graphical interface that represents directed graph. Hence, the main visible difference between the SACIN framework and the Athena is the UI as SACIN features more elements and different views. The CIP/DSS system, implemented by Los Alamos National Laboratory, is designed to assist decision makers in making informed choices3. However, only internal analysts at the laboratory are the intended users, narrowing the applicability of the system. Finally, the third implementation similar to SACIN is Fort Future, developed by the U.S. Army Corps of Engineers, Engineer Research and Development Center and Construction Engineering Research Laboratory 3. This implementation is used to simulate different scenarios. Compared to SACIN, Fort Future focuses on simulations, while
Proc. of SPIE Vol. 9122 912208-2 Downloaded From: http://proceedings.spiedigitallibrary.org/ on 01/15/2015 Terms of Use: http://spiedl.org/terms
SACIN is mainly intended for real-time monitoring. In addition, Kopylec et al.6 developed a system to analyze how physical events can affect cyber environments. Their system setup has three monitors: a disaster plan, a map interface, and a network topology display. Compared to SACIN, main differences are the order and number of the display monitors. Furthermore, Kopylec et al.’s implementation focuses more on the cyber infrastructure environment. No implementations were found that would have had a designated view for actor filtering and the overall status of the monitored elements. Such visual element was used in the SACIN UI and turned out to be an efficient tool to achieve initial SA.
3. SACIN USER INTERFACE The implemented UI consists of four distinguishable views, each displaying event data received from the SACIN framework’s analysis component. The UI setup is shown in figure 2. Each view serves a different purpose when creating SA: one to display a general overview of the monitored environment, one to display geographical distribution of events, one to display logical relationships between actors and, finally, one to display the temporal distribution of events. Individual visual elements include status circles, a timeline, a raw events log, a geographical map, and a logical map. As the displayed data are diverse, multiple display monitors were needed to make the UI more convenient compared to a UI with a single monitor7. Interactions in the UI are JavaScript-based and 10 third party libraries were used in constructing the UI. However, no commercial off-the-shelf products were used when constructing the UI as all libraries used in the implementation are available open source. 3.1 UI environment The UI itself was not the only aspect that was thought through in designing a monitoring system. It was also as important to think about how the individual elements should be placed on an operator’s workstation. The implemented display layout can be seen in figure 2. The most interesting view to users, the one with the timeline and the events log, is placed in the middle so the user is looking forward most of the time and not straining his or her neck. The overview display is placed on the left side and the map view on the right side to support the left-to-right style of reading. In an ideal situation, the operator first looks left to check whether all actors in the monitored infrastructure are working correctly, continues to the timeline view to check recent events, and finally turns to check the map to, for example, look for regional events. The display with logical dependencies is placed on the top of the three other displays as it is assumed to be used infrequently. It is placed so that it is barely seen in users’ vertical field of view so as to not disturb users’ attention from the other displays.
Logical relations display
00
-
.o
Overview display
Temporal display
o
Figure 2. UI environment. All displays are the same size.
Proc. of SPIE Vol. 9122 912208-3 Downloaded From: http://proceedings.spiedigitallibrary.org/ on 01/15/2015 Terms of Use: http://spiedl.org/terms
Map display
3.2 Overview display The overview is designed to offer a quick way to check whether all actors in the monitored critical infrastructure are working as they should. The display supports operators’ global SA by providing a big picture of the situation. It is a critical part of the system to support SA (see Endsley’s SA-oriented design principles8). The display layout, as seen in figure 3, consists of 12 different status circles. In our research, the understanding of CI, the dependencies in CI, and the key concepts of CI protection are based on a model by Lewis9. His model of CI is based on 11 separate industry sectors, and this categorization was used when developing the overview display. One extra sector was added for actors that do not necessarily belong to any other category. The sectors are power, water, information and telecommunication, banking and finance, transportation, chemical industry, defense industry, emergency services, food and agriculture, public health, and postal and shipping. xlaIANIJVS
snWisicuoncudo
dpwivuonc.aao
dew1el1lp1
dNy41,:
wqpuul
daxod
laleM
paUaP$ slua3e 5/1
paUalas slua2e £Jl
t 4± uoueuoAsueil
4IW pood q alnllnpu8e .. Ca2aa1as r
*. r4 leplwato dllsnpw
uoilewlo}ul $ suoueJNnwwoJalal
aauala0 ,Glsnpul
2uplue8 8 aweuy
Afwa3law3 . 2ePpl !
SAS pip.
*110
4IW 411ea41!14nd
8lelsod Rulddi4s pro:slua2e
1a410 pal)ans :slua2e oro
Figure 3. The overview that displays the overall state of CI actors.
Each of the CI sector icons has a status circle around it. An individual status circle, as seen in figure 4, is partitioned into six segments. These segments represent the Federal Agency Incident Categories10. This way, a human operator can achieve an overall understanding of what type of events are occurring in which industries. This is especially important for operators just beginning their shifts as they do not necessarily have prior knowledge of the state of the CI.
Proc. of SPIE Vol. 9122 912208-4 Downloaded From: http://proceedings.spiedigitallibrary.org/ on 01/15/2015 Terms of Use: http://spiedl.org/terms
Water Selected agents: 1/3
Unauthorized acce s
Investigation
ell
S c ans, probes
=
& ttempted
Denial of se ice
access
Malicious code
Improper use
Figure 4. An individual state circle. Category names are presented when a user hovers over any circle segment.
Visual cues are used to convey information about important events in the CI. A classic traffic light analog is used to display whether the CI actors are working alright, if they are having some difficulties, or if their services are severely malfunctioning. Gray-scale industry icons are also used to allow operators to identify industries easily while not interfering with the event colors in the status circle. The incident category names are not displayed in the UI to save space. However, the category names are shown when a user hovers over a segment. Also, as users might not always want to monitor the complete CI, they may click on the industry icons and select the actors they wish to monitor. This filtering option is illustrated in figure 5. In the example, the emergency services icon, the rightmost icon in the middle row in figure 3, is clicked and the user has the option to select which actors to follow.
EMERGENCY
Company
Site
Emergency actor #1
City
Emergency actor #2
City 1
Following
Emergency actor #3
City 2
Follow
1
Follow
Defence industry
Chemical industry Figure 5. Filtering options for emergency service actors.
Proc. of SPIE Vol. 9122 912208-5 Downloaded From: http://proceedings.spiedigitallibrary.org/ on 01/15/2015 Terms of Use: http://spiedl.org/terms
3.3 Temporal display As it is important to keep up with what has happened, it is equally important to know when events have occurred. Temporal information or awareness is also needed to project future changes in system behavior, the highest level of SA. According to Endsley’s guidelines for SA-oriented design, assistance is particularly needed for level 3 SA8. For this reason, one of the displays of the UI is dedicated to a timeline and an events log, which are illustrated in figures 6 and 7. This way, an operator using the system has a temporal view of the CI and has the ability to link consecutive events together even if there are no indications of connections between two entities in any other displays or external sources. When the UI receives new events, the timeline displays a new sequence line for every new CI actor. Every actor is placed on a separate line to prevent confusion between actors. The timeline uses a slightly different color analog for events, replacing the color green with light blue. This is to provide users a better chance in spotting warnings and alerts as they occur. The timeline is also scalable and helps to prevent information overload. As seen in figure 6, for example, events that occur near each other are grouped together and labeled with a number indicating the number of events. The color of the group element is defined by the most severe event it contains. Users can then zoom in, pan, or follow the current time on the timeline when necessary. Likewise, new events are also immediately shown on the common events log next to the timeline. The events log displays the raw data of the event and also has a hide button and a receipt button for interaction. Events can be highlighted by clicking on a specific row, and the corresponding event is shown in other views. This helps users to link between multiple monitors7. Track current time I^ Ok Events ® I Warning Events ® I Error Events 09:55:22 800
09:55:23 0
09:55:24 200
400
600
800
0
09:55:25 200
40357 - SimpleTestEvent (Denial of Service)
Power company (City 2)
warning Telecom company (City 1) 10359 - SimpleTestEvent (Investigation) a
Water supply (City 1)
800
600
0
40366 - SimpleTestEvel (Scans, probes & Attempted
n
40362 - Sim pleTestEvent (Scans, probes & Attempted Access)
ERROR
40365 - SimpleTestEve (Denial of Service)
I_
40358 - SimpleTestEvent ,, probes & Attempted Access)
40367 - SimpleTestEver (Unauthorized Access
Figure 6. Timeline. Events are displayed as individual boxes or grouped elements displaying the number of events in them.
Show 10
entries
Search:I Check
Id
Event
Agent
Severity
Category
Occurred
40365
SimpleTestEvent
Telecom
9
Denial of Service
29. marraskuuta 2013 9:55:25
Hide
3
Scans, probes & Attempted Access
29. marraskuuta 2013 9:55:25
Hide
0
Unauthorized
29. marraskuuta 2013 9:55:25
company
Hide
(City 1)
40366
SimpleTestEvent
Power company (City 2)
40367
SimpleTestEvent
Water supply (City
Access
1)
40360
SimpleTestEvent
2
Power company
4
Improper Use
29. marraskuuta 2013 9:55:24
Figure 7. Common events log that displays the raw event data that the UI receives.
Proc. of SPIE Vol. 9122 912208-6 Downloaded From: http://proceedings.spiedigitallibrary.org/ on 01/15/2015 Terms of Use: http://spiedl.org/terms
3.4 Map display The map display can be seen in figure 8. One aspect of SA in the CI context is to be aware of the geographical distribution of events and actors. The initial idea was to implement a map interface that would be easy to learn. Thus, a common interface type was used that resembles popular map interfaces, such as Google Maps. New events are displayed as markers in the map, and they can be clicked on to show more information regarding the event. In this view, the traffic light analog is also used to differentiate between event severities. Different actors are displayed with blue markers. After new events are displayed on the map, they are also highlighted, as illustrated in figure 8. Users may freely decide what types of markers are shown on the map. In our example, the user has hidden all markers except the error markers. Furthermore, the map interface also has a clustering feature to prevent information overload. Events are grouped based on their severity, and users have the option to zoom in on events to view more closely the distribution of events. In addition, though not shown in figure 8, the map interface is also capable of displaying areal events. A colored area is displayed on the map that works similarly to the event markers. SACIN View
Operational Status
Operational Map
Logical Map
Timeline
Hauldcamaki Pappilanvuor i
Li
Pa lokka ,
Mannisenmäld
Ok's
Warnings 710
Errors 51
,
-
Agents
tOla n pe Ito
Figure 8. Map. A new error event appears on the map. Every other marker type is hidden from the map.
3.5 Logical relations display One main goal for the SACIN system in general is to analyze dependencies between CI actors1. This way, risk analysis, for example, becomes easier. For this reason, one of the displays of the UI is dedicated to visualization of the logical dependencies between CI actors. The purpose of this display is to show actor dependencies in a directed graph so that an operator can easily form critical paths and estimate how long would it take for error events to propagate further. This display supports comprehension in level 2 SA and anticipation in level 3 SA8. The implementation of the directed graph can be seen in figure 9. The arrows in the graph presentation illustrate the service dependencies wherein the actor that is at the pointed end of an arrow is the dependent and the value next to the arrow indicates how long the dependent can function normally without the service source.
Proc. of SPIE Vol. 9122 912208-7 Downloaded From: http://proceedings.spiedigitallibrary.org/ on 01/15/2015 Terms of Use: http://spiedl.org/terms
Figure 9. Logical dependencies. Boxes represent infrastructure actors. Arrows represent any logical dependency. The time values next to the arrows represent the time a dependent can function without the source.
4. METHODS In total, two user tests were conducted during this research, both using different methods for evaluating the implemented UI. As the purpose of this research was to develop a novel UI to monitor the CI, test participants were required to be familiar with similar monitoring systems. Six participants were recruited from a military organization. All of them had previous experience in network or similar monitoring systems. Five of them were in their late twenties and the sixth was in his early thirties. It is also notable that though all were working for the military at the time of the tests, everyone volunteered. The six participants can be considered to be expert users as they already had the initial knowledge to analyze a variety of monitoring data. They were also familiar with the most common interfaces and visualization types, such as map interfaces and logical maps. 4.1 SAGAT The most important testing method during this research was the Situation Awareness Global Assessment Technique (SAGAT), and it was used as the main method of the second user test. It is described as an objective method to evaluate the SA achieved with a given system11. During SAGAT, the evaluated system is presented to the user, and a simulation is run. At random intervals, the simulation is frozen, all screens are blanked, and the user has to answer a series of questions about the current situation of the monitored actors11. After the test, answers to each question can be analyzed to see if the user was able to form an understanding of the current state of the environment. The SAGAT method tries to objectively measure SA but does require short-term memory. The method has been criticized for this12, though it has been shown to have no effect on user performance 13, 14. Also, SAGAT suggests creating queries through a goal-driven task analysis15. Unfortunately, it was not possible to conduct a thorough task analysis during this research. During the second user test, along with SAGAT, participants’ eye movements were also monitored with eye tracking glasses. This way it was possible to gather quantitative data of the most interesting elements in the UI and to see if any particular eye movement paths helped users in achieving SA. 4.2 Visual walkthrough Besides SAGAT, complementary testing methods used during the research included visual and informal walkthroughs. Both of these were used during the first user test. A visual walkthrough has been described as a quick way to identify the important and most appealing features of a UI16. During a visual walkthrough, a test participant is given pictures or snapshots of the system. Then, he or she describes what kinds of elements there are in the system and what he or she thinks they mean.
Proc. of SPIE Vol. 9122 912208-8 Downloaded From: http://proceedings.spiedigitallibrary.org/ on 01/15/2015 Terms of Use: http://spiedl.org/terms
4.3 Informal walkthrough As the visual walkthrough method is not intended to be used as the sole method for a user test16, an informal walkthrough was used to complement it. An informal walkthrough is described as a way to evaluate a UI when specific test tasks are not appropriate due to the context or the phase of development17. In an informal walkthrough, a test participant is given the opportunity to navigate in the UI freely. The aim of the informal walkthrough is that all major features are tested during the informal walkthrough17. 4.4 System Usability Scale To complement the described methods, the System Usability Scale (SUS) was used to gather quantitative data of the UI’s usability18 both during the first and second user tests. In a SUS query, users are presented 10 statements about the evaluated UI. Combined statements produce a single value ranging from 0 to 100 that represents the overall usability of the system18. 4.5 First user test setup The aim of the first user test was to find out whether people with experience with such systems thought the UI could be useful in their work. The idea of the test was also to find out whether the development of the UI had taken the right direction regarding a user-centered design. The first user test was held after four weeks of initial UI development. Because the development was still in a rather early stage, visual and informal walkthroughs were used to evaluate the then current prototype. First, a visual walkthrough was used as a warm up task. After the visual walkthrough, an informal walkthrough was used as the main testing method. Finally, the test participants were asked to answer the SUS survey. The user tests were conducted in a closed environment with no significant background noise. 4.6 Second user test setup The second user test was set up to assess how well the UI actually helps a user in performing his or her work as a CI monitoring operator. The second user test was held four weeks after the first session. Besides evaluating the usability with the SUS scale, the SAGAT method was used to define how well the UI provides the user with information about the monitored CI. The display layout used in the second user test was constructed similarly to the one in figure 2, with the exception that the timeline was in the rightmost display and the map display was in the middle. The initial test setup including a test participant is illustrated in figure 10. In addition, some features presented in the implementation section were not implemented by the time of the second user test and were added after the test feedback. The tests took place in a designated testing room at the Department of Military Technology at the Finnish National Defence University.
Figure 10. Second user test initial setup: a test participant with eye tracking glasses.
Proc. of SPIE Vol. 9122 912208-9 Downloaded From: http://proceedings.spiedigitallibrary.org/ on 01/15/2015 Terms of Use: http://spiedl.org/terms
5. RESULTS AND DISCUSSION 5.1 Visual and informal walkthrough During both the visual and the informal walkthroughs, a vast number of qualitative data were gathered about the visual appearance and the usability of the UI. A summary of the most important findings is presented in table 1. Two users participated in the first user test. As described, in a visual walkthrough, users are given pictures of the UI and asked what they see and how they interpret the UI to work. In the first user test, when shown a picture containing the actor filtering and overall status, similar to figure 3, the participants commented that the image represents an overview of some kind. By the time of the first user test, the 12 status circles shown in figure 3 were presented as a single circle. It was noted that besides being an overview, for example, for a network operator, the view would offer a good glimpse of CI status for an executive-level manager. A picture containing the map view, similar to figure 8, was quickly interpreted as displaying the geographical distribution of the error events. Participants explained that they had interacted with such interface, so it offered nothing new in general but was considered an essential part of a monitoring system. The third picture that was given to the participants represented the logical view, similar to figure 9. Participants were not confident about the actual meaning of the picture, though both implied that it displayed some sort of logical relationships. Finally, the display with the timeline, as shown in figure 6, did not raise any particular thoughts or opinions, but both participants did recognize that the timeline showed recent events. During the informal walkthrough, participants had the opportunity to try out the features of the UI freely. Initially, new error events were visualized as flashing circles in the middle of a single status circle. Both users stated that this feature was annoying. Though acknowledging that the flashing red circle would stimulate their peripheral vision and draw their attention to the circle, they would not want the feature in a live situation. The logical view of the UI (figure 9) raised mixed opinions. The feature was considered essential for an operator’s job, but the implementation was not considered to be of top quality. According to the participants, the dependency arrows were too hard to see in the directed graph, and the impact of new events on the dependency map was considered to be unclear. Overall, the participants commented that the logical view felt a bit disconnected from the other views of the UI. When going through the map (figure 8) and especially the timeline (figure 6), the users commented that the most important events should be easily noticeable. At this point, both users were aware that there was a filtering option for events with different severities, but they did not initially employ this option. This might signify that users are willing to see everything possible in the UI, but the most dangerous events should be somehow highlighted so that they become more noticeable so as to direct the users’ focus. By the time of the first user test, the UI did not have a raw events log for new events, and both users did note it as a lacking feature. They stated this to be a must-have element. The raw events log was later added to the implementation (figure 7). Before the tests, special interest was given to the opinions regarding the overview display (figure 3) as it was purely experimental and not based on any implementation found in the literature review. As we learned, it gained very positive feedback and can be considered one of the key elements when trying to achieve initial SA. Though there was no objective measurement of users’ SA levels at this point, their opinions suggested that this could function as an executivelevel overview to help assess the top-level situation in the CI. Table 1. Main remarks made by test participants during the visual and informal walkthroughs.
Display
Visual walkthrough
Informal walkthrough
Overview display (figure 3)
Good tool for an executive-level manager.
A flashing circle to indicate errors would be annoying.
Temporal display (figure 6)
No particular thoughts.
A raw events log would complement the timeline.
Map display (figure 8)
Familiar and essential display for a monitoring system.
New events should be easily noticeable.
Logical relations display (figure 9)
Identified as a logical map, but its purpose was not intuitive.
It is important in an operator’s job to know the dependencies. The implementation was not of the highest quality.
Proc. of SPIE Vol. 9122 912208-10 Downloaded From: http://proceedings.spiedigitallibrary.org/ on 01/15/2015 Terms of Use: http://spiedl.org/terms
5.2 System Usability Scale The System Usability Scale served as a quick method to see if everything was alright in the UI. The SUS method is intended to offer a subjective assessment of usability18. The SUS survey was conducted during both user tests, and a total of six people answered the survey—two during the first and four during the second user test. The mean SUS score after the first user test was 77.5, and the mean score for the second was 71.25, as can be seen in figure 11. Though the SUS scores themselves give a good idea as to how usable the UI is, an adjective scale clarifies the performance by giving a linguistic measure between “worst imaginable” and “best imaginable”19. On this scale, the developed UI assigns a grade of “good,” which is the third highest rating on a seven point scale. This implies that users are willing to use the UI with little to no problems. Most comments during the second SUS survey noted that the system itself was simple and easy to learn but would require a short introduction by a system expert. Some participants also stated that the UI was simpler than the ones they had used in the military. This might have had a positive effect on the score if the participants compared the tested UI to their previous bad experiences with similar UIs. 100 90 80
i
T
User test. 2
Combined
70 60 50
40 30
20 10 0 User test 1
Figure 11. Mean SUS scores. Error bars represent the 95% confidence intervals.
5.3 Situation Awareness Global Assessment Technique SAGAT was used to test how well the UI supports SA levels 1 and 2. The initial goal of the UI was to offer users an understanding of the current situation, so the research focused on testing levels 1 and 2. The third level, the ability to predict the future status of actors, was not included in the SAGAT queries because the prediction section of the SACIN analysis component was still under development. Furthermore, the sector categorization that is presented in the overview display prevents the user from predicting all industries. For example, a user working in a water supply company most likely has no knowledge of how their problems propagate to completely different industries, such as the chemical industry or finance sector. The SAGAT results are shown in figures 12, 13, and 14. As presented in figure 12, of all error events, the exact location of each event was recalled with a 60% success rate. The success rate means that, for example, out of 10 error event messages, participants were able to pinpoint six on a map, and the other four were either forgotten or misplaced. Warnings and “ok” messages were recalled with approximately 41% and 49% success rates, respectively. The users were also asked where the monitored actors were located, and they were recalled with an approximate success rate of 77%. The data imply that the test participants were able to create a mental model of the situation with the help of the UI. A surprising finding was that each color in the six segmented circle, which is shown in figure 10, was remembered with a success rate of 75%, even though only 15% of total eye dwell time was spent on the status circle. This means that although users spent less time looking at the status circle, they were able to recall the exact color of four or five segments
Proc. of SPIE Vol. 9122 912208-11 Downloaded From: http://proceedings.spiedigitallibrary.org/ on 01/15/2015 Terms of Use: http://spiedl.org/terms
on any given freeze. Apparently, the overview display works very efficiently: The participants were able to create and maintain an accurate global SA. The participants were also asked about single events and their five attributes, which were the actor name, site, event name, event severity, and event category. These attributes could have been noticed, for example, from the event log (see figure 7). On average, 50% of the single event attributes were remembered correctly. The single events asked about during the query consisted of the latest events from a specific actor or the most severe events between SAGAT freezes. As can be seen in figure 13, the average percentage of correct answers increased during the tests. This can be interpreted as an increase in SA. When the questions were divided into SA levels 1 and 2, there was an ascending trend in level 1, while level 2 remained relatively the same, as illustrated in figure 14. This suggests that the elements in the UI are easily noticeable. To summarize the SAGAT results, the data and the qualitative feedback support the claim that the UI helps users achieve better SA. 100
90
L
80 70
i
60
1
50 40 30 20 10
0
1
Warning event locations Actor locations Single event attributes Ok event locations Status circle segments Error event locations Figure 12. Mean success rates of different elements. Error bars represent the 95% confidence intervals.
100
90 80 70 60 50
40 30
20 10
0 Freeze #1
Freeze #2
Freeze
Figure 13. Average score percentage on each freeze. Error bars represent the 95% confidence intervals.
Proc. of SPIE Vol. 9122 912208-12 Downloaded From: http://proceedings.spiedigitallibrary.org/ on 01/15/2015 Terms of Use: http://spiedl.org/terms
Freeze #4
100
90
80 70
60 SA level 1
50
SA level 2
Ti
40 30 20 10 0
1
r
Freeze #1
b. Freeze #2
Freeze #3
Freeze #4
Figure 14. SAGAT score percentages on each freeze. Error bars represent the 95% confidence intervals.
5.4 Eye tracking data The eye tracking glasses provided an overview of how the test participants went through the UI. The areas with greatest dwell times were the newest events on the timeline (seen in figure 6) and the event and agent description in the events log (seen in figure 7). After each session, all participants commented that they felt their main display was the one with the timeline. This is supported by the eye tracking data as approximately 42.5% of the total dwell time was spent on the said display. The average dwell times are shown in figure 15. It was also stated that the timeline was the most interesting element in the UI and that the events log was the easiest view for keeping track of new events. It is notable that the timeline display was initially placed as the rightmost display but still drew the most attention. Because of this, it was placed in the middle as described in the implementation section. As the map display was initially placed in the middle, as shown in figure 10, it can be assumed that at least a portion of the total dwell time derived from the center position. 0,7% 17,1%
Overview
Map
42,5%
Timeline Logical
39,7%
Figure 15. Mean dwell times for each display. Percentages represent the time users were looking at specific displays.
Proc. of SPIE Vol. 9122 912208-13 Downloaded From: http://proceedings.spiedigitallibrary.org/ on 01/15/2015 Terms of Use: http://spiedl.org/terms
6. CONCLUSIONS AND FURTHER RESEARCH The SAGAT results showed that the test participants were able to achieve an initial reasonable understanding of the monitored CI. The confidence intervals for the results are broad, but the feasibility of the UI is supported by the qualitative feedback given by the participants. Most of them stated that the UI is easy to learn and use, and it is considerably better than some UIs they have used in the military. It is possible that the military background of the test participants may have had some effect on the final results, especially the SUS score. The user tests made it evident that the display containing the timeline and the raw events log was considered more interesting than the one with the interactive map. Though the mean dwell times for each display were similar to each other, it is likely that the map interface drew more attention because it was the center display. That the timeline display was deemed most interesting is also supported by the qualitative feedback given by the test participants. Thus, in our implementation, the timeline and the raw events log should be placed in the middle display and the map should be placed in the rightmost display. Initially, the overview display had a single status circle to represent the state of the entire CI. It was noted during the user tests that the display is a good tool to create an overall understanding of the system but that a single circle was considered a bit too abstract of an element. Hence, each industry sector was given a status circle, as was presented in figure 4. As stated earlier, the SACIN framework is still under research, and more interactivity will be added to the UI to make the system more than just a monitoring tool. This serves as a basis for supplementary user tests, and the improved version can be compared to this initial prototype. The current research phase was focused on providing SA levels 1 and 2 through the UI. Further research will focus more on testing the third SA level, which presents an interesting and challenging aspect to developing a CI monitoring tool.
ACKNOWLEDGMENTS We would like to thank Pirkko Oittinen, Mikko Kuhna, and the other personnel of Aalto University Department of Media Technology for their valuable comments regarding the research and for lending the eye-tracking glasses used in the user tests. We would also like to thank Lauri Lääperi from the Finnish National Defence University for his help throughout the research and for his feedback on this paper.
REFERENCES [1] Timonen, J., Lääperi, L., Rummukainen, L., Puuska, S., and Vankka, J., “Situational awareness and information collection from critical infrastructure,” in [CyCon 2014, to appear], NATO CCD COE (2014). [2] Endsley, M. R., “Toward a theory of situation awareness in dynamic systems,” Human Factors: The Journal of the Human Factors and Ergonomics Society 37(1), 32–64 (1995). [3] Pederson, P., Dudenhoeffer, D., Hartley, S., and Permann, M., “Critical infrastructure interdependency modeling: a survey of US and international research,” Idaho National Library, 1–20 (2006). [4] Bush, B., Dauelsberg, L., LeClaire, R., Powell, D., Deland, S., and Samsa, M., “Critical infrastructure protection decision support system (CIP/DSS) project overview,” Proceedings of the 23rd International Conference of the System Dynamics Society, July 17–21 (2005). [5] Case, M. P., Smith, W., and Grobler, F., “Fort Future: Modeling and simulation for collaborative multi-criteria decision support,” in [Computing in Civil Engineering (2005)], 1–10, ASCE (2005). [6] Kopylec, J., D’Amico, A., and Goodall, J., “Visualizing cascading failures in critical cyber infrastructures,” in [Critical Infrastructure Protection], 351–364, Springer (2007). [7] Wang Baldonado, M. Q., Woodruff, A., and Kuchinsky, A., “Guidelines for using multiple views in information visualization,” Proceedings of the working conference on Advanced visual interfaces, 110–119, ACM (2000). [8] Endsley, M. R., “Situation awareness-oriented design,” [The Oxford Handbook of Cognitive Engineering], 272 (2013). [9] Lewis, T. G., [Critical Infrastructure Protection in Homeland Security: Defending a Networked Nation], Wiley (2006).
Proc. of SPIE Vol. 9122 912208-14 Downloaded From: http://proceedings.spiedigitallibrary.org/ on 01/15/2015 Terms of Use: http://spiedl.org/terms
[10] United States Computer Emergency Readiness Team, “Federal Incident Reporting Guidelines,” (n.d.). [11] Endsley, M. R., “Situation awareness global assessment technique (SAGAT),” Aerospace and Electronics Conference, 1988. NAECON 1988, Proceedings of the IEEE 1988 National, 789–795, IEEE (1988). [12] Sarter, N. B. and Woods, D. D., “Situation awareness: A critical but ill-defined phenomenon,” The International Journal of Aviation Psychology 1(1), 45–57 (1991). [13] Endsley, M. R., “Measurement of situation awareness in dynamic systems,” Human Factors: The Journal of the Human Factors and Ergonomics Society 37(1), 65–84 (1995). [14] Endsley, M. R., “A methodology for the objective measurement of pilot situation awareness,” AGARD, Situational Awareness in Aerospace Operations 9, SEE N 90-28972 23-53 (1990). [15] Endsley, M. R., “Direct measurement of situation awareness: Validity and use of SAGAT,” Situation Awareness Analysis and Measurement 10 (2000). [16] Nieminen, M. and Koivunen, M., “Visual walkthrough,” HCI, 95, 86–89 (1995). [17] Riihiaho, S., “User testing when test tasks are not appropriate,” European Conference on Cognitive Ergonomics: Designing beyond the Product-Understanding Activity and User Experience in Ubiquitous Environments, 21, VTT Technical Research Centre of Finland (2009). [18] Brooke, J., “SUS—a quick and dirty usability scale,” Usability Evaluation in Industry 189, 194 (1996). [19] Bangor, A., Kortum, P. T., and Miller, J. T., “An empirical evaluation of the system usability scale,” Intl. Journal of Human-Computer Interaction 24(6), 574–594 (2008).
Proc. of SPIE Vol. 9122 912208-15 Downloaded From: http://proceedings.spiedigitallibrary.org/ on 01/15/2015 Terms of Use: http://spiedl.org/terms
