Situation Awareness Requirements for a Critical Infrastructure Monitoring Operator Lauri Rummukainen∗ , Lauri Oksama, Jussi Timonen, and Jouko Vankka Department of Military Technology, National Defence University, Helsinki, Finland ∗ E-mail:
[email protected]
Abstract—This paper presents a set of situation awareness (SA) requirements for an operator who monitors critical infrastructure (CI). The requirements consist of pieces of information that the operator needs in order to be successful in their work. The purpose of this research was to define a common requirement base that can be used when designing a CI monitoring system or a user interface to support SA. The requirements can also be used during system or user interface evaluation, and as a guide for what aspects to emphasize when training new CI monitoring operators. To create the SA requirements, goaldirected task analysis (GDTA) was conducted. For GDTA, nine interview sessions were held during the research. For a clear understanding of a CI monitoring operator’s work, all interviewees were subject matter experts (SMEs) and had extensive experience in CI monitoring. Before the interviews, a day-long observation session was conducted to gather initial input for the GDTA goal hierarchy and the SA requirements. GDTA identified three goals an operator must achieve in order to be successful in their work, and they were used to define the final SA requirements. As a result, a hierarchy diagram was constructed that includes three goals: monitoring, analysis and internal communication, and external communication. The SA requirements for a CI monitoring operator include information regarding ongoing incidents in the environment and the state of systems and services in the operator’s organization.
I. I NTRODUCTION In a modern society, situation awareness (SA) is critical for various level decision makers in all industries. During a harmful incident, SA is needed to make more informed and better decisions to overcome difficulties in organization operations. In addition, decision makers need good situation awareness to maintain and improve their organizations’ operational performance. Situational information can be conveyed via monitoring systems or other people, so it is important to study the most critical elements in providing SA. The main contribution of this research was the creation of situation awareness requirements for a critical infrastructure (CI) monitoring operator. These requirements include the information an operator needs to maintain SA of the state of the CI. The requirements can thus be used as the basis when designing monitoring systems. The operator’s goal hierarchy was also identified during the research process. The research was conducted using goal-directed task analysis (GDTA), which will be described in section III. The research results from GDTA can be divided into two categories: the operator goal hierarchy and the situation awareness requirements. These topics will be discussed in sections IV and V, respectively. The meaning of 978-1-4799-1737-2/15/$31.00 ©2015 IEEE
the results and suggested future topics will be discussed in sections VI and VII. The research described in this paper is part of a bigger research project, the Digital Security of Critical Infrastructures (DiSCI), that aims to estimate and minimize threats facing the CI of a society. During the DiSCI project, the Situational Awareness of Critical Infrastructure and Networks (SACIN) framework was developed to evaluate CI monitoring [1]. The framework was used to gather information from various industry source systems through a specific agent interface, process and analyze the gathered data, and then display the result as a form of common operating picture that is the UI of the system [1]. Thus, any decision maker working with the source system can then monitor and gain understanding of how well the system works, and can affect its condition with any actions necessary [1]. The outcome of the research described in this paper was used to evaluate the role of decision makers in the DiSCI context, as will be discussed in section VI. In our research, our understanding of SA is based on Endsley’s three-level definition: “Situation awareness is the perception of the elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future” [2]. Our research method, goal-directed task analysis, is based on Endsley’s definition of SA, and is intended to identify the information necessary for achieving all levels of SA [3]. Our understanding of CI is based on Lewis’s categorization [4]. He describes CI as an interconnected network of 11 industry sectors that include power and energy, water supply, and information and telecommunications. Regarding CI monitoring, our definition of incidents is based on the National Initiative for Cybersecurity Careers and Studies’ definition: “Incident: an occurrence that actually or potentially results in adverse consequences to (adverse effects on) (poses a threat to) an information system or the information that the system processes, stores, or transmits and that may require a response action to mitigate the consequences” [5]. II. R ELATED W ORK GDTA has previously been adapted to various CI environments, including smart grid operations [6], facility safety management [7], and army brigade officers [8]. Each study examined monitoring as part of the GDTA. However, to our knowledge, the GDTA method has not been applied to CI
environment monitoring as a whole. This research combines all CI sectors and suggests a common SA requirements base for all environments used to monitor the CI. III. M ETHOD During the research, GDTA was conducted. The purpose was to gather data about CI monitoring and to form the basis for situation awareness requirements. GDTA is generally used to identify the various pieces of information operators need in their work [3]. During GDTA, a series of semistructured interviews is conducted [3]. The interviewees for GDTA are subject matter experts (SMEs) who have extensive knowledge of an operator’s job. The interviews conducted during this research were neither audio or video recorded as it might have affected the interviewees’ opinions [9]. Since the interviews were also intended to be semi-structured, the absence of recording equipment made the interview sessions more relaxed [9]. Although the interviews were not digitally recorded, extensive notes were taken during each session. The notes were later organized based on their topic, and the goal hierarchy was constructed based on them, as suggested by Endsley [3]. In total, 12 people from nine organizations were interviewed. When the interviewees were categorized based on the sector categorization described by Lewis [4], seven sectors were covered. The industries include power, water, information and telecommunication, banking and finance, transportation, chemical industry, and emergency services. As illustrated in Fig. 1, these interviews effectively cover the first and second levels of CI. All interviews were held in each interviewee’s organization’s facilities to ensure as natural an atmosphere as possible. All interviewees had experience in their organization’s monitoring environment, and seven were security managers in their organization. Since the interviews were semi-structured, the topics included organizations’ service dependencies, system monitoring, and information communication during an incident. The interviews lasted between one and two hours. Before the interviews, a day-long observation session was conducted. The session was held in an emergency service situation room, which fits the CI monitoring context. A contextual interview was conducted during the observation session, and the operators were asked to describe their work, their information requirements, and opinions about monitoring work. The same topics were used for discussion as for the following interviews. Several incidents occurred during the observation session, and one employee shift change was observed. IV. O PERATOR G OAL H IERARCHY During the observation session, communication was recognized as the most essential part of a monitoring operator’s work. The observed monitoring operator stated that though situation awareness can be achieved during an incident, the information is useless if it is not communicated forward. In the emergency service case, the operators must forward the information to their superior if needed. The importance of
Fig. 1. Illustration of the critical infrastructure sectors that were covered during interviews. The red outline indicates that a subject matter expert in that sector was interviewed, and a gray outline indicates that a subject matter expert in that sector was not interviewed. The levels shown in this figure should not be confused with the three situation awareness levels.
communication was later confirmed by all the interviewees, and had a major impact in the final goal structure. In most situations, the operator responsible for keeping up to date with the situation is not the same person who makes the necessary decisions to recover from the situation. The situation information is always passed forward, and the operator is responsible for keeping every involved party up to date with the latest information. As a result of the interviews, a GDTA goal hierarchy was constructed. The hierarchy consists of one major goal and three main goals, two of which have subgoals, as illustrated in Fig. 2. The major goal for a critical infrastructure monitoring operator is to increase the situation awareness of critical infrastructure. This high-level goal can be divided into three main goals: monitor the organization environment, analyze incidents and communicate internally, and communicate externally. The three main goals were initially constructed based on the observation session and the first four interviews, and they were later iteratively improved by the following interviewees. The three main goals were also divided into subgoals as illustrated in Fig. 2. The subgoals of the first main goal are: (1) identify harmful incidents and (2) monitor resource management. The operators must achieve these goals to have good SA of the state of the CI. Thus, they need information about the current systems and services through organization resource management, and they need to identify and know about current incidents in the monitored physical or cyber environment. No subgoals were identified for the second main goal. The analysis and internal communication were included in the same goal because, according to the interviewees, operators do not necessarily analyze the gathered situation information; they just forward it. In every case, the interviewees noted that the information would be forwarded to people mostly consisting
Fig. 2. Task hierarchy for a critical infrastructure monitoring operator. Tasks apply to a monitoring operator’s work in general.
of various decision makers. The operators analyze the information themselves only if they have enough knowledge of the monitored environment and are at least partially responsible for its performance. The subgoals for the third main goal, external communication, are the following: (1) determine to whom the incident information should be communicated to, and (2) determine the incidents that are worth mentioning. Based on the interviewee data, it is important to separate the types of communication the operator is responsible for. Regarding external communication, the operator needs to decide which organizations need to know about an incident and what information can be forwarded to them. This differs from internal communication since usually the operator does not have to think about what they can say and what they cannot say regarding organization confidentiality policies or other similar factors. Furthermore, the operator must determine what information must be forwarded since not everything is relevant to the recipients and they might not understand the technical details of the operator’s monitored environment. Multiple tasks and goals were not included in the final goal hierarchy. The most notable type of task left out was related to the operator’s operational tasks. These include facility controls and other features related to operating the system in the organization. The tasks were not included because not every monitoring operator is responsible for operating the monitored system, so those tasks belong to a different employee role. This is also the case when considering decision making. Since the operators do not necessarily have the possibility to make operational decisions themselves, the operational goal can be left out of the hierarchy. This makes the hierarchy more applicable for other environments, too.
V. S ITUATION AWARENESS R EQUIREMENTS
The purpose of GDTA is to discover the information operators need in their work to achieve the goals defined in the goal hierarchy [3]. After constructing the goal hierarchy and analyzing the interview data further, two distinctive blocks of information can be found. The first type is incident-related information. This includes all the possible information that can be gathered during an incident. Incident in this case refers to a situation in which some fault, natural event, or deliberate attack causes a warning or a fault error in a target system [5]. The other type of distinctive information found during the research was system related. This means system characteristics that include system status, its purpose, and service dependencies. The interviewees noted that most of the information about other organizations is merely “nice-toknow” and does not serve a specific purpose. Most of the interviewees stated that they have no practical use for the information, but still thought it was good for their situation awareness. Although the information blocks consist of various necessary materials for decision makers, not everything needs to be present at all times. In the following subsections, the SA requirements in each block are divided into Endsley’s SA levels [2]. The first level, perception, represents the initial information a person needs in their work and does not require any mental processing. The second level, comprehension, represents the result of the person’s evaluation of the current situation. The third and highest level, projection, represents the information that requires heavy mental processing and results in estimation about future events.
Fig. 3. The situation awareness requirements of a critical infrastructure monitoring operator. Each block represents the subgoals presented in Fig. 2.
A. Incidents Incidents serve a specific purpose in the research, since several types of information that must be gathered after every incident. If the incident was caused by another organization, the interviewees agreed that initially they would like to know about the incident according to three Ws: “What has happened?,” “When did it take place?,” and “Where did it happen?” The three Ws form the first level of SA of an incident. To supplement the initial information, the interviewees gradually want to know more about the situation. The complete list of information requirements for a generic incident is: SA Level 1 - Perception – Short vernacular description – Time of occurrence – Location SA Level 2 - Comprehension – Magnitude of incident – Reason behind incident – Relation to other incidents – Whom to contact about the incident – Reliability of incident SA Level 3 - Projection – Duration of incident (or estimation) – Trend (is it escalating or diminishing?) – Effect of incident B. Organization Systems and Services During the research, another important component in SA requirements was the information regarding the operator’s own organization’s systems and services. Although the CI
monitoring operators are not necessarily decision makers, they need information about the organization to provide the decision makers with better SA. Furthermore, the decision makers are not necessarily up to date regarding the system and service status so the CI monitoring operators need to forward that information, too. The complete list of information requirements for a generic system or service is: SA Level 1 - Perception – Location – Purpose of the system or service – Contact information of responsible people – System and service resource requirements SA Level 2 - Comprehension – Operational status – Security status – Priority and criticality – System and service dependencies SA Level 3 - Projection – Sufficiency of critical resources C. Final Situation Awareness Requirements The incidents and service information can be used in the final situation awareness requirements. The complete SA requirements are illustrated in Fig. 3. The requirements for the monitoring goal include ongoing incidents, services and systems affected by the incidents, their priority, and possible future incidents. These requirements were all verified in the interviews: An operator needs to be aware of all the incidents and service statuses in the monitored environment to be able to coordinate repairs or communicate the situation efficiently. The SA requirements for the second main goal include the elements needed in the monitoring task. However, an operator
is also responsible for communicating with other people to create a common and shared SA. The operator may also analyze current incidents with other people. The requirements related to external communication share similarities with the other two main goals. However, in external communication the operators must determine more carefully to whom to forward information. Therefore, they need to know the communication agreements between organizations and evaluate the types of contact. In addition, the operators might need to evaluate whether further collaboration is required between organizations. VI. D ISCUSSION A. Reliability The proposed goal structure and SA requirements were supported by the interviews. Thus, the formed SA requirements and task goals are highly reliable because of the uniformity of the interview answers. This research identified the SA requirements for a generic monitoring operator, but in order to apply these requirements to a real system, national security strategies must be taken into account. Furthermore, as noted in section III, not all CI sectors, as illustrated in Fig. 1, were represented in the interviews. Additional interviews with representatives from these sectors might affect the goal hierarchy, though it can be seen unlikely since most of the interview answers were uniform across sectors. B. Operator Communication Although all interviewees mentioned that communication is essential in an operator’s work, not many incident management or monitoring systems had a communication module installed. For communication, the interviewees relied mainly on mobile phones, and this could cause problems if the mobile network crashed. Mobile communication was also described as nonstructured and slightly complicated. Thus, future research should be directed more toward communication. C. Operator Proactivity Based on the interviews, it also appears that most if not all monitoring operators are reactive rather than proactive. Most common remarks regarding incident evaluation indicated that the operators have no means of predicting future incidents. This means that the operators are unable to proactively spot possible incidents so they only react to new ones. Most interviewees noted that increasing resilience based on information from other systems would be challenging. In addition, although they knew what would happen to organizations on which they are depended, the operators could not prepare for the incidents beforehand. However, mental preparation helps to adjust the mindset right if something actually happens. The preparation helps users to adjust their thinking according to Endsley’s SA level 3.
D. Information Importance and Priority The interviewees observed that sometimes information is more of nice to know rather than of real value. Although decision makers would ideally like to know all information, it might intervene in the process as the most important information gets buried under the nice to know information. Many interviewees stated that during a crisis situation, the atmosphere in a monitoring room is intense. Thus, it is not wise to introduce extra information that has no direct real value. Furthermore, the communication strategy should be thought about before, and the overall amount of information should be examined and reduced to a level where only necessary elements are present. This supports the need for further research in communication between organizations. E. Other Operator Tasks Finally, it is important to separate two tasks in an operator’s work. As several interviewees stated, most often the operator needs to monitor the environment and execute incident-related tasks. This research identified only the tasks related to monitoring and SA. This is because it is not necessarily possible to create a uniform task hierarchy of incident management since every organization has its own method of handling situations. In our earlier research, we stated that decision makers need improved situation awareness in order to make better decisions [1]. However, as we found during this research, most organizations do not directly provide situational information to the decision makers but use an operator in between. This difference should be taken into account when designing a user interface for a monitoring system. F. SA Requirements in DiSCI Context In the context of the DiSCI project [1] [10], protecting the critical infrastructure is the key mission. When the results of this research are applied to the DiSCI context, the operator’s task can be described as an intermediator who focuses on building the right common operating picture (COP) for the associated entities. The elements and relations of operator’s goals and SA requirements in the DiSCI context are illustrated in Fig. 4. The goals for the operator, as illustrated in Fig. 2, can be applied to the DiSCI context to clearly define the role differences between operators and decision makers. VII. C ONCLUSIONS In a critical infrastructure organization, an operator is responsible for monitoring the state of their and partially other organizations’ systems and services. If a harmful incident occurs, the operator is responsible for informing the relevant decision makers about the incident. During this research, goaldirected task analysis was conducted to identify the situation awareness requirements in an operator’s work. These requirements consist of information that the operators need to perform well in their work. In total, 12 subject matter experts from nine organizations were interviewed during the research. The interviews showed that communication is the most essential part of an operator’s work. If an incident occurs and an
[7] J. Irizarry and M. Gheisari, “Situation awareness (sa), a qualitative usercentered information needs assessment approach,” Int. J. Constr. Manag., vol. 13, no. 3, pp. 35–53, 2013. [8] C. A. Bolstad, J. M. Riley, D. G. Jones, and M. R. Endsley, “Using goal directed task analysis with army brigade officer teams,” in Proc. Hum. Factors Ergo. Soc. Ann. Mtg., vol. 46, no. 3. SAGE Publications, 2002, pp. 472–476. [9] M. N. Saunders, M. Saunders, P. Lewis, and A. Thornhill, Research Methods for Business Students, 6th ed. Pearson Education Limited, 2012. [10] L. Rummukainen, L. Oksama, J. Timonen, and J. Vankka, “Visualizing common operating picture of critical infrastructure,” in SPIE Sensing Technology+ Applications. Int. Soc. Opt. Photonics, 2014, pp. 912 208– 912 208.
Fig. 4. Critical infrastructure monitoring operator’s tasks applied to the DiSCI context. The operator monitors various sources and provides information for various decision makers [1].
organization has to take action, an operator is responsible for smooth internal and external communication so that the incident can be handled efficiently. As a result of the goal-directed task analysis, a goal hierarchy was constructed. The three goals of a critical infrastructure monitoring operator are (1) monitor the organization environment, (2) analyze incidents and communicate internally, and (3) communicate externally. Each goal has a specific set of situation awareness requirements, of which the most common were information regarding incidents and the state of own systems and services. The results are highly reliable as the description of incidents and the monitoring operator’s work were mainly uniform across interviews. The research also identified the different roles the operator may have. Though communication was identified as the most important aspect of a monitoring operator’s work, only a few organizations had support systems for communication outside their own organization. The external communication relied mostly on contemporary mobile phones. However, many interviewees stated that better communication during an incident would help their organization recover more efficiently. This serves as a basis for future studies, and further research will focus on improving communication between organizations. The results of this research help form a standardized description of incidents, thus also helping to improve communication efficiency. R EFERENCES [1] J. Timonen, L. Laaperi, L. Rummukainen, S. Puuska, and J. Vankka, “Situational awareness and information collection from critical infrastructure,” in Cyber Conflict (CyCon 2014), 2014 6th International Conference On. IEEE, 2014, pp. 157–173. [2] M. R. Endsley, “Toward a theory of situation awareness in dynamic systems,” Hum. Factors, vol. 37, no. 1, pp. 32–64, 1995. [3] ——, Designing for Situation Awareness: An Approach to UserCentered Design. CRC Press, 2012. [4] T. G. Lewis, Critical Infrastructure Protection in Homeland Security: Defending a Networked Nation. John Wiley & Sons, 2006. [5] National Initiative for Cybersecurity Careers and Studies (NICCS). (2014, Oct.) Cyber glossary. [Online]. Available: http://niccs.us-cert. gov/glossary [6] R. Romero-G´omez, S. Tena, D. D´ıez, and P. D´ıaz, “The application of situation awareness-oriented design to the smart grid domain,” Interraccin 2013.
