Usability Evaluation Using Specialized Heuristics ... - Semantic Scholar

Usability Evaluation Using Specialized Heuristics with Qualitative Indicators for Intrusion Detection System 1

2

Tulsidas Patil , Ganesh Bhutkar and Noshir Tarapore 1

2

2

Research Scholar, Assistant Professor Department of Computer Engineering, Vishwakarma Institute of Technology, Pune - 411 037, India.

Abstract. Network security promises protection of valuable and accessible network resources from viruses, trojans, keyloggers, hijackers and unauthorized access. One of the important subsets of the network security tools is Intrusion Detection System (IDS). It is found that current IDS systems are not easy to use. As a result user has difficulties in judging the quality of the output, i.e. getting efficient alarm and severity level for detected intrusions, information about the detected intrusions as per layers, ports and visitors. Also the problems in installing and configuring the system go unnoticed. Therefore, the usability evaluation is extremely vital to help users in efficient interaction and enhance usage of IDS system. In this paper a specialized set of heuristics combined with objectively defined usability indicators are proposed for the usability evaluation of IDS systems. This study presents the evaluation of specialized set of heuristics based on usability problems and design deficiencies commonly prevalent among IDS systems. This set of specialized heuristics can be used to evaluate various IDS systems or other network security tools with diversified platforms and features. Key words: Usability, Intrusion Detection System, Human Computer Interaction, Usability Heuristics, Usability Indicators, Network Security.

1

Introduction

From the evolution of internet, people have been facing challenges of the network security. To face security challenges, network users utilize various tools such as firewall, antivirus software, ethereal, nmap, nessus, and Intrusion Detection System (IDS) [12]. But according to report of USCERT the rate of incident in year 2010 is nearly six times as compare to year 2005. So, the focus of work is moving towards the usability in security. Among these tools, IDS plays vital role in addressing the issues of network security as it is designed to provide a timely identification of malicious activities and to support effective response to the real-time attacks. But user often fails to get all these functional advantages from the IDS as usability of IDS comes into picture. Users complain about operations and maintenance of IDS [13]. There are two main problems regarding the state of art and state of practice in IDS. The first problem is about the underlying technique that is used in detecting attacks and the second problem is about user interaction to know and quickly respond to the detected attacks [7, 14]. This paper is a one step towards the improvement of usability in the network security tools. In this paper, we used well known heuristic evaluation method for the usability evaluation of IDS. The concept of qualitative indicator is also very much helpful to rate the importance of heuristics. ?

Please note that the LNCS Editorial assumes that all authors have used the western naming convention, with given names preceding surnames. This determines the structure of the names in the running heads and the author index.

2

Lecture Notes in Computer Science: Authors’ Instructions

1.1 Usability Heuristics Usability heuristics are used as guidelines to evaluate the usability of system under consideration. These heuristics are derived from the user as well as expert’s experiences and observations of the IDS systems. With these heuristics user can evaluate the usability of various IDS systems. According to the importance and its impact on the usability of IDS, the formation of heuristics was done from the observations. Usability evaluation is done by various methods such as cognitive walkthrough, formal usability inspection, heuristic evaluation or pluralistic walkthrough [5]. For usability evaluation of IDS, a heuristic evaluation method was selected where heuristics are specially designed for the IDS. The usability problems and design deficiencies commonly prevalent among all three IDS systems were identified and list down as observations. A comprehensive list of relevant problems and observations, which contribute to usability of IDS systems, was prepared. With IDS and usability expert the only observations which are applicable for the usability of IDS are considered as usability heuristics which are further divided into six groups. These six groups are elaborated in next section - Introduction to heuristics. The qualitative usability indicators [4] were identified to measure the compliance. Instead of applying the 1-5 Likert scale uniformly across all parameters, we have chosen an indicator based evaluation method. Some heuristic indicators are checked in term of their absence or presence and some are elaborated in terms of their qualitative attributes. 1.2

Intrusion Detection System

The notion of IDS came from the seminal paper by James Anderson in 1980 [7]. Intrusion detection is the process of monitoring the events occurring in a computer system or network and analyzing them for signs of intrusions, defined as attempts to compromise the confidentiality, integrity, availability to bypass the security mechanisms of a computer or network. Intrusions can be caused by attackers accessing the systems from the internet, authorized users of the systems who attempt to gain additional privileges for which they are not authorized and authorized users who misuse the privileges given them. IDS is a software that automate this monitoring and analysis process[1]. 1.3

Users of IDS Systems

Traditionally IDS is used only by the administrators but because of its advantage over other network security tools IDS users are continuously increasing from network administrator to the daily computer users who wants to have a full picture of the traffic going through one’s PC or LAN segment. These users can be mainly classified as LAN administrators, security professionals and network programmers. • LAN Administrators – The LAN administrator provides support and management of the local area network within a company. This management involves a number of functions that have to do with regular maintenance of the network, overseeing enhancements and upgrades to the local area network. As part of the regular maintenance of the network, the LAN administrator will monitor the daily activity on the network, ensuring that the resources of the company are utilized in ways that are within the standards set for employee usage. • Security Professionals – Security professionals have broad understanding of a growing list of topics and technologies, including identity management, strong authentication, biometrics, anti-virus, intrusion detection, anti-spyware, firewalling and encryption.


3

• Network Programmers – Network programmers are the persons which are deal with the design of network. Based upon the traffic on the network and occurrence of intrusions, network programmer designs the networks and program according to traffic. 2

Related Work

It is observed that Even with strong (financial) incentives, users tend to ignore security indicators, such as absence or invalidity of SSL certificates [17]. Andrew Zhou has studied a IDS system and proposed a set of 6 heuristics for the usability improvement of IDS [1]. In the set of six heuristics 4 heuristics from Nielson’s heuristics and 2 additional heuristics – ‘Display of information’ and ‘Information navigation’; exclusively for IDS. These set of six heuristics are useful but not sufficient as it consider only IDS systems. In another study, a heuristic evaluation of 3 touch screen ventilator systems has been performed using qualitative indicators [4]. We have used the concept of a comprehensive set of usability heuristics and well-defined usability indicators in our study with IDS. The next study has highlighted various challenges while using IDS such as considerations for deployment, configuration of security settings, availability of information about log storage in IDS [6, 13] and requirement of additional software for better operations. These challenges have propelled us to arrive at some vital usability heuristics in our study. Masone and Smith [3] discusses the problems related to the secure email applications based on digital certificates, certification authorities, and public key infrastructures. Similarly, Peter Mell has discussed issues in testing of IDS [12]. These issues have guided us in designing heuristics such as type of IDS output provided, sorting of detected events and provision of attack signatures for IDS in our study. We have come across several usability evaluation studies which are carried out using Nielsen’s heuristics. Ficarra has performed a heuristic evaluation of multimedia products [3] and another study was focused on heuristic evaluation of paper-based web pages [9]. These studies have helped us to get better insights into a method of heuristic evaluation.

3

Research Methodology

It involves following activities during research: 3.1

Selection and Study of IDS Systems

To design heuristics for IDS systems, we had chosen five IDS systems out of which three IDS systems namely Snort, KFSensor and Easyspy are used for the observation and study purpose and two IDS systems namely Sax-2 and X-ray along with Snort are used for evaluation. For the study of IDS systems, we installed IDS systems on three different operating systems likewise Snort on Linux, KFSensor on Microsoft Windows 7 and Easyspy on Microsoft Windows XP. These three IDS systems are widely used and readily available. Operating System (OS) platform supported by IDS is an important aspect in selection of IDS. During the study of IDS systems we have noted down different aspects of IDS systems such as usability issues during installation, configuration, operation and maintenance of IDS systems. All major and differentiating observations providing useful insights into IDS usability were noted down.


4

3.2 Use of Card Sorting Method To classify heuristics we use the card sorting method which is a useful method to determine and understand about how items can be classified [11]. In our research, we have used a set of index cards for classification of heuristics. Each card has an entry of heuristic written on it along with a related identification number. These cards are then provided to IDS experts for classification into suitable categories. The output of this process has provided 6 well-defined categories such as Installation, Output, Event, Customization, Help and Miscellaneous Heuristics. These classified heuristics are discussed in next section.

4

Introduction to Heuristics

We proposed a set of 35 usability heuristics for usability evaluation of IDS. To design heuristics we considered not only the suggestion from IDS experts but also the experience of naïve user. So the final set of specialized heuristics contains the heuristics useful for both naïve user as well as the IDS experts. By considering the inputs from IDS experts, experience from naïve user and suggestion by usability experts the final draft of specialized set of 35 heuristics for the usability evaluation of IDS systems prepared. These heuristics are discussed ahead in details. 4.1 Installation Heuristics The heuristics under this group are useful to select the IDS system for the network. This group is divided into two categories like pre-installation and installation. The heuristics under pre-installation group is helpful when user selects the IDS system among the others. Before installation of IDS systems user must have sufficient information about IDS systems like whether the chosen system is applicable for OS platform, the type of installation. The other category of heuristics is suitable during installation process. The heuristics under this category is required at the time of installation of IDS systems. Table 1. Installation heuristics Pre-Installation Heuristics H1

Operating system platform supporting the IDS

H2

Type of IDS available

H3

Type of installation supported in IDS

Windows (4) Linux (3) FreeBSD (2) Other (1) Freeware (4) Open source(2) Proprietary (0) Offline (2) Online (1)

Installation Heuristics H4

Provide Graphical User Interface (GUI) for installation of IDS

H5

Require installation of additional software for IDS

H6 H7

Allow user to customize installation of IDS Provide facility to select database during installation

Provided (3) Not provided (0) Not required (3) Available at vendor’s website (2) Online (1) Not available (0) Allowed (2) Not allowed (0) Provided (1) Not provided (0)


5

4.2 Output Heuristics This group of usability heuristics contains the heuristics which are related to the ou tput provided by IDS. One of the most important part of IDS systems is the output. So usability of IDS systems is the most critical section. It is observed that users fail to understand the output of IDS systems as in output IDS systems provides unrelated information also it contains too many technical specifications which are not require to user. Use of table and charts in the output of IDS systems makes it more quickly and rapidly understandable. It is always better if the output of IDS contains information about the number of attacks and detailed information for each attack such as its type, time and/or severity. A set of heuristics for output is depicted in table 2. Table 2. Output heuristics

Output Heuristics

H8

Understanding of the output provided by IDS

H9

Customization of GUI for the output of IDS

H10

The type of IDS output provided

H11

Provide printable output report

H12

The additional information available apart from number of attacks in IDS output

H13

Provide tables and charts to represent output information

Easy to understand (4) Difficult to understand (2) Not understood (0) Allowed (3) Not allowed (2) GUI not provided (0) As per ports (2) As per layers (2) As per visitors (2) Not provided (0) Provided in PDF format (2) Provided in Excel Format (2) Provided in other format (1) Not provided (0) Type of intrusion (2) Severity of intrusion (2) Time of intrusion (2) Number of intrusion attempts (2) Not available (0) Charts provided (2) Tables provided (1) Not provided (0)

4.3 Event Heuristics We have proposed 7 heuristics related with usability of events in IDS as shown in table 3. IDS system continuously detects events from the network. So with in less time IDS systems detects large number of events. To get the useful information from the events, these event heuristics are helpful. IDS systems should provide facility to categories events in ascending as well as in descending order. Also color codes as per the severity of intrusion makes the good difference among the detected events. For hiding the events, IDS should provide customization through display options like ‘before today’ and ‘as per the severity level. Assignment of severity level to each event is a vital feature in IDS can be observed in figure 1.


6

a

b

Fig 1. Sample screen shots of events in Easyspy and KFSensor a) severity level b) events to hide

With the help of severity levels, user can not only differentiate events, but also assign appropriate actions to the events. These actions may include an execution of a suitable file, audio alert, email alert or flashing an icon. Users find it difficult to search event(s) based on a particular criteria and so, there is a need of advance search option. Table 3. Event heuristics

Event Heuristics

H14

Amount of false-positive and falsenegative events

Negligible (4) Marginal (3) Higher (0) Don’t know (2)

H15

Type of intelligence technique used in IDS

Rule based (3) Anomaly based (3) Honey pot based (2) Other (1)

H16

Categories provided for customizing the event alerts in IDS

H17

Number of severity levels provided in IDS

H18

Availability of search option for events

H19

Provide color codes for the events in IDS

H20

Sorting of detected events in IDS

H21

Categories provided for loading/hiding events

Execution of file (4) Flashing icon (3) Mobile messaging (3) Audio alert (2) Mobile messaging (2) E-mail alert (1) Not provided (0) 3-5 levels provided (3) Levels out of range (2) Not provided (0) Advanced search available (2) Search available (1) Not available (0) Appropriate (3-5 colors) (2) Confusing (1) Not provided (0) Ascending order (1) Descending order (1) Not possible (0) Before today (1) Low severity (1) Medium severity (1) High severity (1) Not provided (0)


7

4.4 Customization Heuristics Customization heuristics reduces the effort of re-installing the IDS systems. Also customization heuristics reduces the redundancy in information and makes the interaction of user with IDS systems more efficient. IDS systems detect events with the help of signatures. So to reduce false negative and false positive alarm rate support for customized signature is necessary. A set of heuristics for customization is depicted in table 4. Table 4. Customization heuristics

Customization Heuristics

H22

Availability of wizard help option in IDS

Whenever necessary (3) At the end (2) Not available (0)

H23

Provide support for customized signatures

Provided (2) Not provided (0)

H24

Provide facility to set updation policy for IDS

H25

Provide facility for customize activation time of IDS.

Auto update (3) Manually update (2) Not provided (0) Auto-started (1) Need to start explicitly (0)

4.5 Help Heuristics The heuristics under this category is required when user came across core ny networking terminologies of IDS. Also when some errors / warnings and related messages appears on the screen. IDS systems have to provide online help for understanding terminologies and error-handling. IDS also stores logs for the detected events, so it should provide the path of the log file for its users. A set of help heuristics is depicted in table 5. Table 5. Help heuristics

Help Heuristics

H26

Provide help about the networking terminology in IDS

H27

Provide information about the log file in IDS

H28

Provide appropriate error/warning message

H29

Provide help for icons in IDS

H30

Provide multilingual support

Help is provided in software(3) At vendor’s website (2) Not provided (0) Provided with path of log file (2) Provided without path of log file(1) Not provided (0) Provided along with help (4) Provided without help (2) Not provided (0) As a tool tip (2) In user manual document (1) Not provided (0) Provided (1) Not provided (0)

4.6 Miscellaneous Heuristics With the help of card sorting method all 35 heuristics are categorized into 5 categories and the remaining heuristics are group into miscellaneous category. The heuristics under this category is important for the performance

8


as well as usability of IDS. To keep the IDS always updated, provision for attack signatures is needed. Users find convenience in getting the information about attack signatures through notification than downloading it from the vendor’s website. An active IDS should not degrade the system performance. Also the time required to provide new signatures for attack detection should be less than a day. A set of miscellaneous heuristics is depicted in table 6. Table 6. Miscellaneous heuristics

Miscellaneous Heuristics

H31

Provision for attack signatures for IDS

H32

Time required to provide signature for new vulnerability

H33

Provide previous and next options at every screen

H34

Provide information about scalability of IDS

H35

Effect of IDS on system performance

5

Through notification (4) At vendor’s website (2) Not provided (0) Within 24 Hrs (4) Within a week (2) More than a week (0) Don’t know (2) Provided (2) Not provided (0) Provided in help option (3) Provided in user manual (2) Provided at vendor's website (1) Not provided (0) Not affected (2) Marginally affected (1) Severely affected (0)

Evaluation

We have evaluated the usability of IDS systems using the heuristics and usability indicators with following objectives: 1. Measure the usability and overall efficacy of IDS systems in terms of usability index. 2. Study the reliability of the heuristics by involving two more IDS experts to carry out the evaluation of additional two IDS systems. This heuristic evaluation has been carried out by two more Usability Evaluators (UE) with author. In the following table UE1 is referred to author where as UE2 and UE3 are for other two IDS expert involved in the heuristic evaluation of IDS systems. For the heuristic evaluation we used two more IDS systems with Snort IDS namely Sax-2 and X-ray. The usability evaluators have adequate understanding of Human Computer Interaction (HCI). They were sensitized about the proposed heuristics, fundamentals about network security and the usability evaluation of IDS systems. Before the evaluation, Evaluators have hands free session on the all three IDS systems. All the evaluators got the IDS systems on internet after they installed it on the allocated computer system. Before they installed IDS systems, the list of all 35 heuristics are provided to them and ask to give score as per their observation and experience with the IDS systems. Their queries about the heuristics and related evaluation were discussed and then they carried out individually the heuristic evaluation of the IDS systems provided to them. The total scores of usability evaluations by all three usability evaluators are consolidated in table 7.


7

Table 7. Heuristic evaluation of IDS by three evaluators

Heuristic Max. Score

Categories

Usability Evaluators

Scores for IDS Systems IDS-1

IDS-2

IDS-3

UE1

16

05

06

UE2

16

06

06

UE3

16

06

06

UE1

04

05

04

UE2

01

05

04

UE3

01

05

04

UE1

14

20

09

UE2

14

21

13

UE3

14

20

13

UE1

08

20

07

UE2

07

20

07

UE3

08

17

07

UE1

04

08

05

UE2

04

08

05

UE3

04

08

05

UE1

03

06

02

UE2

03

06

02

UE3

03

06

02

UE1

09

04

03

UE2

10

07

04

UE3

10

08

04

UE1

58

68

36

UE2

55

73

41

UE3

56

70

41

Pre19

Installation

Installation

Output

Events

12

29

33

Customizatio 11

n

Help

13

Miscellaneo 21

us

Total

6

138

Validation

The figure 2 shows a graph for a comparison of usability evaluation of three IDS systems under evaluation by three usability evaluators (UE1, 2 and 3). The usability evaluation by other usability evaluators – UE1 differs from UE3 by -2.16 % for IDS-1, 2.17 for IDS-2 and -2.63 for IDS-3 IDS.

10


Fig 2. Usability evaluation resuts of three IDS systems with specialized set of heuristics by three evaluators

Above figure depicts a graph representing closeness in usability evaluation of three IDS systems by three evaluators with the set of specialized heuristics for IDS. The above graph shows the heuristics are very appropriate and gives similar results in the evaluation process by different evaluators.

7 Conclusion A study of IDS systems and the outcomes of their usability evaluation using specialized set of heuristics with indicators show that there are many usability issues as well as design deficiencies, which needs to be addressed. The specialized set of heuristics categorized into relevant groups and the objectively defined usability indicators ensure better understanding and efficiency among IDS systems to make them more user-friendly and humanized. This process helps in better understanding and usage of IDS systems by maximum possible users including novice users. In future, this study can be extended to evaluation of other network security tools.

Acknowledgement We express our gratitude to IDS experts – Axel Kloth, CTO and Vice President Engineering, Parimics Inc., USA and Hal Flynn, Vulnerability Analyst, Symantec Corporation, USA for their timely inputs and feedback during this research work.

References 1. Andrew Zhou, James Blustein and Nur-Zincir-Heywood.: Improving Intrusion Detection System through Heuristic Evaluation. IEEE CCECE 2004-CCGEI, Niagara Falls, pp.1641-1644 (2004). 2. Alain Abran, Adel Khelifi and Witold Suryn.: Usability Meaning and Interpretation in ISO Standards. Software Quality Journal, pp. 325-338 (2003). 3. C. Masone and S. Smith.: Towards Usefully Secure Email, IEEE Technology and Society Magazine 26(1), pp. 25-34 (2007) 4. F. Ficarra.: Evaluation of Multimedia Components. IEEE International Conference,

Ottawa, Ont., Canada, pp. 557-564 (1997). 5. Dinesh Katre Ganesh Bhutkar, and Shekhar Karmarkar.: Usability Heuristics and Qualitative Indicators for the Usability Evaluation of Touch Screen Ventilator Systems. Human Work Interaction Design, Usability in Social, Cultural and Organizational Context - IFIP AICT 136, Springer, 83-97. DOI: 10.1007/978-3-64211762-6_8 (2010). 6. Jakob Neilsen and R. Molich.: Heuristic Evaluation of User Interfaces. ACM SIGCHI Conference on Human factors in Computing Systems: Empowering People, pp. 249-256. ISBN: 0-201-50932-6 (1990). 7. John McHugh, Alan Christie and Julia Allen.: The Role of Intrusion Detection Systems. IEEE Software, pp 42-51 (2000). 8. James Anderson "Computer Security Threat Monitoring and Surveillance”, February 1980, pp.1-56 9. Kevin Baker, Saul Greenber and Carl Gutwin.: Empirical Development of Heuristic Evaluation Methodology for Shared Workspace Groupware. ACM CSCW, New Orleans, Louisiana, USA, pp 96-105 ( 2002). 10. Martuza Ahmed, Rima Pal, Md Mojammel Hossain, Md. Abu Naser Bikas and Md. Khalad Hasan.: A Comparative Study on Currently Existing Intrusion Detection Systems. IACSIT-SC, IEEE, pp. 151-154, (2009). 11. Mureen Allen, Leanne Currie, Suzanne Bakken and Vimlal Patel.: Heuristic Evaluation of Paper-based Web Pages A Simplified Inspection Usability Methodology. Journal of Biomedical Informatics, pp. 412–423 (2006). 12. Neumann.: Audit Trail Analysis and Usage Collection and Processing. Technical Report Project 5910, SRI International (1985). 13. N. Nurmuliani, D. Zowghi and S. Williams.: Using Card Sorting Technique to Classify Requirements Change. IEEE International Requirement Engineering Conference, Kyoto, Japan, pp 240-248 (2004). 14. Nitesh Dhanjani and Justin Clarke.: Network Security Tools. O’reily Media Inc. (2005). 15. Peter Mell and Vincent Hu.: An Overview of Issus in Testing Intrusion Detection System. DARPA (2002). 16. Rodrigo Werlinger, Kristie Hawkey, Kasia Muldner and Pooya Jaferian.: The Challeges of Using an Intrusion Detection System: Is It Worth the Effort. SOUPS, Pittsburg, PA, USA (2008). 17. SANS Institute: Intrusion Detection System: Definition, Need and Challenges (2001). 18. S.E Schecter, R. Dhamija, A. Ozment, and I. Fischer.: The Emperor’s New Security Indicators: An evaluation of website authentication and the effect of role playing on usability studies,” IEEE Symp. on Security and Privacy, Oakland, CA, USA, May 20-23, (2007)