Literature Review

1 downloads 0 Views 4MB Size Report
Apr 12, 2018 - utilization of the Censys Internet-wide scanner and the data feeds found ...... The United Kingdom's National Cyber Security Centre describes.
Scout: A Contactless ‘Active’ Reconnaissance Known Vulnerability Assessment Tool

Jamie O’Hare 40168785

Submitted in partial fulfilment of the requirements of Edinburgh Napier University for the Degree of BEng (Hons) Cyber Security and Forensics

School of Computing

April 2018

Authorship Declaration I, Jamie O’Hare, confirm that this dissertation and the work presented in it are my own achievement. Where I have consulted the published work of others this is always clearly attributed; Where I have quoted from the work of others the source is always given. With the exception of such quotations this dissertation is entirely my own work; I have acknowledged all main sources of help; If my research follows on from previous work or is part of a larger collaborative research project I have made clear exactly what was done by others and what I have contributed myself; I have read and understand the penalties associated with Academic Misconduct. I also confirm that I have obtained informed consent from all people I have involved in the work in this dissertation following the School's ethical guidelines

Signed:

Date:

Matriculation no:

1

Data Protection Declaration Under the 1998 Data Protection Act, The University cannot disclose your grade to an unauthorised person. However, other students benefit from studying dissertations that have their grades attached.

Please sign your name below one of the options below to state your preference.

The University may make this dissertation, with indicative grade, available to others.

The University may make this dissertation available to others, but the grade may not be disclosed.

The University may not make this dissertation available to others.

2

Abstract Internet-wide scanning projects such as Shodan and Censys, scan the Internet and collect active reconnaissance results, and provide access to this information publically through websites. This can be used to potentially identify systems and services which may be susceptible to known-vulnerabilities without interacting with the target devices or networks themselves. This approach can be classed as contactless active reconnaissance. However, the vulnerability identification functionality aspect is very limited. This dissertation looks towards extending this functionality through the creation of a novel tool named ‘Scout’. Through the utilization of the Censys Internet-wide scanner and the data feeds found within the National Vulnerability Database (NVD), Scout passively identifies known-vulnerabilities. This is done by the manufacturing of accurate Common Platform Enumerations (CPE) via service banner analysis of the Censys data, and associating these to relevant Common Vulnerability and Exposures (CVE) from the NVD. Implemented through Python scripts corresponding with the Censys API and a locally stored MongoDB containing the relevant NVD data feeds. Current vulnerability assessment tools operate actively by interrogating services with specialized probes. These tools can contribute to traffic for sustained periods of time, causing disruption to the target network. Through the novel approach of Scout the problems currently associated with the alternative active approach are mitigated. Several experiments were performed. In initial experiments performed on 2571 services across 7 local academic intuitions, 12967 known-vulnerabilities were identified. More focused experiments to evaluate results and compare with industry standard vulnerability assessment tools were carried out. Scout can successfully identify vulnerabilities which has been validated against active industry tools such as Nessus and OpenVAS. Scout has produced an effectiveness score in vulnerability identification of up to 74% when compared to OpenVAS as a baseline.

3

Table of Tables Table 1: Censys' Initial Deployment Scan Schedule (Durumeric, Adrian, Mirian, Bailey, & Halderman, 2015) ....................................................................................................................18 Table 2: Crawler's Operation ...................................................................................................22 Table 3: Results from Heartbleed Queries...............................................................................28 Table 4: Software product obtained from the GPLI database seen in (Sanguino & Uetz, 2017) ..................................................................................................................................................34 Table 5: Search Terms generated by information shown in Table 4 .......................................34 Table 6: The desired output fields ...........................................................................................39 Table 7: Methods of CPE matching in literature ......................................................................43 Table 8: Results from Manual Assessment ..............................................................................57 Table 9: ShoVAT and Scout's Manual Assessment Results ......................................................59 Table 10: Comparative Assessment Results ............................................................................60 Table 11: CVE Assignment Results ...........................................................................................61 Table 12: Vulnerabilities reported for an Apache 2.4.7 service ..............................................62 Table 13: Vulnerabilities reported for an Apache 2.4.10 service ............................................63 Table 14: Vulnerabilities reported for a Lighttpd 1.4.18 service .............................................64 Table 15: Vulnerabilities reported for a Lighttpd 1.4.33 service .............................................64 Table 16: Tool Comparison ......................................................................................................68

4

Table of Figures Figure 1: Lockheed Martin Cyber Kill Chain ...............................................................................9 Figure 2: Screenshot of Shodan detailing Windows 8 services ...............................................10 Figure 3: Categorization of Cyber Scanning (Bou-Harb, Debbabi, & Assi, 2014).....................13 Figure 4: Cyber-attack anatomy...............................................................................................14 Figure 5: Internet-scanning project overview .........................................................................17 Figure 6: Screenshot of Censys detailing Heartbleed vulnerable devices ...............................17 Figure 7: The output observed of first 1000 IP addresses from a Masscan (Graham R. , 2013) ..................................................................................................................................................19 Figure 8: The output of first 1000 IP addresses from a ZMap scan (Graham R. , 2013) .........20 Figure 9: Address Randomization Comparison (Adrian, Durumeric, Singh, & Halderman, 2014) ........................................................................................................................................20 Figure 10: Comparison of ZMap and Masscan (Adrian, Durumeric, Singh, & Halderman, 2014) ........................................................................................................................................21 Figure 11: Crawler’s Algorithm Illustrated ...............................................................................23 Figure 12: Censys and Shodan's scanning behaviour visualised..............................................23 Figure 13: Comparison between Censys and Shodan results across Higher Education Institutions ...............................................................................................................................25 Figure 14: University of F's 277 NTP services as seen on Shodan ...........................................25 Figure 15: Lighttpd 1.4.33 as seen in Shodan ..........................................................................26 Figure 16: Lighttpd 1.4.33 as seen in Censys ...........................................................................27 Figure 17: Screenshot of the Vulnerability Functionality Found within ZoomEye ..................28 Figure 18: Vulnerability assessment tool comparison (Genge & Enăchescu, 2016) ...............29 Figure 19: Number of CVEs by Year since 1997 .......................................................................31 Figure 20: CVE-2017-9788 .......................................................................................................31 Figure 21: CPE composition with corresponding levels labels (Na, Kim, & Kim, 2017). ..........33 Figure 22: The attributes found in the Well-Formed CPE Name .............................................33 Figure 23: Proposed regular expression by (Gawron, Cheng, & Meinel, 2017) ......................33 Figure 24: Search by product and vendor formula ..................................................................34 Figure 25: CPE Tree illustrated with corresponding levels. .....................................................35 Figure 26: IPO Flow Diagram....................................................................................................38 Figure 27: Detailed IPO flow ....................................................................................................39 Figure 28: Comparison of Contact-based and contactless test methods (Simon, Moucha, & Keller, 2017) .............................................................................................................................40 Figure 29: Hypothesised contactless reconnaissance .............................................................40 Figure 30: Internal flow and external dependencies of Scout.................................................41 Figure 31: Code snippet featuring a Censys search .................................................................42 Figure 32: Output for Censys search........................................................................................42 Figure 33: Output visualised by process ..................................................................................43 Figure 34: CPE matching pseudocode .....................................................................................44 Figure 35: UML activity diagram for choosing CPE ..................................................................45 Figure 36: Nginx 1.4.3 candidates ............................................................................................45 Figure 37: Levenshtein distance visualised ..............................................................................46 Figure 38: Apache 2.4.7 CPE Matching without Levenshtein algorithm .................................46 Figure 39: Apache 2.4.7 CPE Matching with Levenshtein algorithm .......................................46 Figure 40: CVE Search code snippet ........................................................................................47

5

Figure 41: Scout Output ...........................................................................................................47 Figure 42: Associated Average CVSS and Outliers ...................................................................52 Figure 43: Associated Average CVSS and Outliers for Academia Test Data ............................53 Figure 44: Percentage of Vulnerabilities by Severity for Academic Institutions .....................53 Figure 45: Experiment design ..................................................................................................55 Figure 46: Results from Manual Assessment...........................................................................57 Figure 47: Manual Assessment Confusion Matrix ...................................................................57 Figure 48: A Sankey Diagram of Manual Assessment Confusion Matrix .................................57 Figure 49: The Output of the CPE Algorithm with the Input "Apache Coyote 1.1" .................58 Figure 50: Algorithm output for “Apache httpd2.2.11” ..........................................................58 Figure 51: Precision, Recall, F-Measure for Scout in Comparison to OpenVAS ......................65 Figure 52: Inconclusive CPE creation .......................................................................................66

6

Table of Contents Abstract ....................................................................................................................................... 3 Introduction ................................................................................................................................ 8 Aim ........................................................................................................................................... 11 Objectives ..................................................................................................................................... 11 Literature Review ...................................................................................................................... 12 Introduction ...................................................................................................................................... 12 Internet Wide Scanning .................................................................................................................... 12 Shodan vs Censys ......................................................................................................................... 16 Scanning and Information Gathering ........................................................................................... 18 Current Vulnerability Assessment Tools ........................................................................................... 29 National Vulnerability Database ...................................................................................................... 30 Service Banner Analysis to Identify Vulnerabilities ........................................................................... 32 Conclusion ......................................................................................................................................... 36 Methodology ............................................................................................................................. 38 Design ............................................................................................................................................... 38 Implementation ................................................................................................................................ 41 Similar Projects ................................................................................................................................. 48 Legal and Ethical Considerations ...................................................................................................... 49 Conclusion ......................................................................................................................................... 51 Initial Operation ........................................................................................................................ 52 Experiments Results, Analysis and Evaluation ............................................................................. 55 Experiment Design ............................................................................................................................ 55 Experiment 1 - Common Platform Enumeration Creation ................................................................ 56 Manual Assessment ..................................................................................................................... 56 Comparative Assessment ............................................................................................................. 60 Experiment 2 - Common Vulnerability Enumeration Assignment .................................................... 61 Evaluation ......................................................................................................................................... 65 Conclusion ......................................................................................................................................... 68 Future Work .............................................................................................................................. 69 Conclusion ................................................................................................................................. 70 Acknowledgement ..................................................................................................................... 72 References................................................................................................................................. 73 Appendix ................................................................................................................................... 83 A ........................................................................................................................................................ 83 B ........................................................................................................................................................ 84 C ........................................................................................................................................................ 86 D........................................................................................................................................................ 86 E ........................................................................................................................................................ 87 F ........................................................................................................................................................ 88 G........................................................................................................................................................ 89 H........................................................................................................................................................ 90 I ......................................................................................................................................................... 91 J ......................................................................................................................................................... 91

7

Introduction Cyber security is emerging as an essential practice for businesses. The growth of cyber security can be attributed to high profile incidents such as WannaCry and NotPetya (BBC News, 2017) (Thomson, 2017). These worldwide cyberattacks exploited known vulnerabilities, which was patched, however those who did not update were susceptible to malware which caused havoc on many organisations and institutions severely disrupting normal business flow (Martin, Kinross, & Hankin, 2017). Another contributing factor to rapid adoption of cyber security practices has been the highly publicized implementation of the General Data Protection Regulation (The European Parliament, 2018). This piece of legislation created by European Union focuses on the protection and regulation of the collection and processing of personal and sensitive information (Information Commissioner's Office, 2017). This legislation significance has been magnified due to the potential for steep fines of up to 4% of annual turnover or €20 million for violations (The Economist, 2017). This has pressured businesses to allocate a greater focus to their risk management procedures related to cyber security. To locate, mitigate and manage cyber security risks, businesses are turning to penetration testing. The United Kingdom’s National Cyber Security Centre describes penetration testing as “a method for gaining assurance in the security of an IT system by attempting to breach some or all of that system’s security, using the same tools and techniques as an adversary might.” (National Cyber Security Centre, 2017). This gain of assurance can come in two forms as described in by the SANS institute, either to increase upper management awareness of cyber security, or to stress and test intrusion detection and response capabilities (SANS Institute, 2002). Over the last 15 years there has been substantial growth in penetration testing in the UK. The Council for Registered Ethical Security Testers, the main body for the growing industry, has grown to over 80 registered companies (CREST, 2017) (McCune, 2016). Security testing is used to audit for compliance for legislation such as for GDPR. As Security testing imitates an attack, it follows similar steps to that of would be attacker, these steps can be seen in Figure 1. Reconnaissance is the first step in testing and attacking, it is especially important in a black box testing scenario, where the attacker is uniformed and has received no previous information about the target system. The recon phase is divided into two different natures, passive followed by active. Passive recon consists of information gathering performed without a direct correspondence to the target systems. This can take place through techniques such as network sniffing or Open-source Intelligence (OSINT). In OSINT, information is collated from public, or open, sources. Examples include news reports, public government data and Internet sources like blogs, social media, repositories. A new inclusion to Internet sources has been that of Internet-wide scanning projects such as Shodan (Schearer, 2010).

8

Figure 1: Lockheed Martin Cyber Kill Chain

Shodan is tool which collates lightweight active reconnaissance results from services operating on all publically available IP addresses, allowing the user to navigate through the 550 million services indexed per month. A screenshot of Shodan can be found in Figure 2, showing the 96,026 Windows 8 services it has identified. This falls under OSINT and passive recon, as the user does not correspond with target, only with Shodan. The communication Shodan makes with target is of its own accord, and is through techniques usually performed early on during active reconnaissance. As Shodan has already performed this active reconnaissance, the information when accessed as a user is therefore passive. Shodan’s correspondence with the target system leave evidence, in this case through log files, this is a key principle in active reconnaissance. Through this active reconnaissance Shodan is able to identify a few known-vulnerabilities, most notably Heartbleed (CVE-2014-0160). This is not as extensive as active tools such OpenVAS and Nessus. OpenVAS and Nessus are vulnerability assessment tools which enumerate through a target range. Attempting to identify targets and their exploitable vulnerabilities, this tends to involve interrogating running services to try to highlight versions, through a fingerprinting process, which may have weaknesses and probe them with specialized packets to exploit specific known-vulnerabilities. This can be potentially disruptive to the target network causing denial of service through the considerable time taken to operate (Skybox Security, 2016). Due to this these tools require permission from the target organisation to be included within the test plan.

9

Figure 2: Screenshot of Shodan detailing Windows 8 services

This piece of work will look to investigate the possibility of expanding upon the vulnerability functionality present in Internet-wide scanning projects by identifying potential knownvulnerabilities using the information provided. If possible this research could provide a significant impact in the anatomy of security testing, by providing the ability of identifying vulnerabilities prior to active recon phase without interaction with the target. There has been limited work in this concept currently, with the main initial piece of literature being ShoVAT (Genge & Enăchescu, 2016). The importance of this research grows by the day as more cases of known vulnerability attacks are publicised. As reported by the SANS institute, in more than 80% of the security incidents known flaws were exploited (SANS Analyst Program, 2017). Gartner predicts that this statistic will rise through by 2020, where it believes that 99% of vulnerabilities exploited will be known prior to their use (Panetta, 2016). High profile incidents have come from this approach as WannaCry and NotPetya both exploited issues which were addressed in patches prior to exploitation (Martin, Kinross, & Hankin, 2017).

10

Aim The main aim of the project is to explore and evaluate how well information indexed by Internet-wide scanning projects could be harnessed to identify potential knownvulnerabilities in Internet connected systems. This aim will be supported by completing the following Objectives. Objectives  Objective 1 – Literature Review  Through a comprehensive literature review, conduct critical analysis of prior research, comparing and contrasting previously work in the field of Internet scanning and vulnerability assessment.  Objective 2 – Methodology  Using findings from the literature review to inform the design of a methodology, to build and evaluate a system designed to identify vulnerabilities from Internet scanning results.  Objective 3 – Experiments  Conduct experiments with the reproducible procedures outlined in prior research. Critique the procedures and also produce improved procedures where possible.  Objective 4 – Evaluation  Compare and analyse results of the experimentation, in particular against the aim of the project and prior research. In this process, it is paramount that the same evaluation methods are implemented as used in prior research. New methods of evaluation could also be hypothesized and implemented, so future research can analyse.  Objective 5 – Poster  Produce an appealing poster with illustrations and concise paragraphs. This poster should display in-depth in knowledge of the project area, identification of key processes, issues and outcomes. This dissertation is organized to reflect the objectives outlined above. Starting with a literature review, followed by a methodology. Which is used in experiments thereafter is evaluated.

11

Literature Review Introduction This section looks to review literature surrounding 4 distinct topics: Internet Wide Scanning, Current Vulnerability Assessment Tools, Vulnerability Databases, and Service Banner Analysis to Identify Vulnerabilities. There is a growing body of knowledge around the topic of Internet Wide Scanning, the scope of the review will look at current literature and include a detailed look at the current projects which make publicly available the information they collate. This analysis will not include tools which allow a user to perform internet scanning themselves. Current Vulnerability Assessment Tools will consist of a detailed investigation of the tools available on the market. This investigation is significant as it will be at the heart of evaluation. The Vulnerability Databases topic will look to analyse the current landscape of literature and resources in this area. The organization of vulnerabilities is essential for a wide range of technologies, including that of Vulnerability Assessment Tools. Incorporating themes found in the topics beforehand, Service Banner Analysis to Identify Vulnerabilities will combine the findings of the prior sections and investigate methods detailed in current literature. The literature surrounding these topics varied, with a mixture of academic literature and documentation, often from the author of referenced materials.

Internet Wide Scanning Internet Wide Scanning is fast becoming a key instrument in conducting research on the Internet. One of the main obstacles hindering research in this field was the sheer size of Internet, through technological advancements the impact of this impediment has been reduced. The concept of scanning is obviously central to Internet Wide Scanning. Scanning has two different natures, passive and active (Bou-Harb, Debbabi, & Assi, 2014). This can be observed in Figure 3. Where Cyber Scanning is seen to have two distinct natures. Passive scanning is the process of identifying network services by observing and analysing traffic (Xu, Wan, Zhang, Tong, & Wu, 2011). This requires the scanner to be located somewhere along the transmission, this can be done along the network or on the hosts themselves. This process requires sufficient traffic to analyse, without traffic there is no data for identification to be deduced from. Ironically, active scans contribute to traffic which help passive scans as they correspond with idle services, which passive monitoring techniques would miss (Bartlett, Heidemann, & Papadopoulos, 2007). Wireshark is one of the most common used tools in this field (Combs, 2008). PADS is another tool in this area, this system differs by using signatures in an attempt to match traffic to known services (Shelton, 2018). Active scanning is the process of identifying network services by corresponding with targets. This is can be used maliciously by external threat actors or legitimately by network operator to keep track of open services. This can be accomplished through several means such as a TCP handshake, UDP packets or ICMP pings (Stevens, 1994) (Panjwani, Tan, Jarrin, & Cukier, 2005). These methods contribute to the traffic of the network and therefore, leave evidence of the process. This process finds all services currently running, including those that are idle, but it can be blocked by firewalls. Often these methods are used in penetration testing, collecting information such as what services are running and where. These are then investigated further

12

by performing more active procedures such as banner grabbing, where the identification of software versions can be deduced (Collins, 2017) (Pinto & Stuttard, 2011).

Figure 3: Categorization of Cyber Scanning (Bou-Harb, Debbabi, & Assi, 2014) These cyber scanning natures align with that of the cyber-attack anatomy. The reconnaissance phase of a cyber-attack or penetration test has two distinct sections, passive and active (Sanghvi & Dahiya, 2013). The difference between the two is that passive does not interact with the target, gather information from other means. Passive scanning falls under passive reconnaissance where as active falls under active. This can be seen in Figure 4, where Reconnaissance phase of the cyber-attack anatomy has two distinct phases.

13

Figure 4: Cyber-attack anatomy

Internet scanning is a form of active reconnaissance, as internet scanning actively probes the internet. This scanning behaviour is often attributed to botnets and worms (Bou-Harb, Debbabi, & Assi, 2014). One such botnet was Carna botnet. This botnet was used to implement techniques such as ICMP pings, common ports probing, reverse DNS lookups as well as SYN scans. The findings of which was used to conduct the Internet Census 2012 (Anonymous, 2012). This is an example of how internet scanning which often is attributed to malicious activity is now quickly developing into becoming a valuable methodology for security research as it can be used to reveal new vulnerabilities and track their spread as well as deployment of mitigations (Durumeric, Wustrow, & Halderm, 2013). The oldest, most popular and well-known internet scanning project is Shodan (Shodan, 2017). Founded in 2009 by John Matherly, allows the user to query the search engine to find service banners of internet-connected devices (Matherly, Shodan, 2017). Shodan advertises itself as a tool to conduct market research, perform external security checks and to search the Internet of Things (Shodan, 2017). With the implications of the latter, such as identifying traffic lights, gas stations and even a command and control system for nuclear power plants, earned Shodan the title of “the scariest search engine on the Internet” by CNN (Goldman, 2013). John Matherly credits his scary search engine not on the engine itself but that it exposes the vast number of networks who operate under the premise of security as obscurity (Chris, 2016). Which is the reliance of being the needle in a haystack, and that you will not be likely targeted. Shodan’s inclusion in news reports have been of the sensationalist nature (Goldman, 2013) (Franceschi-Bicchierai, 2017). Matherly hypothesizes that this is due to its ability of to create fear-inducing headlines (Matherly, 2014). A far less hyperbolic exposure of Shodan was it featuring on season 3 of NBC’s ‘Mr Robot’ (Varmazis, 2017). Despite the mainstream exposure for several years now, the research and industrial use of Shodan has not been substantial.

14

Some research has been conducted with Shodan as part of an analysis of a nation’s cyber space. With research conducted on Industrial Control Systems found in Finland, Italy and Poland (Kiravuo, Tiilikainen, & Manner, 2015) (Chiappetta, Fadlon, & Norcini Pala, 2017) (RT, 2017). Further work has been done on a geographical scale, Trend Micro published in late 2017 published a series of white papers showcasing what Shodan exposes across multiple cities worldwide. Their findings were mostly in line with the greater the population the greater the number of exposed devices (Trend Micro, 2017). Despite this work by Trend Micro being advertised as using Shodan to provide the data analysed, not all of the data used was actually from Shodan. Another similar Internet-wide scanner called Censys also looks to have been used by the researchers (Tohru, 2016). Censys stems from a research group in the University of Michigan. Censys, like Shodan, scans the entire internet to index publicly available devices, the intricacies of each product is discussed later. The sheer number of Internet-of-Things (IoT) devices found in Trend Micro’s white papers was staggering. Most research encompassing IoT devices, and Intenet wide scanning, seems like it is focussed specifically on Industrial Control Systems (ICS) and Supervisory Control and Data Acquisition (SCADA). This research focuses on the capability to index and identify ICS and SCADA devices, as well as ways to mitigate the effectiveness (Bodenheim, Impact of the Shodan Computer Search Engine on Internet-facing Industrial Control System Devices, 2014) (Bodenheim, Butts, Dunlap, & Mullins, 2014) (Trend Micro, 2017) (Samtani, Yu, Zhu, & Patton, 2016) (Abe, Fujimoto, Horata, Uchida, & Mitsunaga, 2016). The research concludes that Shodan is a good tool to identify ICS devices, excelling at identifying when the devices are poorly configured. The results are also easily observed through its visualization of the scanning output (Leverett, 2011) (Ercolani, Patton, & Chen, 2016). Limited research has explored the idea of solely using Shodan as a tool to find vulnerabilities. It is often used in conjunction with other pre-existing tools, examples include Google (Simon, Moucha, & Keller, 2017) and Censys (Arnaert, Bertrand, & Boudaoud, 2016). Shodan is not the only Internet scanning project, and others found in the literature were IVRE, ZoomEye, and Censys. IRVE (Lalet, Monjalet, & Mougey, 2017) is an acronym for Instrument de veille sur les réseaux extérieurs (IVRE, 2017). Irve is the French word for drunk, which is also an acronym for the tool, Dynamic Recon of Unknown NetworKs (IVRE, 2017). Ivre is an open source project sponsored by the Commission of Atomic Energy and Alternative Energies (CEA) (IVRE, 2017). Ivre uses tool such as Nmap (Lyon, 2009), ZMap (Durumeric, Wustrow, & Halderm, 2013) and Masscan (Graham, McMilan, & Tentler, 2014). There is currently no published research using Ivre. ZoomEYE (ZoomEye, 2017) is a cyberspace search engine, similar to Shodan it allows the user to query for services located across the world. Developed by Knownsec Inc from Beijing, it was first released in 1st July 2013 (Baroni, 2016). Due to the geographic source of scans being China, this may cause inaccuracies in the data gathered. The inner workings of the search engine do not seem to be documented. Most research available using the tool is written in Chinese, those written in English often use the tool as a recording tool in conjunction with Shodan (Quarta, et al., 2017) (TrendMicro, 2017). Founded from a research project from the University of Michigan, Censys (Censys, 2017), is search engine allowing researchers to query the makeup of the internet enabled by daily

15

ZMap scans of the IPv4 address range (Arnaert, Bertrand, & Boudaoud, 2016). Censys allows its user to query protocol banners and parsed fields, to help to researchers solve security problems (Durumeric, Adrian, Mirian, Bailey, & Halderman, 2015). Research surrounding Censys and ZMap mostly come from the founder and creator himself (Durumeric, et al., 2014). However, research is gaining traction, due to the privileged access Censys provides for researchers. Of these Internet-scanning projects described above, Shodan and Censys stand out as the most appropriate to investigate in further detail. They are the most researched and most widely used throughout academic publications compared to lack of research currently surrounding IVRE and ZoomEye. Shodan has a long history within the information security community with research dating back to its founding (Schearer, 2010) (Leverett, 2011). Censys’ creation is documented through publications (Durumeric, Adrian, Mirian, Bailey, & Halderman, 2015), although much of this research is based on the underlying ZMap technology (Durumeric, Wustrow, & Halderm, 2013) (Adrian, Durumeric, Singh, & Halderman, 2014). This technology looks to not be used in alternative tools such as ZoomEye. For these reasons, it seems apt for further analysis of Shodan and Censys. Shodan vs Censys This section looks to provide a comparison between the major internet scanning projects, Shodan and Censys. Unlike other internet-scanning projects, these tools are available to the public and have a library of prior research related to both. This section is broken into several subsections, in each subsection analysis of each tool will take place. Specifically looking at comparing and contrasting the two scanning projects. The subsections are as follows:  Architecture  Scanning and Information Gathering  Indexing and Filtering  Vulnerability Functionality  Historical Analysis  Application Programming Interface Architecture The information gathered and displayed on Shodan and Censys is derived from internetscanning. Both Censys and Shodan employ similar techniques to obtain this data however there are differences. A main overview of how these projects operate can be observed in Figure 5. This figure displays the internet scanning being performed by a web crawler which reports back to a centralized back end database. This database is then accessed by the user through a front-end web application.

16

Figure 5: Internet-scanning project overview

Figure 6: Screenshot of Censys detailing Heartbleed vulnerable devices

Shodan’s front end web application is displayed in Figure 2, an example of Censys’ similar front-end web application can be seen in Figure 6. Both have been sanitized to remove any possible identifying information. From these figures, a similarity in how they display information to the user can be seen. To alternatively access the data, an API can be used. Both Shodan and Censys have an official Python API as well as a REST API. To access this data, an account is needed as both sites restrict usage until an account is made to prevent abuse (Censys, 2017). Shodan accounts are free for those who sign up with an academic email (Shodan, 2015), if not possible the price for a lifetime membership is $49. Once activated the account runs on a supply of credits which are updated monthly. A query credit is used when searching using a filter, or surpassing the first page of results (Matherly, 2017). A user has 20000 credits a month. A scan query is used when requesting for Shodan to scan a specific host (Matherly, 2017). A user has 60000 scan credits a month. An export credit is used for every 10000 results exported from Shodan using its export tool to acquire CSV files (Matherly, 2017). 100 export credits are given upon the creation of account, however unlike query credits these are no replenished at the start of every calendar month. Upon expiration, the user must pay $5 for every credit (Shodan, 2017). At the start of this project Censys allowed all these features for free with the only restriction being a limit on the rate of API

17

output, however during this project Censys commercialised. With commercialisation came several different plans, the Censys Free plan allowed Web and API access, with a limit of 250 queries a month. Even upon upgrade to Censys Basic, which costs $99, the user is restricted to 2500 queries a month. This drastically less than what Shodan offers. Scanning and Information Gathering Shodan boasted the indexing 550 million services a month, this was accomplished through the usage of 24/7, distributed and randomized crawlers (Matherly, 2014)(Matherly, 2016). Shodan’s crawlers run perpetually, and are continuously updating the underlying database in real-time (Matherly, 2016). This is significant as the data provided must remain contemporary to retain value. Obsolete data can provide a radically different picture than the latest. While Shodan scans perpetually, Censys’ scans are scheduled daily, biweekly and weekly as seen in Table 1. These scans are performed over a 24-hour period, even though the underlying ZMap can perform this faster (Durumeric, 2017) (Adrian, Durumeric, Singh, & Halderman, 2014). This prolonged process means Censys will have regimental updates to its underlying database. This scheduling means Censys may miss services with short life spans due to the time between scans, such as those connected to Internet via mobile communications such as 3G. Table 1: Censys' Initial Deployment Scan Schedule (Durumeric, Adrian, Mirian, Bailey, & Halderman, 2015)

Protocols HTTP, HTTPS and CWMP SSH, Modbus, FTP, DNS, NTP, UPnP, SMTP and SSH IMAP, IMAPS, POP3, POP3, POP3S, HTTPS/SSLv3 and HTTPS/Heartbleed

Scan Scheduling Daily Biweekly Weekly

To comprehensively scan the entire internet, Shodan distributes crawlers world-wide to minimize the effect of a geographic bias. An example of potential geographic bias is the banning of Chinese IP ranges by network administrators in America (Matherly, 2016). This is a particular problem for ZoomEye (ZoomEye, 2017). By using distributed crawlers, Shodan negates the effect of this geographic blocking by being able to scan from many countries around the world as well as reducing the likelihood of being detected of scanning. Upon detection network administrators may block the responsible IPs. This is significant as this practice ensures the dataset that Shodan provides is as complete as possible. Upon initial deployment Censys’ scans were conducted from the University of Michigan (Durumeric, 2017) (Adrian, Durumeric, Singh, & Halderman, 2014) (Durumeric, Adrian, Mirian, Bailey, & Halderman, 2015). This has not been addressed since however, due to the inclusion in multiple published works. It seems it has not been changed. This is problematic, due to issues raised prior with the effect of geo-blocking and detection. This may be the cause of the requests to be blacklisted. The demographics of Censys’ blacklist can be observed in Appendix C. As Shodan distributes their scanners, while Censys does not, Shodan has the advantage of a more comprehensive dataset.

18

Shodan’s crawlers are randomized, using a similar method to ZMap, although this is not well documented (Matherly, 2014). This randomization is to avoid the problems caused via incremental mass scanning as seen in previous projects (Graham, McMilan, & Tentler, 2014). During incremental scanning, the scanner would be blocked automatically by firewalls from accessing networks due to their predictable behaviour and occasionally receive threats. Besides reducing the possibility of being blocked when scanning this feature also avoids any information bias too (Matherly, 2016). This is an appropriate measure to maintain the adequacy of Shodan’s data set. Without this procedure, the dataset wouldn’t be as diverse, leading to conclusions being formed from only a subsection, which wouldn’t reflect the overall dataset. Censys uses ZMap as its scanning architecture. ZMap was released in 2013 and upgraded later in 2014 (Durumeric, Wustrow, & Halderm, 2013) (Adrian, Durumeric, Singh, & Halderman, 2014). When the upgraded ZMap was compared to Shodan in previous research, it was found that Shodan while able to index the services uses as a test subject, they were not able to return comparable results. ZMap was able to match, and at times, exceed the results Shodan returned (Forbis, 2015). When upgraded it was compared to Masscan (Adrian, Durumeric, Singh, & Halderman, 2014). Masscan as seen as DEFCON 22, claims to be the fastest port scanner, at the time 10 times faster than any alternative. Boasting a 25 million packets/second scan rate, theoretically able to scan the entire internet in under 3 minutes, although it requires prerequisite technology (Graham R. , 2013) (Graham, McMilan, & Tentler, 2014). Masscan’s leapfrog technology is its use of a self-described ‘encryption’ algorithm (Graham R. , 2013). This encryption is used on a monotonically increasing index to both randomize the output as well as maintain a singular correspondence between input and output. The algorithm used in Masscan is known to the author to have problems, although it was deemed suitable for use for randomizing IP addresses and port numbers. Although upon visualization its suitability is questioned (Graham R. , 2013). Another particular problem is the computational cost for this cryptographic algorithm, to reduce cost the algorithm only uses 3 round DES instead of 16 rounds, reducing the randomization. When comparing the randomization of both scanning projects visualization is used. This is done to spot glaring artefacts in the randomization process, using the human eye instead of relying on algorithms. Upon comparison when visualized, Masscan was considerably less random than ZMap (Adrian, Durumeric, Singh, & Halderman, 2014). This can be displayed in the following figures.

Figure 7: The output observed of first 1000 IP addresses from a Masscan (Graham R. , 2013)

19

Figure 8: The output of first 1000 IP addresses from a ZMap scan (Graham R. , 2013)

Figure 9: Address Randomization Comparison (Adrian, Durumeric, Singh, & Halderman, 2014)

Figure 7 displays quite clear vertical banding, this artefact proves that the algorithm used in Masscan is not as random as ZMap. Figure 8 displays no apparent artefact in the algorithm incorporated in the ZMap’s randomization process. This comparison has been performed since (Figure 9) and reaches the same conclusion. Although ZMap’s cyclical address randomization is computationally intensive it achieves far better statistical randomness than the custom feistel network used by Masscan (Adrian, Durumeric, Singh, & Halderman, 2014). Masscan’s inadequate randomization increases the potential of overloading destination networks, a possible explanation to the results observed in Figure 10 (Adrian, Durumeric, Singh, & Halderman, 2014).

20

Figure 10: Comparison of ZMap and Masscan (Adrian, Durumeric, Singh, & Halderman, 2014)

Figure 10 displays a drop in hit rate for ZMap after 4 Mpps, however, this drop never drops further than max hit rate achieved by Masscan even at the maximum speed achieved. As previously mentioned this may be due to overloading destination networks due to inadequate randomization. ZMap was made with internet scanning in mind, and due to this fact has several qualities which make it appropriate to do so. ZMap has the capability to scan the full IPv4 address space in under 45 minutes. One of the features which enable this speed is the fact there is no retransmissions. Only a set number of packets is set per target. The default is one packet per target, this have been observed to have a 98% coverage of the address space. This helps ZMap achieve a scanning rate 1300 times faster than related tools such as NMap (Durumeric, Wustrow, & Halderm, 2013). Unlike other tools ZMap does not require modification to the kernel or specialized hardware (Deri, 2004) (Han, Jang, Park, & Moon, 2010) (Graham R. , 2013). However, a year after release ZMap was updated. This motivation behind the additional development to ZMap was simply due to the access to a new 10GigE connection. With this came with a large performance increase, leading to comparable results to Masscan, however the same prerequisites are needed (Adrian, Durumeric, Singh, & Halderman, 2014). This update to ZMap saw a tenfold increase in the maximum scan rate. This increase enabled ZMap to scan the entire public IPv4 address space for one TCP port in 4.5 minutes, a huge decrease. This reduced scan time was due to new features, including paralleled address generation. This is used to spread address generation across multiple threads and cores. During this address generation, checks to the blacklist must be made. The way to this is done has been streamlined and improved. The most drastic change was ZMap switching to PF-RING technology as seen in other projects such as Masscan (Graham R. , 2013). This bypasses the kernel and reduces memory load, however requires the aforementioned specialized hardware. When benchmarked using only one 10-gbps Ethernet card, Masscan was found to observe a peak of 7.4Mpps. On the same hardware, ZMap was observed to obtain a peak of

21

14.1Mpps. A 6.7Mpps difference between the two scanners, with ZMap obtaining a peak nearly double of which Masscan obtained (Adrian, Durumeric, Singh, & Halderman, 2014). Although, at one time Masscan may have been the fastest scanner and subjectively the better scanner prior to ZMap’s upgrade, the upgrade to ZMap leapfrogs all competitors. With problems such as randomization associated with Masscan, the upgraded ZMap is far more appropriate to be used to conduct research on the internet. Which in turn makes Censys, a reputable research tool. From the three qualities Shodan’s crawlers have, Censys is only able to better in one quality, Randomization. Although upon further investigation the scanning project with the better data value is in contention. This can be observed by investigating the scanner’s behaviour. The crawlers Shodan deploys to gather the data be performing the steps seen in Table 2. Table 2: Crawler's Operation

Step 1. 2. 3. 4.

Action Generate a random IPv4 address Generate a random port to test from the list of ports that Shodan understands Check the random IPv4 address on the random port and grab a banner Go to Step 1

Step 2 in Table 2 uses the phrase ‘list of ports that Shodan understands’, this is not the full range of ports available. The list is 271 ports long (Matherly, 2017). This is done to avoid having to interrogate the possible 65535 ports for each IP address. This saves time by eliminating many banner grabs that will most likely fail. However, this practice restricts Shodan’s ability to deliver a comprehensive view of the Internet. The full list of ports can be found in Appendix B. In comparison, Censys only surveys 16 different protocols, as seen in Table 1, the process in which these are interrogated can be observed in Appendix C (Durumeric, Adrian, Mirian, Bailey, & Halderman, 2015). The protocols Censys scans are more focused on the main ports used such as HTTP, HTTPS and SSH, as well as focusing on SCADA ports such as Modbus. The ports Shodan interrogates are far more diverse including all the port Censys scans and much more. Due to this fact Shodan is often used to identify more interesting and novel services. Even though research documents Censys scanning 16 different protocols, this is not the case, evidence of which is presented later. The information gathered in step 3 are the most commonly service banners, this is the information that is displayed upon attempting to connect to a service, these can contain information such as system specification and warning messages (Jielin, 2007) (McClure, Scambray , & Kurtz, 2009) (Pinto & Stuttard, 2011) (Collins, 2017). This information is then collated to be indexed in the respective searchable interface (Mirataheri, et al., 2013).

22

Figure 11: Crawler’s Algorithm Illustrated

The process detailed in Table 2, can be graphically represented as seen in Figure 11. Where the colour coded steps correspond to that found in Table 2. Shodan’s crawlers must perform the process seen in Figure 11, for every port and address possible. This is known a vertical scan (Bou-Harb, Debbabi, & Assi, 2014). This process takes 3 weeks to drain the pool of all possible addresses, and start again (Matherly, 2014) (Trend Micro, 2017). This is where Censys’ regimented and split scans will perform better as the pools are separate. This type of scanning, is known as a horizontal scan (Bou-Harb, Debbabi, & Assi, 2014). Each horizontal scan is scheduled in accordance to Table 1. The nature, approach and categorization of Censys and Shodan’s scanning methodology can be seen in Figure 12. This figure depicts the active single source horizontal scanning method used by Censys as well as the active distributed vertical scanning method used by Shodan.

Figure 12: Censys and Shodan's scanning behaviour visualised

Research has further expanded upon the 3-step process found Table 2 and Figure 11, coining the process Shodan’s 3 phases of identification (Bodenheim, Butts, Dunlap, & Mullins, 2014).

23

1. The first phase is “Scan Initialization”. This stage encapsulates the previously mentioned behaviour of the crawler, progression onto step 2 is only granted upon a successful scan. 2. The following step is “First Successful Banner Grab”. This entails, a successful banner grab. The authors first recorded this stage occurring after just one day. Although the success rate for this step is dependent on the speed of the crawlers as well as luck. 3. The final stage is “Web interface identification”. Shodan allows the user to access its data through two mediums, web interface and API. After a device is successfully scanned, it is not immediately available via the web interface. This latency between scan and identification can wildly vary. The authors’ noted that it ranged from 1 day to 19 days. This third stage contradicts “the latest picture of the internet” that Shodan boasts (Matherly, 2016). However, due to the difference in publication dates between statements, this may require further investigation due to potential improvements made over time. This is an important piece of research as, if Shodan does in-fact have a substantial latency, especially to the extent observed by previous researchers (Bodenheim et al., 2014), then findings made using data provided by Shodan will be outdated and thus potentially incorrect. John Matherly admits that his scanning method as not being the fastest available and refuses to upgrade to these speedier methods in favour of flexibility (Matherly, 2014). Both Shodan and Censys utilize stateless scanning. Stateless scanning is a method similar to that used by ZMap, Masscan and other asynchronous/SYN Cookie scanners, where there is no per-connection state (Matherly, 2014) (Durumeric, Wustrow, & Halderm, 2013) (Graham, McMilan, & Tentler, 2014) (Bernstein, 1996) (Rohrmann, 2017 ). In this method valuable information is stored in unused packet fields, instead of being held in memory. Upon receiving a response, the sender is able to understand who sent the message, by decoding the information in miscellaneous fields. This process filters out background traffic. This feature removes the timeout features in processes such as TCP handshake (Stevens, 1994), in return for tremendous speed increase. This speed increase is extremely beneficial as both Censys and Shodan experience a sizeable failure rate due to their lack of ability to bypass firewalls or navigate through NAT. Knowledge of this behaviour is widespread. Thus, it is common to restrict interaction by using practices such as NAT and or Firewalls. To compare both Censys and Shodan, several searches and their findings were recorded. For this comparison, academic institutions in Scotland were chosen as the focus. The reasoning being the choice of academia is due to the nature of these organisations. They are less diverse than a geographic filter or net range, this is good as in order to compare Shodan and Censys, the comparison must take place in the overlapping features. In a more diverse scope, Shodan will return greater number of results due the fact that Shodan scans more services than Censys. Limiting to academia will remove this diversity of services to a range of devices both scan for.

24

Comparing Censys and Shodan Results Across Higher Education Institutions 700 600 500 400 300 200 100 0 University A

University B

University C

University D

Censys

University E

University F

University G

Shodan

Figure 13: Comparison between Censys and Shodan results across Higher Education Institutions

From Figure 13, it can be observed that it is more common for Censys to have gathered more information from services than Shodan. This ranges from 2 services for University A to 59 services for University C. There are 2 instances in which Shodan indexes more services than Censys. There may be a reason for this due to the way NTP services are indexed. For University F, the difference between the two results is likely to be due to Shodan indexing NTP services while Censys seems not to, despite research saying otherwise. This lack of NTP services being identified is shown in Figure 14. For University G, the reason must be hypothesised as no explanation is given. It seems that University G may be one of institutions who requested to be blacklisted from scanning. The demographics from the blacklist can be observed in Appendix C, this does include Universities. From this investigation the shortcomings of both scanning projects can be observed. The latency of Shodan as well as the blacklist and failure to index of Censys.

Figure 14: University of F's 277 NTP services as seen on Shodan

In comparison to Shodan and in terms of performance, the founders of Censys say that through their experiment, in which they compared three academic institutions across 3

25

services (FTP, HTTP and HTTPS), Censys discovered 598% FTP hosts, 222% HTTP hosts and 808% HTTPS hosts in comparison to Shodan. This experiment took place over a set period of time, explaining the discrepancy in results. This alludes to a reason for the results noted in Figure 13. Censys was also able to index these devices 23 days faster than Shodan (Durumeric, Adrian, Mirian, Bailey, & Halderman, 2015). This is in line with the approximation that Shodan takes 3 weeks to perform a full scan as well as display these results (Bodenheim, Butts, Dunlap, & Mullins, 2014) (Trend Micro, 2017). Indexing and Filtering Both search engines index their findings as discussed in Scanning and Banner Grabbing, each then amend features upon their findings such as mapping to a geographical location (Matherly, The Complete Guide to Shodan, 2017). Shodan goes a step further than Censys by taking screenshots of some services once connected. This only happens to select services including remote desktop protocol (RDP) and virtual networking computing (VNC). This feature adds a novel use as through these screenshots a user can have a greater knowledge base to make the assumption of what the device is or who is using it. This feature is best showcased in the images search engine within Shodan available at images.shodan.io (Shodan, 2017). Shodan and Censys have overlap in their filtering functions, examples include filtering by geographic location, port number and IP range. Shodan has 47 filters while Censys boasts an impressive 2390 filters (Matherly, 2017) (Censys, 2017). Of these Shodan has 19 generic filters and 28 service specific filters such as ssl.cert.extension and http.status. While Censys has 48 generic filters and 2322 service specific filters, offering the ability to filter in excruciating detail. This filtering is needed to explore the data as an IP address can have multiple entries. A service is an application running on a device, which uses a port as a way to deliver the application. An example of that a web server often runs on port 80, that IP address can also have a SMTP server on port 25. Each of these services will be indexed under the corresponding IP. This gulf stems from the main difference between Shodan and Censys in terms of filtering is the fact Shodan only performs filtering through plaintext and amended data, while Censys differs by allowing filtering through parsed fields (Durumeric, Adrian, Mirian, Bailey, & Halderman, 2015) (Durumeric, 2017). To demonstrate this the same service was found on both Shodan and Censys and the data provided was compared. These services can be observed in Figure 15 and Figure 16.

Figure 15: Lighttpd 1.4.33 as seen in Shodan

26

Figure 16: Lighttpd 1.4.33 as seen in Censys

Vulnerability Reporting Functionality With Shodan is often used to quantify the potential threat a new amplification factor has, for example upon discovery the TFTP DDoS amplification attack, a journalist reached the conclusion that the there was a potential 96,000 TFTP services that could be exploited (Sieklik, Macfarlane, & Buchanan, 2016) (Layden, 2016). This conclusion was reached through a simple search for TFTP on Shodan. Although this conclusion is making the assumption that the services are correctly identified and running a vulnerable configuration. The latter has some merit as identified that services have vulnerable defaults such as Memcached (Matherly, 2015). This vulnerable default was used in the Memcrashed tool, which harnessed the Shodan API to identify potential Memcached services to launch a DDoS attack (@037, 2018) (Khandelwal, 2018). This tool was used to synthesize the largest known DDoS ever was recorded, peaking at 1.35Tbps, beating the previous record of the Dyn attack which was 1.2Tbps (Skottler, 2018) (Ranger, 2018). Not all vulnerabilities found using Shodan need to be assumptions as the Shodan crawlers tests for specific vulnerabilities (Trend Micro, 2017) (Simon, Moucha, & Keller, 2017). The findings from these tests can be accessed using a filter however its implementation has been limited to only a few vulnerabilities. Those are DVR configuration vulnerability, argument injection, Heartbleed, FREAK, Jetty credential vulnerability and Ticketbleed(Ramos, 2013)(NVD, 2013)(NVD, 2017) (Synopsys, 2017) (NVD, 2014) (NVD, 2016) (NVD, 2017). In comparison, upon release of Censys, support for the testing of only two vulnerabilities were available and was accredited to the modularity of ZMap (Durumeric, et al., 2014). These vulnerabilities are Heartbleed and POODLE (NVD, 2017) (Synopsys, 2017) (NVD, 2014) (Möller, Duong, & Kotowicz, 2014). Both Shodan and Censys can identify services vulnerable to Heartbleed. The Shodan query to perform this check is “vuln:cve-2014-0160”. For Censys, this query is performed by searching for “443.https.heartbleed.heartbleed_vulnerable: True” (Durumeric, et al., 2014). To investigate this functionality, several searches will be performed and the results will be recorded. These results are displayed in Table 3. For each country tested in Table 3, there is a difference in results returned, however the results are returned within similar range. This discrepancy may be due to the scanning discrepancy as seen in Figure 13, however, this can’t be certain. This shows the uncertainty in the built-in vulnerability checking mechanisms. This is an area which requires future research.

27

Table 3: Results from Heartbleed Queries

Test Data Origin Great Britain Ireland Canada France

Shodan 3645 856 3301 5500

Censys 2557 657 3149 4060

As an aside, ZoomEye also displays vulnerability functionality tied to another Knownsec project, Seebug (Seebug, 2018). Exploits for vulnerabilities related to the returned results, are given with the corresponding results, they are indexed by Seebug Security Vulnerability ID, along with date submitted, level of difficulty to execute and title. This can be seen in Figure 17.

Figure 17: Screenshot of the Vulnerability Functionality Found within ZoomEye

Historical Data Retention Upon comparison to Shodan, Censys’ founders claim that Shodan’s results do not expire (Durumeric, Adrian, Mirian, Bailey, & Halderman, 2015). While this is true, as Shodan’s results only update upon observing a change within the indexed result, however their further claim of the lack of timeline present for indexed results is inaccurate. A result’s historical data can be found by using the history flag present within the application programming interface, this feature was used to great effect in ShoVAT (Shodan, 2017) (Genge & Enăchescu, 2016). Shodan recently implemented the ability to access the images, captured within the scanning process that are stored within the history of the service to be downloaded and created into a time-lapse gif (Shodan Help Center, 2017). Whereas Censys’ historical records cannot be accessed through the search interface or API. To access any historical data surrounding Censys scans they must be accessed through the repository scans.io (Durumeric, Adrian, Mirian, Bailey, & Halderman, 2015) (Censys, 2017) (Scans.io, 2018).

28

Current Vulnerability Assessment Tools A common way to identify known-vulnerabilities is through the use of vulnerability assessment tools. This section will look to investigate the eco-system of surrounding these tools, through personal use and consultation of current literature. The previously mentioned tools OpenVAS and Nessus are both active in nature, this is not always the case as seen in Figure 18, were depicted is a vulnerability assessment tool comparison found in recent research (Genge & Enăchescu, 2016). In this comparison it can be observed that tools featured are either of an active or passive nature, this aligns with cyber-attack anatomy, as seen in Figure 4.

Figure 18: Vulnerability assessment tool comparison (Genge & Enăchescu, 2016)

Included within the comparison are Nmap, Hershel and ZMap, the latter of which is used in Censys. However, these are asset discovery tools not vulnerability assessment tools. They do have the ability to return identifiers of services, however this data is not further analysed. Therefore, they fall outside the scope of this investigation and therefore, will not be discussed further (Durumeric, Wustrow, & Halderm, 2013) (Lyon, 2009) (Shasmi, 2017). P0f is a purely passive traffic fingerprinting tool designed to identify those involved in any TCP/IP communication, commonly used during routine network monitoring and provide reconnaissance on penetration testing (Zalewski, 2017). It does not intercept packets thus, being passive in nature. When operating on a networked device, p0f analyses ongoing connections, providing information such as IP addresses, operating systems and applications. No further analysis is done beyond supplying the information found within the packets, this leads to results found in Figure 18. Similarly PRADS (Passive Real-Time Asset Detection Systems) is based upon P0f and is a tool designed to passively listen to network traffic, gathering data about hosts and services, which is further analysed to provide a map the network being listened to (Sanders & Smith, 2013). A problem with a passive approach to network monitoring is that services which are idle will not be identified. Ironically, an active approach helps passive monitoring by create a correspondence with these idle services.

29

NetGlean is a distributed security scanner that continuously monitors the network which it is hosted on. Subsequently providing real time and historical views of large and complex networks (Manes, Schulte, Guenther, & Shenoi, 2005). Using distributed sensors across a network, which individually fingerprint systems via active probing and passive packet sniffing, to update the results in a corresponding database. SinFP does not use a distributed approach but does follow a similar structure, with option to actively and passively fingerprint systems, with the results updating an underlying database (GomoR, 2012). Tenable’s Nessus was founded in 1998 by Renauld Deraison and is a remote security scanning tool (Beale, Haroon, van der Walt, & Deraison, 2004). When operated Nessus actively probes services in order to test for known vulnerabilities and possible service configuration weaknesses (Genge & Enăchescu, 2016). These probes are available through plugins, these plugins test for specific known-vulnerabilities specified by their related CVE (Wendlandt, 2010) (Tenable, 2018). Plugins usually are tied to vulnerabilities indexed within the National Vulnerability Database. Plugin 54646 is an example of a plugin with a concomitant CVE, CVE2011-0419 (Tenable, 2011) (NVD, 2011). When used this plugin activates a correspondence with the target performing a check through supplying a command to identify the current configuration of the system. Through this process, a network correspondence is made. This is significant as these can be monitored using tools such as p0f and PRADS. The plugins exist in families, these include Windows, Linux, Cisco and Database (Kumar, 2014). These plugins become available soon after announcement of the vulnerabilities (Tittel, 2016) (Mukhopadhyay, Goswami, & Mandal, 2014). A similar tool to Nessus which is omitted in Figure 18 is OpenVAS. It is an open source fork of Nessus, performing similar actions by actively communicating with the target service using Network Vulnerability Tests (NVTs). These NVTs sometimes have associated CVEs. OpenVAS is publically available for free on Kali Linux distributions. A problem associated with an active approach is that the tools can contribute to Denial of Service attacks through their disruptive nature on the target network. This approach can take considerable time to operate, sustaining the potential of DoS attack (Skybox Security, 2016).

National Vulnerability Database The National Vulnerability Database (NVD) is a public repository using the Security Content Automation Protocol (SCAP) to collate vulnerability management data Security (Quinn, Waltermire, Johnson, Scarfone, & Banghart, 2011). The NVD is managed by the National Institute of Standards and Technology (NIST, 2016). The NVD includes several separate databases, such as for checklist references, software flaws, misconfigurations, product names and impact metrics. Since inception in 1997, the NVD has published information about more than 94,000 vulnerabilities affecting more than 17,000 applications (Zhang, Caragea, & Xinming, 2011) (MITRE, 2017). Figure 19 features the numbers of vulnerabilities released per year.

30

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

18000 16000 14000 12000 10000 8000 6000 4000 2000 0

1997

Number of CVEs

Number of CVEs by Year

Year

Figure 19: Number of CVEs by Year since 1997

Of the feeds contained within the NVD mentioned prior, the most applicable to Internet-wide scanning is that of Common Platform Enumeration (CPE), as the data need to create a CPE can be sourced through fingerprinting (Buttner & Ziring, 2008). A CPE is a unique identifier for a specific version for a particular hardware, service or operating system. A more in-depth look at CPEs is discussed later. They are used to partition issues identified in other data feeds within the NVD to an exact application, as these issues maybe fixed and patched in different version. An example of a feed they are used in is the Common Vulnerabilities and Exposures (CVE). This database contains known-vulnerabilities and for each one there is information such as a description, severity through Common Vulnerability Score System (CVSS) and vulnerably configurations through CPEs. An example of this can be observed in Figure 20.

Figure 20: CVE-2017-9788

31

Shortly after publication of CVEs, security researchers will verify the vulnerabilities by creating proof of concept (PoC) exploits. These are only to verify the potential of exploitation, they are not to manipulate or exploit the target. It is these PoC’s which are used by vulnerability assessment scanners such as Nessus and OpenVAS to probe targets. Despite the best intentions, the NVD suffers from a plethora of problems. One identified problem is the lack of reliability of data prior to 2006. This data is often excluded from predictive studies (Khadilkar, Rachapalli, & Thuraisingham, 2010). A particular problem within the NVD are the accuracy of the initial entries upon release of a new vulnerability. 89% of new vulnerability entries are edited within the first 30 days typically to improve their accuracy. Only 4% are not changed within the first 150 days (Karlsson, 2012). With ‘vulnerable versions’ being the most edited field, this highlights that there is an issue with the reliability of the data for new vulnerabilities. This finding highlights the potential of reductions in the reliability of the NVD information overall. As this information is widely used by many researchers this could consequently have impact upon the validity of findings. In line with reliability, research has identified that known vulnerabilities associated with Google Chrome have an erroneous rate of 25% (Hung Nguyen & Massacci, 2013). The aforementioned editing may be of some cause of this error rate, nonetheless this error rate dampens the reliability of the NVD. Another problem within the NVD is synchronisation. Research has identified numerous cases of asynchronous data. These contain close to 1,000 CVE entries without related CPEs, despite the CPE existing. Similarly, there are over 100,000 CVE entries with CPE entries that do not exist (Sanguino & Uetz, 2017). This causes problems, especially with matching and navigating vulnerabilities. On top of all of this the NVD only catalogues an estimated 50% of all vulnerabilities in existence (Coman, 2017). As well as this, an estimated 6,000 vulnerabilities in 2015 went ‘unassigned’, meaning vulnerabilities were discovered and disclosed to the NVD however they did not receive an identifier. This number was followed by 7,900 in 2017 (Ragan, 2016) (Tara, 2018). This demonstrates the incompleteness of the National Vulnerability Database

Service Banner Analysis to Identify Vulnerabilities When identifying vulnerabilities using the National Vulnerability Database, the use of elements such as Common Platform Enumeration (CPE) and Common Vulnerability and Exposure (CVE) are needed. CPE is the standard set for uniquely naming assets, while CVE is the standard for vulnerabilities. Each entry in the CVE dictionary contains the concomitant CPEs for assets that are vulnerable to the CVE. Both of these standards are encompassed in the framework known as Security Content Automation Protocol (SCAP) (Fitzgerald & Foley, 2013). In previous research, authors investigating varying topics such as log analysis and patch monitoring have proposed and implemented ways to create Common Platform Enumerations. The following section contains an investigation into how current literature addresses the creation of CPEs. To clarify what a CPE is, it is a unique name comprised of several fields detailing the specific version of a hardware, software or operating system. This structure can be observed in Figure 21.

32

Figure 21: CPE composition with corresponding levels labels (Na, Kim, & Kim, 2017). Creators of CPEs, the National Institute of Standards and Technology, have released a specification for performing CPE name matching (National Insititute of Standards and Technology, 2011). In this specification NIST outlines an official method to create CPEs to then be validated. Their method uses the Well-Formed CPE Name (WFN), which is void of the structure seen in Figure 21. A WFN splits the structure of a CPE into fields corresponding to its composition, performing matching by using an AND operator with fields identified. An example of a WFN for internet explorer 8 can be found in Figure 22. This figure omits unspecified attributes values such as those found in Figure 21. This is the set out standard however it is not used throughout literature.

Figure 22: The attributes found in the Well-Formed CPE Name

Gawron et al explored CPE creation in their research surrounding patch monitoring this topic through processing the system logs on local machines (Gawron, Cheng, & Meinel, 2017). The authors propose a new passive vulnerability detection method by analysing system logs and creating identifies to tie to known vulnerabilities. For Windows and Unix systems, they parse the relevant log files using regular expression to capture program and version for each entry. Their proposed regular expression for Unix logs can be found in Figure 23. However, the regular expression used is inadequate to fulfil the full composition of a CPE as it omits the vendor field (Buttner & Ziring, 2008) (National Insititute of Standards and Technology, 2011). This was omitted due to their analysis of system logs that don’t include the vendor. This omission obstructs the possibility of manufacturing a CPE without additional computation to associate the correct vendor, this omission also could create a problem that may arise when programs share the same name but not the same vendor causing incorrect results. This method also omits any consideration for typographical errors discussed in relation to the NVD (Sanguino & Uetz, 2017). This displays the level of detail needed to obtain a correct CPE. Despite their methodology, the authors’ proposed method contains noticeable pitfalls that need addressing when approaching CPE creation, this is applicable to CPE creation through service banner analysis.

Figure 23: Proposed regular expression by (Gawron, Cheng, & Meinel, 2017)

33

Sanguino et al explores CPE creation in relation to Vulnerability Management Systems (VMS), discussing a method to perform Common Platform Enumeration creation, matching the CPE to a third-party software database through a 3 step process (Sanguino & Uetz, 2017). In the first step, Generate Search Term, WFNs are created by scraping the third-party software database known as GLPI database. From the information gained, observed in Table 4, WFNs terms are created, observed in Table 5. Interestingly the search terms created displays Microsoft appearing in both vendor and product, this is significant as it is done to consider CPEs which do not adhere to the correct semantics outlined in the specification, a culprit for this is Nginx and Lighttpd (Buttner & Ziring, 2008). This is one of the steps the authors’ employ to avoid the inconsistencies which can affect the matching success rate, however this requires having a database of precompiled information. Table 4: Software product obtained from the GPLI database seen in (Sanguino & Uetz, 2017)

Vendor Product Version

Microsoft Corporation Microsoft .NET Framework 4.5.2 4.5.51209

Table 5: Search Terms generated by information shown in Table 4

Vendor Product

mircosoft_corporation, microsoft, corporation microsoft_.net_framework_4.5.2, microsoft_.net_framework, micrsoft_.net, microsoft, .net_framework_4.5.2, .net_framework, framework, .net, 4.5.2

The second step, Search by Product and Vendor. In this step the previously created search terms are used to generate the CPE WFNs attributes, these are then checked against the CPE dictionary. Where using the formula featured in Figure 24 is used.

Figure 24: Search by product and vendor formula

The distinction must be made with the inclusion of the approximately equal (≃) symbol instead of the equal (=) symbol. This is because the formula uses the Levenshtein distance algorithm to mitigate the previously mentioned problem of typographical errors. This algorithm returns the number of dissimilarities between two strings (Lee & Peter In, 2017). For this formula the Levenshtein distance is set to being less than or equal to 2, if successful the CPE is considered a match. This threshold of 2 was determined by analysing the deprecated CPEs in the CPE dictionary, commonly depreciation took place due to a typo in the product with a Levenshtein distance of one or two. This step has the ability to produce more than one CPE, therefore, the last step which Order by Version is required. In this step the numerous CPEs found in the previous step are ordered by version to then perform a CVE lookup. By not selected one CPE, the list of CPEs returned is full of false positives.

34

From the limited research into creating CPE identifiers from data, exists a smaller subsection of research focuses on creating CPE identifiers from service banner information. It seems only 2 approaches are outlined in current literature. Na et al propose operating system identification through service banners (Na, Kim, & Kim, 2017). The method used includes the generation of CPE tree based on the CPE dictionary. This tree is used after conducting keyword analysis upon a service-banner, from which the keywords extracted are used in conjunction with the CPE tree to create the closest CPE. The tree consists of 6 levels, observed in Figure 21. Vendor (level 1) from which has a child note product information (level 2). Sequential levels are created in the same manner. Considerations are made for the specifications of each level. Keyword analysis is performed on the service banner, specifically looking for level 1 and 2 information as a collective. If successful consecutive levels conduct keyword analysis dependent on the results of the previous levels (1 and 2) analysis. For version information, a version listed 2.10.3 the highest level of version detail is taken into account, for the example given 2 is used. This process can be observed in Figure 25. Combining the keywords together, they are converted into CPE from level 1 in order. The longest CPE is created however, if through another keyword branch there is a possible longer CPE then that is also created. The outlined approach has several potential weaknesses, one such weakness if the inability to operate on incomplete data provided by a service banner. If a banner does not include the Vendor and Product information then the method stops. This reliance on vendor and product, creates a dismissal of the importance of intricate version numbers, thus, having the potential of returning the wrong CPE, creating false positives. This proposed method doesn’t include considerations for errors and issues identified in previously mentioned research on National Vulnerability Database (Sanguino & Uetz, 2017). As outlined prior, this is significant as the level of impact this may have on potential results is of an alarming scale. This problem could be amplified through the proposed method’s selection of the longest CPE. A longer CPE maybe one that includes a spelling mistake (Sanguino & Uetz, 2017).

Figure 25: CPE Tree illustrated with corresponding levels. The approach found within Shodan Vulnerability Assessment Tool, ShoVAT, utilises hash tables to store elements of CPEs corresponding to version numbers. Instead of searching for specific sub strings through the large CPE database, ShoVAT identifies version numbers

35

matching the regular expression pattern, Vpat = “I.(I.)*I”. Once the pattern is identified, the hash table is used to associate the other elements of the CPE such as Vendor and Product. If this association is not decisive then surrounding information in the banner is used to identify the most appropriate banner. A problem arises in this approach through the need to update, store and manage the hash table. This hash table may also include depreciated and erroneous CPEs identified in related literature (Sanguino & Uetz, 2017). Another problem is the reliance on identifying a pattern. This may not identify CPEs which do not have a version number, or a version number that is not applicable to the regular expression patterned identified.

Conclusion In this literature review, the aim was to conduct a critical analysis of key research in the topics surrounding Internet-wide scanning and vulnerability assessment. This section was structured in 4 subsections around the aim of the project: Internet-wide Scanning, Current Vulnerability Assessment Tools, National Vulnerability Database and Service Banner Analysis to Identify Vulnerabilities. In Internet-wide Scanning, the section identified multiple projects currently providing this service, however of those only Censys and Shodan were identified as appropriate due to their functionality, supporting documentation and the associated academic research (Durumeric, Adrian, Mirian, Bailey, & Halderman, 2015). Censys and Shodan were then analysed further, from which some interesting conclusions were drawn. The research surrounding Shodan and Censys has shown that they approach Internet-wide scanning in differing ways, in that Censys opts for a regimented horizontal scan using ZMap, and is focused on the most frequently used Internet services (Bou-Harb, Debbabi, & Assi, 2014) (Durumeric, Adrian, Mirian, Bailey, & Halderman, 2015). Whereas Shodan uses a comprehensive vertical scanning method using an approach similar to ZMap, including novel and uncommon services (Bou-Harb, Debbabi, & Assi, 2014) (Matherly, 2014). This difference in the scanning approach highlights Censys having more up to date scanning data, more accurate, and less randomised available data than Shodan. The way this data is presented and available also differ between the platforms. Shodan provides the data collected for the service banner is provided in its raw form and in plain text. While Censys provides the data, abstracted into a number of formatted fields. With this difference having implications on the approach taken to process the data provided, with the plain text providing the banners and the abstracted data only providing certain parts. Both Shodan and Censys provide limited vulnerability assessment capabilities but do some limited service fingerprinting. Shodan is able to identify 6 known vulnerabilities, and Censys provides support to identify 2 known vulnerabilities via tags for those specific filters. The only vulnerability both tools are able to identify is Heartbleed, and upon investigating the Heartbleed results across platforms it was evident there seemed to be discrepancies in the output. It seems this is probably due to the different scanning processes analysed earlier. The current vulnerability functionality available is far less advanced than standard standalone vulnerability assessment tools, and this seems like it could be an avenue for exploration. An investigation into the current array of vulnerability assessment tools was done. In this field, the tools look to be split two approaches, passive or active. These approaches align with that of the reconnaissance phase of cyber-attack anatomy (Weidman, 2014) (Genge & Enăchescu, 2016). Passive vulnerability scanners such as p0f and PRADs are not disruptive to

36

the target network, however they have very little functionality in identifying vulnerabilities. While active vulnerability scanners such as OpenVAS and Nessus can be disruptive to the target network, can take considerable time to operate, however they are capable of identifying known vulnerabilities (Skybox Security, 2016). This looks to have possibly identified a gap in current offerings, by providing the capability of identifying vulnerabilities in a passive manner by passively using the active scanning data from Internet-wide scanners. And this approach could negate the associated active vulnerability scanning issues. When reviewing literature around Service Banner Analysis to Identify Vulnerabilities, all approaches seem to utilize the common Security Content Automation Protocol (SCAP) standards: Common Platform Enumeration (CPE) and Common Vulnerability Enumeration (CVE). This clear trend prompted an investigation into the literature surrounding National Vulnerability Database (NVD) which stores CPE and CVE data. One of the more significant findings to emerge from this study is that despite its application in many research projects and industry tools the NVD is deeply flawed. Work in the field has shown that the data present within the NVD is mutable, and can contain errors in descriptions or have problems with asynchronous data across relations (Karlsson, 2012) (Hung Nguyen & Massacci, 2013). A significant problem relevant to service banner analysis was simply the incorrectly spelling of CPEs, which could have a significant impact as CPEs are used as a stepping stone from service banners to vulnerabilities (Sanguino & Uetz, 2017). By matching to the wrong CPE, then the wrong vulnerabilities could be associated. This specific problem has failed to be addressed in all but one of the pieces of literature in service banner analysis. In the sole piece of research identified which considers the errors present in the NVD, the Levenshtein distance algorithm was utilized to perform matching on approximately equal strings, rather than an exact match (Sanguino & Uetz, 2017).

37

Methodology This methodology will outline the design and implementation of a tool which will use Internetwide scanning data to manufacture industry standard identifiers for services then associate these identifiers to known-potential vulnerabilities. The design will be influenced by comparable works found in current literature. The method outlined will be compared to that of similar projects.

Design In this section, the proposed approach will be put forward. With each process being thoroughly discussed. The operational flow can be broken down into 3 sections: input, process and output. This can be observed in Figure 26. With the Input coming from an Internet-scanning project, the Process being the vulnerability assignment and the Output being the presentation of results.

Figure 26: IPO Flow Diagram

For Input, Censys was chosen as the Internet-scanning project. Censys has many qualities which make it more appropriate than the alternative Shodan. Censys is far more up to date, the top 1 million websites are scanned daily, port 80 for the full IPv4 range is scanned weekly, with other ports scanned fortnightly. In comparison Shodan takes 3 weeks or so to scan the full IPv4 range. This time difference consequently makes Censys’ data far more valuable. In line with scanning procedure, Censys’ scanning process has seen to be unmatched in terms of randomness, this is significant as the randomness decreases the likelihood of bias within the result set. Randomness also reduces the probability of being identified and excluded by firewalls, meaning Censys’ data is more likely to be complete. These reasons combined contribute to the increased worth of input data, subsequently the tool’s output worth. For implementation, Censys differs by having pre-processed fields in their output, whereas Shodan provides banner information within a plain text field thus, requiring a procedure to extract information. This extra procedure adds more complexity to an already intricate process. Although performed in prior research, this process could be another project in itself (Genge & Enăchescu, 2016). Including the external factor of a time constraint placed upon on this project, as well as producing valuable output, Censys was deemed the most suitable Internet-scanning project to take onwards to implementation. For the process procedure, the goal is find associated vulnerabilities with the given input. To achieve this the SCAP standards Common Platform Enumeration and Common Vulnerability Enumerations will be used. The latter on only data feeds in rather obese Json or XML formats. To avoid the approach of downloading and using these files, which requires updating

38

regularly, an approach found in pervious literature was employed. This approach uses a tool known as cve-search is used to obtain information from the NVD (Dulaunoy, Moreels, & Vinot, 2018). Cve-search culminates several data feeds including those of the NVD within a MongoDB, which can be accessed and searched easily by Python. Potential use of the combination of Censys’ Python API and CVE-Search lead to the adoption of Python to be used as the language for implementation.

Output is vital to present the information gained through process in a meaningful way, this will contribute to quantifying the usefulness. The output should display all relevant information associated to each result. The fields can be found in Table 6. Table 6: The desired output fields

Field IP Address Metadata Common Platform Enumeration Vulnerabilities Common Vulnerability Enumeration

Description IP Address for the given result Relevant information for identifier Identifier formed from Metadata A List of Vulnerabilities Individual Vulnerabilities

As detailed above, the specific approach for each process have been chosen and justified. The flow of the operation was originally defined in Figure 26, however this has been refined through selecting and justifying the method for each process, as detailed above. This new detailed operational flow can be observed in Figure 27.

Figure 27: Detailed IPO flow

Through this outlined approach, this design can be classed as contactless reconnaissance (Simon, Moucha, & Keller, 2017). This differs from the active approach found in tools such as NMap, OpenVAS and Nessus as these tools directly contact the target systems. Whereas the approach outlined prior employs an indirect method by communicating with a search engine which has already collated active correspondence with the target systems. A graphically comparison of these approaches can be found in the research by Simon et al (2017), this can be observed in Figure 28. Compared to the contact-based approach, found on the left-hand side, contactless approach, found on the right-hand side, adds the inclusion of Search Engine. This addition adds a critical layer of removal from the user to the target however, this interpretation omits the possibility of latency. By retrieving information from a search engine there is an inherit latency contributed to their information gathering process. The latency is significant as time progresses the likelihood of the data being accurate diminishes, as accuracy decreases so does the worth of the data. For Shodan and Censys this latency varies from a day to many weeks. To more accurately consider and represent this latency a new figure was designed, this can be seen in Figure 29.

39

Figure 28: Comparison of Contact-based and contactless test methods (Simon, Moucha, & Keller, 2017)

Figure 29: Hypothesised contactless reconnaissance

A similarity noted through the comparison of contact and contactless approaches was that the same information can be obtained. This is due to the same active reconnaissance methods being used to collect the information with the only discrepancy being the collection is done by a third party for contactless. To properly explain this and distinguish itself from the reconnaissance performed on more classical search engines, I propose the term contactless ‘active’ reconnaissance. This term encompasses the use of active reconnaissance results collected by an associated third party. As an example, using Shodan or Censys would be classed contactless ‘active’ reconnaissance. Upon designing the tool, a name had to be chosen. The only similar tool is rather unimaginatively named ShoVAT which stands for Shodan Vulnerability Assessment Tool. In an

40

effort to stand out from the monotony of named research project tools, a relevant but simple and catchy name was needed. When exploring synonyms of words related to reconnaissance, Scout stood out as a fitting name. With connotations of exploration and discovery, Scout was chosen to be the rather apt name for the tool.

Implementation To implement the above design, several impactful decisions were made. This section looks to detail the implications of these decisions and record how the design was applied in the creation of the resulting tool, Scout. As seen in the design phase, this tool relies on two external properties, Censys and The National Vulnerability Database. These dependencies in relation to the internal flow can be observed in Figure 30. External dependencies are depicted as actions out with the domain of Scout. Using Figure 30 as a basis of the following section, each process featured will discussed at length.

Figure 30: Internal flow and external dependencies of Scout

Generate Query To program a Censys search through the python API package, the requirements needed are a valid set of API keys, a query and the desired fields of output. An example of the requirements, implemented can be observed in the query follows the same syntax used in the front-end web application, while the fields are a desired subsection of the results found on the web application. Fields are required and limited to a maximum number of 20, only a subsection of the overall results that can be observed in the web application can be returned. This prompted an investigation into the possible fields which could be utilized by Scout further along in the operation. By reviewing the documentation provided by Censys, 14 unique services were identified as possible data for Scout, however some of these services were spread across multiple ports. For example, HTTP is scanned for by Censys on port 80, 8080 and 8888. To consider all the possible services across all the possible ports they may operate on more than the maximum 20 fields are needed. Thus, a compromise is needed.

41

Figure 31: Code snippet featuring a Censys search

An approach put forward was to use two Censys searches. The first search would take the user defined query and locate all ports indexed by Censys in the specified query. For example, the query used would be “IP:192.168.0.0/16”, and specified fields would be simply “tags”. The results would return all the unique ports used in the given IP range, as an example “80, 22”. The second search would use the same user defined query however use the list of returned ports found in the first query to specific the necessary fields to further Scouts operation. For example, “IP:192.168.0.0/16” and the specified field would be “‘80.http.get.metadata.description’, ‘22.ssh.v2.metadata.description’”. This approach would return the greatest number of services which is can then be operated by Scout, with no redundant or omitted data. Although this approach has a significant problem, as discussed prior Censys uses a credit system to monetize their application. In this approach 2 credits are used, 1 for each query used, this significantly reduces the number of uses of the tool per month. In an addition to limiting the uses of Scout, this could and would prove disruptive during implementation and testing. Therefore, another approach was required. The approach which was chosen was to simply use one query. This query was similar to the second query used in the approach detailed above however, this query differed by using preselected fields over ones which Censys located. The chosen preselected fields were the most popular fields found through use of Censys, ‘80.http.get.metadata.description’, ‘22.ssh.v2.metadata.description’ were included. An example of output from this search can be observed in Figure 32.

Figure 32: Output for Censys search

Censys Results The results of the Censys search are provided as a generator, this is the structure that can be observed in each line in Figure 32. From the figure, differing output can be seen with some generators returning ‘80.http.get.metadata.description’ and ‘22.ssh.v2.metadata.description’ or just ‘80.http.get.metadata.description’ or nothing more than an IP address. This is due to the results from Censys always returning an IP address, however only returning the fields applicable. This complication adds complexity to the further operation of Scout. As Scout is dependent on the metadata provided by fields such ‘80.http.get.metadata.description’. This dependency is depicted in Figure 33.

42

Figure 33: Output visualised by process

Figure 33 shows that the processing hinges upon the metadata given by the Input, depending on the metadata field contents or lack thereof, the process for this IP Address will stop there. These metadata contents will then be used in an attempt to assign an associated Common Platform Enumeration. Common Platform Enumeration Matching In this step, the information provided by Censys is analysed in an attempt to assign the service an identifier. The identifier chosen in the design, indicative of the literature review conclusions. Table 7: Methods of CPE matching in literature

Literature Passive Vulnerability Detection (Gawron, Cheng, & Meinel, 2017) Software Vulnerability Analysis Using CPE and CVE (Sanguino & Uetz, 2017)

A Study on the Service Identification of InternetConnected Devices Using Common Platform Enumeration (Na, Kim, & Kim, 2017) Shodan-based Vulnerability Assessment Tool (Genge & Enăchescu, 2016)

Method Used A regular expression for Program and Version Multiple search terms then use of Levenshtein algorithm Hierarchical Tree

Multiple Hierarchical Trees

As discussed in the literature review there are services with related CPEs which are incorrect, depreciated or non-existent. It is significantly detrimental to the accuracy of the data if incorrect CPE is used. Most of the current literature, as seen in Table 7 neglect considerations for these possibilities however, Sanguino & Uetz (2017) detailed an approach in which the NVD errors were factored into the design. This was accomplished using an external software database to create elements of a well formatted name CPE. Different combinations of these elements are searched against the CPE dictionary with the applicable Levenshtein distance algorithm used to consider spelling mistakes. This method has the potential to return multiple CPEs, of which the most suitable candidate, determined by version number, is chosen.

43

Although this approach was well thought out and considerate of the errors with the NVD. The approach has flaws. The requirement of an additional software details database, adds another cumbersome dependency to the approach. Through the utilization of this database multiple candidate CPEs are made, then listed in order of version number as potentially correct. This does provide clarity on if it is reasonably safe to upgrade the software but results in many false positives. The chosen approach was a blend of previous approaches, using the Common Platform Enumeration dictionary found within the NVD and available through the cve-search tool. This dictionary was iterated over and for each iteration the given metadata returned by Censys was symmetrical compared to the CPE. If the result of this symmetric difference was 0 then the CPE was appended to the elected candidate list. If the difference was less than 2 then CPE was appended to a vetted candidate and if difference was equal to two then it was appended to accepted candidate. This approach can be observed in this pseudocode found in Figure 34.

Figure 34: CPE matching pseudocode

Once enumerated through the CPE dictionary, the 3 lists are analysed. Starting with elected then vetted, then finally accepted candidates, each length of list is checked, if the result returns 0 then the next list is analysed, if the length of returns 1 then that result is treated at the final CPE, however if the length is greater than 1 then the matching finishes inconclusively. This operation can be visualised using a UML activity diagram, this can be seen in Figure 35.

44

Figure 35: UML activity diagram for choosing CPE

Figure 36 shows when provided nginx 1.4.3, the algorithm returns “cpe:2.3:nginx:nginx:1.4.3”, which is the correct CPE. It reached this conclusion through the use of the tiered lists. There is 1 result for both elected and vetted candidate and a plethora of accepted candidates. Through the prioritisation of the candidate lists the result of in the elected candidate list is treated as the CPE over the candidates found in vetted and accepted.

Figure 36: Nginx 1.4.3 candidates

When provided any Apache httpd product, problems arise as they does not correspond directly to a CPE, the correct corresponding CPEs follow the syntax of

45

“cpe:2.3:a:apache:http_server:x.x.x”. Through the usage of the Levenshtein distance algorithm, with the given httpd against http_server. A distance of 7 will be returned. This is less than the length of both strings combined which is 16. This is due to common ‘http’ as observed in Figure 37. Through the assumption that any results that finds a smaller Levenshtein distance than the combined length is due to an inconsistency with the CPE syntax. This works correctly for apache httpd services as can be seen when comparing Figure 38 and Figure 39. The former depicts the matching algorithm without the addition of the Levenshtein algorithm returning in an inconclusive result, whereas the latter shows that when implemented the Levenshtein algorithm is able to narrow down the results to only the correct one.

Figure 37: Levenshtein distance visualised

Figure 38: Apache 2.4.7 CPE Matching without Levenshtein algorithm

Figure 39: Apache 2.4.7 CPE Matching with Levenshtein algorithm

CVE Assignment With the CPE decided in the previous section, it is then used in the CVE assignment. Included within CVEs, is a field of the vulnerable configurations. This field is a list of CPEs which correspond to services vulnerable to the CVE. By using cve-search this field can be searched with the chosen CPE. This is performed by using the code seen in Figure 40, a find function is used to search CVEs by their “vulnerable_configuration” field. For each CVE that is returned the ID and corresponding CVSSv2 score is stored.

46

Figure 40: CVE Search code snippet

Present Output The final step in Scout’s operation is to present the findings. The output should be clear and understandable. The method chosen for Scout was simply to print the results in a JSON like format, with clear indentation to isolate each result. This caters itself well to command line interface, which is the preferred environment for this tool. An example of output can be seen in Figure 41.

Figure 41: Scout Output

47

Similar Projects In early 2018, a project called AutoSploit was released on Github (NullArray, 2018). AutoSploit uses Shodan and MetaSploit in an attempt to automate the exploitation of external hosts. AutoSploit works by collecting targets through a user defined query on Shodan’s API, from then this specific query is matched with relevant modules within the MetaSploit Framework and then performed automatically (NullArray, 2018). A problem with the project is that the Shodan query must align with the name of modules found within the framework. This limit the targeting capabilities of the project, thus the overall effectiveness (Gallagher, 2018). The release of AutoSploit created waves within the information security community, with most voices declaring its release dangerous, particularly the potential danger to ICS devices (Leyden, 2018). The danger comes from the automation, Chris Morales of Vectra said, “...these two highly automated tools are combined to make life even easier for someone to hack systems lowers the bar much more” (Robinson, 2018). Richard Bejtlich of TaoSecurity went as far as to say “There is no need to release this…This will end in tears” (Bejtlich, @TaoSecurity, 2018) (Gallagher, 2018) (Cox, 2018). Bejtlich is on record warning that Shodan would usher in Intrusion as a Service (IaaS) (Bejtlich, Shodan: Another Step Towards Intrusion as a Service, 2009) and in ways AutoSploit is several steps further down the IaaS path. Previously referenced Robert Graham explains that AutoSploit is nothing new (Graham R. , @ErrataRob, 2018), this is reinforced as previously referenced Dan Tentler said that the fear and outrage of AutoSploit is misplaced as Dan delivered talks on how to pipe Shodan into the Hail Mary feature of Cobalt Strike nearly a decade ago, however never received comparable reception to that of AutoSploit (Gallagher, 2018) (Tentler, 2018). Scout does not go as far as AutoSploit by automating exploitation. This was a chosen design choice as doing so would break the computer misuse act 1990. Through a series of published works, Bela Genge et al discuss the novel idea of identifying vulnerabilities passively, culminating in their tool known as ShoVAT (Genge & Enachesu, 2015) (Genge & Enăchescu, 2016; Genege, Haller, & Enăchescu, 2016). ShoVAT uses Shodan, hence the name, to perform contactless active reconnaissance, although that term was coined later. ShoVAT takes Shodan input and creates CPEs then associates them to known-vulnerabilities. The methodology utilizes a heavy importance on identifying version numbers, to correspond with an entry in a hash table containing possible CPEs. ShoVAT is the first and still only published work about a tool relying solely an internet scanning project to identify known vulnerabilities. This novelty can be seen in its inclusion in Figure 18. ShoVAT discovered a total of 3922 known vulnerabilities across 1501 services from 12 education institutions. A criticism of research published about ShoVAT is that it focuses too heavily on the performance aspect of the tool over the accuracy (Simon, Moucha, & Keller, 2017). This can be seen in the evaluation of ShoVAT where the experiments used did not meet a reproducible standard. Scout differs from ShoVAT by using Censys over Shodan and by having a less dependent CPE creation process considerate of the erroneous NVD. In the forthcoming experiments Scout will be evaluated using reproducible experiments with clear metrics and test data.

48

Legal and Ethical Considerations This section looks to outline the legal and ethical considerations within the scope of the Scout project. This is significant within the project due to the delicate nature of the context surrounding it. This section focuses on the legality of scanning, the storing of the information gathered and the ethics behind releasing a tool which uses these procedures to the public. The legality of port scanning lies within the ever-advancing field of information and communication technology. Due to this perpetually forward motion, there is a disconnect between current considerations on laws and ethics (Bailey, Dittrich, Kenneally, & Maughan, 2012). Due to law reactively responding to this growing area of interest, the latency caused creates confusion to where the responsibility of researcher lies (Forbis, 2015). In relation to port scanning this is associated to the availability of data that is publicly available, however maybe there unknowingly. To address this research in this area has explicitly detailed their ethically considerations. In the work of Durumeric et al, they have outlined “Scanning Best Practices”, these contain considerations for resources out with the administration of the research, providing simple means of opting out, as well as the consideration of potential legal differences across jurisdictions (Durumeric, Wustrow, & Halderm, 2013). The latter of which was not considered fully considered during the undertaking on the Internet Census 2012 (Anonymous, 2012). As previously mentioned the findings of this research were collated using the Carna botnet. This involves uploading a binary to insecure devices, from which these devices probe the internet to collect information on the devices connected. This is illegal to perform in many countries, in particular the act of unauthorized entry and uploading of binary is illegal in the UK, under the Computer Misuse Act 1990 (UK Goverment, 1990). Although Scout does perform port scanning, it lives in the same disconnect as Scout has the potential to identify vulnerabilities in devices, which is a grey area. Presenting these possible catastrophic vulnerabilities in an understandable format. Although no interaction takes place between the Scout user and the target systems, the information gained through the operation of Scout maybe used with malicious intent. The possibility of this is perhaps increased through the lack of complexity and knowledge required to operate Scout. Lending itself towards the aforementioned Intrusion as a Service label as seen with AutoSploit (Bejtlich, 2018). To reduce the possibility of Scout being used with malicious intent, as well as to avoid any potential legal trouble which may follow suit, Scout will not be released to the public. Scout will be made available to researchers upon request to help further academic research in this area. Although it should be stressed that Scout is for research purposes only and should not be used to exploit known vulnerabilities. This practice can be found in other similar projects (Durumeric, Wustrow, & Halderm, 2013). The ethical considerations about storing and indexing the results of port scanning is only mentioned by Durumeric et al when including an opt-out option, the legality of such a practice is not discussed (Durumeric, Wustrow, & Halderm, 2013). To provide clarity on the legality of this practice GDPR will be looked at. The General Data Protection Regulation (GDPR) is European Union regulation with the intention to unite and strengthen the data protection across the EU (The European Parliament, 2018). The date of implementation is the 25th of May 2018, a few short weeks after this project finishes. GDPR is relevant to this piece of research as it highlights that some online data may be personal information therefore protected under the regulation. An example of this online data is given in recital 30 where the

49

definition of online identifiers includes IP addresses (The European Parliament, 2018) (Information Commissioner's Office, 2018). This could affect the way in which internetscanning projects this research relies upon operate, however in October 2016, the Court of Justice of the European Union provided clarity on the status of IP addresses as personal information in the case of Patrick Breyer v Germany. With this ruling being perhaps an example of EU’s thoughts on PII which will be developed in GDPR. In Patrick Breyer vs Germany, the court ruled, “a dynamic IP address registered by an online media services provider when a person accesses a website that the provider makes accessible to the public constitutes personal data within the meaning of that provision, in relation to that provider, where the latter has the legal means which enable it to identify the data subject with additional data which the internet service provider has about that person.” (Patrick Breyer v Bundesrepublik Deutschland, 2016). Deconstructing this ruling, one can observe that the distinction between Static and Dynamic IP addresses is needed. This is due to Directive 95/46/EC, detailing static IP addresses are personal identifying information. Later on, the crux of the ruling is found. If the party has additional data to identify the individual from the IP address then it constitutes as personal identifying information (Messori, 2016) (Peers, 2017). This sentiment is also seen in Data Protection Act in the UK (UK Goverment, 1998) (Pinsent Masons, 2008). The ruling seems to mostly apply to ISPs as they have both PII of the user as well as their IP address, from which this information can be used for marketing reasons. An internet-scanning project is not an online media service provide. However, the ruling includes the relevancy due to its inclusion of the argument that dynamic IP addresses are not personal identifying information. Internet-scanning projects will correspond with both static and dynamic IP addresses, however it has no way to determine which type of IP address each is without consultation of a third-party service. Internet scanning projects, despite their commercialization, these tools are for reference and analysis. Some projects such as Shodan, add context to IP addresses by associating with the organization that owns the IP (Matherly, 2016). This information can be found by referring to IANA and the regional internet registries (Internet Assigned Numbers Authority, 2018). However, this does not include information specified as personal identifying information (Boardman, Mullock, & Mole, 2017). This section set out to outline the legal and ethical considerations which encompass the scope of the project. One of the most significant points of discussion is the legality of port scanning and its lack of regulation. Due to this several considerations on the release of Scout were made. Restricting its release will mitigate any controversy surrounding the project. Another point of discussion was the legality of storing IP addresses. In investigation of current rulings and future laws has identified that the storing of IP addresses is legal unless the organisation storing the information has the ability to tie the IP address to PII. As Internet-scanning projects do not have the required contextual information, this is not a legal consideration. Although as mentioned, once laws respond to the changing climate, this may need to be readdressed. A look into laws, from around the world, concerning the practices mentioned could be a fruitful area for further work.

50

Conclusion This methodology included a deliberation on several key factors which contributed to the approach implemented and was then compared to a small number of similar projects. Also included was a discussion on the legality and ethics of the project with relevant analysis of surrounding legal and ethical frameworks. In the design phase, the key findings found in the literature view were assessed, with the most appropriate findings being implemented. The choice of Censys over Shodan was an example of this, Censys was deemed as the more appropriate due to their approach resulting in data with greater value. Using this data, Scout’s output can be held with the same regard. Upon implementation, findings discussed within the literature review were considered. A specific example of this is the utilization of the Levenshtein algorithm to mitigate the errors found within the NVD. This approach was first used in work by Sanguino et al, however their approach required alteration to fit Scout’s limited resources (Sanguino & Uetz, 2017). When implemented, usage of the Levenshtein algorithm contributed to greater accuracy. When consulting surrounding literature, it seemed the approach detailed through design and implementation had never been defined. To address this, a proposal was put forward coining the term “contactless ‘active’ reconnaissance”. Differing from contactless reconnaissance by using data specifically collated through active reconnaissance (Simon, Moucha, & Keller, 2017). The lack of definition before this proposal shows just how novel the approach undertaken is, this position was reinforced when consulting relevant legal frameworks. It appeared that Scout’s approach and subsequent output were not covered under any existing or forthcoming legalisation, existing within a grey area, thus relying upon ethical consideration. After examining the approaches of other researches, the ethical principles found within the Menlo report were followed (Bailey, Dittrich, Kenneally, & Maughan, 2012). This report outlines guidelines for ethical approaches to ICT research, the most applicable guideline to Scout is that of beneficence. Nevertheless, as Scout can be used maliciously to collect information on possible vulnerable services for further illicit activity, the potential benefit to society to release this tool to the public is outweighed by the risk of malicious usage of Scout.

51

Initial Operation Once implemented, a large quantity of data was used to test Scout and look to document a potential use. With the output provided, graphical charts can be produced which then can be utilized in an analysis and comparison of a sole or group of organizations. To demonstrate this, several education institutions will be used as test data and the findings will be displayed. Education institutions were chosen as their online addresses are publicly available, they are used in related literature as use case assessment and also to provide insight into the security eco-system of Academia (Genge & Enăchescu, 2016). This is particular relevant as Cyberattacks have been targeting Higher Education (Rogers, 2018)(Havergal, 2016) (The Week, 2017). The academic sector has reported a doubling in breaches over the past two years (Yeung & Bennett, 2017). With notable attacks such as a data breach at University of Greenwich, ransomware on University College London and a DDOS bringing down the Janet network (Kelion, 2016)(Coughlan, 2017) (Havergal, 2015). By focusing this example of output on Academia, potential insight into security ecosystem could be gained. Scout was operated on the data provided by Censys about 7 local academic institutions. In total 12967 services were processed. The number of services associated to each institutions range in size, therefore to perform valid comparison individual statistics will be compared. In previous literature only average and outliers for organizations were calculated (Genge & Enăchescu, 2016), on top of this calculation, proportion will be provided to allow for greater insight and analysis. In analysis of ShoVAT, Gene & Enăchescu reported the majority of Universities having a CVSS score average revolve around 5 and the results are spread across the 0-10 spectrum. These findings can be observed in Figure 42. In comparison Scout also features an average revolving around a CVSS score of 5, with a common spread of across the 0-10 spectrum. These results can be observed in Figure 43. In both cases there is an outlying result, that of University 5 in Figure 42 and University B in Figure 43. Of these results University B is more akin to the common results than University 5 as it has a larger spread. University 5 has a much smaller range of results, however a comparable average to the rest.

Figure 42: Associated Average CVSS and Outliers

52

Average CVSSv2 Score and Outliers

CVE's Average CVSSv2 and Outliers from Academia 10 9 8 7 6 5 4 3 2 1 0 University A University B University C University D University E University F University G

Academic Institution Figure 43: Associated Average CVSS and Outliers for Academia Test Data

Although the results in Figure 42 and Figure 43 provide grounds of comparison, they somewhat obfuscate the picture for each institution. To provide clarity, the vulnerabilities were grouped in to the respective CVSSv2 severity, the bands of severity were discussed earlier, and then the percentage of each severity was calculated for each university accordingly. The results of this can be observed in Figure 44. From this figure a much clearer image can be seen. Findings include that medium severity vulnerabilities are the most common, followed by high severity. The number of low severities is minimal, with high severities in some cases such as University A and F ranging into the 25-35% range. From this figure, it can be said that University A is the most susceptible to potential high severity known vulnerabilities.

Percentage

Percentage of Vulnerabilites by CVSSv2 Severity found in Academic Institiutions 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0%

High Medium Low

Academic Institution Figure 44: Percentage of Vulnerabilities by Severity for Academic Institutions

53

From the output Scout provides, such as that seen in Figure 41, graphs, such as Figure 43 and Figure 44, can be used to create greater understanding of the raw data. When included in reports and audits, this can be especially useful for to portray the significance of the findings to individuals of a less technical knowledge. This is just one of the potential uses of Scout. A natural progression of this work is to explore further ways Scout could be used.

54

Experiments Results, Analysis and Evaluation Through this section of the dissertation, a clear experimental design will be put forward. The results of the experiments conducted will be analysed with comparisons made to prior research and industry tools. From which these will be evaluated to quantify the usefulness of Scout as was the aim of this section.

Experiment Design The design of the forthcoming experiments was based upon those encountered within current literature. The themes found literature revolve around comparison to industry tools which perform similar functions (Genge & Enăchescu, 2016) (Simon, Moucha, & Keller, 2017). In an effort to diversify the experiments and build upon current research, the authors of some of the aforementioned research were contacted, and generally helpful however, could not supply their tools or datasets. This was particularly unfortunate as the method used in their work was not of reproducible standards. In the effort to design a sound experiment with reproducible qualities, both the requirements of Scout and easy duplication are needed. Scout requires Censys to be able to scan the service under investigation, this requires the service to be publically accessible and not contained within Censys’ blacklists. To use the vulnerability assessment tools used in current literature, permission must be granted. If not the user runs the risk of breaching the Computer Misuse Act 1990, as these tools actively probe for vulnerabilities, potentially causing harm to the target system (UK Goverment, 1990).

Figure 45: Experiment design

Figure 45 depicts the topology implemented in the following experiments to evaluate Scout. Scout is featured at the end of a two-way correspondence with Censys, representing the API calls needed to retrieve the scanning results. Censys is seen to have a two-way communication with the target, symbolizing the scanning process. In addition, this communication is tied to a clock illustrating the latency which will be present in the scan results given to the Scout user. In comparison, OpenVAS has a sole two-way communication

55

directly to the target. The decision to use OpenVAS was because it is free and found within the highly popular Kali Linux distribution. Nessus is OpenVAS’ main competitor with similar functionality as seen in the literature review, however access seems to cost considerable investment. By selecting OpenVAS over Nessus, the requirements needed for reproducing these experiments are less drastic. The target used over the course of the experiments was an Amazon Web Services EC2 instance - a free tier Ubuntu 14.4. After applying, permission was granted to perform the necessary testing, proof of which can be seen in Appendix I. AWS was chosen as it was free to students and was able to assign the instance with a public IPv4 address, thus allowing for Censys to scan it. Due to the decision to use AWS, the experiments were restricted to only services located on port 80. This was due to the Censys tag problem outlined in Implementation and in part to the lack of technical knowledge required to establish other services.

Experiment 1 - Common Platform Enumeration Creation One major procedure within Scout is the Common Platform Enumeration manufacturing and matching process, as seen in the implementation, this task is difficult and complex. The procedure is vital to Scout’s operation as it bridges the output given by Censys and associates it with an entry within the Common Platform Enumeration dictionary. This can then be used to search the National Vulnerability Database to find known-vulnerabilities. The importance of this process cannot be underestimated as it acts as a bridge between the two external data sources, this procedure is a main factor in determining the adequacy of the approach and thus the tool. Fitting of this importance, it is paramount the evaluation method is scientifically relevant and reproducible for further research and critique. To ensure this, methods found previously within associated literature have been adopted. These also allow for comparisons to be drawn. This experiment is split into two, a manual assessment in which the algorithm is examined for possible mistakes, the second assessment is a comparison with tools which perform the same process. Manual Assessment In previous research, the CPE creation accuracy is compared against manually performing the procedure and Precision and Recall will be used as according to literature it is criteria for quality evaluation (Genge & Enăchescu, 2016) (Mohammed, 2016) (Tung, Tseng, Shih, & Shan, 2013). The reasoning for such an experiment is to mathematically quantify the success and failure rate as well as other statistics for the approach employed. Upon comparison with other approaches, the most suitable approach can be determined. For this experiment, a Scottish university’s IP range was chosen as the test data. The query for the database returned 1420 results, from this 52 unique services were identified, this set of services was the data for the experiment. As seen in design and implementation stage currently Scout is only able to operate on the data retrieved by Censys via the field ‘80.http.get.metadata.description’. So therefore, the test data is only information available via this field. This data is passed through Scout and from there the results are then analysed for possible errors such as false negatives. False negatives in this assessment are defined as entries in which a common platform enumeration exists, but cannot be determined by the approach used. The data for this assessment, which can be used for reproduction, can be found in Appendix E. The results are displayed within Figure 46, Figure 47 and visualised within Figure 48.

56

Table 8: Results from Manual Assessment Number of Unique Services Identified

Number of Positive Results

Number of False Results

Number of False Positives

Number of False Negatives

False Result Percentage

52

39

13

1

12

25%

Figure 46: Results from Manual Assessment

Figure 47: Manual Assessment Confusion Matrix

Figure 48: A Sankey Diagram of Manual Assessment Confusion Matrix

The true positive results returns the most appropriate (correct) Common Platform Enumeration. Although this occurred a considerable number of times it was pipped to the most common response by true negative results were. The true negative results highlight the short comings of the National Vulnerability Database. As all of these results occurred due to the versions of products not having corresponding CPEs. The reason for which varied between recently released to versions which had overlapping life time with new products. From Figure 46, the main observation is the sizeable false result percentage, of which 92% are false negatives. 8 of these false negatives come from versions of Apache, 2 come from versions IIS and Nginx and the last from an application called Webrick. The IIS versions which result in false negatives can be explained through a change in the syntax of CPE for this service. “cpe:2.3:a:microsoft:iis:7.5” upon version increment changed to “cpe:2.3:a:microsoft:internet_information_services:8.0”. Scout did not identify this as corresponding to the service. This has identified a flaw within the approach employed. The false negatives associated to varying versions of Apache, differ in reasons of failure. Given “Apache Coyote 1.1”, no CPE was found. This is due to the corresponding CPE being “cpe:2.3:a:apache:coyote_http_connector”, this was found via the algorithm, as seen in

57

Figure 49. However, it was competing with other CPE candidates. The Nginx false negative also follows this pattern. This identifies a flaw with the decisiveness of the algorithm used, in particular that of the Levenshtein algorithm’s usage.

Figure 49: The Output of the CPE Algorithm with the Input "Apache Coyote 1.1"

One particular and novel reason for a false negative was seen given the input “Apache httpd2.2.11”. This is not a spelling mistake, “2.2.11” is not separated with “httpd” by a space. As seen in the Appendix G, this is the only occurrence of such an incident, which resulted in the algorithm not identifying the version, consequently affecting the quality of CPE candidates, as seen in Figure 50. This flaw demonstrates Scout’s reliance on a specific format, identifying room for future improvement.

Figure 50: Algorithm output for “Apache httpd2.2.11”

The last reason for false negatives was attributed to Common Platform Enumeration dictionary used, as although existing, some CPEs did not appear in the dictionary. Notably those associated to “Apache httpd 2.2.31”, “Apache httpd 2.4.25” and “WEBrick 1.3.1”. Contributing to list of false negatives. This flaw is out with the scope of the algorithm, for once, however is not out of scope of the approach. The lone false positive can be explained as it occurred when Censys provided “Apache httpd 2.4”, which when passed through Scout resulted in “cpe:2.3:a:apache:mod_python:2.4”. Although this is the incorrect CPE, it funnily enough returned a single vulnerability. The problem stems from apache version have a 3-part syntax, starting at 2.4.0. But as only 2-parts are provided, an incorrect match was made. The cause of this seems to be that Apache version 2.4.0 is advertised as 2.4 through the metadata field. To scientifically quantify the results found in Figure 47, methods utilized in related literature will be transferred over in an attempt to increase the reproducibility and comparably of the findings. The methods used are Precision, Recall and F-measure, these assessment methods gauge differing aspects of the tool (Mohammed, 2016) (Tung, Tseng, Shih, & Shan, 2013) (Simon, Moucha, & Keller, 2017) (Fonseca, Vieira, & Madeira, 2007) (Suto, 2010) (Antunes & Vieira, 2010) (Nair & Devi, 2015). Their use in current literature focuses on web vulnerability scanning, specifically looking at measuring the identification of SQL injection and cross site scripting attacks.

58

Precision quantifies accuracy and is the ratio of correctly matched CPEs to number of all matched CPEs. Equation 1: Scout's precision in CPE manufacturing

𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 =

𝑇𝑃 17 17 = = = 0.94 𝑇𝑃 + 𝐹𝑃 17 + 1 18

Recall is the measure of the comprehensiveness and calculated by dividing the ratio of correctly matched CPEs to the number of potentially correctly matched CPEs. Equation 2: Scout's recall in CPE manufacturing

𝑅𝑒𝑐𝑎𝑙𝑙 =

𝑇𝑃 17 17 = = = 0.56 𝑇𝑃 + 𝐹𝑁 17 + 13 30

𝐹1 Score is the measure of the effectiveness, using both Precision and Recall, to calculate their harmonic mean. Equation 3: Scout's F1 score in CPE manufacturing

𝐹1 =

2 × 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 × 𝑅𝑒𝑐𝑎𝑙𝑙 2 × 0.94 × 0.56 1.05 = = = 0.68 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 + 𝑅𝑒𝑐𝑎𝑙𝑙 0.95 + 0.56 1.51

It seems the only project in which this method could be compared to using these metrics is ShoVAT. ShoVAT however does not detail their full true results, which are needed for the above calculations. They also do not detail their banners in which they analysed, thus making their experiment is less reproducible and therefore, less acceptable for comparison. However, a comparison could be made and the results may provide some insight. Table 9 provides the results for both ShoVAT and Scout. Table 9: ShoVAT and Scout's Manual Assessment Results

Tool

Banners analysed

False results

False Positives

ShoVAT Scout

296 52

46 13

23 1

False False Positives % Negatives 7.77% 1.92%

23 12

False Negatives % 7.77% 23.08%

Table 9 shows a disparity between number of banners analysed. This is due the difference of in functionality of tools, as ShoVAT can operate on more diverse services, however some generalizations in approach can be made. Scout seems to be less accurate than ShoVAT, this can be said as Scout is more likely to return a false result. Of these false results, Scout is more likely to return a false negative where as ShoVAT’s false results are on equally spread among positives and negatives. As such Scout appears to be more precise as it is less likely to return a false positive. This is important as this may be the determining factor which would deter a user from using the application. The trade-off for Scout is results are more likely to be false, from which they are less likely to false flags and more likely to be erroneous. The trade-off for ShoVAT is that false results happen less often but are just as likely to be false flags as errors.

59

From the sweeping generalizations made, it is clear that Scout and ShoVAT differ. ShoVAT might appear to the better of the two, however both have considerable false result rate. Improving this false result rate may be an area of consideration for future work. Despite this rate, both still appear to be functional with a degree of accuracy. In order to determine if this is potentially useful tool a greater comparison to current tools is needed. Comparative Assessment In previous research, the CPE creation accuracy is compared against industry tools which perform the same procedure (Genge & Enăchescu, 2016) (Simon, Moucha, & Keller, 2017). Ideally this would be the method in which Scout’s CPE creation would be evaluated, however, gaining permission to use these tools on a large sample size would be prove to be unfeasible. So therefore, the experiment was conducted on a smaller sample size in which permission was gained for. This test data was cumulated from instances provided by AWS. The purpose of this comparative assessment was to analyse how well Scout can manufacture Common Platform Enumerations in comparison to OpenVAS, which performs the same function. Table 10 presents an overview of the experiment results, what is interesting about the data in this table is that Scout outperforms OpenVAS. With Scout able to identify 1 more CPE than OpenVAS. This is a rather unexpected result as the limitations of the approach implemented in Scout were discussed earlier. Both tools only correctly matched the Common Platform Enumeration when enough correct information was supplied. For OpenVAS a caveat to this rule was the need for this information to be given in the correct order. Omitted from the results is the false positive which Scout returned, this was due to the advertised service being ‘lighttpd 1.4.19’, Scout returns ‘cpe:2.3:a:lighttpd:lighttpd:1.4.19’ although a valid CPE is not the correct one as the service running is actually ‘lighttpd 1.4.18’. For this example OpenVAS does not return a Common Platform Enumeration. This result suggests that OpenVAS must perform a form of verification of the advertised service through correspondence. As Scout cannot perform this same step of verification, hence the false positive return on a falsely advertised service. These results suggest that Scout is able to perform CPE assignments in a more diverse dataset OpenVAS, although does return a false positive in certain situations. Table 10: Comparative Assessment Results

Advertised Service ‘Apache/2.4.7’ ‘Apache/2.4.10’ ‘Apache/2’ ‘Apache httpd’ ‘lighttpd 1.4.18’ ‘1.4.18 lighttpd’ ‘lighttpd 1.4.19’ ‘lighttpd 1.4.33’ ‘lighttpd’ ‘nginx 1.4.6’ ‘nginx’

60

Correct CPE ‘apache:http_server:2.4.7’ ‘apache:http_server:2.4.27’ ‘apache:http_server:2.4.27’ ‘apache:http_server:2.4.27’ ‘lighttpd:lighttpd:1.4.18’ ‘lighttpd:lighttpd:1.4.18’ ‘lighttpd:lighttpd:1.4.18’ ‘lighttpd:lighttpd:1.4.33’ ‘lighttpd:lighttpd:1.4.33’ ‘nginix:nginx:1.4.6’ ‘nginix:nginx:1.4.6’

OpenVAS ✓ ✓ ✕ ✕ ✓ ✕ ✕ ✓ ✕ ✓ ✕

Scout ✓ ✓ ✕ ✕ ✓ ✓ ✕ ✓ ✕ ✓ ✕

Experiment 2 - Common Vulnerability Enumeration Assignment Similar to the experiment prior, in the current literature this CVE assignment is compared against industry tools which perform a similar function (Genge & Enăchescu, 2016) (Simon, Moucha, & Keller, 2017). The referenced experiments seem to not provide adequate test data to reproduce them, nor did they have metrics in which comparisons could be made from. In an effort to generate a sounder method of evaluation, surrounding literature was consulted. In literature related to web vulnerability scanning, the metrics Precision, Recall and F1 measure were used to quantify accuracy, comprehensiveness and effectiveness of a web scanner (Mohammed, 2016) (Tung, Tseng, Shih, & Shan, 2013) (Fonseca, Vieira, & Madeira, 2007) (Suto, 2010) (Antunes & Vieira, 2010) (Nair & Devi, 2015). Theses metrics seemed appropriate to transfer due to their test data having a ground truth with binary state of being susceptible to a vulnerability or not. To distinguish between potential vulnerabilities and those that could be exploited by an attacker, the National Vulnerability Database was included in the comparison to provide reference to potential vulnerabilities while OpenVAS was used to provide a comparison to vulnerabilities the service is susceptible to. A service may have more potential vulnerabilities than is susceptible to due to necessary features being enabled or required configurations. There is also the possibility of the vulnerability have being patched through the process of backporting. Table 11 presents an overview of outputs from the National Vulnerability Database, OpenVAS and Scout. The OpenVAS can be found in Appendix J. The National Vulnerability Database information used in the process, was taken from their website on the 19th of March 2018. Table 11: CVE Assignment Results

Service Apache httpd 2.4.7 Apache httpd 2.4.10 Lighttpd 1.4.18 Lighttpd 1.4.33

NVD 18 16 14 5

OpenVAS 18 14 6 4

Scout 15 13 12 3

What is interesting about the data in Table 11 is that the NVD outnumbers both OpenVAS and Scout. This is expected in OpenVAS due to the aforementioned possibilities a service may not be vulnerable to known vulnerability. This is unexpected in Scout as the methodology implemented Scout looks to return all possible known vulnerabilities. Interestingly, in most cases OpenVAS reports more vulnerabilities than Scout. This is also unexpected as it was hypothesized that Scout would return more due to the aforementioned possibilities. To explore the reasoning and further assess the usefulness of Scout a dissection of the results reported in Table 11 is needed. What follows is a detailed examination of the individual results, comparing the vulnerability assessment tools to the National Vulnerability Database and between the two tools. The results obtained from the analysis of an apache 2.4.7 service are set out in Table 12. The differences between the NVD and vulnerability assessment are highlighted in this table. Of

61

these differences Scout reported excluding 3 vulnerabilities found in the NVD whereas OpenVAS reported excluding 4 vulnerabilities, while including 4 vulnerabilities not found in the NVD. To scientifically quantify this, precision can be used to assign a percentage of the correct relevant vulnerabilities to the total number of vulnerabilities returned. Treating the NVD as the ground truth in this equation would disregard the service’s susceptibility to exploitation, therefore using OpenVAS results as the ground truth would be more appropriate to evaluate the usefulness of the output provided by Scout. To calculate precision, true positives and false positives are used. True positives are results which appear in the output of both OpenVAS and Scout whereas false positives are results confined to only Scout’s results. The workings and result for this equation can be observed in Equation 4. Table 12: Vulnerabilities reported for an Apache 2.4.7 service

NVD

OpenVAS

Scout

CVE-2004-0230 CVE-2013-6438

CVE-2013-6438

CVE-2014-0098

CVE-2014-0098

CVE-2014-0117

CVE-2014-0117

CVE-2014-0117

CVE-2014-0118

CVE-2014-0118

CVE-2014-0118

CVE-2014-0226

CVE-2014-0226

CVE-2014-0226

CVE-2014-0231

CVE-2014-0231

CVE-2014-0231

CVE-2014-3523

CVE-2014-3523

CVE-2014-3523

CVE-2014-8109

CVE-2014-8109

CVE-2014-8109

CVE-2015-0228

CVE-2015-0228

CVE-2015-3183

CVE-2015-3183

CVE-2015-3184 CVE-2015-3185

CVE-2015-3184 CVE-2015-3185

CVE-2016-0736

CVE-2015-3185 CVE-2016-0736

CVE-2016-2161

CVE-2016-2161

CVE-2016-5387

CVE-2016-5387

CVE-2016-8743

CVE-2016-8743

CVE-2016-2161 CVE-2016-8743

CVE-2017-3167 CVE-2017-3169 CVE-2017-7679 CVE-2017-9788

CVE-2017-9788

CVE-2017-9788

CVE-2017-9798

CVE-2017-9798

CVE-2017-9798

Differences

+4/-4 Differences

+0/-3 +4/-7

Equation 4: Scout's precision to OpenVAS results

𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 =

62

𝑇𝑃 11 11 = = = 0.73 𝑇𝑃 + 𝐹𝑃 11 + 4 15

Scout has a precision of 73%. This is a significant result as this proves the possibility of identifying exploitable vulnerabilities, usually found through active reconnaissance, can be found through a contactless ‘active’ approach, although these results may be inaccurate. To further evaluate the usefulness of Scout, recall can be used to quantify the proportion of true vulnerabilities returned to all true vulnerabilities. To calculate recall, true positives and false negatives are used. True positives are the same as outlined in the precision calculation while false negatives are results that are present in OpenVAS’ output but not in Scout’s output. The workings and result for this equation can be observed in Equation 5. Equation 5: Scout's recall to OpenVAS results

𝑅𝑒𝑐𝑎𝑙𝑙 =

𝑇𝑃 11 11 = = = 0.61 𝑇𝑃 + 𝐹𝑁 11 + 7 18

Scout has a recall of 61%. This is an important result as this quantifies the comprehensiveness of the results Scout. With a recall of 61% Scout is able to identify more than half, close to two thirds, of the applicable vulnerabilities for this apache service. Again to further evaluate the usefulness of Scout, F-Measure can be used to give an indication of the effectiveness of a vulnerability assessment tool by combining the accuracy and comprehensiveness. To calculate the F-Measure precision and recall is used. The workings and result for this equation can be observed in Equation 6. Equation 6: Scout's F-Measure to OpenVAS results

𝐹𝑚𝑒𝑎𝑠𝑢𝑟𝑒 =

2 × 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 × 𝑅𝑒𝑐𝑎𝑙𝑙 2 × 0.73 × 0.61 0.89 = = = 0.66 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 + 𝑅𝑒𝑐𝑎𝑙𝑙 0.73 + 0.61 1.34

Scout has an F-Measure of 66%, the effectiveness of Scout can be quantified as nearly two thirds of OpenVAS. This metric only applies to the Apache 2.4.7 service. Table 13: Vulnerabilities reported for an Apache 2.4.10 service

NVD CVE-2014-3583 CVE-2014-8109 CVE-2015-0228 CVE-2015-3183 CVE-2015-3184 CVE-2015-3185 CVE-2016-0736 CVE-2016-2161 CVE-2016-5387 CVE-2016-8743 CVE-2017-3167 CVE-2017-3169 CVE-2017-7668 CVE-2017-7679 CVE-2017-9788

63

OpenVAS CVE-2004-0230 CVE-2014-3583 CVE-2014-8109 CVE-2015-0228 CVE-2015-3183 CVE-2015-3185 CVE-2016-2161 CVE-2016-5387 CVE-2016-8743 CVE-2017-3167 CVE-2017-3169 CVE-2017-7679 CVE-2017-9788

Scout CVE-2014-3583 CVE-2014-8109

CVE-2015-3184 CVE-2015-3185 CVE-2016-0736 CVE-2016-2161 CVE-2016-8743 CVE-2017-3167 CVE-2017-3169 CVE-2017-7668 CVE-2017-7679 CVE-2017-9788

CVE-2017-9798

CVE-2017-9798

CVE-2017-9798

Differences

+1/-3 Differences

+0/-3 +3/-4

Table 14: Vulnerabilities reported for a Lighttpd 1.4.18 service

NVD

OpenVAS

Scout

CVE-2004-0320 CVE-2008-0983 CVE-2008-1111 CVE-2008-1270 CVE-2008-1531 CVE-2008-4298 CVE-2008-4360 CVE-2010-0295 CVE-2011-4362 CVE-2013-1427 CVE-2013-4559 CVE-2013-4560 CVE-2014-2323 CVE-2014-2324 CVE-2015-3200

Differences

CVE-2008-0983 CVE-2008-1111 CVE-2008-1270

CVE-2008-4360 CVE-2010-0295

CVE-2014-2323 CVE-2014-2324 CVE-2015-3200

+1/-9 Differences

CVE-2008-4298 CVE-2008-4360 CVE-2010-0295 CVE-2011-4362 CVE-2013-1427 CVE-2013-4559 CVE-2013-4560 CVE-2014-2323 CVE-2014-2324

+0/-2 +8/-2

Table 15: Vulnerabilities reported for a Lighttpd 1.4.33 service

NVD

OpenVAS

Scout

CVE-2004-0320 CVE-2011-4362 CVE-2013-4508 CVE-2014-2323 CVE-2014-2324 CVE-2015-3200

Differences

CVE-2014-2323 CVE-2014-2324 CVE-2015-3200

+1/-2 Differences

CVE-2013-4508 CVE-2014-2323 CVE-2014-2324

+0/-2 +1/-2

To avoid unnecessary length, the other services contained within Table 11 will not as analysed to the same level of detail to the extent done for the results of Apache 2.4.7. The same procedure will be performed for the result found in in Table 13, Table 14, Table 15, with the relevant calculations found in the appendix H. A summary of all the results

can be found in Figure 51.

64

Figure 51: Precision, Recall, F-Measure for Scout in Comparison to OpenVAS

The results of the equations associated to all the services analysed are presented in Figure 51. From this data, we can see that there is a variance in results. In overall effectiveness, measured F1 score, the effectiveness ranged from a high of 74% for Apache 2.4.10 to a low of 44% for Lighttpd 1.4.18. Closer inspection of the data shows that reason for the moderate range of F1 score is due the significant range in precision. Precision ranges from 33% to 77%, this 40% range is near considerably more that of the range for Recall, which is 29%. Overall, these results indicate that Scout varies moderately in effectiveness as vulnerability assessment tool, this variance is due to the deviation in accuracy more so than the comprehensiveness.

Evaluation In both experiments, Scout was compared against OpenVAS, with ShoVAT only featuring to compare against manual assessment results. This is due to ShoVAT seeming to have very little reproducibility in their experiments, meaning new experiments had to be conceived with reproducibility and validity in mind. This contributed to proving evaluation being a difficult task. To the goal set out for each experiment was to rigorously evaluate each section of the methodology implemented. To evaluate CPE creation the manual assessment and comparative assessment experiments were used. To evaluate CVE assignment, experiment 2 was conceived. These experiments were not without complication. This section will look to detail these obstacles, how they formed, how they were handled, as well as how they could be addressed during future research. This is section will also give an overview of Scout in comparison to similar tools. During the initial operation, when using Scout across academia, there was an interesting result involving the CPE creation module. When ‘Apache httpd 2.2.3’ services were identified, the approach did not produce a conclusive answer due to the presence of multiple accepted candidates. This can be observed in Figure 52. The correct CPE in this case is one which has ‘http_server’ however, due to the competition from the other accepted candidates, it was not selected. This provides evidence that further work is needed on the approach used in CPE creation.

65

Figure 52: Inconclusive CPE creation

When evaluating Scout’s CPE assignment accuracy through Experiment 1, the task was broken down into two sections: A manual section in which comparison was made to human assignment and a comparative section in which comparison was made to industry vulnerability assessment tools which perform the same task. As it was already shown, this methodology was backed by previous research and was applicable to this dissertation as the usefulness of the project has to be quantified. For manual assessment within experiment 1, the test data was produced by identifying unique services from a densely populated academic subnet. The reasoning behind this decision as it would return a less diverse dataset than a geographic filter, but more diverse than a service specific filter, with the diversity reflecting upon the organisation’s size, culture, goals and sector. This is significant as it removes the abnormalities such as home modems from a geographical filter, but refrains from conforming to uniformity as seen in service filtering. These organisations are also prone to not changing drastically overtime unlike geographic areas. By using an organisation as a test data, the findings better reflect that of potential tool use within industry. In retrospect, the dataset used in the manual assessment was not particularly comprehensive, potentially skewing the results. In comparison to the dataset used in ShoVAT, Scout’s dataset is 6 times smaller. An argument could be voiced that ShoVAT is able to perform analysis on a far more diverse test dataset than Scout, as Scout is isolated on services assigned at port 80, while ShoVAT is able to perform analysis on wider range. Potentially, this could have been avoided if the authors provided their test data however, it seems that this dataset is not publically available. To avoid this issue, the test data used for Scout’s manual assessment are located within Appendix F. This should increase the reproducibility and, therefore, integrity of Scout’s results. In continuation with the issue of reproducibility, this assessment provided results which could be quantified into scientific metrics. These metrics were chosen as they appeared in previous research and were applicable to this work, as they align with the binary nature of vulnerability confirmation. By including these metrics, Scout’s results can be used in further research. On the contrary, while ShoVAT provided some of the results needed for calculating certain metrics, the data given was not suitable to produce comparable metrics. For the comparative assessment OpenVAS was used to perform a scan on preconfigured instances hosted by Amazon Web Services. AWS has a restricted free tier which was utilized as a dataset. The restriction placed upon the account were the number of hours per month the instance is online, this is set at 750 hours a month. In running one instance for a full month this uses 730 of the allotted 750 hours. This did not provide enough hours to perform a horizontal vulnerability assessment across multiple instances. Therefore, each service had to be configured and testing in a linear fashion. Configuring these different services in time with Censys’ scans was not possible within the timeframe of the project. So therefore, the services chosen were those which existed within Scout output, gained through Initial Operation, with a comparison after the fact. This does raise the validity of the test data. A complication with this process was gaining the permission to use these tools on the cloud instance. Thankfully

66

this was an easy process as Amazon have catered to this possibility and gaining permission was as simple as filling in an online form. The results of which can be found in Appendix I. This process should have been done further in advance to alleviate any problems with the integrity of test data by allowing Censys to scan the test experiments. Despite being deemed the most appropriate tool in the methodology, Censys proved a nuisance through the experiment procedure as test data was gathered using Censys and therefore search credits were used. The usage of these credits was a strain on the testing and evaluation of this tool. Censys transitioned to a commercial product during the course of this project, which introduced the usage of search credits: With each search (and execution of Scout) a credit is used. These credits are reset every calendar month to a total of 250. A similar practice is in place for Shodan, however they are far more generous, by providing 65535 tokens per month. The only way to gain more credits is to buy Censys Basic, which costs $1188 per year. Although research access of Censys was provided, extra credits were not received. This demonstrated one of the main downsides in the contactless ‘active’ approach, one such downfall is the reliance on a third-party entity such as Shodan, or in Scout’s case Censys’. Making sure these entities are providing the correct as well as concurrent information is out with the user’s control. The need for a contactless ‘active’ tool may succeed the current organisations’ approach requiring tweaking, or they may outlive the organisations’ operation entirely. If the user results to starting to do the requisite scans themselves then contact has been made with potential target systems, breaking the contactless approach. A related downfall is the nature of contactless ‘active’ relying on the information provided. This can be seen as Scout’s effectiveness can be hindered by the manipulation and or removal of information which is used to make identifiers, seen in the comparative assessment. Through this process, Scout’s ability to create correct or any identifier is impeded. In experiment 2, the same AWS instance from experiment 1 were used. Originally it was planned to test Scout against services which there was no control of configuration over, opting to use services in which permission could be gained. Permission was gained to operate on services in control of a gracious academic, however the operating of OpenVAS on these machines proved difficult. Scans would finish in 20 seconds, this points towards firewalls being operated within the network. Due to this only services configured on AWS instances were used, with the same after the fact comparison caveat. From the prior experiments, the subsequent analysis and comparison, Scout has shown to be a useful vulnerability assessment tool. This can be verified by identifying services greater than OpenVAS, as seen in comparative assessment. This was further analysed in the manual assessment against ShoVAT, where Scout is shown to be more accurate however, is more likely to return a false result than ShoVAT. In the limited experiments in comparison to OpenVAS, a significant finding to emerge from this comparison was the effectiveness of Scout to identify known vulnerabilities. In this aspect Scout performed worse than OpenVAS, however the effectiveness of Scout measured in F1-score ranged from 33-74%. Notwithstanding the limitation to services located on port 80, this comparison suggests that there is a moderate to significant usefulness to the results provided. This usefulness is enhanced by the novel contactless approach which by design mitigates any impact on the target network seen in a typical active vulnerability assessment tool (Skybox Security, 2016). Due to these attributes, Scout has potential to be a first line of reactive security protocol upon

67

the release of a new high-profile vulnerability. A natural progression of this project is to further develop Scout in an effort to improve the effectiveness. To show the nature of Scout in comparison to other tools, similar to that observed in Figure 18. A new comparison was made comprising of tools mentioned in this dissertation. This can be observed in Table 16. Table 16: Tool Comparison

Tool Active Passive Custom Banner CPE/CVE ZMap ●●● ● ● ● Shodan ● ●●● ● ●● Censys ● ●●● ● ●● p0f ● ●●● ● ● Nessus ●●● ● ●● ●●● OpenVAS ●●● ● ●● ●●● ShoVAT ● ●●● ●● ●●● Scout ● ●●● ●● ●●●

Conclusion This section contained experiments which were conducted, these experiments were similar to those found in research cited throughout the methodology however, were improved to create a reproducible process. The findings of these experiments were evaluated through comparison to previous research and industry tools. Through this comparison the value of reproducibility was carried on, by using applicable scientific measurements to quantify the results, which in turn allowed for evaluation against the aim of the project. Both the manual and comparative CPE experiments conducted have a basis in cited literature (Simon, Moucha, & Keller, 2017) (Genge & Enăchescu, 2016). When conducted in evaluation of ShoVAT, no raw test data was provided and the analysis of the experiments was limited. To improve this, included in this dissertation is the test data used and thorough analysis of each individual result. Through the inclusion of scientific metrics such as precision, recall and F1 score, a clear process of comparison can be made to Scout in further research. This reproducible nature carries through in CVE experiment through the use of the same scientific metrics and also through the use of an open source tool. The procedure used in the CVE experiment is based from a similar experiment found within the ShoVAT evaluation (Genge & Enăchescu, 2016). Like the CPE experiment, the procedure found in the literature lacked the quality of reproducibility. To increase reproducibility and thus help future comparison, this dissertation contains the exact test data and method used.

68

Future Work This project has had a considerable time constraint, due to this the scope has narrowed as time progressed. This contributed to omitting detail that could be explored in future work. Although Censys was the most appropriate tool to use as concluded during the literature review, other tools do exist, each with their differing approach. These tools include Shodan and ZoomEye as well as other lesser known projects. Comparing and contrasting the projects within the scope of contactless ‘active’ reconnaissance would be an area of investigation that could be explored at a later date. This body of work has highlighted problems with the National Vulnerability Database. These problems effected the approach employed through the methodology. Therefore, an investigation into employing alternatives to the National Vulnerability Database would be an interesting area of future research. When designing Scout, considerations for the shortcomings present within the National Vulnerability Database were made. A combination of lack of technical skill and time management to a problem out with the scope of the project led to the considerations not performing as well as intended. Advancing research in this area could prove useful to Scout. Upon reflection the Scout tool itself, does not provide the data it has produced in an interesting manner. This was never considered throughout the project due to it falling out with the scope of the project. The National Vulnerability Database, despite its downfalls, does provide fascinating resources related to each vulnerability. Not all of these fields are included in Scout’s output. Investigating the use of this information and how to display it in an easily readable or perhaps graphical format may prove useful. The evaluation identified Scout as having a considerable false result rate. As alluded to prior, this could be the area of further research. This research would encompass the most effective and suitable way to link service banners to Common Platform Enumerations. If more time was dedicated to this topic during the project, better results may have occurred. During the time in which this project took place two high profile tools within the surrounding topic gained attention, AutoSploit and Memcrashed. Both of these projects were discussed in detail during the Related Work section. What these projects, along with Scout, highlight is the room for further applications of Internet Scanning projects. With dedication of time and resources, more novel applications could be implemented.

69

Conclusion The aim of the project was to explore and evaluate how well Internet-wide scanning projects could be used to identify potential known-vulnerabilities. To accomplish this aim, the creation of a novel tool, Scout, was undertaken. Scout utilizes the Internet-wide scanning project Censys to source publically available services and the National Vulnerability Database in an attempt to assign these services with applicable known vulnerabilities. This approach employed by Scout has shown to be successful by correctly identifying vulnerabilities in Internet services. Through the process of creating this tool, the initial objectives were accomplished. Through completion of a comprehensive the literature review, key findings were identified. By investigating literature surrounding Internet-wide Scanning, the main tools Censys and Shodan were identified. Upon further analysis it was identified Censys’ approach to Internetwide scanning resulted in Censys having more valuable scanning data than Shodan as it was found to be more up to date, more accurate, and less biased (Durumeric, Adrian, Mirian, Bailey, & Halderman, 2015). These tools have vulnerability assessment capabilities, however is far less advanced than standard standalone vulnerability assessment tools, this justifies the aim of the project. When vulnerability assessment tools were investigated the tools could be split two approaches, passive or active (Bou-Harb, Debbabi, & Assi, 2014). Passive vulnerability scanners are not disruptive to the target network, however they have very little functionality in identifying vulnerabilities. While active vulnerability scanners are known to be potentially disruptive to the target network, however they are capable of identifying known vulnerabilities (Skybox Security, 2016). This provides justification of the project as well as an approach to compare against. The literature review also identified key considerations for the methodology. One of the more significant findings to emerge from this review is that the NVD is deeply flawed, with problems such as mutable, erroneous and asynchronous data (Karlsson, 2012) (Hung Nguyen & Massacci, 2013). When applied to service banner analysis there was only one approach which included consideration for these problems, this approach utilized Levenshtein distance algorithm to perform approximate matching (Sanguino & Uetz, 2017). The methodology incorporated a deliberation on several key factors identified through the literature review which contributed to the approach implemented. This approach was evaluated through an analysis of similar tools and approaches. Due to the nature of the tool an analysis on the legality and ethics of the project was also conducted. In the design phase, Censys was chosen as the internet-wide scanning project Scout would be built upon, as it was deemed the most appropriate due to their approach resulting in data with greater value. Upon implementation, consideration for the tools utilized in the approach was made, these were based around findings discussed within the literature review. A specific example of this is the utilization of the Levenshtein algorithm, resulting in greater accuracy, this required alteration from that found in the literature due to the limited resources of Scout (Sanguino & Uetz, 2017). When consulting surrounding literature, it seemed the novel approach implemented had not been properly defined before, therefore “contactless ‘active’ reconnaissance” was proposed. Differing from contactless reconnaissance by using data specifically collated through active reconnaissance (Simon, Moucha, & Keller, 2017). When

70

consulting legal frameworks this novelty was confirmed as it appeared that Scout’s approach was not covered under any existing or forthcoming legalisation. Thus existing within a grey area. To provide guidance the ethical principles found within the Menlo report were followed (Bailey, Dittrich, Kenneally, & Maughan, 2012). The Menlo report outlines ethical guidelines to ICT research where there is a disconnect between research methods and current legislation. Beneficence was identified as the most applicable Menlo report guideline to Scout, as the potential malicious use outweighs the potential benefit to society upon release this tool. In accordance to this guideline, Scout will not be released openly to the public but will be available to researches upon request. This methodology was tested through experiments which were used to analyse specific decisions made. These experiments had basis in research cited throughout the methodology, with alterations made to create a clear reproducible process (Simon, Moucha, & Keller, 2017) (Genge & Enăchescu, 2016). 3 distinct experiments were conducted. 2 CPE creations experiments, one focused on the accuracy and one experiment focused on obfuscated banners. The last experiment focused on the accuracy, comprehensiveness and effectiveness in relation to CVE assignment. The reproducibility was furthered through the use of applicable scientific measurements to quantify results. These metrics allow for evaluation against similar tools such as ShoVAT and OpenVAS. Upon comparison to ShoVAT’s CPE creation module, Scout is found to be more likely to return a false result however, of these false results Scout is less likely to return a false positive. This means Scout is more likely to be potentially incorrect but is far less likely to be inaccurate. When comparing Scout to OpenVAS at the ability to identify known-vulnerabilities, Scout obtained an effectiveness of 74% for a specific apache service, this is a remarkable result considering Scout does not correspond directly with the service. However, this effectiveness ranges across results with a 44% effectiveness observed against OpenVAS’ findings on a Lighttpd 1.4.18. The reason for this difference in effectiveness is mostly due to the accuracy from which it is derived from, although the comprehensiveness of the results also plays a part. These statistics can also be used to evaluate the project as a whole against the aim of the project, which is to evaluate how well Internet-wide scanning results could be used to identify potential known-vulnerabilities in Internet connected systems.

71

Acknowledgement I would like to express my appreciation to Alex Arbelet, Charlie Celice and Miguel Marques, as with their permission I was able to showcase some of my research through a presentation at a joint 2600 and DEFCON Edinburgh event. Through this presentation I was introduced Éireann Leverett, whom provided assistance which was greatly appreciated. Coincidentally I had referenced Éireann’s work prior to meeting him. Alex and Charlie also introduced me to Paul Ritchie, whom provided time and resources to answer questions related to their project ‘cve-offline’. I wish to also acknowledge Lachlan Kidson, for his assistance to remove redundancies within the outlined approach to the complex CPE algorithm. I wish to extend my special thanks to the staff of Censys for granting myself research access to their data, which although did not prove useful throughout this project, has provided motivation to carry out further research related to their product. Finally, I would like to express my utmost appreciation to Richard Macfarlane for his highly valuable, constructive suggestions and recommendations throughout this lengthy ordeal. His willingness to give his time so generously and selflessly has proved to be invaluable.

72

References @037. (2018, March 7). MEMCRASHED DDOS EXPLOIT TOOL. Retrieved from GitHub: https://github.com/649/Memcrashed-DDoS-Exploit/ Abe, S., Fujimoto, M., Horata, S., Uchida, Y., & Mitsunaga, T. (2016). Security Threats of Internet-Reachable ICS. 55th Annual Conference of the Society of Instrument and Control Engineers of Japan, SICE 2016 (pp. 750-755). Tokoyo: Society of Instrument and Control Engineers. Adrian, D., Durumeric, Z., Singh, G., & Halderman, J. A. (2014). Zippier ZMap: Internet-Wide Scanning at 10 Gbps. 8th USENIX Workshop on Offensive Technologies (WOOT '14) (p. N/A). San Diego: Usenix. Anonymous. (2012, December N/A). Internet Census 2012. Retrieved from Census 2012: http://census2012.sourceforge.net/paper.html Antunes, N., & Vieira, M. (2010). Benchmarking Vulnerability Detection Tools for Web Services. IEEE International Conference on Web Services (p. N/A). Miami: IEEE. Arnaert, M., Bertrand, Y., & Boudaoud, K. (2016). Modeling Vulnerable Internet of Things on SHODAN and CENSYS: An Ontology for Cyber Security. SECURWARE 2016 : The Tenth International Conference on Emerging Security Information, Systems and Technologies, (pp. 299-302). Bailey, M., Dittrich, D., Kenneally, E., & Maughan, D. (2012). Thee Menlo Report. DC: IEEE Computer and Reliability Societie. Baroni, F. (2016, January 13). ZoomEye : the CyberSpace Search Engine You Never Heard of. Retrieved from Pentest Guru: http://www.pentest.guru/index.php/2016/01/13/zoomeye-the-valid-iot-searchengine-you-never-heard-of/ Bartlett, G., Heidemann, J., & Papadopoulos, C. (2007). Understanding Passive and Active Service Discovery . 7th ACM SIGCOMM conference on Internet measurement (pp. 5770). San Diego: Association for Computing Machinery. BBC News. (2017, October 27). NHS 'could have prevented' WannaCry ransomware attack. BBC News Technology, p. N/A. Beale, J., Haroon, M., van der Walt, C., & Deraison, R. (2004). Nessus Network Auditing: Jay Beale Open Source Security Series. Syngress. Bejtlich, R. (2009, November 25). Shodan: Another Step Towards Intrusion as a Service. Retrieved from TaoSecurity Blog: https://taosecurity.blogspot.co.uk/2009/11/shodan-another-step-towardsintrusion.html Bejtlich, R. (2018, January 31). @TaoSecurity. Retrieved from Twitter: https://twitter.com/taosecurity/status/958670200357998592 Bernstein, D. (1996, N/A N/A). SYN Cookies. Retrieved from Crypto: https://cr.yp.to/syncookies.html BGPview. (2017, December 17). AS786 Jisc Services Limited. Retrieved from BGPview: https://bgpview.io/asn/786#prefixes-v4 Boardman, R., Mullock, J., & Mole, A. (2017). Guide to the General Data Protection Regulation. London: Bird & Bird. Bodenheim, R. C. (2014). Impact of the Shodan Computer Search Engine on Internet-facing Industrial Control System Devices. Fort Belvoir: AIR FORCE INSTITUTE OF

73

TECHNOLOGY WRIGHT-PATTERSON AFB OH GRADUATE SCHOOL OF ENGINEERING AND MANAGEMENT. Bodenheim, R. C., Butts, J., Dunlap, S., & Mullins, B. (2014). Evaluation of the ability of the Shodan search engine to identify Internet-facing industrial control devices. International Journal of Critical Infrastructure Protection, 114-123. Bou-Harb, E., Debbabi, M., & Assi, C. (2014). Cyber Scanning: A Comprehensive Survey. IEEE COMMUNICATIONS SURVEYS & TUTORIALS, 16(3), 1496-1519. Buttner, A., & Ziring, N. (2008). Common Platform Enumeration (CPE) – Specification. Washington: MITRE. Censys. (2017). Censys. Retrieved from Censys: www.censys.io Censys. (2017, Decemeber 2). Login. Retrieved from Censys: https://censys.io/login Chiappetta, F., Fadlon, L., & Norcini Pala, A. (2017, June 26). An Analysis on the security perception of Italian Cyberspace. Models and Technologies for Intelligent Transportation Systems (MT-ITS), 5. Chris. (2016, December 9). Shodan Demonstrates Why Closing Unused IoT Ports is Critical to Cyber Security. Retrieved from Spiceworks' Community: https://community.spiceworks.com/topic/1948062-shodan-demonstrates-whyclosing-unused-iot-ports-is-critical-to-cyber-security Collins, M. (2017). Network Security Through Data Analysis: From Data to Action 2nd Edition. Sebastpol: O'Reilly Media. Coman, J. (2017, April 12). SOURCE Boston Opening Keynote. SOURCE Boston . Combs, G. (2008). Wireshark User’s Guide. Coughlan, S. (2017, June 15). Top university under 'ransomware' cyber-attack. BBC News, p. N/A. Council of The European Union. (2018). General Data Protection Regulation. Brussels: THE EUROPEAN PARLIAMENT AND OF THE COUNCIL. Cox, J. (2018, January 31). New Tool Automatically Finds and Hacks Vulnerable InternetConnected Devices. Motherboard, p. N/A. CREST. (2017, November 10). UK Approved Member Companies. Retrieved from CREST: http://www.crest-approved.org/uk/members/index.html Deri, L. (2004). Improving Passive Packet Capture: Beyond Device Polling. 4th International System Administration and Network Engineering Conference (p. N/A). Amsterdam: USENIX. Dulaunoy, A., Moreels, P.-J., & Vinot, R. (2018, March 1). Main Page. Retrieved from CVESearch Project: http://cve-search.org/ Durumeric, Z. (2017). Fast Internet-Wide Scanning: A New Security Perspective. Ann Arbor: University of Michigan . Durumeric, Z., Adrian, D., Mirian, A., Bailey, M., & Halderman, J. A. (2015). A Search Engine Backed by Internet-Wide Scanning . ACM Conference on Computer and Communications Security . 22nd , p. N/A. Denver: ACM. Durumeric, Z., Kasten, J., Adrian, D., Halderman, J. A., Bailey, M., Li, F., . . . Paxson, V. (2014). The Matter of Heartbleed. Internet Measurement Conference (pp. 2-14). Vancover: Association for Computing Machinery . Durumeric, Z., Wustrow, E., & Halderm, J. A. (2013). ZMap: Fast Internet-wide Scanning and Its Security Applications. 22nd USENIX Security Symposium (pp. 605-619). Washington: USENIX Association.

74

Ercolani, V., Patton, M. J., & Chen, H. W. (2016). Shodan Visualized. IEEE International Conference on Intelligence and Security Informatics: Cybersecurity and Big Data, ISI 2016 (pp. 193-195). Beijing: IEEE. Fitzgerald, W., & Foley, S. (2013). Avoiding Inconsistencies in the Security Content Automation Protocol. Communications and Network Security. Fonseca, J., Vieira, M., & Madeira, H. (2007). Testing and Comparing Web Vulnerability Scanning Tools for SQL Injection and XSS Attacks. 13th Pacific Rim International Symposium on Dependable Computing (p. N/A). Melbourne: IEEE. Forbis, S. (2015). Integration of ZMap with Shodan for Comprehensive Internet of Things Research . Tucson: THE UNIVERSITY OF ARIZONA . Franceschi-Bicchierai, L. (2017, July 29). More Than 120,000 Internet Connected Cameras Can Be Easily Hacked, Researcher Warns. Motherboard. Gallagher, S. (2018, Febuary 1). Threat or menace? “Autosploit” tool sparks fears of empowered “script kiddies. Arstechnica, p. N/A. Gawron, M., Cheng, F., & Meinel, C. (2017). PVD: Passive Vulnerability Detection. 8th International Conference on Information and Communication Systems (ICICIS) (pp. 322-327). Irbid, Jordan: International Conference on Information and Communication Systems. Genege, B., Haller, P., & Enăchescu, C. (2016). Beyond Internet Scanning: Banner Processing for Passive Software Vulnerability Assessment. INTERNATIONAL JOURNAL OF INFORMATION SECURITY SCIENCE, 81-91. Genge, B., & Enăchescu, C. (2016). ShoVAT: Shodan-based Vulnerability Assessment Tool for Internet-facing Services. Security and Communication Networks, 2696–2714. Genge, B., & Enachesu, C. (2015). Non-Intrusive Historical Assessment of Internet-Facing. International Conference on Recent Achievements in Mechatronics, Automation, Computer Science and Robotics (p. N/A). N/A: De Gruyter. GomoR. (2012, September 21). SinFP3 Operating System Fingerprinting Tool Released. Retrieved from MetaBrik: https://www.metabrik.org/blog/2012/09/21/sinfp3operating-system-fingerprinting-tool-released/ Graham, R. (2013). C10M Defending The Internet At Scale. Shmoocon (p. N/A). Washington D.C: Shmoocon. Graham, R. (2013, December 31). Masscan: Designing My Own Crypto. Retrieved from Errata Security Blog: http://blog.erratasec.com/2013/12/masscan-designing-myown-crypto.html Graham, R. (2013, September 14). Masscan: The Entire Internet in 3 Minutes. Retrieved from Errata Security's Blog: http://blog.erratasec.com/2013/09/masscan-entire-internetin-3-minutes.html Graham, R. (2018, January 31). @ErrataRob. Retrieved from Twitter: https://twitter.com/ErrataRob/status/958792631810711553 Graham, R., McMilan, P., & Tentler, D. (2014, August 9). Massscanning the Internet. DEFCON 22. Las Vegas, Nevada, USA: DEFCON. Han, S., Jang, K., Park, K., & Moon, S. (2010). PacketShader: a GPU-accelerated software router. SIGCOMM '10 (pp. 195-206). New Delhi: Association for Computing Machinery . Havergal, C. (2015, December 8). Cyber attack on Janet network disrupts university internet connections. Times Higher Education, p. N/A.

75

Havergal, C. (2016, March 16). UK universities 'losing cyber security battle'. Times Higher Education, p. N/A. Hoole, A. M. (2016). Security Vulnerability Verification through Contract-Based Assertion Monitoring at Runtime. Victoria: University of Victoria. Hung Nguyen, V., & Massacci, F. (2013). The (Un)Reliability of NVD Vulnerable Versions Data: an Empirical Experiment on Google Chrome Vulnerabilities. 8th ACM Symposium on Information, Computer and Communications Security. Hangzhou: Asia CCS. Information Commissioner's Office. (2017, November 15). Overview of the General Data Protection Regulation (GDPR). Retrieved from Information Commissioner's Office: https://ico.org.uk/for-organisations/data-protection-reform/overview-of-the-gdpr Information Commissioner's Office. (2018, March 18). Guide to GDPR. Retrieved from Information Commissioner's Office: https://ico.org.uk/for-organisations/guide-tothe-general-data-protection-regulation-gdpr/ Internet Assigned Numbers Authority. (2018, March 18). Internet Assigned Numbers Authority. Retrieved from Internet Assigned Numbers Authority: https://www.iana.org IVRE. (2017, December 29). IVRE - Network Recon Framework. Retrieved from IVRE: https://ivre.rocks/#about Jielin, D. (2007). Network Dictionary . Javvin Technologies Inc. Karlsson, M. (2012). The Edit History of the National Vulnerability Database. Zurich: Swiss Federal Institute of Technology Zurich. Kelion, L. (2016, February 17). Students hit by University of Greenwich data breach. BBC News, p. N/A. Khadilkar, V., Rachapalli, J., & Thuraisingham, B. (2010). Semantic Web Implementation Scheme for National Vulnerability Database. Dallas: University of Texas. Khandelwal, S. (2018, March 7). Memcached DDoS Exploit Code and List of 17,000 Vulnerable Servers Released. The Hacker News, p. N/A. Kiravuo, T., Tiilikainen, M., & Manner, J. (2015). Peeking under the skirts of a nation: Finding ICS Vulnerabilities in the Critical Digital Infrastructure. ECCWS2015 - Proceedings of the 14th European Conference on Cyber Warfare and Security (p. 137). Hatfield: Academic Conferences and Publishing International. Kumar, H. (2014). Learning Nessus for Penetration Testing. Birmingham: Packt Publishing. Lalet, P., Monjalet, F., & Mougey, C. (2017). IVRE - a network recon framework. https://ivre.rocks/. CEA: the French Alternative Energies and Atomic Energy Commission. Layden, J. (2016, March 9). Trivial path for DDoS amplification attacks found by infosec bods. The Register, p. N/A. Lee, T., & Peter In, H. (2017). Levenshtein Distance-based Regularity Measurement of Circadian Rythm Patterns. Journal of Theoretical and Applied Information Technology, 4358-4366. Leverett, E. (2011). Quantitatively Assessing and Visualising Industrial System Attack Surface. Cambridge: University of Cambridge. Leverett, E., & Kaplan, A. (2017). Towards estimating the untapped potential: a global malicious DDoS mean capacity estimate. Journal for Cyber Policy, 195-208.

76

Leyden, J. (2018, Febuary 2). Hey, you know what the internet needs? Yup, more industrial control systems for kids to hack. Retrieved from The Register: https://www.theregister.co.uk/2018/02/02/industrial_control_system_security/ Lubell , J., & Zimmerman, T. (2017). CHALLENGES TO AUTOMATING SECURITY CONFIGURATION CHECKLISTS IN MANUFACTURING ENVIRONMENTS. International Conference on Critical Infrastructure Protection XI (pp. 175-191). Springer. Lyon, G. (2009). NMap Network Scanning: The Official NMap Project Guide to Network Discovery and Security Scanning. N/A: NMap Project . Manes, G., Schulte, D., Guenther, S., & Shenoi, S. (2005). NetGlean: A Methodology for Distributed Network Security Scanning. Journal of Network and Systems Management, 329-344. Martin, G., Kinross, J., & Hankin, C. (2017). Effective Cybersecurity is Fundamental to Patient Safety. London: BMJ. Matherly, J. (2014). Inside The World’s Most Dangerous Search Engine. ShowMeCon. Matherly, J. (2015, December 17). Memory As A Service. Retrieved from Shodan Blog: https://blog.shodan.io/memory-as-a-service/ Matherly, J. (2017, October 26). Shodan. Retrieved from Shodan.io: www.shodan.io Matherly, J. (2017). The Complete Guide to Shodan. Austin: leanpub. McClure, S., Scambray , J., & Kurtz, G. (2009). Hacking Exposed: Network Security Secrets and Solutions. New York: The McGraw-Hill Companies. McCune, R. (2016). The Rise and Fall (?) of Penetration Testing. Scottish Justice Matters. Messori, G. (2016, N/A N/A). Dynamic IP addresses are now "personal data": why you should care about it. Retrieved from Chino: https://www.chino.io/blog/ip-addresses-arepersonal-data-why-you-should-care-about-it-2/ Mirataheri, S. M., Dinçtürk, M. E., Hooshmand, S., Bochmann, G. V., Jourdan, G.-V., & Onut, I. V. (2013). A Brief History of Web Crawlers. Conference of the Center for Advanced Studies on Collaborative Research (pp. 40-54). Ontario: IBM Corp. MITRE. (2017, December 9). Common Vulnerabilities and Exposures: The Standard for Information Security Vulnerability Names. Retrieved from Homepage: https://cve.mitre.org/index.html Modeling Vulnerable Internet of Things on SHODAN and CENSYS : An Ontology for Cyber Security. (2016). SECURWARE 2016, The Tenth International Conference on Emerging Security Information, Systems and Technologies (pp. 299 - 302). Nice: The International Academy, Research and Industry Association (IARIA). Mohammed, R. (2016). Assessment of Web Scanner Tools. International Journal of Computer Applications, 133(5), N/A. Möller, B., Duong, T., & Kotowicz, K. (2014). This POODLE Bites: Exploiting The SSL 3.0 Fallback. Palo Alto: Google. Mukhopadhyay, I., Goswami, S., & Mandal, E. (2014). Web Penetration Testing using Nessus and Metasploit Tool. IOSR Journal of Computer Engineering, 16(3), 126-129. Na, S., Kim, T., & Kim, H. (2017). A Study on the Serivce Identification of Internet-Connected Devices Using Common Platform Enumeration. Advanced Multimedia and Ubiquitous Engineering (pp. 237-241). Sinapore: Springer. Nair, G. V., & Devi, S. (2015). Assessing and Comparing Vulnerability Detection In Web. IOSR Journal of VLSI and Signal Processing (IOSR-JVSP), 11-23.

77

National Cyber Security Centre. (2017, August 9). Guidance: Penetration Testing. Retrieved from National Cyber Security Centre's Guidance: https://www.ncsc.gov.uk/guidance/penetration-testing National Insititute of Standards and Technology. (2011). Common Platform Enumeration: Name Matching Specification Version 2.3. Gaithersburg: U.S. Department of Commerce. National Insitute of Standards and Technology. (2011). Common Platform Enumeration: Naming Specification Version 2.3. Gaithersburg: U.S. Department of Commerce. NIST. (2016, September 21). National Vulnerability Database (NVD). Retrieved from National Institute of Standards and Technology: https://www.nist.gov/programsprojects/national-vulnerability-database-nvd NullArray. (2018, January 30). AutoSploit. Retrieved from GitHub: https://github.com/NullArray/AutoSploit NVD. (2011, 05 24). CVE-2011-1928 Detail. Retrieved from National Vulernability Database: https://nvd.nist.gov/vuln/detail/CVE-2011-1928 NVD. (2013, November 11). CVE-2013-1899 Detail. Retrieved from National Vulnerability Database: https://nvd.nist.gov/vuln/detail/CVE-2013-1899 NVD. (2014, March 11). CVE-2014-0204 Detail. Retrieved from National Vulnerability Database: https://nvd.nist.gov/vuln/detail/CVE-2014-0204 NVD. (2014, October 10). CVE-2014-3566 Detail. Retrieved from National Vulnerability Database: https://nvd.nist.gov/vuln/detail/cve-2014-3566 NVD. (2016, July 10). CVE-2015-2080 Detail. Retrieved from National Vulnerability Database: https://nvd.nist.gov/vuln/detail/CVE-2015-2080 NVD. (2017, 12 15). CVE-2014-0160 Detail. Retrieved from National Vulnerability Database: https://nvd.nist.gov/vuln/detail/CVE-2014-0160 NVD. (2017, Febuary 2). CVE-2016-9244 Detail. Retrieved from National Vulnerability Database: https://nvd.nist.gov/vuln/detail/CVE-2016-9244 O’Harrow Jr., R. (2012, June 3). Cyber Search Engine Shodan Exposes Industrial Control Systems to New Risks. Retrieved from The Washington Post: http://www.washingtonpost.com/investigations/cyber-search-engine-exposesvulnerabilities/2012/06/03/gJQAIK9KCV_story.html%5Cnhttp://www.washingtonpos t.com/investigations/cyber-search-engine-exposesvulnerabilities/2012/06/03/gJQAIK9KCV_print.html Panetta, K. (2016). Gartner's Top 10 Security Predictions 2016. New York: Gartner. Panjwani, S., Tan, S., Jarrin, K. M., & Cukier, M. (2005). An Experimental Evaluation to Determine if Port Scans are Precursors to an Attack. International Conference on Dependable Systems and Networks (pp. 602-611). Yokohama: IEEE. Patrick Breyer v Bundesrepublik Deutschland, C-582/14 (Court of Justice of the European Union October 19, 2016). Peers, S. (2017, January 4). IP addresses as personal data - the CJEU's judgment in C-582/14 Breyer. Retrieved from EU Law Analysis: http://eulawanalysis.blogspot.co.uk/2017/01/ip-addresses-as-personal-datacjeus.html PenTestIT. (2017, May 1). Leviathan: An All In One Security Audit Toolkit! Retrieved from PenTestIt: http://pentestit.com/leviathan-security-audit-toolkit/ Pinsent Masons. (2008, March). IP addresses and the Data Protection Act. Retrieved from Out-Law: https://www.out-law.com/page-8060

78

Pinto, M., & Stuttard, D. (2011). The Web Application Hacker's Handbook: Finding and Exploiting Security Flaws. Indianapolis: Wiley. Quarta, D., Pogliani, M., Polino, M., Maggi, F., Zanchettin, A. M., & Zanero, S. (2017). An Experimental Security Analysis of an Industrial Robot Controller. IEEE Symposium on Security and Privacy (pp. 268-286). San Jose: IEEE. Quinn, S., Waltermire, D., Johnson, C., Scarfone, K., & Banghart, J. (2011). NIST SP 800-126: Technical Specification for the Security Content Automation Protocol, SCAP Version 1.0. Gaithersburg: National Institute of Standards and Technology. Ragan, S. (2016, Spetember 22). Over 6,000 vulnerabilities went unassigned by MITRE's CVE project in 2015. Computer Security Online, p. N/A. Ramos, A. (2013, January 28). Hunt CCTV Credential Disclosure. Retrieved from Packet Storm Security: https://packetstormsecurity.com/files/119871/Hunt-CCTV-CredentialDisclosure.html Ranger, S. (2018, March 1). GitHub hit with the largest DDoS attack ever seen. ZDNet, p. N/A. Reintjes, K. (2014). A benchmark approach to analyse the security of web frameworks. Nijmegen: Radboud University Nijmegen. Ritchie, P. (2017, December 1). cve-offline. Retrieved from GitHub Repository: https://github.com/cornerpirate/cve-offline Robinson, T. (2018, January 31). Autosploit marries Shodan, Metasploit, puts IoT devices at risk. SC Magazine, p. N/A. Rogers, C. (2018, January 16). Challenges facing education and cybersecurity. EdQuater, p. N/A. Rohrmann, R. R. (2017 ). Large Scale Anonymous Port Scanning . Tuscon: THE UNIVERSITY OF ARIZONA. RT. (2017, September 22). In Our Bid to Take Over the World We Hunt ICS Devices Using Maltego. Retrieved from Maltego Blog: http://maltego.blogspot.co.uk/2017/09/inour-bid-to-take-over-world-we-hunt.html Samtani, S., Yu, S., Zhu, H., & Patton, M. C. (2016). Identifying SCADA Vulnerabilities Using Passive and Active Vulnerability Assessment Techniques. EEE International Conference on Intelligence and Security Informatics: Cybersecurity and Big Data, ISI 2016 (pp. 25-30). Beijing: IEEE. Sanders, C., & Smith, J. (2013). Applied Network Security Monitoring: Collection, Detection, and Analysis. Elsevier. Sanghvi, H. P., & Dahiya, M. S. (2013). Cyber Reconnaissance: An Alarm before Cyber Attack. International Journal of Computer Applications , 63(6), N/A. Sanguino, L., & Uetz, R. (2017). Software Vulnerability Analysis Using CPE and CVE. SANS Analyst Program. (2017). Cyber Security Trends: Aiming Ahead of the Target to Increase Security in 2017. Boston: SANS Institute. SANS Institute. (2002). Conducting a Penetration Test on an Organization. SANS Institute. Scans.io. (2018, January 4). Internet-Wide Scan Data Repository. Retrieved from Scans.io: https://scans.io/ Schearer, M. (2010). Shodan for Penetration Testers. DEFCON (p. N/A). Las Vegas: DEFCON. Seebug. (2018, January 4). Seebug Vulnerability Platform. Retrieved from Seebug Website: https://www.seebug.org/ Shasmi, Z. (2017). Hershel: Single-Packet OS Fingerprinting. IEEE/ACM Transcations on Networking, 2196-2209.

79

Shelton, M. (2018, March 1). Passive Asset Detection System. Retrieved from Sourceforge: http://passive.sourceforge.net Shodan. (2015, June 4). Shodan's Tweet. Retrieved from Twitter: https://twitter.com/shodanhq/status/606513250709184512?lang=en Shodan. (2017, Decemeber 19). Buy Export Credits. Retrieved from Shodan: https://www.shodan.io/store/credits Shodan. (2017, December 19). Images Search Engine. Retrieved from Shodan: images.shodan.io Shodan. (2017). Shodan Search Engine . Retrieved from Shodan : www.shodan.io Shodan. (2017, October 26). Shodan's Twitter. Retrieved from Twitter: www.twitter.com/shodanhq Shodan Help Center. (2017, December 14). Create a GIF from an IP Image History. Retrieved from Shodan Help Center: https://help.shodan.io/developer-tutorials/creating-gifsfromhistory?utm_content=buffera5ade&utm_medium=social&utm_source=twitter.com &utm_campaign=buffer Sieklik, B., Macfarlane, R., & Buchanan, W. J. (2016). Evaluation of TFTP DDoS amplification attack. Computers & Security, 67-92. Simon, K., Moucha, C., & Keller, J. (2017). Contactless Vulnerability Analysis using Google and Shodan. Journal of Universal Computer Science, 23(4), 404-430. Skottler. (2018, March 1). February 28th DDoS Incident Report. Retrieved from GitHub Engieering Blog: https://githubengineering.com/ddos-incident-report/ Skybox Security. (2016). 2015 Enterprise Vulnerability Management Trends Report. Los Angelos: Skybox Security . Stevens, W. (1994). TCP/IP Illustrated: The Protocols, Volume 1. Addison-Wesley Professional. Suto, L. (2010). Analyzing the Accuracy and Time Costs of Web Application Security Scanners. London: BeyondTrust. Synopsys. (2017, December 18). The Heartbleed Bug. Retrieved from Heartbleed: http://heartbleed.com Tara, E. (2018, February 16). 7900 Vulnerabilities Didn't Make It into the CVE Database in 2017. Infosecurity-magazine, p. N/A. Techopedia. (2017, Decemeber 28). Remote Desktop Protocol (RDP). Retrieved from Techopedia : https://www.techopedia.com/definition/3422/remote-desktopprotocol-rdp Techopedia. (2017, December 29). Virtual Networking Computing (VNC). Retrieved from Techopedia: https://www.techopedia.com/definition/3308/virtual-networkcomputing-vnc Tenable. (2011, N/A N/A). Apache 2.2.x < 2.2.18 APR apr_fnmatch DoS. Retrieved from Nessus' Plugins: https://www.tenable.com/plugins/index.php?view=single&id=54646 Tenable. (2017, Novemeber 1). Passive Vulnerability Scanners. Retrieved from Tenable Documentation: https://docs.tenable.com/sccv/5_3/Content/PassiveVulnerabilityScanners.htm Tenable. (2018, Febuary 1). Nessus Professional. Retrieved from Tenable Products: https://www.tenable.com/products/nessus/nessus-professional

80

Tentler, D. '. (2018, January 31). @Viss. Retrieved from Twitter: https://twitter.com/Viss/status/958798468486590464 The Economist. (2017, May 2017). Data is Giving Rise to a New Economy. The Economist, p. N/A. The European Parliament. (2018). General Data Protection Regulation. Brussels: European Parliament and Council. The European Parliament. (2018). The General Data Protection Regulation: Recital 30. In T. E. Parliament. Brussels: The European Parliament. The Week. (2017, September 5). Hackers target research from UK universities. The Week, p. N/A. Thomson, I. (2017, June 28). Everything you need to know about the Petya, er, NotPetya nasty trashing PCs worldwide. The Register, p. N/A. Tittel, E. (2016, November 8). Tenable Nessus Vulnerability Scanner: Product overview. TechTarget, p. N/A. Tohru, Y. (2016, June 28). The infrastructure exposed to the outside is protruding from Tokyo ~ SHODAN 5 prefectures (Trend Micro). Retrieved from Scan Net Security: https://scan.netsecurity.ne.jp/article/2016/06/28/38644.html Trend Micro. (2017). US Cities Exposed: Industries and ICS . Los Angelos: Trend Micro Forward-Looking Threat Research (FTR) Team. Trend Micro. (2017). Western European Cities Exposed: A Shodan-based Security Study on Exposed Cyber Assets in Western Europe. Los Angeles: Trend Micro Forward-Looking Threat Research (FTR) Team. TrendMicro. (2017). Rogue Robots: Testing the Limits of an Industrial Robot’s Security. Cork: Trendlabs. Tung, Y.-H., Tseng, S., Shih, J.-F., & Shan, H.-L. (2013). A Cost-Effective Approach to Evaluating Security Vulnerability Scanner. Asia-Pacific Network Operations and Management Symposium, 15, N/A. UK Goverment. (1990). Computer Misuse Act 1990. London: The Stationary Office. UK Goverment. (1998). Data Protection Act 1998. London: The Stationary Office. Varmazis, M. (2017, October 12). Mr Robot season 3 episode Eps3.0_Power-Saver-Mode.H – the security review. Naked Security . Weidman, G. (2014). Penetration Testing: A Hands-On Introduction to Hacking. New York: No Starch Press. Wendlandt, D. (2010, January 1). Nessus : A security vulnerability scanning tool. Retrieved from dwendlan: http://www.cs.cmu.edu/~dwendlan/personal/nessus.html Xu, X., Wan, J., Zhang, W., Tong, C., & Wu, C. (2011). PMSW: a passive monitoring system in wireless sensor networks. International Journal of Network Management, 21(4), N/A. Yeung, P., & Bennett, R. (2017, Septemeber 5). University secrets are stolen by cybergangs. The Times, p. N/A. Zalewski, M. (2017, November 1). p0f v3 (version 3.09b). Retrieved from lcamtuf: http://lcamtuf.coredump.cx/p0f3/ Zhang, S., Caragea, D., & Xinming, O. (2011). An Empirical Study on Using the National Vulnerability Database to Predict Software Vulnerabilities. DEXA 2011: Database and Expert Systems Applications (pp. 217-231). Toulouse: International Conference on Database and Expert Systems Applications.

81

ZoomEye. (2017, December 29). Cyberspace Search Engine. Retrieved from ZoomEye: https://www.zoomeye.org/

82

Jamie O’Hare Appendix

40168785

A

Initial Project Overview SOC10101 Honours Project (40 Credits) Title of Project: Analysis of Passive Vulnerability Assessment Tools Overview of Project Content and Milestones To research and analyse methods use to passively assess networked devices for vulnerabilities.

The Main Deliverable(s): • • • •

A literature review Evaluation of assessment experiments A poster featuring findings A report of findings

The Target Audience for the Deliverable(s): • Security Professionals and Researchers • Networking Professionals and Researchers

The Work to be Undertaken: Research techniques and tools used to passively assess devices for vulnerabilities and how the tools are evaluated. Develop experiments to test the performance of these tools and evaluating the experiments.

Additional Information / Knowledge Required: • Service banner uses and implementation • Knowledge on network management

Information Sources that Provide a Context for the Project: Shodan. (2017). Shodan.io. Retrieved 28 September 2017, from http://www.shodan.io

• Shodan allows the user to search through service banners, this can be used as reference.

The Importance of the Project: Attacks making the news are most commonly old vulnerabilities that have been patched however the user has not installed the relevant patch. A prime example being the Equifax breach. Using passive vulnerability assessment tools, a user can

83

B Port Number 7 11 13 15 17 19 21 22 23 25 26 37 49 53 67 69 70 79 80 81 82 83 84 88 102 104 110 111 119 123 129 137 143 161 175 179 195 311 389 389 443 443 444 445 465 500 502 503 515 520 523 554 587 623 626 636 666 771 789 873 902 992 993 995 1010

84

Port Description Echo Systat Daytime1 Netstat Quote of the day Character generator FTP SSH Telnet SMTP SSH rdate TACACS+ DNS DHCP TFTP, BitTorrent Gopher Finger HTTP, malware HTTP, malware HTTP, malware HTTP HTTP Kerberos Siemens S7 DICOM POP3 Portmapper NNTP NTP Password generator protocol NetBIOS IMAP SNMP IBM Network Job Entry BGP TA14-353a OS X Server Manager LDAP CLDAP HTTPS QUIC TA14-353a, Dell SonicWALL SMB SMTPS IKE (VPN) Modbus Modbus Line Printer Daemon RIP IBM DB2 RTSP SMTP mail submission IPMI OS X serialnumbered LDAPS Telnet Realport Redlion Crimson3 rsync VMWare authentication Telnet (secure) IMAP with SSL POP3 with SSL malware

Port Number 4500 4567 4070 4800 4840 4911 4949 5006 5007 5008 5009 5060 5094 5222 5269 5353 5357 5432 5577 5632 5672 5900 5901 5938 5984 6000 6001 6379 6666 6667 6881 6969 7218 7474 7548 7777 7779 8010 8060 8069 8087 8090 8099 8112 8139 8140 8181 8333 8334 8443 8554 8880 8888 8889 9001 9002 9009 9042 9051 9100 9151 9160 9191 9418 9443

Port Description IKE NAT-T (VPN) Modem web interface HID VertX/ Edge door controller Moxa Nport OPC UA Niagara Fox with SSL Munin MELSEC-Q MELSEC-Q NetMobility Apple Airport Administration SIP HART-IP XMPP XMPP Server-to-Server mDNS Microsoft-HTTPAPI/2.0 PostgreSQL Flux LED PCAnywhere RabbitMQ VNC VNC TeamViewer CouchDB X11 X11 Redis Voldemort database, malware IRC BitTorrent DHT TFTP, BitTorrent Sierra wireless (Telnet) Neo4j database CWMP (HTTPS) Oracle Dell Service Tag API Intelbras DVR Roku web interface OpenERP Riak Insteon HUB Yahoo SmartTV Deluge (HTTP) Puppet agent Puppet master GlassFish Server (HTTPS) Bitcoin Bitcoin node dashboard (HTTP) HTTPS RTSP Websphere SOAP HTTP, Andromouse SmartThings Remote Access Tor OR Tor OR Julia Cassandra CQL Tor Control Printer Job Language Tor Control Apache Cassandra Sierra wireless (HTTP) Git Sierra wireless (HTTPS)

1023 1025 1099 1177 1200 1234 1400 1434 1521 1604 1723 1741 1833 1900 1911 1962 1991 389 389 443 443 444 445 465 500 502 503 515 520 523 554 587 623 626 636 666 771 789 873 902 992 993 995 1010 1023 1025 1099 1177 1200 1234 1400 1434 1521 1604 1723 1741 1833 1900 1911 1962 1991 2000 2081 2082 2083 2086 2087

85

Telnet Kamstrup Java RMI malware Codesys udpxy Sonos MS-SQL monitor Oracle TNS Citrix, malware PPTP CiscoWorks MQTT UPnP Niagara Fox PCworx malware LDAP CLDAP HTTPS QUIC TA14-353a, Dell SonicWALL SMB SMTPS IKE (VPN) Modbus Modbus Line Printer Daemon RIP IBM DB2 RTSP SMTP mail submission IPMI OS X serialnumbered LDAPS Telnet Realport Redlion Crimson3 rsync VMWare authentication Telnet (secure) IMAP with SSL POP3 with SSL malware Telnet Kamstrup Java RMI malware Codesys udpxy Sonos MS-SQL monitor Oracle TNS Citrix, malware PPTP CiscoWorks MQTT UPnP Niagara Fox PCworx malware iKettle, MikroTik bandwidth test Smarter Coffee cPanel cPanel WHM WHM

9595 9600 9869 10001 10001 10243 11211 17000 17185 12345 13579 14147 16010 18245 20000 20547 21025 21379 23023 23424 25105 25565 27015 27017 28017 30313 30718 32400 37777 44818 47808 49152 49153 50070 51106 53413 54138 55443 55553 55554 62078 64738 2181 2222 2323 2332 2375 2376 2404 2455 2480 2628 3000 3306 3310 3386 3388 3389 3460 3541 3542 3689 3780 3787 4000 4022 4040

LANDesk Management Agent OMRON OpenNebula Automated Tank Gauge Ubiquiti Microsoft-HTTPAPI/2.0 Memcache Bose SoundTouch VxWorks WDBRPC Sierra wireless (Telnet) Media player classic web interface Filezilla FTP Apache Hbase General Electric SRTP DNP3 ProconOS Starbound Matrikon OPC Telnet Serviio Insteon Hub Minecraft Steam A2S server query, Steam RCon MongoDB MongoDB (HTTP) Gardasoft Lighting Lantronix Setup Plex Dahuva DVR EtherNet/IP Bacnet Supermicro (HTTP) WeMo Link HDFS Namenode Deluge (HTTP) Netis backdoor Toshiba PoS McAfee Metasploit Metasploit Apple iDevice Mumble Apache Zookeeper SSH, PLC5, EtherNet/IP Telnet Sierra wireless (Telnet) Docker Docker IEC-104 CoDeSys OrientDB Dictionary ntop MySQL ClamAV GTPv1 RDP RDP malware PBX GUI PBX GUI DACP Metasploit Ventrilo malware udpxy Deprecated Chef web interface

2123 2152

GTPv1 GTPv1

4063 4064 4369 4443 4444

ZeroC Glacier2 ZeroC Glacier2 with SSL EPMD Symantec Data Center Security malware

C Port 80 443 443 443 7547 502 21 143 993 110 995 25 22 53 123 1900

Protocol HTTP HTTPS HTTPS HTTPS CWMP MODBUS FTP IMAP IMAPS POP3 POP3S SMTP SSH DNS NTP UPnP

Sub Protocol GET / TLS SSLv3 Heartbleed GET / Device ID Banner Grab Banner Grab Banner Grab Banner Grab Banner Grab Banner Grab RSA OpenResolver Get Time Discovery

D Type of Organisation Small/Medium Business Home User Other Corporation Academic Institution Government/Military Internet Service Provider Unknown Total

86

Number of Responses 41 38 17 22 15 2 10 145

E 'Apache/2.4.18 (Ubuntu)' 'Apache' 'lighttpd/1.4.33' 'Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.2k-fips PHP/7.0.20' 'Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.2k-fips PHP/7.0.20’ 'Microsoft-IIS/7.5' 'Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.2k-fips PHP/5.6.27' 'Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.2k-fips PHP/5.6.28’ 'lighttpd/1.4.33' 'nginx/1.10.3 (Ubuntu)' 'Microsoft-HTTPAPI/2.0' 'Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.2k-fips PHP/5.6.29' 'Microsoft-HTTPAPI/2.0' 'Apache/2.4.25 (Fedora) OpenSSL/1.0.2k-fips PHP/7.0.18' 'Microsoft-HTTPAPI/2.0' 'Microsoft-IIS/8.5' 'Apache/2.4.27 (Fedora) OpenSSL/1.0.2m-fips' 'Microsoft-HTTPAPI/2.0' 'Microsoft-IIS/7.5' 'Microsoft-HTTPAPI/2.0' 'Microsoft-IIS/8.5' 'nginx/1.10.3 (Ubuntu)' 'Microsoft-HTTPAPI/2.0' 'Microsoft-IIS/8.5' 'Microsoft-IIS/7.5' 'Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.2k-fips PHP/5.6.29' 'Microsoft-IIS/8.5' 'nginx' 'Microsoft-IIS/7.5' 'Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.2k-fips PHP/7.0.21' 'Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_fcgid/2.3.9 mod_wsgi/3.4 Python/2.7.5' 'Microsoft-IIS/7.0' 'Apache/2.4.7 (Ubuntu)' 'Apache/2.4.18 (Ubuntu)' 'Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.2k-fips PHP/5.6.27' 'Microsoft-HTTPAPI/2.0' 'nginx' 'Microsoft-HTTPAPI/2.0' 'Microsoft-IIS/7.5' 'Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.2k-fips PHP/5.6.28'

87

F ['Apache httpd 2.4.6', 'Apache httpd', 'Microsoft IIS 8.5', 'Apache httpd 2.4.27', 'Apache httpd 2.4', 'Apache Coyote 1.1', 'Apache httpd 2.2.3', 'Microsoft HTTPAPI 2.0', 'Microsoft IIS 6.0', 'Microsoft IIS 7.5', 'Apache httpd 2.2.14', 'nginx', 'Apache httpd 2.2.15', 'Apache httpd 2.2', 'Apache httpd 2.2.21', 'Bomgar Bomgar', 'nginx 1.12.2', 'Apache httpd 2.4.3', 'Jetty', 'Apache httpd 2.4.7', 'Apache httpd 2.4.10', 'nginx 1.10.3', 'Apache httpd 2.2.22', 'Microsoft IIS 8.0', 'Apache httpd 2.2.31', 'Apache httpd 2.4.18', 'Apache httpd 2.4.25', 'Zope', 'Microsoft IIS 10.0', 'RStudio', 'nginx 1.13.7', 'Apache httpd2.2.11', 'nginx 1.12.1', 'Apache httpd 2.2.20', 'Apache httpd 2.2.24', 'lighttpd 1.4.33', 'Apache httpd 2.4.28', 'nginx 1.10.2', 'lighttpd 1.4.35', 'Apache httpd 2.4.29', 'Apache httpd 2.2.26', 'nginx 1.4.6', 'gunicorn 19.7.1', 'nginx 1.12.0', 'eHTTP', 'PRTG 18.1.37.12158', 'ELOG', 'Apache httpd 2.2.19', 'nginx 1.10.1', 'WSGIServer 0.2', 'nginx 1.13.3', 'WEBrick 1.3.1']

88

G Metadata CPE Match TP TN FP FN Apache httpd 2.4.6 Apache httpd Microsoft IIS 8.5 Apache httpd 2.4.27 Apache httpd 2.4 Apache Coyote 1.1 Apache httpd 2.2.3 Microsoft HTTPAPI 2.0 Microsoft IIS 6.0 Microsoft IIS 7.5 Apache httpd 2.2.14 nginx Apache httpd 2.2.15 Apache httpd 2.2 Apache httpd 2.2.21 Bomgar Bomgar nginx 1.12.2 Apache httpd 2.4.3 Jetty Apache httpd 2.4.7 Apache httpd 2.4.10 nginx 1.10.3 Apache httpd 2.2.22 Microsoft IIS 8.0 Apache httpd 2.2.31 Apache httpd 2.4.18 Apache httpd 2.4.25 Zope Microsoft IIS 10.0 RStudio nginx 1.13.7 Apache httpd2.2.11 nginx 1.12.1 Apache httpd 2.2.20 Apache httpd 2.2.24 lighttpd 1.4.33 Apache httpd 2.4.28 nginx 1.10.2 lighttpd 1.4.35 Apache httpd 2.4.29 Apache httpd 2.2.26 nginx 1.4.6 gunicorn 19.7.1 nginx 1.12.0 eHTTP PRTG 18.1.37.12158 ELOG Apache httpd 2.2.19 nginx 1.10.1 WSGIServer 0.2 nginx 1.13.3 WEBrick 1.3.1 Total

89

✓ ✕ ✕ ✓ ✓ ✕ ✕ ✕ ✓ ✓ ✕ ✕ ✓ ✕ ✓ ✕ ✕ ✓ ✕ ✓ ✓ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✓ ✓ ✕ ✕ ✓ ✕ ✓ ✕ ✕ ✕ ✕ ✕ ✕ ✓ ✓ ✕ ✕ ✕ 18

✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ 17

✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ 23

✓ 1

✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ 12

H Table 12

𝑇𝑃 11 11 = = = 0.73 𝑇𝑃 + 𝐹𝑃 11 + 4 15

𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 = 𝑅𝑒𝑐𝑎𝑙𝑙 =

𝐹1 =

𝑇𝑃 11 11 = = = 0.61 𝑇𝑃 + 𝐹𝑁 11 + 7 18

2 × 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 × 𝑅𝑒𝑐𝑎𝑙𝑙 2 × 0.73 × 0.61 0.89 = = = 0.66 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 + 𝑅𝑒𝑐𝑎𝑙𝑙 0.73 + 0.61 1.34

Table 13

𝑇𝑃 10 10 = = = 0.77 𝑇𝑃 + 𝐹𝑃 10 + 3 13

𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 = 𝑅𝑒𝑐𝑎𝑙𝑙 =

𝐹1 =

𝑇𝑃 11 11 = = = 0.71 𝑇𝑃 + 𝐹𝑁 11 + 3 14

2 × 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 × 𝑅𝑒𝑐𝑎𝑙𝑙 2 × 0.77 × 0.71 1.34 = = = 0.74 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 + 𝑅𝑒𝑐𝑎𝑙𝑙 0.77 + 0.71 1.64

Table 14

𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 = 𝑅𝑒𝑐𝑎𝑙𝑙 =

𝐹1 =

𝑇𝑃 4 4 = = = 0.33 𝑇𝑃 + 𝐹𝑃 4+8 12

𝑇𝑃 4 4 = = = 0.67 𝑇𝑃 + 𝐹𝑁 4+2 6

2 × 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 × 𝑅𝑒𝑐𝑎𝑙𝑙 2 × 0.33 × 0.67 0.34 = = = 0.44 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 + 𝑅𝑒𝑐𝑎𝑙𝑙 0.33 + 0.67 0.91

Table 15

𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 = 𝑅𝑒𝑐𝑎𝑙𝑙 =

𝐹1 =

90

𝑇𝑃 2 2 = = = 0.67 𝑇𝑃 + 𝐹𝑃 2+1 3

𝑇𝑃 2 2 = = = 0.50 𝑇𝑃 + 𝐹𝑁 2+2 4

2 × 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 × 𝑅𝑒𝑐𝑎𝑙𝑙 2 × 0.67 × 0.50 0.67 = = = 0.57 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 + 𝑅𝑒𝑐𝑎𝑙𝑙 0.67 + 0.50 1.17

I

J Overleaf is OpenVAS results starting with Apache 2.4.7. Followed by Apache 2.4.10, Lighttpd 1.4.18 and finally Lighttpd 1.4.33

91