Collective detection of potentially harmful requests directed at web ...

Collective detection of potentially harmful requests directed at web sites Marek Zachara AGH University of Science and Technology, 30 Mickiewicza Av., 30-059, Krakow, Poland [email protected]

Abstract. The number of web-based activities and websites is growing every day. Unfortunately, so is cyber-crime. Every day, new vulnerabilities are reported and the number of automated attacks is constantly rising. Typical signature-based methods rely on expert knowledge and the distribution of updated information to the clients (e.g. anti-virus software) and require more effort to keep the systems up to date. At the same time, they do not protect against the newest (e.g. zero-day) threats. In this article, a new method is proposed, whereas cooperating systems analyze incoming requests, identify potential threats and present them to other peers. Each host can then utilize the findings of the other peers to identify harmful requests, making the whole system of cooperating servers “remember” and share information about the threats.

1

Introduction

There is little doubt that an increasing part of peoples’ activities has moved over to the Internet. From online shopping and digital banking to personal blogs and social networking sites the number of websites is constantly growing, with the recent number of 800 million websites (of which almost 200 million were considered active) reported by the end of 2013 [14], While the underlying operating systems, web servers, firewalls, and databases are usually well known and tested products that are subject to continuous scrutiny by thousands of users, the applications themselves are often much less tested (if at all). However, even if all the underlying software is secure, a successful attack against the web application may compromise the data in its database. At the same time, the attacks directed at the web application are performed via normal http requests, which pass undetected by typical security means (e.g. a firewall). More about applications vurnelabilities can be found in [9], and the most common ones are periodically published be CWE [5].

2

The Unprotected Web Applications

The reason why firewalls do not protect websites from harmful requests is that their ability is only to filter traffic at the lower layers of the OSI model, while the

2

Collective detection of harmful requests

identification of harmful requests is only possible at layer 7. There are specialized firewalls, known as Web Application Firewalls (WAF), but their use is limited mainly because of the effort required to configure and maintain them. An answer to this issue could be a learning WAF proposed in [15], but it is still in the early stages. The mentioned paper also describes the reasons for the low adoption of the application firewalls in more details. As a result, each web application has to deal with security issues on its own. This means that each web developer should be familiar with security issues and be able to address them in their application. Experience and data shows however, that this is not the case. 2.1

The state of websites security

According to the report [8], 99% of the web applications they had tested had at least one vulnerability (35 on average), with 82% of the websites having at least one high/critical vulnerability. This report was based on their customer base of over 300 companies. In another report [18], WhiteHat Security states that 86% of the web applications they had tested had at least one serious vulnerability, with an average of 56 vulnerabilities per web application. On the other hand, Symantec claims that while running 1400 vulnerability scans per day, they have found approximately 53% of the scanned websites to have at least one unpatched vulnerability [17]. Also, in a single month of May of 2012 the LizaMoon toolkit was responsible for at least a million successful SQL Injections attacks. In the same report, they also state that approximately 63% of websites used to distribute malware were actually legitimate websites compromised by attackers. Since these companies do not publish details on their methodology it’s hard to precisely assess the number of vulnerable websites, but an exact number is not that important. Whether it is 50% or 90% of all, this means that there are hundreds of millions of websites with unpatched vulnerabilities on the Internet.

3

Proposed Method for Prevention of Attacks

It does not seem feasible to propose a single, monolithic security system to defend the websites against various attacks. Since the attack vectors often change and evolve, the security systems need to do the same. Actually, an accepted approach to building secure systems is named “defense in depth” [1] and requires each part (layer) of the system to be secured with the best tools and knowledge. In this paper, a method to deal with specific (but popular) attack vectors is proposed, which employs methods similar to those used by the human immune system. First, let’s look at example patterns presented in Figure 1. This figure is made with actual data from a running web server. The requests arriving at the server are grouped into sessions (consecutive requests from the same source), and then presented as a graph, starting at the “Start” node in the center. For reasons of clarity, only a few sessions are presented. The sessions in the top left corner are


3

“legitimate” sessions, namely requests from regular users who visit the web page and do not attempt any harmful actions. The other three sessions, can be noted as actual attack attempts. As can be seen in this figure, these requests try to access a popular database management module [2] via various URLs at which it may be available if installed.

Fig. 1. Sample users’ sessions presented as a part of a graph, with attack attempts marked by red background

It is hard to identify whether there was a real user behind these presumed attack attempts, or if they were carried out by an automated script or a malware. The latter is actually more likely, since the majority of attack attempts are automated, either directly coming from malware or a large number of “script kiddies” [11] who download and use ready-made attack tool-kits. These two groups have the ability to perform attacks attempts on a large scale. However, these attacks are usually identical, or very similar to each other. Since they are performed by programs and usually rely on one or just a few attack vectors, they generate similar requests to a large number of web servers. 3.1

Key concepts of the proposed methodology

For clarity reasons, let’s reiterate the features of the attacks performed by malware and “script kiddies”: – Distributed. The attacks are usually targeted towards a large number of web hosts, not at the same time, but within a limited time-frame. – Similar. Malware and “script kiddies” both use single pieces of software with incorporated attack vectors. This software produces identical or very similar requests to various hosts. – Unusual. The requests generated by the automated software are not tailored for the specific host, so they vary from its usual usage patterns.

4


Fortunately, these features can be utilized to detect and neutralize the majority of the threats coming from these sources. In order to do so, a method similar to one of the immunology processes referred to above is used. First, potential unusual requests are identified whether they seem to be “suspicious” or not. This is done based on the server response and a behavioral evaluation of the request sequence. Requests identified as suspicious are stored locally for reference and also “presented” to other hosts, somewhat mimicking the way antigens are presented by various cells to the T-cells of the immune system. Periodically, each host retrieves the lists of such suspicious requests identified by its peers. The information received from other hosts, together with the local evaluation is then used to assess the request and take an appropriate action.

3.2

Identification of suspicious requests

The simplest way to identify suspicious requests is to look for HTTP 404 responses from the web server, which are results of requests for nonexistent resources or URLs. For better results, a heuristic behavior-based method is also proposed. Anomaly-based detection is a common approach for security purposes [12], [13], [4]. The proposed method is based on a dynamically build usage graph, which contains information about typical usage of the website. For the purpose of this task, it is proposed to use an oriented, weighted graph where nodes represent requested URLs and the edges denote transitions between URLs by the users. The weight of each edge is equivalent to the frequency of the particular transition (number of transitions during the recent time window). Obviously, users’ session tracking (e.g. using cookies) is necessary for reliable results. In Figure 2 a sample graph is presented.

/pl/students_res

/pl/ /pl/contact/

3

19

/pl/students

3

2 34

7

9

/pl/home

26 11 6

2

/pl/marek_marks 5 15

17

/pl/students/marek

5

Start

3

1 6

/pl/research

Fig. 2. Sample graph representing a time-framed usage of a website

Furthermore, Figure 3, shows the typical situation where there are a number of edges with high weights and a number of edges with very low (even equal to 1) weights. These low-weighted edges usually point to nodes (i.e. URL requests) that can accurately be considered as “suspicious”.


5

Fig. 3. Distribution of edge weights in a usage graph created for a real website

3.3

Collective assessment of the requests

The result of the process described in the previous section, means that a server hosting a website can label certain incoming requests as potentially suspicious. However, the results of experiments show that such a list contains a large portion of false positives (i.e. legitimate requests that are incorrectly labeled as potentially dangerous. This list must be narrowed down, which can be done by utilizing the distributed knowledge from several websites. If a particular request is identified as suspicious locally and has also been received at other servers (and labeled by them as such), the likeliness of it being a real attack attempt is very high. To facilitate the exchange of information between web servers, various means can be utilized. It seems that the simplest way would be for the server to maintain a dedicated web page accessible from outside at a certain URL that contains a list of its findings (requests locally identified as “suspicious”). This page obviously needs to be regularly updated. Other servers can then periodically retrieve these lists from their peers and use them to verify their local findings. 3.4

Protection of sensitive information

Certainly, publishing information about received requests could be a security issue in its own right (including a potential security and/or confidentiality breach). Fortunately, it is not necessary to disclose full requests’ URLs to other peers. Since each host just needs to check if the URL it marked as suspicious has also been reported by other hosts, it is enough if the hosts compare the hashes (digests) of the requests’ URLs. Hashes (like SHA-256 and MD6) are generated using one-way functions and are generally deemed irreversible. If a host only publishes hashes of the requests that it deems suspicious, it allows other hosts to verify their findings, but at the same time protects its potentially confidential information. For the proposed method to work, it is enough if hosts just publish a list of hashes of the received requests that they consider suspicious, but other data

6


(e.g. type of anomaly, the time it was detected, etc.) may improve the interoperability and possible future heuristics. Therefore it is suggested that hosts use a structured document format, e.g. based on [10]. A sample published document may look like the one in Listing 1.1. This will be referred to as a Server List of Suspicious Requests, abbreviated SLSR. The list presented in 1.1 is a part of a list taken from an actual running instance of the reference implementation. Listing 1.1. Sample SLSR with additional debug information { { { { { {

C :O , C :O , C :F , C :F , C :F , C :O ,

3.5

T :M , T :M , T :M , T :M , T :M , T :B ,

A :57 , A :53 , A :32 , A :31 , A :24 , A :17 ,

MD5 :2 c f 1 d 3 c 7 f e 2 e a d b 6 6 f b 2 b a 6 a d 5 8 6 4 3 2 6 MD5 :2370 f 2 8 e d a e 0 a f c d 8 d 3 b 8 c e 1 d 6 7 1 a 8 a c MD5 :2 f 4 2 d 9 e 0 9 e 7 2 4 f 4 0 c d f 2 8 0 9 4 d 7 b e a e 0 a MD5 :8 f 8 6 1 7 5 a c d e 5 9 0 b f 8 1 1 5 4 1 1 7 3 1 2 5 d e 7 1 MD5 : e e e 5 c d 6 e 3 3 d 7 d 3 d e a f 5 2 c a d e b 5 9 0 e 6 4 2 MD5 : b d 9 c d b f e d c a 9 8 4 2 7 c 8 0 a 4 1 7 6 6 f 5 a 3 7 8 3

} } } } } }

Maintenance of the lists

For the process to work as intended, each server must not only identify suspicious requests, but also generate and publish the list of them and retrieve similar lists from other servers. However, exchanging of the SLRS leads to an issue of data retention, and two questions need to be answered: – How long should an SLSR contain an entry about a suspicious request after such a request was received. – Should the hashes received from other peers be preserved locally if the originating server does not list them any more, and if so, for how long? Both issues are related to the load (i.e. number of requests per second) received either by the local or the remote server. Servers with very high loads and a high number of suspicious requests will likely prefer shorter retention times, while niche servers may only have a few suspicious requests per week and would prefer a longer retention period. It is difficult to currently propose a justified approach to these issues, until the proposed method is more widely accepted, which would provide necessary statistical data. Therefore, as a “rule of thumb” it may be assumed that a record (i.e. a suspicious request) should not be listed for more than a couple of weeks and it may be removed earlier if the SLSR grows too long (e.g. more than a couple of hundred records). Experience shows that some attack attempts last for months or years (e.g. the attempts at phpMyAdmin) but some attacks, particularly if they are malware-based, tend to die off quickly as the vulnerability they exploit is patched.

4

Implementation and deployment

A reference implementation has been prepared to verify the feasibility of the proposed method and to prove this concept. The application has been programmed in Java and executed on a few real web servers. The architecture of the application is presented in Figure 4.


7

Fig. 4. The architecture of the reference implementation

The application monitors the requested access log files of Apache Web Server and retrieves incoming requests. This is one of a well-established security monitoring methods [7], [16], [3]. The requests for unavailable resources are directly forwarded to the aggregation of suspicious requests. Other requests are first analyzed by session extractor, which groups them into user sessions. After filtering out requests for the web page elements (e.g. images), the primary requests for web pages are used to build the behavior graph explained earlier in this article. Requests that do not seem to match typical behavior (i.e. they do not match the built graph) are considered suspicious and forwarded to the “suspicious activity detector” module. This module aggregates all suspicious requests and matches them against pattern-based rules, primarily to eliminate requests that are well known and harmless, yet for some reason are still reported to this module. Such requests are mainly automated requests from browsers for the website icon ’favicon.ico’ or requests from automated crawlers “robots.txt” which are often missing in websites. Finally, the remaining suspicious requests are formatted according to the sample presented in Listing 1.1, and are stored in a document inside the web server directory to be accessible to other hosts. 4.1

Reasoning

The last module (named ’Reasoning’) receives the current request, which is the list of requests locally considered as suspicious, and, at the same time periodically retrieves similar lists from other hosts. With each request, the module has to decide whether it should report it for human (e.g. administrator’s) attention or not. There are various strategies that can be implemented here, depending on the type of application being protected and the number and type of peer servers it receives the data from. Since the reference test environment consisted of only three hosts, a simple algorithm was used. This algorithm reported requests that were both locally considered an anomaly and listed by at least one other peer server. Even this simple strategy provided considerable results, as will be demonstrated later in the paper.

8


5

Test method and achieved results

The prototype application was deployed on three web servers. These were: – A small commercial B2B application used to coordinate work between one company and its customers. – A website used by a group of people to inform others about local events. – A personal (author’s) website. All the web servers were located in the same city. The relatively large number of requests served by the third (personal website) server was due to the fact that it contained teaching materials and was therefore used by the author’s students. The number of all received requests is relatively high because it includes all requests for all the resources (not only web pages, but all images, scripts, etc.) The results received from the application were compared against a semimanual analysis of log files (negative selection against a set of patterns representing legitimate requests). These results are presented in Table 1. Table 1. Server A (small B2B dynamic application): requests statistics Server A

ServerB

ServerC

small B2B

community site

author’s site

33 706

143 211

193 241

Reported attempts of which most common /w00tw00t.* /wp-login.php phpMyAdmin variations

306

278

377

75 48 47

75 31 37

45 ? 29

Manually identified attempts of which most common /w00tw00t.* /wp-login.php phpMyAdmin variations

979

612

1 134

124 50 359

115 34 96

118 ? 313

All received requests

In terms of the efficiency of the described method, the reference single-thread implementation (in Java) was able to process over 4,000 requests per second on a typical PC (Intel, 3GHz). This was achieved without any specific optimizations and was a couple orders of magnitude higher than required - considering the real load that the servers experienced.

6

Conclusions and Future Work

The method proposed in this paper aims at providing automated detection of potentially harmful requests with the minimum level of involvement from the


9

system administrators. Inspired by some mechanisms that evolved within the human immune system, it works by presenting potentially harmful requests to other peers for verification. A final decision is then reached by considering the votes of other peers along with local judgment. The method may significantly hinder the modus operandi of most malware, due to the fact that after initial the attempts to attack a number of servers, it will become increasingly difficult for a malware to successfully attack new servers. These servers will recognize the attacks because of the knowledge acquired from its peers. This way, the whole system will develop an immune response similar to the one observed in living organisms. The reference implementation shows the benefits for deploying the proposed method, even in a very small test scenario of only three server nodes. The limited detection rate of attacks directed at phpMyAdmin results from various permutations of the URL, including package version numbers. This is the area that certainly requires further attention, because introducing some heuristics generalizing the request URLs should greatly boost the detection rate. Also, deploying these application on a larger number of servers will likely greatly increase its effectiveness. It is important to note, that the proposed method provides unique features, that make it difficult to compare to other anomaly-based methods published so far. While most of these methods require some kind of training with labeled data sets (e.g. [6], [12]) to distinguish the attacks, the method described in this paper does not have such prerequisite. Also, compared to the mentioned methods, which tend to report a number of false positives, this method provides virtually no such false alarms - a feature of great value to potential end users. Further work that could improve the detection ratio should mainly be focused on the evaluation of decision criteria that result in reporting of a particular request (i.e. the “reasoning” module). These should include: The impact of the number of peers on reasoning accuracy, the retention period for both local and remote SLSRs and the form of the decision algorithm (e.g. input weights, rules, etc.). The generalization of request URLs has already been mentioned.

References 1. Defense in Depth: A practical strategy for achieving Information Assurance in today’s highly networked environments. http://www.nsa.gov/ia/\_files/ support/defenseindepth.pdf, http://www.nsa.gov/ia/_files/support/ defenseindepth.pdf 2. PhpMyAdmin. http://sourceforge.net/projects/phpmyadmin/ 3. Agosti, M., Crivellari, F., Di Nunzio, G.: Web log analysis: a review of a decade of studies about information acquisition, inspection and interpretation of user interaction. Data Mining and Knowledge Discovery 24(3), 663–696 (2012), http://dx.doi.org/10.1007/s10618-011-0228-8;http://www. bibsonomy.org/bibtex/29b85b7d3c5587c5f0920f0d602ba93b1/sdo 4. Auxilia, M., Tamilselvan, D.: Anomaly detection using negative security model in web application. In: Computer Information Systems and Industrial Management Applications (CISIM), 2010 International Conference on. pp. 481–486 (2010)

10


5. Christey, S.: 2011 CWE/SANS Top 25 Most Dangerous Software Errors. Tech. rep. (2011), http://cwe.mitre.org/top25/archive/2011/2011_cwe_sans_top25.pdf 6. Cova, M., Balzarotti, D., Felmetsger, V., Vigna, G.: Swaddler: An Approach for the Anomaly-Based Detection of State Violations in Web Applications. In: Kruegel, C., Lippmann, R., Clark, A. (eds.) Recent Advances in Intrusion Detection, Lecture Notes in Computer Science, vol. 4637, pp. 63–86. Springer Berlin Heidelberg (2007), http://dx.doi.org/10.1007/978-3-540-74320-0_4 7. Iv´ ancsy, R., Vajk, I.: Frequent Pattern Mining in Web Log Data. Acta Polytechnica Hungarica 3(1) (2006), http://citeseerx.ist.psu.edu/ viewdoc/summary?doi=10.1.1.101.4559;http://www.bibsonomy.org/bibtex/ 2f29f4627c9ae99370fc7ba005982e2e6/sdo 8. iVIZ: Web Application Vulnerability Statistics Report. http://www. securitybistro.com/?p=4966 (2013) 9. Johari, R., Sharma, P.: A Survey on Web Application Vulnerabilities (SQLIA, XSS) Exploitation and Security Engine for SQL Injection. In: Communication Systems and Network Technologies (CSNT), 2012 International Conference on. pp. 453–458 (2012) 10. JSON: A lightweight data-interchange format. http://www.json.org 11. Kayne, R.: What Are Script Kiddies. http://www.wisegeek.com/ what-are-script-kiddies.htm 12. Kruegel, C., Vigna, G.: Anomaly Detection of Web-Based Attacks. pp. 251–261. ACM Press (2003), http://www.bibsonomy.org/bibtex/ 2099e1b9a6e57960e4b3e02410e83cb64/liangzk 13. Kruegel, C., Vigna, G., Robertson, W.: A multi-model approach to the detection of web-based attacks. Computer Networks 48(5), 717–738 (Aug 2005), http://dx. doi.org/10.1016/j.comnet.2005.01.009 14. Netcraft: Web Server Survey. http://news.netcraft.com/archives/2013/11/01/ november-2013-web-server-survey.html (2013) 15. Palka, D., Zachara, M.: Learning web application firewall - benefits and caveats. In: Proceedings of the IFIP WG 8.4/8.9 international cross domain conference on availability, reliability and security for business, enterprise and health information systems. Lecture Notes in Computer Science, vol. 6908, pp. 295–308. SpringerVerlag (2011), http://dl.acm.org/citation.cfm?id=2033973.2033999 16. Sun, Z., Sheng, H., Wei, M., Yang, J., Zhang, H., Wang, L.: Application of web log mining in local area network security. In: EMEIT. pp. 3897– 3900. IEEE (2011), http://dblp.uni-trier.de/db/conf/emeit/emeit2011. html#SunSWYZW11;http://dx.doi.org/10.1109/EMEIT.2011.6023097;http: //www.bibsonomy.org/bibtex/23badf5326d9486b9b17c48ab47576eaa/dblp 17. Symantec: Internet Security Threat Report. http://www.symantec.com/ security\_response/publications/threatreport.jsp (2013) 18. WhiteHat: Website Security Statistics Report. http://info.whitehatsec.com/ 2013-website-security-report.html (2013)