On the Horizon Editor: O. Sami Saydjari,
[email protected]
Worm and Attack Early Warning Piercing Stealthy Reconnaissance
W
arfare’s first stage usually involves intelligence gathering. Attackers locate their would-be victims, identify their defensive capabilities, and determine what weaponry they’ll use.
Stealth is important during reconnaissance to maintain the element
of surprise. Attackers don’t want to forewarn their targets of an impending assault. The same is true for cyberattacks. Attackers perform many ongoing nefarious Internet activities, including stealthy reconnaissance, low and slow scanning, and probing, to identify opportunities to penetrate and launch exploits. These activities can be arbitrary and random, rather than directed or strategic. How can the “good guys” detect strategic stealthy reconnaissance to identify future attacks? Is it possible to identify attackers’ intent and predict their behavior, or are their activities so undirected that they’re indistinguishable from that of random-probing worms? In this installment of On the Horizon, I describe some aspects of the Worminator project (http:// worminator.cs.columbia.edu), a collaboration of academic institutions pursuing R&D of intelligent predictive and proactive technologies that detect, report, and defend against early preattack cyberevents—specifically network observables—that are precursors to malicious activities during a later attack stage. The project aims to measure and significantly in-
crease the warning time for a zeroday attack (an attack against an unreported vulnerability) to give security analysts and decision makers time to take preventive steps and for automated adaptive response mechanisms to reconfigure IT infrastructures to minimize the impact and losses. Worminator addresses two broad areas: • perimeter detection and early warning of potential worm propagations and precursors to zero-day attack against a secured site, and • the impact on network anomalydetection systems to discover potential new malicious attacks that have pierced the perimeter defenses, including automating the generation of zero-day attack signatures. The key requirement for these pursuits is a distributed set of sensors that can accurately detect stealthy reconnaissance activities at network gateways’ exterior perimeters.
Distributed sensors Antura Recon (called Recon here) is a commercial product (made by System Detection, Inc.; www.sysd.
PUBLISHED BY THE IEEE COMPUTER SOCIETY
■
com/index.html) resulting from DARPA-sponsored research at Columbia University. It detects scans and probes against an enclave, a secured computer network protected by perimeter defenses defined by a common set of security policies and implemented by firewalls and intrusion-detection systems (IDS). Attackers perform port scanning and probing, looking for vulnerable services to attack or ways to gain a detailed map of available hosts and open ports. They attempt to breach an enclave’s defenses by learning which internal IPs they can reach and what ports and services are available for attack. Recon resides outside a network’s firewall and sniffs all traffic directed to the site, but it is not a honeypot. While a honeypot sees a subset of available traffic—only the traffic destined to its services— Recon is a passive sniffer and is not addressable on the network. The most important class of surveillance activities that Recon detects are those considered to be stealthy—reconnaissance activities from an outside source IP conducted slowly so that commercial off-the-shelf (COTS) systems can’t detect it. In a formal DARPAfunded comparative evaluation, a standard IDS detected 23 verified stealthy scans, while Recon detected 8,600. In those field tests, a significant portion of the detected nonstealthy and fast surveillance activities included ongoing Code Red and Nimda activities months after their first appearance.
1540-7993/04/$20.00 © 2004 IEEE
■
IEEE SECURITY & PRIVACY
SALVATORE J. STOLFO Columbia University
73
On the Horizon
Correlated watchlists Recon includes a watchlist that stores detected IPs (organized according to their stealthiness behavior) as a database table of alerts. The watchlist has several important uses, including reprioritizing alert outputs from internal detectors and potentially predicting future attacks from watchlisted source IPs. Thus, Recon provides important new information about attack and attacker behavior that’s valuable for fine-tuning the priority of detected attacks, and also might provide early warning capabilities. The Worminator project aims to validate this conjecture. Recon also detects distributed scans—with common probe profiles—emanating from many disparate source IPs. A distributed set of Recon installations with correlated watchlists provides additional valuable information, and could serve as an early-warning system for worm propagation and coordinated attacks. Detecting common sets of source IPs executing scanning and probing activities across a wide area of victims indicates that worm propagations are underway. Recon’s particular focus on detecting slow stealthy scans and probes would therefore detect stealthy worm propagations that attempt to avoid standard IDS system detection. Generally, bursts in probe activity are easy to spot but slow and methodical propagations would go unnoticed if we focused only on burst activity. Columbia University’s Worminator group is studying the specific case of detecting scans and probes among distributed sites as a general model for coordinated attacks against multiple targets. Worminator includes efficient and tamper-proof communication among the coordinated sites for high-speed real-time alert distribution. For the actual data structure to be exchanged among the distributed sensor network’s nodes, we provide an implementation of bloom filters, a method for representing elements (detected scanning IPs, in this case) to support mem74
IEEE SECURITY & PRIVACY
■
MAY/JUNE 2004
bership queries in correlating distributed sets of detected sources. We might be able to detect classes of potential coordinated worm propagations through advanced data analysis. It is technically feasible for us to analyze a source IP’s scanning and probing behavior. Then, we can cluster it into groups of sources that exhibit common behavior characteristics. We could accomplish this by analyzing scan behavior (for example, intraprobe delays) as well as payload characteristics (for example, common characteristics of both control and content data in the scanning packets’ payloads). Currently, Columbia University and the Georgia Institute of Technology are running Recon sensors and conducting informal experiments. Other sites that will join this study include the Florida Institute of Technology, Syracuse University, and the Massachusetts Institute of Technology. This ongoing study aims to provide evidence that uncoordinated attacks and worm propagations might be detectable by correlated watchlists from a set of distributed sites and that these sites also might serve as a realistic test bed for mitigation experiments. In one field test, we found evidence that the premise was sound: Recon detected a scanning source against a target approximately three hours prior to the actual attack. Unfortunately, the amount of data available was limited in real-time duration and, hence, prior scanning activities of that source IP were not detectable simply because there was no data gathered for that time period. Another recent Worminator field test that exchanged Recon alert data between Columbia and Georgia Tech discovered three common sources of stealthy surveillance: Beijing, the Philippines, and a small western US community college. We are watching these sources to determine what behavior they exhibit over weeks and months. We don’t know why these three sources are in-
terested in Columbia and Georgia Tech, but we theorize that they are seeking exploitable university machines for nefarious purposes. Part of Worminator’s core research includes: • Distributing sensors. Determining the number and strategic placement of distributed sensors (for example, at an enclave’s gateway, at an upstream peering point, or both) for a particular-size enclave maximizes coverage and minimizes communication cost and time to detect propagations and attack precursors. • Inferring intent. The relationships among common targeted victims suggest what a scanning source’s intent might be. For example, a common source of stealthy scanning from an attacking IP address directed toward a set of unrelated victims appears fundamentally different than an attacker scanning a set of IP addresses all owned by, say, several different banks. • Profiling behavior. A longitudinal study of attacker behavior and intent and their attacks against victims might suggest precursor observables we can use to predict future attacker behavior. Attack tools are commonly available and shared. Thus, using common tools might provide sufficient repeated behavior to accurately predict future attack steps. • Classifying activities. We need a way to quickly classify worms and scan or probe activity into useful clusters and profiles according to their characteristics (destination ports, interprobe delay, and payload length, for example) and behavior. What might we do with the information derived from these studies and analyses? One very useful outcome is to better defend secured networks from crafty new zero-day attacks. The distributed sensors and real-time sharing of attack alert information among distributed sites could provide an early attack warn-
On the Horizon
ing system for all participating sites who then could respond by updating firewall defenses or implementing other mitigation strategies.
Positive outcomes Recon sensor alert output— produced at one site or shared among sites—could contribute to several possible positive outcomes, advanced warning to blacklist sources and prevent their attacks, and detecting penetrations of zero-day attacks by watchlisted IPs. A complete autonomic defense and attack-prevention system might use the additional information about ongoing activities at the protected site’s perimeter of which it otherwise wouldn’t be aware. For example, if watchlisted source IPs—found hours earlier by Recon and via shared alerts from other Worminator sites—also established “good” connections with a target IP that the Recon sensor grid covers, the suspect source IP’s good network connection should generate an alert. (A good network connection means that the source IP did not generate a particular alert, either by a standard COTS IDS or by an anomaly detector, yet it did establish a connection with an internal service.) The fact that the source IP conducted a stealthy reconnaissance activity might indicate that it had launched a new attack for which no standard IDS has a known signature. Furthermore, any alerts that a watchlisted source IP generates should have an impact on the response that an autonomic defense and prevention system generates—perhaps a rapid update to the firewall rules blocking that source IP, regardless of other evidence such a system might use in generating response and mitigation decisions. In either case, early scan and probe activity warning and identification would enrich the information available about the security posture and state of an enclave possibly under attack, or one that might soon be. The Worminator software also will provide a new packet payload
anomaly detector. This new sensor—PAYL—generates alerts whenever it sees an unusual packet datagram directed toward some service or port. The PAYL sensor is a true anomaly detector; it is trained in an environment and learns a model of normal, expected data traffic to some selected service port. A Worminator version now under development will include a means of sharing PAYL alerts among sites, with a signature of a new anomalous payload that has a high likelihood of being a new worm. We believe coupling PAYL’s alert output with evidence that Recon’s sensors acquired and shared will detect zero-day worm propagations, whether slow or fast moving.
ther organizations have indicated interest in deploying Recon sensors, joining the Worminator grid to share information about common sources of stealthy attacks, and profiling attackers over time to develop criteria for advanced warning of impending attacks. Registered users can check out the Worminator site hosted at Columbia (mentioned earlier) to see ongoing results and trends discovered over time by long-term correlated watchlists and attacker behavior. We also have joined in a collaboration with a European research group at the Institute of Computer Science of the Foundation for Re-
O
search and Technology–Hellas (www.ics.forth.gr), which has proposed the Noah project. Noah uses widely distributed honeypots within the participating sites’ protected LANs and thus is most similar to the Network Telescope project at the University of California, San Diego. It will be particularly interesting to determine what a detection grid might uncover about attackers and attacked sites across a significantly large area of the Internet.
Acknowledgments Researchers participating on the Worminator project include Phil Gross, Shlomo Hershop, Gail Kaiser, Angelos Keromytis, Wei-Jen Li, Michael Locasto, Tal Malkin, Vishal Misra, Janak Parekh, Morris Pearl, and Ke Wang at Columbia University, Pete Manolios and Wenke Lee at the Georgia Institute of Technology, Philip Chan at the Florida Institute of Technology, Roman Markowski and Steve Chapin at Syracuse University, and Jeffrey Schiller at the Massachusetts Institute of Technology. Salvatore J. Stolfo is professor of computer science at Columbia University and cofounder and chief scientific advisor of System Detection, Inc., makers of Recon. His research interests include intrusion- and anomaly-detection systems, data mining, and distributed systems. He has a BS in computational mathematics from Brooklyn College and a PhD from New York University Courant Institute of Mathematical Sciences. Contact him at
[email protected]. edu; www.cs.columbia.edu/~sal.
A new department editor
R
eaders might notice that On the Horizon has a new department editor. O. Sami Saydjari has received the baton from founding editors Nancy R. Mead and Gary McGraw, who will continue to contribute to S&P in other capacities. We thank them for their hard work and expertise in getting S&P in general and On the Horizon in particular off to a great start. Under Sami’s direction, On the Horizon will continue to explore trends and developments that will impact security and privacy technologies today and tomorrow.
Sami is founder and president of the Cyber Defense Agency, a consulting company that creates defenses against cyberattacks. Previously, he was a senior staff scientist in SRI International’s Computer Science Laboratory and held various titles at the National Security Agency and DARPA. He has an MS in computer science from Purdue University and has published more than a dozen technical papers in the field of information security. Contact him at ssaydjari@cyberdefense agency.com.
www.computer.org/security/
■
IEEE SECURITY & PRIVACY
75
