An Efficient Online Anomalies Detection Mechanism ...

Anomaly Detection Mechanism for High-Speed Networks

Motivations

An Efficient Online Anomalies Detection Mechanism for High-Speed Networks

Background Network Monitoring CUmulated SUM Count-Min Sketch

Osman Salem, Sandrine Vaton, Annie Gravey

Proposed approach Change point detection over sketch Sketch inversion

Evaluation

ENST Bretagne Departement of Computer Science Brest, France

Conclusions

MonAM 2007 LAAS, Toulouse, France, 5 and 6 November 2007

Anomaly Detection Mechanism for High-Speed Networks

Motivations

Outline

1

Motivations

2

Background Network Monitoring CUmulated SUM Count-Min Sketch

3

Proposed approach Change point detection over sketch Sketch inversion

4

Evaluation

5

Conclusions

Background Network Monitoring CUmulated SUM Count-Min Sketch

Proposed approach Change point detection over sketch Sketch inversion

Evaluation Conclusions

Anomaly Detection Mechanism for High-Speed Networks

Motivations Goals:

Motivations Background Network Monitoring CUmulated SUM Count-Min Sketch

Proposed approach Change point detection over sketch Sketch inversion

Anomaly detection over high speed links Problems addressed: Fast per-packet update Scalable with high-speed links Small memory usage and quick access Fast and accurate attacks detection & identification DoS, DDoS, PortScan, NetScan, etc.

Evaluation Conclusions

Contributions: Derive the minimum memory requirement s.t an error rate Propose an efficient method for sketch inversion Use a sequential algorithm that is memory efficient and allows fast detection Build a tool for attacks detection and classification from a large volume of traffic in real time

Anomaly Detection Mechanism for High-Speed Networks

Motivations Two existing approaches:

Motivations Background Network Monitoring CUmulated SUM Count-Min Sketch

Proposed approach Change point detection over sketch Sketch inversion

Evaluation

Signature-based: looking for known patterns They cannot detect unknown network threats Worm spreads and gains control of network in a few minutes

Statistics-based: abnormal deviation and big changes from statistical profile Prior knowledge is not required

Conclusions

Motivations: Existing IDS are insufficient: They are mostly HIDS or located on end routers Not scalable to high speed networks Most are based on overall traffic & cannot provide information even if they find some anomalies Most cannot simultaneously detect different types of anomalies

Anomaly Detection Mechanism for High-Speed Networks

Network Monitoring

Motivations Background Network Monitoring CUmulated SUM Count-Min Sketch

Proposed approach Change point detection over sketch

DBMS (Oracle, mysql)

DB analysis is slow & expensive

















  

  





  

  





 







 

  











Network Monitoring • What are the top-M (100) dst seen over the last minutes?

Sketch inversion

Evaluation

• SELECT COUNT (src, dst) FROM Table

Conclusions

WHERE dst = src

The challenge Potentially tens of millions of time series/records ! Finding needle in a haystack

Existing approaches not directly applicable

Anomaly Detection Mechanism for High-Speed Networks

Anomaly detection algorithm CUmulated SUM (CUSUM)

Motivations Background Network Monitoring CUmulated SUM Count-Min Sketch

Proposed approach

Uses CUSUM algorithm for attack detection Change point detection algorithm TSA based on overall aggregation of TCP SYN packets With fixed false alarm rate & small delay detection Attack is detected if current profile violates normal profile

Change point detection over sketch Sketch inversion

Evaluation Conclusions

How to detect deviations? Hypothesis testing: H0 before attack and H1 after attack Observe: X = (X1 , X2 , . . . , Xγ , Xγ+1 . . . , Xn ) = (X1 , X2 ) X1 , X2 , . . . , Xγ from fθ (x) with θ = θ0 before t0 Xγ+1 . . . , Xn from gθ (x) with θ = θ1 after t0

Anomaly Detection Mechanism for High-Speed Networks

Basic idea behind CUSUM 



  θ

Motivations Background Network Monitoring CUmulated SUM Count-Min Sketch

Proposed approach

 



 

Change point detection over sketch

θ

Sketch inversion

Evaluation 0

Conclusions

60 −200

50

Alarm time

−400

30

Sk

#SYN

40

−600

20

h

10 −800 mk

0 −10 0

50

100 Time

150

(a) Change point

200

−1000 0

50

100 Time

150

(b) Variation of Sk

200

Anomaly Detection Mechanism for High-Speed Networks

Multichannel-CUSUM is more accurate Problems:

Motivations Background Network Monitoring CUmulated SUM Count-Min Sketch

Proposed approach Change point detection over sketch Sketch inversion

Evaluation Conclusions

CUSUM based on overall SYN packets counter CUSUM raises only alarm when attack is detected No information about victim server or attack type Multichannel CUSUM: CUSUM over many time series Better accuracy than single CUSM Per-flow analysis is too slow & expensive Not adequate for real time (potentially tens of millions of time series!) Statistical detection unscalable for flow-level detection Need scalable aggregation Sketch provides a random aggregation

Anomaly Detection Mechanism for High-Speed Networks

Count-Min Sketch Definition (Sketch)

Motivations

Compact summary of data stream

Background

Sketching is randomized aggregation Array of hash table: C[d][w]

Network Monitoring CUmulated SUM Count-Min Sketch

Proposed approach

Multi-stage bloom filter Small summary as an array of w × d in size Use d hash functions to map vector entries to [1, . . . , w] Probabilistic Guarantees $ ! " # −

Change point detection over sketch Sketch inversion

Evaluation Conclusions

2

(

/

=

01

* (

+

=)

, -.

*

$

)

2

%

'

−

2

2

2

2

2

2

2

2

2

2

&

2

2 2

2

2

2

2

2

2

2

2

2

2

2

2

2

2

2

2

2

Anomaly Detection Mechanism for High-Speed Networks

Motivations Background Network Monitoring CUmulated SUM Count-Min Sketch

Proposed approach Change point detection over sketch Sketch inversion

Evaluation Conclusions

Count-Min Sketch Update Sketch Update Input stream: (key, update) Update (key, u): C[j][hj (key)]+ = u f or(j = 0, · · · , d − 1) 2-universal hash function: hj (key) = aj × key + bj mod(w) Ex: key = IP1 = 10.0.0.1 = 167772161 ⇒ h0 (IP1 ) = 1 C 7 58 9 6 3 4 P NQ R O M M

W

TX Y U V S S −

B

@A

:?

@A

; ?

@A

< ?

@A

=?

@A

> ?

@A

:

G

E

F

C

G

H

H

C

C

H

C

C

C

C

C

C

H

H

I

I

J

J

G

I

J

I

I

I

I

I

J

I

I

I

J

J

J

G

J

J

I

J

K

H

I

I

J

J

I

I

I

I

I

I

H

H

I

H

L

C

I

I

J

−

D

Anomaly Detection Mechanism for High-Speed Networks

Count-Min Sketch Query

Motivations Background Network Monitoring CUmulated SUM Count-Min Sketch

Proposed approach

CMS Estimate: Approximate Answer Estimate: CM S − Query(key) = minj (C[j][hj (key)]) Ex: CM S − Query(IP1 ) = minj (2, 3, 2, 2) = 2

Change point detection over sketch Sketch inversion

^ \_` ] Z [ x vy z w u u

Evaluation Conclusions



|€  } ~ { { −

i

gh

af

gh

bf

gh

cf

gh

df

gh

ef

gh

a

j

n

o

p

n

o

r

t

n

o

o

r

q

o

o

r

r

n

o

n

o

p

r

r

o

o

n

p

r

n

l

m r

n

o

o

o

s

−

r

r

o

n

n

r

n

r

r

n

s

r

n

r

r

p

r

r

r

s

r

o

r

r

n

o

k

Anomaly Detection Mechanism for High-Speed Networks

Motivations Background Network Monitoring CUmulated SUM Count-Min Sketch

Proposed approach Change point detection over sketch Sketch inversion

MNP-CUSUM over Sketch cells

CUSUM over Sketch Uses d × w CUSUM function One CUSUM function per cell MNP-CUSUM only raises alarm in buckets with big changes in time series

Evaluation ‚

Conclusions ‚

‡

‰

−

ˆ

†

…

„

−

ƒ

Anomaly Detection Mechanism for High-Speed Networks

Motivations Background Network Monitoring CUmulated SUM

MNP-CUSUM over Sketch cells

Problems: Sketch does not store any information about its key entry

Count-Min Sketch

Proposed approach Change point detection over sketch Sketch inversion

Sketch is not reversible Storage of key is required to determine its value keeping track of per-key value is expensive

Evaluation Conclusions

‹

Š

“

Œ

Ž

”

•

’

‘ –

Number of keys N = 232 if we keep track of source IPs Number of keys N = 2104 if we keep track of 5-tuples (srcIP, dstIP, srcPort, dstPort, proto) as in Netflow

Anomaly Detection Mechanism for High-Speed Networks

Motivations

Reverse Sketch Problem

Solution:

Background

A novel approach for sketch inversion

Network Monitoring CUmulated SUM

A software compliant approach Proposed framework is based at 2 sketches:

Count-Min Sketch

Proposed approach Change point detection over sketch

1

Sketch inversion

2

Multi-Layer Reversible Sketch (MLRS). Count-Min Sketch (CMS).

Evaluation Conclusions

Attacks classification: 1

DDoS: identification of DIP, DP or DIP

2

PortScan: identification of SIP, DIP

3

NetScan: identification of SIP, DP

4

DoS: identification of SIP, DIP, DP

Anomaly Detection Mechanism for High-Speed Networks

Motivations

Sketches in Action System design MNP-CUSUM over Multi-Layer Reversible Sketch

Background Network Monitoring CUmulated SUM Count-Min Sketch

³´ µ µ¶˜·¸¹

´ Ü Ä

º»˜›

³Å µÅ ´

Mutli-Layer Reversible Sketch

Change detection module

Proposed approach Change point detection over sketch Sketch inversion

Evaluation Conclusions

«¬ ¬­®¯ °±²

ENDACE DAG 3.6ET

Traces of given Time slots

— ˜™š˜›  ¡¢

— ˜™š˜ ›

œ ™ž Ÿ™š

£¤¥ ¦§¨©ª

¼½¾ ¼½¾ ¿¼¾ £½ ¾¿¼½¾

ÀÀ ¤¤

£½¾ ¿¼¾

ÂÂ

ÁÁ

Attacks Classificat° Algorithm

Anomaly Detection Mechanism for High-Speed Networks

Motivations Background Network Monitoring CUmulated SUM Count-Min Sketch

Proposed approach Change point detection over sketch Sketch inversion

Evaluation Conclusions

Proposed approach

Anomaly Detection Mechanism for High-Speed Networks

Motivations Background Network Monitoring CUmulated SUM Count-Min Sketch

Proposed approach

Experiment results Dataset Well documented trace (3h of background traffic mixed with generated attacks) OTIP trace: 3 days of collect (6.9GB & 896.105 Netflow records)

Change point detection over sketch Sketch inversion

Evaluation Conclusions

Efficient data recording Hardware: Intel Pentium 1.7Ghz and 1GB of RAM memory Implemented over DAG-3.6ET (high speed data sniffing) Work in real time Only a few hundred KB of memory are used Efficient attacks detection and key inference even when not directly recorded

Anomaly Detection Mechanism for High-Speed Networks

Experiment results: Mixed traces 4

x 10

500 450

Motivations Background

#Packets

300 #SYN

Count-Min Sketch

2

350

Network Monitoring CUmulated SUM

#Packets in Auckland trace #Packets in Mixed trace

2.5

400

250 200

1.5

1

Proposed approach

150 100

Change point detection over sketch

0.5

50 0 0

Sketch inversion

Evaluation

2000

4000

6000 Time (sec)

8000

10000

0

4

x 10

Conclusions

2000

4000

6000 Time (sec)

8000

10000

2000

4000

6000 Time (sec)

8000

10000

4

x 10 3

MNP−CUSUM raised Alaram

2.5

#SYN

2

1.5

1

0.5

0

2.5

2

1.5

1

0.5

2000

4000

6000 Time (sec)

8000

10000

0 0

Anomaly Detection Mechanism for High-Speed Networks

Experiment results: OTIP Trace 4

3.5

Network Monitoring CUmulated SUM Count-Min Sketch

#Packets

Background

Proposed approach Change point detection over sketch

7000

2.5

6000

2

5000

1.5

4000

1

3000

0.5

2000

0 0

Sketch inversion

8000

3

#SYN

Motivations

x 10

1000 500

1000

1500 2000 2500 Time (min)

3000

3500

500

4000

1000

1500 2000 2500 Time (min)

3000

3500

Evaluation 4

3.5

Conclusions

DIP|DP:251.36.255.40:21 DIP|DP:231.117.189.150:80 DIP|DP:231.29.226.114:25 DIP|DP:13.209.95.186:81 DIP|DP:247.19.52.134:6667 DIP|DP:10.120.119.81:XXXX

6000 5000

3000 2000

2 1.5 1

1000 0

3 2.5 #Packets

#SYN

4000

x 10

0.5

500

1000

1500 2000 2500 Time (min)

3000

3500

0 0

500

1000

1500 2000 2500 Time (min)

3000

3500

4000

Anomaly Detection Mechanism for High-Speed Networks

Conclusions Proposed framework is:

Motivations Background Network Monitoring CUmulated SUM

Scalable

Can handle tens of millions of flows

Count-Min Sketch

Proposed approach Change point detection over sketch Sketch inversion

Evaluation Conclusions

Efficient

Small memory usage & small nb of memory access Decouple the recording stage of sketches from the detection stage Attacks detection & classification Software compliant Accurate & extensible

High intensity attacks can be detected quickly and accurately Provable probabilistic accuracy guarantees UDP flooding, ICMP flooding, Smurf attack, TCP Reset attack, etc.

Anomaly Detection Mechanism for High-Speed Networks

Any Question?

Motivations Background Network Monitoring CUmulated SUM Count-Min Sketch

Proposed approach Change point detection over sketch Sketch inversion

Evaluation Conclusions

Thank you!