Computer Networks 55 (2011) 1909–1920
Contents lists available at ScienceDirect
Computer Networks journal homepage: www.elsevier.com/locate/comnet
Graption: A graph-based P2P traffic classification framework for the internet backbone Marios Iliofotou a,⇑, Hyun-chul Kim b, Michalis Faloutsos a, Michael Mitzenmacher c, Prashanth Pappu d, George Varghese e a
University of California, Riverside, CA, USA Seoul National University, Gwanak-gu, Seoul, Korea c Harvard University, Cambridge, MA, USA d Conviva, Inc., San Mateo, CA, USA e University of California, San Diego, San Diego, CA, USA b
a r t i c l e
i n f o
Article history: Received 6 February 2010 Received in revised form 18 November 2010 Accepted 29 January 2011 Available online 13 February 2011 Keywords: Traffic classification Behavioral-approach Peer-to-peer Graph mining
a b s t r a c t Monitoring network traffic and classifying applications are essential functions for network administrators. Current traffic classification methods can be grouped in three categories: (a) flow-based (e.g., packet sizing/timing features), (b) payload-based, and (c) host-based. Methods from all three categories have limitations, especially when it comes to detecting new applications, and classifying traffic at the backbone. In this paper, we propose the use of Traffic Dispersion Graphs (TDGs) to remedy these limitations. Given a set of flows, a TDG is a graph with an edge between any two IP addresses that communicate; thus TDGs capture network-wide interactions. Using TDGs, we develop an application classification framework dubbed Graption (Graph-based classification). Our framework provides a systematic way to classify traffic by using information from the network-wide behavior and flow-level characteristics of Internet applications. As a proof of concept, we instantiate our framework to detect P2P traffic, and show that it can identify 90% of P2P flows with 95% accuracy in backbone traces, which are particularly challenging for other methods. Ó 2011 Elsevier B.V. All rights reserved.
1. Introduction An important task when monitoring and managing large networks is classifying flows according to the application that generates them. Such information can be utilized for network planning and design, QoS and traffic shaping, and security. In particular, detecting P2P traffic is a potentially important problem for ISPs that want to manage such traffic, and for specific groups such as the entertainment industry in legal and copyright disputes. Detecting P2P traffic also has particular interest since it represents a large portion of the Internet traffic. In fact, a resent work by Labovitz et al. [27] shows that at least ⇑ Corresponding author. Tel.: +1 951 880 3699. E-mail address:
[email protected] (M. Iliofotou). 1389-1286/$ - see front matter Ó 2011 Elsevier B.V. All rights reserved. doi:10.1016/j.comnet.2011.01.020
20% of all Inter-domain traffic on the Internet is P2P. Even though percentage wise P2P traffic is not the dominant Internet traffic contributor today, its overall traffic volume is constantly increasing. In addition, other studies using advanced deep-packet inspection show P2P traffic to be up to 70% of the overall volume in some networks [20]. Most current application classification methods can be naturally categorized according to their level of observation: payload-based signature-matching methods [30,28], flow-level statistical approaches [10,32], or host-level methods [25,40]. Each existing approach has its own pros and cons, and no single method clearly emerges as a winner. Relevant problems that need to be considered include identifying applications that are new, and thus without a known profile; operating at backbone links [26,25]; and detecting applications that intentionally alter their
1910
M. Iliofotou et al. / Computer Networks 55 (2011) 1909–1920
behavior. Flow-level and payload-based classification methods require per application training and will thus not detect P2P traffic from emerging protocols. Behavioral-host-based approaches such as BLINC [25] can detect traffic from new protocols [25], but have weak performance when applied at the backbone [26]. In addition, most tools including BLINC [25] require fine-tuning and careful selection of parameters [26]. We discuss the limitations of previous methods in more detail in Section 4. In this paper, we use the network-wide behavior of an application to assist in classifying its traffic. To model this behavior, we use graphs where each node is an IP address, and each edge represents a type of interaction between two nodes. We use the term Traffic Dispersion Graph or TDG to refer to such a graph [19]. Intuitively, with TDGs we enable the detection of network-wide behavior (e.g., highly connected graphs) that is common among P2P applications and different from other traffic (e.g., Web). While we recognize that some previous efforts [6,9] have used graphs to detect worm activity, they have not explored the full capabilities of TDGs for application classification. This paper is an extension of a workshop paper [18] and the differences will be clarified in the related work section (Section 4). We propose a classification framework, dubbed Graption (Graph-based classification), as a systematic way to combine network-wide behavior and flow-level characteristics of network applications. Graption first groups flows using flow-level features, in an unsupervised and agnostic way, i.e., without using application-specific knowledge. It then uses TDGs to classify each group of flows. As a proof of concept, we instantiate our framework and develop a P2P detection method, which we call Graption-P2P. Compared to other methods (e.g., BLINC [25]), Graption-P2P is easier to configure and requires fewer parameters. The highlights of our work can be summarized in the following points: Distinguishing between P2P and client–server TDGs. We use real-world backbone traces and derive graph theoretic metrics that can distinguish between the TDGs formed by client–server (e.g., Web) and P2P (e.g., eDonkey) applications (Section 2.2). Practical considerations for TDGs. We show that even a single backbone link contains enough information to generate TDGs that can be used to classify traffic. In addition, TDGs of the same application seem fairly consistent across time (Section 2.3). High P2P classification accuracy. Our framework instantiation (Graption-P2P) classifies 90% of P2P traffic with 95% accuracy when applied at the backbone. Such traces are particularly challenging for other methods (Section 3.2.2). Comparison with a behavioral-host-based method. Graption-P2P performs better than BLINC [25] in P2P identification at the backbone. For example, Graption-P2P identifies 95% of BitTorrent traffic while BLINC identifies only 25% (Section 3.3). Identifying the unknown. Using Graption, we identified a P2P overlay of the Slapper worm. The TDG of Slapper was never used to train our classifier. This is a promis-
ing result showing that our approach can be used to detect both known and unknown P2P applications (Section 3.4). The rest of the paper is organized as follows. In Section 2 we define TDGs, and identify TDG-based metrics that differentiate between applications. In Section 3 we present the Graption framework and our instantiation, GraptionP2P. In Section 5 we discuss various practical issues. In Section 4 we discuss related work. Finally, in Section 6 we conclude the paper. 2. Studying the TDGs of P2P applications 2.1. Traffic dispersion graphs (TDGs) Definition. Throughout this paper, we assume that packets can be grouped into flows using the standard 5-tuple {srcIP, srcPort, dstIP, dstPort, protocol}. Given a group of flows S, collected over a fixed-length time interval, we define the corresponding TDG to be a directed graph G(V, E), where the set of nodes V corresponds to the set of IP addresses in S, and there is a link (u, v) 2 E from u to v if there is a flow f 2 S between them. In this paper, we consider bidirectional flows. We define a TCP flow to start on the first packet with the SYN-flag set and the ACK-flag not set, so that the initiator and the recipient of the flow are defined for the purposes of direction. For UDP flows, direction is decided upon the first packet of the flow. Visualization examples. In Fig. 1, we show TDG examples from two different applications. In order to motivate the discussion in the rest of the paper, we show the contrast between a P2P and a client–server TDG. From the figure we see that P2P traffic forms more connected and more dense graphs compared to client–server TDGs. In Section 2.2, we show how we can translate the visual intuition of Fig. 1 into quantitative measures that can be used to classify TDGs that correspond to different applications. Data set. To study TDGs, we use three backbone traces from a Tier-1 ISP and the Abilene (Internet2) network. These traces are summarized in Table 1. All data are IP anonymized and contain traffic from both directions of the link. The TR-PAY1 and TR-PAY21 traces were collected from an OC48 link of a commercial US Tier-1 ISP at the Palo Alto Internet eXchange (PAIX). To the best of our knowledge, these are the most recent backbone traces with payload that are available to researchers by CAIDA [5]. The TR-ABIL trace is a publicly available data set collected from the Abilene (Internet2) academic network connecting Indianapolis with Kansas City. The Abilene trace consists of five randomly selected five-minute samples taken every day for one month, and covers both day and night hours as well as weekdays and weekends. Extractingground truth. We used a Payload-based Classifier (PC) to establish the ground truth of flows for the 1 The authors thank CAIDA for providing this set of traffic traces. Additional information for these traces can be found in the DatCat, Internet Measurement Data Catalog [8], indexed under the label ‘‘PAIX’’.
1911
M. Iliofotou et al. / Computer Networks 55 (2011) 1909–1920 864 1337
1335
1330
710
1323
663 1257
1338
1245
1243
1336
1331
1324
502 1032
990
987
952
1271 1258
1246
1244
272 827
374
920 1033
991
988 865
102
953 1016 828
9 711
664 58
375 1003
503 853 267 903 103 641
506 273
499 1239 993
59
854
615
1312 46 398
1342
788
1108
1240
498 862
823 10
1234
473
45 973 240 200 208
353 507 904 863
492
983
207
509
824
150 284
956 971
241
1202 1038 177
151 559
115 236
984
190
759 1039
604
1259 275 264 887
532
199 1201
237
985
336 352 587 1040
21
901
114
981
588 1041
22 266 560 1144
1343
189 338
1311
982
533 265
621
70
1077
1344 783
535 23 468 337 1308 622 992
33
1102
346 332 949 1377 649 1300 71 638
771
607
943 1141
586 752
1280
19
688
639
1120
608
1188 89
1119 270 860 536
1096
1281
20 88 1184
34 24
467
1345
126 1362 798 297
738 1283
1232
1055
175 718 687 167 17
698
557 1284
1233 1154
1163
817
1252 91
1197 583
18
677
950
1292
1036 1329
92 462
818
290 1351
298
837
958 1079
1098
1250
996 152 797 932 1293
699
775 1008
719
112
514 960 573 1178
333 577 629
450
760
558 930
259
704 574
446 1191 515
1030 424 291
205
184
13 471 951 1315
1156
441 685 1157
563 193
729
1247
113 153 655
601 433
372
271 1261
959
1225 14
997
447 440
1162
907 825
636 488
740
776
1347
829 411
204
1113 315 659 258 308 656
57 1037 444
946
1169 891 351
1213 1207 1192
1182
631 472 1305
548 454
214
911
703 1348 397
1369
819 945
76 326
940
101
1374 701 1015
1327
131
1214 1100
869 368 1078 994
1014
1052 814
523
871
119 605
1371 1208 246 963 835
1058
260
1194 90
1222 169 466 602
564
309 1319 96 369
1328
528
192
412
594
875
962 582
1364
439 1215 1019
669
1340 795
226
972
725
1095
1339 35 585
762
661
147
2 1161
1074
1376
1290
327
905
1065 233
857
620
1326
842
1195
944 933
686 299
646
26
1174 282
106
1172
786
785 1277
1
1196
896
1231 926
42
542
845
144
883
154 1047 457
1159
745
384 180 442
886
262
965
670
660
363 1238
410 456
261
1200
1262
909 626
453 913 995
1235 215 730
234
320 1012
746
100 143 280 224
347
894
107
787
882 805
417
632
1212 747 672
826
872 1370 65
321 310
1341
136
1013 566 129 5
210
1017 404 1237 1093
170
1321 77
25 572
867
1094 881 247 279
850
658 41 673
328
36 578 64
463
801
634 399 322
1220
832
1021 155 8 181 934 377
7
520
879
1266
1153
436 1114
248
438 1224
884 1365
218
360
281 633 194 556 117
999 1206
176
128
361
302
111
209 118
62 627 355 125 1025
624 301
1066 967 408 183 804
497
1204 277
244
435
792 741 341
6
356 595
319 616
110
756 171 311 1075 593 749 496
1189
491 838
613
690 257
906
715
754 816
1140
735 576 538 511
1101 979 561
592 534
580 878
252 714 731
986
1285
1128 182 512
448
493
1166
198
83
109 1147 1011
693
206
706 276
373
295
1090
49
123 964
1359
1133
897 350 429
445
188
127
423
968
469
899
81
966
201
60
87
348
1164
980 461 568
420
889
628
159
674
890
888
1092
1127
1199
836
339
666 833 160
50
531
1278
609
1103
1264
936
570
1004
799
1050
393
464
1291 340 870
477
948
161
108
239 530
28
680 648
212
162
552 376
15
856 1048
874
914
413
575
51
1325 1057
82
1282
263
562
784
465 174
975
938 1001
61
915
137
1254 1056
841
313
245
614
211
1051 668
794
124
132
789
312
970
584
619
135
331
925
165
1049 1217
1248 545
1000
773
479
947 1260
343
1251
484
149 1099 403
590
1170
597 140
1373 32
69 292 16
317
758
640
227
567
1296
893 1106
342
748
1105
1009
500
31
364
449
1353
976
1205 1137
225
86
644
1272
571 1372
230
1190
1045
618
831 1241
635
187 1303 419
1176 768
148
537 122
755
303 318
1121 191
235 732
37
158
505 232
196
40
851
432
868
1183
782
27
485
39 316
421
630
1088
598
130
716 978
723
1097
1112
344
510
1352
278
1288
84 254 392 525
418
138
1216
808
774
475
383
1148
428 1129
63
222
612
844 324
1028
791
569 1046
596
85
394
414
255
253 1089
1255
812
52 164
1273 1022 728
242
139
898
1165 1116 1333
876
1267
1322
274
1356
443
1242
354 1210
653
689
378
1059
1151
1302 487
671
405
727 43
610 974 314
679
157
813
811 1173
855
961
902 1289
955
1142
1060
268
1203
489
684
163
1018 892
217
717
1256
98
529
843
494 793
751
809
365 551
540
231
852
1031
1069 451 724
1320 1005
486
866
349
599
68
908
1186
643
722 611
1269
858
1027 806
834 637
895
753
1136
1035
654
116
796
1334
810
579 495
1185
839 554
647
650
474
937
38
504 1086
939
1064
1209
422
591
989
478
734 1061
168
508
323
516
1053
134 396 517
1279
1375 603
916 1249 1349
1218
1363
1010
78
665
846
1117
452
105
733
1316
524 606
44 589
757 1076 969
935
362 678 1171
388
395
657
1198
1135
900
1043 1301
977
565
1346
800 1275
737
283
409
195
458 1024
381 387 807
186
1287
705
700
736
172
455
750
243
1070 1134
400
391 781 97
156 623
1054
1152
1044
145 73
1228 830
1350
790
1317
345
1085
642
720 166
1229
146 675
173
1081
1236 406
1354 847
1118
1087 539
1160 269
1274 104
815 216 555 402
1355
1268 1263 221 1297
676
820 401 1304
12 80
1062
803
694
726 848 721 617 849 1357
553
1313
927 1143
928
219 859 885
1270
121
1020 1042
922
1265 1080 99 778 29 840 300
238
780 1358
1126
1314 645 3
72
366 1286
1181 921 1219
223 861 779 600 998 79 1091 1332 220 702
407
581
434 912
821 667 1223 476 954
95
289
367
1115 917 482
1193
120 213
11
1067
761 1158
1227
1187
54 513 1107
1023
1211 652
877 185 359 697 802
1175 1130
133 1063 880
251
777
910
873 1253
931 480 30 197
67 66 929 695 1068 4
1155 481 1221 1072
53 625
772
483 1029 459 1007
696
142
1073
1026
770
1034
56
293 1082
763
460
651 1083 681
712 1368 742
74
430
490
1071
1104 1226 1084
1002
713 427 437 329
47 1006
93
707 296
1230 141 75 1109 1318 470 1110
325 522
382 822
739
957 541
256 431 501 94 521 769 55 294
1111
330
1360
249
48
285
287
178
202
228
1179
1177 1276 334
1361
250
304
286
306
288
370
179
203
229
379
1180
357
335
518
305
766
307
662
526
385
389
425
416
371
380
358
682
691
519
543
546
767
549
527
386
918
390
923
708
683
692
1122
1124
544
547
426
743
764
550
1131
941
919
924
709
1294
1138
1123
1125
1132
1145
1149
744
765
1167
415
942
1366
1298
1295
1139
1146
1150
1306
1309
1168
1367
1299
1307
1310
723
744
872 810
547
873
599
601
710
743
772
774
779
722
871 542
598
267
270
280
600
272
284
638
442
444
731
159
113
855
760
709
1001
310
452
326
50
188
375
266
269
271
283
868
242
143
499
279
203 666
866
649 286
205
1041
118
438
674 467
362
703 660
183
162
290
909
841
867
384
538
436
275
204
1040
437
466
210
374
702
386
537 752
632
707
116
235 350
979
808
318
612
951
441 193 229
901
355
634
751
530
977
1035 181
701
981
289
167
201
176
202
228
861 428
900 334
678
241
579
1005
14
963
768
1008
260
191
427
724
800
816
166
539
529 259
470
8
518
994
357
980 257 985
742
517
258
860
778
741
885
754
976
694
424 753
153
305
200
975
693
423
335
63 501
152
304
548
249
814
151
513 196
105
248
1034
164
736 512 69
195
149
973 735 745
68
80
103
333
972
803
687
477 15
680
476
592
551
806
232
173 950
936
393
591 361
312
231
172
805
716
595
683
158
413
120
90
970
324
682
214
433
38
157
212 78
77
75
293
628
968 676
630
584
581 625
623
573
627
211
508
506
407
302 491
495
401
298 482
489
399
396
391
382
475
621
569
669
665
663
967
406
400
398
395
390
381
507
505
494
490
488
481
474
471
668
662
965
959
956 1042
952
947
941 1038
929 1030
926 1027
1019
922
848
801
664
1043
629
966
960
583
580 957 624
953
1039
622
948
942
572 930 1031
620
655
1028
1020
618
923
927
568
566
849
462
802
378
376
351
656
916
619
567
705
472
463
377
379
450
706
352
920
755
448
917
969
408
303
653 299
451
750
640
371
76
294
449
677
586
739
884
794
37
1015
48
914
27
852
832
49
642
464
502
128
925
28
469
840
409
720
155
594
522
372
430
915
83
607
643
585
434
938
215
422
523
608
129
611
325
108
431
156
410
96
504
788
1036
681
842
243
23
121
414
730
1018
540
91
84
141
208
184
837
60
783
109
394
142
263
924
222
875
609
92
220
983
571 268
20
870
715
5
971
300
24
1000
185
264
838
180
230
658
146
605
217
388
163
250
839 492
606
93
1033
207
21
320
6
301
179
418
626
815
122
65
902
1044
245
147
79
276
186
244
419
688
493
826
251
1003
81
29
425
988
718
311
906
189
321
25
39
426
66
526
339
746
935
992
94
699
420
11
30
893
140
54
402
637
169
82
59
907
812
58
26
125
40
9
403
857
690
712
35
903
329
691
262
95
12
974
561
55
785
150
177
738
556
555
858
879
10
828
261
255
727
36
904
937
106
692
161
459
199
525
535
342
897
524
766
224
178
859
747
792
343
256
509
831
1023
174
869
356
265
288
863
910
761
557
281
711
160
898
943
670
496
322
986
536
61
225
758
138
932
949
570
767
833
273
171
383
175
139
813
64
416
763
596
795
777
487
62
336
51
34
757
856
18
1
221
918
516
853
587
98
19
2
834
360
331
695
52
854
851
890
415
114
989
380
16
198
696
498
520
846
252
197
883
239
503
112
85
33
218
7
22
209
1048
99
392
510
369
945
737
330
528
238
562
115
990
667
44
165
933
511
588
749
689
104
604
253
483
661
934
697
89
439
17
219
748
708
908
1013
654
534
370
206
119
190
86
137
421
295
636
123
721
337
233
223
45
835
798
962
13
456
457
770
519
552
338
995
698
1024
314
756
527
358
110
987
1014
274
315
728
136
836
650
134
700
955
282
455
297
124
254
582
1016
593
765
148
521
799
684
359
500
130
216
484
323
645
73
479
850
111
719
465
192
1046
468
944
829
131
100
226
1045
543
67
87
440
411
135
545
675
905
1025
964
940
306
349
560
234
610
53
107
590
317
43
740
1047
984
862
47
313
732
865
1017 227
480
776
74
497
1029
576
344
713
807
830
790
672
285
991
679
88
704
46
899
577
389
1021
345
993
316
563
615
485
921
939
824
194
213
671
405
657
844
101
659
1026
277
954
635
822
56
1032
307
631
478
958
614
42
404
102
417
843
132
364
72
363
809
931
97
429
240
319
328
821
825
117
432
978
126
602
894
685
154
57
182
603
308
597
791
144
133
575
473
804
845
847
514
3
170
784
4
486
31
332
771
385
458
864
127
946
70
895
913
41
348
531
187
435
32
928
544
982
309
168
823
71
387
880
686
714
145
633
1049
412
532
717
278
574
578
613
373
644
764
1004
648
769
397
876
515
541
827
236
246
911
533
896
759
291
762
1002
729
641
340
346
912
793
919
589
1022
296
961
353
365
367
891
789
1037
673
446
546
237
453
460
549
553
558
247
616
287
292
646
327
725
733 341
347 651
564
773
775
780 781
443
445
447
454
786
354
366
368
892
796
461
817
550
877
881 554
886
819
559
565
888
617
996
639
998
647
726
1006
1009
1011
652
734
782
811
874
997
999
1007
1010
787
797
818
878
882
887
820
889
1012
(a) The Gnutella TDG (P2P application).
(b) The HTTPS TDG (client-server application). Fig. 1. Two TDG visualization contrasting a P2P (top) with a client–server application (bottom). Largest component is with bold (black) edges – hence the graphs are best viewed on a computer screen or a colored print-out. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
Table 1 Set of backbone traces from the Cooperative Association for Internet Data Analysis (CAIDA). Statistics for the TR-ABIL trace, are reported only for the first five-minute interval since IPs were anonymized differently at each five-minute sample. Name
Date/time
Duration
Flows
TR-PAY1 TR-PAY2 TR-ABIL
2004-04-21/17:59 2004-04-21/19:00 2002-09/(N/A)
1h 1h 1 month
38,808,604 37,612,752 2,057,729
TR-PAY1 and TR-PAY2 traces. Both traces contain up to 16 bytes of payload in each packet, thereby allowing the labeling of flows using the signature matching techniques described in [26,25]. Running the PC over the TR-PAY1 and TR-PAY2 traces we find 14% of the traffic to be P2P, 27% Web, 7% DNS, and the rest to belong to other applications. A detailed application breakdown is summarized in Table 2. In the table we further report the traffic information for the top six P2P applications with the highest
1912
M. Iliofotou et al. / Computer Networks 55 (2011) 1909–1920
Table 2 Application breakdown for TR-PAY1 and TR-PAY2 traces. Values in parenthesis show the percentage of each P2P application over the entire identified P2P traffic (All-P2P). Name
% in flows
% in bytes
% in packets
Gnutella eDonkey FastTrack Soribada MP2P BitTorrent All-P2P Web (80/443) SMTP DNS Games Unknown No-Payload Rest
0.95(6.78) 2.96(21.16) 0.55(3.92) 7.76(55.44) 0.41(2.93) 0.60(4.26) 13.85 27.45 1.65 6.65 0.71 0.97 28.06 20.66
0.17(1.59) 2.22(21.17) 0.74(7.10) 0.07(0.63) 0.01(0.14) 4.59(43.81) 9.19 41.99 1.10 0.32 0.54 1.98 0.46 44.42
0.81(6.32) 2.84(22.21) 0.97(7.61) 0.97(7.63) 0.07(0.53) 4.37(34.24) 12.10 35.51 2.431 1.586 2.84 3.22 6.04 36.27
number of flows in our data set. These six applications contribute 95% of the flows and 75% of the overall P2P bytes. In our traces, we also have flows by P2P applications that are not longer in use, such as WinMX, Direct Connect, Goboogy, Ares, SoulSeek, and EarthStation. We observed that some of these protocols were used for transferring very large files and contributed many bytes, but their overall number of flows was significantly lower. In our evaluation we include all P2P flows. For our study, we remove the 1% of traffic that remained unknown and the 28% that contained no payload. The small number of unknown flows is strongly correlated with the small number of encrypted flows in our traces. 2.2. Identifying P2P TDGs Identifying the right metrics to compare graph structures is a challenging question that arises in many disciplines [31]. Our approach is to consider several graph metrics, each capturing a potentially useful characteristic, until a set of metrics is found that distinguishes the target graphs. To select an appropriate set of metrics, we generate a large number of TDGs using all our traces (Table 1), thus observing TDGs over two different locations at the backbone. For the TR-PAY1 and TR-PAY2 traces, we use the payload-based classifier (PC) in order to select which flows belong to each TDG. Since the TR-ABIL trace does not have any payload information, we use port numbers [26] to assign flows to applications. We can use port numbers for the TR-ABIL trace since it was collected in 2002 where most P2P applications used their default port numbers [13,23]. We only use the TR-ABIL trace to verify our TDG observations over a second location in the backbone and we do not use it in the final evaluation of our classifier. By using the month-long TR-ABIL trace, we can study the consistency of TDGs over different times of the day and over weekdays and weekends. We observe TDGs over five-minute intervals. This interval length gives good classification results and stability of TDG metrics over time as we show later in this section. For each TDG we generate a diverse set of metrics. Our
metrics capture various aspects of TDGs including the degree distribution, degree correlations, connected components, and distance distribution. For additional details about these metrics we refer the reader to [19,31]. To select the right set of metrics we use various graph visualizations and trial and error. Finding a less ad hoc approach is beyond the scope of this work. Two TDG visualization examples are shown in Fig. 1. We see that FastTrack (P2P) has a denser graph than HTTPS, or a higher average is given by degree, where the average node degree k k ¼ 2jEj=jVj. We utilize two other metrics that capture the directionality of the edges in the graph and the distances between nodes. The directionality is useful since we know that pure clients only initiate traffic, pure servers should never initiate traffic, and that some P2P nodes play both roles. To capture this quantitatively, we define InO to be the percentage of nodes in the graph that have both incoming and outgoing edges. The distance between two nodes is defined as the length of their shortest path in the graph. The diameter of a graph is defined as the maximum distance between all pairs of nodes, which is sensitive as a metric [31]. For a more robust metric, we use the effective diameter (EDiam), which we define as the 90-th percentile of all pairwise distances in the graph. We show the high graph diameter of the BitTorrent P2P TDGs in Fig. 2. The graph represents the BitTorrent interactions between 3000 nodes from the TR-PAY1 trace. The visualization is suggestive of a spider’s web with large paths between nodes at opposite edges. This intuition is precisely captured by the large value of the effective graph diameter. In general TDG visualizations are suggestive, but the metrics make the intuitions precise and allow classification. From our measurements, we empirically derive the fol > 2:8 lowing two rules for detecting P2P activity. Rule 1: k and InO > 1%; Rule 2: InO > 1% and EDiam > 11. With these simple rules, we can correctly identify all P2P TDGs from both backbone locations (Abilene backbone and Tier-1 ISP). Intuitively, P2P hosts need to be connected with a large set of peers in order to perform tasks such as answering content queries and sharing files, which can explain the higher average degree compared to client–server applications. An additional characteristic of P2P applications is the duality of roles, with many hosts acting both as client and server. The duality of roles is in turn captured by the high InO value. We further speculate that the decentralized architecture of some P2P applications (such as BitTorrent), can explain the high diameters in some P2P TDGs. We do not claim that our thresholds are universal, but our measurements suggest that small adjustments to these simple parameters allow our methodology to work on different backbone links. Distinguishing collaborative applications from P2P: Some well-known applications other than P2P exhibit collaborative behavior, such as DNS and SMTP. This is not surprising since in these applications servers communicate with each and servers act both other and with other clients (high k), as clients and servers (high InO). This is exactly what our metrics are set out to detect. Next, we outline two
1913
M. Iliofotou et al. / Computer Networks 55 (2011) 1909–1920 2042
1906
2885
2060
3213
3654
2706
1498
3512
3227
3563
3006
3646
3174
2066
3511
1243
3501
3498
3497
3490
3489
3487
3486
2050
2620
2614
2613
515
3483
1001
3482
3473
2592
2591
3472
3467
3466
3040
2043 39 2011
2953
2238
2237
2949
2948
2947
2946
2276
2274
2631
1592
2211
496
980
1331
1757
2630
1782
2593
1278
2590
2589
2588
3217
1799 846
3294
1737
2227
2788
1347
3386
1156
3263
516 2130
741
2975
2974
2983
2982
704
2116
2115
1708
1707
2586
3340
2743
3289
270
2529
2212
1941
2585
857 38
1923
2918 362
2275
1884
1783
1114
1738
688
1876
2242
2391
2124
2677
2629
3240
140 1064 1348
1293
1279
1157
1676
1675
2633
838
2815
2579
2578
2379
3038
742
705 839 956
2813
1504
180
2122 1169
1281
136
88
1924
2173 2990
2989
363
2676
1054
2628
271
1636
1546
1115
2028
2109
1414
385
1705
1305 1200
723
141
1394
726
3427
1393
2026
1065
3008
2110
2700 957
1647
26
1375
932
3007
1168
3454
25
622
3453
1908
1280
3471 1306
3255
771
1646
3525
137
1075
1275
3378
722
1055
87
89
1127 1342 808
131
52
1128 384
1407
3010
3009
2608
3448
3447
124
809
3371
2323
2781 1467 2966
3298 1890
2727
53
1744
1704
404 2680
3146
1044 1363
3449
1814 2504 2351
3530
2731
3544
3543
3551
3550
1340
1339
1299
62
3444
2941
2123
2624
3443
2047 2715
1191 2157
1408
132
2132
853 3606 1059 856 155
3587 2716 3162
1028
396
401
3436
3435
3018
637 293
1349 1188
2695
520
1639
1406 2182 2315
434
1259
1629
3313 2147
2693
2718 146
973 3417
854
1082
130
1686
2574
2573
2559
2558
1007 3576
2065
1677
397
2104
156
519
1638
1058 855
1390 982
145
3228
1107
255 3570
892
3569
2846
1173
1172
1165
1164
294 2454
121
2004
400
2155
1102
518
2399
654
46
1100 1049
2770
1048
1443 1537 1006
357 3577
1953
129
2064
1106
2236
2235
1389 587
58
504 3492
1310
2270 221
1118 1770
28
24
47 3390
517
419 1514 254 1557 1921 2400
3375
1927
274
1180
1125
655
1672 1518
2322 2005
2836
2778
967
29 405
59
2051 220
815
2094
122
543
2534
2533
23
588
356
1062
1266
361
111
1886
225
1311 2370
2250
2249
1309
1070
27
930
163
2168 966 1485 803 1061
1060
2059
923
922
2015
275
3613
1440
2523
816
3176
1928
2643
406
2881
914
224
2522
444 1124
773
1004
832 2644
913 944
273
1490
2450
473 534 288
2734
1267
772
888
2515
2514
452 112
2239
287
1286
1897
3395
1784
1618
3585
3584
3593
3592
868
867
185
453
2366 802
3309
1690
309
1095 3178 998
2079
3437
1529 2344
833 1005
1530
2088
2087
2082
2081
533
1962 770 2823 3214
3002
615
700
3080 992
3072
3419
1473
1288
3133
1815
1645
2785
1477 3468
2616
756 3337
985
984
1508
3539
663
987 747
7
11
3179
776
465
1777
454
310
1538
3224
480 2220
3601
3600
3362
3015 2474
474
1596 3187
1591
481
68
8
1336
1619 2747 3132 989
988
799
699
798
286
2615
2935
1533
2926
2934
769 69 2744
738
2849
3225
761
760
2933
2610
451 234
633 3215
12 1610
2401
455
1507
662
240
1388
1899
1813
1534
2925
1462 1867
324 3014
1476
2959
505
1620
3123
2075 678
285
1595
677
1594
2705
680
3252
2845
2455
2748
2447 1033
1632
679 2177
960 2061
2062
3609
2353
3612
3611
682
1480
1898
573
681
3231
1812
1811
1785 3614
1881
447
450 1896
3124
3145 2780
634
1902
262
844
843
1249
1248
3045 1727
1739
2494
942
601 2213 238
1034 2980
1680
1631
337
442
3186
2503
2502
2021
599 2449
560 559
2174
3098
3272
2348 862
2567
472 3586
532
2311
1669
1422
2283
1901 380
1719 2564
2726
602
381
1959
2510
551 443
1588
421
600
572
598
566
565
1671
1670
3404
2001
1531 1766
1853
2437
1341
1585 1681 3005 531
1587
298
647
420
2464
1710 2264
471 2428
2957 861
782
547
546
1874
2057
417
2056
3567
2438
1578
959
2623 18
91
1987
931
1579
3588 2565
552
438
437
399
398
3171
754
753
529 2855
1523 1586
378
2789
1765
1141
2427
2928
2927
379 17 376
2263
3517 2958 3023
3022
3025
3024
2302
1939
1088
648
1087
2930
1709
521
997
716
902
1558
3655
2657
2754
236
958 1417
90
596
595
1418
841
330
840
1789
1788
1780
1779
703
1424
1622
1319
483
329
970
429
969
237
2810 3109
1412 996
1612 430
545 748
544
1758
1327
377
3573
1433
3027
1067 667
1303
3026
292
2746
291
1068
1506 2921
811
968
3622 1320 1261
1260
3523
284
283
1161
3403
462 2825
2092
2136
3222 2472 1520 2884 113
3616
1016
3615 3496
2225 1519
3459
730 216
1977 1469 1295
1326
577
594
1434
249
248
245
244
3402
3401
3011
3583
2073
3241
2317
1826 2462
3481
1017
463
983 3122
2659
609
606
448 578
829
501
1186
2178
2113
1752
3383
3575
3382
338
1404
114 881
217
1415
1405
593
339
910
1296
909
1236
1235
1743 2156
1742
493
2252
459
500
3623
2251
135
134
3251
340
214
3380
1936
714
2690
3477
2330
2074
513
3112
2681 1177
715
3147
610
1751 828 154
95
3242
2542
528
476 2669
2668
2671
2670
870
1864
3282
2486
2485
2568 630 1869
241 403
898 2689
458
3148
875 1847
402
876
104 2720 3306
3568
1427
514
642
2496
96 477 103 2463 1277
153
1940
1051
2814
1868
2831
641
1193
920
2911
2910
2900
2837
2902
2901
64
2916 2152
2673
2267
2672
3157
3518
2868
3041
3548
1593 495
762
3653
1964 2044
1197
200 1750
2188 2291
1192
3432
2863 2685
2189
1491
2686
35
34
1163
101
1945
583
389 2571
1134 1590
2398
2032
494
206 2151 431
921
1946
2646 1094
2490 65
2554
3029
3028
2443
763
3344
2430
1492
3470
1354
1135
1572 1411
3199
817
308
307
3149
1699
2229
2228
1252
388 439
2117
584
345 1423 1225
3627
3626
3629
3628
3633
3632
3643
3642
1589
6
2745
3032
205
2397
1015 207
3158 109 60 1152
3374
2327
1580
2484
1952
2483
2300 905
971
1050
1573
2808
5
2246
2497
110
1345
1344
1253 684
3167
346
818
3336
2161 344 3269
31
2923
1877 3475
3479
800
3059
3617
3311
2476
2475
1276 1184 1358
157 2306
1297
183
2804 1176
1513
208
139
3594
159
2165
1653
1895
61
1382
706
333
3634
10
638
176
2069 3161
30
1762
972
639
1761
1047 1982 3508 3491
3
2728
2114
181 2736
3638
175 2260
460
1460 1181
57
508
1468
1359
158
3182
2971
751
1195
935
1911
934
1495
1823
1494
3076 1425
807
3610 707 1493
160
1912 3619
2894
2893
2699
1410
9
1461 138
1346 530
1983
56
1111
1957 946 182
1372
464
1455
2126
3097
507 2030
2662
71
2247
3192
2018
2331 1196
2208
965
280
964
2888
3141
2887
813 1024 1022
243
2835
1409
392
736
1837 70
1233
1797 659
737 607
281
393 394
812
2688 3637 801 2107
2709
2883
1037
2108
2691
824 242
2684
2226
806
1160
608 1201
2793
2809
278
2995 2505
860
1845
152
2048 1401
1083 1768
2943 3229
1958
1512
3207 1846 1505
2800
695
694
1202
2872
2871
1723
1023
1822 2932
825
1657 1240
440 395
3560
2742
1833
2193
1817
1254
2363 1428
2905
2904
797 2759
3042
260
629
1973
628
1665
2223
2222
1605 2599
1693
1661
1445
3183 408 3636
1194
1606 2507
2801
1656 1767 167 2245 2392
1241
1875
2072
2459 1444
3305
1913
994
3421 261 2240
1521 1140
1722
2040
2039
1660
3134
2192 1972
3343 3036
3035
251
250
20
19
1522
168
871
1432 2886
1026
2506
796
1511
661
1996 1368
691 872
470
2493
2358 1703
314
2896
3259
1384
1234
1698
2149
355
539
2880 2917 2456
424
2460
187
186
98
97
3377
2167
3376
2641 491
3356
3064
625
1215 118 3087
669
668
467
1273
926
2261
466
259
268
2687
232
1844
1397
624
2280
2617
1745 3519
3253 2694
835
1697
1999
1112
354
605
538
3349
192
1385 40
1145 690
418
1992
2994
1231
692
2762
1222
1232
1221
2148
660
585
3220
1900 537
536
1270
2618
3405
3529
1966
2224
1604
1730 3218
3052
41
3051
1204
3347
2906
2664
233
1849
2769 1313
2248 2217 269
1000 2761
2478 94
1426
2090 117
1076
2929
2461
258
2553 877
15 3054
2203
3314 341
3504
3055
561
211
2531
37
2204
1220 1109
1010
2287
2663
2730
3117
2433
586
3221
2469
1827
744
315
2184 999
213
335
2798
368
1717
100
99 449
2183
1891
360
570
3115
50
953 2372 2290 1398
2704
2703
556
2076
555 3335 2329
1085
869
1623
963
2457
651
1778
1413 1144
698
312
711
710
1547 2944
3334
3333
2991
342
36
3393
120
1903
3070
1219
2083
209
3118
3173 1294 2314
1861
548
353
1644
3431 210
2848 977 2920
1441
306
1947
1892
743
2350 1532 3058
3057
256
2384 334 3101
2230
2000 383
391
426
1624 2721
44
2162
2601 890
1482
3455
562
93
569
708
1380
371
370
1466
1625
1465
2436
2435
1621 3571
390 794
479
2826 1330
3061
1948
51
1086 2919
904
3003
2757
305
228
1338 3502
3060
903
457
673
3033
382
2067
1383 3039
2717
576
2912
162
119
190
2285
2442
2570
456
229
3412
1626
189
2052
2181
952
2625
3439
13
195 2640
2931 203
2328
161
1
2063
1041
2143
1942
945 672
3063
3062
3067
3066
580
874
2636
372
1198
3327
3326
478
823
1133
246
2842
204 1329
851
822
522
1092
720
1679 1373
3169
2070
1949
226
3170
883
194
2652
3414
1502 804
746
745
1652
3318
943
414
427
2760 267
191
3086
3528
3082
1039
884
2879
2281
1040
3325
3324
721 579 1206
374 525
14
247 1484
92 3111
597
2468
2146
1244
74
1374
2416 125
2576
2889 2091
1283
413
266
375 1561
2550
54
827
826
1635
171
1746
144
1855
1986 2071
3545
1205 1678
805
2577
852
373
73
1187
3323
3322
301
2185
2272
75
1247
1860
1963
1535
116 895 492
2389
1716
986
3034
1314
2725 2390
343
2013
3056
302
1772
784 3400
3652
612
3596
611
674
2552
3281
3075
2415
1673
143 1781
468
2772
2419
2361
133 3151
2867
1798
1854 115 1179
2499
2129
3250
759
3194
671
3461
3065
2807
2362 1078
1178
1701
55
1369
750
147
1486
764 1370
2420
3495
2258
482 2940 1282
2937
670
1077 1155
178
1149 3233
2936
1158
718
1526
1791
783
2031 820
1648
810 1429
2862
222
780
1143
2584 845
2811
2861
2277
1302
1568
2166
1096
3004
148
1920
223 786
387 2833
2445
2575
1536
469
2619
3053
1599
1497
3206
785
1362 788 732
3540
1081
2903
749
2068
1436
3360
3268
2465
1614 2954 2771 765
758
656
2517
1159 719 2383 618
2674
1148
617
1848
1142 2365
1696
1976
2207 1470
202
2278
177 366
2812
2086
787
2446
3285
169
2913 1008
2100
2956
1021
766
2543
1119 1794
2945
563
1792
2555
1121
650 2029
3205
735
791
734
1069 675
1437 165
657 1615
1642
2832 367
1079
2859
2714
2713
2241
1773
1032 386 1598
2145
821
3155
1002
1056
3321
3320
3301
729
2431
1353
1151 1459
1666
2914
3410 2284
2963
1667
1052
166
1029
961
1117
142
1150
2012
3413 3463
2190
3181
3074
3073
564
859
858
1663
2095
1419
3267
1392
410 3140
2422
2797
127 1333
792
649
2796
2214
2096
676
2084
3566
757 1515
911 1131
1263
1053
1379
2765 3532
550
1655
962 2257
1080
1400
2256
3096
409
1003
2432
2596
1019
1018
2023
2022
1332 2163
2572 2560 350
2035 3649
850
170
849 2877
1352 2221
511 1479
1147
2508
2041
1458 2675
3079
3078
3077
1350
3089
3088
3091
3090
1020
1285
1284
2852
1167
2851
1933
793
1689
2305 2847
2219
1643
948
524
1961 3331
2262
303
2875
2135
2547 1378
697
2444
1421
1230
2556
2144
1654 78
2279
3513
3257
2683
128
2367
3630 2255
1207
1300
2045
3308
3139
2563
2834
2034
1301
2639 349
3484 3016
3017 1171
3232
549
557
3291
1123
512 289
3526
2298
696
523
2137 1613
1229
603
1166 263
1965
2418
2417
1843
2412
2404
2403
1878
2544
1255
2737
2792
2539 2583
900 3507
908
3372
941 2600
940
2259
1775
635
1218 2876 2180
2053
2254
2289
3128
929
928
955
954
317
2480
1925 1979
1544
1728 3142
1031
950 1356
322
150
894
3416 1915
3527
2099
1994
1035
2985 1971 778
3127
43 32
2312 901
3180
1499
3254
42 1036
81
1748 2054
276
2355
1694
2866 318
3049
321 3188 1909
277 2612
80
423
1980
1132
951
2017
2016
2978 1804
2702
1183
3172
3303
1873
2282 1483 33 1803
3354
1138
2102
422
2434
1545
2858
2609
1387 2191 896
1209
1208
3193
149
2206
2080
2205
1889
1471
3476 2645
1105
3346
1664
3559
949 82
2635
3384
1747
1214
1182
1093
3595 107
2101 1449 2821
1937 2951 108
2532
1386
1269
1268 3524
2528 2724 2243
2304
1754 3631 702
3302 2458
3236
2557
2338
2196
1517
3288
3287
252
2414
1116
219
574
83 174
978
2105
1013
1879
1103 2197
2537 1850
2648 3366
173 2952
701
1753
979
1014
1104
1308
1307
3280
3279
1720 1706
2402
2332
590
2869
2696
621 3012 1721
218 1258
1692 2118
3297
3154
1323
3635 866
939
937
1829 3105
3104
3505 3500
1315
3648
2660
2265
2860
575
3278
3277
199
1830
1265
296
264
2538
589
620
2511
3189
3415
645
789
3203
2530
2753
2752
2764
2763
1760
2121
865
2170
938
2308
936
3013
2394
2393
2388
2387
731
687
1700
3130
3565 915
1396
1395
1403
924
1402
1431
1430
1447
1446
1628
646
1223
790
2964
1224
1553
1602
265
1759
2120
3350
1464
1463
1324
2169
2307
1316
2967 2055 3129
1819
1541
1540
1543
1542
3564
1818
1551
1550
1556
1555
1563
916
925
1712
1562
1802
1711
1584
1488
2965
1583
1582
1581
3204
1597
990
1608
1607
1634
1633
1641
1487
1640
1650
1649
2200
2199
709 554
327 885 1825
1810
1824
1840
1839
1866
1865
1872
1871
2319
2318
2321
567
2325
2324
1554
3160
3159
3164
3163
831
3168
1888
2119
2139
2138
2160
1887
1603
1715
2159
2968
1917
1714
1732
1916
1919
1918
1930
1929
1932
1931
1956
1955
1968
1967
1989
1687
1988
1998
1997
2008
2007
2818
2817
2899
2266
1805
2310
2309
2175
2791
2790
2195
2194
2202
2201
1926
428
2767
2176
2766
2774
2773
2787
2786
2334
2333
3191
3190
2336
2335
2024
2025
2347
2346
2360
2359
2369
2368
2375
2374
2378
2377
2382
2381
2386
2385
1435 553
328 3138
3137
3144
3143
499
1943
2795
2794
1617
2802
412
3197
3196
3202
3201
3212
3211
3223
2999
3244
3243
3247
3246
3261
3260
3271
3270
3275
3274
2602 2605
411
2603
Fig. 2. Graphical representation of the BitTorrent TDG with 3000 IPs, showing the formation of long paths connecting P2P hosts. The largest connected component is highlighted with darker color edges.
2.3. Practical considerations for TDGs 2.3.1. Stability of metrics over time If thresholds derived from TDGs changed significantly with time, then the classifier must be trained constantly and that would detract from its value. Fortunately this does not appear to be the case. We show the stability of TDGs in time and space using traces from different points in time at two different backbone locations for Abilene and a commercial Tier 1 ISP (Section 2.2).
To test stability over time, we measured P2P TDGs from our longest trace (TR-ABIL) that span over one month and include samples taken over day and night hours, and both weekdays and weekends. To the best of our knowledge such month-long traces with payload are not available. Stability was also observed in all the one-hour long traces (TR-PAY1, TR-PAY2). Fig. 3 summarizes the stability study for TR-ABIL showing the average degree metric. As we see, average degree takes values in a small range over the entire month. Soribada tends to have very small graphs ranging from 30 to 1000 nodes and hence the higher variability. Stability was also observed for the diameter and InO metric with the figures not included for brevity. For this study, we used TDGs representing five-minutes of traffic. In Section 2.3.2
Average Degree
approaches that can help to separate between collaborative applications (e.g., DNS) and P2P: (1) One solution is to rely on port numbers or payload information for this task. In fact, it has been reported recently [26] that port numbers are fairly accurate in identifying such legacy applications, although they fail to identify P2P and other applications with dynamic use of port numbers. Therefore, one could use legacy ports to pinpoint and isolate such collaborative applications and then use graph metrics on the remaining traffic. In addition to port inspection, we can also examine the payload of a flow in order to verify that it follows the expected application-layer interactions. (2) In a recent work on dynamic graphs [16], we show that measuring how much TDGs change over time gives additional information about the nature of the applications. In particular, we observed that P2P TDGs are more dynamic (i.e., how frequently nodes and edges change over time) than legacy applications, such as DNS. If the classification problem at hand requires this further separation, we can incorporate such dynamic techniques in Graption.
30 28 26 24 22 20 18 16 14 12 10 8 6 4 2 0
min = 23.73
0
Gnutella FastTrack Soribada
WinMX
eDonkey
MP2P
Fig. 3. The average degree for various P2P protocols over one month in trace TR-ABIL. Candle sticks show the maximum and minimum recorded values together with the average (horizontal line) and ± the standard deviation. For visualization purposes, for the MP2P application we show only the minimum value which is very high (23.73).
1914
M. Iliofotou et al. / Computer Networks 55 (2011) 1909–1920
we show experiments with TDGs where we vary the observation interval from five seconds to 15 min. Note that BitTorrent is not shown in Fig. 3 since it was not well-known in 2002 and resulted only in a handful of flows in the TRABIL trace. 2.3.2. Selecting the interval of observation The effect of the observation interval on the graph metrics. We observe that by increasing the observation interval, graph metrics are more stable across successive TDGs of an application. To show this, we measure the variation of all our graph metrics over time as a function of the interval of observation. We vary the interval from one second up to 15 min. For each interval length, we generate consecutive graph snapshots with non-overlapping time intervals. For each interval, we extract the graph and calculate the average degree, InO, and effective diameter metrics. We examine the variability of each metric over all intervals of observation. We use the commonly-used range metric, which is the difference between the maximum and minimum observed values over all intervals for each graph metric. For each metric, we then normalize the range by the average value calculated over the time series. This normalization makes the range for different metrics somewhat comparable. For each application, we report the average range over all three graph metrics, and we plot this average range versus the duration of the observation interval in Fig. 4. Our experiments showed very high range values for intervals smaller than 60 s. The range shows significant decrease for intervals larger than a minute. Increasing the interval from five minutes to 10 and 15 does not significantly effect the range. This trend is shown in Fig. 4, where we report the average range over all metrics for a set of our applications. Even though we have variability in our results, there was a clear trend in all our measurements. Similar trends we observed in all our traces (Table 1). For the rest of our study, we use five minute intervals. 3. The Graption framework The Graption traffic classification framework consists of the following three steps. 2.5
DNS EDONKEY FASTRACK GNUTELLA HTTP NTP POP3 SMTP MP2P
Variation Range
2
1.5
1
0.5
0
0
100
200
300
400
500
600
700
800
900
Interval Length (sec)
Fig. 4. The effect of changing the interval of observation for TDGs ranging from five seconds up to 15 min over a large set of protocols. To reduce variability we can choose to use longer intervals. After 300 s the reduction in variability is not significant.
Step 1: Flow isolation. The input is network traffic in the form of flows as defined in Section 2. The goal of this first optional step is to utilize external information to isolate any flows that can already be classified. This knowledge could be based on payload signatures, port numbers, or IP address (e.g., exclude flows from a particular domain such as google.com). Step 2: Flow grouping. We use similarity at the flow and packet level to group flows. The definition of similarity is flexible in our proposed methodology. We can use flow statistics (duration, packet sizes, etc.) or payload if this is available. Eventually, the output of this step is a set of groups with each group ideally containing flows from a single application (e.g., Gnutella, NTP, etc.). However, at this step, the exact application of each group is not known. Step 3: Group classifier. For each group of flows, we construct a TDG. Next, we quantify each TDG using various metrics. The classifier uses these metrics to identify the application for each group of flows. For the classification decision, we use a set of rules which in general depend on the focus of the study. Although this paper focuses on P2P detection, we believe Graption can be used for general application classification by choosing metrics and parameters appropriately. We next describe how we specialized Graption to detect P2P traffic (Graption-P2P).
3.1. Implementation details of Graption-P2P Step 1: This is an optional step in our methodology. Experiments without this step are discussed later in the section. Recent work [26] suggests that port-based classification works very well for legacy applications, as legacy applications use their default ports and tunneling of P2P at such ports is not very common. Thus, in this study, we isolated flows with port 80 for Web, port 53 for DNS, and port 25 for SMTP. These applications turn out to be about 65% of the total number of flows. In our traces, the proportion of P2P actually using one of these ports is as low as 0.1%. Step 2: To implement flow grouping we use the fact that application-level headers are likely to recur across flows from the same application. Therefore, payload similarity can be used to group flows. In Graption-P2P, we only use the first sixteen bytes from each flow. As we show, sixteen bytes are sufficient to give very good classification results. This observation agrees with findings in [30,28]. Even though we use the payload bytes, our grouping is agnostic to application semantics, as each byte is considered as a single independent categorical feature. We consider each byte as a single categorical feature in the range {0, 1, . . ., 255}. The flow grouping step comprises two sub-steps: cluster formation and cluster merging.
1915
M. Iliofotou et al. / Computer Networks 55 (2011) 1909–1920
Step 3: The outcome of the previous step is a set of groups of flows, with each group consisting of flows that we hope stem from a single application. In order to classify each group, we generate a TDG on the group in the same way as described in Section 2. Each group yields a TDG that can be summarized using graph metrics. To identify P2P TDGs, we used the rules extracted from Section 2.2. When a group is labeled as P2P then all the flows of that group are classified as P2P flows.
3.2. Evaluating Graption-P2P To evaluate Graption-P2P, we use traces TR-PAY1 and TR-PAY2, where we have the ground truth using the payload classifier (Section 2). We compute the True Positives, False Positives, and False Negatives. The True Positives (TP) measures how many instances of a given class are correctly classified; the False Positives (FP) measures how many instances of other classes are confused with a given class; and the False Negatives (FN) measures the number of misclassified instances of a class. In our comparisons, we used the following standard metrics: Precision (P), defined as P = TP/(TP + FP); Recall (R), defined as R = TP/(TP + FN); and the F-Measure [38], defined as F = 2P R/(P + R), combining P and R. In all our results, the metrics above report the classification performance in terms of flows. 3.2.1. Forming clusters The K-means algorithm generates k clusters and assigns each flow to a different cluster. We first test the effectiveness of K-means to form pure clusters. A pure cluster will ideally contain flows from a single application. To label the clusters, we use the dominant heuristic [3]. In this heuristic, using the ground truth of flows we label each cluster as belonging to the application with the majority of flows in the cluster. All the flows of a cluster are then classified to belong to this dominant application. Using the dominant heuristic we assign clusters to applications and calculate the precision (P) and recall (R) for different values of k. The clustering results as we increase k for both traces are shown in Fig. 5. We observe that with sufficiently large k (>120) we achieve very good results with P and R above 90%. By increasing k, on one hand we have more ‘‘pure’’ clusters but on the other we make the cluster merging step harder. In the extreme case, each cluster will contain a single flow giving 100% P and R, but making cluster merging challenging. We will return to this topic later in Section 3.5. The Graption-P2P classifier is not sensitive to the exact value for k, since we do not require that each application maps to exactly one cluster. Instead, we only need enough clusters so that flows from different applications do not share a cluster. Hence, it is reasonable to slightly overestiClustering Performance for K-means(%)
(a) Forming clusters. Given the set of discriminating features, the next step is to cluster ‘‘similar’’ flows together. We use the term cluster to describe the outcome of an initial grouping using the selected features. Clusters may be merged in the next function of this step to form groups, which produces the final output. Feature-based clustering is a well-defined statistical data mining problem. For this task we used the popular K-means algorithm [38]. This algorithm has been commonly used for unsupervised clustering of network flows [10,30], with very good results and low computational cost. K-means operates with a single parameter that selects the number of final clusters (k). As we show later in our evaluation, our classifier gives very good results over a large range of k. The similarity between two flows is measured by the Hamming [38] distance calculated over the 16 categorical features (i.e., the payload bytes). Even though more involved similarity measures such as edit-distance (also known as Levenshtein distance) exist, Hamming distance has been used successfully before [15] and performs very well in our application. (b) Cluster merging. During clustering, it is likely that the same application generates multiple clusters. For example, many P2P protocols exhibit a variety of interaction patterns, such as queries (UDP flows) and file transfers (TCP flows), each with significantly different flow and packet characteristics [23]. This motivates mergings clusters that we expect to belong to the same application into groups. This grouping provides a more complete view of the application and aids in understanding the structure of the P2P protocol, as we show in Section 3.2. Cluster merging cannot be based on the chosen set of flow-level features that were already used to create the clusters originally. Instead, in the case of a P2P protocol, it is natural to assume that the TDGs corresponding to each cluster of the same protocol would share a large number of common nodes (IP addresses). Based on these observations, we use an Agglomerative (Hierarchical) Clustering Algorithm that recursively merges clusters with significant similarity in IP addresses. We used the following metric to calculate similarity between clusters: Sim(C1, C2) = (Number of flows having their source or destination IPs present in both clusters)/(The number of flows of the smaller cluster). The cluster merging process starts by hierarchically merging clusters with high similarity and stops when the similarity between all new cluster pairs is below a similarity threshold (ST). As we show later in our evaluation, our classifier gives very good results over a large range of similarity thresholds.
96 94 92 90 88 86 84
Precision (TR-PAY1) Recall (TR-PAY1) Precision (TR-PAY2) Recall (TR-PAY2)
82 80 0
40
80 120 160 Number of Clusters (k)
200
240
Fig. 5. Evaluating K-means: With sufficiently large k (>120) K-means can efficiently separate the flows of different application. All classification metrics are in terms of flows.
1916
M. Iliofotou et al. / Computer Networks 55 (2011) 1909–1920
mate the k. As we show next, with k in the range 120–240 Graption-P2P has high classification performance. 3.2.2. Identifying P2P traffic Using Graption-P2P, we achieve high flow P2P F-Measure over a range of values of k (K-means) and similarity thresholds (ST). We show this in Fig. 6, where we vary the ST from 0.01 to 1 and use a sufficiently large k (see Fig. 5). All experiments are averaged over each disjoint five minute interval of both traces.Intuitively, by using a very large ST, the clusters of an application are not grouped together, which results in TDGs that are harder to classify as P2P. On the other hand, with a very small ST, clusters belonging to different applications are merged together leading to poorer classification performance. The results in Fig. 6 show that we achieve good classification performance (>90% F-Measure), over a large range of similarity thresholds and number of clusters (k). In Fig. 7, we compare our approach with labeling each cluster using the ground truth (i.e. without merging any clusters and labeling each cluster using the dominant heuristic). Intuitively, for a given clustering of flows, the ground truth shows the best that our cluster labeling mechanism can achieve. For merging, we use a ST of 0.5. From Fig. 7, we see that Graption-P2P deviates only slightly from labeling clusters using the ground truth. In the same plot, we also compare Graption-P2P without the cluster
F-Measure of Graption-P2P (%)
100
k=120 k=160 k=240
95 90 85 80 75 70 65 60
0
0.1
0.2
0.3
0.4 0.5 0.6 Similarity Threshold
0.7
0.8
0.9
1
Fig. 6. Graption-P2P achieves > 90% flow F-Measure over a large range of similarity thresholds and number of clusters (k).
merging step, highlighting the benefit of merging clusters of the same application together. Using a ST of 0.5 and k = 160, Graption-P2P achieves above 90% Recall and above 95% Precision over all disjoint five minute intervals for both traces. To apply GraptionP2P to other backbone links, the same selection process can be repeated to adjust the values of ST and k. Our experiments show that the classification performance can degrade with a bad choice of parameters. However, as shown in Figs. 5–7, for reasonable choices for k and ST, our method provides very good results. The Graption framework is flexible enough to incorporate state-of-theart clustering methods. 3.3. Comparison with BLINC [25] We used BLINC to classify traffic for both TR-PAY1 and TR-PAY2 traces. BLINC was optimized after several trial and error efforts to achieve its best accuracy for classifying P2P traffic over these traces. A detailed description of the selection process for the parameters for BLINC is provided in [26]. As reported by Kim et al. [26], tuning the 28 parameters of BLINC is a time consuming process. This is because small changes to the thresholds have a significant effect on the classification results. The Recall and Precision for BLINC are 84% and 89% respectively compared to 90% and 95% of our method. The classification performance per P2P protocol for both BLINC and Graption-P2P are shown in Fig. 8. Given that both methods do not distinguish between particular P2P protocols (e.g., Gnutella vs BitTorrent), it only make sense to report the portion of identified traffic (recall) for each protocol. Graption-P2P can identify more traffic for all P2P applications tested with the exception of MP2P. We can see from Fig. 3 that MP2P has a much higher average node degree than the other P2P applications which can explain the high performance of BLINC over this protocol. Going back to Fig. 8, we see that BLINC has significantly lower performance for some P2P applications. For example, Graption-P2P detects 95% of BitTorrent traffic, while BLINC detects only 25%! In addition, Graption-P2P detects 91% of the flows from the three popular P2P applications {BitTorrent, Gnutella, and eDonkey} while BLINC detects only 73%. Our experiments suggest that BLINC and possible other behavioral-host-based approaches work well when applied Graption−P2P BLINC
95
100
90
80
85 Ground Truth Graption-P2P: Merging Graption-P2P: No Merging
80
Recall (%)
F-Measure of Graption-P2P (%)
100
75 70 65
60 40 20
60 80
100
120
140 160 180 Clusters (k) in K-means
200
220
240
Fig. 7. Graption-P2P compared to cluster labeling based on ground truth. Results are also compared with and without cluster merging.
0 BitTorrent Gnutella eDonkey FastTrack MP2P Soribada
Rest
Fig. 8. The percentage of P2P flows detected by Graption-P2P and BLINC. Flow precision for Graption-P2P is 95% and for BLINC is 89%.
M. Iliofotou et al. / Computer Networks 55 (2011) 1909–1920
at the edge, where a large fraction of host flows are observed and hence enough evidence is collected to profile each node. However, this is not always true for backbone monitoring points which can explain BLINC’s lower performance. These observations are also supported by findings in [26]. We acknowledge that BLINC provides a methodology for detecting traffic from multiple classes of applications (e.g., HTTP, FTP, Chat, etc.) and not only for P2P. Our results only show that Graption-P2P does better in P2P detection. 3.4. Identifying the unknown Our goal here is to answer the following question: Can Graption identify ports used by unknown P2P applications in our traces? To achieve this, we configure Graption as follows. For step 1, we isolate legacy flows using the same approach as in Graption-P2P. In addition, we isolate flows from ports that belong to known P2P applications (e.g., port 1214 for FastTrack). At step 2, we cluster flows using their port number. This produces one cluster for each distinct port in the trace. For merging clusters, we use the same similarity threshold as in the Graption-P2P configuration. Finally, for classifying each group (step 3) we use the same rules as in Graption-P2P. We will refer to this configuration as Graption-P2P-Ports. By manually inspecting the TDGs that Graption-P2PPorts reports as P2P, we discovered interesting behaviors in the Abilene Trace (TR-ABIL). Our simple method was able to detect three ports (2002, 4156, 1978) that we initially thought were false positives. On further research, we found that these three ports were used by the Slapper2 worm. Slapper has been reported to use these ports in order to form a P2P botnet overlay for launching DDoS attacks. Even though Graption-P2P-Ports was never explicitly trained to detect the Slapper overlay, it used the TDG profile of known P2P applications (e.g., eDonkey, Gnutella, etc.) to identify it. This is an advantage of using network-wide behavior to classify traffic. Graption-P2P-Ports also captured an unusual behavior of IPs executing traceroute to each other forming a near clique. Such highly connected group of hosts was not observed in any of the P2P applications present in our traces. We speculate these interactions are by host performing active network measurements inside the Internet2 (Abilene) backbone. Given this unique connection pattern, we can always isolate such unusual behavior and distinguish it from a potentially new P2P application. Besides the active measurement traffic and the flows from the Slapper worm, Graption-P2P-Ports reported only 0.3% other flows as P2P. With no payload information, we were not to able to manually match these flows to a known exploit or P2P application. 3.5. Other Graption configurations For completeness, we show here preliminary results with other system configurations assuming that access to payload is limited or isolation cannot be used. We proceed by describing these configurations. 2
http://www.cert.org/advisories/CA-2002-27.html
1917
Graption-P2P-NI: without isolation. This is the same configuration as the Graption-P2P, but without using isolation. By increasing k in K-means above 300 we achieve good results; >92% precision and >85% recall. A larger k was to be expected, since by including all the Web, SMTP, and DNS flows the total number of signatures increased as well. Our preliminary experiments showed that by increasing the number of clusters, it makes the cluster merging step more difficult. This observation highlights the advantage of using isolation. Graption-P2P-NP: using only flow-level features. This configuration is the same as the Graption-P2P, but assuming that payload and port numbers cannot be used. We achieved >90% precision and >88% recall over a range of k and similarity thresholds. For each flow, we extracted more than 40 flow features ranging from packet size information (size of first 10 packets, max/min packet size, etc.), timing information (flow duration, min and max inter-packet gap, etc.), TCP flags, total volumes in bytes, number of packets, etc. In order to test the relevance of each feature we applied the Information Gain Ranking Filter [38] over various time intervals. For this configuration, we used the most prominent features: packet size information (i.e. min, max, and the size of the first five packets) and protocol (UDP or TCP). A recent work [11] also supports our observation that packet sizes are good features to cluster/classify network flows. 4. Related work This journal paper extends our six-page workshop version [18]. In addition to the work presented in the workshop version, we here include: (a) A study on practical considerations for using TDGs (see Section 2.3, Figs. 3 and 4). (b) We provide additional details regarding our data sets and our classification results (see Sections 2.2 and 3.5, Fig. 2, and Table 2). (c) We provide a more detailed comparison with BLINC (see Section 3.3 and Fig. 8). (d) We include additional experiments showing how Graption can be used to detect previously unknown traffic (see Section 3.4). 4.1. Traffic classification As an alternative to port-based methods, some work use payload [30,28]. Other approaches use machine learning (ML) algorithms to classify traffic using flow features (e.g., packet sizes). For an exhaustive list and comparison of ML algorithms we refer the reader to [32,26]. The problem of training a ML algorithm on one trace and applying it on another was recently addressed in [37]. Even though supervised solutions to traffic classification are continuously improving, all supervised methods require per application training and will thus not detect traffic from new applications. Our work has more in common with unsupervised methods which group similar flows together. All previous methods [3,29,10] require manual labeling of clusters. Our work bridges this gap by providing a method to automatically label clusters of flows based on their network-wide behavior. In BLINC [25], the authors characterize the connection patterns (e.g., if it behaves like using P2P) of a single host at the transport layer and use these patterns to label the
1918
M. Iliofotou et al. / Computer Networks 55 (2011) 1909–1920
flows of each host. BLINC uses graph models called graphlets to model a host’s connection patterns using port and IP cardinalities. Unlike TDGs, graphlets do not represent network-wide host interaction. In some sense, TDGs represent a further level of aggregation, by aggregating across hosts as well. Thus it is perhaps fair to say that while BLINC hints at the benefit of analyzing the node’s interaction at the ‘‘social’’ level, it ultimately follows a different path that focuses on the behavior of individual nodes. As we show, our approach performs better than BLINC in our backbone traces (see Section 3.3). Similar to BLINC, other host-based methods [2,22] target the identification of P2P users inside a university campus (i.e., network edge). The connection patterns of neighboring host (e.g., their degree distribution) were also used as features to profile network hosts [1]. Unlike Graption-P2P, in [2,22,25,1] they do not use network-wide host interaction. In [7], the authors use a port-based method to identify P2P users, using their temporal appearance and connection patterns in a trace. A different approach to host profiling was introduced by Trestian et al. [36]. They used readily available information from the Web to classify traffic using the Google search engine. They show very good results for classifying flows for legacy application, but their results are not promising for P2P detection because of the dynamic nature of P2P IP hosts. Our method can thus be used to complement the work in [36]. The problem of classifying traffic from applications that use sophisticated obfuscation techniques is the topic in [17]. The solution proposed uses advanced community detection algorithms to group related hosts together. It then requires external seed information about the application used by few hosts in each group in order to classify the entire group. The solution in [17] is very different from this work since it does not classify applications using networkwide behavioral informatio. We believe that Graption-P2P can provide a way to identifying P2P communities without relying on external seeding information. 4.2. Worm detection Graphs have been used for detecting worm activities within enterprise networks [9]. Their main goal was to detect the tree-like communication structure of worm propagation. This characteristic of worms was also used for post-mortem trace analysis (for the identification of the source of a worm outbreak, the so-called patient zero) using backbone traces [39]. More recent studies use graph techniques to detect hit-list worms within an enterprise network, based on the observation that an attacker will alter the connected components in the network [6]. 4.3. Measurements on network-wide interactions Statistical methods are used in [40] for automating the profiling of network hosts and ports numbers. The connectivity behavior and habits of users within enterprise networks is the focus of many papers, including [35]. In [34], the authors study P2P overlays using passive measurements, but target mainly the profiling of P2P hosts. The most resent work on network-wide interactions in by Jin et al. [21]. In their work [21], they use graph-partitioning methods to ex-
tract and study the evolution of smaller communities within a TDG. None of the above papers targets P2P detection.
5. Discussion Enhancing the isolation step. To improve isolation we can enforce payload inspection in addition to port-based filtering. For example, we can test all DNS flows at port 53 to see if they also have a DNS payload signature or if another protocol is tunneling its traffic under the DNS port. If payload is encrypted, then we can choose to use flow-level feature such as packet sizes [26] or whitelisted IP addresses [36]. Applying our framework to other backbone locations. In our study we show that is feasible to correctly identify P2P TDGs from all our backbone traces using a single set of thresholds. In other words, we do not use different classification threshold for each individual trace. Even though TDGs from different locations may have differences between them, in our study, we always find TDGs from the same family (e.g., P2P) to have similar characteristics (e.g., high average degree). In Section 2.2, we explicitly state that our thresholds are not universal. Therefore, it might not always be possible to use exactly the same thresholds to identify particular TDGs in all backbone locations. Nevertheless, our measurements suggest that small adjustments to our simple parameters (see Section 2.2) allow our methodology to work on different backbone links. TDG formation and packet sampling. Packet sampling is used in order to reduce the overhead of processing every single packet by a monitoring system. Sampling effectively reduces the overall number of observed flows by the monitor. Less overall flows translates to fewer observed edges over time when forming TDGs. In our experiments in Section 2.3, we observe that the longer we collect edges for a TDG (i.e., applying a longer observation interval) the number of edges and the stability of metrics increases. With sampling, we believe the curves in Fig. 4 may change. In particular, in order to reduce TDG variability in the presence of sampling, the monitoring interval will need to be increase. Encryption in P2P applications. If an application encrypts all traffic classification methods using payload [33,28,15, 30] will be out-dated. Note, however, that it has been shown before [4] that even if an application encrypts its traffic there are ways to distinguish it from other flows. To show that even in the case where P2P encryption becomes prevalent our framework will still be useful, we also experimented with Graption-P2P-NP using only flow level features and observed good results with recall and precision of above 88% in both traces. Obfuscation of P2P applications at multiple levels. In the future, obfuscation might become more common. In the case where IPsec (encryption up to Layer-4 headers) becomes prevalent, transport layer [24], and host-based [25] methodologies will most likely not work. If P2P protocols start using polymorphic blending methods [12,14] to masquerade their traffic at the packet (e.g., packet sizes) and payload level to resemble a legacy application (e.g., Web), then all current machine learning [32] methods will not work. In such a scenario, TDGs (IP-to-IP graphs) might still prove to be useful because the interaction patterns seem hard to hide.
M. Iliofotou et al. / Computer Networks 55 (2011) 1909–1920
6. Summary and conclusions The underlying theme of our work is to go beyond just monitoring individual packets, flows, or hosts, to also monitoring the interactions of a group of hosts. Towards this end, we introduce TDGs and show their potential to generate novel tools for traffic classification. Based on TDGs, we developed Graption, a graph-based framework, which we then specialize (Graption-P2P) for the detection of P2P applications. We show that Graption-P2P classifies more than 90% of P2P traffic with above 95% precision when tested on real-world backbone traces. While our best results are obtained assuming access to some initial payload bytes, Graption still performs well (90% precision with 88% recall) even when payload is not used. We believe the network-wide characteristics of P2P traffic (e.g., InO, average degree, and diameter) are not as easy to hide without significantly altering P2P functionality.
Acknowledgments This work was supported partly by NSF NECO 0832069, a CISCO URP grant, ARL CTA W911NF-09–2-0053, by NAP of Korea Research Council of Fundamental Science and Technology, and the ITRC support program [NIPA-2010C1090–1011-0004] of MKE/NIPA. Michael Mitzenmacher was supported in part by NSF grant CNS-0721491 and research grant from CISCO. George Varghese was supported by grants from NSF and CISCO. The SDSC’s TeraGrid and compute resources are supported by the NSF grant CONMI CRI-0551542. Support for CAIDA’s Internet traces is provided by the National Science Foundation, the US Department of Homeland Security, and CAIDA Members.
References [1] E.G. Allan, W.H. Turkett, E.W. Fulp, Using network motifs to identify application protocols, in: IEEE GLOBECOM, 2009. [2] G. Barlett, J. Heidemann, C. Papadopoulos, Inherent behaviors of online detection of peer-to-peer file sharing, in: IEEE GI, 2007. [3] L. Bernaille, R. Teixeira, K. Salamatian, Early application identification, in: ACM CoNEXT, 2006. [4] D. Bonfiglio, M. Mellia, M. Meo, D. Rossi, P. Tofanelli, Revealing Skype Traffic: when randomness plays with you, in: ACM SIGCOMM, 2007. [5] CAIDA. http://www.caida.org. [6] M.P. Collins, M.K. Reiter, Hit-List worm detection and bot identification in large networks using protocol graphs, in: RAID, 2007. [7] F. Constantinou, P. Mavrommatis, Identifying known and unknown peer-to-peer traffic, in: IEEE NCA, 2006. [8] DatCat, Internet Measurement Data Catalog. http://www.datcat.org. [9] D. Ellis, J. Aiken, K. Attwood, S. Tenarglia, A behavioral approach to worm detection, in: ACM CCS WORM, 2004. [10] J. Erman, M. Arlitt, A. Mahanti, Traffic classification using clustering algorithms, in: ACM SIGCOMM MineNet, 2006. [11] A. Este, F. Gringoli, L. Slagerelli, On the stability of the information carried by traffic flow features at the packet level, ACM SIGCOMM CCR 39 (3) (2009) 13–18. [12] P. Fogla, M. Sharif, R. Perdisci, O. Kolesnikov, W. Lee, Polymorphic blending attacks, in: USENIX Security Symposium, 2006. [13] A. Gerber, J. Houle, H. Nguyen, M. Roughan, S. Sen, P2P, the Gorilla in the cable, in: National Cable and Telecommunications Association (NCTA), 2003. [14] G. Gu, R. Perdisci, J. Zhang, W. Lee. BotMiner: clustering analysis of network traffic from protocol- and structure-independent botnet detection, in: USENIX Security Symposium, 2008. [15] P. Haffner, S. Sen, O. Spatscheck, D. Wang, ACAS: automated construction of application signatures, in: ACM SIGCOMM MineNet, 2005.
1919
[16] M. Iliofotou, M. Faloutsos, M. Mitzenmacher, Exploiting dynamicity in graph-based traffic analysis: techniques and applications, in: ACM CoNEXT, 2009. [17] M. Iliofotou, B. Gallagher, G. Xie, T. Eliassi-Rad, M. Faloutsos, Profiling-by-association: a resilient traffic profiling solution for the internet backbone, in: ACM CoNEXT, 2010. [18] M. Iliofotou, H. Kim, P. Pappu, M. Faloutsos, M. Mitzenmacher, G. Varghese, Graph-based P2P traffic classification at the internet backbone, in: IEEE GI, 2009. [19] M. Iliofotou, P. Pappu, M. Faloutsos, M. Mitzenmacher, S. Singh, G. Varghese, Network monitoring using traffic dispersion graphs (TDGs), in: ACM IMC, 2007. [20] IPOQUE – Bandwidth Management with Deep Packet Inspection, Internet Study: The Impact of P2P File Sharing, Voice over IP, Instant Messaging, One-Click Hosting and Media Streaming on the Internet, 2009. [21] Y. Jin, S. Esam, Z.L. Zhang, Unveiling core network-wide communication patterns through application traffic activity graph decomposition, in: ACM SIGMETRICS, 2009. [22] W. John, S. Tafvelin, Heuristics to classify internet backbone traffic based on connection patterns, in: IEEE ICOIN, 2008. [23] T. Karagiannis, A. Broido, N. Brownlee, K. Claffy, M. Faloutsos, Is P2P dying or just hiding? in: IEEE GLOBECOM, 2004. [24] T. Karagiannis, A. Broido, M. Faloutsos, K. claffy, Transport layer identification of P2P traffic, in: ACM IMC, 2004. [25] T. Karagiannis, K. Papagiannaki, M. Faloutsos, BLINC: multi-level traffic classification in the dark, in: ACM SIGCOMM, 2005. [26] H. Kim, K. Claffy, M. Fomenkov, D. Barman, M. Faloutsos, K. Lee, Internet traffic classification demystified: myths, caveats, and the best practices, in: ACM CoNEXT, 2008. [27] C. Labovitz, S. Lekel-johnson, J. Oberheide, F. Jahanian, Internet interdomain traffic, in: ACM SIGCOMM, 2010. [28] J. Ma, K. Levchenko, C. Kreibich, S. Savage, G.M. Voelker, Unexpected means of protocol inference, in: ACM IMC, 2006. [29] A. McGregor, M. Hall, P. Lorier, J. Brunskill, Flow clustering using machine learning techniques, in: PAM, 2004. [30] A. Moore, K. Papagiannaki, Toward the accurate identification of network applications, in: PAM, 2005. [31] M.E.J. Newman, The structure and function of complex networks, SIAM Review 45 (2003) 167. [32] T.T.T. Nguyen, G. Armitage, A survey of techniques for internet traffic classification using machine learning, IEEE Communications Surveys and Tutorials, 4th ed., March, 2008. [33] S. Sen, O. Spatscheck, D. Wang, Accurate, scalable in-network identification of p2p traffic using application signatures, in: WWW, 2004. [34] S. Sen, J. Wang, Analyzing peer-to-peer traffic across large networks, IEEE/ACM Transaction on Networking 12 (2) (2004) 219–232. [35] G. Tan, M. Poletto, J. Guttag, F. Kaashoek, Role classification of hosts within enterprise networks based on connection patterns, in: USENIX Annual Technical Conference, 2003. [36] I. Trestian, S. Ranjan, A. Kuzmanovic, A. Nucci, Unconstrained endpoint profiling (Googling the Internet), in: ACM SIGCOMM, 2008. [37] G. Urvoy-Keller, T. En-Najjary, Challenging statistical classification for operational usage: the ADSL case, in: ACM IMC, 2009. [38] I.H. Witten, E. Frank, Data Mining: Practical Machine Learning Tools and Techniques, 2nd ed., Morgan Kaufman, 2005. [39] Y. Xie, V. Sekar, D. Maltz, M. Reiter, H. Zhan, Forensic analysis of epidemic attacks in federated networks, in: IEEE ICNP, 2006. [40] K. Xu, Z. Zhang, S. Bhattacharyya, Profiling internet backbone traffic: behavior models and applications, in: ACM SIGCOMM, 2005.
Marios Iliofotou received the B.S. degree in Electrical and Computer Engineering from the Technical University of Crete, in Greece, and the M.S. and Ph.D. degree in Computer Science from the University of California, Riverside, in 2007 and 2011, respectively. His current research investigates the effectiveness of graph-based algorithms for managing and protecting large IP networks. During the course of his Ph.D., Marios has worked as an intern at Microsoft Research in Cambridge, Telefonica Research in Barcelona, the Lawrence Livermore National Laboratory, Narus, and Cisco Systems.
1920
M. Iliofotou et al. / Computer Networks 55 (2011) 1909–1920
Hyun-chul Kim is an assistant professor in the School of Computer Science and Engineering at Seoul National University. His research interests include content-centric networking, measurement, analysis and modeling of Internet traffic and topology, and user behavior patterns in a variety of largescale applications such as IPTV, VoD, VoIP, P2P file sharing, massively multi-player online role-playing games (MMORPGs), and the Web. Kim received his Ph.D. in computer science from the Korean Advanced Institute of Science and Technology (KAIST).
Michalis Faloutsos is a faculty member at the Computer Science Department in University of California, Riverside. He got his bachelor’s degree at the National Technical University of Athens and his M.Sc and Ph.D. at the University of Toronto. His interests include, Internet protocols and measurements, peer-to-peer networks, network security, BGP routing, and ad-hoc networks. He is actively involved in the community as a reviewer and a TPC member in many conferences and journals. With his two brothers, he co-authored the paper on powerlaws of the Internet topology SIGCOMM’99, which is one of the top ten most cited papers of 1999. His most recent work on peerto-peer measurements have been widely cited in popular printed and electronic press such as slashdot, ACM Electronic News, USA Today, and Wired. Most recently he has focused on the classification of traffic and identification of abnormal network behavior. He also works in the area of Internet routing (BGP), and ad hoc networks routing, and network security, with emphasis on routing.
Michael Mitzenmacher is a Professor of Computer Science in the School of Engineering and Applied Sciences at Harvard University. He graduated summa cum laude with a B.A. in mathematics and computer science from Harvard in 1991. After studying mathematics for a year in Cambridge, England, on the Churchill Scholarship, he obtained his Ph. D. in computer science at U.C. Berkeley in 1996. His work on low-density parity-check codes shared the 2002 IEEE Information Theory Society Best Paper Award and won the 2009 ACM SIGCOMM Test of Time Award. His textbook on randomized algorithms and probabilistic tech-
niques in computer science was published in 2005 by Cambridge University Press.
Prashanth Pappu received his Doctor of Science degree from Washington University in St. Louis where his research was focused on design of scalable routers and included QoS architectures, packet lookup and scheduling algorithms. Later, he was a Technical Leader at Cisco Systems where he led the design of Cisco’s data center, storage and Ethernet switches. He’s currently the Director of Product Management at Conviva Inc. that is creating a real-time, viewer-centric platform for delivering the highest resolution, broadcast quality video on the Internet.
George Varghese worked at DEC for several years designing DECNET protocols and products (bridge architecture, Gigaswitch) before obtaining his Ph.D. in 1992 from MIT. He worked from 1993–1999 at Washington University. He joined UCSD in 1999, where he currently is a professor of computer science. He won the ONR Young Investigator Award in 1996, and was elected to be a Fellow of the Association for Computing Machinery (ACM) in 2002. Together with colleagues, he has 16 patents awarded in the general field of Network Algorithmics. Several of the algorithms he has helped develop have found their way into commercial systems including Linux (timing wheels), the Cisco GSR (DRR), and Microsoft Windows (IP lookups). He also helped design the lookup engine for Procket’s 40 Gbps forwarding engine. He has written a book on building fast router and endnode implementations called ‘‘Network Algorithmics’’, which was published in December 2004 by Morgan-Kaufman. In May 2004, he co-founded NetSift Inc., where he was the President and CTO. NetSift was acquired by Cisco Systems in 2005. From Aug 2005 to Aug 2006, he worked at Cisco Systems to help equip future routers and switches to detect traffic patterns for measurement and security.
