A graph-based P2P traffic classification framework for the internet ...

23 downloads 67564 Views 1MB Size Report
20% of all Inter-domain traffic on the Internet is P2P. Even ... ioral-host-based approaches such as BLINC [25] can detect .... To the best of our knowledge,.
Computer Networks 55 (2011) 1909–1920

Contents lists available at ScienceDirect

Computer Networks journal homepage: www.elsevier.com/locate/comnet

Graption: A graph-based P2P traffic classification framework for the internet backbone Marios Iliofotou a,⇑, Hyun-chul Kim b, Michalis Faloutsos a, Michael Mitzenmacher c, Prashanth Pappu d, George Varghese e a

University of California, Riverside, CA, USA Seoul National University, Gwanak-gu, Seoul, Korea c Harvard University, Cambridge, MA, USA d Conviva, Inc., San Mateo, CA, USA e University of California, San Diego, San Diego, CA, USA b

a r t i c l e

i n f o

Article history: Received 6 February 2010 Received in revised form 18 November 2010 Accepted 29 January 2011 Available online 13 February 2011 Keywords: Traffic classification Behavioral-approach Peer-to-peer Graph mining

a b s t r a c t Monitoring network traffic and classifying applications are essential functions for network administrators. Current traffic classification methods can be grouped in three categories: (a) flow-based (e.g., packet sizing/timing features), (b) payload-based, and (c) host-based. Methods from all three categories have limitations, especially when it comes to detecting new applications, and classifying traffic at the backbone. In this paper, we propose the use of Traffic Dispersion Graphs (TDGs) to remedy these limitations. Given a set of flows, a TDG is a graph with an edge between any two IP addresses that communicate; thus TDGs capture network-wide interactions. Using TDGs, we develop an application classification framework dubbed Graption (Graph-based classification). Our framework provides a systematic way to classify traffic by using information from the network-wide behavior and flow-level characteristics of Internet applications. As a proof of concept, we instantiate our framework to detect P2P traffic, and show that it can identify 90% of P2P flows with 95% accuracy in backbone traces, which are particularly challenging for other methods. Ó 2011 Elsevier B.V. All rights reserved.

1. Introduction An important task when monitoring and managing large networks is classifying flows according to the application that generates them. Such information can be utilized for network planning and design, QoS and traffic shaping, and security. In particular, detecting P2P traffic is a potentially important problem for ISPs that want to manage such traffic, and for specific groups such as the entertainment industry in legal and copyright disputes. Detecting P2P traffic also has particular interest since it represents a large portion of the Internet traffic. In fact, a resent work by Labovitz et al. [27] shows that at least ⇑ Corresponding author. Tel.: +1 951 880 3699. E-mail address: [email protected] (M. Iliofotou). 1389-1286/$ - see front matter Ó 2011 Elsevier B.V. All rights reserved. doi:10.1016/j.comnet.2011.01.020

20% of all Inter-domain traffic on the Internet is P2P. Even though percentage wise P2P traffic is not the dominant Internet traffic contributor today, its overall traffic volume is constantly increasing. In addition, other studies using advanced deep-packet inspection show P2P traffic to be up to 70% of the overall volume in some networks [20]. Most current application classification methods can be naturally categorized according to their level of observation: payload-based signature-matching methods [30,28], flow-level statistical approaches [10,32], or host-level methods [25,40]. Each existing approach has its own pros and cons, and no single method clearly emerges as a winner. Relevant problems that need to be considered include identifying applications that are new, and thus without a known profile; operating at backbone links [26,25]; and detecting applications that intentionally alter their

1910

M. Iliofotou et al. / Computer Networks 55 (2011) 1909–1920

behavior. Flow-level and payload-based classification methods require per application training and will thus not detect P2P traffic from emerging protocols. Behavioral-host-based approaches such as BLINC [25] can detect traffic from new protocols [25], but have weak performance when applied at the backbone [26]. In addition, most tools including BLINC [25] require fine-tuning and careful selection of parameters [26]. We discuss the limitations of previous methods in more detail in Section 4. In this paper, we use the network-wide behavior of an application to assist in classifying its traffic. To model this behavior, we use graphs where each node is an IP address, and each edge represents a type of interaction between two nodes. We use the term Traffic Dispersion Graph or TDG to refer to such a graph [19]. Intuitively, with TDGs we enable the detection of network-wide behavior (e.g., highly connected graphs) that is common among P2P applications and different from other traffic (e.g., Web). While we recognize that some previous efforts [6,9] have used graphs to detect worm activity, they have not explored the full capabilities of TDGs for application classification. This paper is an extension of a workshop paper [18] and the differences will be clarified in the related work section (Section 4). We propose a classification framework, dubbed Graption (Graph-based classification), as a systematic way to combine network-wide behavior and flow-level characteristics of network applications. Graption first groups flows using flow-level features, in an unsupervised and agnostic way, i.e., without using application-specific knowledge. It then uses TDGs to classify each group of flows. As a proof of concept, we instantiate our framework and develop a P2P detection method, which we call Graption-P2P. Compared to other methods (e.g., BLINC [25]), Graption-P2P is easier to configure and requires fewer parameters. The highlights of our work can be summarized in the following points:  Distinguishing between P2P and client–server TDGs. We use real-world backbone traces and derive graph theoretic metrics that can distinguish between the TDGs formed by client–server (e.g., Web) and P2P (e.g., eDonkey) applications (Section 2.2).  Practical considerations for TDGs. We show that even a single backbone link contains enough information to generate TDGs that can be used to classify traffic. In addition, TDGs of the same application seem fairly consistent across time (Section 2.3).  High P2P classification accuracy. Our framework instantiation (Graption-P2P) classifies 90% of P2P traffic with 95% accuracy when applied at the backbone. Such traces are particularly challenging for other methods (Section 3.2.2).  Comparison with a behavioral-host-based method. Graption-P2P performs better than BLINC [25] in P2P identification at the backbone. For example, Graption-P2P identifies 95% of BitTorrent traffic while BLINC identifies only 25% (Section 3.3).  Identifying the unknown. Using Graption, we identified a P2P overlay of the Slapper worm. The TDG of Slapper was never used to train our classifier. This is a promis-

ing result showing that our approach can be used to detect both known and unknown P2P applications (Section 3.4). The rest of the paper is organized as follows. In Section 2 we define TDGs, and identify TDG-based metrics that differentiate between applications. In Section 3 we present the Graption framework and our instantiation, GraptionP2P. In Section 5 we discuss various practical issues. In Section 4 we discuss related work. Finally, in Section 6 we conclude the paper. 2. Studying the TDGs of P2P applications 2.1. Traffic dispersion graphs (TDGs) Definition. Throughout this paper, we assume that packets can be grouped into flows using the standard 5-tuple {srcIP, srcPort, dstIP, dstPort, protocol}. Given a group of flows S, collected over a fixed-length time interval, we define the corresponding TDG to be a directed graph G(V, E), where the set of nodes V corresponds to the set of IP addresses in S, and there is a link (u, v) 2 E from u to v if there is a flow f 2 S between them. In this paper, we consider bidirectional flows. We define a TCP flow to start on the first packet with the SYN-flag set and the ACK-flag not set, so that the initiator and the recipient of the flow are defined for the purposes of direction. For UDP flows, direction is decided upon the first packet of the flow. Visualization examples. In Fig. 1, we show TDG examples from two different applications. In order to motivate the discussion in the rest of the paper, we show the contrast between a P2P and a client–server TDG. From the figure we see that P2P traffic forms more connected and more dense graphs compared to client–server TDGs. In Section 2.2, we show how we can translate the visual intuition of Fig. 1 into quantitative measures that can be used to classify TDGs that correspond to different applications. Data set. To study TDGs, we use three backbone traces from a Tier-1 ISP and the Abilene (Internet2) network. These traces are summarized in Table 1. All data are IP anonymized and contain traffic from both directions of the link. The TR-PAY1 and TR-PAY21 traces were collected from an OC48 link of a commercial US Tier-1 ISP at the Palo Alto Internet eXchange (PAIX). To the best of our knowledge, these are the most recent backbone traces with payload that are available to researchers by CAIDA [5]. The TR-ABIL trace is a publicly available data set collected from the Abilene (Internet2) academic network connecting Indianapolis with Kansas City. The Abilene trace consists of five randomly selected five-minute samples taken every day for one month, and covers both day and night hours as well as weekdays and weekends. Extractingground truth. We used a Payload-based Classifier (PC) to establish the ground truth of flows for the 1 The authors thank CAIDA for providing this set of traffic traces. Additional information for these traces can be found in the DatCat, Internet Measurement Data Catalog [8], indexed under the label ‘‘PAIX’’.

1911

M. Iliofotou et al. / Computer Networks 55 (2011) 1909–1920 864 1337

1335

1330

710

1323

663 1257

1338

1245

1243

1336

1331

1324

502 1032

990

987

952

1271 1258

1246

1244

272 827

374

920 1033

991

988 865

102

953 1016 828

9 711

664 58

375 1003

503 853 267 903 103 641

506 273

499 1239 993

59

854

615

1312 46 398

1342

788

1108

1240

498 862

823 10

1234

473

45 973 240 200 208

353 507 904 863

492

983

207

509

824

150 284

956 971

241

1202 1038 177

151 559

115 236

984

190

759 1039

604

1259 275 264 887

532

199 1201

237

985

336 352 587 1040

21

901

114

981

588 1041

22 266 560 1144

1343

189 338

1311

982

533 265

621

70

1077

1344 783

535 23 468 337 1308 622 992

33

1102

346 332 949 1377 649 1300 71 638

771

607

943 1141

586 752

1280

19

688

639

1120

608

1188 89

1119 270 860 536

1096

1281

20 88 1184

34 24

467

1345

126 1362 798 297

738 1283

1232

1055

175 718 687 167 17

698

557 1284

1233 1154

1163

817

1252 91

1197 583

18

677

950

1292

1036 1329

92 462

818

290 1351

298

837

958 1079

1098

1250

996 152 797 932 1293

699

775 1008

719

112

514 960 573 1178

333 577 629

450

760

558 930

259

704 574

446 1191 515

1030 424 291

205

184

13 471 951 1315

1156

441 685 1157

563 193

729

1247

113 153 655

601 433

372

271 1261

959

1225 14

997

447 440

1162

907 825

636 488

740

776

1347

829 411

204

1113 315 659 258 308 656

57 1037 444

946

1169 891 351

1213 1207 1192

1182

631 472 1305

548 454

214

911

703 1348 397

1369

819 945

76 326

940

101

1374 701 1015

1327

131

1214 1100

869 368 1078 994

1014

1052 814

523

871

119 605

1371 1208 246 963 835

1058

260

1194 90

1222 169 466 602

564

309 1319 96 369

1328

528

192

412

594

875

962 582

1364

439 1215 1019

669

1340 795

226

972

725

1095

1339 35 585

762

661

147

2 1161

1074

1376

1290

327

905

1065 233

857

620

1326

842

1195

944 933

686 299

646

26

1174 282

106

1172

786

785 1277

1

1196

896

1231 926

42

542

845

144

883

154 1047 457

1159

745

384 180 442

886

262

965

670

660

363 1238

410 456

261

1200

1262

909 626

453 913 995

1235 215 730

234

320 1012

746

100 143 280 224

347

894

107

787

882 805

417

632

1212 747 672

826

872 1370 65

321 310

1341

136

1013 566 129 5

210

1017 404 1237 1093

170

1321 77

25 572

867

1094 881 247 279

850

658 41 673

328

36 578 64

463

801

634 399 322

1220

832

1021 155 8 181 934 377

7

520

879

1266

1153

436 1114

248

438 1224

884 1365

218

360

281 633 194 556 117

999 1206

176

128

361

302

111

209 118

62 627 355 125 1025

624 301

1066 967 408 183 804

497

1204 277

244

435

792 741 341

6

356 595

319 616

110

756 171 311 1075 593 749 496

1189

491 838

613

690 257

906

715

754 816

1140

735 576 538 511

1101 979 561

592 534

580 878

252 714 731

986

1285

1128 182 512

448

493

1166

198

83

109 1147 1011

693

206

706 276

373

295

1090

49

123 964

1359

1133

897 350 429

445

188

127

423

968

469

899

81

966

201

60

87

348

1164

980 461 568

420

889

628

159

674

890

888

1092

1127

1199

836

339

666 833 160

50

531

1278

609

1103

1264

936

570

1004

799

1050

393

464

1291 340 870

477

948

161

108

239 530

28

680 648

212

162

552 376

15

856 1048

874

914

413

575

51

1325 1057

82

1282

263

562

784

465 174

975

938 1001

61

915

137

1254 1056

841

313

245

614

211

1051 668

794

124

132

789

312

970

584

619

135

331

925

165

1049 1217

1248 545

1000

773

479

947 1260

343

1251

484

149 1099 403

590

1170

597 140

1373 32

69 292 16

317

758

640

227

567

1296

893 1106

342

748

1105

1009

500

31

364

449

1353

976

1205 1137

225

86

644

1272

571 1372

230

1190

1045

618

831 1241

635

187 1303 419

1176 768

148

537 122

755

303 318

1121 191

235 732

37

158

505 232

196

40

851

432

868

1183

782

27

485

39 316

421

630

1088

598

130

716 978

723

1097

1112

344

510

1352

278

1288

84 254 392 525

418

138

1216

808

774

475

383

1148

428 1129

63

222

612

844 324

1028

791

569 1046

596

85

394

414

255

253 1089

1255

812

52 164

1273 1022 728

242

139

898

1165 1116 1333

876

1267

1322

274

1356

443

1242

354 1210

653

689

378

1059

1151

1302 487

671

405

727 43

610 974 314

679

157

813

811 1173

855

961

902 1289

955

1142

1060

268

1203

489

684

163

1018 892

217

717

1256

98

529

843

494 793

751

809

365 551

540

231

852

1031

1069 451 724

1320 1005

486

866

349

599

68

908

1186

643

722 611

1269

858

1027 806

834 637

895

753

1136

1035

654

116

796

1334

810

579 495

1185

839 554

647

650

474

937

38

504 1086

939

1064

1209

422

591

989

478

734 1061

168

508

323

516

1053

134 396 517

1279

1375 603

916 1249 1349

1218

1363

1010

78

665

846

1117

452

105

733

1316

524 606

44 589

757 1076 969

935

362 678 1171

388

395

657

1198

1135

900

1043 1301

977

565

1346

800 1275

737

283

409

195

458 1024

381 387 807

186

1287

705

700

736

172

455

750

243

1070 1134

400

391 781 97

156 623

1054

1152

1044

145 73

1228 830

1350

790

1317

345

1085

642

720 166

1229

146 675

173

1081

1236 406

1354 847

1118

1087 539

1160 269

1274 104

815 216 555 402

1355

1268 1263 221 1297

676

820 401 1304

12 80

1062

803

694

726 848 721 617 849 1357

553

1313

927 1143

928

219 859 885

1270

121

1020 1042

922

1265 1080 99 778 29 840 300

238

780 1358

1126

1314 645 3

72

366 1286

1181 921 1219

223 861 779 600 998 79 1091 1332 220 702

407

581

434 912

821 667 1223 476 954

95

289

367

1115 917 482

1193

120 213

11

1067

761 1158

1227

1187

54 513 1107

1023

1211 652

877 185 359 697 802

1175 1130

133 1063 880

251

777

910

873 1253

931 480 30 197

67 66 929 695 1068 4

1155 481 1221 1072

53 625

772

483 1029 459 1007

696

142

1073

1026

770

1034

56

293 1082

763

460

651 1083 681

712 1368 742

74

430

490

1071

1104 1226 1084

1002

713 427 437 329

47 1006

93

707 296

1230 141 75 1109 1318 470 1110

325 522

382 822

739

957 541

256 431 501 94 521 769 55 294

1111

330

1360

249

48

285

287

178

202

228

1179

1177 1276 334

1361

250

304

286

306

288

370

179

203

229

379

1180

357

335

518

305

766

307

662

526

385

389

425

416

371

380

358

682

691

519

543

546

767

549

527

386

918

390

923

708

683

692

1122

1124

544

547

426

743

764

550

1131

941

919

924

709

1294

1138

1123

1125

1132

1145

1149

744

765

1167

415

942

1366

1298

1295

1139

1146

1150

1306

1309

1168

1367

1299

1307

1310

723

744

872 810

547

873

599

601

710

743

772

774

779

722

871 542

598

267

270

280

600

272

284

638

442

444

731

159

113

855

760

709

1001

310

452

326

50

188

375

266

269

271

283

868

242

143

499

279

203 666

866

649 286

205

1041

118

438

674 467

362

703 660

183

162

290

909

841

867

384

538

436

275

204

1040

437

466

210

374

702

386

537 752

632

707

116

235 350

979

808

318

612

951

441 193 229

901

355

634

751

530

977

1035 181

701

981

289

167

201

176

202

228

861 428

900 334

678

241

579

1005

14

963

768

1008

260

191

427

724

800

816

166

539

529 259

470

8

518

994

357

980 257 985

742

517

258

860

778

741

885

754

976

694

424 753

153

305

200

975

693

423

335

63 501

152

304

548

249

814

151

513 196

105

248

1034

164

736 512 69

195

149

973 735 745

68

80

103

333

972

803

687

477 15

680

476

592

551

806

232

173 950

936

393

591 361

312

231

172

805

716

595

683

158

413

120

90

970

324

682

214

433

38

157

212 78

77

75

293

628

968 676

630

584

581 625

623

573

627

211

508

506

407

302 491

495

401

298 482

489

399

396

391

382

475

621

569

669

665

663

967

406

400

398

395

390

381

507

505

494

490

488

481

474

471

668

662

965

959

956 1042

952

947

941 1038

929 1030

926 1027

1019

922

848

801

664

1043

629

966

960

583

580 957 624

953

1039

622

948

942

572 930 1031

620

655

1028

1020

618

923

927

568

566

849

462

802

378

376

351

656

916

619

567

705

472

463

377

379

450

706

352

920

755

448

917

969

408

303

653 299

451

750

640

371

76

294

449

677

586

739

884

794

37

1015

48

914

27

852

832

49

642

464

502

128

925

28

469

840

409

720

155

594

522

372

430

915

83

607

643

585

434

938

215

422

523

608

129

611

325

108

431

156

410

96

504

788

1036

681

842

243

23

121

414

730

1018

540

91

84

141

208

184

837

60

783

109

394

142

263

924

222

875

609

92

220

983

571 268

20

870

715

5

971

300

24

1000

185

264

838

180

230

658

146

605

217

388

163

250

839 492

606

93

1033

207

21

320

6

301

179

418

626

815

122

65

902

1044

245

147

79

276

186

244

419

688

493

826

251

1003

81

29

425

988

718

311

906

189

321

25

39

426

66

526

339

746

935

992

94

699

420

11

30

893

140

54

402

637

169

82

59

907

812

58

26

125

40

9

403

857

690

712

35

903

329

691

262

95

12

974

561

55

785

150

177

738

556

555

858

879

10

828

261

255

727

36

904

937

106

692

161

459

199

525

535

342

897

524

766

224

178

859

747

792

343

256

509

831

1023

174

869

356

265

288

863

910

761

557

281

711

160

898

943

670

496

322

986

536

61

225

758

138

932

949

570

767

833

273

171

383

175

139

813

64

416

763

596

795

777

487

62

336

51

34

757

856

18

1

221

918

516

853

587

98

19

2

834

360

331

695

52

854

851

890

415

114

989

380

16

198

696

498

520

846

252

197

883

239

503

112

85

33

218

7

22

209

1048

99

392

510

369

945

737

330

528

238

562

115

990

667

44

165

933

511

588

749

689

104

604

253

483

661

934

697

89

439

17

219

748

708

908

1013

654

534

370

206

119

190

86

137

421

295

636

123

721

337

233

223

45

835

798

962

13

456

457

770

519

552

338

995

698

1024

314

756

527

358

110

987

1014

274

315

728

136

836

650

134

700

955

282

455

297

124

254

582

1016

593

765

148

521

799

684

359

500

130

216

484

323

645

73

479

850

111

719

465

192

1046

468

944

829

131

100

226

1045

543

67

87

440

411

135

545

675

905

1025

964

940

306

349

560

234

610

53

107

590

317

43

740

1047

984

862

47

313

732

865

1017 227

480

776

74

497

1029

576

344

713

807

830

790

672

285

991

679

88

704

46

899

577

389

1021

345

993

316

563

615

485

921

939

824

194

213

671

405

657

844

101

659

1026

277

954

635

822

56

1032

307

631

478

958

614

42

404

102

417

843

132

364

72

363

809

931

97

429

240

319

328

821

825

117

432

978

126

602

894

685

154

57

182

603

308

597

791

144

133

575

473

804

845

847

514

3

170

784

4

486

31

332

771

385

458

864

127

946

70

895

913

41

348

531

187

435

32

928

544

982

309

168

823

71

387

880

686

714

145

633

1049

412

532

717

278

574

578

613

373

644

764

1004

648

769

397

876

515

541

827

236

246

911

533

896

759

291

762

1002

729

641

340

346

912

793

919

589

1022

296

961

353

365

367

891

789

1037

673

446

546

237

453

460

549

553

558

247

616

287

292

646

327

725

733 341

347 651

564

773

775

780 781

443

445

447

454

786

354

366

368

892

796

461

817

550

877

881 554

886

819

559

565

888

617

996

639

998

647

726

1006

1009

1011

652

734

782

811

874

997

999

1007

1010

787

797

818

878

882

887

820

889

1012

(a) The Gnutella TDG (P2P application).

(b) The HTTPS TDG (client-server application). Fig. 1. Two TDG visualization contrasting a P2P (top) with a client–server application (bottom). Largest component is with bold (black) edges – hence the graphs are best viewed on a computer screen or a colored print-out. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

Table 1 Set of backbone traces from the Cooperative Association for Internet Data Analysis (CAIDA). Statistics for the TR-ABIL trace, are reported only for the first five-minute interval since IPs were anonymized differently at each five-minute sample. Name

Date/time

Duration

Flows

TR-PAY1 TR-PAY2 TR-ABIL

2004-04-21/17:59 2004-04-21/19:00 2002-09/(N/A)

1h 1h 1 month

38,808,604 37,612,752 2,057,729

TR-PAY1 and TR-PAY2 traces. Both traces contain up to 16 bytes of payload in each packet, thereby allowing the labeling of flows using the signature matching techniques described in [26,25]. Running the PC over the TR-PAY1 and TR-PAY2 traces we find 14% of the traffic to be P2P, 27% Web, 7% DNS, and the rest to belong to other applications. A detailed application breakdown is summarized in Table 2. In the table we further report the traffic information for the top six P2P applications with the highest

1912

M. Iliofotou et al. / Computer Networks 55 (2011) 1909–1920

Table 2 Application breakdown for TR-PAY1 and TR-PAY2 traces. Values in parenthesis show the percentage of each P2P application over the entire identified P2P traffic (All-P2P). Name

% in flows

% in bytes

% in packets

Gnutella eDonkey FastTrack Soribada MP2P BitTorrent All-P2P Web (80/443) SMTP DNS Games Unknown No-Payload Rest

0.95(6.78) 2.96(21.16) 0.55(3.92) 7.76(55.44) 0.41(2.93) 0.60(4.26) 13.85 27.45 1.65 6.65 0.71 0.97 28.06 20.66

0.17(1.59) 2.22(21.17) 0.74(7.10) 0.07(0.63) 0.01(0.14) 4.59(43.81) 9.19 41.99 1.10 0.32 0.54 1.98 0.46 44.42

0.81(6.32) 2.84(22.21) 0.97(7.61) 0.97(7.63) 0.07(0.53) 4.37(34.24) 12.10 35.51 2.431 1.586 2.84 3.22 6.04 36.27

number of flows in our data set. These six applications contribute 95% of the flows and 75% of the overall P2P bytes. In our traces, we also have flows by P2P applications that are not longer in use, such as WinMX, Direct Connect, Goboogy, Ares, SoulSeek, and EarthStation. We observed that some of these protocols were used for transferring very large files and contributed many bytes, but their overall number of flows was significantly lower. In our evaluation we include all P2P flows. For our study, we remove the 1% of traffic that remained unknown and the 28% that contained no payload. The small number of unknown flows is strongly correlated with the small number of encrypted flows in our traces. 2.2. Identifying P2P TDGs Identifying the right metrics to compare graph structures is a challenging question that arises in many disciplines [31]. Our approach is to consider several graph metrics, each capturing a potentially useful characteristic, until a set of metrics is found that distinguishes the target graphs. To select an appropriate set of metrics, we generate a large number of TDGs using all our traces (Table 1), thus observing TDGs over two different locations at the backbone. For the TR-PAY1 and TR-PAY2 traces, we use the payload-based classifier (PC) in order to select which flows belong to each TDG. Since the TR-ABIL trace does not have any payload information, we use port numbers [26] to assign flows to applications. We can use port numbers for the TR-ABIL trace since it was collected in 2002 where most P2P applications used their default port numbers [13,23]. We only use the TR-ABIL trace to verify our TDG observations over a second location in the backbone and we do not use it in the final evaluation of our classifier. By using the month-long TR-ABIL trace, we can study the consistency of TDGs over different times of the day and over weekdays and weekends. We observe TDGs over five-minute intervals. This interval length gives good classification results and stability of TDG metrics over time as we show later in this section. For each TDG we generate a diverse set of metrics. Our

metrics capture various aspects of TDGs including the degree distribution, degree correlations, connected components, and distance distribution. For additional details about these metrics we refer the reader to [19,31]. To select the right set of metrics we use various graph visualizations and trial and error. Finding a less ad hoc approach is beyond the scope of this work. Two TDG visualization examples are shown in Fig. 1. We see that FastTrack (P2P) has a denser graph than HTTPS, or a higher average  is given by degree, where the average node degree k  k ¼ 2jEj=jVj. We utilize two other metrics that capture the directionality of the edges in the graph and the distances between nodes. The directionality is useful since we know that pure clients only initiate traffic, pure servers should never initiate traffic, and that some P2P nodes play both roles. To capture this quantitatively, we define InO to be the percentage of nodes in the graph that have both incoming and outgoing edges. The distance between two nodes is defined as the length of their shortest path in the graph. The diameter of a graph is defined as the maximum distance between all pairs of nodes, which is sensitive as a metric [31]. For a more robust metric, we use the effective diameter (EDiam), which we define as the 90-th percentile of all pairwise distances in the graph. We show the high graph diameter of the BitTorrent P2P TDGs in Fig. 2. The graph represents the BitTorrent interactions between 3000 nodes from the TR-PAY1 trace. The visualization is suggestive of a spider’s web with large paths between nodes at opposite edges. This intuition is precisely captured by the large value of the effective graph diameter. In general TDG visualizations are suggestive, but the metrics make the intuitions precise and allow classification. From our measurements, we empirically derive the fol > 2:8 lowing two rules for detecting P2P activity. Rule 1: k and InO > 1%; Rule 2: InO > 1% and EDiam > 11. With these simple rules, we can correctly identify all P2P TDGs from both backbone locations (Abilene backbone and Tier-1 ISP). Intuitively, P2P hosts need to be connected with a large set of peers in order to perform tasks such as answering content queries and sharing files, which can explain the higher average degree compared to client–server applications. An additional characteristic of P2P applications is the duality of roles, with many hosts acting both as client and server. The duality of roles is in turn captured by the high InO value. We further speculate that the decentralized architecture of some P2P applications (such as BitTorrent), can explain the high diameters in some P2P TDGs. We do not claim that our thresholds are universal, but our measurements suggest that small adjustments to these simple parameters allow our methodology to work on different backbone links. Distinguishing collaborative applications from P2P: Some well-known applications other than P2P exhibit collaborative behavior, such as DNS and SMTP. This is not surprising since in these applications servers communicate with each  and servers act both other and with other clients (high k), as clients and servers (high InO). This is exactly what our metrics are set out to detect. Next, we outline two

1913

M. Iliofotou et al. / Computer Networks 55 (2011) 1909–1920 2042

1906

2885

2060

3213

3654

2706

1498

3512

3227

3563

3006

3646

3174

2066

3511

1243

3501

3498

3497

3490

3489

3487

3486

2050

2620

2614

2613

515

3483

1001

3482

3473

2592

2591

3472

3467

3466

3040

2043 39 2011

2953

2238

2237

2949

2948

2947

2946

2276

2274

2631

1592

2211

496

980

1331

1757

2630

1782

2593

1278

2590

2589

2588

3217

1799 846

3294

1737

2227

2788

1347

3386

1156

3263

516 2130

741

2975

2974

2983

2982

704

2116

2115

1708

1707

2586

3340

2743

3289

270

2529

2212

1941

2585

857 38

1923

2918 362

2275

1884

1783

1114

1738

688

1876

2242

2391

2124

2677

2629

3240

140 1064 1348

1293

1279

1157

1676

1675

2633

838

2815

2579

2578

2379

3038

742

705 839 956

2813

1504

180

2122 1169

1281

136

88

1924

2173 2990

2989

363

2676

1054

2628

271

1636

1546

1115

2028

2109

1414

385

1705

1305 1200

723

141

1394

726

3427

1393

2026

1065

3008

2110

2700 957

1647

26

1375

932

3007

1168

3454

25

622

3453

1908

1280

3471 1306

3255

771

1646

3525

137

1075

1275

3378

722

1055

87

89

1127 1342 808

131

52

1128 384

1407

3010

3009

2608

3448

3447

124

809

3371

2323

2781 1467 2966

3298 1890

2727

53

1744

1704

404 2680

3146

1044 1363

3449

1814 2504 2351

3530

2731

3544

3543

3551

3550

1340

1339

1299

62

3444

2941

2123

2624

3443

2047 2715

1191 2157

1408

132

2132

853 3606 1059 856 155

3587 2716 3162

1028

396

401

3436

3435

3018

637 293

1349 1188

2695

520

1639

1406 2182 2315

434

1259

1629

3313 2147

2693

2718 146

973 3417

854

1082

130

1686

2574

2573

2559

2558

1007 3576

2065

1677

397

2104

156

519

1638

1058 855

1390 982

145

3228

1107

255 3570

892

3569

2846

1173

1172

1165

1164

294 2454

121

2004

400

2155

1102

518

2399

654

46

1100 1049

2770

1048

1443 1537 1006

357 3577

1953

129

2064

1106

2236

2235

1389 587

58

504 3492

1310

2270 221

1118 1770

28

24

47 3390

517

419 1514 254 1557 1921 2400

3375

1927

274

1180

1125

655

1672 1518

2322 2005

2836

2778

967

29 405

59

2051 220

815

2094

122

543

2534

2533

23

588

356

1062

1266

361

111

1886

225

1311 2370

2250

2249

1309

1070

27

930

163

2168 966 1485 803 1061

1060

2059

923

922

2015

275

3613

1440

2523

816

3176

1928

2643

406

2881

914

224

2522

444 1124

773

1004

832 2644

913 944

273

1490

2450

473 534 288

2734

1267

772

888

2515

2514

452 112

2239

287

1286

1897

3395

1784

1618

3585

3584

3593

3592

868

867

185

453

2366 802

3309

1690

309

1095 3178 998

2079

3437

1529 2344

833 1005

1530

2088

2087

2082

2081

533

1962 770 2823 3214

3002

615

700

3080 992

3072

3419

1473

1288

3133

1815

1645

2785

1477 3468

2616

756 3337

985

984

1508

3539

663

987 747

7

11

3179

776

465

1777

454

310

1538

3224

480 2220

3601

3600

3362

3015 2474

474

1596 3187

1591

481

68

8

1336

1619 2747 3132 989

988

799

699

798

286

2615

2935

1533

2926

2934

769 69 2744

738

2849

3225

761

760

2933

2610

451 234

633 3215

12 1610

2401

455

1507

662

240

1388

1899

1813

1534

2925

1462 1867

324 3014

1476

2959

505

1620

3123

2075 678

285

1595

677

1594

2705

680

3252

2845

2455

2748

2447 1033

1632

679 2177

960 2061

2062

3609

2353

3612

3611

682

1480

1898

573

681

3231

1812

1811

1785 3614

1881

447

450 1896

3124

3145 2780

634

1902

262

844

843

1249

1248

3045 1727

1739

2494

942

601 2213 238

1034 2980

1680

1631

337

442

3186

2503

2502

2021

599 2449

560 559

2174

3098

3272

2348 862

2567

472 3586

532

2311

1669

1422

2283

1901 380

1719 2564

2726

602

381

1959

2510

551 443

1588

421

600

572

598

566

565

1671

1670

3404

2001

1531 1766

1853

2437

1341

1585 1681 3005 531

1587

298

647

420

2464

1710 2264

471 2428

2957 861

782

547

546

1874

2057

417

2056

3567

2438

1578

959

2623 18

91

1987

931

1579

3588 2565

552

438

437

399

398

3171

754

753

529 2855

1523 1586

378

2789

1765

1141

2427

2928

2927

379 17 376

2263

3517 2958 3023

3022

3025

3024

2302

1939

1088

648

1087

2930

1709

521

997

716

902

1558

3655

2657

2754

236

958 1417

90

596

595

1418

841

330

840

1789

1788

1780

1779

703

1424

1622

1319

483

329

970

429

969

237

2810 3109

1412 996

1612 430

545 748

544

1758

1327

377

3573

1433

3027

1067 667

1303

3026

292

2746

291

1068

1506 2921

811

968

3622 1320 1261

1260

3523

284

283

1161

3403

462 2825

2092

2136

3222 2472 1520 2884 113

3616

1016

3615 3496

2225 1519

3459

730 216

1977 1469 1295

1326

577

594

1434

249

248

245

244

3402

3401

3011

3583

2073

3241

2317

1826 2462

3481

1017

463

983 3122

2659

609

606

448 578

829

501

1186

2178

2113

1752

3383

3575

3382

338

1404

114 881

217

1415

1405

593

339

910

1296

909

1236

1235

1743 2156

1742

493

2252

459

500

3623

2251

135

134

3251

340

214

3380

1936

714

2690

3477

2330

2074

513

3112

2681 1177

715

3147

610

1751 828 154

95

3242

2542

528

476 2669

2668

2671

2670

870

1864

3282

2486

2485

2568 630 1869

241 403

898 2689

458

3148

875 1847

402

876

104 2720 3306

3568

1427

514

642

2496

96 477 103 2463 1277

153

1940

1051

2814

1868

2831

641

1193

920

2911

2910

2900

2837

2902

2901

64

2916 2152

2673

2267

2672

3157

3518

2868

3041

3548

1593 495

762

3653

1964 2044

1197

200 1750

2188 2291

1192

3432

2863 2685

2189

1491

2686

35

34

1163

101

1945

583

389 2571

1134 1590

2398

2032

494

206 2151 431

921

1946

2646 1094

2490 65

2554

3029

3028

2443

763

3344

2430

1492

3470

1354

1135

1572 1411

3199

817

308

307

3149

1699

2229

2228

1252

388 439

2117

584

345 1423 1225

3627

3626

3629

3628

3633

3632

3643

3642

1589

6

2745

3032

205

2397

1015 207

3158 109 60 1152

3374

2327

1580

2484

1952

2483

2300 905

971

1050

1573

2808

5

2246

2497

110

1345

1344

1253 684

3167

346

818

3336

2161 344 3269

31

2923

1877 3475

3479

800

3059

3617

3311

2476

2475

1276 1184 1358

157 2306

1297

183

2804 1176

1513

208

139

3594

159

2165

1653

1895

61

1382

706

333

3634

10

638

176

2069 3161

30

1762

972

639

1761

1047 1982 3508 3491

3

2728

2114

181 2736

3638

175 2260

460

1460 1181

57

508

1468

1359

158

3182

2971

751

1195

935

1911

934

1495

1823

1494

3076 1425

807

3610 707 1493

160

1912 3619

2894

2893

2699

1410

9

1461 138

1346 530

1983

56

1111

1957 946 182

1372

464

1455

2126

3097

507 2030

2662

71

2247

3192

2018

2331 1196

2208

965

280

964

2888

3141

2887

813 1024 1022

243

2835

1409

392

736

1837 70

1233

1797 659

737 607

281

393 394

812

2688 3637 801 2107

2709

2883

1037

2108

2691

824 242

2684

2226

806

1160

608 1201

2793

2809

278

2995 2505

860

1845

152

2048 1401

1083 1768

2943 3229

1958

1512

3207 1846 1505

2800

695

694

1202

2872

2871

1723

1023

1822 2932

825

1657 1240

440 395

3560

2742

1833

2193

1817

1254

2363 1428

2905

2904

797 2759

3042

260

629

1973

628

1665

2223

2222

1605 2599

1693

1661

1445

3183 408 3636

1194

1606 2507

2801

1656 1767 167 2245 2392

1241

1875

2072

2459 1444

3305

1913

994

3421 261 2240

1521 1140

1722

2040

2039

1660

3134

2192 1972

3343 3036

3035

251

250

20

19

1522

168

871

1432 2886

1026

2506

796

1511

661

1996 1368

691 872

470

2493

2358 1703

314

2896

3259

1384

1234

1698

2149

355

539

2880 2917 2456

424

2460

187

186

98

97

3377

2167

3376

2641 491

3356

3064

625

1215 118 3087

669

668

467

1273

926

2261

466

259

268

2687

232

1844

1397

624

2280

2617

1745 3519

3253 2694

835

1697

1999

1112

354

605

538

3349

192

1385 40

1145 690

418

1992

2994

1231

692

2762

1222

1232

1221

2148

660

585

3220

1900 537

536

1270

2618

3405

3529

1966

2224

1604

1730 3218

3052

41

3051

1204

3347

2906

2664

233

1849

2769 1313

2248 2217 269

1000 2761

2478 94

1426

2090 117

1076

2929

2461

258

2553 877

15 3054

2203

3314 341

3504

3055

561

211

2531

37

2204

1220 1109

1010

2287

2663

2730

3117

2433

586

3221

2469

1827

744

315

2184 999

213

335

2798

368

1717

100

99 449

2183

1891

360

570

3115

50

953 2372 2290 1398

2704

2703

556

2076

555 3335 2329

1085

869

1623

963

2457

651

1778

1413 1144

698

312

711

710

1547 2944

3334

3333

2991

342

36

3393

120

1903

3070

1219

2083

209

3118

3173 1294 2314

1861

548

353

1644

3431 210

2848 977 2920

1441

306

1947

1892

743

2350 1532 3058

3057

256

2384 334 3101

2230

2000 383

391

426

1624 2721

44

2162

2601 890

1482

3455

562

93

569

708

1380

371

370

1466

1625

1465

2436

2435

1621 3571

390 794

479

2826 1330

3061

1948

51

1086 2919

904

3003

2757

305

228

1338 3502

3060

903

457

673

3033

382

2067

1383 3039

2717

576

2912

162

119

190

2285

2442

2570

456

229

3412

1626

189

2052

2181

952

2625

3439

13

195 2640

2931 203

2328

161

1

2063

1041

2143

1942

945 672

3063

3062

3067

3066

580

874

2636

372

1198

3327

3326

478

823

1133

246

2842

204 1329

851

822

522

1092

720

1679 1373

3169

2070

1949

226

3170

883

194

2652

3414

1502 804

746

745

1652

3318

943

414

427

2760 267

191

3086

3528

3082

1039

884

2879

2281

1040

3325

3324

721 579 1206

374 525

14

247 1484

92 3111

597

2468

2146

1244

74

1374

2416 125

2576

2889 2091

1283

413

266

375 1561

2550

54

827

826

1635

171

1746

144

1855

1986 2071

3545

1205 1678

805

2577

852

373

73

1187

3323

3322

301

2185

2272

75

1247

1860

1963

1535

116 895 492

2389

1716

986

3034

1314

2725 2390

343

2013

3056

302

1772

784 3400

3652

612

3596

611

674

2552

3281

3075

2415

1673

143 1781

468

2772

2419

2361

133 3151

2867

1798

1854 115 1179

2499

2129

3250

759

3194

671

3461

3065

2807

2362 1078

1178

1701

55

1369

750

147

1486

764 1370

2420

3495

2258

482 2940 1282

2937

670

1077 1155

178

1149 3233

2936

1158

718

1526

1791

783

2031 820

1648

810 1429

2862

222

780

1143

2584 845

2811

2861

2277

1302

1568

2166

1096

3004

148

1920

223 786

387 2833

2445

2575

1536

469

2619

3053

1599

1497

3206

785

1362 788 732

3540

1081

2903

749

2068

1436

3360

3268

2465

1614 2954 2771 765

758

656

2517

1159 719 2383 618

2674

1148

617

1848

1142 2365

1696

1976

2207 1470

202

2278

177 366

2812

2086

787

2446

3285

169

2913 1008

2100

2956

1021

766

2543

1119 1794

2945

563

1792

2555

1121

650 2029

3205

735

791

734

1069 675

1437 165

657 1615

1642

2832 367

1079

2859

2714

2713

2241

1773

1032 386 1598

2145

821

3155

1002

1056

3321

3320

3301

729

2431

1353

1151 1459

1666

2914

3410 2284

2963

1667

1052

166

1029

961

1117

142

1150

2012

3413 3463

2190

3181

3074

3073

564

859

858

1663

2095

1419

3267

1392

410 3140

2422

2797

127 1333

792

649

2796

2214

2096

676

2084

3566

757 1515

911 1131

1263

1053

1379

2765 3532

550

1655

962 2257

1080

1400

2256

3096

409

1003

2432

2596

1019

1018

2023

2022

1332 2163

2572 2560 350

2035 3649

850

170

849 2877

1352 2221

511 1479

1147

2508

2041

1458 2675

3079

3078

3077

1350

3089

3088

3091

3090

1020

1285

1284

2852

1167

2851

1933

793

1689

2305 2847

2219

1643

948

524

1961 3331

2262

303

2875

2135

2547 1378

697

2444

1421

1230

2556

2144

1654 78

2279

3513

3257

2683

128

2367

3630 2255

1207

1300

2045

3308

3139

2563

2834

2034

1301

2639 349

3484 3016

3017 1171

3232

549

557

3291

1123

512 289

3526

2298

696

523

2137 1613

1229

603

1166 263

1965

2418

2417

1843

2412

2404

2403

1878

2544

1255

2737

2792

2539 2583

900 3507

908

3372

941 2600

940

2259

1775

635

1218 2876 2180

2053

2254

2289

3128

929

928

955

954

317

2480

1925 1979

1544

1728 3142

1031

950 1356

322

150

894

3416 1915

3527

2099

1994

1035

2985 1971 778

3127

43 32

2312 901

3180

1499

3254

42 1036

81

1748 2054

276

2355

1694

2866 318

3049

321 3188 1909

277 2612

80

423

1980

1132

951

2017

2016

2978 1804

2702

1183

3172

3303

1873

2282 1483 33 1803

3354

1138

2102

422

2434

1545

2858

2609

1387 2191 896

1209

1208

3193

149

2206

2080

2205

1889

1471

3476 2645

1105

3346

1664

3559

949 82

2635

3384

1747

1214

1182

1093

3595 107

2101 1449 2821

1937 2951 108

2532

1386

1269

1268 3524

2528 2724 2243

2304

1754 3631 702

3302 2458

3236

2557

2338

2196

1517

3288

3287

252

2414

1116

219

574

83 174

978

2105

1013

1879

1103 2197

2537 1850

2648 3366

173 2952

701

1753

979

1014

1104

1308

1307

3280

3279

1720 1706

2402

2332

590

2869

2696

621 3012 1721

218 1258

1692 2118

3297

3154

1323

3635 866

939

937

1829 3105

3104

3505 3500

1315

3648

2660

2265

2860

575

3278

3277

199

1830

1265

296

264

2538

589

620

2511

3189

3415

645

789

3203

2530

2753

2752

2764

2763

1760

2121

865

2170

938

2308

936

3013

2394

2393

2388

2387

731

687

1700

3130

3565 915

1396

1395

1403

924

1402

1431

1430

1447

1446

1628

646

1223

790

2964

1224

1553

1602

265

1759

2120

3350

1464

1463

1324

2169

2307

1316

2967 2055 3129

1819

1541

1540

1543

1542

3564

1818

1551

1550

1556

1555

1563

916

925

1712

1562

1802

1711

1584

1488

2965

1583

1582

1581

3204

1597

990

1608

1607

1634

1633

1641

1487

1640

1650

1649

2200

2199

709 554

327 885 1825

1810

1824

1840

1839

1866

1865

1872

1871

2319

2318

2321

567

2325

2324

1554

3160

3159

3164

3163

831

3168

1888

2119

2139

2138

2160

1887

1603

1715

2159

2968

1917

1714

1732

1916

1919

1918

1930

1929

1932

1931

1956

1955

1968

1967

1989

1687

1988

1998

1997

2008

2007

2818

2817

2899

2266

1805

2310

2309

2175

2791

2790

2195

2194

2202

2201

1926

428

2767

2176

2766

2774

2773

2787

2786

2334

2333

3191

3190

2336

2335

2024

2025

2347

2346

2360

2359

2369

2368

2375

2374

2378

2377

2382

2381

2386

2385

1435 553

328 3138

3137

3144

3143

499

1943

2795

2794

1617

2802

412

3197

3196

3202

3201

3212

3211

3223

2999

3244

3243

3247

3246

3261

3260

3271

3270

3275

3274

2602 2605

411

2603

Fig. 2. Graphical representation of the BitTorrent TDG with 3000 IPs, showing the formation of long paths connecting P2P hosts. The largest connected component is highlighted with darker color edges.

2.3. Practical considerations for TDGs 2.3.1. Stability of metrics over time If thresholds derived from TDGs changed significantly with time, then the classifier must be trained constantly and that would detract from its value. Fortunately this does not appear to be the case. We show the stability of TDGs in time and space using traces from different points in time at two different backbone locations for Abilene and a commercial Tier 1 ISP (Section 2.2).

To test stability over time, we measured P2P TDGs from our longest trace (TR-ABIL) that span over one month and include samples taken over day and night hours, and both weekdays and weekends. To the best of our knowledge such month-long traces with payload are not available. Stability was also observed in all the one-hour long traces (TR-PAY1, TR-PAY2). Fig. 3 summarizes the stability study for TR-ABIL showing the average degree metric. As we see, average degree takes values in a small range over the entire month. Soribada tends to have very small graphs ranging from 30 to 1000 nodes and hence the higher variability. Stability was also observed for the diameter and InO metric with the figures not included for brevity. For this study, we used TDGs representing five-minutes of traffic. In Section 2.3.2

Average Degree

approaches that can help to separate between collaborative applications (e.g., DNS) and P2P: (1) One solution is to rely on port numbers or payload information for this task. In fact, it has been reported recently [26] that port numbers are fairly accurate in identifying such legacy applications, although they fail to identify P2P and other applications with dynamic use of port numbers. Therefore, one could use legacy ports to pinpoint and isolate such collaborative applications and then use graph metrics on the remaining traffic. In addition to port inspection, we can also examine the payload of a flow in order to verify that it follows the expected application-layer interactions. (2) In a recent work on dynamic graphs [16], we show that measuring how much TDGs change over time gives additional information about the nature of the applications. In particular, we observed that P2P TDGs are more dynamic (i.e., how frequently nodes and edges change over time) than legacy applications, such as DNS. If the classification problem at hand requires this further separation, we can incorporate such dynamic techniques in Graption.

30 28 26 24 22 20 18 16 14 12 10 8 6 4 2 0

min = 23.73

0

Gnutella FastTrack Soribada

WinMX

eDonkey

MP2P

Fig. 3. The average degree for various P2P protocols over one month in trace TR-ABIL. Candle sticks show the maximum and minimum recorded values together with the average (horizontal line) and ± the standard deviation. For visualization purposes, for the MP2P application we show only the minimum value which is very high (23.73).

1914

M. Iliofotou et al. / Computer Networks 55 (2011) 1909–1920

we show experiments with TDGs where we vary the observation interval from five seconds to 15 min. Note that BitTorrent is not shown in Fig. 3 since it was not well-known in 2002 and resulted only in a handful of flows in the TRABIL trace. 2.3.2. Selecting the interval of observation The effect of the observation interval on the graph metrics. We observe that by increasing the observation interval, graph metrics are more stable across successive TDGs of an application. To show this, we measure the variation of all our graph metrics over time as a function of the interval of observation. We vary the interval from one second up to 15 min. For each interval length, we generate consecutive graph snapshots with non-overlapping time intervals. For each interval, we extract the graph and calculate the average degree, InO, and effective diameter metrics. We examine the variability of each metric over all intervals of observation. We use the commonly-used range metric, which is the difference between the maximum and minimum observed values over all intervals for each graph metric. For each metric, we then normalize the range by the average value calculated over the time series. This normalization makes the range for different metrics somewhat comparable. For each application, we report the average range over all three graph metrics, and we plot this average range versus the duration of the observation interval in Fig. 4. Our experiments showed very high range values for intervals smaller than 60 s. The range shows significant decrease for intervals larger than a minute. Increasing the interval from five minutes to 10 and 15 does not significantly effect the range. This trend is shown in Fig. 4, where we report the average range over all metrics for a set of our applications. Even though we have variability in our results, there was a clear trend in all our measurements. Similar trends we observed in all our traces (Table 1). For the rest of our study, we use five minute intervals. 3. The Graption framework The Graption traffic classification framework consists of the following three steps. 2.5

DNS EDONKEY FASTRACK GNUTELLA HTTP NTP POP3 SMTP MP2P

Variation Range

2

1.5

1

0.5

0

0

100

200

300

400

500

600

700

800

900

Interval Length (sec)

Fig. 4. The effect of changing the interval of observation for TDGs ranging from five seconds up to 15 min over a large set of protocols. To reduce variability we can choose to use longer intervals. After 300 s the reduction in variability is not significant.

Step 1: Flow isolation. The input is network traffic in the form of flows as defined in Section 2. The goal of this first optional step is to utilize external information to isolate any flows that can already be classified. This knowledge could be based on payload signatures, port numbers, or IP address (e.g., exclude flows from a particular domain such as google.com). Step 2: Flow grouping. We use similarity at the flow and packet level to group flows. The definition of similarity is flexible in our proposed methodology. We can use flow statistics (duration, packet sizes, etc.) or payload if this is available. Eventually, the output of this step is a set of groups with each group ideally containing flows from a single application (e.g., Gnutella, NTP, etc.). However, at this step, the exact application of each group is not known. Step 3: Group classifier. For each group of flows, we construct a TDG. Next, we quantify each TDG using various metrics. The classifier uses these metrics to identify the application for each group of flows. For the classification decision, we use a set of rules which in general depend on the focus of the study. Although this paper focuses on P2P detection, we believe Graption can be used for general application classification by choosing metrics and parameters appropriately. We next describe how we specialized Graption to detect P2P traffic (Graption-P2P).

3.1. Implementation details of Graption-P2P Step 1: This is an optional step in our methodology. Experiments without this step are discussed later in the section. Recent work [26] suggests that port-based classification works very well for legacy applications, as legacy applications use their default ports and tunneling of P2P at such ports is not very common. Thus, in this study, we isolated flows with port 80 for Web, port 53 for DNS, and port 25 for SMTP. These applications turn out to be about 65% of the total number of flows. In our traces, the proportion of P2P actually using one of these ports is as low as 0.1%. Step 2: To implement flow grouping we use the fact that application-level headers are likely to recur across flows from the same application. Therefore, payload similarity can be used to group flows. In Graption-P2P, we only use the first sixteen bytes from each flow. As we show, sixteen bytes are sufficient to give very good classification results. This observation agrees with findings in [30,28]. Even though we use the payload bytes, our grouping is agnostic to application semantics, as each byte is considered as a single independent categorical feature. We consider each byte as a single categorical feature in the range {0, 1, . . ., 255}. The flow grouping step comprises two sub-steps: cluster formation and cluster merging.

1915

M. Iliofotou et al. / Computer Networks 55 (2011) 1909–1920

Step 3: The outcome of the previous step is a set of groups of flows, with each group consisting of flows that we hope stem from a single application. In order to classify each group, we generate a TDG on the group in the same way as described in Section 2. Each group yields a TDG that can be summarized using graph metrics. To identify P2P TDGs, we used the rules extracted from Section 2.2. When a group is labeled as P2P then all the flows of that group are classified as P2P flows.

3.2. Evaluating Graption-P2P To evaluate Graption-P2P, we use traces TR-PAY1 and TR-PAY2, where we have the ground truth using the payload classifier (Section 2). We compute the True Positives, False Positives, and False Negatives. The True Positives (TP) measures how many instances of a given class are correctly classified; the False Positives (FP) measures how many instances of other classes are confused with a given class; and the False Negatives (FN) measures the number of misclassified instances of a class. In our comparisons, we used the following standard metrics: Precision (P), defined as P = TP/(TP + FP); Recall (R), defined as R = TP/(TP + FN); and the F-Measure [38], defined as F = 2P  R/(P + R), combining P and R. In all our results, the metrics above report the classification performance in terms of flows. 3.2.1. Forming clusters The K-means algorithm generates k clusters and assigns each flow to a different cluster. We first test the effectiveness of K-means to form pure clusters. A pure cluster will ideally contain flows from a single application. To label the clusters, we use the dominant heuristic [3]. In this heuristic, using the ground truth of flows we label each cluster as belonging to the application with the majority of flows in the cluster. All the flows of a cluster are then classified to belong to this dominant application. Using the dominant heuristic we assign clusters to applications and calculate the precision (P) and recall (R) for different values of k. The clustering results as we increase k for both traces are shown in Fig. 5. We observe that with sufficiently large k (>120) we achieve very good results with P and R above 90%. By increasing k, on one hand we have more ‘‘pure’’ clusters but on the other we make the cluster merging step harder. In the extreme case, each cluster will contain a single flow giving 100% P and R, but making cluster merging challenging. We will return to this topic later in Section 3.5. The Graption-P2P classifier is not sensitive to the exact value for k, since we do not require that each application maps to exactly one cluster. Instead, we only need enough clusters so that flows from different applications do not share a cluster. Hence, it is reasonable to slightly overestiClustering Performance for K-means(%)

(a) Forming clusters. Given the set of discriminating features, the next step is to cluster ‘‘similar’’ flows together. We use the term cluster to describe the outcome of an initial grouping using the selected features. Clusters may be merged in the next function of this step to form groups, which produces the final output. Feature-based clustering is a well-defined statistical data mining problem. For this task we used the popular K-means algorithm [38]. This algorithm has been commonly used for unsupervised clustering of network flows [10,30], with very good results and low computational cost. K-means operates with a single parameter that selects the number of final clusters (k). As we show later in our evaluation, our classifier gives very good results over a large range of k. The similarity between two flows is measured by the Hamming [38] distance calculated over the 16 categorical features (i.e., the payload bytes). Even though more involved similarity measures such as edit-distance (also known as Levenshtein distance) exist, Hamming distance has been used successfully before [15] and performs very well in our application. (b) Cluster merging. During clustering, it is likely that the same application generates multiple clusters. For example, many P2P protocols exhibit a variety of interaction patterns, such as queries (UDP flows) and file transfers (TCP flows), each with significantly different flow and packet characteristics [23]. This motivates mergings clusters that we expect to belong to the same application into groups. This grouping provides a more complete view of the application and aids in understanding the structure of the P2P protocol, as we show in Section 3.2. Cluster merging cannot be based on the chosen set of flow-level features that were already used to create the clusters originally. Instead, in the case of a P2P protocol, it is natural to assume that the TDGs corresponding to each cluster of the same protocol would share a large number of common nodes (IP addresses). Based on these observations, we use an Agglomerative (Hierarchical) Clustering Algorithm that recursively merges clusters with significant similarity in IP addresses. We used the following metric to calculate similarity between clusters: Sim(C1, C2) = (Number of flows having their source or destination IPs present in both clusters)/(The number of flows of the smaller cluster). The cluster merging process starts by hierarchically merging clusters with high similarity and stops when the similarity between all new cluster pairs is below a similarity threshold (ST). As we show later in our evaluation, our classifier gives very good results over a large range of similarity thresholds.

96 94 92 90 88 86 84

Precision (TR-PAY1) Recall (TR-PAY1) Precision (TR-PAY2) Recall (TR-PAY2)

82 80 0

40

80 120 160 Number of Clusters (k)

200

240

Fig. 5. Evaluating K-means: With sufficiently large k (>120) K-means can efficiently separate the flows of different application. All classification metrics are in terms of flows.

1916

M. Iliofotou et al. / Computer Networks 55 (2011) 1909–1920

mate the k. As we show next, with k in the range 120–240 Graption-P2P has high classification performance. 3.2.2. Identifying P2P traffic Using Graption-P2P, we achieve high flow P2P F-Measure over a range of values of k (K-means) and similarity thresholds (ST). We show this in Fig. 6, where we vary the ST from 0.01 to 1 and use a sufficiently large k (see Fig. 5). All experiments are averaged over each disjoint five minute interval of both traces.Intuitively, by using a very large ST, the clusters of an application are not grouped together, which results in TDGs that are harder to classify as P2P. On the other hand, with a very small ST, clusters belonging to different applications are merged together leading to poorer classification performance. The results in Fig. 6 show that we achieve good classification performance (>90% F-Measure), over a large range of similarity thresholds and number of clusters (k). In Fig. 7, we compare our approach with labeling each cluster using the ground truth (i.e. without merging any clusters and labeling each cluster using the dominant heuristic). Intuitively, for a given clustering of flows, the ground truth shows the best that our cluster labeling mechanism can achieve. For merging, we use a ST of 0.5. From Fig. 7, we see that Graption-P2P deviates only slightly from labeling clusters using the ground truth. In the same plot, we also compare Graption-P2P without the cluster

F-Measure of Graption-P2P (%)

100

k=120 k=160 k=240

95 90 85 80 75 70 65 60

0

0.1

0.2

0.3

0.4 0.5 0.6 Similarity Threshold

0.7

0.8

0.9

1

Fig. 6. Graption-P2P achieves > 90% flow F-Measure over a large range of similarity thresholds and number of clusters (k).

merging step, highlighting the benefit of merging clusters of the same application together. Using a ST of 0.5 and k = 160, Graption-P2P achieves above 90% Recall and above 95% Precision over all disjoint five minute intervals for both traces. To apply GraptionP2P to other backbone links, the same selection process can be repeated to adjust the values of ST and k. Our experiments show that the classification performance can degrade with a bad choice of parameters. However, as shown in Figs. 5–7, for reasonable choices for k and ST, our method provides very good results. The Graption framework is flexible enough to incorporate state-of-theart clustering methods. 3.3. Comparison with BLINC [25] We used BLINC to classify traffic for both TR-PAY1 and TR-PAY2 traces. BLINC was optimized after several trial and error efforts to achieve its best accuracy for classifying P2P traffic over these traces. A detailed description of the selection process for the parameters for BLINC is provided in [26]. As reported by Kim et al. [26], tuning the 28 parameters of BLINC is a time consuming process. This is because small changes to the thresholds have a significant effect on the classification results. The Recall and Precision for BLINC are 84% and 89% respectively compared to 90% and 95% of our method. The classification performance per P2P protocol for both BLINC and Graption-P2P are shown in Fig. 8. Given that both methods do not distinguish between particular P2P protocols (e.g., Gnutella vs BitTorrent), it only make sense to report the portion of identified traffic (recall) for each protocol. Graption-P2P can identify more traffic for all P2P applications tested with the exception of MP2P. We can see from Fig. 3 that MP2P has a much higher average node degree than the other P2P applications which can explain the high performance of BLINC over this protocol. Going back to Fig. 8, we see that BLINC has significantly lower performance for some P2P applications. For example, Graption-P2P detects 95% of BitTorrent traffic, while BLINC detects only 25%! In addition, Graption-P2P detects 91% of the flows from the three popular P2P applications {BitTorrent, Gnutella, and eDonkey} while BLINC detects only 73%. Our experiments suggest that BLINC and possible other behavioral-host-based approaches work well when applied Graption−P2P BLINC

95

100

90

80

85 Ground Truth Graption-P2P: Merging Graption-P2P: No Merging

80

Recall (%)

F-Measure of Graption-P2P (%)

100

75 70 65

60 40 20

60 80

100

120

140 160 180 Clusters (k) in K-means

200

220

240

Fig. 7. Graption-P2P compared to cluster labeling based on ground truth. Results are also compared with and without cluster merging.

0 BitTorrent Gnutella eDonkey FastTrack MP2P Soribada

Rest

Fig. 8. The percentage of P2P flows detected by Graption-P2P and BLINC. Flow precision for Graption-P2P is 95% and for BLINC is 89%.

M. Iliofotou et al. / Computer Networks 55 (2011) 1909–1920

at the edge, where a large fraction of host flows are observed and hence enough evidence is collected to profile each node. However, this is not always true for backbone monitoring points which can explain BLINC’s lower performance. These observations are also supported by findings in [26]. We acknowledge that BLINC provides a methodology for detecting traffic from multiple classes of applications (e.g., HTTP, FTP, Chat, etc.) and not only for P2P. Our results only show that Graption-P2P does better in P2P detection. 3.4. Identifying the unknown Our goal here is to answer the following question: Can Graption identify ports used by unknown P2P applications in our traces? To achieve this, we configure Graption as follows. For step 1, we isolate legacy flows using the same approach as in Graption-P2P. In addition, we isolate flows from ports that belong to known P2P applications (e.g., port 1214 for FastTrack). At step 2, we cluster flows using their port number. This produces one cluster for each distinct port in the trace. For merging clusters, we use the same similarity threshold as in the Graption-P2P configuration. Finally, for classifying each group (step 3) we use the same rules as in Graption-P2P. We will refer to this configuration as Graption-P2P-Ports. By manually inspecting the TDGs that Graption-P2PPorts reports as P2P, we discovered interesting behaviors in the Abilene Trace (TR-ABIL). Our simple method was able to detect three ports (2002, 4156, 1978) that we initially thought were false positives. On further research, we found that these three ports were used by the Slapper2 worm. Slapper has been reported to use these ports in order to form a P2P botnet overlay for launching DDoS attacks. Even though Graption-P2P-Ports was never explicitly trained to detect the Slapper overlay, it used the TDG profile of known P2P applications (e.g., eDonkey, Gnutella, etc.) to identify it. This is an advantage of using network-wide behavior to classify traffic. Graption-P2P-Ports also captured an unusual behavior of IPs executing traceroute to each other forming a near clique. Such highly connected group of hosts was not observed in any of the P2P applications present in our traces. We speculate these interactions are by host performing active network measurements inside the Internet2 (Abilene) backbone. Given this unique connection pattern, we can always isolate such unusual behavior and distinguish it from a potentially new P2P application. Besides the active measurement traffic and the flows from the Slapper worm, Graption-P2P-Ports reported only 0.3% other flows as P2P. With no payload information, we were not to able to manually match these flows to a known exploit or P2P application. 3.5. Other Graption configurations For completeness, we show here preliminary results with other system configurations assuming that access to payload is limited or isolation cannot be used. We proceed by describing these configurations. 2

http://www.cert.org/advisories/CA-2002-27.html

1917

Graption-P2P-NI: without isolation. This is the same configuration as the Graption-P2P, but without using isolation. By increasing k in K-means above 300 we achieve good results; >92% precision and >85% recall. A larger k was to be expected, since by including all the Web, SMTP, and DNS flows the total number of signatures increased as well. Our preliminary experiments showed that by increasing the number of clusters, it makes the cluster merging step more difficult. This observation highlights the advantage of using isolation. Graption-P2P-NP: using only flow-level features. This configuration is the same as the Graption-P2P, but assuming that payload and port numbers cannot be used. We achieved >90% precision and >88% recall over a range of k and similarity thresholds. For each flow, we extracted more than 40 flow features ranging from packet size information (size of first 10 packets, max/min packet size, etc.), timing information (flow duration, min and max inter-packet gap, etc.), TCP flags, total volumes in bytes, number of packets, etc. In order to test the relevance of each feature we applied the Information Gain Ranking Filter [38] over various time intervals. For this configuration, we used the most prominent features: packet size information (i.e. min, max, and the size of the first five packets) and protocol (UDP or TCP). A recent work [11] also supports our observation that packet sizes are good features to cluster/classify network flows. 4. Related work This journal paper extends our six-page workshop version [18]. In addition to the work presented in the workshop version, we here include: (a) A study on practical considerations for using TDGs (see Section 2.3, Figs. 3 and 4). (b) We provide additional details regarding our data sets and our classification results (see Sections 2.2 and 3.5, Fig. 2, and Table 2). (c) We provide a more detailed comparison with BLINC (see Section 3.3 and Fig. 8). (d) We include additional experiments showing how Graption can be used to detect previously unknown traffic (see Section 3.4). 4.1. Traffic classification As an alternative to port-based methods, some work use payload [30,28]. Other approaches use machine learning (ML) algorithms to classify traffic using flow features (e.g., packet sizes). For an exhaustive list and comparison of ML algorithms we refer the reader to [32,26]. The problem of training a ML algorithm on one trace and applying it on another was recently addressed in [37]. Even though supervised solutions to traffic classification are continuously improving, all supervised methods require per application training and will thus not detect traffic from new applications. Our work has more in common with unsupervised methods which group similar flows together. All previous methods [3,29,10] require manual labeling of clusters. Our work bridges this gap by providing a method to automatically label clusters of flows based on their network-wide behavior. In BLINC [25], the authors characterize the connection patterns (e.g., if it behaves like using P2P) of a single host at the transport layer and use these patterns to label the

1918

M. Iliofotou et al. / Computer Networks 55 (2011) 1909–1920

flows of each host. BLINC uses graph models called graphlets to model a host’s connection patterns using port and IP cardinalities. Unlike TDGs, graphlets do not represent network-wide host interaction. In some sense, TDGs represent a further level of aggregation, by aggregating across hosts as well. Thus it is perhaps fair to say that while BLINC hints at the benefit of analyzing the node’s interaction at the ‘‘social’’ level, it ultimately follows a different path that focuses on the behavior of individual nodes. As we show, our approach performs better than BLINC in our backbone traces (see Section 3.3). Similar to BLINC, other host-based methods [2,22] target the identification of P2P users inside a university campus (i.e., network edge). The connection patterns of neighboring host (e.g., their degree distribution) were also used as features to profile network hosts [1]. Unlike Graption-P2P, in [2,22,25,1] they do not use network-wide host interaction. In [7], the authors use a port-based method to identify P2P users, using their temporal appearance and connection patterns in a trace. A different approach to host profiling was introduced by Trestian et al. [36]. They used readily available information from the Web to classify traffic using the Google search engine. They show very good results for classifying flows for legacy application, but their results are not promising for P2P detection because of the dynamic nature of P2P IP hosts. Our method can thus be used to complement the work in [36]. The problem of classifying traffic from applications that use sophisticated obfuscation techniques is the topic in [17]. The solution proposed uses advanced community detection algorithms to group related hosts together. It then requires external seed information about the application used by few hosts in each group in order to classify the entire group. The solution in [17] is very different from this work since it does not classify applications using networkwide behavioral informatio. We believe that Graption-P2P can provide a way to identifying P2P communities without relying on external seeding information. 4.2. Worm detection Graphs have been used for detecting worm activities within enterprise networks [9]. Their main goal was to detect the tree-like communication structure of worm propagation. This characteristic of worms was also used for post-mortem trace analysis (for the identification of the source of a worm outbreak, the so-called patient zero) using backbone traces [39]. More recent studies use graph techniques to detect hit-list worms within an enterprise network, based on the observation that an attacker will alter the connected components in the network [6]. 4.3. Measurements on network-wide interactions Statistical methods are used in [40] for automating the profiling of network hosts and ports numbers. The connectivity behavior and habits of users within enterprise networks is the focus of many papers, including [35]. In [34], the authors study P2P overlays using passive measurements, but target mainly the profiling of P2P hosts. The most resent work on network-wide interactions in by Jin et al. [21]. In their work [21], they use graph-partitioning methods to ex-

tract and study the evolution of smaller communities within a TDG. None of the above papers targets P2P detection.

5. Discussion Enhancing the isolation step. To improve isolation we can enforce payload inspection in addition to port-based filtering. For example, we can test all DNS flows at port 53 to see if they also have a DNS payload signature or if another protocol is tunneling its traffic under the DNS port. If payload is encrypted, then we can choose to use flow-level feature such as packet sizes [26] or whitelisted IP addresses [36]. Applying our framework to other backbone locations. In our study we show that is feasible to correctly identify P2P TDGs from all our backbone traces using a single set of thresholds. In other words, we do not use different classification threshold for each individual trace. Even though TDGs from different locations may have differences between them, in our study, we always find TDGs from the same family (e.g., P2P) to have similar characteristics (e.g., high average degree). In Section 2.2, we explicitly state that our thresholds are not universal. Therefore, it might not always be possible to use exactly the same thresholds to identify particular TDGs in all backbone locations. Nevertheless, our measurements suggest that small adjustments to our simple parameters (see Section 2.2) allow our methodology to work on different backbone links. TDG formation and packet sampling. Packet sampling is used in order to reduce the overhead of processing every single packet by a monitoring system. Sampling effectively reduces the overall number of observed flows by the monitor. Less overall flows translates to fewer observed edges over time when forming TDGs. In our experiments in Section 2.3, we observe that the longer we collect edges for a TDG (i.e., applying a longer observation interval) the number of edges and the stability of metrics increases. With sampling, we believe the curves in Fig. 4 may change. In particular, in order to reduce TDG variability in the presence of sampling, the monitoring interval will need to be increase. Encryption in P2P applications. If an application encrypts all traffic classification methods using payload [33,28,15, 30] will be out-dated. Note, however, that it has been shown before [4] that even if an application encrypts its traffic there are ways to distinguish it from other flows. To show that even in the case where P2P encryption becomes prevalent our framework will still be useful, we also experimented with Graption-P2P-NP using only flow level features and observed good results with recall and precision of above 88% in both traces. Obfuscation of P2P applications at multiple levels. In the future, obfuscation might become more common. In the case where IPsec (encryption up to Layer-4 headers) becomes prevalent, transport layer [24], and host-based [25] methodologies will most likely not work. If P2P protocols start using polymorphic blending methods [12,14] to masquerade their traffic at the packet (e.g., packet sizes) and payload level to resemble a legacy application (e.g., Web), then all current machine learning [32] methods will not work. In such a scenario, TDGs (IP-to-IP graphs) might still prove to be useful because the interaction patterns seem hard to hide.

M. Iliofotou et al. / Computer Networks 55 (2011) 1909–1920

6. Summary and conclusions The underlying theme of our work is to go beyond just monitoring individual packets, flows, or hosts, to also monitoring the interactions of a group of hosts. Towards this end, we introduce TDGs and show their potential to generate novel tools for traffic classification. Based on TDGs, we developed Graption, a graph-based framework, which we then specialize (Graption-P2P) for the detection of P2P applications. We show that Graption-P2P classifies more than 90% of P2P traffic with above 95% precision when tested on real-world backbone traces. While our best results are obtained assuming access to some initial payload bytes, Graption still performs well (90% precision with 88% recall) even when payload is not used. We believe the network-wide characteristics of P2P traffic (e.g., InO, average degree, and diameter) are not as easy to hide without significantly altering P2P functionality.

Acknowledgments This work was supported partly by NSF NECO 0832069, a CISCO URP grant, ARL CTA W911NF-09–2-0053, by NAP of Korea Research Council of Fundamental Science and Technology, and the ITRC support program [NIPA-2010C1090–1011-0004] of MKE/NIPA. Michael Mitzenmacher was supported in part by NSF grant CNS-0721491 and research grant from CISCO. George Varghese was supported by grants from NSF and CISCO. The SDSC’s TeraGrid and compute resources are supported by the NSF grant CONMI CRI-0551542. Support for CAIDA’s Internet traces is provided by the National Science Foundation, the US Department of Homeland Security, and CAIDA Members.

References [1] E.G. Allan, W.H. Turkett, E.W. Fulp, Using network motifs to identify application protocols, in: IEEE GLOBECOM, 2009. [2] G. Barlett, J. Heidemann, C. Papadopoulos, Inherent behaviors of online detection of peer-to-peer file sharing, in: IEEE GI, 2007. [3] L. Bernaille, R. Teixeira, K. Salamatian, Early application identification, in: ACM CoNEXT, 2006. [4] D. Bonfiglio, M. Mellia, M. Meo, D. Rossi, P. Tofanelli, Revealing Skype Traffic: when randomness plays with you, in: ACM SIGCOMM, 2007. [5] CAIDA. http://www.caida.org. [6] M.P. Collins, M.K. Reiter, Hit-List worm detection and bot identification in large networks using protocol graphs, in: RAID, 2007. [7] F. Constantinou, P. Mavrommatis, Identifying known and unknown peer-to-peer traffic, in: IEEE NCA, 2006. [8] DatCat, Internet Measurement Data Catalog. http://www.datcat.org. [9] D. Ellis, J. Aiken, K. Attwood, S. Tenarglia, A behavioral approach to worm detection, in: ACM CCS WORM, 2004. [10] J. Erman, M. Arlitt, A. Mahanti, Traffic classification using clustering algorithms, in: ACM SIGCOMM MineNet, 2006. [11] A. Este, F. Gringoli, L. Slagerelli, On the stability of the information carried by traffic flow features at the packet level, ACM SIGCOMM CCR 39 (3) (2009) 13–18. [12] P. Fogla, M. Sharif, R. Perdisci, O. Kolesnikov, W. Lee, Polymorphic blending attacks, in: USENIX Security Symposium, 2006. [13] A. Gerber, J. Houle, H. Nguyen, M. Roughan, S. Sen, P2P, the Gorilla in the cable, in: National Cable and Telecommunications Association (NCTA), 2003. [14] G. Gu, R. Perdisci, J. Zhang, W. Lee. BotMiner: clustering analysis of network traffic from protocol- and structure-independent botnet detection, in: USENIX Security Symposium, 2008. [15] P. Haffner, S. Sen, O. Spatscheck, D. Wang, ACAS: automated construction of application signatures, in: ACM SIGCOMM MineNet, 2005.

1919

[16] M. Iliofotou, M. Faloutsos, M. Mitzenmacher, Exploiting dynamicity in graph-based traffic analysis: techniques and applications, in: ACM CoNEXT, 2009. [17] M. Iliofotou, B. Gallagher, G. Xie, T. Eliassi-Rad, M. Faloutsos, Profiling-by-association: a resilient traffic profiling solution for the internet backbone, in: ACM CoNEXT, 2010. [18] M. Iliofotou, H. Kim, P. Pappu, M. Faloutsos, M. Mitzenmacher, G. Varghese, Graph-based P2P traffic classification at the internet backbone, in: IEEE GI, 2009. [19] M. Iliofotou, P. Pappu, M. Faloutsos, M. Mitzenmacher, S. Singh, G. Varghese, Network monitoring using traffic dispersion graphs (TDGs), in: ACM IMC, 2007. [20] IPOQUE – Bandwidth Management with Deep Packet Inspection, Internet Study: The Impact of P2P File Sharing, Voice over IP, Instant Messaging, One-Click Hosting and Media Streaming on the Internet, 2009. [21] Y. Jin, S. Esam, Z.L. Zhang, Unveiling core network-wide communication patterns through application traffic activity graph decomposition, in: ACM SIGMETRICS, 2009. [22] W. John, S. Tafvelin, Heuristics to classify internet backbone traffic based on connection patterns, in: IEEE ICOIN, 2008. [23] T. Karagiannis, A. Broido, N. Brownlee, K. Claffy, M. Faloutsos, Is P2P dying or just hiding? in: IEEE GLOBECOM, 2004. [24] T. Karagiannis, A. Broido, M. Faloutsos, K. claffy, Transport layer identification of P2P traffic, in: ACM IMC, 2004. [25] T. Karagiannis, K. Papagiannaki, M. Faloutsos, BLINC: multi-level traffic classification in the dark, in: ACM SIGCOMM, 2005. [26] H. Kim, K. Claffy, M. Fomenkov, D. Barman, M. Faloutsos, K. Lee, Internet traffic classification demystified: myths, caveats, and the best practices, in: ACM CoNEXT, 2008. [27] C. Labovitz, S. Lekel-johnson, J. Oberheide, F. Jahanian, Internet interdomain traffic, in: ACM SIGCOMM, 2010. [28] J. Ma, K. Levchenko, C. Kreibich, S. Savage, G.M. Voelker, Unexpected means of protocol inference, in: ACM IMC, 2006. [29] A. McGregor, M. Hall, P. Lorier, J. Brunskill, Flow clustering using machine learning techniques, in: PAM, 2004. [30] A. Moore, K. Papagiannaki, Toward the accurate identification of network applications, in: PAM, 2005. [31] M.E.J. Newman, The structure and function of complex networks, SIAM Review 45 (2003) 167. [32] T.T.T. Nguyen, G. Armitage, A survey of techniques for internet traffic classification using machine learning, IEEE Communications Surveys and Tutorials, 4th ed., March, 2008. [33] S. Sen, O. Spatscheck, D. Wang, Accurate, scalable in-network identification of p2p traffic using application signatures, in: WWW, 2004. [34] S. Sen, J. Wang, Analyzing peer-to-peer traffic across large networks, IEEE/ACM Transaction on Networking 12 (2) (2004) 219–232. [35] G. Tan, M. Poletto, J. Guttag, F. Kaashoek, Role classification of hosts within enterprise networks based on connection patterns, in: USENIX Annual Technical Conference, 2003. [36] I. Trestian, S. Ranjan, A. Kuzmanovic, A. Nucci, Unconstrained endpoint profiling (Googling the Internet), in: ACM SIGCOMM, 2008. [37] G. Urvoy-Keller, T. En-Najjary, Challenging statistical classification for operational usage: the ADSL case, in: ACM IMC, 2009. [38] I.H. Witten, E. Frank, Data Mining: Practical Machine Learning Tools and Techniques, 2nd ed., Morgan Kaufman, 2005. [39] Y. Xie, V. Sekar, D. Maltz, M. Reiter, H. Zhan, Forensic analysis of epidemic attacks in federated networks, in: IEEE ICNP, 2006. [40] K. Xu, Z. Zhang, S. Bhattacharyya, Profiling internet backbone traffic: behavior models and applications, in: ACM SIGCOMM, 2005.

Marios Iliofotou received the B.S. degree in Electrical and Computer Engineering from the Technical University of Crete, in Greece, and the M.S. and Ph.D. degree in Computer Science from the University of California, Riverside, in 2007 and 2011, respectively. His current research investigates the effectiveness of graph-based algorithms for managing and protecting large IP networks. During the course of his Ph.D., Marios has worked as an intern at Microsoft Research in Cambridge, Telefonica Research in Barcelona, the Lawrence Livermore National Laboratory, Narus, and Cisco Systems.

1920

M. Iliofotou et al. / Computer Networks 55 (2011) 1909–1920

Hyun-chul Kim is an assistant professor in the School of Computer Science and Engineering at Seoul National University. His research interests include content-centric networking, measurement, analysis and modeling of Internet traffic and topology, and user behavior patterns in a variety of largescale applications such as IPTV, VoD, VoIP, P2P file sharing, massively multi-player online role-playing games (MMORPGs), and the Web. Kim received his Ph.D. in computer science from the Korean Advanced Institute of Science and Technology (KAIST).

Michalis Faloutsos is a faculty member at the Computer Science Department in University of California, Riverside. He got his bachelor’s degree at the National Technical University of Athens and his M.Sc and Ph.D. at the University of Toronto. His interests include, Internet protocols and measurements, peer-to-peer networks, network security, BGP routing, and ad-hoc networks. He is actively involved in the community as a reviewer and a TPC member in many conferences and journals. With his two brothers, he co-authored the paper on powerlaws of the Internet topology SIGCOMM’99, which is one of the top ten most cited papers of 1999. His most recent work on peerto-peer measurements have been widely cited in popular printed and electronic press such as slashdot, ACM Electronic News, USA Today, and Wired. Most recently he has focused on the classification of traffic and identification of abnormal network behavior. He also works in the area of Internet routing (BGP), and ad hoc networks routing, and network security, with emphasis on routing.

Michael Mitzenmacher is a Professor of Computer Science in the School of Engineering and Applied Sciences at Harvard University. He graduated summa cum laude with a B.A. in mathematics and computer science from Harvard in 1991. After studying mathematics for a year in Cambridge, England, on the Churchill Scholarship, he obtained his Ph. D. in computer science at U.C. Berkeley in 1996. His work on low-density parity-check codes shared the 2002 IEEE Information Theory Society Best Paper Award and won the 2009 ACM SIGCOMM Test of Time Award. His textbook on randomized algorithms and probabilistic tech-

niques in computer science was published in 2005 by Cambridge University Press.

Prashanth Pappu received his Doctor of Science degree from Washington University in St. Louis where his research was focused on design of scalable routers and included QoS architectures, packet lookup and scheduling algorithms. Later, he was a Technical Leader at Cisco Systems where he led the design of Cisco’s data center, storage and Ethernet switches. He’s currently the Director of Product Management at Conviva Inc. that is creating a real-time, viewer-centric platform for delivering the highest resolution, broadcast quality video on the Internet.

George Varghese worked at DEC for several years designing DECNET protocols and products (bridge architecture, Gigaswitch) before obtaining his Ph.D. in 1992 from MIT. He worked from 1993–1999 at Washington University. He joined UCSD in 1999, where he currently is a professor of computer science. He won the ONR Young Investigator Award in 1996, and was elected to be a Fellow of the Association for Computing Machinery (ACM) in 2002. Together with colleagues, he has 16 patents awarded in the general field of Network Algorithmics. Several of the algorithms he has helped develop have found their way into commercial systems including Linux (timing wheels), the Cisco GSR (DRR), and Microsoft Windows (IP lookups). He also helped design the lookup engine for Procket’s 40 Gbps forwarding engine. He has written a book on building fast router and endnode implementations called ‘‘Network Algorithmics’’, which was published in December 2004 by Morgan-Kaufman. In May 2004, he co-founded NetSift Inc., where he was the President and CTO. NetSift was acquired by Cisco Systems in 2005. From Aug 2005 to Aug 2006, he worked at Cisco Systems to help equip future routers and switches to detect traffic patterns for measurement and security.