SNMP libvirt UDP

8istributed Lightweight 8ata Streaming and Processing for Monitoring Floud Infrastructures Iris Leitner*, Bashar Ahmad and Michael T. Krieger A Forresponding author) irisNleitnerRrisc4softwareNat

—bstract

UDP

One of the advantages of cloud infrastructures is their flexibility obtained by virtualization technologiesN With this benefit cloud infrastructures can become very complexN Therefore new approaches are needed for appropriate monitoringN We present a lightweight distributed data streaming and processing frameworkE which adapts to any infrastructure architectureN

SNMP

110 001 110

1

110

11

011

00

010

11

011

01

1

Collector Parser Parser

110 001

1

001

11

1

10 1

110

1

011

00

110

01 1

010

110001110101011

011

00 0

110

11

110

110

Parser

1

011

00

010

11

Live Database 011

Filter

1

Filter 11

1010011

1

110 110 011

11

010 011

11

0110

0110

0101011

Analyzer

11000111

1

00

01

1 11

01

10

01

10

10

01

11

Historical Database

0111

1011001

001

01

1

10

01

10

10 1

01

0

10

11

01

00 1

11

010011

Aggregator

1100

1100011

110 0

01

11

00

1100 011101 10011

—rchitecture

There are several data collectors and parsers for log filesE SNMP and libvirt data or UDP trafficN New data sources can easily be added thanks to our focus on extensibilityN The collected data is parsed and transferred to a unified object based on a predefined meta descriptionN We provide different standardised stream implementations to allow low over4head data forwarding within the network as well as a proper interface for applications like data visualizationN

Collector

Collector

The elasticity and increasing size of IFT infrastructures present a challenge to monitoring systemsN — flexible tool is requiredE which grows with the infrastructure and adapt to any architectureN — future4orientated way of data collection and processing is needed to extend the monitoring to new types of data sourcesN —t the same time this data must be standardised to allow the use of proven methods and increase the performance in data transferE persistence and analysesN —nother aim for monitoring toolsE is to minimize their impactN To fulfill these requirementsE we developed a flexible lightweight data streaming framework for monitoring cloud infrastructures [1]N

Data Collection, Parsing and Streaming

110001110101011

1 10 0 01 1 01 1 01 0

Introduction

Our framework consists of several components which can be deployed and extended with easeN New components like filteringE analysisE prediction and visualizationN In cooperation with JKU1 a monitoring visualization was developed on top of our data streaming framework [J]N The data streaming approach provides the flexibility to distribute the workload through the whole network and adapt the processing strategy to the existing resourcesN The system allows to run data collectionE processing and storing processed data on local or remote sitesN 8ifferent operations on the generated streams are possibleE including splitting and mergingN These operations can be configured and reconfigured as requiredN Layers for data filtering and privacy preservation can be introduced at any point of the streaming processN

libvirt

Visualization 1001

11

Figure 1: This figure represents the Flexibility of our modul based streaming framework. Different data streams can be collected and forwarded to several processing components. The workload can be distributed in the whole infrastructure.

Aggregation 8ata aggregation is an important tool for analysis pre4 processing and saving bandwidth and storageN Our framework supports a wide range of statistical aggregation methods for custom time periodsN

Persistence Since we provide live data as well as historical dataE our persisting strategy is based on the lambda architecture [5]N —t the moment two database controllers are implementedN The in4memory database RE8IS5 is ideally suited for real time data accessE while FassandraJ is applicable for persisting a huge amount of time series dataN

References

Fonclusion and Further Work

[1] /N —hmadE IN LeitnerE and MN TN KriegerN Pv8) — Lightweight 8istributed Monitoring —rchitecture for Floud InfrastructuresN In IEEE Hth Symposium on Network Floud Fomputing and —pplicationsE 501GN

Our framework is successfully running on two completely different infrastructures — an in4house cloud infrastructure and the infrastructure of a web service provider with 5NH million visitors per monthN We are currently working on including data analysis and prediction features based on time series dataN

[5] NN MarzE and JN WarrenN /ig 8ata) Principles and best practices of scalable realtime data systemsN Manning Publications FoNE 501GN [J] HN StitzE SN GratzlE MN KriegerE and MN StreitN FloudGazer) — 8ivide4 and4Fonquer —pproach for Monitoring and Optimizing Floud4 /ased NetworksN Proceedings of the IEEE Pacific Visualization Symposium ,PacificVis k1GBE 501GN Acknowledgements The research presented in this poster received funding in the project PIPES4 VS48—MS from the program IFT of the future of the —ustrian Research Funding —gency ,FFGHB under the grant agreement number (H05J5N /ackground image under Freative Fommons by 8errick Foetzee from /erkeleyE F—E US— [FF0]E via Wikimedia Fommons

1

http)22wwwNjkuNat2cg 5 http)22redisNio J http)22cassandraNapacheNorg H http)22wwwNffgNat