Supporting Autonomic Management of Clouds

Supporting Autonomic Management of Clouds: Service-Level-Agreement, Cloud Monitoring and Similarity Learning by Rafael Brundo Uriarte [email protected] Under the Supervision of: Prof. Rocco De Nicola and Prof. Francesco Tiezzi

Doctoral Thesis Defense - March 30th, 2015 - Lucca, Italy

Contents 1

Introduction

2

SLA for Clouds

3

Cloud Monitoring

4

Similarity Learning

5

Polus Framework

6

Conclusions Rafael Brundo Uriarte

1/51

Introduction

Introduction

Rafael Brundo Uriarte

2/51

Cloud Computing

Introduction


3/51

Cloud Characteristics

I

Services

I

Heterogeneity

I

Virtualization

I

Large-Scale

I

Complexity

Introduction


4/51

Autonomic Computing

Introduction


5/51

Autonomic Computing

Introduction


6/51

Knowledge for the Self-Management

I

Policies

I

Service Definition and Objectives

I

Status of the Cloud and Services

I

Specific Knowledge

Introduction


7/51

Challenge and Scope

Introduction


8/51

Research Questions Research Question 1 How to describe services and their objectives in the cloud domain?

Research Question 2 What is data, information, knowledge and wisdom in the autonomic cloud domain?

Research Question 3 How to collect and transform operational data into useful knowledge without overloading the autonomic cloud? Introduction


9/51

Research Questions

Research Question 4 How to produce a robust measure of similarity for services in the domain and how can this knowledge be used?

Research Question 5 How to integrate different sources of knowledge and feed the autonomic managers?

Introduction


10/51

SLA for Clouds

SLA for Clouds


11/51

Service-Level-Agreement (SLA)

I

Contract

I

Service Description

I

Quality-of-Service

I

Formalism

I

Guarantees!

SLA for Clouds


12/51

SLA for Cloud Computing - SLAC

I

Domain Specific

I

Multi-Party

I

Deployment Models

I

Formalism

I

Ease-of-Use

SLA for Clouds


13/51

Formal Business General

Yet Another SLA Definition Language? Features

WSOL

WSLA

SLAng

WSA

SLA*

SLAC

Deployment Models

Broker Support

-

-

-

-

-

Pricing Schemes

-

Semantics

-

-

-

-

Verification

-

-

-

-

“” feature covered “” feature partially covered “-” no support

SLA for Clouds


14/51

Main Concepts

I

Predefined Metrics - Involved Parties and Unit

I

Intervals for Metrics - Template and Variations

I

Groups - Multiple Service, Community Cloud

I

Constraint Solving Problem

SLA for Clouds


15/51

Example

SLA for Clouds


16/51

Business Aspects

I

Business Actions

I

Flat and Variable Models

I

Pricing Schemes - Exchange, Auction, Tender, Bilateral, Fixed, Posted

SLA for Clouds


17/51

Implementation

I

Editor for SLAs (Ecplise-based using Xtext)

I

SLA Evaluator (Z3 Solver)

I

Integration with the Monitoring System

SLA for Clouds


18/51

Cloud Monitoring

Cloud Monitoring


19/51

DIKW in the Domain

I

Data

I

Information

I

Knowledge

I

Wisdom

Cloud Monitoring


20/51

Cloud Monitoring

The Role of the Monitoring System in Clouds: I

Collect data and Provide Information and Knowledge

I

No Wisdom - Related to Decision-Making

I

Sensor of MAPE-K Loop

Cloud Monitoring


21/51

Related Works Property

PCMONS

Monalytics

Lattice

Wang

Cloud

-

-

-

Autonomic Integration

-

-

-

-

Scalability

-

Adaptability

-

-

Resilience

-

-

-

-

Timeliness

-

-

Extensibility

“” feature covered “” feature partially covered “-” no support Cloud Monitoring


22/51

Panoptes

I

Multi-agent system

I

Monitoring in different levels

I

Monitoring Modules - What needs to be monitored and how to process the data

Cloud Monitoring


23/51

Architecture

Cloud Monitoring


24/51

Architecture

Communication:

Adaptativeness:

I

Publish/Subscribe

I

Priority for Modules

I

Private Message

I

Change of Roles

Cloud Monitoring


25/51

Architecture: Autonomic Integration

I

Urgency Mechanism

I

Decentralised Architecture

I

On-the-Fly Configuration

I

Multiple Abstractions

Cloud Monitoring


26/51

Experiments

I

Self-Protection System

I

Urgency Mechanism

I

Scalability

Cloud Monitoring


27/51

Similarity Learning

Similarity Learning


28/51

Specific Knowledge

Generation of Knowledge for a Specific Purpose, i.e. not applicable in all clouds. For example, similarity. But what is similarity? I

How much an object (service) resembles other

Similarity Learning


29/51

Applications of Similarity

Applications in the Domain: Cluster Services: I

I

I

Anomalous Behaviour Detections

I

Service Scheduling

I

Application Profiling

I

SLA Risk Assessment

Group Similar Services Different Algorithms (K-Means, PAM, EM)

Similarity Learning


30/51

Domain Requirements

I

Categorical Characteristics of Services

I

On-line Prediction

I

Large Number of Characteristics

I

Fast Prediction

Similarity Learning


31/51

Random Forest

Clustering with Random Forest I

Originally Developed for Classification

I

Calculate the Similarity

I

Clustering Algorithm (PAM)

Similarity Learning


32/51

Similarity Using RF: Criteria

Similarity Learning


33/51

Problems

I

Similarity Matrix (Big Memory Footprint)

I

Re-cluster on Every New Observation

I

Cannot be Used in the Domain

Similarity Learning


34/51

Solution: RF+PAM

Similarity Learning


35/51

Solution: RF+PAM

Similarity Learning


36/51

Experiments I

Compared the performance of our algorithm to other 2 methodologies

I

Compared the performance of RF+PAM with the standard off-line similarity learning

I

Use Case: I Scheduler deploys together the most dissimilar services I Similarity based on their SLAs

Similarity Learning


37/51

Polus Framework

Polus Framework


38/51

Polus Framework

Polus Framework


39/51

Use Case

Polus Framework


40/51

Use Case

Polus Framework


41/51

Use Case

Polus Framework


42/51

Conclusions

Conclusions


43/51

Summary

Conclusions


44/51

Research Questions Research Question 1 How to describe services and their objectives in the cloud domain? SLAC Research Question 2 What is data, information, knowledge and wisdom in the autonomic cloud domain? DIKW Hierarchy Research Question 3 How to collect and transform operational data into useful knowledge without overloading the autonomic cloud? Panoptes Conclusions


45/51

Research Questions

Research Question 4 How to produce a robust measure of similarity for services in the domain and how can this knowledge be used? RF+PAM

Research Question 5 How to integrate different sources of knowledge and feed the autonomic managers? Polus Framework

Conclusions


46/51

Limitations

I

Intelligence of Autonomic Managers

I

Wide Range of Specific Knowledge

I

Off-line Training of RF+PAM

Conclusions


47/51

Contributions

I

A theoretical and practical framework for the generation and provision of knowledge for the autonomic management of clouds (Polus Framework): I I I

Conclusions

SLAC - SLA Definition and Evaluation Panoptes - Monitoring RF+PAM - Similarity Learning


48/51

Publications 1.

R. B. Uriarte, S. Tsaftaris and F. Tiezzi. Service Clustering for Autonomic Clouds Using Random Forest. In Proc. of the 15th IEEE/ACM CCGrid [In Press], 2015.

2.

R.B. Uriarte, F. Tiezzi, R. De Nicola, SLAC: A Formal Service-Level-Agreement Language for Cloud Computing. In IEEE/ACM 7th International Conference on Utility and Cloud Computing (UCC), 2014.

3.

R.B. Uriarte, C.B. Westphall, Panoptes: A monitoring architecture and framework for supporting autonomic Clouds, In Proc. of the 16th IEEE/IFIP Network Operations and Management Symposium (NOMS), 2014.

4.

R.B. Uriarte, S.A. Chaves, C.B. Westphall, Towards an Architecture for Monitoring Private Clouds. In IEEE Communications Magazine, 49, pages 130-137, 2011.

Conclusions


49/51

Future Works

I

Dynamic SLAs

I

Negotiation of SLAs

I

Cloud Case Study

Conclusions


50/51

Thank you! Questions?

Rafael Brundo Uriarte [email protected] Conclusions


51/51

SLAC - Expressivity

I

Core Language

I

Extensions - Business Aspects

I

Formal Definition for Extensions

Conclusions


51/51

SLAC - Implementation

Compatibility only with OpenNebula I Toy Implementation I

Conclusions

Easily adapted


51/51

SLAC - Cloud Metrics

DTMF Cloud Computing Service Metrics Description I Recent Document (Still a Draft) I

Creation of a Model for the Definition of Metrics

I

The SLAC Metrics can be Adapted for this Model

Conclusions


51/51

SLAC Violation

I

Violation and Penalty are Separated Concepts

I

“Violation” Concept Flexible

I

Easy to Understand

Conclusions


51/51

Panoptes - Scalability

I

Designed to be scalable

I

Adapt itself

I

Experiments suggest it is scalable

I

More experiments for future works

Conclusions


51/51

Panoptes - Analysis of Apache Broklyn

I

Not Focused on Monitoring

I

Does Not Process the Data

Conclusions


51/51

Panoptes - Analysis with CSPARQL

I

Data is not Decorated (e.g. RDF)

I

Impact of Decorated Monitoring Data (Scalability)

I

Very Interesting Option

Conclusions


51/51