Storage CloudSim: A Simulation Environment for Cloud Object Storage Infrastructures http://github.com/toebbel/StorageCloudSim
[email protected], {foud.jrad, achim.streit}@kit.edu STEINBUCH CENTRE FOR COMPUTING - SCC
KIT – Universität des Landes Baden-Württemberg und nationales Forschungszentrum in der Helmholtz-Gemeinschaft
www.kit.edu
Agenda Introduction & Use Cases Motivation STaaS Simulation Concept Implementation Evaluation Conclusion & Future Work
2
03.04.2014
[email protected], {foud.jrad, achim.streit}@kit.edu
Steinbuch Centre for Computing
Cloud Computing and STaaS Cloud computing: scalable hardware and service provisioning [5] Deployment Models
X+Y Public
Software
IaaS
PaaS
Service Models
SaaS
Community
Private
Hybrid
VMs / STaaS
Platform Infrastructure
Storage as a Service (STaaS) Online / near-line, non-volantine storage at low costs [6] Object or block storage 3
03.04.2014
Tobias Sturm
Steinbuch Centre for Computing
CDMI (Cloud Data Management Interface) Standard API to access object STaaS, released by SNIA, 2012 [3] RESTful, HTTP based Organizes objects in containers, user separation, access via IDs or names
4
03.04.2014
Tobias Sturm
Steinbuch Centre for Computing
CloudSim Popular Java eventbased simulation environment for IaaS, developed by CLOUDS Lab at University of Melbourne [1] [2] Lacks for STaaS modeling: No standard Cloud user interface Not realistic disk model No simulation of concurrent use of bandwidths Coarse file size model
5
03.04.2014
Tobias Sturm
Steinbuch Centre for Computing
Motivation Simulations required to Estimate costs for users in multi cloud scenarios Compare different constellations of hardware components & policies
No known STaaS simulation environment
StorageCloudSim Develop extension for CloudSim to simulate STaaS Accurate models for servers & disks Realistic simulation of IO limitations Use of STaaS standards like CDMI Model multi-Cloud STaaS usage
6
03.04.2014
Tobias Sturm
Steinbuch Centre for Computing
Implementation - Class Diagram EventTracker
TrackableResource
UsageHistory
CdmiEntity
ReportGenerator StorageCloud
CdmiRootContainer
CdmiObjectContainer
CdmiCloudCharacteristics
CdmiMetadata
CdmiObject
StorageBlob
TimeawareResource GETContainerRequest
IObjectStorageDevice
CdmiRequest
StorageBlobLocation
GETObjectRequest
ScheduleEntry
StorageBroker
ObjectStorageServer
PUTObjectRequest
DELETEObjectRequest ...
CdmiResponse
/
StorageMetaBroker
7
03.04.2014
Tobias Sturm
UserRequest
SLARequest
UsageSequence
TrackableResource
*
SLARequirement
UsageSequenceFileGenerator
Steinbuch Centre for Computing
Implementation - Components EventTracker
Monitoring
TrackableResource
UsageHistory
CdmiEntity
ReportGenerator StorageCloud
CdmiRootContainer
CdmiObjectContainer
CdmiCloudCharacteristics
CdmiMetadata
CDMI
CdmiObject
StorageBlob
TimeawareResource
CDMI
GETContainerRequest
Storage Models IObjectStorageDevice
CdmiRequest
GETObjectRequest
StorageBlobLocation
ScheduleEntry
StorageBroker
ObjectStorageServer
PUTObjectRequest
DELETEObjectRequest ...
CdmiResponse
UserRequest
/
03.04.2014
StorageMetaBroker
Tobias Sturm
*
SLARequirement
Usage Models
8
SLARequest
UsageSequence
TrackableResource
UsageSequenceFileGenerator
Steinbuch Centre for Computing
Example STaaS Request Workflow (Single Cloud)
9
03.04.2014
Tobias Sturm
Steinbuch Centre for Computing
Implementation - SLA based StaaS Brokering SLA requirements are defined in UsageSquence UsageSquence independent from each other MetaBroker chooses best provider for each UsageSquence using SLA matching policies and SLA rankings Example SLA matching policies: Supports Capability (WebDav export, metadata modification, …) Does not have Restriction (max. container/object size) Max./min. allowed feature metric (max. latency, min bandwidth, max storage cots per GB,…)
Example SLA Ranking: Assign Score to each Cloud: 𝑠𝑐𝑜𝑟𝑒 =
𝑐𝑜𝑛𝑠𝑡 𝑝𝑟𝑖𝑐𝑒 𝑝𝑒𝑟 𝑠𝑡𝑜𝑟𝑒𝑑 𝐺𝐵
Rate availability of capabilities
10
03.04.2014
Tobias Sturm
Steinbuch Centre for Computing
Evaluation – STaaS Clouds Setup Modelling of a single- (Amazon S3) and multi-Cloud (all three) scenario Linear pricing model Amazon S3
SCC intra
Swift Cloud
#servers / #disks per server
6/6
1/3
4/4
write/read rate, write/read latency capacity per disk
156 MB/s 9.5 ms / 8.5 ms 2 TB
64 MB/s / 156 MB/s 156 MB/s 11 ms / 9 ms 9.5 ms / 8.5 ms 1 TB 2 TB
# allowed replica
3
3
1
Max. obj. size
Unlimited
16 GB
unlimited
total capacity
72 TB
3 TB
32 TB
$ per uploaded GB $ per stored GB $ per down. GB 11
03.04.2014
Tobias Sturm
0.0002 $ 0.05 $ 0.01 $
0.04 $ 0.0002 $
0.01 $ 0.1 $ Steinbuch Centre for Computing
Evaluation – Modeled STaaS UsageSquences Simulations with two types of UsageSquences Three experiments: mixed (50, 500, 5000 input sequences), normal only and scientific only (250 input sequences) total traffic per sequence
Sequence A
Sequence B
Gamma distr. 𝛼 = 2, 𝛽 = 3, max 15 GB
1..100 GB (uniform distr.)
upload-download ratio
3:1
uploads only
size of biggest object
1 KB .. 1 GB
1..100 GB (uniform distr.)
Idle time between two requests
10 ms – 30 s
Bursts of 5 operations, 510 min idle between bursts
SLAs
cdmi_create_container cdmi_delete_container
cdmi_create_container cdmi_delete_container No max_container_size max_object_size > X SLA available capacity > Y 1 1 + 𝑠𝑡𝑜𝑟𝑒 𝑢𝑝𝑙𝑜𝑎𝑑
SLA available capacity > Y 1 1 1 + + 𝑠𝑡𝑜𝑟𝑒 𝑢𝑝𝑙𝑜𝑎𝑑 𝑑𝑜𝑤𝑛𝑙𝑜𝑎𝑑
Rating 12
03.04.2014
Tobias Sturm
Steinbuch Centre for Computing
Evaluation – Effect of Object Size MetaBroker selects Clouds with lowest Costs with respect of SLA SCC Cloud selected for small objects as it is the cheapest Cloud For big objects, Swift is preferred to Amazon as it is cheaper
50 mixed sequences, Mutli-Cloud experiment. Used storage per Cloud 13
03.04.2014
Tobias Sturm
Steinbuch Centre for Computing
Evaluation – STaaS requests SLA Violations Due to over capacity, all sequences after minute 28000 are declined. Only 15 requests failed due to SLA violation (object size can not be satisfied)
Single Cloud, 5000 mixed sequence types 14
03.04.2014
Tobias Sturm
Steinbuch Centre for Computing
Evaluation – Total Usage Costs Multi-Cloud is more cost-saving compared to single Cloud (in terms of costs per succeeded requests)
50, 500 and 5000 mixed sequences – Mutli & Single Cloud 15
03.04.2014
Tobias Sturm
Steinbuch Centre for Computing
Evaluation – Effect of UsageSquence Type Restrictive SLA (as in Scientific sequences) leads not to cost-savings for the multi-Cloud usage
Type A
16
03.04.2014
Tobias Sturm
Type B
Steinbuch Centre for Computing
Conclusion Development of STaaS extension for CloudSim simulation environment Modelling of different SLA requirements and SLA matching policies for STaaS Clouds Evaluation of used Storage and costs for single and multi-Cloud scenario with different UsageSquence types Usage of multiple Clouds lead to cost-savings if SLA is not too restrictive
Future Work More complex SLA matching policies (location, throughput) More complex price models for STaaS Dynamic Broker Decisions Modeling of different storage controller policies (example: OpenStack Swift Ring) 17
03.04.2014
Tobias Sturm
Steinbuch Centre for Computing
References [1] Rodrigo N Calheiros, Rajiv Ranjan, Anton Beloglazov, C´esar AF De Rose, and Rajkumar Buyya. Cloudsim: a toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms. Software: Practice and Experience, 41(1):23–50, 2011. [2] Rodrigo N Calheiros, Rajiv Ranjan, C´esar AF De Rose, and Rajkumar Buyya. Cloudsim: A novel framework for modeling and simulation of cloud computing infrastructures and services. arXiv preprint arXiv:0903.2525, 2009. [3] Cloud Data Management Interface (CDMI) Version 1.0.2. [4] Amazon Inc. Amazon s3, cloud computing storage for files, images, videos, 2013. [Online; accessed 17-July-2013]. [5] Peter Mell and Timothy Grance. The nist definition of cloud computing (draft). NIST special publication, 800(145):7, 2011. [6] A. Schill and T. Springer. Verteilte Systeme: Grundlagen und Basistechnologien. Springer London, Limited, 2007. 18
03.04.2014
Tobias Sturm
Steinbuch Centre for Computing
Questions ?
19
03.04.2014
Tobias Sturm
Steinbuch Centre for Computing
Implementation – Cloud User Interface
20
03.04.2014
Tobias Sturm
Steinbuch Centre for Computing
Implementation – Storage Models CdmiObject
StorageBlob TimeawareResource IObjectStorageDevice
StorageBlobLocation
ObjectStorageServer
21
03.04.2014
Tobias Sturm
Steinbuch Centre for Computing
Implementation – Storage Models CdmiEntity
StorageCloud
CdmiRootContainer
CdmiObjectContainer
CdmiCloudCharacteristics
CdmiMetadata
CdmiObject
CdmiRequest
ScheduleEntry
CdmiResponse
22
03.04.2014
Tobias Sturm
Steinbuch Centre for Computing
Implementation – Usage Models CdmiRequest
SLARequirement *
StorageBroker UserRequest
1
CdmiResponse
SLARequest
/
StorageMetaBroker
UsageSequence
UsageSequenceFileGenerator
23
03.04.2014
Tobias Sturm
Steinbuch Centre for Computing
Simulation Workflow Plots, Logs, CSV
Cloud Scenario
Stat. Generator
Sample Streams, Traces, Logs
24
03.04.2014
Tobias Sturm
SequenceGenerator
Simulation
UsageSequences in XML
Steinbuch Centre for Computing
Implementation – Provider Side Hardware Modeling
Used Capacity Used UsedCapacity 100% 100%
Operation Operation 44
60% 60%
Operation Operation22 Operation Operation 11
t1
25
03.04.2014
Operation Operation33
t2 t2
Tobias Sturm
t3 t3
t4 t4
t5t5
t6
Operation 5
t7t7
t8
Time
Steinbuch Centre for Computing
