Reinventing Cloud Storage Services Through Innovation and ...

0 downloads 133 Views 3MB Size Report
Their goal was to deliver an new level enterprise class level of ... Open Source. •Enterprise support and quality. •
Reinventing Cloud Storage Services Through Innovation and Collaboration Key properties for a cloud storage system

Background

• A major Global Telco provider was looking to enter the Storage as a Service market • Their goal was to deliver an new level enterprise class level of service to their customers • After a thorough review of a wide range of storage technologies and vendors, the customer selected Amplidata to develop a storage platform to meet their vision • Based upon this experience the following presentation is a brief overview of some of the areas to consider when developing your own public or private cloud implementation 2

AmpliStor® and Himalaya™ are trademarks of Amplidata. All other trademarks are the property of their respective owners.

Amplidata Confidential

Considerations

• Requirements – – – – – –

Exabyte scalable cloud That has built-in high durability: 15+ nines With guaranteed data consistency At low cost To offer cloud storage services to millions of users Across the globe

• Where would you start?

3

AmpliStor® and Himalaya™ are trademarks of Amplidata. All other trademarks are the property of their respective owners.

Amplidata Confidential

Choosing Your Platform Open Source

• Free community support • Expandable • COTS hardware

Commercial Storage System

• Expensive support & consultancy • Invest in in-house competency • In-house testing and tuning • No guaranteed solutions / SLA

• Enterprise support • Expensive license and support and quality • Long term lifecycle • Locked-in proprietary HW / release management

Software Enabled • Enterprise support and quality • Long term lifecycle / release management • Inexpensive COTS certified HW

4

AmpliStor® and Himalaya™ are trademarks of Amplidata. All other trademarks are the property of their respective owners.

Amplidata Confidential

High Durability at Low Overhead Erasure Encoding vs. 3 copy cloud • High Data Durability • 15 nines durability • •

99.9999999999999% Chance of losing an object per year is 1 / 1.000.000.000.000.000



Amazon is 11 nines

• Dynamically tunable in Policy • •

Tune to 2,3,4,5,6,…19 failure protection Update policy on-line

• Lower overhead = lower TCO • Only 43% storage overhead •

3 Copy Cloud Store

18/6 3 GEO AmpliStor

20/4 1 site AmpliStor

Source data

1.00 TB

1.00 TB

1.00 TB

Raw disk

3.00 TB

1.71 TB

1.43 TB

Overhead

200%

71%

43%

Technology

for 4 failure protection

• Compared to 200% for 3 copy

*This calculation is based on 3TB disks, 5% annual failure rate (AFR), 1PB AmpliStor® and Himalaya™ are trademarks of Amplidata. All other trademarks are the property of their respective owners.

Data Consistency Don’t settle for eventual consistency

by Wyatt Lloyd, Michael J. Freedman, Michael Kaminsky, David G. Andersen, April 21, 2014 http://queue.acm.org/detail.cfm?id=2610533

6

AmpliStor® and Himalaya™ are trademarks of Amplidata. All other trademarks are the property of their respective owners.

Amplidata Confidential

3D Elastic scalability

Scale global namespace

Scale throughput

Scale storage capacity

7

AmpliStor® and Himalaya™ are trademarks of Amplidata. All other trademarks are the property of their respective owners.

Amplidata Confidential

BitSpread® True 3D Elastic Scale-out ANY encoded block from ANY object in ANY bucket can be stored on ANY disk – Added disks become immediately available to store new data blocks – When disks are added, the existing data does not need to be moved – Buckets are NOT limited to subset of storage nodes – Spread NOT determined by physical installation of storage nodes – Hierarchy aware: spread wide to minimize impact of rack / DC failure Other technologies and limitations BitSpread Hierarchy aware spread 3D Elastic Scale-out

RAIN RAID pool across Storage Nodes Fixed hardware layout Expansion requires fixed # nodes Repair to same nodes in order

RING / CRUSH map Mapped to hardware layout Data movement at HW change

3 replica map 18/3 Pool

TRUE Exabyte Global pool

Copyright © Amplidata 2011-2014 8

AmpliStor® and Himalaya™ are trademarks of Amplidata. All other trademarks are the property of their respective owners.

6/2 Pool 1 geo Pool

6/1 Pool 3 geo Pool

Amplidata Confidential

12/4 EC map

6/2 EC map

GeoSpread® Multi-Site Disaster Prevention •

GeoSpread = hierarchy aware and configurable policy – WAN latency optimized consistency – No need to match exact hardware deployment to achieve Geo-protection



Protects against a full datacenter outage and more – Policy flexibility: DC failure + simultaneous disk/node/rack failure(s)



Local read performance – Automatically reads more data locally at any location, no configuration 1PB USER DATA Traditional Storage •

18/6 policy GeoSpread



3 replica + 25% RAID-6 overhead



1.71PB total RAW capacity



3.75 PB total RAW capacity

.57PB

+

.57PB

+

.57PB

1.25PB

+

1.25 PB

+

1.25PB

*Policy such as 18/7 can be set to protect against one datacenter + an additional node or rack in a second datacenter.

DC1

9

DC2

AmpliStor® and Himalaya™ are trademarks of Amplidata. All other trademarks are the property of their respective owners.

DC3

DC1

Amplidata Confidential

DC2

DC3

Small File Support

Requirement: •

Any bucket supports optimization for small files





10

Not acceptable: •

Configurable file size threshold that automatically optimizes for small files within bucket

The need to modify applications to redirect small files to different buckets



Same durability policy for small and large files: same failure protection level

Replication with low durability for small files vs. high durability for large files



Some systems don’t offer any protection at all for small files: they can be lost and might need to be regenerated



Dedicated hardware required for small files



No dedicated/specific hardware required for small files



High read performance, eliminates high number of disk IOPS for small files

AmpliStor® and Himalaya™ are trademarks of Amplidata. All other trademarks are the property of their respective owners.

Amplidata Confidential

The Customer solution: Introducing the Himalaya Reference Architecture for Service Providers A software defined object storage solution architecture Designed to meet true telco class service levels on a global scale Scalability

• 100+ Trillions of objects global namespace capacity • 100+ of Exabytes capacity

Durability

• Flexible policy definition supports multiple durability levels in a single system and online policy updates • True data consistency • Built-in robust data integrity, audit and repair functions

Performance

• 400+ MB/sec single stream performance • Linear performance scalability • Small file performance optimization

Security

• Network SSL/TSL and AES256 data encryption • LDAP authentication

Manageability

• Hot swap for HDD/SDD • HDD fail-in-place without degraded performance

Platform support

• Runs on Intel-based, Commercial-Off-The-Shelf hardware • Several suppliers certified

AmpliStor® and Himalaya™ are trademarks of Amplidata. All other trademarks are the property of their respective owners.

Amplidata Confidential

12

AmpliStor® and Himalaya™ are trademarks of Amplidata. All other trademarks are the property of their respective owners.

Amplidata Confidential

Service Provider vs. Enterprise Requirements Service provider •

Enterprise

Scale: Exabytes in multi-PB blocks



Great Out of the box experience



Self managing: little knowledge and infrequent operator intervention required



Broad API compatibility



Fully locked security



Storage operation API's



Start small, scale large in-place



User operation API’s self service portal, metering



Integration with account/authentication system



Bulk customer on-boarding



Certified Partner Ecosystem



Broad COTS HW support



Broad COTS HW support

Himalaya™ Reference Architecture available in 2 products: - Himalaya Service Provider/OEM - Himalaya Enterprise 13

AmpliStor® and Himalaya™ are trademarks of Amplidata. All other trademarks are the property of their respective owners.

Amplidata Confidential

Amplidata offering

Stay tuned for joint customer announcement tomorrow

AmpliStor® and Himalaya™ are trademarks of Amplidata. All other trademarks are the property of their respective owners.

Amplidata Confidential

Summary • When designing a public or private storage cloud, there are many things to consider. • The main goals should be to deliver:  enterprise/telco class service durability and performance levels  elastically 3D scale to meet capacity, performance and namespace requirements now and in the future  a software defined solution that contains cost by supporting COTS hardware  strong data consistency to make the storage easy to consume by users and applications 15

AmpliStor® and Himalaya™ are trademarks of Amplidata. All other trademarks are the property of their respective owners.

Amplidata Confidential

Himalaya Delivers 3D Elastic Scalability

Scale Throughput

Scale Global Namespace • •

Created new “Sharded” database architecture Scale by simply adding SSDs to system





Add Scale Servers Add Controllers

Scale Storage Capacity • •

Add Storage Nodes Storage Nodes will maximize different drive sizes

Plug & Play design allows system to scale linearly as hardware is added.

17

AmpliStor® and Himalaya™ are trademarks of Amplidata. All other trademarks are the property of their respective owners.

Amplidata Confidential

Customer Solution: Himalaya™ Reference Architecture

• A software defined object storage solution architecture • Massively scalable • Enterprise class performance • Flexible to meet variable SLA levels • Runs on Intel-based, Commercial-Off-The-Shelf hardware • Himalaya™ Reference Architecture available in 2 products – Himalaya Service Provider/OEM – Himalaya Enterprise

18

AmpliStor® and Himalaya™ are trademarks of Amplidata. All other trademarks are the property of their respective owners.

Amplidata Confidential