Sep 1, 2014 - Values. Beta. Delta. Alpha. Charlie. 0. 1. 2. 3. Dictionary. 1. 3. 1. 2. Values. Merge. Read Operations. Data Modifying. Operations. Me mo ry. Sto.
An Approach for Hybrid-Memory Scaling Columnar In-Memory Databases ! ! ! !
*Bernhard Höppner, °Ahmadshah Waizy, *Hannes Rauhe
!
* SAP SE ° Fujitsu Technology Solutions GmbH
! ! ! !
ADMS’14 in conjunction with 40th VLDB
Hangzhou, China September 1, 2014
Memory – Linking Desire and Reality Cost
$/GB
10 $/GB
Bandwidth
GB/s
μs
MB/s
ns
Latency
GB
TB
Capacity © 2014 SAP SE and Fujitsu Technology Solutions GmbH. All rights reserved.
2
Memory – Linking Desire and Reality Cost
DRAM
HDD
$/GB
10 $/GB
Bandwidth
GB/s
μs
MB/s
ns
Latency
GB
TB
Capacity © 2014 SAP SE and Fujitsu Technology Solutions GmbH. All rights reserved.
3
Memory – Linking Desire and Reality Cost
DRAM
HDD
$/GB
10 $/GB
Bandwidth
GB/s
μs
MB/s
ns
Latency
GB
TB
SSD © 2014 SAP SE and Fujitsu Technology Solutions GmbH. All rights reserved.
Capacity 4
Memory – Linking Desire and Reality Cost
DRAM
HDD
$/GB
Cheaper than DRAM Latency close to DRAM
10 $/GB
Bandwidth
GB/s
μs
MB/s
ns
Latency
GB
Bandwidth close to DRAM Denser than DRAM
SSD © 2014 SAP SE and Fujitsu Technology Solutions GmbH. All rights reserved.
TB
Capacity
+Non-Volatile +Byte-addressable +More energy efficient than DRAM Storage Class Memory ( SCM ) 5
Memory – Linking Desire and Reality Cost
DRAM
HDD
$/GB
Cheaper than DRAM Latency close to DRAM
Available 2018+ 10 $/GB
Bandwidth
GB/s
μs
MB/s
ns
Latency
GB
Bandwidth close to DRAM Denser than DRAM
SSD © 2014 SAP SE and Fujitsu Technology Solutions GmbH. All rights reserved.
TB
Capacity
+Non-Volatile +Byte-addressable +More energy efficient than DRAM Storage Class Memory ( SCM ) 6
Memory – Linking Desire and Reality Cost
DRAM
Offer separate
memory allocator
Fixed amount of DRAM to buffer data from SSD
HDD
!
$/GB
Accessed data via load/store operations 10 $/GB
Bandwidth
GB/s
μs
MB/s
Hybrid-Memory PCIe SSD as backing storage
SSD © 2014 SAP SE and Fujitsu Technology Solutions GmbH. All rights reserved.
ns
Latency
GB
Use paging to buffer data of SSD on DRAM TB
!
Increase addressable amount of memory Capacity
SCM 7
IMDBMS – SAP HANA based Architecture Data Modifying Operations
Read Operations
Main Store
Delta Store
Values Dictionary 0 1 2 3
Alpha Beta Charlie Delta
Values
Merge
1 3 1 2
Dictionary 0 1 2 3
Data Volumes
© 2014 SAP SE and Fujitsu Technology Solutions GmbH. All rights reserved.
Log Volumes
Storage
Persistence
Beta Delta Alpha Charlie
Memory
1 0 1 2 3 2
8
IMDBMS – SAP HANA based Architecture Data Modifying Operations
Read Operations
Main Memory Loaded Delta Store Column(s)
Values
0 1 2 3
Alpha Beta Charlie Delta
Values
Merge
1 3 1 2
Dictionary 0 1 2 3
Data Volumes
© 2014 SAP SE and Fujitsu Technology Solutions GmbH. All rights reserved.
Unloaded Log Volumes Table
Storage
Persistence
Beta Delta Alpha Charlie
Memory
Dictionary
Data Request on Column
1 0 1 2 3 2
Limited amount of Memory
Main Store
9
IMDBMS – SAP HANA based Architecture Data Modifying Operations
Read Operations
Main Memory Loaded Delta Store Column(s)
0 1 2 3
Alpha Beta Charlie Delta
Values
Merge
1 3 1 2
Dictionary 0 1 2 3
Data Volumes
© 2014 SAP SE and Fujitsu Technology Solutions GmbH. All rights reserved.
80
Beta Delta 60 Alpha Charlie 40
20
0
2
4
Storage
Persistence
Memory
Dictionary
Data Request on Column
1 0 1 2 3 2
Table Load Time [s]
Values
Limited amount of Memory
Main Store
6
8
10
In-Memory Table Size [GB]
Unloaded Log Volumes Table
10
IMDBMS – Scaling the Traditional Way Scale-Up • Increase the amount of available memory in a single server by adding DRAM • Limitations:
!
• Fujitsu RX-600 S6
2 TB
• Fujitsu RX-900 S2
4 TB
• Fujitsu 2800B
12 TB
Scale-Out • Add additional cooperating instances • Instances are separate (shared nothing) • Additional complexity and cost e.g. on the infrastructure level
© 2014 SAP SE and Fujitsu Technology Solutions GmbH. All rights reserved.
11
IMDBMS – Scaling w/ Hybrid-Memory Data Modifying Operations
Read Operations
Main Memory Loaded Delta Store Column(s)
loa d
m
Un
De
On
Merge
1 3 1 2
Dictionary 0 1 2 3
Data Volumes
© 2014 SAP SE and Fujitsu Technology Solutions GmbH. All rights reserved.
Unloaded Log Volumes Table
Storage
Persistence
Beta Delta Alpha Charlie
Memory
Column(s)
Values
Data Request on Column
1 Dictionary 0 0 Alpha 1 1 Beta 2 2 Charlie Hybrid-Memory 3 Delta Loaded 3 2
Limited amount of Memory
Lo
ad
Values
an d
Main Store
12
IMDBMS – Scaling w/ Hybrid-Memory Data Modifying Operations
Read Operations
Main Memory Loaded Delta Store Column(s)
1 3 1 2
0 1 2 3
Beta600 Delta 500 Alpha 400 Charlie 300 200 100 0
2
4
Storage
Persistence
Approximated Hardware Cost [1000Euro]
loa d
m
Un
De
On
Merge
Dictionary
IMDBMS using DRAM only IMDBMS using DRAM and Hybrid-Memory
Memory
Column(s)
Values
Data Request on Column
1 Dictionary 0 0 Alpha 1 1 Beta 2 2 Charlie Hybrid-Memory 3 Delta Loaded 3 2
Limited amount of Memory
Lo
ad
Values
an d
Main Store
6
8
10
12
Database Size [TB]
Data Volumes
© 2014 SAP SE and Fujitsu Technology Solutions GmbH. All rights reserved.
Unloaded Log Volumes Table
13
IMDBMS – Scaling w/ Hybrid-Memory DRAM
Data Modifying Operations
Read Operations
HybridMemory
Main Store Values
3 2 1 1 0 1
1 0 1 2 3 2
Dictionary 0 1 2 3
Alpha Beta Charlie Delta
Values
Merge
1 3 1 2
Dictionary 0 1 2 3
Data Volumes
Log Volumes
Storage
Persistence
Beta Delta Alpha Charlie
Memory
Values
Delta Store
SSD © 2014 SAP SE and Fujitsu Technology Solutions GmbH. All rights reserved.
14
Hybrid-Memory – Integrating DRAM
Data Modifying Operations
Read Operations
• Hybrid-Memory is introduced on column level if data does not fully fit into DRAM HybridMemory
Values
Delta Store
• Minimize latency overhead by storing less frequently used Values columns on Hybrid-Memory first 1 0• 1 2 3• 2
!
!
Dictionary
Values
Dictionary
Memory
3 2 1 1 0 1
!
Main Store
Mergeon column Higher access skew increases the probability 1 0level Beta 0 Alpha 3 1 Delta of cache hits 1 Beta 2 Charlie Reduce 3 Deltalarge
!
1 2
2 3
Alpha Charlie
range selects on Hybrid-Memory as data is paged in 4kB chunks
Persistence
Data Volumes
Log Volumes
Storage
• Avoid scan operations as sequential operations are not beneficial on Hybrid-Memory
SSD © 2014 SAP SE and Fujitsu Technology Solutions GmbH. All rights reserved.
15
Hybrid-Memory – Integrating While nodes in AST
Generate LFU list for columns
Fetch SQL Plan Cache
Analyze column access
Parse SQL Plan Cache, generate AST
Column used for Projection Get next node from AST Column used for Selection
Use existing information, do not generate additional statistics
Append SeqAcc to AccFreq
Append RndAcc to AccFreq
© 2014 SAP SE and Fujitsu Technology Solutions GmbH. All rights reserved.
Sort SeqAcc, RndAcc ascending
Not indexed
Add to SeqAcc
Indexed
Add to RndAcc
Add to RndAcc
Parse SQL statements executed in the past
Calc. usage frequency for SeqAcc, RndAcc
Aggregate SeqAcc, RndAcc
16
Evaluation – Partitioning TPC-C 80% of all columns used in less than 2% of all queries
Less frequently used columns cause 60% of the data volume © 2014 SAP SE and Fujitsu Technology Solutions GmbH. All rights reserved.
17
Evaluation – TPC-C Hybrid-Memory 80% : DRAM 20%
DBMS Throughput [%]
80% of all columns on Hybrid-Memory,
20% on DRAM
TPC-C performance in tpm% compared to unchanged system
Upper touchpoint, no paging
100
90 10% DRAM buffer reduces tpm by 12%, index vector size reduced by 54%
80
70
60
DRAM buffer size for Hybrid-Memory is adjusted
5 10 20
40
60
80
100
Hybrid-Memory Buffer Size [%] © 2014 SAP SE and Fujitsu Technology Solutions GmbH. All rights reserved.
18
Conclusion and Future Work Conclusion: • DRAM + SSD combined to Hybrid-Memory • Hybrid-Memory made available to applications via a separate allocator • Extended SAP HANA to use Hybrid-Memory as an additional data store • Analyzed TPC-C to be partitioned for two data stores • Ran TPC-C based evaluation reducing the index vector size by 54% while keeping the IMDBMS performance at 88%
© 2014 SAP SE and Fujitsu Technology Solutions GmbH. All rights reserved.
19
Conclusion and Future Work Future Work: • Evaluate our approach based on real data and workload • Integrate column partitioning directly into HANA • Introduce paging to other data stores (e.g. dictionary) • Design a cost model for columns to be stored on Hybrid-Memory including row level information
© 2014 SAP SE and Fujitsu Technology Solutions GmbH. All rights reserved.
20
Thank You! Questions? Comments? An Approach for Hybrid-Memory Scaling Columnar In-Memory Databases
!
*Bernhard Höppner, °Ahmadshah Waizy, *Hannes Rauhe
!
* SAP SE ° Fujitsu Technology Solutions GmbH
!
ADMS’14 in conjunction with 40th VLDB
Hangzhou, China September 1, 2014 © © 2014 SAP SE and Fujitsu Technology Solutions GmbH. All rights reserved.
Hybrid-Memory – Integrating While nodes in AST
Analyze column access
Fetch SQL Plan Cache
Column used for Projection
Parse SQL Plan Cache, generate AST
Get next node from AST Column used for Selection
Analyze SQL Plan Cache on column level
Append SeqAcc to AccFreq
Separate into random and sequential access
Append RndAcc to AccFreq
© 2014 SAP SE and Fujitsu Technology Solutions GmbH. All rights reserved.
Sort SeqAcc, RndAcc ascending
Not Indexed indexed
Add to SeqAcc
Not Indexed indexed
Add to RndAcc
Add to RndAcc
Maintain statistics in two independent lists
Calc. usage frequency for SeqAcc, RndAcc
Aggregate SeqAcc, RndAcc
22
Hybrid-Memory – Integrating While nodes in AST
Analyze column access
Fetch SQL Plan Cache
Parse SQL Plan Cache, generate AST
Column used for Projection Get next node from AST Column used for Selection
Append SeqAcc to AccFreq
Append RndAcc to AccFreq
© 2014 SAP SE and Fujitsu Technology Solutions GmbH. All rights reserved.
Sort SeqAcc, RndAcc ascending
Calc. usage frequency for SeqAcc, RndAcc
Not indexed
Add to SeqAcc
Indexed
Add to RndAcc
Add to RndAcc
Aggregate SeqAcc, RndAcc
23
Hybrid-Memory – Integrating While nodes in AST
Analyze column access
Fetch SQL Plan Cache
Parse SQL Plan Cache, generate AST
Column used for Projection Get next node from AST Column used for Selection
Workload adapted priority list for Hybrid-Memory
Append SeqAcc to AccFreq
Add randomly accessed columns first as they are more suited for Hybrid-Memory
Append RndAcc to AccFreq
© 2014 SAP SE and Fujitsu Technology Solutions GmbH. All rights reserved.
Sort SeqAcc, RndAcc ascending
Not indexed
Add to SeqAcc
Indexed
Add to RndAcc
Add to RndAcc
Reduce access on HybridMemory by keeping frequently used columns on DRAM
Calc. usage frequency for SeqAcc, RndAcc
Aggregate SeqAcc, RndAcc
24
© 2014 SAP SE or an SAP affiliate company.
All rights reserved. No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP SE or an SAP affiliate company. SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP SE
(or an SAP affiliate company) in Germany and other countries. Please see http://global12.sap.com/corporate-en/legal/copyright/index.epx for additional trademark information and notices. Some software products marketed by SAP SE and its distributors contain proprietary software components of other software vendors. National product specifications may vary. These materials are provided by SAP SE or an SAP affiliate company for informational purposes only, without representation or warranty of any kind, and SAP SE or its affiliated companies shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP SE or
SAP affiliate company products and services are those that are set forth in the express warranty statements accompanying such products and
services, if any. Nothing herein should be construed as constituting an additional warranty. In particular, SAP SE or its affiliated companies have no obligation to pursue any course of business outlined in this document or any related presentation, or to develop or release any functionality mentioned therein. This document, or any related presentation, and SAP SE’s or its affiliated companies’ strategy and possible future developments, products, and/or platform directions and functionality are all subject to change and may be changed by SAP SE or its affiliated companies at any time for any reason without notice. The information in this document is not a commitment, promise, or legal obligation to deliver any material, code, or functionality. All forward-looking statements are subject to various risks and uncertainties that could cause actual results to differ materially from expectations. Readers are cautioned not to place undue reliance on these forward-looking statements, which speak only as of their dates, and they should not be relied upon in making purchasing decisions.
© 2014 SAP SE and Fujitsu Technology Solutions GmbH. All rights reserved.
