③Mapping Into Oracle BI ... Introducing The Performance Layer >> Oracle's View
On Data Warehouse ... Build the tables using INFA & DAC - Complete. Oracle ...
Performance Tuning Oracle’s BI Applications Jeff McQuigg | Senior BI Architect April 24, 2013
Start Here
© KPI Partners Inc.
Contact Us 510.818.9480 | www.kpipartners.com
Agenda
2
Agenda
①Introducing The Performance Layer ②Building The Performance Layer ③Mapping Into Oracle BI ④Implementation Considerations ⑤Q&A
Performance Tuning Oracle’s BI Applications
3
Introducing The Performance Layer
4
Introducing The Performance Layer
Targeted At Organizations Who Have:
Large Data Volumes Highly Customized Tables Questionable Design Extensions Aggressive Performance Targets Slow Hardware Only a Data Warehouse
Performance Tuning Oracle’s BI Applications
5
Introducing The Performance Layer
These performance concepts are applicable to any BI system
BI Apps (OBIA) Stars Custom Stars in OBIA Custom Built Warehouses SQL Server
Performance Tuning Oracle’s BI Applications
6
Introducing The Performance Layer
Wide tables carry more data than you need Dashboards only need a few fields
Smaller is faster Performance Tuning Oracle’s BI Applications
7
Introducing The Performance Layer
Eliminate other and conflicting priorities Singular focus on performance
Performance starts with clear design goals Performance Tuning Oracle’s BI Applications
8
Introducing The Performance Layer
Great performance requires perfect design for how it is used A Dashboarding environment is an “Application” Use a top-down design approach to support that application
Specialized design for specialized usage Performance Tuning Oracle’s BI Applications
9
Introducing The Performance Layer
Pre-built logic Clean star models Reduced data weight Tables which match usage by Oracle BI
Top-Down design yieldsF
Performance Tuning Oracle’s BI Applications
10
Introducing The Performance Layer >> Oracle’s View On Data Warehouse Architecture
Performance Tuning Oracle’s BI Applications
11
① ② ③ ④
Build a new data model Copy data from BI Apps/DW tables Bring only what you need Denormalize & pre-calculate
*Optimize To Usage* Performance Tuning Oracle’s BI Applications
12
ETL Performance Mini-Fridge
Keep the BI Apps/DW model mostly as-is & add a Performance LayerF
BI Apps Data Fridge
Introducing The Performance Layer
Introducing The Performance Layer - Takeaways
① The Performance Layer is an industry standard architecture ② Design is driven only by report performance improvement ③ Travel light ④ No need to alter BI Apps or DW Performance Tuning Oracle’s BI Applications
13
Building The Performance Layer
14
Building The Performance Layer
1. Identify a priority area (select a Fact table) 2. Identify common use cases (reports w/ prompts & data security)
3. Analyze resulting physical SQL 4. Try to tune the BI Apps model first! (Indexes, etc)
Performance Tuning Oracle’s BI Applications
15
Building The Performance Layer
5. Prototype a new data model to match those needs (SQL handcrafting of new tables) 6. Adjust SQL & benchmark (SQL handcrafting needed) 7. Map into Oracle BI & test (Unit & Regression) 8. Benchmark the Oracle BI report using prototyped tables (Reports have many SQLsn)
Performance Tuning Oracle’s BI Applications
16
Building The Performance Layer
9. Build the tables using INFA & DAC - Complete Oracle BI RPD mapping 10. Formal Regression Test 11. Deploy 12. Enjoy praise from users
Performance Tuning Oracle’s BI Applications
17
Building The Performance Layer
Reduce I/O with extreme prejudice • Tune the BI Apps model first! It may work for you with low effort • Employ techniques to eliminate I/O wherever possible • Partition Elimination, Compression, Indexes, Aggregates, Star Transformations
• Let the Performance Layer do the work, not the report query • Follow the KISS principle: Use a simple and clean Star. No Snowflakes! • Ensure OBI is mapped properly and uses correct tables with perfect SQL • Favor a general approach as opposed to a case-by-case approach
• A rising tide lifts all boats Performance Tuning Oracle’s BI Applications
18
Building The Performance Layer
There are 4 kinds of tables in the Performance Layer: 1. 2. 3. 4.
Skinny Dimension and Fact tables New Dimension tables Mini-Dimension tables Fact Aggregate tables
BI Apps or DW
Built directly from the base BI Apps or DW tables
Goal: use these tables in as many reports as possible
Guiding principles and performance influences: 1. 2. 3. 4. 5. 6.
Application use cases drive the layer’s design Use minimal data for the job at hand Aggregate Fact data when needed Denormalize dimensions to eliminate extra joins Pre-Build calculations to eliminate extra joins Pre-Split data sets based on logical usage
Performance Tuning Oracle’s BI Applications
19
Perf. Layer
Building The Performance Layer
Table Partitioning
Query Indexing (4 Types)
For all Fact tables of a reasonable size (e.g., > 5M rows)
① Single column, local bitmap indexes on all Fact table FKs (_WIDs) and filter fields
Usually partition on Month (Range
(DELETE_FLG)
or Interval)
② Single column bitmap indexes on all dimensional fields used in any sort of prompt or report filter
The Database can easily eliminate the majority of the table Allows for smaller, local indexes
③ Special composite B-Tree indexes to assist Snowflaked areas
Before Beginning: Tune the OOTB Model Performance Tuning Oracle’s BI Applications
④ Composite B-Tree indexes on large dimensions for join backs (for list reports)
20
Building The Performance Layer
Skinny Tables are highly selective versions of the BI Apps or DW tables •
“Horizontal Aggregation” – E.g., 10 columns vs. 100 in the base table
Both Dimensions and Facts Very easy to build and use Goal: Reduce Avg. Row Length to 1/5th - 1/20th original size Include only the columns you will need for top-down reporting analysis • •
If you don’t need Customer Address, don’t include it Ignore Meta Data columns (e.g., INTEGRATION_ID, etc.)
Row sets are identical (1:1) with the base tables •
For Dimensions use the same ROW_WIDs - can be used with existing fact tables easily
Performance Tuning Oracle’s BI Applications
21
Building The Performance Layer
CREATE TABLE WC_ACCT_BUDGET_SF COMPRESS NOLOGGING PARALLEL (DEGREE 8) PARTITION BY RANGE(PERIOD_END_DT_WID) INTERVAL(NUMTOYMINTERVAL(1, 'MONTH')) (PARTITION Part_01 VALUES LESS THAN (20100101)) AS SELECT /*+ PARALLEL(F,8) */ F.PERIOD_END_DT_WID, F.X_PERIOD_END_DT_WID, F.COMPANY_ORG_WID, F.GL_ACCOUNT_WID, F.X_POSTED_TOTAL_AMT, case when GL_D."GL_ACCOUNT_NUM" = 'S250' then F."X_POSTED_TOTAL_AMT" end as PLAN_CASES, FROM W_ACCT_BUDGET_F F, W_GL_ACCOUNT_D GL_D WHERE F.GL_ACCOUNT_WID = GL_D.ROW_WID;
Build using: 1. 2. 3.
Create Table as Select (CTAS) Insert /*+ APPEND */ Materialized Views
Compress the table Use Parallel hints & options Don’t forget partitions Enhance the tables with calculation logic Database is very fast at these operations – expect only a few minutes for 100M rows Performance Tuning Oracle’s BI Applications
22
Building The Performance Layer
Enhance _SF tables with logic Identify CASE WHEN statements which require other dimensions •
Potential great benefit if the join can be eliminated
•
Don’t over do it – table will get less skinny with each column
case when GL_D."GL_ACCOUNT_NUM" = 'S250' then F."X_POSTED_TOTAL_AMT" end as PLAN_CASES, FROM W_ACCT_BUDGET_F F, W_GL_ACCOUNT_D GL_D WHERE F.GL_ACCOUNT_WID = GL_D.ROW_WID;
Identify any data set splitting from the RPD •
HR Workforce Events table has both Events and Snapshots records but they are always used separately in the RPD
•
Usage drives design: Split them out!
•
Huge benefit for Event counting metrics (~10% of table)
Performance Tuning Oracle’s BI Applications
Create table WC_WRKFC_EVT_EVENTS_SF … … WHERE SNAPSHOT_IND = 0 Create table WC_WRKFC_EVT_MONTH_SNP_SF … … WHERE SNAPSHOT_IND = 1
23
Building The Performance Layer
Typical to get a 10X to 20X and even 50X I/O benefit in the _SF vs. the base _F in size
Reduced AVG_ROW_LEN COMPRESSION Record Set Splitting All without any aggregation
Real Examples Sub Ledger (custom) Workforce Snap Workforce Events GL Balance Acct Budget
I/O Benefit 24X 11X 56X 21X 32X
Skinny Dimensions also have benefits: 1. 2. 3. 4. 5.
All I/O is a killer and slows down the entire system De-normalize into a Star (eliminate snowflakes & outer joins) Real World Ex. #1: 2 wide dims 2 skinny dims: 6X query improvement Real World Ex. #2: One query going from 11s to 4s with one skinny dim Real World Ex. #3: GL Account Dimension: 37X I/O benefit
Performance Tuning Oracle’s BI Applications
24
Building The Performance Layer
Pre-build major pieces of commonly used but complex logic into the Data Model • •
Over-relying on the RPD or Reports for logic can harm performance Let the ETL for the Performance Layer do the work not the query
Example #1: Large binning and bucketing CASE WHEN FACT.ORDER_AMT BETWEEN 0 and 100 THEN ‘0-100’ ELSE CASE WHEN FACT.ORDER_AMT BETWEEN 101 and 200 THEN ‘101-200’ … END
•
Build a new dimension table to hold these values – WC_CUST_ORDER_QTY_BAND_D
Example #2: Date format conversions – dynamically building a new column with a string concatenation statement: substring(T66755."PER_NAME_MONTH" , 1, 4) , '') + '-' + isnull(right(T66755."PER_NAME_MONTH" , 2) , '')
•
Build a new column in the W_DAY_D table & index it
Performance Tuning Oracle’s BI Applications
25
Building The Performance Layer
Create table WC_EMPLOYEE_MD COMPRESS as select ROWNUM AS ROW_WID, W_ETHNIC_GRP_DESC, WC_RACE_ETHNIC_DIVRSE_GRP_DESC, W_SEX_MF_CODE, W_SEX_MF_DESC, WC_NON_EMPLOYEE_VENDOR_NAME from ( select distinct W_ETHNIC_GRP_DESC, WC_RACE_ETHNIC_DIVRSE_GRP_DESC, W_SEX_MF_CODE, W_SEX_MF_DESC, WC_NON_EMPLOYEE_VENDOR_NAME from W_EMPLOYEE_D);
Simply a higher level or levels of a larger dimension • •
A combination of several Kimball concepts Granularities will be mixed
Make a new table from the large, base dimension • • •
Contains distinct combinations Use only commonly used fields Get cues from dashboard prompts, column selectors, report filters
Create a new ROW_WID Compress and index as normal, Parallel if needed for creation
This real world example created 5,400 records from a W_EMPLOYEE_D of 9+ Million rows.
Easy to build and map Performance Tuning Oracle’s BI Applications
26
Building The Performance Layer
_MD tables are used in the Performance Layer in two places: 1.
Link into Skinny Facts •
Use a separate FK in addition to the base _WID
•
Fact table has both EMPLOYEE_WID and EMPLOYEE_MD_WID
Thus the _SF can join to all of the following: •
New Mini Dimension (_MD)
(~1% rows, some columns)
•
New Skinny Dimension (_SD)
(100% rows, some columns)
•
Base BI Apps/DW Dimension (_D) (100% rows, 100% columns)
The OBI RPD can select which one is best for each query
Benefits of linking into the _SF 1. 2. 3. 4.
The same set of fact rows are selected – no benefit Reduced dimension I/O, CPU and buffer space Faster join-back on list reports Very fast prompts, especially when constrained
Performance Tuning Oracle’s BI Applications
27
Conceptual Size Differences _MD
_SD
_D
Building The Performance Layer
2. Use them for very high level Fact Aggregates •
Build a Fact aggregate at the Mini-Dimension level
Allows greater field inclusion with excellent aggregation ratios •
Multiple fields are available - not just one
A good Mini Dimension and Skinny Fact/Aggregate can serve a large % of dashboard queries MD’s & Fact Aggregates offer extreme performance: 1. 2.
Real World Ex #1: From time-out after 10 minutes to 4 seconds Real World Ex #2: GL Account MD: 131X I/O benefit
Performance Tuning Oracle’s BI Applications
28
Building The Performance Layer
Aggregates are used when summary reports exist Pre-aggregate the dataset to make it smaller & faster Sometimes they are the only solution Typically a minimum of a 10:1 ratio is used Use all database tools as with any fact table •
Partitioning, Indexing, Star Transformations, Compression
For extreme needs, consider merging facts together •
Ex: Monthly Actuals and Budgets
•
Be mindful of gaps in datasets and non-conformed dimensions
Advanced implementations use partition management to build only changed data for faster load times
Performance Tuning Oracle’s BI Applications
29
Building The Performance Layer- Takeaways
① Building the underlying tables is relatively simple ② But strong SQL & Tuning expertise is needed ③ Take cues from dashboard, report & RPD configuration ④ Any savings in I/O helps the overall system ⑤ Use all of the available database performance tools
Performance Tuning Oracle’s BI Applications
30
Mapping Into Oracle BI
31
Mapping Into Oracle BI
Link as much as possible to allow for the best performance across all scenarios Best
Performance Layer
Better
BI Apps Base
Performance Tuning Oracle’s BI Applications
32
Mapping Into Oracle BI
The 3 Fact tables are mapped like any aggregate
_A
The Skinny Fact (_SF) will have fewer dimensions and fewer metrics mapped to it • Along the Employee dimension however it is the same as the base _F
_SF
OBI will prefer to use the _SF over the _F • Uses regular aggregate navigation concepts • Uses the _F when needed as a “backup plan” Performance Tuning Oracle’s BI Applications
33
_F
Mapping Into Oracle BI
Raise the priority group on the base _D to have OBI prefer the _SD As both LTS grains are identical, OBI needs more info to make a choice
_D
Performance Tuning Oracle’s BI Applications
_SD
34
Mapping Into Oracle BI
Create a dummy hierarchy level and map the LTSs for the Mini Dimension and Fact Aggregate to it The grain of the Mini Dimension is arbitrary As long as OBI knows it is higher than the other LTSs it will be preferred (Priority groups not needed)
_MD
_A
Performance Tuning Oracle’s BI Applications
35
Mapping Into Oracle BI - Takeaways
Table Mapping The mapping of tables is straightforward
Link Tables As Much As Possible Let Oracle BI make the best choice
Performance Tuning Oracle’s BI Applications
36
Implementation Considerations
37
Implementation Considerations
The whole prototyping process can be done on a simple star in roughly two weeks Allow for more time if you have: • • • • •
Large data volumes Difficult performance targets More complex models and logic Many disparate report patterns or lots of reports to consider More stars are needed (e.g., Actuals and Budgets together)
Development effort depends on # new objects • •
Typically only another two weeks needed (ETL & OBI RPD) A few more for regression test and deployment
Performance Tuning Oracle’s BI Applications
38
Implementation Considerations
Use Production data volumes for accurate analysis Use Production DDL, ETL code, OBI RPD and OBI Webcat Quiet, unused machine for accurate benchmarking Use hardware that is as similar to Prod as possible •
KPI uses a database benchmarking tool to compare environments
Performance Tuning Oracle’s BI Applications
39
Implementation Considerations
Additional ETL & RPD Development •
Use SQL scripts instead of Informatica mappings (less effort, faster execution)
Additional Testing – Regression test is easy Additional ETL Run Time – may be critical Additional Database size - minor Customization Propagation / Impact Analysis •
True of any aggregate
Complex logic will be more difficult • •
E.g. #1: Financial Analytics uses snowflakes with multiple segment hierarchies E.g. #2: HR Workforce Event & Snapshot logic uses effective dates for future dated events
Performance Tuning Oracle’s BI Applications
40
Q&A
41
www.kpipartners.com
Transform Data Into Insight Strategic Consulting | Systems Implementation | Training
Staff built from Oracle/Siebel/Hyperion engineering teams On-site, off-shore and blended shore delivery models Exclusive pre-built solutions for Oracle BI & E-Business Suite Depot Repair Analytics Fixed Asset Analytics Manufacturing Analytics Salesforce.com Analytics
The Leader In Oracle BI & EPM
42
Student Info Analytics Subledger (SLA) Analytics and moreF
Oracle BI Hyperion Endeca Exalytics
Contact Us
Email:
[email protected] Web: kpipartners.com/contact KPI World Headquarters
North America Offices
39899 Balentine Drive Suite #375 Newark, CA 94560 Phone: (510) 818-9480
New York, NY Chicago, IL Boston, MA
Global Offices Bangalore, India
The Leader In Oracle BI & EPM
Minneapolis, MN San Diego, CA Greensboro, NC
43
Hyderabad, India
www.kpipartners.com
44
