Testing data warehouse with key data indicators - CGI
Recommend Documents
of Data Warehouse Support. Focus IT management on the big picture, improve ... reporting or analytic objects that use th
TestQ Technologies Limited. Page 1. Dhananjay Patil. Data Warehouse / MIS
Testing: Corporate Information Factory. Introduction. Data warehouse commonly
...
become fundamental in data warehousing. For example, the E/R conceptual model, that is widely used for designing operational systems, does not explicitly ...
data warehouse testing, data migration and compliance ... in testing of data migration by providing extensive ... manage
Exhaustive testing of a Data warehouse during its design and on an ongoing
basis. (for the incremental activities) comprises Data warehouse testing. This type
...
Nov 6, 2009 - H.4.2 [Information Systems Applications]: Types of. SystemsâDecision support; D.2.5 .... testing, and performance testing are proposed as the main testing steps. ... ETL procedures and accessed by front-end tools. However,.
Data Warehouse. 2/20. Sistem Basis Data Lanjut. Outline. • Konsep dan
Arsitektur Data. Warehouse. • Alur Data Warehouse. • Teknologi dan Peralatan
Data.
Nov 4, 2012 ... warehouse. • Users typically do not access a warehouse directly but instead
access data through non- updatable views. Data Testing on ...
These ideas were later also applied to software engineering [3], .... Song, Khare, and Dai [7] developed the SAMSTAR method that is a semi-automated approach ...... CMU/SEO-2010-TR-033, Carnegie-Mellon University, November 2010. 24.
and reduce cost in logistics system [1]. Warehouses are .... model by calculating the efficiency level and can be used for benchmarking .... Book. Company(2002). 6. J.X. Gu, M. Goetschalckx, and L.F. McGinnis,. Research on ... F.H. Staudt, G. Alpan,
Katko, t. s. 2008: Groundwater sustainability indicators: test- ing with ... abundance in proportion to population and water use, as well as its quality and vulner-.
The Big Debate: Inmon Versus Kimball. ▫ In the beginning there were basically
two approaches to modeling the data warehouse. ▫ Inmon popularized the term
...
PDF Building a Scalable Data Warehouse with Data Vault 2.0, Building a Scalable Data Warehouse with Data Vault 2.0 Free
The Big Debate: Inmon Versus Kimball. ▫ In the beginning there were basically
two approaches to modeling the data warehouse. ▫ Inmon popularized the term
...
BEDAWA tool, which supports a rapid development of prototypes by automatically ... Grow into the Final System: Used as a construction tool, a prototype is a full implementation of an .... information that appears in response to a data warehouse query
stored data) from process models (i.e., software modules). .... cleaning standards/specs and business rules. - Schemata of any data marts. - List of available pre- ...
Data analysis as part of business intelligence solutions is a growingly demanded needs. .... Create a transformation to
Dec 1, 2009 ... Design Tip #109 Dos and Don'ts on the Kimball Forum. Design Tips 2008 ...
Design Tip #106 Can the Data Warehouse Benefit from SOA?
Data quality is a multi-dimensional concept that evaluates the suitability of data ... Furthermore, it provides monitoring of the parameters during operations and ...
data warehouse project better and to help them construct an efficient, robust, and
... to describe the structure and contents of a data warehouse in real-world ...
[4] Huang Yu, Zhang Xiao-yi, Yuan Zhen, Jiang. Guo-quan.â A universal data cleaning framework based on user Modelâ. ISECS International. Colloquium on ...
A SURVEY ON VARIOUS CLUSTERING TECHNIQUES IN DATA MINING ... automatic analysis of large quantities of data to extract previously unknown interesting ... The main challenges in text summarization are extracting the overall.
data warehouse framework with improved transformation phase that creates the ..... tured Data and Textual Analytics into Business Intelligence: Prentice Hall ...
Presentation. Front ... the system in a normalized form, a direct implementation of the model is ... database design must support goals of the data warehouse.
Testing data warehouse with key data indicators - CGI
Testing data warehouses with key data indicators ... Quality assurance of
historical data for Basel II, IFRS etc. ..... Fast reports with results in MS Excel and
PDF.
General Problem Test in the project / regression test •
Non-recurring assurance of the data quality in a project within a specified project plan
•
Test and retest multiple deliveries of mass-data
•
Quality assurance of historical data for Basel II, IFRS etc.
Plan Testbegin
Scheduled end of project Corrected Datadelivery
Datadelivery
Test preparation
Test
approval Re-Test
Time 4
General Problem Data verification •
Recurring assurance of data quality within production
•
Continuous check of the delivery of mass data
•
Additional sources of errors within recurring data deliveries
Reporting
Reporting
Datamart
Datamart
DWH Code X
X
X
Code X
X
X X
5
Problem Problem within the Project
Concrete Problem within the Project Project
Root-Systems
Basel II DWH (min. 5 years history)
Subsequent processing
Build and test of a DWH for historical Basel II data
ETL Processes
•
eg. calculation of parameters or regulatory reporting
7
Concrete Problem within the Project The original plan •
Non-recurring historical data delivery and test of this data set (inclusive Re-Test)
•
Handover of the daily data delivery within production
•
No usage of testing tools intended
8
Concrete Problem within the Project Scope of testing Around 50 tables –500 fields – several millions of data records • Around 500 test cases within 3 levels (Possible value range – Data integrity – End-to-End-Test)
•
Test execution •
Manual execution and documentation of the tests
•
Individual execution of every test case
•
Documentation of the test execution within a MS Access testing database
9
Concrete Problem within the Project Actual condition •
Recurring historical data delivery because of changes and incidents Time- and resources consuming (Duration of a complete test cycle around 20 person days) Partial abort of the test because of a new data delivery Concentration on one defined test data (One historical month)
Additional requirement •
Recurring verification of the data quality in production
10
Concrete Problem within the Project Plan Testbegin
Scheduled end of project Corrected Datadelivery
Datadelivery
Test preparation
approval Re-Test
Test
Time Scheduled end of project
Current situation Testbegin
Corrected Datadelivery
Datadelivery
Test preparation
Test
Corrected Datadelivery Test
New Datadelivery
Re-Test
Test
Re-Test ?!
Time 11
The Idea
The Idea No time-consuming repeat of all test cases! Fast test execution!
Automatization!
Design of a slim test tool using predefined data quality indicators 13
Reconciliation between data sources Reconciliation between data sources Reconciliation with the Balance-Sheet Reconciliation with the Balance-Sheet Direct Validation, Integrity Direct Validation, Integrity
Number of customers Sum of balance Sum of credit cards balance Number of defaulted accounts without being past due Number of accounts without scoring
Accounts
22
Indicators Technique
System Landscape
Indicators & Rules
Execution
Results
Export TestingDatabase
From a technical point of view the indicators are summary functions according to the SQL standard (SAS): Function
Description
Example
SUM AVG|MEAN COUNT|FREQ|N
Sum Average Counting values
SUM(Balance) AVG(Scorevalue) COUNT(*)
NMISS
Counting missing values Smallest value Maximum value
