Swearingen and Sinha (2001) carried out a study to investigate not only the ..... repetitively a certain tabular dataset with N records indexed by [n] and M ...... Decision-Support Applications", Addison-Wesley Longman Publishing Co., Inc., ...
BEN-GURION UNIVERSITY OF THE NEGEV FACULTY OF ENGINEERING SCIENCES DEPARTMENT OF INDUSTRIAL ENGINEERING AND MANAGEMENT
Enhancing Business-Intelligence Tools with Value-Driven Recommendations
THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE M.Sc. DEGREE
Yoav Kolodner
November 2009
BEN-GURION UNIVERSITY OF THE NEGEV FACULTY OF ENGINEERING SCIENCES DEPARTMENT OF INDUSTRIAL ENGINEERING AND MANAGEMENT
Enhancing Business-Intelligence Tools with Value-Driven Recommendations
THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE M.Sc. DEGREE
By: Yoav Kolodner Supervised by: Dr. Adir Even Author: Yoav Kolodner
……………….
Date: ………………
Supervisor: Dr. Adir Even
……………….
Date: ………………
Chairman of Graduate Studies Committee: Prof. Josef Kreimer ……………….
November 2009
Date: ………………
i
Acknowledgments First, I would like to thank my supervisor, Dr. Adir Even, for his dedication, commitment and active involving in all aspects of this research. Thank you for sharing with me your experience, ideas, time and most importantly, thank you for giving me a strong feeling of friendship. My study was supported by a grant from Microsoft R&D Israel, and I thank them for their support. I would like to express my appreciation to Roy Varshavsky and Daniel Sitton from Microsoft, who helped me in the development of some of the new concepts presented in this research and their support in the design of the experimental environment. Many thanks are due to Prof. Nava Pliskin who guided me throughout the 4th year project, and for always being there for a good advice. I would also like to appreciate Prof. Joachim Meyer. Discussions and cooperation with Prof. Meyer were essential for the success of the experiment conducted in the study. Lastly, for my Mom, Dad and my dearest family, this work would not have happened without your love and everlasting support. You are my inspiration… My deepest love to Jenny. Thank you for your love, your advice and for supporting me throughout the happy and more challenging times of this project.
ii
Abstract
Business-intelligence (BI) tools are broadly adopted in organizations today, supporting activities such as business analytics, decision making, and performance measurement. However, usage of BI tools is not simple and could be improved if feedback about usage and how-to-use recommendations were available to users. This study proposes a feedback and recommendation mechanism (FRM), embedded in a BI tool, whose visual cues and guidelines are based on assessment of previous usage. Moreover, this study introduces value-driven usage metadata - a novel methodology for tracking and communicating a quantitative assessment of the value gained from using the data resources. This study investigates the usefulness and the impact of integrating the proposed FRM via a comprehensive laboratory experiment with 200 participants. The findings confirm that FRM integration can improve the usability of BI tools and increase the benefits gained from data resources, highlighting the potential benefits of collecting and using value-driven usage metadata to generate usage recommendations.
Keywords: Business Intelligence (BI), Data Warehouse, Metadata, Data Value, Recommender System, Decision Support System (DSS)
iii
Table of Contents 1.
Introduction ...................................................................................................................... 1
2.
Literature Review ............................................................................................................. 3
2.1.
Business Intelligence ........................................................................................................ 3
2.2.
2.3.
2.1.1.
Data as a Critical Organizational Resource ............................................................. 4
2.1.2.
Transactional versus Analytical Data Usage ........................................................... 5
2.1.3.
The Data Warehouse ............................................................................................... 7
2.1.4.
Business Intelligence ............................................................................................. 10
2.1.5.
Metadata in Business Intelligence Environments .................................................. 13
Data Usage and the Associated Business Value ............................................................. 15 2.2.1.
Data Usage and Utility .......................................................................................... 15
2.2.2.
Tracking Data Usage – Solutions and Issues......................................................... 18
Recommender Systems .................................................................................................. 21 2.3.1.
Recommender Systems Evaluation ....................................................................... 24
2.3.2.
Accuracy and Trust Issues in Recommender Systems .......................................... 25
2.4.
Literature Review Summary........................................................................................... 26
3.
Conceptual Framework................................................................................................... 28
3.1.
Frequency-Driven versus Value-Driven Usage Metadata .............................................. 28
3.2.
Value Driven Feedback and Recommendation Mechanisms ......................................... 35
4.
Method ............................................................................................................................ 38
4.1.
Prototyping a Value-Driven Usage Metadata Module ................................................... 39 4.1.1.
4.2.
Query Analysis and Value Allocation ................................................................... 40
Experimental Lab Testing .............................................................................................. 43 4.2.1.
Experiment Settings............................................................................................... 44
4.2.2.
Experiment Task .................................................................................................... 46
4.2.3.
Experiment Tools .................................................................................................. 47
5.
Experiment Results ......................................................................................................... 51
5.1.
Data Collection and Preparation ..................................................................................... 51
5.2.
FRM Integration and Decision Outcomes ...................................................................... 55
5.3.
FRM Integration and Usage Style .................................................................................. 58
5.4.
The Perceived Contribution of FRM Integration ........................................................... 61
iv 5.5.
Discussion....................................................................................................................... 65
6.
Conclusions .................................................................................................................... 67
6.1.
Limitations and Directions for Future Research............................................................. 68 6.1.1.
Value-driven Usage Metadata ............................................................................... 68
6.1.2.
FRM Integration .................................................................................................... 70
6.1.3.
Integration and Evaluation in Real-World Business Contexts .............................. 71
7.
References ...................................................................................................................... 72
8.
Appendices ..................................................................................................................... 79
8.1.
Appendix 1: Metadata Schema ....................................................................................... 79
8.2.
Appendix 2: Experimental System Specification ........................................................... 82
8.3.
Appendix 3: Participant's Agreement Form ................................................................... 85
8.4.
Appendix 4: Experiment Guidelines .............................................................................. 86
8.5.
Appendix 5: Questionnaire ............................................................................................. 89
8.6.
Appendix 6: Statistical Analysis .................................................................................... 90
8.7.
8.6.1.
Normality Tests ..................................................................................................... 90
8.6.2.
Decision Value – FRM Effect ............................................................................... 91
8.6.3.
Clicks and Time – FRM Effect.............................................................................. 92
8.6.4.
Questionnaire – FRM Related Questions only ...................................................... 93
8.6.5.
Questionnaire – General BI Questions .................................................................. 94
8.6.6.
Questionnaire – Cronbach's Alpha ........................................................................ 95
8.6.7.
Correlations ........................................................................................................... 99
Appendix 7: ECIS 2009 Submission ............................................................................ 100
List of Figures Figure 1: Research's focus on analysis decision tasks (Nutt, 2002) ................................................ 3 Figure 2: The BI environment ......................................................................................................... 4 Figure 3: Star schema and Snow flake schema (Chaudhuri and Dayal, 1997) ............................... 8 Figure 4: Basic DW architecture (Kimball et al., 2002).................................................................. 9 Figure 5: Reporting using spreadshits ........................................................................................... 10 Figure 6: Oracle OLAP tool .......................................................................................................... 11 Figure 7: Dashboard with KPI in a Proclarity BI tool ................................................................... 11
v Figure 8: Proactive statistical analysis........................................................................................... 12 Figure 9: Collaborative filtering approach – users' similarity affects the rating prediction .......... 22 Figure 10: Amazon.com collaborative based recommender system ............................................. 23 Figure 11: (a) Frequency-driven versus (b) Value-driven usage metadata ................................... 30 Figure 12: Example of frequency-driven and value-driven usage metadata assessment ............. 33 Figure 13: Value-driven collection of usage metadata .................................................................. 34 Figure 14: A BI tool with FRM (Rates presented next to possible next steps of data analysis) .. 36 Figure 15: Pivot table with FRM ................................................................................................... 37 Figure 16: BI tool with collaborative based FRM ......................................................................... 37 Figure 17: Metadata module architecture ...................................................................................... 40 Figure 18: Example of table Value_Dist ....................................................................................... 42 Figure 19: BI tool without FRM .................................................................................................... 48 Figure 20: BI tool with FRM ......................................................................................................... 49 Figure 21: Decision screen ............................................................................................................ 50 Figure 22: Campaign report........................................................................................................... 50 Figure 23: Control group, the effect of repetitive usage ............................................................... 56 Figure 24: First-session performance per group ............................................................................ 56 Figure 25: Performance in session 2 ............................................................................................. 57 Figure 26: FRM by session effect ................................................................................................. 58 Figure 27: FRM by session effect on decision time ...................................................................... 59 Figure 28: Clicks session effect ..................................................................................................... 60 Figure 29: FRM related questions only ......................................................................................... 63 Figure 30: FRM related acceptance ............................................................................................... 64 Figure 31: Metadata repository table diagram ............................................................................... 79 Figure 32: Example of table Integrate ........................................................................................... 82 Figure 33: Business DB diagram ................................................................................................... 83 Figure 34: User Management DB diagram ................................................................................... 84
List of Tables Table 1: OLTP versus OLAP .......................................................................................................... 7 Table 2: Literature review summary ............................................................................................. 27 Table 3: Comparison of the two generic data stakeholders ........................................................... 29
vi Table 4: Participant demographics along the different experiment groups ................................... 51 Table 5: Survey questions ............................................................................................................. 53 Table 6: Summary statistics for all participants ............................................................................ 54 Table 7: Control variables effect ................................................................................................... 54 Table 8: Decision value analysis ................................................................................................... 55 Table 9: Time analysis ................................................................................................................... 58 Table 10: Clicks analysis ............................................................................................................... 59 Table 11: Session 1 correlations .................................................................................................... 61 Table 12: Session 2 correlations .................................................................................................... 61 Table 13: Perceived performance .................................................................................................. 62 Table 14: Usefulness, ease of use and acceptance......................................................................... 62 Table 15: Usefulness, Ease of Use and Acceptance analysis (Session 2) ..................................... 63 Table 16: Usefulness, rase of use, and acceptance analysis (Session 2, FRM-related questions) 64 Table 17: Table Trace – columns description ............................................................................... 80 Table 18: Table Business_Task – columns description ................................................................. 80 Table 19: Table Value_Input – columns description ..................................................................... 81 Table 20: Table Integrate – columns description .......................................................................... 81 Table 21: Table Value_Dist – columns description ...................................................................... 82
vii
List of Abbreviations
API
Application programming interface
BI
Business Intelligence
BPM
Business Process Management
COTS
Commercial Of The Shelf
DBMS
Database Management System
DDL
Data Definition Language
DL
Digital Library
DML
Data Manipulation Language
DQM
Data Quality Management
DSS
Decision Support System
DW
Data Warehouse, Data Warehousing
ER
Entity Relationship
ERD
Entity Relationship Diagram
ETL
Extraction, Transformation and Loading
FRM
Feedback and Recommendation mechanism
IP
Internet Protocol
IS
Information Systems
IT
Information Technology
OLAP
On Line Analytical Processing
OLTP
On Line Transaction Processing
RDBMS
Relational Database Management System
SQL
Structured Query Language
1
1.
Introduction Data repositories, along with the information systems (IS) utilizing them, have long been
recognized as critical organizational resources. Recent years have witnessed a major transition toward extended usage of data resources for business analytics, performance measurement, and managerial decision support. This transition is driven by the notion that decisions based on evidence and data are likely to be better and thus help the organization thrive (Pfeffer and Sutton, 2006). Indeed, evidence exists about firms that gained competitive advantage by investing in the development of data analysis capabilities and data-driven analytics (Davenport, 2006). This transition toward data-driven management is well-supported by the rapid progress in the capacity and performance of information technologies (IT) for utilizing large data resources. Most notable is the broad adoption of business intelligence (BI) platforms and tools, which permit rapid development and distribution of data analysis and decision support utilities. In the 2009 Gartner survey, responding chief information officers (CIOs) ranked BI as their #1 top technology-investment priorities (Gartner, 2009). High complexity, however, is a major limitation of current BI environments (Watson and Wixom, 2007). The common end-user, in search of an answer to a business question, often finds large data repositories too difficult to navigate for reaching the right data, and BI tools too complex to use for coming up with the right answer. Furthermore, in many cases the user knows neither the right business question to ask, nor the full range of capabilities offered by the BI tools and the underlying data repositories (Lawton, 2006). To overcome these complexities, this study proposes to integrate a feedback and recommendation mechanism (FRM) into a BI tool and investigates the proposed integration. The proposed FRM generates textual and/or graphical visual cues, guiding the user to consider the use of certain data subsets and/or analysis forms. Recommender systems, like the proposed FRM, have been intensively explored and broadly adopted in contexts such as E-commerce and digital libraries, but rarely in the BI context. The main purpose of this research is to investigate whether the proposed FRM integration does improve the functionality of BI tools and thereby business decisions by facilitating effective and efficient navigation and by revealing the undiscovered potential of unused data and analysis forms. In the proposed value-driven FRM, the recommendations are based on quantitative assessment of business-value gains and their attribution to the data resources being used. Since
2 data resources contribute value through their usage and integration within business processes, tracking the usage of data resources is essential for managing them effectively, and for understanding the associated value contribution. Usage tracking, or the collection of "usage metadata" (Shankaranarayanan and Even, 2004), is a relatively new concept in database environments which has not been significantly researched so far, and is currently supported only by a limited number of commercial tools. To facilitate the proposed FRM, this study develops a new form of usage metadata that tracks not only the frequency of using data objects, as commonly done today, but also includes quantitative assessments of the value gained by data usage, assigning value to different data elements (e.g., subsets of records and/or attributes) according to their former value contribution. The study also introduces a conceptual design and working prototype of the metadata layer that captures these value assignments. The proposed FRM and the new concepts incorporated in it were investigated in a laboratory experiment involving 200 participants who were asked to use different FRM variants integrated into a BI tool to analyze a simulated sales database and to create promotion campaigns for a new product, based on their analysis. Following a set of rules, which guided the database simulation, each campaign was assigned with an objective value measure, later used for quantitative evaluation of the decision outcomes. Aspects of the participants' usage style and perceived contribution of FRM were also measured. The results show that the proposed valuebased FRM significantly improves the effectiveness of using BI tools and also contribute important insights regarding the impact of FRM on usage behavior and on the perceived usefulness of BI tools. The next sections provide the background for this research, introducing two novel concepts: the collection of value-driven usage metadata and the integration of FRM into BI tools. This is followed by a description of the laboratory experiment, analysis of its results, and a discussion of the findings and their implications. To conclude, limitations are discussed along with a few directions for future research and three potential contributions are outlined. First, this study proposes value-driven usage metadata, i.e., a novel methodology for tracking the usage of data resources which integrates assessments of the frequency of data usage and the associated value gains. Second, it presents the concept of integrating value-driven FRM into BI tools and highlights approaches for generating them. Finally, it explores the potential contribution of collecting value-driven metadata and generating value-based FRM through a comprehensive laboratory experiment.
3
2.
Literature Review The following literature review, which sets the background for the research, begins with a
survey of the BI environment, continues to discuss previous research on the IS usage and the associated business value, data resources in particular, and proceeds to cover recommender systems, their application in IS environments, and ways to measure and evaluate their success. 2.1.
Business Intelligence This study focuses on assessing the usage patterns and the value-contribution of large data
repositories, and ways to improve their use for analytical and decision-support purposes. Notably, not all forms of decisions need to be supported by vast amounts of data. Nutt (2002) defines four types of organizational decisions, different from each other by the clarity of objectives and the availability of means for decision making (Figure 1). The main focus of this research is on decisions in the "Analysis" upper-left quadrant of Figure 1 with well-defined goals and measureable performance that rely on the data available in the data repositories. The other decision types - Judgment, Bargaining and Inspiration - lack either a clear objective and/or the means for making the decision, while important and common organizations, are not within the scope of this research.
Figure 1: Research's focus on analysis decision tasks (Nutt, 2002)
The BI environment (Figure 2) has evolved from the concept of Decision Support Systems (DSS) as a dominant concept for supporting analytical usage (March and Hevner, 2007). At a high level, a typical BI architecture consists of (Kimball et al., 2002 Sen and Sinha, 2005):
4 1) Data Sources – data used for analysis are extracted from multiple source systems, including internal – e.g., operative systems inside the organization or external – e.g., the internet and commercial data vendors. 2) Extraction, Transformation and Loading (ETL) Mechanisms – data extracted from the data sources are transformed into new data structures that are more suitable for analysis and loaded into a repository. Data transformation may include data cleansing (e.g. correcting misspellings and filling-in missing elements), aligning data formats between multiple sources and indexing to permit fast retrieval 3) Data Warehouse (DW) – the main data repository which organizes and stores archives of cleansed and well-structured data that meet the requirements of analytical usage. 4) BI Applications – front-end tools that the user interacts with, designed to support managerial decision making and permit effective and efficient data analysis.
Figure 2: The BI environment
BI components, concepts and relevant studies are reviewed next.
2.1.1.
Data as a Critical Organizational Resource Most organizations today depend on effective and efficient management of data
repositories to support day-to-day business processes, meet regulatory requirements (Even and Shankaranarayanan, 2007), sell data as a commodity to other organizations (e.g., www.reuters.com, which sells financial quotes), and use analytically for decision support (Rhodes, 1993). Database management systems (DBMS) that manage data repositories are implemented and used in almost any industry and business domain including, for example, industrial production, commerce, finance, travel, and entertainment (Cooper et al., 2000; Watson et al., 2001; Davenport, 2006). A study of 32 leading organizations that collect and analyze
5 business data on a regular basis showed that these firms attribute much of their success to the masterful exploitation of data (Davenport, 2006). Data repositories have thus become among the most important organizational assets. The management of data as a critical resource has to address the growing complexities, while supporting the fundamental goals of the enterprise (Levitin and Redman, 1998). The data repositories that organizations manage grow continuously and accumulate large volumes in the magnitude of Petabytes (see for example http://www.businessintelligencelowdown.com, which lists the world‟s largest databases in 2009), and their implementation requires a significant investment in data-management technology and formulation of companywide data-management strategies (Redman, 1996). Managing large data repositories is challenging for several possible reasons. First, data are not always aligned with the organization‟s business strategy, possibly leading to situations where vast amounts of data are collected, but the particular data elements needed to execute corporate strategy cannot be effectively accessed and remain unused (Levitin and Redman, 1998). Second, resource investment in data acquisition and data management technologies are high and often underestimated (Even and Shankaranarayanan, 2004). Third, supporting all business processes requires availability and accessibility of data resources to the many different data consumers which are not always guaranteed (Levitin and Redman, 1998). Fourth, maintaining high data quality, to avoid inaccuracy, incompleteness, duplications, inconsistency, or invalidity, can be very costly (Redman, 1996). Fifth, to deal with rapid changes in business environments, data repositories have to permit fast and inexpensive response to change. Finally, to deal with information-security breaches, systems that store and manage data must prevent unauthorized access, theft or loss of critical data (Elmasri and Navathe, 2006). Large data repositories require constant maintenance and quality-assurance efforts to ensure effective use (Redman, 1996), including investment in implementing data quality monitoring tools, establishing data auditing and correction procedures, and appointing personnel for maintaining the data repositories (Wang, 1998). Data maintenance and quality assurance tasks are often being done without considering the economic aspects (Even, 2007). This study suggests that better understanding and linking usage of data to economic tradeoffs can make important contributions to the implementation of better data administration policies.
2.1.2.
Transactional versus Analytical Data Usage Data usage can be classified into two key categories: transactional data usage, often
termed as On-Line Transactional Processing (OLTP), versus analytical data usage, often termed
6 as On-Line Analytical Processing (OLAP). This classification has major implications for design and administration of data resources (Kimball et al., 2002) and therefore understanding the fundamental differences between these two forms of data usage is important (Inmon, 1993). A key role of OLTP applications is to automate the data processing (e.g. banking transactions, production events) that supports day-to-day operations and business processes in organizations. Such applications typically require detailed and up-to-date operational data which focus on a specific business entity or event (e.g., a customer, a product, or a sale transaction). Since data consumers retrieve and update only a small number of specific records during transaction processing, OLTP systems are required to support fast search, retrieval and processing of specific records, and guarantee high availability and recoverability, thus typically focusing on the most recent transactions and maintaining only limited historical data (Kimball et al., 2002). OLAP applications, on the other hand, typically support intelligent business analytics and managerial decision making. Such tasks often require access to large and diverse data resources, with the ability to aggregate vast amounts of data that have been collected over a long period of time; hence, OLAP databases need to store large volumes of data, and to support dataaccess queries, enabling rapid aggregation of data elements and permitting complex calculations (Chaudhuri and Dayal, 1997). OLTP and OLAP applications are geared towards supporting different types of decisions. Typically the former support routine decisions, while for the latter support non-routine decisions which often require in-depth data analysis. Two examples for such business analysis tasks are: (a) Identifying the targeted list of customers for a marketing campaign – an important decision that a marketing manager would cope with when designing marketing campaigns is segmenting the customers database (typically managed in a CRM system), toward identifying which customer segments will be targeted by a certain campaign (Even et al., 2007); and (b) Identifying products for which the inventory policy has to change – decisions about which product to keep and at what stock level are important for retail and production firms that seek to lower their inventory costs and manage it efficiently (Ramanathan, 2006). Table 1 summarizes few key differences between OLTP and OLAP systems (Singh, 1998; Shankaranarayanan and Even, 2004). The difference most relevant to this work is the separation of the data environments that support OLAP from OLTP applications, with data warehouses supporting OLAP applications as described next.
7
Purpose
OLTP Run day-to-day operation by data collection and update
Common Data model
Relational
Access – front-end and back-end Volume of data Type of data Condition of data Frequency of changes Characteristics of queries
SQL, data entry and editing, client-server applications Relatively low volume Data that runs the business Incomplete – changing Rapidly changing Update/retrieval of a single business transaction Highly normalized Focus on current RDBMS
Normalization Currency Storage
OLAP Information retrieval and analysis, toward supporting managerial decision-making Dimensional - star/snowflake schema. BI tools, DSS, Data mining tools. High volume Data to analyze the business Descriptive - summarized Relatively stable Aggregation of large datasets Redundancy allowed Archives of historical data RDBMS and OLAP Cubes
Table 1: OLTP versus OLAP
2.1.3.
The Data Warehouse A DW supports the collection and the assessment of the critical business intelligence
needed to understand and evaluate the firm within its environmental context (March and Hevner, 2007). The DW concept, first introduced by Inmon (1993), is aimed at managing data resources needed to support complicated analyses and serve as the foundation for improved managerial decision making. Providing consistent and reliable organizational data that can be easily accessed, DW data are often integrated into unified schemas through often-complex ETL processes from both internal (e.g., OLTP systems such as ERP) and external (e.g., online data feeds such as from the internet) sources. The DW architecture and design are optimized for aggregating large datasets towards supporting reporting and data analysis needs. Methodologies such as the Entity-Relationship model (Chen, 1976), and the relational/normalized database design (Codd, 1970), while permitting optimizing databases for higher speed of retrieval, effective modification of specific data records, and prevention of data redundancies, are not optimal in terms of aggregation and retrieval of large datasets into a DW (Kimball et al., 2002). Hence, data warehouses often use a dimensional model composed of a fact table with a multi-attribute key that contains attributes, whose values are generally numeric and additive, with several tables, describing the data dimension, joining to the fact table as needed (Chaudhuri and Dayal, 1997). Dimensional models
8 have proved to be understandable, predictable, extendable, and highly resistant to the ad-hoc queries from groups of business users because of their relatively simple structure (Kimball et al., 2002; Sen and Sinha, 2005). The characteristic structure of the physical representation of a dimensional model, called a star schema, can be extended to a snowflake schema by removing the low cardinality attributes in the dimensions and placing them in separate tables which are linked back into the dimension table with artificial keys (Figure 3). The data in the main DW repository can be re-arranged into subsets, commonly known as Data Marts (Figure 4), often aggregated and configured to accommodate specific decision-making processes, analyses or business questions via a variety of BI tools (Kimball et al., 2002). A DW and data mart can be organized in an OLAP cube structure, i.e., a data model which renders data into multidimensional perspectives, more efficiently supporting OLAP queries. For example, a data cube holding data for a sales application can have dimensions of product, location and time representing each of the sale transactions. The cube can be sliced on its different dimensions in order to get summarized data on one of the different products or time unit. Each of the dimensions can be summarized at different levels, ranging from the most summarized to the most detailed perspective and creating a dimensional hierarchy – each level of the hierarchy can be viewed by different users at different situations (Datta and Thomas, 1999). A Commonly-used commercial DBMS products (e.g., Oracle, MS-SQL and Sybase) support OLAP cubes alongside with the more traditional relational/tabular database structures.
Figure 3: Star schema and Snow flake schema (Chaudhuri and Dayal, 1997)
9
Figure 4: Basic DW architecture (Kimball et al., 2002)
The implementation and the usage of DW have been associated with substantial benefits in organizations, as integrating multiple data sources, each reflecting different business aspects, and finding new sources of revenue in the given business environment by permitting rapid development of new analytical data usages (Watson et al., 2001), but at the same time, with significant difficulties. A few studies have described successful DW implementations in fields such as marketing, logistics and manufacturing, and demonstrated major contributions in large companies (Cooper et al., 2000; Even, 2007; Watson et al., 2001), e.g., reduce the amount of service calls in quality assuring department, help monitoring and analyze IRS tax payers, help marketing analysis for a financial services company, redefining corporate strategies, and supporting customer relationships management. Other studies highlighted possible difficulties with DW projects, which might prevent firms that invest in DW solutions from achieving the desired performance (Lee et al., 2004), such as lack of managerial support (Wixom and Watson, 2000)., and severe data quality defects as a result of unsuccessful integration (Nelson et al., 2005). Large DWs, require constant maintenance and quality-assurance efforts to ensure effective use (Redman, 1996), especially since the DW stores data from heterogeneous databases associated with several transactional systems (Chaudhuri and Dayal, 1997), and given differences in naming, domain definitions, identification numbers, and the like. For example, upon changes in source database schemas, ETL processes must be adjusted accordingly to meet the changed definitions (Elmasri and Navathe, 2006). In addition, using heterogeneous set of databases as DW data sources requires getting full and ongoing support from all parts of the organization and senior management involvement (Davenport, 2006) since DW implementations often consist of major organizational changes and
10 involve interactions with many stakeholders from multiple organizational departments (Elmasri and Navathe, 2006).
2.1.4.
Business Intelligence The term BI describes applications and technologies used to gather, provide access to and
analyze data about the organization for better business decisions (Druzdzel and Flynn, 2000; Lawton, 2006). In a survey of 1,500 CIOs, Gartner found that BI projects were the number one technology priority for 2009 (Gartner, 2009), and 68% of executives in top US firms believe that their company has to improve analytical capabilities to remain competitive (Accenture, 2009). Many companies, including retailers, telecommunications providers, travel agencies, and manufacturers, use BI for activities such as customer profiling, customer support, market research and segmentation, product profitability, and inventory and distribution analysis (Lawton, 2006). Mobile tools and wireless Internet protocols will likely to mark be among the next BI developments (Shim et al., 2002; Watson and Wixom, 2007). BI incorporates mainly four different forms of analytical tools and data usage styles: 1. Reporting - Tools for creating and distributing batch versus ad-hoc reports. Batch reports are typically created by an expert user and incorporate pre-defined structure and format, and distributed to users while ad-hoc reports are created by the users themselves. Spreadsheets (e.g., Microsoft Excel) are a common platform for generating reports (Figure 5), typically offering some BI capabilities such as pivot tables, and additional addons offered by BI vendors (Few, 2006).
Figure 5: Reporting using spreadshits
11 2. OLAP tools – Tools for interactive inquiry of OLAP data cubes (Figure 6), which present aggregations of data and permit "Slice and Dice" and "Drill Down" into cube measures and dimensions (Chaudhuri and Dayal, 1997).
Figure 6: Oracle OLAP tool
3. Digital Dashboards – Tools for snapshot overview of current business status, providing business performance measurement and key performance indicators (KPI) review and enabling business performance management (BPM) initiatives (Few, 2006). In a single or few screens, users can quickly see how actual performance compares to goals, benchmarks, and previous performance (Figure 7).
Figure 7: Dashboard with KPI in a Proclarity BI tool
4. Statistical Analysis tools – Tools for advanced reporting and data visualization which emphasize statistical tests (e.g. SPSS, SAS). Unlike simple reporting, statistical analysis
12 tools represent a proactive approach by not only reflecting what already happened but predicting future behavior (Figure 8).
Figure 8: Proactive statistical analysis
5. Data Mining - Tools for analyzing and finding knowledge in large datasets, which are typically automated or with minimal end-user intervention (Smyth, 2002). A BI environment would typically incorporate a few of the tools described above. Watson and Wixom (2007) note a few facilitating conditions that encourage value generation from the BI environment. First, it is crucial that the use of information and analytics would be part of the organization‟s culture. Second, there need to be alignment between the business and BI strategies and senior management support of the BI strategy. Finally, it is important that users have the necessary training, and support to use the BI tools. BI is especially valuable when the amount of available information elevate the intuition of an unaided human decision maker, and in which precision and optimality are important (Druzdzel and Flynn, 2000). Organizations realize the full value from their BI environment only when many employees use BI tools and BI benefits are spread to many departments and projects, thus getting more use out of the investment (Lawton, 2006). The implementation of BI environments can be risky and many BI projects fail because of inadequate planning, missed tasks, missed deadlines, undelivered business requirements, or poor quality deliverables (Moss and Atre, 2003). Other risk factors relate to features of the BI application. First, the user interface must be intuitive and easy to use to support users who are managers without computer training. To meet this challenge, vendors are either developing better interfaces, or incorporate BI tools into applications with well-known interfaces, such as Web browsers and Microsoft Excel (Lawton, 2006).
13 In addition, the vast amounts of data in the DW may cause information overload whereby the excess data available to the user makes it difficult to cope, delaying the decision-making process and causing cognitive confusion (Beller, 2006). According to Edmunds and Morris, (2000) information overload may reduce the quality of business decisions and decrease the economic performance. Possible solutions for information overload in the BI context are better data visualization with charts that effectively represent the data, use of contrast and colors and avoiding unnecessary decorations, avoiding too much details and using the right measures to be presented (Kearse, 2009). Without simple means to overcome information overload, the average employee cannot use BI to find useful data. This research suggests a way for reducing information overload by embedding recommendations into the front-end BI applications that guide users to the more valuable data that can be analyzed with the BI tool. The suggestion incorporates the concept of metadata discussed next.
2.1.5.
Metadata in Business Intelligence Environments Metadata is often described as “data about data” – an abstracted information about the
structure, the contents, and the integration of data resources (Marco, 2000). Historically, metadata has been treated as a „second class citizen‟ in data management literature, as its business value was not as clear as other investments in data projects (Sen, 2004). As a result, metadata collection often gets cut from IS projects plan despite the acknowledgment that it is important (Kimball et al, 2002). However, with the expending use of metadata to cover many aspects of database management, it has been recognized as having important managerial implications (Shankaranarayanan and Even, 2004). Today, metadata management in BI platforms is well recognized as an important capability needed for successful implementation of BI environments (Gartner, 2009). It has been suggested that metadata can be used to support a better use of complex BI tools (Moss and Atre, 2003) – a notion that this study extends. The metadata architecture in an analytical environment is complex; however, critical for uniformity and cohesion between the different components in a BI environment (Singh, 1998; Elmasri and Navathe, 2006). The unique needs of a DW and BI, stemming from the large volume and complexity of data resources that are managed in such environments, require managing different forms of metadata such as definitions of the source database, documentation of the
14 transformation rules that convert source data into DW formats, descriptions of back-end and front-end tools for managing the warehouse, definitions of the DW schema, data mart locations and contents, and data quality measurements (Shankaranarayanan and Even, 2004). While the importance of metadata from a technical perspective has long been recognized, only recently studies have started examining metadata from the business perspective, trying to understand the potential benefits of exposing certain aspects of metadata to the business user. Notably, despite evidence that support the argument for potential benefits, the integration of metadata into front-end tools is still very rare in commercial BI. Hess and Wells (2002), emphasize the importance of metadata in the BI context by presenting two case-studies of the use of a BI infrastructure in large companies. In both organizations, end-users were given metadata about the data they analyze. The results show that metadata integration improved the sense of reliability when end-users used the BI tools for making business decisions. Notably, the study showed that analysts appreciated metadata integration, despite the low quality of the metadata maintained. Chengular-Smith et al. (1999) combined metadata on the reliability of data in tools aiding the decision making process. They found that the inclusion of data-quality information is more valuable in situations where the decision-maker needs to find the best alternative while it may be less valuable when the decision process calls for the comparison of averages of larger subgroups. They conclude that database designers should include data quality metadata, since it affects the decision making process. However the complexity of the decision task is likely to affect the needed data quality measures and ways of presentation. A follow-up study (Fisher et al.,2003) claimed that incorporating quality metadata into databases is costly; hence it is important to understand the characteristics of the users that may benefit from this kind of metadata. They found that experts use the metadata substantially more than novice users. Also, older experts pay more attention to the metadata than younger experts. Decision support tools were also examined in the context of process metadata integration (Shankaranarayanan et al., 2006). Process metadata tend to support and aid contextual quality assessment; it describes how a particular data set was created and delivered. Process metadata claimed to be an additional source of value that organizations can provide to their decision-makers to improve decision outcomes. Empirical validation suggest that integrating process metadata do affect the decision making process which affects decision outcomes.
15 This study examines the integration of another form of metadata - "Usage Metadata", which tracks the use of data and applications. Despite the recognition of usage metadata as an important metadata form (Shankaranarayanan and Even, 2004), it has rarely been researched and today it is supported only by a limited number of commercial metadata products. The unique approach developed in this study for the collection of usage metadata and the generic module that was implemented to support such collection are described in detail in a later section.
2.2.
Data Usage and the Associated Business Value This study suggests that the benefits that firms can gain from data resource, particularly in
the context of BI, can be significantly improved through better understanding of usage and the associated economic outcomes. A plethora of studies have examined the usage of IS, and of data resources in particular, from different angles. This section starts with reviewing relevant aspects of data usage and the concept of associating it with utility, as a measure for the value generated from data use. Later, it highlights the challenge of tracking the use of data and associating it with economic outcomes, reviews usage metadata as a possible way to capture IS usage, and discusses the strengths and the weaknesses of current solutions for data usage tracking.
2.2.1.
Data Usage and Utility In recent years, enormous amounts of data are generated by traditional sources, such as
OLTP systems, as well as by new sources such as point-of-sale records, click stream data (Bucklin and Sismeiro, 2003), and radio frequency identification (RFID) (Wang and Liu, 2005). The increasing investments in data management raise the issue of benefit – what benefits and business value do organizations gain by possessing and managing large data resources? The question of value and return on investment has been raised in broader IS contexts, as spending on IT is rapidly increasing and, in recent years, often reaches 30-50% of the capital investment in firms (Carr, 2004). A plethora of studies have examined the question of IS value – how firms benefit from IS implementation, and how successfully investments in IS translate into revenues and/or promote them. The usage of IS has long been identified as an important factor in explaining IS success and the associated value gained (DeLone and McLean, 1992). Actual usage of IS strongly explains their success and payoff – even more than the level of ICT-investments made (Deveraj and Kohli, 2003). Burton-Jones and Straub (2006) review a few conceptualizations of IS usage in past research – e.g., based on the extent, the nature, and the
16 frequency of use. They state that usage event is a triplet comprising of user, system and task. The extent, to which the user employs the system to carry out the task, is the richest usage data that can be collected according to them. Although studies of IS usage emphasize it as a key for IS success, none of them suggest to capture the actual data usage being made through the use of the IS. Ahituv (1980) suggests the concept of utility as a tool for assessing value creation through the use of information systems. Utility can be conceptualized, for example, as the difference in the outcome between a decision based on certain information and the one that could be taken without having the information available. Ahituv (1980) suggests linking information systems and resources to value and benefits by defining a multi-variable utility function, which maps information system's attributes and configuration decisions to a measurable outcome. Using the utility mapping, he suggests a structured way for evaluating alternative data model designs, and figure out the most cost-effective choice. The question of usage, utility and value is particularly relevant in the field of data management – as the market for data management hardware, software, data resources, and consulting services is estimated in $100's of billions in the US alone (Wixom and Watson, 2001). As the volume and the complexity of data resources grow, it is critical to understand and evaluate economic factors such as the business value those resources generate, costs for managing them, and their overall contribution to the organizational economic performance (Even, 2007). The benefits that can be gained from large data resources are apparent in many business fields, but these benefits are often difficult to quantify and/or express in monetary terms. Unlike other physical resources, data resources are not subject to market forces and pricing mechanisms; hence, the need to assess the value of data in a more contextual – depending on the specific organizational setting and task domain (Watson et al. 2001). Moreover, data resources do not have “stand alone” value, but their value contribution materializes as they are being used - to enable business processes, support managerial decision making, and/or generate revenues as a commodity. These processes are dependant in a specific business, time and organizational context (Levitin and Redman, 1998; Even et al., 2007). Ignoring the value of data might lead to incorrect data management decisions – refrain from investing in data with high potential business contribution, or over-investing in lessimportant data. Further, the costs associated with acquiring, storing, and maintaining data are high. Beyond storage and maintenance cost – too-large data resources may damage search,
17 retrieval, usage performance and overload the decision maker with unnecessary information (Levitin and Redman, 1998). Viewing data as an important resource, Ronen and Spiegler (1991) have argued that data management can rely on known methodologies (e.g. "Just-in-time") that enable the application of inventory management of other, more traditional, resources. The view of data as a resource is broadly adopted in the data quality management literature – studies in that field (e.g., Wang, 1998) promote adopting common quality management methodologies, such as the Total Quality Management (TQM), and applying them to data resources as well. Adopting the utility concept, Even et al. (2007a) define "data utility" as a numeric measure that reflects the value gained by using data resources within a specific business/task context. They propose an analytical framework for optimizing data management design and maintenance decisions, through understanding the link between dataset characteristics and utility/cost tradeoffs. Certain dataset and system attributes, such as quality level, time span, choice of attributes and system configuration are seem by this framework as decision variables that may affect utility, costs, and the overall net-benefit in data environment. The optimization framework suggested by Even et al. (2007a) maps these decisions to economic outcomes and can help optimize their configuration. The utility associated with data records, as reflecting the business benefit gained, may have significant variability (Even et al., 2007b). Inequality in large databases can be analyzed, using common methods, used in other research fields, such as Lorentz's Curve, Gini's index and ANOVA tests. Even et al. (2007b) demonstrate "utility inequality" in an alumni database of a large university. The alumni database, storing data about potential donors, can be viewed as a special form of CRM (Customer Relationship Management) – a class of systems which manage data on costumers in commercial business. Inequality in this database has been found in a few aspects - the chance of getting a new donation from a person whose most recent gift was a long time ago appears to be very small, leading to the possible conclusion that older records are generally less significant than newer ones. The different schools in the university also show high inequality in the number of donations made by the school graduates, leading to the possible conclusion that the data that is saved on certain schools generates higher utility then other schools. Inequality in the utility of data may have some important managerial implications (Even et al., 2007b). In the presence of high inequality – the organization may consider managing differently records with high utility contribution in terms of targeted quality level, inclusion/exclusion of attributes, time span coverage and data granularity. Another decision that
18 can benefit from adopting an economic perspective and assessing inequality is the how to treat older and/or unused data – sustaining, archiving, eliminating it entirely (Even et al., 2007a). Investment in a powerful DW infrastructure will be harder to justify if the vast majority of the utility can be obtained from a small fraction of the data resources, which can be effectively managed with a simple and inexpensive system. Utility inequality can impact data acquisition and retention policies as the value gained from purchasing and maintaining large data volumes is not necessarily obvious (Even et al., 2007b). This research considers possible inequality in data resources, and links it to the tracking of usage. This inequality approach is equivalent, to a great extent, to the ABC analysis which is very common in managing the inventory of tangible resources (Ramanathan, 2006), ABC suggests that inhabitants of certain entities, like inventory, events and even people, conceal within them, a natural inequality. The approach suggests classifying entities into groups (A, B and C) and managing each group differently according to its characteristics. The main difference between the utility measurement approaches in former studies to the one presented in this study is that here the utility of data is measured not only by analyzing the data itself, but rather by observing the usage of data as the utility generator. In addition, while most of the research done so far on data utility was focused on data management, this research is focused on leveraging data utility for enhancing front-end BI tools.
2.2.2.
Tracking Data Usage – Solutions and Issues This study introduces a novel form of metadata for tracking the usage of data resources
and associating it to business value. Academic literature does not offer comprehensive research on methods and architectures for collecting usage metadata in databases and only small number of commercial products offers solutions for monitoring data usage (Ambeo, 2009; Zetapoint, 2009). However, there is a greater recognition in recent years of the need to track usage patterns Bollen and Van de Sompel (2006), for example claim that a user-driven revolution is underway in which end-user services are no longer solely based on top-down design decisions, but have come to prominently include the analysis of user actions and preferences. A plethora of studies addressed usage tracking aspects in the Internet domain (e.g., Cooley et al., 1997; Shahabi et al., 1997; Nasraoui et al., 2000). Internet technologies permit the collection of "clickstream" data - tracking and recording web-usage behavior, which can be seen as a form of usage metadata (a listing of many web usage tracking utilities can be found at http://www.uu.se/Software/Analyzers/Access-analyzers.html). Clickstream data gives researchers
19 and practitioners the opportunity to study how users browse or navigate websites and to assess site performance in various ways. Analysis of clickstream data can help gaining important insights toward improving website design, customization, and ongoing monitoring of performance (Bucklin and Sismeiro, 2003). Joshi et al. (2002) describe an approach for creating a DW that is populated by web logs – tables which contain data on user sessions in a web site. Analyzing and exploring the data in such a DW and, by that, finding regularities in the behavior of the users accessing a web site can improve website performance, enhance the quality and delivery of Internet information services to the end user, and identify population of potential customers for electronic commerce. Clickstream data collects usage metadata mainly on the userinterface; however, it does not track the usage of the actual data in a repository, which is a key concept of this study. A few studies have explored metadata solution for tracking the usage of data in the backend databases management system itself, rather than in BI tools. Saharia and Babad (2000) propose an enhancement to the traditional data warehouse structure, by building an adaptive query cache in which the definitions of recently written queries as well as their answers and access paths are kept as part of the metadata. The objective is to add intelligence at the warehouse level so that the system remembers recent work it has performed, and by that speeds up the answers given to future queries. Saharia and Babad focus mainly on improving technical aspects of DW and BI performance, but do not address using this metadata in order to improve front-end tools that analyze the data such as BI applications. Bollen and Van de Sompel (2006) suggest architecture for collecting usage data of scholarly information services (e.g. digital libraries). The data collected for each usage event can be described as a triplet containing answers to three main questions: 1. What is the item for which the usage was recorded, e.g. a journal article? 2. Who is the originator of the event, e.g. the user? 3. When is the time at which the event occurred, i.e. the event‟s timestamp? The resulting collection of this initiative consisted of 3,507,484 unique usage events. From the recorded events, usage patterns were derived, and were used to build a number of tools and applications, related to the field of digital libraries: 1. Recommendation tools (e.g. if a user asked for a certain article, related articles were recommended to).
20 2. Usage impact ranking which ranks the journals according to the number of events in which they were asked by users. 3. Journal level-of-interest mapping that enables easy clustering of journals into topics according to similar usage patterns. Bollen and Luce (2002) propose a quantitative approach to Digital Library (DL) evaluation that analyzes the retrieval habits of users, and helps to reduce managerial decisions that are based on the management intuition. On-line DL services generally maintain a log of user requests in a set of log files, similar to the web-logs that are often collected by web servers. These log files contain detailed information regarding user requests such as the originating IP number, date and time of the request, the requested document, etc. The DL logs are used to find relations between articles and to generate a cluster analysis and network graphs for visual inspection of the usage metadata. The researches described above track usage in large data resources in order to improve the search process of academic article, but do not link, and does not collect the data necessary for value estimation of the data in this repositories. Although recording of usage data is common in scholarly information services, its exploitation for the creation of value added services remains limited (Bollen and Van de Sompel, 2006). Bollen and Van de Sompel (2006) describe several issues in existing usage metadata solutions. They point mainly user privacy, data validity and the lack of accepted standards for the representation, sharing and aggregation of usage metadata as the major drawbacks that constraint the creation of value-added services, based on usage metadata. They point out future research to be directed towards an investigation of the scalability of different aggregation architectures of usage data, models to protect user privacy, mechanisms to ensure data validity and a range of technical issues associated mainly with referent identification. Until recently, commercial products that manage metadata did not offer comprehensive solutions for usage tracking. For example, a 2003 survey of commercial product (Shankaranarayanan and Even, 2004), has shown very little evidence for usage metadata support in BI/DW products. Today, a few commercial solutions are suggested for usage tracking in large data resources. Among them are "STATSPACK" by Oracle (http://www.oracle.com), "Usage Tracker"
by
Amebo
(http://www.ambeo.com),
and
"DB*CLASSIFY"
by
Zetapoint
(http://www.zettapoint.com). For example, DB*CLASSIFY (Zetapoint, 2007) is used by practitioners to track and collect usage metadata and then classify the collected data into certain groups according to the frequency of usage events that accessed it, where frequently-used data is
21 considered to be more valuable then seldom-used data. The classification is performed by analysis of actual database activity. DB*CLASSIFY captures in real-time database queries, along with contextual information such as the time, username, application and all other session-related information. The Zetapoint's optimization and data classification methods, as well as the methods used in other commercial products, is driven mostly by technical aspects – reducing storage space, accelerating data processing (ETL), and query retrieval speed. This approach assumes that the data with the most value is the data that is used more often and vice versa, without correlating the content of the data to its business context. For example, according to Zetapoint's optimization schema – if a certain dataset is queried or updated often, it is automatically classified as important (Zetapoint 2007). Arguably, this approach might turn out to be a major drawback if the business value gained by using this specific data subset is low. This study explores an important usage-tracking aspect, not addressed before - linking data usage to the associated business value. It contributes a new, generic, module for collecting value-driven usage-tracking metadata. This module does not only address technical aspects (as done, for example, in Zetapoint's solution), but also informs on value contribution of data. As shown in this study, the value-driven usage metadata captured can be used to recommend users to use certain valuable data items when analyzing the data for decision making. 2.3.
Recommender Systems Recommender system is an automated mechanism, which provides end-users with some
rating of information items not seen so far (Adomavicius and Tuzhilin, 2005). Recommender systems aim at improving usability and decision-making outcomes, enhancing the end-user‟s experience, and reducing information overload (Wei et al., 2005; Sun et al., 2008). Such systems are very common today in E-commerce and social-network websites. However, there is almost no evidence, neither in the academic nor in the practitioner literature, for implementation of recommender systems in BI environments. This study attempts to address this gap by proposing a new form for developing and integrating recommender systems in BI environments. This gap deserves further investigation, as BI environments manage vast amounts of data, have a strong impact on managerial decision making and business performance (Davenport, 2006), and are often subject to information overload issues (Beller, 2006). Adomavicius and Tuzhilin (2005) identify two categories of methods for generating recommendations – "content-based" versus "collaborative-filtering". Methods under the former
22 category recommend items that are similar to those preferred by the user in the past, while methods under the latter category recommend items that other users with similar preferences have liked. Collaborative filtering methods typically estimate the similarity of two users according to their past ratings. It estimates the rating of a user to un-seen item according to the ratings of similar users to the same item (Figure 9). The more similar the two users are - the higher is the weight that will be given to the known rating of one, when predicting the preferences of the other and assessing the rate, accordingly (Adomavicius and Kwon, 2008). To improve prediction accuracy, some collaborative filtering techniques also refer to the ratio between the number of users and the items to be fitted, the ratings scale and other properties of the underlying data set (Herlocker et al., 2004). Adomavicius and Tuzhilin (2005) describe a few recommender-systems methods, which can be seen as a hybrid between collaborative filtering and content based approaches. The recommendation form that is tested in the current study, described further later, can be seen as such a hybrid.
Figure 9: Collaborative filtering approach – users' similarity affects the rating prediction
A typical recommendation process starts with the specification of initial rating set, either provided explicitly by the user, or inferred by the system (Sun et al., 2008). Users offer feedback on purchased or consumed items and the recommender system uses the information to predict their preferences for yet unseen items and, subsequently, recommends items with the highest predicted relevance (Adomavicius and Kwon, 2008). Once the initial ratings are specified, a recommender system tries to estimate the rating function for the user. Importantly, most recommender systems techniques today are highly domain dependant and their performance relies significantly on the underlying data's characteristics; hence, the ability to generalize a
23 recommender system beyond the field in which it was tested is limited (Herlocker et al., 2004; Adomavicius and Kwon, 2008). Recommender systems are a common practice today in commercial and social websites (McNee et al., 2003; Adomavicius and Kwon, 2008). A well-known recommender system is the one used by Amazon (www.amazon.com) which recommends users of books and other products they might be interested in (Figure 10). The system generates recommendations based on a collaborative-filtering approach, stating the products in which people with similar preferences to the user have expressed interest.
Figure 10: Amazon.com collaborative based recommender system
Another well-known application of recommender system is "Cinematch", used by Netflix (www.netflix.com) which recommends users movies that they may want to rent, based on common interests with other users. A content based music recommender system was developed by Pandora (www.pandora.com), which recommends songs to users, based on mapping the musical attributes of songs that the user has rated as high. Recommendations can be also found in social networks such as Facebook (www.facebook.com) and Linkedin (www.linkedin.com), which recommends people as potential friends and/or colleagues. Digital libraries are another important information-retrieval context in which recommender systems were studied (Song et al., 2006). Sun et al., (2008) have showed that analysis of user's logs can generate personalized recommendations for academic documents, based solely on the implicit feedback from the user.
24
2.3.1.
Recommender Systems Evaluation Defining the metrics on which the recommender systems should be measured is a
complicated task, and the literature offers a variety of quantitative metrics and qualitative evaluation techniques (Herlocker et al., 2004). A common approach for recommender systems evaluation is to assess the accuracy of the recommendations, e.g., by measuring the MSE of the predicted ratings versus the actual user preference (Song et al., 2006). However, accurate recommendations alone do not guarantee positive user experience - additional aspects such as the extent to which a user is satisfied by the recommendations and the degree to which the recommendations made are not obvious must be examined as well (Cosley et al., 2003; Herlocker et al., 2004). Swearingen and Sinha (2001) carried out a study to investigate not only the accuracy of predictions but also the perceived usefulness, ease of use and usability of recommender systems. Such perceptive measurement would typically use a survey and interview methods. On the other hand, process factors such as the amount of time and effort required to complete a task would typically be measured implicitly by logging user behavior (Herlocker et al., 2004). Recommender system evaluation can be done offline (e.g., by analyzing a dataset of certain user's ratings), or online with users who actually use the recommendations during the experiment (Sun et al., 2008). A major weakness of offline approach is that such analyses are limited to objective evaluation of prediction results, but cannot reasonably assess user preferences, satisfaction, and intention to use the recommendation. Offline analysis cannot determine, for example, whether users prefer a particular system because of its predictions capabilities, or because of other less objective criteria such as the aesthetics of the user interface. An alternative approach to offline analysis is to conduct a live-user experiment, such as controlled lab studies (e.g., with random assignment of subjects to different conditions), or field studies where a particular system is made available to a community of users (Herlocker et al., 2004). A lab study typically tests pre-defined hypotheses under controlled conditions, while a field study can reveal what users do in their own actual context (Herlocker et al., 2004). Lab experiments also differ from field studies by the time window of measurement - the former, usually implies a relatively short time period, while the latter permits a longer period in which dimensions such as long term dissatisfaction from the system can be evaluated (Turpin and Hersh, 2001).
25 This study includes a task-based lab experiment. In such experiments, subjects are assigned a set of clearly-defined tasks to complete, while various recommender algorithms or interfaces can be tested to find out which ones lead to the best task outcomes. For example, McDonald (2001) tests a system that was developed within a particular organizational task-based context. That study attempted to situate users in a task context that would lead them to evaluate the recommendations given within that context. Subjects were given a rich scenario for evaluating recommendations, which specified a general topic area and a specific problem. Taskbased lab experiment was also conducted by Shankaranarayanan et al. (2006), who tested the integration of data-quality assessments and data-process maps into a decision support tool.
2.3.2.
Accuracy and Trust Issues in Recommender Systems A significant challenge in designing recommender systems is ensuring the accuracy of
recommendations, and building up the trust of users in the recommendations made. Cosley et al. (2003) point out a possible bias in generating new recommendations, caused by exposure to previous ones. When presenting subjects with sets of movies they had rated in the past, that have found out that the predicted ratings did influence user ratings, causing a significant bias in the direction of the prediction. If such a bias indeed exists, user-interface design should let users concentrate on rating, while minimizing the focus on previous predictions. Cosley et al., also found that the control group which didn‟t get false recommendations expressed significantly more satisfaction than the experimental group who did get false recommendations. Experimental users sensed that predictions were inaccurate and that this inaccuracy led to an overall decrease in liking of the recommender system; hence the importance of choosing an accurate algorithm for generating recommendations. Swearingen and Sinha (2001) suggest that in order to develop trust in a recommender system, the functionality of the underlying algorithm needs to be transparent to the user to an extent. Another possible way to facilitate trust would be to generate some recommendations on items that the user is likely to have known or rated before, as users might get confused if all recommendations are unrelated to unknown items. In order to increase user's trust in the system, McNee et al., (2003) suggest that recommendation system would be added with confidence metrics (e.g., based on the number of ratings available for an item) indicating how confident the system is in the recommendation given. They found that adding a confidence display to a recommender system increases the overall user satisfaction, although it decreases the trust of
26 experienced users in the system. They have also examined the effect of training users on how to use the confidence metric. Providing training to new users increased user satisfaction over just adding the confidence display to the system. Providing training to experienced users increased their usage of the confidence system, however, decreased their overall satisfaction with the recommender (McNee et al., 2003) In the experiment described by Swearingen and Sinha (2001), subjects used three different recommender systems. Based on the results of this experiment, which included surveys and interviews, Swearingen and Sinha suggest few design guidelines for recommender systems. If additional ratings would lead to better predictions it would be more beneficial from the perceived usefulness perspective to start giving recommendations only after the initial rating set would be large enough. In addition satisfaction and ease-of-use ratings were higher for the systems that also supplied some basic contextual information about the item being rated on the same page (e.g., a picture). Recommender systems can include recommendations of new, just released items, but such recommendations should be a separate category of recommendations, leaving the choice of accessing them to the user (Swearingen and Sinha ,2001). 2.4.
Literature Review Summary This study lies on three premises, which were reviewed in this section – a) Business
Intelligence (BI), b) Data usage and the associated business value, and c) Recommender Systems. Table 2 lists a few key studies in each of these research streams that has influenced this study and summarizes their key contributions.
27 Research Stream Business Intelligence Environments
Key Studies
Contributions and Influences
Chaudhuri and Dayal, 1997 Marco, 2000 Watson et al., 2001 Kimball et al., 2002 Sen, 2004 Sen and Sinha, 2005 Davenport, 2006 Lawton, 2006 March and Hevner, 2007 Watson and Wixom, 2007
Usage and Utility of Data Resources
Ahituv, 1980 Wixom and Watson, 2001 Deveraj and Kohli, 2003 Even and Shankaranarayanan, 2004 Burton-Jones and Straub, 2006 Bollen and Van de Sompel, 2006 Elmasri and Navathe, 2006 Even and Shankaranarayanan, 2006 Even et al., 2007 Zetapoint, 2009
Data management challenges and costs Data utility as a numeric measure Data resources design to maximize utility Utility inequality in large data resources Usage explains IS success and payoff Usage as frequency of use Usage metadata as a way for representing data usage Current solutions for usage tracking
Recommender Systems
Swearingen and Sinha, 2001 Cosley et al., 2003 Herlocker et al., 2004 Adomavicius and Tuzhilin, 2005 Shankaranarayanan et al., 2006 Adomavicius and Kwon, 2008 Sun et al., 2008
Distinction between content based, collaborative filtering and hybrid systems Recommender system applications in web and e-commerce Measuring recommender systems success – objective and subjective Recommender systems research methodologies
Distinction of OLTP and OLAP DW architecture Benefits of analytical use of data BI as a tool that enable better business decisions BI as a top priority technology DW/BI as a complex environment The importance of metadata in the DW context
Table 2: Literature review summary
An important goal of this study is bridging on some gaps in the related streams of research. Data management and BI literature emphasizes the importance of data in today's organizations, and highlighted the need to treat data and the systems that manage and use it as a critical resource. Despite this criticality, and the high cost of DW/BI implementation, there is very little discussion in literature and almost no commercial solutions that take cost-benefit tradeoffs into account systematically when designing a BI environment, the DW, and ETL processes. Another issue that is raised often is the high complexity of front-end BI tools, which
28 often require extensive training and advanced analytical skills. Embedding metadata within the tools was suggested as a possible solution for improving the decision making process and outcome. Very little research was done on this topic and none of it addresses the integration of usage metadata. Several studies suggest that the business value of IS and data stems from their use. However, the literature about economic perspective of data management does not offer much practical solutions for quantifying the utility of data resources in a systematic way. Commercial tools that monitor data usage are mainly geared towards improving data administration policies, rather than enhancing end-users experience and supporting their needs. Recommender systems are a common practice in e-commerce and other web-oriented domains. However, there is a gap in the academic literature regarding the use of recommender systems in the BI context. Current literature on recommender systems focus mainly on the algorithmic side, with some discussion on ways for designing and evaluating such systems. This research examines a recommender system embedded in the BI environment, toward facilitating better business decisions. As emphasized in the literature, the design of recommender system must take into account issues such as the accuracy of the recommendations given, and end-users trust.
3.
Conceptual Framework The conceptual-framework development starts with a distinction of two usage metadata
forms: the common approached, termed as frequency driven usage metadata, versus the novel approached proposed, termed as value driven usage metadata. It then defines a recommendation mechanism in the BI context, based on the collection of value-driven usage metadata. 3.1.
Frequency-Driven versus Value-Driven Usage Metadata Research often describes a complex data environment, such as DW/BI implementations,
as a manufacturing process, consisting of interconnected stages such as acquisition, processing, storage, retrieval and usage (Ballou et al., 1998; Shankaranarayanan et al. 2006). For the purpose of developing the framework for this study, this manufacturing process can be conceptualized as having two high-level stages – data administration versus data consumption (Table 3) - each associated with different stakeholders, goals, motivations and tasks. Data administration would typically address technical aspects – providing the IT capacity needed to store and process data
29 (e.g., hardware, databases, and back-end processes), and the tools for implementing informationproduct outcomes (e.g., datasets, reports, and analyses). Data consumption, on the other hand, would seek to transform data resources and information products into business value through their effective usage. Data consumers would typically concern more about the availability and the quality of data resources, and less about the technical aspects associated with managing them.
Data Administrator Maximizing efficiency – providing the capacity and performance needed to support data usage at the lowest efforts and costs
Data Consumer Maximizing effectiveness – gaining Motivation business benefits and increasing profitability, by supporting business processes and decision making tasks with the appropriate data resources Choosing the data management IT. Having the right data at the right time Goals Determining appropriate data and the right place. acquisition, retention, storage, and quality improvement policies Information Which data components are used Which data components hold valuable more often? inputs for the decision task? needed Table 3: Comparison of the two generic data stakeholders
Tracking the usage of data subsets (e.g., tables, attributes, and records) and applications in DW/BI environments has been identified as an important form of metadata (Shankaranarayanan and Even, 2006). Usage-tracking utilities are offered by some specialized commercial solutions and, to an extent, by database management and BI platforms (e.g., DB*CLASSIFY, Oracle STATSPACK). The common approach implemented by today‟s solutions is termed by this research as frequency-driven usage metadata (Figure 11a), as this approach is based on tracking data-retrieval requests and identifying the data subsets being most-frequently used (e.g., by parsing the underlying SQL statements). Usage tracking utilities today are geared mainly toward supporting data administration – For example, Zetapoint (www.zetapoint.com), a provider of a powerful usage metadata tool, targets mainly data administrators in its marketing efforts. Indeed, frequency-driven metadata collection may provide important inputs to the data administrator, toward improving system design and prioritizing data and system administration efforts. It is common in databases that some records and attributes are accessed more frequently than others. The assumption that motivates the frequency-driven approach is that frequent usage reflects higher importance. Accordingly, the results of frequency-driven usage tracking may lead to important data management decisions such as giving frequently-used data subsets higher
30 priority in terms of data-quality improvement – i.e., watch these data subsets more closely, detect and correct defects, and make sure to keep them up-to-date. Detecting frequently used items may also have important implications for the configuration of system capacities - a data administrator may consider, for example, transferring less-frequently used data subsets to archives in order to reduce load on active systems and improve data-retrieval performance.
(a)
(b) Data Administration
Data Resources
Retrieval
Data Administration
Data Consumption
Information Products
Usage
Business Value
Data Resources
Retrieval
Data Consumption
Information Products
Usage
Business Value
Metadata Layer
Metadata Layer Frequency-Driven Usage Metadata
Metadata Integration
Frequency-Driven Usage Metadata Value-Driven Usage Metadata
Figure 11: (a) Frequency-driven versus (b) Value-driven usage metadata
While seeing the merits of collecting frequency-driven usage metadata for data administration, this study questions – to what extent does it benefit data consumers? Frequent usage may reflects higher significance of certain data subsets to data consumers; hence, to an extent, a higher value-contribution potential. Conversely, one could argue that frequent usage reflects certain stagnation and tendency to "dig into the same well" – i.e., re-using certain data subsets repetitively, while possibly ignoring unused subsets with high contribution potential (Kirs et al., 2001). Therefore, a potential risk with basing data management decisions solely on frequency-driven metadata is a possible loss of opportunity to benefit from data subsets not frequently used so far - e.g., due to data quality or accessibility issues. There is possibly no "clear cut" answer to this question, as it largely depends on business context, the timing, and the nature of usage tasks. However, this study suggests that important insights can be gained from tracking not only the frequency of data usage, but also the decisions made based on the data being used and the associated value gains. This novel approach is termed as value-driven usage metadata (Figure 11b). The notion underlying this approach is that the purpose of using data and information resources is generating value. Shapiro and Varian (1999) argue that information resources and products don‟t possess “stand-alone” value, but can rather be attributed with value only through usage and experience.
31 The benefits gained from the use of information products have been conceptualized as utility (Ahituv, 1980), which can be measured in terms of revealed value (objective, measurable), perceived value, and/or willingness to pay. Organizations commonly capture value measurements at the business process and/or at the firm level (e.g., production increase, sales activity, revenues and costs) and store them in dedicated data repositories. However, organizations rarely link these value measurements neither to the data resources nor to the decision-support tools that were used in the process of value generation. Aral et al. (2006) argue for the importance of creating such a link, and show that it can help explaining information-workers productivity. Utility assessments have been used to optimize the configuration of data processes (Ballou et al., 1998), DW datasets (Even et al., 2007), and data-quality improvement policies (Even and Shankaranarayanan, 2008) – all are typical data administration tasks. This research suggests that, beyond the benefits offered to data administration, collecting quantitative value assessment as a form of metadata can improve data consumption as well. The novel usagetracking approach promoted in this study (Figure 11b), extends the current common approach (Figure 11a), by combining measurements of both the frequency of use and the associated business-value assessments. In certain cases, value assessments can be based on data that exists within the same data resource (e.g., sale transactions, linked to marketing campaigns that were based on analysis of previous sales). In other cases – such assessments may reside in other information resources such as CRM and accounting systems. The integration is done by observing data at the decision-task level – observing the value gained by decision task outcomes and linking it to the queries that have supported each task. A key challenge with the value-driven metadata collection approach is the fact that most data environments today are not designed to establish an explicit link between decision outcomes and the underlying data and queries. An implicit link can be possibly created through inference mechanisms - e.g., by comparing user identifiers and time-stamps (e.g., Even and Shankaranarayanan, 2008). The difficulty of linking decision outcomes to the data being used can certainly be seen as a limitation of this study. This issue is further discussed in the concluding section, and some directions for future research to address it are proposed. In this study, the evaluation is done in a specific marketing context in which a reasonable link between data usage and decision value can be established, as further explained in following sections. Once the link between decision outcomes and data usage is established (e.g., through the parsing of queries), different methods can be considered for attributing value to specific data
32 subsets and objects (i.e., records, attributes) within those subsets. To permit a parsimonious development of the novel usage metadata concept, this study initially uses a relatively simple value-attribution method, which attributes the value to the last in a sequence of queries that support a decision task. The assumption is that, to support a certain decision task, users query repetitively a certain tabular dataset with N records indexed by [n] and M attributes indexed by [m]. Users run Q queries indexed by [q], each associated with a value Vq, which reflects the business benefits gained by the decision outcome The binary indicator Rqn indicates whether record [n] was retrieved by query [q] (Rqn=1), or not (Rqn=0). Similarly, Rqm indicates whether attribute [m] participated in query [q] or not. The value of a certain query (Vq) is attributed between the participating data items, using a certain value-attribution function Vqn,m=u(Vq, Rqn, Rqm), such that Vq=ΣnΣmVqn,m. The preliminary method used in this study applies an equal attribution of value among all participating data items. Accordingly, the overall value attributed to a certain data items Vn.m is given by (Eq. 1):
(1) Vn,m q1..QVnq,m q1..Q u V q , Rnq , Rmq q1..Q Rnq RmqV q Q, q M, N Vq, Vqn,m uRqn, Rqm -
n1.. N
m1..M
Rnq Rmq , where
The number of queries performed, and the corresponding index, respectively The number of attributes (indexed [m]) and records (indexed [n]), respectively Query [q] value, its attribution to data item [n,m], respectively The attribution function used Binary indicators of the participation (=1) of record [n] and attribute [m] in query [q], respectively
Value-driven metadata is illustrated through a simplified example of a table used by marketing associates to decide which customers will be approached when promoting a new product (Figure 12). This example would be the base for the experiment described next. An associate use a BI tool to investigate previous sale transactions. The tool generates queries directed to the tables, such as those demonstrated. It is assumed that each query has led to a certain promotion campaign in which a group of customers has been approached. Some customers may have responded to the campaign by making purchases, and the overall value attributed to a query is the total purchase amount. As illustrated, this value proxy may significantly vary among queries. The value is now being spread (Eq. 1) to assess the relative value of each data object (the cumulative value is indicated per record and per attribute). For comparison, frequency-driven metadata is also calculated – i.e., the number of times that each record or attribute participated in queries (this number is indicated in brackets, near the value score).
33 Customers …
#
Customer
Gender
Income
Children
Status
Value
Value
1
Abraham
Male
High
0
Single
1000 (1)
0
2
Sarah
Female
Low
1
Married
510 (2)
< 100
3
Isaac
Male
Medium
2
Married
50 (1)
< 1000
4
Rebecca
Female
Low
0
Single
10 (1)
≥ 1000
5
Jacob
Male
Medium
3
Married
50 (3)
6
Lea
Female
High
2
Married
1510 (3)
7
Rachel
Female
Low
4
Single
0
Color
… Value
515 (3)
2000 (1)
60
(2)
500 (1)
Queries SQL-WHERE condition
Attributes Used
Records Used
Value
Gender = „Male‟ and Children > 0
Gender, Children
[3], [5]
100
Gender = „Female‟ and Children < 3
Gender, Children
[2], [4], [6]
30
Gender = „Female‟ and Status = „Married‟
Gender, Status
[2], [6]
1000
Income = „High‟
Income
[1], [6]
2000
Figure 12: Example of frequency-driven and value-driven usage metadata assessment
As illustrated by color-coding – some records and attributes possess higher value than others. Moreover, the “value-map” looks significantly different than the one based on the attribution of usage frequency. For example, the Income attribute, which was less frequently used, is associated with the highest value, while the Children attribute, which was more frequently used, is associated with lower value. Insights as such can be transformed into valuable recommendations for marketing associates the next time they plan to run a similar campaign. The value-driven usage metadata module suggested by this study records the business-value associated with the decisions involved in specific instances of certain business tasks. As noted earlier the value driven usage metadata can support both data administrator to the data sets used more frequent and both the data consumer who need a direction to the more valuable datasets (Figure 13). By this approach the metadata module close the loop of past decision outcomes to the new decisions being made.
34 Business Task
Data Resource
Input data
Decision Task
Decision Outcome
Value gained
Queries Performed
Economic Value Data Components Used
Usage Metadata
Data Consumer
Which components in a data resource hold valuable inputs for the decision task
Data Administrator
Which components in a data resource are used more often
Figure 13: Value-driven collection of usage metadata
The methodology presented in this research can be interpreted as a form of taking advantage of the "wisdom of the crowd" to gain important knowledge. James Surowiecki, in "Wisdom of the crowd" (2005), presents numerous examples in which the wisdom collected from the decisions or beliefs of many individuals outperform the decisions made by experts and specialists. The individual decisions on how to use the organizational data represented as queries, are collected from the crowd of data consumers. The decisions of the crowd are aggregated, analyzed and presented as recommendations in the BI tool. The approach presented in this research can be found in a certain variation also in Google's PageRank algorithm (Brin and Page, 1998). The PageRank algorithm is based on using each link from web site A to web site B as a vote from A to B. the more votes a web site got have a direct impact on its value, and eventually on its rank in the search answers presented to the user. But PageRank algorithm doesn‟t look only
35 on the number of votes each web site got in order to determine its rank, it also weigh each of the votes according to the relative importance of the web site that voted. The value driven usage metadata module uses a similar approach, when each of the queries "votes" for certain data elements. Similar to the concept of PageRank algorithm the value of the votes of each query is different and relates to the value of the decision the query supported. 3.2.
Value Driven Feedback and Recommendation Mechanisms As a contribution to improving the effectiveness of DW/BI environment, this research
proposes to integrate a feedback and recommendation mechanism (FRM), a form of a recommender system, into BI tools. The assumption is that FRM capabilities would help the enduser navigate through complex data resources and highlight new usage directions with a high benefit potential, while maintaining simple and easy-to-learn functionality. FRM, in the form visual cues, would provide the end-user with feedback on the actions taken so far, and some guidelines for further actions to consider – e.g., approach certain data subsets and/or apply certain analysis forms. The integration of recommender systems into BI tools is not a common practice today, and has not been significantly explored so far, although some commercial software vendors have introduced recommendation utilities to an extent (e.g., Bissantz, in their data mining tool, http://www.bissantz.com/deltamaster). The motivations that drive the integration of recommender systems into websites and digital libraries - improving decision-making outcomes, enhancing end-user‟s experience, and reducing information overload – are all strong motivations in BI environments as well. Similar enhancement of BI systems may have important contribution to a better usage of the BI tools, and improve the decisions made.
36
Figure 14: A BI tool with FRM (Rates presented next to possible next steps of data analysis)
Figure 14 offers a simplified illustration of integrating FRM capabilities into a BI tool that lets the end-user navigate through sales data, and slice it along certain customer characteristics (e.g., Income, Occupation, and Gender), towards identifying profitable customer segments. The FRM-enhanced version of the BI tool provides color-coded numeric rating of the different attributes, and of the data values in each attributes. These rates provide certain recommendations to the user on how to slice the data further at any given point of time (the rating method used in the experiment will be described in detail later). In this illustrative example – when a characteristic is given a high rate (in the example above, “Education”), this would mean a recommendation to slice the data along that characteristic, indicating that the underlying data values (e.g., "High", "Medium", "Low") differentiate well between customers with high profit potential, versus less-profitable customers. Similarly, a high data-value rate (here, the age group of “30-60”) would identify a customer's segment which is more likely to be profitable and worth further investigation. Notably, navigation decisions in this tool are left to end-users; however, they are now provided with visual cues on how to navigate more effectively. Another example of FRM integration is given in Figure 15. The user sees the data attributes available when using a pivot-table based tool. The data attributes are sorted according to their past value contribution in the type of task the user performs. Each attribute linked to a numeric value which states its value rate.
37 Pivot Table Editor Choose Task: - Marketing - Finance
Drag Dimensions to Here
Current Value: 315 Current Inequality: 0.61 Drag Dims to Here
Best Next Step: Drag Measures to Here
Drag & Drop Attributes to the Table
Data Attributes
Value
Order Amount
220
Order Date
200
Costumer Age
150
Ship City
20
Ship Country
15
Ship Address
10
Figure 15: Pivot table with FRM
Another example for embedding recommendations into BI tools can be seen in Figure 16, where different views available in the tool, are differentiated by color coding representing their utility. This figure demonstrates a possible implementation of collaborative filtering approach which recommends the user of certain data sets or attributes used by other users with similar usage patterns (upper left corner). This example shows a possible FRM extension for BI functionality which is not based on value driven usage metadata but may rely solely on frequency driven usage tracking. Business Intelligence – Sales Tool Associates who used this, also used:
Choose View
- Product Attribute
- Quantity per Customer
- Date Attribute
- Product per Customer
- Location Attribute
- Amount per Customer - Product per Customer
Sales Date Location Product Customer Amount Quantity Total Value: 470
Figure 16: BI tool with collaborative based FRM
38 Obviously, there are other possible forms for visualizing FRM besides color-coding (e.g., textual/graphical pop-up messages, “mouse-over” tool-tips, and side bars). Such FRM forms could indicate, in addition to the actual recommendations, the level of confidence and relevance of each recommendation based on the parameters that construct it. This research explores a novel approach for generating FRM, based on past usage of the underlying dataset and the associated value gains. Other possible forms of FRM are surveyed in the discussion.
4.
Method This study offers two novel contributions, the first being the collection of value-driven
usage metadata and the second being the utilization of this metadata for generating and embedding recommendations in BI tools. Accordingly, this section first describes the development of a prototype for a generic module that collects value-driven usage metadata in RDBMS environments and then describes a laboratory experiment that evaluates the impact of value-driven FRM on end-users' decisions. As value-driven metadata and FRM are novel concepts in the design of BI environments, this study can be seen as following the design-science paradigm. Design science seeks to extend the boundaries of human and organizational capabilities by creating new and innovative artifacts (Hevner et al., 2004). The outcome of design-science research is a viable artifact which aims to be relevant to a business problem. The new artifact must be demonstrated and justified through well-executed evaluation methods, and to contribute to the area of the design artifact. Design science research is similar to a search process in which the artifact is evaluated and being changed accordingly in an iterative manner. Design-science research contributions are achieved through exploring the designed artifact and evaluating it in real-world scenarios. As the designed artifact ought to be relevant to a business problem it must be presented effectively both to technology-oriented as well as management-oriented audiences (Hevner et al., 2004). Evaluation of the designed artifact can be done through observational or experimental methods, through studying the artifact in depth in business environment and/or through studying it in a controlled environment. In this study, the two novel artifacts (value driven metadata and FRM) are evaluated through implementing a prototype and conducting a laboratory experiment methods that were identified by Hevner et al. (2004) as favorable for assessing the impact and the contribution of new artifacts in a design-science research. A construction of a prototype artifact in
39 a design-science research is defined as a necessary first step in order to later deploy it as a full scale artifact. Prototype development enables to iteratively identify drawbacks in a new artifact and creatively develop solutions to address them. The use of prototype can be helpful also in assessing the progress toward the desired result, and to act as a proof of concept for the new design artifact (Hevner et al., 2004). Laboratory experiments have been used in a few former studies in the areas of BI and recommender systems (Herlocker et al., 2004; Shankaranarayanan et al., 2006; Jourdan et al., 2008). A laboratory experiment permits taking measurements at a higher degree of precision and better monitoring of cause-and-affect relationships. On the other hand, these relationships may not be generalizable for other times, settings and populations as a laboratory experiment has low degree of realism of context when compared to a field study (Scandura and Williams, 2000). The choice of evaluation methods in this study can be justified by the fact that both the value-driven metadata and the embedding of FRM in BI tools are novel concepts that have not yet been applied in real-world settings. Hence, it is reasonable to prototype these concepts first then evaluate them in a controlled environment as a proof of concept, as a first step towards implementing and evaluating them in real-world settings. 4.1.
Prototyping a Value-Driven Usage Metadata Module The purpose of developing a prototype of the value-driven usage metadata module was to
gain a better understanding of this new form of metadata, and to support the laboratory experiment that tested it later. The prototype helped defining and demonstrating some key factors in the design and the integration of such module - the inputs required for generating the metadata, the metadata repository database scheme, the metadata consolidation and aggregation processes, and the outputs that permit the integration of such metadata presentation tools (these are further describes in following paragraphs). The prototype development went through two main iterations the first being a preliminary version developed with MS-EXCEL that demonstrated some key module features and characteristics. Later on, the prototype was fully implemented in a more robust MS-SQL server 2005 environment, using C#-language code for computing the metadata contents and populating it into the database schema.
40
Query Analyzer
Integration
Module Output Interface
Value Database (Business Performance)
Module Input Interface
Data Resource (Decision Input)
Usage Metadata Repository
Metadata Module
Administrator Presentation Tools Consumer Presentation Tools
Figure 17: Metadata module architecture
The module architecture consists of four main components (Figure 17):
Monitoring tool and query analyzer – this component trace the queries directed towards the data resource and analyze which data sets were used by each. The monitoring uses SQL SERVER PROFILER (Microsoft, 2006), which has the ability to filter them along certain query characteristics, such as the query time, the applications that executed the queries, the database that the queries were directed to, the data users, time frames, etc.
Integration engine – this component links the value associated with a business task to the queries that supported that task. The value allocation method is further described in the following paragraph.
Metadata repository – this component stores the queries and their associated value, attributes to the data elements in the traced data resource. The metadata repository is autonomous and not attached to a specific DB which is being monitored. The scheme is made up of five tables, which support the metadata creation process
Application Programming Interface (API) – this component permits access to the usage metadata repository from different BI applications through function calls.
Appendix 1 provides further explanation on the module's architecture and database schema.
4.1.1.
Query Analysis and Value Allocation A key challenge that faces the implementation of value-driven usage metadata is the
"missing link" between the value generated as a result of a decision task and the queries that supported that task. Creating a robust link will require tagging each query and assigning these
41 tags to the decisions and the associated value – a tagging schema that does not exist in information systems today. As a reasonable proxy to this tagging schema – the prototype uses the user ID and the time frame to link between the query and the decision tasks. All of the queries that were executed within the timeframe of the business-task instance by the person who is defined as the business instance's owner are linked to the business-task instance. After the integration is completed, the value associated with the business instance is allocated between the associated queries. The prototype uses a simple allocation method – allocating the value to the last query involved. Other allocation methods are possible (e.g., allocating the value equally along all of the queries), and can be further investigated in future extensions to this study. The integration was implemented in SQL code executed by the Integration engine written in C#. The Integration engine procedure created the association between traced queries and business instances according to their date and owner. Then, the value of each business instance spread along the associated queries. After associating a business-task instance and the queries that supported it, the overall value per query is allocated among the specific columns and rows accessed. The query syntax is separated into atomic phrases, and each phrase is compared to the traced data resource column names (Appendix 1). The allocation of query value along the associated data is done in this research according to a method in which the query value is being spread equally along all of the columns that were used in the query syntax, without differentiating the columns of the that were used in the 'SELECT' clause and the ones used in the 'WHERE' clause. This method may be changed - for example, one might say that the value of the query is generated mainly thanks to the columns on which the query was executing the where clause, hence it is more reasonable to allocate more value to the columns of this group. The current research is based on an assumption that the syntax used in the queries is standard SQL and that the traced data resource is a standard relational DB, although the concepts presented here can be easily transformed in order to meet the different structure of other data schemes. For example, in order to support the value distribution over an OLAP cube or a relational DW, the "Select group" can be referred as the Measures in the cube, and the "Where group" can be referred as the Dimensions in the cube.
The analyzing procedure and value allocation across the data resource columns is demonstrated in the following pseudo code:
42
For each row in Integrate do Analyze query syntax For each source column in query syntax do If column is used in query's select clause Then Integrate.Source_Column_Select True End if If column is used in query's where clause Then Integrate.Source_Column_Where True End if End for In addition to the value allocation across the columns of the data resource, the query analyzing procedure spreads the value among all of the different categories of the columns, enabling the module to calculate the distribution of value across the data resource's different categories in each of the rows (e.g. Male and Female in column Gender), and the associated value inequality. The query analyzing procedure runs over each of the rows in the source DB, and aggregates the value of each data cell in the row by adding the values associated to this data row by all of the queries answer sets. The cell's content is examined and the value associated with it is recorded in table Value_Dist (Figure 18) The amount of combinations of cell content and its value is very big, hence the values is clustered into value ranges lowering the amount of possible combinations of category and value.
Figure 18: Example of table Value_Dist
The following steps describe the algorithm used by the value distribution creation procedure: a. define categories for each data column
43 b. define value ranges for each category c. Set the value for each row in the traced table according to the value of the queries that used it. d. For each row, divide its value to the row data cells e. For each data cell, determine its value according to
CellValue
Query _ Value where, Query Row Rowcount NumOfColumns
Query_Value – The value allocated to the query by the business instance they were executed for. Rowcount – The number of rows in the query's answer set NumOfColumns - The number of columns in the query's answer set
f. For each data cell, determine the category it relates to g. If the data cells' value and category fit an existing row in table Value_Dist raise the count attribute of the row by one h. If the data cells' value and category are a new combination not found in table Value_Dist create a new row that have the cells' combination and set its count attribute to one. Although the procedure may have high complexity and requires long calculations, it replaces the need for a lot of storage space to hold the metadata, and creates valuable information about the value distribution along the traced DB. Certain value indicators are derived from tables Integrate and Value_Dist and presented as the modules API (Appendix 1). The indicators are numerical measures, which give both frequency-driven as well as value-driven usage metadata. These indicators are the total value of a certain column in the traced DB, the inequality of the value along the categories of the column and the number of times each column or category were used. 4.2.
Experimental Lab Testing The laboratory experiment described next simulated an organizational setting in which
participants were asked to perform a marketing task that required analyzing data and taking business decisions accordingly. The main goal of the experiment was to assess the impact and the potential contribution of the novel artefacts' presented in this study - value-driven usage tracking, and the integration of value-driven FRM into BI tools. The design of this experiment was directed by the need to answer the following questions:
44 Does FRM integration improve decision outcomes? The research main hypothesis is that FRM integration will significantly improve decision-making outcomes. However, repetitive use is another important factor that may explain such improvement. As end-users repetitively use a decision-support tool, they get more familiar with the underlying data and the decision task – therefore, it is reasonable to assume that some improvement to the decision outcome will be observed, regardless FRM integration. However, the research hypothesis is that decision-outcome improvement with FRM integration will be above and beyond the improvement gained by repetitive use alone. Does FRM integration affect usage style? this research assume that FRM integration will significantly change the way people use a BI tool in terms of making data navigation more focused. Again, this research also assume that repetitive usage will play an important role in changing usage style, but that the changes detected with FRM integration will be significantly greater, above and beyond the changes caused by repetitive usage alone. Do end-users recognize the contribution of FRM integration? End-users‟ perception plays an important role in the success of IS artefacts. The TAM (technology acceptance model) and other theoretical models suggest that a sense of usefulness and ease of use will increase the likelihood of acceptance and adoption (Venkatesh et al., 2003). Hence, there is an importance in assessing the end-users‟ perception of FRM integration, assuming that its contribution to better usability and performance will be well recognized. The design of the survey instrument used in this research to measure end-users‟ perception was indeed influenced by previous TAM studies. With respect to all the questions above, would different forms of FRM lead to significantly different results? When discussing earlier frequency-driven versus value-driven collection of usage metadata, it was suggested that the latter form is likely to be superior to the former from the end-user's perspective. The experiment explores whether FRM generated by value-driven usage tracking indeed outperform the frequency-driven approach. Further, this research explore whether certain forms of value assessment and attribution will significantly outperform others.
4.2.1.
Experiment Settings The experiment simulated a marketing decision task – given a list of customers, choose
those customer segments that will be targeted in a promotion campaign that offers a certain product. To aid this task, users were provided with a BI tool, similar to the one shown in Figure
45 14, and a simulated dataset of past sale transactions that could be analyzed by the given BI tool. All sessions were conducted in labs with similar room conditions, each with identical personal computers, and were all scheduled to similar hours (See Appendix 2 for experimental system specifications). The experiment was conducted with 200 participants, all undergraduate engineering students. All participants were asked to sign a written agreement (Appendix 3) and received course credit for their participation. Additional cash prizes were offered to 5 participants picked by a raffle, in which the chance of winning was correlated to the participant objective performance. Each participant was asked to attend two one-hour sessions. In the beginning of each session, the participants were given a scripted description of the task and instructions on using the tool (Appendix 4). After this introduction, the participants were asked to perform the same decision task 6 times repetitively, given a maximum of 5 minutes per repetition, after which the performance measures per repetition (units sold, costs, and net-benefit) were recorded. The experimental system has also recorded a few measures per repetition that reflect users‟ interaction - the time spent per decision task, the number of customers and segments chosen, and the number of mouse-clicks made. Upon completing 6 repetitions, the participants were asked to fill out a survey (one per session), containing questions on a Likert scale of 1-7 (Appendix 5). All experiment sessions were conducted in labs with similar room conditions, each with identical personal computers, and were all scheduled to similar hours (See Appendix 2 for experimental system specifications). In the first session, all participants were provided with the identical BI-tool version with no FRM included (Figure 19). This session served a few purposes – first, to form a baseline for assessing the impact of FRM usage later. Second, to familiarize the participants with the task and with the BI tool, and third, to collect usage metadata – as soon as each task repetition was completed, the decision value (the net-benefit, as described later), and the segments selection were passed to a usage-tracking metadata module. Using these inputs, the module calculated how frequently each data item in the customer dataset was used and the value attribution among data item, following the attribution method described earlier. For the second session, the 200 participants were divided randomly into 5 groups, each provided with a different FRM variant:
46 1. No FRM (39 participants) – participants in this control group received the exact same version of the BI tool in both sessions. 2. Frequency-Driven FRM (42 participants) – the FRM in this version were based on the frequency of usage. The rating of a characteristic and of a data category within a characteristic reflected the number of times this characteristic and/or category was used for defining a customer segment along all of the campaigns created in the first session. 3. Value-Driven FRM (40 participants) – here, the FRM reflects the relative value contribution, based on the net-benefit achieved in each decision task (net-benefit calculation is further explained in the following section). The rate reflects the cumulative value attributed to each characteristic category, based on the value attribution method described earlier. 4. Expert FRM (40 participants) – this version is similar to the value-driven FRM, but here the value attribution was based only on the best 20% scores achieved (the 20% rate was chosen arbitrarily). 5. Subjective FRM (39 participants) – here, the users‟ perceived performance in the first session was used as a value proxy. Unlike the 3 other FRM forms that rely on the objective value gained by users' decisions, this FRM form relies on users‟ subjective assessments. An important difference in the scoring of this group has to be noted - while the objective value scores were collected one per decision, the subjective score was collected only once per session; hence, the same score was attributed to all the 6 tasks within a session – what may bias the results of this group to an extent.
4.2.2.
Experiment Task The experimental task was defined as - given a list of customers, choose customer
segments that will be targeted in a promotion campaign that offers a certain product. Due to certain promotion costs (e.g., printing charges, mailing fees, and call time), promoting the product to the entire list would be sub-optimal, as only some of the targeted customers are likely to make any purchase. An optimal decision would therefore be to target only those customers with a high likelihood to purchase enough units in a manner that will justify the promotion cost.
47 This decision can be formulated as maximizing an objective function:
(2)
V
m1..M
V M,{Im} P, Qm C F, C V-
I m PQm C V C F , where Net-benefit, the decision value The total number of customers (indexed [m]), and the set of binary decision variables, each indicating whether to include customer [m] in the campaign (=1) or not (=0), respectively Unit price, and the expected quantity of units that customer [m] will purchase (Qm≥0), respectively Fixed campaign cost, and variable promotion cost per customer, respectively
The optimal decision would be to include only customers for which expected revenue is greater than promotion cost (i.e., Im=1, if and only if PQm>Cv). However, while costs and unit price are typically known, the expected quantity of units is subject to uncertainty. Marketing professionals often estimate purchase intents by analyzing past sales, and identifying segments of customer who are more likely to accept a promotion offer (Van den Poel, 2003). Segments can be defined along certain customer characteristics and the associated values (e.g., “target male customers with no children who are between 30-45 years old”) (Figure 21). Accordingly, the task was defined more precisely as – given (a) a list of customers, each with a known set of characteristics values (e.g., age, gender, occupation, etc.), (b) data on past purchases, and (c) unitprice and cost parameters, choose the customer segments that would maximize the campaign‟s net-benefit assuming that past purchases reflect the likelihood of responding positively to the campaign.
4.2.3.
Experiment Tools To aid the experimental task, participants were provided with a BI tool, offered in two
base versions – with and without FRM. The version with no FRM (Figure 19) permitted the following BI functionality: (a) “Slicing”: upon selecting a customer characteristic, the bar graph at the bottom summarizes the number of customers under each category, and the total sales associated with those customers. The tool presented 12 categorical customer characteristics, each with 3 possible values (e.g., High/Medium/Low Income). To prevent biased pre-assumptions on the predictive power of each characteristic, the characteristic names were coded with capital letters (A, B, C, …, L) and the associated value with enumerated lower-case letters - e.g., instead
48 of High/Medium/Low Income, users would see a characteristic named A with associated values {a1, a2, a3}, (b)“Filtering”: when slicing along a certain characteristic, a user would have the choice to limit the presentation to show only certain values (e.g., under characteristic F, show only values f1 and f3), and (c) “Drilling”: after slicing along a certain characteristic, a user would have the choice to slice the data further along others. The numbers would then be summarized and presented along the value-combinations of all the characteristic included – for example, if H(h1, h2) and G(g1, g3) are chosen, the tool presents summaries for the combinations {h1, g1}, {h1, g3}, {h2, g1}, and {h2, g3}.
Figure 19: BI tool without FRM
The FRM-enhanced version included all the above BI functionality, implemented in a similar manner. However (as illustrated in Figure 20), a certain color-coded rate was added per characteristic and per characteristic category, indicating a certain usage recommendation, which reflects potential contribution to a better decision. A high characteristic-value rate (r) would indicate a favorable customer segment - e.g., if under D, r(d1) >> r(d2), the recommendation would be to prefer customers who belong to category d1. A high characteristic rate (R) would indicate a high variance among the rates among characteristic values; hence, a higher likelihood to single out a better customer segment that possessed a certain characteristic-value. If, for example, R(J) >> R(K) – this would mean a high variability between the rates of {j1, j2, j3} versus low variability between the rates of {k1, k2, k3}; hence, a recommendation to prefer slicing the data along J rather than along K. Importantly, the FRM-enhanced tool recalculates the rates dynamically, depending on the characteristic-value combination observed, meaning that at
49 each point of time the user would get a recommendation how to proceed which depends on the “Slicing”, “Filtering” and “Drilling” choices made up to that point.
Figure 20: BI tool with FRM
The simulated database included a list of 1000 customers, each with 12 associated characteristic values (e.g., {a1, b3, …, k2, l1}, which were randomly drawn using given value distributions (e.g., a1:0.3, a2:0.5, a3:0.2). Using a set of rules, which associated certain value combinations with certain levels of purchase intent, each customer [m] was assigned with a set of likelihood numbers {Pm,z} of purchasing z (an integer number between 0 and 5) units in a given campaign, such that ΣzPm,z,=1, and Qm=ΣzZPm,z, is the expected quantity of units purchased. A random engine then generated a list of sale transactions per customer, by simulating a sequence of campaigns and randomly drawing the quantity of items purchased per campaign (including only transactions with quantity greater than 0). This random draw used the set of purchaselikelihood numbers per customer {Pm,z}, with some level of randomness added.
50
Figure 21: Decision screen
Users were also provided with a tool for selecting one or more customer segments to be targeted (Figure 21). Upon completing the selection – the campaign performance would be evaluated. The participant would then get a campaign report, including the number of customers targeted, units sold, costs, and the net benefit (Figure 22).
Figure 22: Campaign report
At the end of each session, the participant filled a questionnaire regarding his familiarity with BI tools, experience in marketing, sense of usefulness, ease-of-use and acceptance of the
51 tool and the recommendations (Appendix 5). In addition the participant filled questions regarding his/her perceived performance. A preliminary pilot study was undertaken with a group of 8 graduate students in order to find unclear questions. The questionnaire was analyzed using cronbach's-alpha methodology and was proven to be internally consistent, as described in the following section.
5.
Experiment Results This section reviews the experimental results and their analysis, starting with a description
of data collection and preparation procedures. The analysis follows the sequence of experiment questions that were stated earlier (in section 4.2), first examining the effect of FRM integration on decision outcomes, then its effect on usage style, and finally the perceived contribution, where at each stage the difference between the various forms of FRM is examined. The analysis was performed with STATISTICA 7.0 and SPSS 13.0. This section summarizes and discusses the relevant analysis results, while the full statistics tables are enclosed in Appendix 6. 5.1.
Data Collection and Preparation The experiment was conducted with 200 participants, all undergraduate engineering
students around the same age. Most participants were majoring in IS engineering, in their 3 rd year of study or later. Some participants have indicated previous exposure to marketing tasks and/or BI applications; hence, these effects were controlled for as well. All participants completed the assignments successfully and no data collection issues could be detected. The participant groups and the associated demographics are summarized in Table 4.
FRM Group
Participants
Age µ/σ
(M/F)
3rd
IS
Israeli
Marketing
BI
Year
Major
Born
Experience
Experience
1. None
39 (20/19)
26.3/1.8
87%
95%
95%
21%
15%
2. Frequency
42 (21/21)
26.2/1.2
86%
93%
88%
17%
17%
3. Value
40 (15/25)
25.8/1.6
100%
95%
88%
10%
20%
4. Expert
40 (18/22)
25.9/1.8
95%
90%
93%
10%
10%
5. Subjective
39 (18/21)
26.1/1.7
93%
90%
92%
10%
10%
Overall
200 (92/108)
26.0/1.7
92%
93%
91%
14%
86%
Table 4: Participant demographics along the different experiment groups
52
The following variables were measured per session and per task (6 per session): Decision outcome: the Decision Value (net-benefit) was measured 12 times, one per decision task. The Decision Value per session was calculated as the average of the 6 scores within that session. Usage style: the Time (in seconds) spent, and the number of Clicks (a click is defined as a use of the computer mouse to navigate through the BI tool) were measured 12 times, one per decision task. The Time and Clicks per session were calculated as the corresponding averages of the 6 scores within that session. Perceived contribution: using the survey instrument, four constructs that reflect user perception were calculated, one per session - Performance, Usefulness, Ease-of-Use, Acceptance – each operationalized with a few questions. The questions were presented in a mixed order, some asked in both sessions while others and a few FRM-related questions were included only in the second session (not for the control group). Table 5 lists an English translation of the survey-instrument questions (See Appendix 5 for the full questionnaire in Hebrew), and Cronbach‟s-Alpha scores per session (See Appendix 6 - 8.6.6 for detailed reliability analysis).
Acceptance
Ease-of-Use
Usefulness
Performance
53
Both sessions (S1, S2)
α (S1, S2)
I have been successful in my assignment The assignment was well understood, and I knew how to perform it well I wasn‟t clear about what I was doing I found the assignment to be too difficult Using the BI tool I was provided with have helped performing the task If I was a marketing person, a BI tool as such would improve my work If I was a marketing person, a BI tool as such would have made my work easier If I was a marketing person, a BI tool as such would have not helped me I easily learned how to operate the BI tool I was clear about the information presented by the BI tool The BI tool used in the experiment was too complex to use The BI tool I used could not support the actions I needed to perform I would have recommend a company to invest in a BI tool such as the one presented If I was a marketing person in a firm, I would have requested to use such a tool If I was a manager in a firm, I would have recommended to invest in such a tool
0.683, 0.768
0.803, 0.820
FRM-related (S2 α (S2) Only) The FRM provided 0.830 helped improving my performance The FRM provided were meaningless with respect to performing the task The FRM provided 0.760 can help using the tool I didn‟t use the FRM provided by the tool
0.823, 0.851
The FRM were 0.777 presented clearly and in an understandable manner I couldn‟t understand how to use the FRM
0.904, 0.937
I would recommend 0.770 a firm to invest in FRM A company should not invest in FRM such as those provided
Table 5: Survey questions
The summary statistics of these variables are summarized in Table 6. A Chi-Square test for normality (Appendix 6 - 8.6.1) shows that all of the variables are normally distributed with high significance (P-value < 0.05).
54 Variable
S1: µ
S1: σ
S2: µ
S2: σ
Decision
Decision
Outcome
Value
30.90
498.00
574.52
593.11
Usage Style
Time
163.74
90.02
156.24
94.85
Clicks
82.37
39.76
46.75
24.89
S2/F: µ
S2/F: σ
Perceived
Performance
4.88
1.12
5.19
1.14
5.10
1.78
Contribution
Usefulness
4.82
1.20
4.81
1.18
5.24
1.50
Ease of Use
4.79
1.28
4.80
1.24
5.23
1.40
Acceptance
4.37
1.35
4.32
1.34
5.38
1.43
S1/2: Session 1 or 2, /F – FRM-related measurements; µ: Avg. , σ: STDEV Table 6: Summary statistics for all participants
The following table summarizes the effect of gender, natives, and former marketing/BI experience (No experience compared to some experience) on all of the variables measured. As all P-values are relatively high (nearly all are above 0.05), the control variable don't seem to have any significant effect on the measured variables.
Gender
Israeli Born
Marketing Experience
BI Experience
F(1,198)=1.1036, p=.29476 F(1,198)=2.5712, p=.11042
F(1,198)=.82869, p=.36376 F(1,198)=.07554, p=.78372
F(1,198)=.82869, p=.36376 F(1,198)=.00012, p=.99112
F(1,198)=2.7334, p=.09985 F(1,198)=.63479, p=.42656
Clicks
F(1,198)=3.8434, p=.05134
F(1, 198)=.37415, p=.54145
F(1, 198)=.00990, p=.92085
F(1, 198)=.03126, p=.85985
Perceived Performance
F(1, 198)=1.2342, p=.26794
F(1, 198)=.00774, p=.93000
F(1, 198)=.25699, p=.61276
F(1, 198)=.29870, p=.58531
Usefulness
F(1, 198)=2.7466, p=.09905 F(1, 198)=.01977, p=.88832 F(1,198)=3.9064, p=.04949
F(1, 198)=.17923, p=.67250 F(1,198)=3.0077, p=.08442 F(1, 198)=.09952, p=.75274
F(1, 198)=2.6904, p=.10254 F(1, 198)=1.4706, p=.22669 F(1, 198)=.43383, p=.51088
F(1, 198)=1.5487, p=.21480 F(1, 198)=1.3793, p=.24163 F(1, 198)=.33864, p=.56128
Decision Value Time
Ease of Use Acceptance
Table 7: Control variables effect
55 Normality tests were done for all of the measures collected. All of the measures were found to be normally distributed according to Chi-Square test (P-Value 0 Gender = „Female‟ and Children < 3 Gender = „Female‟ and Status = „Married‟ Income = „High‟
Attributes Used Gender, Children Gender, Children Gender, Status Income
Records Retrieved [3], [5] [2], [4], [6] [2], [6] [1], [6]
Illustrative Example (part 1) - Assessment of Frequency-Driven Usage Metadata
105 To illustrate the creation of frequency-driven metadata, we use a simplified example above (extended later). The table in this example is used by marketing associates to decide which customers will be approached when promoting a new product. An associate would use a BI tool to investigate previous sale transactions, and the tool will generate queries directed to the customers tables, such as those demonstrated, to specify the subset of customers that will be targeted. Each query can be analyzed to detect which records and attributes are used to specify the selection (e.g., by parsing the WHERE clauses in the SQL statement) and, accordingly, the frequency of usage can be calculated. Frequency-driven metadata collection may provide important inputs to the data administrator, toward improving system design and prioritizing administration efforts. It is common in databases, as highlighted by the example above, that some records and attributes are accessed more frequently than others. In a larger real-world databases, this differentiation may lead to a decision to grant the more frequently used records and attributes a higher priority in terms of data quality maintenance – i.e., watch these database objects more closely, detect and correct defects, and make sure to keep them up-to-date. While seeing the merits of collecting metadata on usage frequency for data administration, we question does it truly address the needs of data consumers? One could argue that, to an extent, frequent usage reflects higher significance of certain data components to data consumers; hence, higher valuecontribution potential. On the other hand, we suggest that frequent usage may reflect certain stagnation and a tendency to "dig into the same well" - re-using certain data subsets repetitively, while possibly ignoring unused subsets with high contribution potential. Therefore, a potential risk with basing data management decisions solely on frequency-driven metadata (e.g., due to a removal of data that is lessfrequently used from the active repository into an archive) is a possible loss of opportunity to benefit from records and attributes that data consumers have neglected to use so far, which may permit new forms of data usage. There is possibly no "clear cut" answer to this question, as it largely depends on the business context and the usage tasks. However, we suggest that important insights can be gained from tracking and considering not only actual data usage, but also the associated value gains. The benefits gained from the use of information products have been conceptualized as utility [Ahituv, 1980]. Utility assessments have been used to optimize the configuration of data processes and resources [Ballou et al., 1998; Even et al., 2007] – tasks typically associated with data administration. We suggest that, beyond the benefit offered to data administration, collecting quantitative assessment of the business-value gained as a form of metadata can improve data consumption as well. Business value can be measured, for example, in terms of decision outcomes (e.g., production increase, customers‟ purchase intent), revenues and profitability. Organizations capture such value measurements today, but rarely link them to the data resources and the decisionsupport tools that were used in the process of value generation. Value-driven collection of usage metadata (Figure 2) aims at establishing such a link. To demonstrate this approach, we have successfully implemented a working prototype of a module that captures and stores value-driven usage tracking as a metadata layer. In its base, the module applies a similar approach to the one described earlier for collecting frequency-driven metadata - capturing the queries directed at a data resource, and parsing them into specific components. However, the module also collects different types of value measures (e.g., throughput, performance, and income), which are associated with a specific decision task. In certain cases, value assessments can be based on the same data resource (sale transactions, for example, can often be linked to a specific marketing campaigns that were based on a certain analysis of previous sales). In other cases – such assessments may use other information resources such as CRM and accounting systems. The module associates the value with certain decision tasks and then, through a mechanism of inference (e.g., by comparing the user-name and the time stamp), to the queries that have supported each tasks. Establishing this link between decision tasks and the underlying queries permits the creation of integrated metadata that associate business value with specific data components. An API (Application Programming Interface) can provide this form of usage metadata upon demand through function calls. Such metadata can be integrated into front-end tools, enhance the presentation, and communicate important information on the frequency of usage and on the associated value to both data consumers and administrators.
106 Once the link between decision tasks and queries is established, different methods can be considered for attributing value to specific data objects. For illustration, we describe here a relatively simple method, which assumes that value is attributed to the last in a sequence of queries that support a decision task. We assume that to support a certain decision, users query repetitively a certain tabular dataset with N records indexed by [n] and M attributes indexed by [m]. We consider Q queries indexed by [q], each associated with a business value Vq. The binary indicator Rqn indicates whether record [n] was retrieved by query [q] (Rqn =1), or not (Rqn =0). Similarly, Rqm indicates whether attribute [m] participated in query [q] or not. The value of a certain query (Vq) is attributed between the participating data items, using a certain valueattribution function Vqn,m=u(Vq, Rqn, Rqm), such that Vq = ΣnΣm Vqn,m. For simplification, we use here an equal attribution of value among all participating data items. Accordingly, the overall value of a certain data items Vn.m is given by: (1) Vn,m q1..QVnq,m q1..Q u V q , Rnq , Rmq q1..QV q R q R q , where n1.. N m1..M n m
Q, q M, N Vq, Vqn,m, u Rqn, Rqm -
The number of queries performed, and the corresponding index, respectively The number of attributes (indexed [m]) and records (indexed [n]), respectively Query [q] value, its attribution to data item [n,m], and the attribution function used Binary indicators of the participation (=1) of record [n] and attribute [m] in query [q]
To demonstrate the value allocation described above, we extend the previous example. We assume that each query has led to a certain promotion campaign in which a group of customers has been approached. Some customers may have responded to the campaign by making certain purchases, and the overall value attributed to a query is the total purchase amount. As illustrated, this value proxy may significantly vary among queries. We now use the allocation (Eq. 1) to assess the relative value of each data object. As illustrated by color-coding – some records, and attributes may turn out to have significantly higher value than others, and the value attribution “map” may look significantly different than the one when basing the attribution of usage frequency.
Customers # 1 2 3 4 5 6 7 … Value
Customer Abraham Sarah Isaac Rebecca Jacob Lea Rachel
Gender Male Female Male Female Male Female Female
Income High Low Medium Low Medium High Low
Children 0 1 2 0 3 2 4
Status Single Married Married Single Married Married Single
515
2000
60
500
…
Value 1000 510 50 10 50 1510 0
Value 0 0 Gender = „Female‟ and Children < 3 Gender = „Female‟ and Status = „Married‟ Income = „High‟
Attributes Used Gender, Children Gender, Children Gender, Status Income
Records/Value [3], [5] [2], [4], [6] [2], [6] [1], [6]
Total Value 100 30 1000 2000
Illustrative Example (part 2) - Assessment of Value-Driven Usage Metadata We can potentially gain important insights by analyzing the value distribution, along with the assessment of frequency of use. For example, the Income attribute, which was not frequently used, is associated with the highest value, while the Children attribute, which was more frequently used, is associated with lower value. Insights as such can be transformed into valuable recommendations for a marketing associate the next time s/he plans to run a similar campaign. The decision value of each of the participants is being saved in a metadata repository. Using the API, a BI tool, which was designed to access to the value-driven usage metadata can now demonstrate value differentials and distribution, toward improving the decisions made, as demonstrated in the experimental design described next.
107
Experiment Design This section describes the design of a lab experiment, currently under final preparation stages, that tests the integration of FRM based on previous usage and the associated value.
Research Model and Working Hypotheses The first research stage will be directed by the theoretical model shown in Figure 3. Some model variables will be measured with the test experiment tool described in the following section, while others will be assessed using a previously-tested questionnaire. Motivation Experience Learning curve Familiarity with similar tools
H4
Expertise H6 H1
H3
Controls Age Gender Language fluency
H5
H7 H2
Performance Decision outcome Time to complete Usefulness Ease of use
Acceptance
FRM Inclusion
Figure 3
Research Model
Dependent Variable – Performance: The dependent variable, reflecting the ability of a user to effectively perform a task with tool support, will take one of two forms: Objective – actual decision outcome and time it takes to complete the decision task Subjective – perceived usefulness and ease of use Previously tested models, such as TAM (technology acceptance model [Davis, 1989]), have suggested that a higher sense of usefulness and ease of use increase the likelihood of user acceptance. While this study intends to focus mainly on performance, the experiment described below will permit assessing acceptance and validating the anticipated link between performance and acceptance. Independent Variables: The independent variables will be Experience and FRM inclusion: a) Experience can be measured in terms of: Learning curve – the time a user spends using and mastering the tool Familiarity – the extent to which the user has previously used similar tools in the past It is reasonable to assume that an experienced user (in terms of learning curve and/or familiarity) will perform better that a non-experienced user; hence,
H1: Usage Experience positively affects Performance
b) Our key assumption is that the inclusion of an FRM will offer a major improvement in the usability of BI tools and therefore in user performance; hence,
H2: FRM inclusion positively affects Performance
As discussed in the previous section, we suggest that value-driven collection of usage metadata is superior to frequency-driven collection; hence:
H2a: The Performance effect of FRM that are based on value-driven metadata will be superior to the effect of FRM based on frequency-driven usage metadata alone
It is reasonable to assume a possible synergistic effect between the two independent variables, i.e., that the overall effect of Experience and FRM inclusion is higher than the effect of each alone. Hence,
H3: The interaction effect between Experience and FRM inclusion is positive
108 Moderating Variables: It is reasonable to assume that certain user characteristics will moderate the effect of Experience and FRM inclusion on Performance. The moderating variables that will be tested are Motivation, the user‟s motivation to perform well, and Expertise, the extent to which the user is knowledgeable in the particular task domain. Studies (e.g., Siegel and Watts-Sussman [2003]) have shown Motivation (or involvement) and Expertise to have moderation effects on the usefulness of information resources and hence on their acceptance and adoption. Hence,
H4: The greater the user’s Motivation, the more Experience affects Performance
H5: The greater the user’s Motivation, the more FRM inclusion affects Performance
H6: The greater the user’s Expertise, the more Experience affects Performance
H7: The greater the user’s Expertise, the more FRM inclusion affects Performance
Control Variables: The experiment will control for a few additional variables - age, gender, language fluency, and possibly others.
Experiment Procedures and Tool The model and the derived hypotheses will be tested in a laboratory setting. In the planned experiment, all participants will be asked to perform a certain decision task repetitively, aided by a BI tool. The decision outcomes, as well as the actual usage of the tool and the data resources will be tracked and measured. This will enable data collection that will allow measuring some of the variables (as described later in Table 1). In addition to tracking decision outcome and actual usage, users will be asked to complete a questionnaire, which will enable data collection on remaining variables. Due to space limitations, we do not describe here in details all the experiment preparation procedures, but rather explain the principles that guide its design. The decision task: the participants will act as marketing associates on behalf of a firm that offers a certain product or service to its customers (e.g., a vacation package). To decision will be aided by a large database that includes two main tables: A list of customers, in which each associated with a given set of attributes (e.g., Income, Gender, Marital Status, and Number of Children). Based on the mix of attribute value – each customer [t] will be associated with a set of likelihood numbers Pt, z (z = 0, 1, 2,…) of purchasing z units within a given time period, such that ΣzPt, z=1, and Qt = ΣzZPt, z is the mean number of purchases. Purchases transactions, based on the purchase likelihoods defined per customer. A random generator will produce a large number of purchase transactions for a broad period of time. Given access to this database, the participants will be asked to choose a customer segment that will be targeted. Approaching a customer and offering him/her a promotion has a given cost (e.g., the mail delivery fee, or the time needed for a phone call); hence, the larger is number of customers approached – the higher is the cost. Each customer is associated with certain likelihood to purchase a certain quantity of the service and, accordingly, the overall decision value is defined as: (2) V t 1..T I t (SQt C) , where
VTS, CQt It -
The overall decision The number of customers (indexed [t]) The revenue per service item sold, and the promotion cost per customer, respectively The expected number of item that customer [t] will purchase (Qt≥0) A decision whether to include customer [t] in the promotion campaign (=1) or not (=0) A decision to include a customer may increase revenue, but at a cost. Obviously, a decision maker would prefer to include only customers for which the expected revenue is higher than the cost (i.e., only [t] for which SQt > C). The expected quantity Qt is defined in advance, but not exposed to the decision maker explicitly. The user will be asked to infer which customers are likely to purchase the service by observing customers‟ past purchases. Moreover, the decision maker will not select specific customers, but
109 rather will be asked to define customer segments. The segment definition will be based on a given set of customer attributes and a selection of a certain criterion per attribute (e.g., “All high-income married male customers, with 2 children or more”), where the user may choose to avoid defining a selection criteria in certain attributes. To aid the decision, each participant will be provided with a BI tool, such as the one illustrated by Figure 1. The BI tool will permit exploring past transactions, analyze purchase activity, and determine the revenue associated so far with each customer segment. The tool presents the distribution of certain measures (Revenues associated with past transactions) along certain customer attribute. The visual display imitates a decision tree. Starting at the high-level node, which reflects the entire population of customers, the user may choose an attribute (e.g., Children) along which he wishes to segment the data. For a given attribute value (e.g., Children = 2), the user may choose to segment the data along another attribute (e.g., Status), and so on. Based on the different customer segmentations that are explored by using the BI tool – the user finally selects the customer segments that will be targeted. Once the selection is made – the overall value of the selection is calculated (Eq. 2), the value is attributed among the different attributes and records (Eq. 1), and the attribution is saved in the value-driven usage metadata module. As the experiment participants keep performing the decision tasks repetitively – the value-driven usage metadata is accumulated and enhanced. The left-hand side of Figure 1 illustrates the BI tool in its basic form, which does not include FRM. The enhance form, illustrated in the right-hand side of Figure 1, includes certain FRM enhancements – indication of the total value and the value distribution (a variance measurement) associated with the different attributes, at each node. The recommendations change dynamically, depending on the node that the user selects. The FRM enhancements are based on the usage metadata that was accumulated while participants keep performing the decision task repetitively. To help testing the hypotheses, the group of participants will be divided into a few sub-groups, and some variability will be created in the tasks that each sub-group is asked to perform (Table 1). Hypothesis H1: Experience affects Performance H2: FRM affects Performance H2A: Value-Driven FRM are superior to FrequencyDriven FRM H3: Experience-FRM interaction H4, H5: Motivation moderation H6, H7: Expertise moderation Control
Table 1.
How the hypotheses will be tested Participants will be asked to perform the same task repetitively Some participants will provided with the same BI tool in all sessions Participants will be asked about past experience with similar tools Initially, participants will perform the task using BI tools with no FRM Later, some participants will be offered BI tools with FRM Some participants will perform the task with no BI support at all Some participants will be provided with FRM enhancement based on value and frequency assessments, while others will be provided with FRM enhancement based on frequency alone Experience/FRM – same as the above Certain statistical regression methods permit testing interaction Experience/FRM – same as the above Some participants will be offered performance-based compensation Experience/FRM – same as the above Participants will be recruited from different expertise populations The test will validate assumption that the control variables have no substantial effect in the given setting
Hypotheses Testing
Conclusions Our research investigates the integration of feedback and recommendation mechanisms (FRM) into BI tools. The working hypothesis that guides our study is that the integration of FRM into BI tools will improve their usability and increase the benefits that end-users and organizations can gain from data resources. We have described an experiment, currently under preparation, for testing the usability and the benefits of such integration in terms of improving decision-making processes. We see this experiment as a
110 first “proof of concept” step of testing the FRM-integration idea, toward gaining insights on its usability and benefits. The controlled lab environment, in which we intend to apply the test, will permit a more precise data collection on usage patterns and value generation – what is often hard to achieve in real-world environments. Nevertheless, testing FRM integration in real-world environments would be an important follow-up step in furthering this line of research. Another key contribution of our study, which links to the previous, is the introduction of a novel approach for usage tracking in data environments. This approach suggests that integrating quantitative assessments of usage-frequency together with the associated value gained may offer substantial benefits to data administration and consumption. Joint frequency and value assessments can help identifying unused data subsets with high value-contribution potential, may highlight flaws with repetitive use of data and, consequently, motivate new usage forms. Further, value assessment can direct design decisions, and help prioritizing data maintenance efforts. Relying on usage frequency alone might promote usage stagnation and loss of opportunity to gain new forms of benefits. Complementing frequency assessments with value assessments may help “closing the loop”, in terms of providing feedback based on usage performance, and reducing the potential risks. First, value allocation gives higher weight to past usages with high contributions. Second, it can reflect variability in the importance of different subsets depending on the usage context. Lastly, it can help detecting data subsets with high contribution potential that have not been frequently used. Obviously, future extensions to our study will need to address some key limitations of this approach: (a) Quantifying value – organization maintain performance measurements (e.g., productivity, income, and profitability) that can be possibly linked to decision tasks. However, decision performance may depend on other resources such as human knowledge and financial assets. Further, the value depends on the usage context, and value assessment for a certain type of usage tasks does not necessarily apply to others. Further, value is time-dependent, as data that can be used effectively at a certain point of time, might become obsolete later. We hasten to say that the value-allocation methodology, which we apply in this study, appears to be a better fit to operational environments in which decision tasks have a high degree of repetition, and causal relations between data usage and business performance are easier to establish. Promotion-campaign management, such as in our illustrative example would be a good representative for this type of decision-making. Financial-investment decisions would be another example for data-driven decisions, in which outcomes are measurable (e.g., the chabge in the value of the financial asset) and linkable to the data resources being used. Conversely, quantifying the value of decision outcomes might turn out to be more challenging in strategic decision scenarios, which are not repetitive in nature and often relay on information sources other that organizational data repositories. (b) Linking value to specific queries - performance assessments are rarely linked explicitly to the data resources and tools used. Our preliminary prototype includes inference mechanisms for creating implicit links – e.g., based on the user name, and/or time proximity. Obviously, implicit links cannot be absolutely precise and might bias the value allocation significantly. Establishing explicit links will require stronger metadata integration between systems and, likely, redesign of data environments (e.g., joint codes that link each decision task and queries). One could question whether or not making such a high investment in redesigning data environments and BI tools would justify the benefits gained. (c) Attributing value to specific data objects – the attribution system has critical impact on the results. Our prototype attributes value only to the last query in the sequence that generated the decision, and distributes the value equally between all the data that were retrieved. A different allocation method may consider, for example, spreading the usage value along all queries and/or consider possible interactions among attributes – hence, unequal allocation. Finally, we would suggest that future extensions of this study should further explore links to the research of recommender systems. Recommender systems are common in web-based user interfaces (e.g., rating systems in E-commerce sites), but less so in BI tools. Adomavicius and Tuzhilin [2005] identify the need to incorporate contextual information into the recommendation process. As value assessments depend on the context in which data is used [Even and Shankaranarayanan, 2008], we would suggest that the valuedriven metadata approach may help such incorporation.
111
Acknowledgments This project was supported by a grant from Microsoft R&D, Israel. We wish to thank Prof. Nava Pliskin from the Department of Industrial Engineering and Management at Ben-Gurion University of the Negev for her advice and support.
תקציר ארגונים רבים מאמצים בימינו כלי בינה עסקית על מנת לתמוך באנליזות עסקיות ,קבלת החלטות ומדידת ביצועים. למרות זאת ,השימוש בכלי הבינה העסקית אינו פשוט ויכול להשתפר באם ינתנו למשתמשים הוראות המבוססות על שימוש קודם .מנגנון של פידבק והמלצות (מפ"ה) יכול להיות מוטמע בכלי בינה עסקית וכך לספק הנחיות ורמזים ויזואליים על אופן השימוש בכלי .מחקר זה מציע ובוחן מפ"ה המבוסס על שימוש עבר שנעשה בכלי הבינה העסקית. נוסף על כך ,מחקר זה מציג את המטה-דטה מבוססת הערך – מתודולוגיה חדשנית למעקב והפקת מדדים כמותיים לערך המופק משימוש במקורות נתונים .התועלת וההשפעה של שילוב סוג זה של מפ"ה נבחן במסגרת ניסוי מקיף בן 200 משתתפים .הממצאים תומכים בכך ששילוב מפ"ה יכול לשפר את השימושיות של כלי הבינה העסקית ולהגביר את התועלות המופקות ממקורות הנתונים .בכך מודגשים התועלות הפוטנציאליות הנובעות מאיסוף מטה-דטה מבוססת ערך ושימוש בה על מנת להפיק המלצות.
מילות מפתח :בינה עסקית ,מחסן נתונים ,מטהדטה ,ערך נתונים ,מערכות המלצה ,כלים תומכי החלטה
אוניברסיטת בן-גוריון בנגב הפקולטה למדעי ההנדסה המחלקה להנדסת תעשייה וניהול
שילוב מנגנוני המלצה מבוססי ערך בכלי בינה עסקית חיבור זה מהווה חלק מהדרישות לקבלת תואר מגיסטר בהנדסה
מאת :יואב קולודנר מנחה :ד"ר אדיר אבן
מחבר :יואב קולודנר
.....................
תאריך....................
מנחה :ד"ר אדיר אבן
.....................
תאריך....................
יו"ר ועדת תואר שני מחלקתית :פרופ' יוסף קרמר .....................
נובמבר 2009
תאריך....................
אוניברסיטת בן-גוריון בנגב הפקולטה למדעי ההנדסה המחלקה להנדסת תעשייה וניהול
שילוב מנגנוני המלצה מבוססי ערך בכלי בינה עסקית
חיבור זה מהווה חלק מהדרישות לקבלת תואר מגיסטר בהנדסה
מאת :יואב קולודנר
כסלו תש"ע
נובמבר 2009
![Obstacles to Enhancing Professional Development with Digital Tools ... [PDF]](https://m.moam.info/img/260x300/obstacles-to-enhancing-professional-development-wi_6493b0ad098a9e661b8b4570.jpg)
