Organizational Data Mining (ODM): An Introduction - CiteSeerX

Organizational Data Mining: An Introduction 1

Chapter I

Organizational Data Mining (ODM): An Introduction Hamid R. Nemati University of North Carolina at Greensboro, USA Christopher D. Barko University of North Carolina at Greensboro, USA

ABSTRACT An increasing number of organizations are struggling to overcome “information paralysis” — there is so much data available that it is difficult to understand what is and is not relevant. In addition, managerial intuition and instinct are more prevalent than hard facts in driving organizational decisions. Organizational Data Mining (ODM) is defined as leveraging data mining tools and technologies to enhance the decision-making process by transforming data into valuable and actionable knowledge to gain a competitive advantage (Nemati & Barko, 2001). The fundamentals of ODM can be categorized into three fields: Artificial Intelligence (AI), Information Technology (IT), and Organizational Theory (OT), with OT being the core differentiator between ODM and data mining. We take a brief look at the current status of ODM research and how a sample of organizations is benefiting. Next we examine the evolution of ODM and conclude our chapter by contemplating its challenging yet opportunistic future. Copyright © 2004, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.

2 Nemati & Barko

SETTING THE STAGE The competitiveness of the new global economy requires immediate decision capability. A recent study of more than 800 U.S. business decision-makers found that most respondents are making more decisions in the same amount of time but are missing opportunities because their decisions are not made quickly enough. In addition, these decision makers are not fully utilizing available resources and are often unable to gather sufficient information to make a fact-based decision (Wessel, 2002). The amount of data available today is doubling every five years, and corporate America is able to utilize less than 7 percent of the information it manages (Anonymous, 2001). Research from IBM also revealed that organizations use less than 1 percent of their data for analysis (Brown, 2002). As noted in the preface, this is the fundamental irony of the Information Age we live in — organizations possess enormous business information, yet have so little real business knowledge. In the past, companies have struggled to make decisions because of the lack of data. But in the current environment, more and more organizations are struggling to overcome “information paralysis” — there is so much data available that it is difficult to determine what is relevant. Organizations today routinely collect and manage terabytes of data in their databases, thereby making information paralysis a key challenge in enterprise decision-making. Once the essential data elements are identified, the data must be reformatted, processed and analyzed to generate knowledge. The resulting knowledge is then delivered to the decision makers for collaboration, review and action. Once decided upon, the final decision must be communicated to the appropriate parties in a rapid, efficient and cost-effective manner.

ORGANIZATIONAL DATA MINING The manner in which organizations execute this intricate decision-making process is critical to their well-being and industry competitiveness. Those organizations making swift, fact-based decisions by optimally leveraging their data resources will outperform those organizations that do not. A robust technology that facilitates this process of optimal decision-making is ODM, which is defined as leveraging data mining tools and technologies to enhance the decision-making process by transforming data into valuable and actionable knowledge to gain a competitive advantage (Nemati & Barko, 2001). ODM eliminates the guesswork that permeates so much of corporate decision-making. By adopting ODM, an organization’s managers and employees are able to act sooner rather than later, be proactive rather than reactive and know rather than guess. ODM spans a wide array of technologies, including, but not limited to, e-business intelligence, data analysis, OLAP, CRM, e-CRM, EIS, digital dashboards, information portals, etc. ODM enables organizations to answer questions about the past (what has happened), the present (what is happening) and the future (what might happen). Armed with this capability, organizations can generate valuable knowledge from their data, which in turn enhances enterprise decisions. This decision-enhancing technology enables many advantages in operations (faster product development, increased market share with quicker time to market, optimal supply chain management), marketing (higher profitability and increased customer loyalty through more effective marketing campaigns Copyright © 2004, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.

Organizational Data Mining: An Introduction 3

and customer profitability analyses), finance (improved performance through financial analytics and economic evaluation of business units and products) and strategy implementation (business performance management (BPM), the Balanced Scorecard and related strategy alignment and measurement systems). The result of this enhanced decision-making at all levels of the organization is optimal resource allocation and improved business performance. Profitability in business today relies on speed, flexibility and efficiency at quality levels thought unobtainable just a few years ago. The slightest imbalance along the supply chain can increase costs, lengthen internal cycle times and delay new product introductions. These imbalances can eventually lead to a loss in both market share and competitive advantage. Meanwhile, organizations are also forging closer relationships with their customers and suppliers by defining tighter agreements in terms of shared processes and risks. As a result, many businesses are deeply immersed in continuously re-engineering their processes to improve quality. Six sigma and Balanced Scorecard type efforts are increasingly prevalent. ODM enables organizations to remove supply chain imbalances while improving the speed, flexibility and efficiency of their business processes. This leads to stronger customer and partner relationships and a sustainable competitive advantage.

ODM VERSUS DATA MINING Data mining is the process of discovering and interpreting previously unknown patterns in databases. It is a powerful technology that converts data into information and potentially actionable knowledge. However, obtaining new knowledge in an organizational vacuum does not facilitate optimal decision-making in a business setting. The unique organizational challenge of understanding and leveraging ODM to engineer actionable knowledge requires assimilating insights from a variety of organizational and technical fields and developing a comprehensive framework that supports an organization’s quest for a sustainable competitive advantage. These fields include data mining, business strategy, organizational learning and behavior, organizational culture, organizational politics, business ethics and privacy, knowledge management, information sciences, and decision support systems. These fundamental ODM elements can be categorized into three main fields: AI, IT, and OT. Our research and industry experience suggest that successfully leveraging ODM requires integrating insights from all three categories in an organizational setting typically characterized by complexity and uncertainty. This is the essence of ODM. The core differentiator between ODM and straightforward data mining is OT.

OT AND ODM Organizations are primarily concerned with studying how operating efficiencies and profitability can be achieved through the effective management of customers, suppliers, partners and employees. To achieve these goals, research in OT suggests that organizations use data in three vital knowledge-creation activities. This organizational knowledge creation and management is a learned ability that can only be achieved via Copyright © 2004, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.

4 Nemati & Barko

an organized and deliberate methodology. This methodology is a foundation for successfully leveraging ODM within the organization. The three knowledge creation activities (Choo, 1997) are: • Sense making is the ability to interpret and understand information about the environment and events happening both inside and outside the organization. • Knowledge making is the ability to create new knowledge by combining the expertise of members to learn and innovate. • Decision-making is the ability to process and analyze information and knowledge in order to select and implement the appropriate course of action. First, organizations use data to make sense of changes and developments in the external environments — a process called sense making. This is a vital activity wherein managers discern the most significant changes, interpret their meanings, and develop appropriate responses. Second, organizations create, organize and process data to generate new knowledge through organizational learning. This knowledge creation activity enables the organization to develop new capabilities, design new products and services, enhance existing offerings, and improve organizational processes. Third, organizations search for and evaluate data in order to make decisions. This data is critical. Since all organizational actions are initiated by decisions and all decisions are commitments to actions, the consequences of which will, in turn, lead to the creation of new data. Adopting an OT methodology enables an enterprise to enhance the knowledge engineering and management process. In another OT finding, researchers and academic scholars have observed that there is no direct correlation between IT investments and organizational performance. Erik Brynjolfsson, a professor of information systems at the MIT Sloan School, discovered that identical IT investments in two different companies may give a competitive advantage to one company but not the other. He realized a key factor for the competitive advantage in an organization is not the IT investment, but the effective utilization of information as it relates to organizational performance. This finding emphasizes the necessity of integrating OT practices with robust IT and AI techniques in successfully leveraging ODM.

ONGOING ODM RESEARCH Given the scarcity of past research in ODM and its growing acceptance and importance in organizations, we have conducted empirical research over the past few years that explored the utilization of ODM in organizations along with project implementation factors critical for success. We surveyed ODM professionals from multiple industries in both domestic and international organizations. Our initial research examined the ODM industry status and best practices, identified both technical and business issues related to ODM projects, and elaborated on how organizations are benefiting through enhanced enterprise decision-making (Nemati & Barko, 2001). The results of our research suggest that ODM can improve the quality and accuracy of decisions for any organization willing to make the investment. After exploring the status and utilization of ODM in organizations, we decided to focus subsequent research on how organizations implement ODM projects and the Copyright © 2004, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.

Organizational Data Mining: An Introduction 5

factors critical for its success. Similar to our initial research, this was pursued in response to the scarcity of empirical research investigating the implementation of ODM projects. To that end, we developed a new ODM Implementation Framework based on data, technology, organizations and the Iron Triangle (soon to be published). Our research demonstrated that selected organizational data mining project factors, when modeled under this new framework, have a significant influence on the successful implementation of ODM projects. Our latest research has focused on a specific ODM technology known as Electronic Customer Relationship Management (e-CRM) and its data integration role within organizations. We developed a new e-CRM Value Framework to better examine the significance of integrating data from all customer touch points with the goal of improving customer relationships and creating additional value for the firm. Our research findings suggest that despite the cost and complexity, data integration for e-CRM projects contributes to a better understanding of the customer and leads to higher return on investment (ROI), a greater number of benefits, improved user satisfaction and a higher probability of attaining a competitive advantage.

ODM ADVANTAGES A 2002 Strategic Decision-making study conducted by Hackett Best Practices determined that “world-class” companies have adopted ODM technologies at more than twice the rate of “average” companies (Hoblitzell, 2002). ODM technologies provide these world-class organizations greater opportunities to understand their business and make informed decisions. ODM also enables world-class organizations to leverage their internal resources more efficiently and effectively than their “average” counterparts who have not fully embraced ODM. Many of today’s leading organizations credit their success to the development of an integrated, enterprise-level ODM system. For example, Harrah’s Entertainment has saved over $20 million per year since implementing its Total Rewards CRM program. This ODM system has given Harrah’s a better understanding of its customers and enabled the company to create targeted marketing campaigns that almost doubled the profit per customer and delivered same-store sales growth of 14 percent after only the first year. In another notable case, Travelocity.com, an Internet-based travel agency, implemented an ODM system and improved total bookings and earnings by 100 percent in 2000. Gross profit margins improved 150 percent, and booker conversion rates rose to 8.9 percent, the highest in the online travel services industry. In another significant study, executives from 24 leading companies in customerknowledge management, including FedEx, Frito-Lay, Harley-Davidson, Procter & Gamble and 3M, realized that in order to succeed, they must go beyond simply collecting customer data and translate it into meaningful knowledge about existing and potential customers (Davenport, Harris & Kohli, 2001). This study revealed that several objectives were common to all of the leading companies, and these objectives can be facilitated by ODM. A few of these objectives are segmenting the customer base, prioritizing customers, understanding your customer’s Internet behaviors, engendering customer loyalty and increasing cross-selling opportunities.

Copyright © 2004, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.

6 Nemati & Barko

ODM EVOLUTION Past Initially, IT systems were developed to automate expensive manual systems. This automation provided cost savings through labor reductions and more accurate, faster processes. Over the last three decades, the organizational role of IT has evolved from efficiently processing large amounts of batch transactions to providing information in support of tactical and strategic decision-making activities. This evolution from automating expensive manual systems to providing strategic organizational value led to the birth of Decision Support Systems (DSS), such as data warehousing and data mining. Operational and DSS are now a vital part of many organizations. The organizational need to combine data from multiple stand-alone systems (financial, manufacturing, distribution) grew as corporations began to acknowledge the power of combining these data sources for reporting. This spurred the growth of data warehousing, where multiple data sources were stored in a format that supported advanced data analysis. The slowness in adoption of ODM techniques in the ’90s was partly due to an organizational and cultural resistance. Business management has always been reluctant to trust something it does not fully understand. Until recently, most businesses were managed by instinct, intuition and “gut feel.” The transition over the past 20 years to a method of managing by the numbers is both the result of technology advances, as well as a generational shift in the business world as younger managers arrive with IT and experience.

Present Many current ODM techniques trace their origins to traditional statistics and AI research from the 1980s. Today, there are extensive vertical data-mining applications providing analysis in the domains of banking and credit, bioinformatics, CRM, health care, human resources, e-commerce, insurance, investment, manufacturing, marketing, retail, entertainment and telecommunications. Our latest survey findings indicate that the banking, accounting/financial, e-commerce and retail industries display the highest ODM maturity level to date. The need for service organizations (banking, financial, health care and insurance) to build a holistic view of their customers through a mass customization marketing strategy is critical to remaining competitive. And organizations in the ecommerce industry are continuing to improve online customer relationships and overall profitability via e-CRM technologies (Nemati & Barko, 2001). Continuous technological innovations now enable the affordable exploration of enormous volumes of data. It is the combination of technological innovation, creation of new advanced pattern-recognition and data-analysis techniques, ongoing research in organizational theory and the availability of large quantities of data that have guided ODM to where it is today.

ENVISIONING THE FUTURE The number of ODM projects is projected to grow more than 300 percent in the next decade (Linden, 1999). As the collection, organization, and storage of data rapidly

Copyright © 2004, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.

Organizational Data Mining: An Introduction 7

increases, ODM will be the only means of extracting timely and relevant knowledge from large corporate databases. The growing mountains of business data coupled with recent advances in OT and technological innovations provides organizations with a framework to effectively use their data to gain a competitive advantage. An organization’s future success will depend largely on whether or not they adopt and leverage this ODM framework. ODM will continue to expand and mature as the corporate demand for oneto-one marketing, CRM, Web personalization and related interactive media increases. As IT advances, organizations are able to collect, store, process, analyze and distribute an ever-increasing amount of data. Data and information are rampant, but knowledge is scarce. As a result, most organizations today are governed by managerial intuition and historical reporting. This is the by-product of years of system automation. However, we believe organizations are slowly moving from the Information Age to the Knowledge Age, where decision makers will leverage ODM and Internet technologies to augment intuition in order to allocate scarce enterprise resources for optimal performance. As organizations set a strategic course into the Knowledge Age, there are a number of difficulties awaiting them. As its name suggests, ODM is part technological and part organizational. Organizations are comprised of individuals, management, politics, culture, hierarchies, teams, processes, customers, partners, suppliers and shareholders. The never-ending challenge is to successfully integrate data mining technologies with organizations to enhance decision-making with the objective of optimally allocating scarce enterprise resources. As many consultants, professionals, industry leaders and the editors of this book can attest, this is not an easy task. The media can oversimplify the task, but successfully implementing ODM is not accomplished without political battles, project management struggles, cultural shocks, business process reengineering, personnel changes, short-term financial and budgetary shortages, and overall disarray. ODM is a journey, not a destination, so there must be a continual effort in revising existing knowledge bases and generating new ones. But the benefits far outweigh both the technical and organizational costs, and the enhanced decision-making capabilities can lead to a sustainable competitive advantage. Recent ODM research revealed a number of industry predictions that are expected to be key ODM issues in the future (Nemati & Barko, 2001). About 80 percent of survey respondents expect Web farming/mining and consumer privacy to be significant issues, while more than 90 percent predict ODM integration with external data sources to be important. We also foresee the development of widely accepted standards for ODM processes and techniques to be an influential factor for knowledge seekers in the 21st century. One attempt at ODM standardization is the creation of the Cross Industry Standard Process for Data Mining (CRISP-DM) project that developed an industry and tool neutral data-mining process model to solve business problems. Another attempt at industry standardization is the work of the Data Mining Group in developing and advocating the Predictive Model Markup Language (PMML), which is an XML-based language that provides a quick and easy way for companies to define predictive models and share models between compliant vendors’ applications. Lastly, Microsoft’s OLE DB for Data Mining is a further attempt at industry standardization and integration. This specification offers a common interface for data mining that will enable developers to embed data-mining capabilities into their existing applications. One only has to consider

Copyright © 2004, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.

8 Nemati & Barko

Microsoft’s industry-wide dominance of the office productivity (Microsoft Office), software development (Visual Basic) and database (SQL Server) markets to envision the potential impact this could have on the ODM market and its future direction. Although many improvements have materialized over the last decade, the knowledge gap in many organizations is still prevalent. Enterprises that see the strategic value of evolving into knowledge organizations by leveraging ODM will benefit directly in the form of improved profitability, increased efficiency, and a strategic competitive advantage.

REFERENCES Anonymous. (2001, February 17). The slow progress of fast wires. The Economist, London, (358)8209. Brown, E. (2002, April 1). Analyze this. Forbes, 96-98. Choo, C. W. (1997). The knowing organization: How organizations use information to construct meaning, create knowledge, and make decisions. Retrieved from Oxford University Press website: http://www.choo.fis.utoronto.ca/fis/ko/default.html. Davenport, T. H., Harris, J. G., & Kohli, A. K. (2001). How do they know their customers so well? Sloan Management Review, 42(2), 63-73. Hoblitzell, T. (2002, July). Disconnects in today’s BI systems. DM Review, 56-59. Linden, A. (1999, July 7). CIO update: Data mining applications of the next decade. Gartner Group - Inside Gartner Group. Nemati, H. R., & Barko, C. D. (2001). Issues in organizational data mining: A survey of current practices. Journal of Data Warehousing, 6(1), 25-36. Wessel, D. K. (2002). Decision making in the digital age. DM Review, Resource Guide, 16-18.

Copyright © 2004, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.