Predicting Consumer Behavior at Unprecedented Scale - Yottamine ...

Executive Brief Whatsthebigdata.com

Predicting Consumer Behavior at Unprecedented Scale Yottamine Delivers Breakthrough Machine Learning for Big Data

Gil Press

The Perils of Predictions Remember the old saying “Those who live by the crystal ball are doomed to die by eating broken glass”? We all want to know what will happen next and look for experts to tell us. The business of prediction, however, has always been risky and highly inaccurate. In the largest and best-known test of the accuracy of expert predictions, the average expert was found to be only slightly more accurate than a dart-throwing chimpanzee. Many experts would have done better if they had made random guesses. But what if you had a crystal ball made of software? Machine learning software has been around for a while and has proven to be a reliable tool for making accurate predictions. Today, however, the amount of data points the software can “learn” from has exploded, suffocating traditional approaches to machine learning. The age of Big Data requires a new approach to efficient and accurate predictions facilitated by a new type of machine learning software. Yottamine provides such a breakthrough approach, designed from the ground up to take advantage of very large volumes of data.

Big Data: Lots of digital crumbs, no cake Like all buzzwords, Big Data is suffering from both exaggerated expectations and misguided maligning. One fact, however, is indisputable: The amount of digital data created and replicated worldwide has been growing by leaps and bounds over the last decade and a half. Consider the following milestones in the march of digitization and the resulting data deluge:  In 2002, digital storage surpassed non-digital storage for the first time  By 2007, 94% of storage capacity in the world was digital, a complete reversal from 1986, when 99.2% of all storage capacity was analog  The amount of digital data created each year has grown by a factor of 9 between 2005 and 2010. In 2005, each person on the Internet produced and consumed 130 gigabytes on average. That has grown to 613 gigabytes in 2010  In 2012, less than 1% of the world’s data was analyzed. And we’ve seen nothing yet. In addition to the 4 or 5 billion people that are expected to get on the Internet by 2020, the “Internet of Things” will become a new driver of data growth. By 2020, there will be 26 times more connected things than people; the amount of digital data created annually will reach 40 trillion gigabytes, resulting in a 50-fold growth from the beginning of 2010. We can conclude from these estimates (from IDC and others) that the flood of digital data is recent, accelerating rapidly, and largely wasted. Most of the digital crumbs consumers are leaving on their Internet journeys end up stored somewhere in a rarely-accessed data warehouse or archive, never to provide new insights to the owners of the data. To cost-effectively benefit from this data deluge rather than drown in it, businesses must increase their data analysis efforts and focus them on where they will get the biggest return. Finding out what consumers will do next is a very profitable area of data analysis, one where, with the right tools, the number of new insights gained grows with the amount of data collected.

Skating to where the puck is going to be The future is not what it used to be. The connected consumer killed the long-term strategic plan and turned business reality into a continuously evolving fabric of rapidly changing habits, desires, competitors, technologies, and channels. Like Wayne Gretzky, one of the most successful ice hockey players of all time, organizations don’t want to skate to where the puck is, they would like to skate to where it is going to be. They would like to find out in real-time what their customers and potential customers are going to do next. (See the sidebar “What’s a CMO to Do?”) Until the era of Big Data, skating to where the puck is going to be was based on the intuition and experience of the organization’s leaders. Having the right leaders at the right time at all levels of the organization, helped in making the right decisions and anticipating correctly market disruptions, competitor moves, and rising opportunities. But intuition and experience, while still valuable, may not be adequate in the era of Big Data. Moreover, the nature of prediction and its goals have been drastically altered by the changing business environment.

Executive Brief: Predicting Consumer Behavior at Unprecedented Scale

Page 2

Take for example companies that provide their customers a discount six or seven months after they subscribe to their services because they found out that’s the peak month for defections. These companies could save a lot of money if they knew which individual customers were growing unhappy and offer only them the promotion just as they were becoming disgruntled. The connected consumer killed “demographics” and “market segments.” Once the bedrock of marketing and sales planning, group-based forecasts are being replaced by predictions about individuals. Which do you prefer to know: How many people will buy your product in the Chicago area next year or which specific individuals will buy or will be willing to buy your product next month or in the next hour? A number of leading-edge companies, government agencies, and nonprofits already are using predictions to skate to where the puck will be. Law enforcement agencies reduce crime rates by predicting where and when a crime is going to be committed. Utilities anticipate when outages are going to occur. Traffic monitoring systems alert drivers to traffic jams ahead of them and suggest alternative routes. Hospitals watch for the beginnings of a potentially fatal infection in premature babies, and the retail industry is investing in upgrading its ability to make predictions about specific customers. The organizations already benefiting from predictions based on the analysis of Big Data have been using a variety of data sources, including data from traditional databases, images, HTML, email, sensors, IT systems logs, click streams, social networks, Web-based transactions, and location/GPS. They have found that the analysis of the data that their systems collect, sometimes automatically, can answer questions that intuitively do not seem related to the data. Combining data from multiple sources also lead to discovery of new insights and opportunities. In many situations, size does matter and the more data you use in the analysis, the better your predictions become. Leading-edge organizations have started to capitalize on the predictive benefits of Big Data. They are doing it with the help of a software crystal ball—machine learning software algorithms.

The process of learning by doing We all learn by experience and by generalizing from specific instances in our experience, generalizations that serve as a basis for predictions. Wayne Gretzky knew where to skate because he has developed, based on his experience, a mental model of hockey playing that helped him process large amount of data very quickly, predicting where every player—and most importantly, the puck—is going to be in the next few seconds. The application of computer technology to learning has been called “machine learning,” a branch of artificial intelligence that has been making steady progress in theory and practice for the last few decades. Similar to our basic learning process, the computer is “trained” by data which is labeled or classified based on previous outcomes, and its software algorithms “learns” how to predict the classification of new data that is not labeled or classified. For example, after a period of training in which the computer is presented with spam and non-spam email messages, a good machine learning program will successfully identify, (i.e., predict,) which email message is spam and which is not without human intervention. Executive Brief: Predicting Consumer Behavior at Unprecedented Scale

What’s a CMO to Do? In a recent survey of 1700 CEOs, the IBM Institute for Business Value found that generating greater consumer insight is the top priority for CEOs worldwide. Most of these CEOs also agree that deeper customer insight will come from a much better use of data and analytics. While CEOs expect CMOs to deliver on their top priority, a recent study indicated that more than 70% of CMOs feel they are underprepared to manage the explosion of data and feel they lack true insight. The CMO Survey has been talking to top marketers in the United States twice per year since 2008. Here are highlights from the most recent edition of the survey:  60.1% of CMOs reported increasing pressures from their CEOs (or board) to prove the value of marketing  Approximately 60% collected online customer behavior data for targeting purposes and 88.5% expected to increasingly do so over time  Spending on marketing analytics expected to increase 40% in three years  Most projects fail to use marketing analytics; this is becoming worse as Big Data grows  Most companies do not have the right talent to fully leverage marketing analytics. CEOs and CMOs are struggling to turn the deluge of data flooding their organizations into consumer insight that can improve the bottom-line. They should focus their marketing analytics efforts on Big Data machine learning software that takes advantage of the size of the data analyzed to provide accurate and affordable predictions of consumer behavior.

Page 3

In addition to spam filtering, machine learning has been applied successfully to problems such as hand-writing recognition, machine translation, fraud detection, and product recommendations. Many successful Web-native companies such as Google, Amazon and Netflix have built their fortunes with the help of clever machine learning algorithms. The real-world experiences of these companies have proved how successful machine learning can be in using lots of data from variety of sources to predict consumer behavior. Using lots and lots of data makes predictive models more robust and predictions more accurate. But even these organizations are still learning to deal with the challenges that are preventing the widespread adoption and establishment of processes and practices necessary for predicting consumer behavior. In many situations, conventional approaches to machine learning are not cost-effective and accurate, as they were not designed for handling Big Data.

Big Data challenges to conventional machine learning Big Data machine learning requires lots of processing power and large amounts of computer memory and disk storage. Most predictive analytics and machine learning software programs running on desktop PCs or on servers were not designed for the Big Data era and typically operate on small data sets. Conventional machine learning software can only execute one instruction on one core of one CPU of one computer at a time, and it cannot handle large volumes of data in memory. These programs have no data parallelism, making them very slow to execute. Most machine learning software cannot take advantage of advances in the performance and memory and storage capacity of desktop computers and data center servers. Even if you have the budget to use or rent a very large computer, conventional software will only use a fraction of the computer to operate. Conventional machine learning software also requires the presence of highly-skilled and difficult-to-find data scientists. But the traditional machine learning process decreases the productivity of these expensive resources—it is highly iterative, requiring many hours of trial-and-error guesswork by the data scientists. This lengthy and costly process degrades one of the most important metrics in the fast-paced environment in which the data scientists—and their business managers—work: Time-to-prediction or the time it takes to build the predictive model. In addition to the cost and productivity of the resources deployed, traditional machine learning approaches are sometime stymied by the very nature of what they are trying to predict. More often than not the goal is to predict something important that doesn’t happen a lot. The data is “unbalanced”—the number of spam messages is much smaller than the number of non-spam messages, for example—and the algorithms must include unique capabilities for automatically classifying marginal data correctly. The algorithms also should not fail in the face of the complexity and non-linearity of consumer behavior and must be able to handle a very large number of variables and take into account the interaction among those variables. Big Data is too big to handle for conventional approaches to machine learning which fail on both counts—the speedy processing of the data and the sophistication of the algorithms used for prediction.

Yottamine: Breakthrough Machine Learning Software for Big Data Yottamine’s machine learning solution attacks the traditional limitations on both fronts—the processing of the data and the accuracy of the predictive algorithms, to produce a solution that is fast, accurate, scalable, and affordable. Yottamine’s software has a hyper-parallel architecture which enables the parallel processing of data and results in far greater speed and scale, at much lower cost than any other approach to machine learning. Designing the software from ground up for parallelization ensures that it uses every core of every processor on as many computers as needed to build the desired model. And it takes full advantage of in-memory technology, holding an enormous number of data points in computer memory, where they can be accessed for processing much faster than if they are kept on a disk storage.

Executive Brief: Predicting Consumer Behavior at Unprecedented Scale

Page 4

Typical Machine Utilization

Optimum Machine Utilization

The proprietary algorithms developed by Yottamine advance the accuracy and efficiency of developing and applying predictive models. These algorithms are self-optimizing and self-correcting, thus reducing computer cycles and data scientists’ time. They include mechanisms for handling noisy data, (e.g., mis-labeled data or missing data), allowing data scientists to focus on modeling rather than on data cleaning. And they are specifically enhanced for “finding the needle in the haystack,” for dealing with highly “unbalanced data,” where the number of observations of one type is significantly larger than for another, (e.g., predicting the presence of one buyer in a million clicks). As such, these algorithms are designed to sift through very large volumes of data to find the few occurrences that matter for an accurate prediction.

Benchmarking the Performance of Big Data Machine Learning Software Making predictions based on known examples is addressed today by a variety of approaches. Some of these approaches are difficult to tailor to a specific problem and require highly skilled human design. Others are more generic and flexible. One particularly versatile and powerful method, known as the Support Vector Machine (SVM) has proven to be both easy to apply and capable of producing results that range from good to excellent in comparison to other approaches. Libsvm is a popular open source library of algorithms of the SVM type. A major drawback of Libsvm is that it uses single-threaded software and, as a result, it can process only one program instruction at a time. This means it cannot take advantage of multi-core processors, multi-processor servers, and multi-server clusters. In contrast, Yottamine is built on highly parallel software that can use multiple cores, processors, and servers at the same time, dramatically reducing the time-to-prediction and the cost of accurately predicting consumer behavior. This difference is demonstrated in a side-by-side comparison of Libsvm and Yottamine. What takes Yottamine a few hours, takes Libsvm many days. Figure 1 summarizes the results of a test using a very large data set with more than 8 million data points with more than 6 billion numbers which Yottamine handles in a record six hours.

Figure 1. Big Data Benchmark Comparison

To take full advantage of the highly parallel nature of Yottamine’s software, it is best to run it on a public or private cloud with its automated provisioning of compute and memory resources, scaling with the data and the task at hand. With its cluster computing approach, Yottamine’s software makes one computer from many, using multiple computers, each with just one or two high core count processors that are linked by a high speed network (see Figure 2).

Executive Brief: Predicting Consumer Behavior at Unprecedented Scale

Page 5

Figure 2. Yottamine Core-to-Cloud Parallel Computing By addressing the challenges of predicting consumer behavior on both fronts—the machine learning algorithms and the way the data is processed—Yottamine has come up with a breakthrough approach to prediction with Big Data. It is setting records for speed, scale, accuracy, low cost, and data scientists’ productivity with its proprietary package of Big Data machine learning algorithms, specifically designed for on-demand cluster computing architecture.

The perfect melding of minds and machines Many predictions are what the experts want the future to be or simply an extension of what they are familiar and comfortable with. Big Data machine learning removes this potential bias from the process of prediction, relying instead on lots and lots of data to provide a true representation of the real world and on sophisticated algorithms to model this world accurately. It does not, however, removes humans from the equation, relying on their domain knowledge to develop and tweak the models. This is what computer technology has always excelled at—not replacing humans completely but rather augmenting their intelligence. But not all machine learning programs are created equal. Yottamine represents a concerted effort to address the limitations of conventional approaches in handling Big Data, by taking care of the two key challenges: The processing of unprecedentedly large amounts of data and the development of predictive algorithms that can handle the complexity of the real world as mirrored by Big Data. This breakthrough in machine learning software for Big Data provides an affordable entry to businesses of all sizes into the world of accurate predictions of consumer behavior.

gPress is a marketing, publishing, and research consultancy. Gil Press, Managing Partner, has held senior marketing and research management positions at NORC, DEC and EMC. Most recently, he was Senior Director, Thought Leadership Marketing at EMC, where he launched the Big Data conversation with the UC Berkeley “How Much Information?” and the IDC “Digital Universe” studies. Press is a contributor to Forbes.com www.forbes.com/sites/gilpress/ and blogs at whatsthebigdata.com and infostory.com. gPress is the sole copyright holder of this publication. Design by C. Veit at www.clv-design.com. Executive Brief: Predicting Consumer Behavior at Unprecedented Scale

Page 6