prediction markets

PREDICTION MARKETS

Prediction Markets and the “Trough of Disillusionment” 7dZh[Wi=hW[\[ PREVIEW: Foresight’s Spring 2011 issue included Robert Rieg and 3BNPOB4DIPEFSTBSUJDMFi$PSQPSBUF1SFEJDUJPO.BSLFUT1JUGBMMT BOE#BSSJFSTwPOUIFIVSEMFTUIBUNVTUCFPWFSDPNFUPQSPQFSMZ FTUBCMJTIBDPSQPSBUFQSFEJDUJPONBSLFU $1. 5IFZFOVNFSBUFE four principal obstacles: (1) identifying the appropriate subject to be forecast; (2) defining the subject clearly enough for people to know what they are betting on; (3) NPUJWBUJOHBOETFDVSJOHBOBEFRVBUFOVNCFSPGQBSUJDJQBOUTBOE FNCFEEJOHUIF$1.XJUIJO UIFDPSQPSBUFDVMUVSF5IFJSDPODMVTJPOPOMZTFMFDUJWFVTFPGB$1.XPVMECFGFBTJCMFBOEQSBDUJDBM*O the following response, Foresight’s1SFEJDUJPO.BSLFU&EJUPS"OESFBT(SBFGFOPUFTUIBUDFSUBJOPGUIF DIBMMFOHFTDBOSFBEJMZCFPWFSDPNF BOEUIBUBQSPNJTJOHSPMFGPS$1.TMJFTBIFBEoJGUIFDPOEJUJPOT for success are better understood.

THE LACK OF EVIDENCE FOR CPM

In this year’s Spring issue of Foresight, Robert Rieg and Ramona Schoder (2011) reviewed evidence on the adoption of corporate prediction markets (CPMs) as a method for business forecasting. The authors argue that the empirical record of the use of CPMs in corporations is weak and describe several obstacles that hinder their practical implementation.

Figure 1 shows how many searches have been done for the term “prediction markets” relative to the total number of searches on Google during the five years from July 2006 to June 2011. The tally peaked in the fall of 2008 – the most recent U.S. presidential election – and then plummeted in a free fall until January 2009. Since that time, relative interest in prediction markets appears to be consistently lower than in 2006, at a slightly decreasing trend.

It’s true that prediction markets are still not widely used for business forecasting. Despite Figure 1- Relative frequency of Google searchessearches for Figure 1: Relative frequency of Google for “prediction markets “prediction markets” (July 2006 to June 2011) more than a decade of research on the subject, (July 2006 to June 2011) Rieg and Schoder found little more than a handful of empirical studies conducted within organizations, the most recent in 2009. Of these, many were published on the websites of either the companies themselves or software vendors, which certainly may raise questions about their reliability. Google Insights for Search, a service that provides information on how often Internet users search for certain keywords on Google, reveals numbers suggesting that interest in prediction markets is generally decreasing.

BARRIERS TO CORPORATE PREDICTION www.forecasters.org/foresight FORESIGHTMARKETS 43

Key Points t0 WFS UIF QBTU GJWF ZFBST JOUFSFTU JO prediction markets has waned. An important reason for the decline within the field of business forecasting is a lack of evidence to date that prediction markets can improve forecast accuracy relative to more traditional TPVSDFTPGKVEHNFOU t/FWFSUIFMFTT JU XPVME CF XSPOH UP XSJUF off the CPM as a valuable tool for business forecasting. While there are barriers to the adoption of prediction markets within a business, some are relatively easy to solve by appropriate market design. Predictionmarket vendors have recognized this need and have started to offer consulting services in order to rescue clients from failed trials. t1SFEJDUJPO NBSLFUT TIPVME CF VTFGVM JG B CVTJOFTT SFRVJSFT NBOZ GPSFDBTUT UIBU IBWF to be continuously updated. It could also be worthwhile to implement prediction markets as a supplement to forecasting methods that are already in place. This would allow for checking the validity of the internal forecasts and provide an additional measure of uncertainty. t5IF JEFB PG VTJOH QSFEJDUJPO NBSLFUT GPS QSPKFDU NBOBHFNFOU SFNBJOT B QBSUJDVMBSMZ interesting field of application.

organizations, noting four particular challenges: finding appropriate items for a prediction market to forecast; clearly specifying what the market should forecast; motivating adequate participation; and overcoming cultural barriers within an organization. Except for the last, which is a new insight, these issues are well known and have been frequently discussed, including in this journal (e.g., Graefe, 2008). While they can indeed provide barriers to the adoption of prediction markets, they are also relatively easy to solve by appropriate market design. For example, in response to the problem of “thin” markets (few participants), researchers have developed automated market-maker mechanisms to provide additional liquidity (Hanson, 2003; Pennock, 2004). Such mechanisms can solve the thin-market problem, as participants don’t need other participants to trade with. Instead, they can trade with the market maker whenever they want. In general, many of the issues raised can be avoided by providing guidance to organizations wishing to experiment with CPMs and taking them through the implementation process. Prediction-market vendors have recognized this need and have started to offer consulting services in order to save their clients from failed trials.

I contend that the main reason prediction markets have not been widely adopted for business forecasting is really much simpler: This development is also reflected by the there’s still not enough evidence to prove that classifications of prediction markets in the the method provides accurate forecasts that “Gartner Hype Cycle for Social Software,” can aid business decisions. which is published every year (http://www. In an attempt to summarize published gartner.com/technology/research/hype- empirical studies on the relative accuracy of cycles/). According to Gartner, prediction prediction markets for business forecasting markets reached the “peak of inflated (Graefe, 2011), I identified only five relevant expectations” in 2008. Since 2009, the method articles, which included eight comparisons of is stuck in the “trough of disillusionment.” prediction markets and alternative forecasting methods. Of these, some studies used similar BARRIERS TO CORPORATE data or analyzed similar problems. For PREDICTION MARKETS example, five of the eight comparisons were Rieg and Schoder highlight several reasons conducted within the movie industry, four of why CPMs have failed to be adopted by

44

FORESIGHT Fall 2011

which used data from the Hollywood Stock Exchange. Over the eight comparisons, there were no differences in the relative accuracy of prediction markets and alternative methods. While prediction markets were more accurate than naïve and simple econometric models, evidence on the relative performance of prediction markets and individual and combined expert judgment was mixed. The lack of evaluative studies on CPMs in the arena of business forecasting was one of the major conclusions from this review.

THROUGH THE TROUGH OF DISILLUSIONMENT

To advance the field and to put prediction markets into practical use within organizations, we need additional evidence on the relative accuracy of prediction markets for business forecasting. Future research should focus on specific conditions where organizations could benefit from prediction markets. For example, prediction markets should be valuable in situations where dispersed information becomes frequently available, as the market could continuously incorporate such information. In contrast, asking experts to reveal individual forecasts, participate in a Delphi, or attend a meeting are oneoff activities that need to be triggered by someone with the authority to set them in motion. Prediction markets might already have aggregated information by the time a decision maker recognizes the necessity to obtain information. Apart from the initial cost of setting up a prediction market, the marginal costs for running one or more markets are low. Prediction markets should therefore be useful if someone needs many forecasts that need to be continually updated. It could also be valuable to implement CPMs as a supplement to forecasting methods that are already in place. This would allow for checking the validity of the existing forecasts and provide an additional measure of uncertainty. If the deviations from the

prediction-market forecast and the internal forecasts are large, this might alert decision makers to have a closer look.

"The main reason prediction markets have not been widely adopted for business forecasting is really much simpler: there’s still not enough evidence to prove that the method provides accurate forecasts that can aid business decisions." The idea of using prediction markets for project management remains a particularly interesting field of application. Cherry (2007) reported on an internal market implemented at Microsoft that illustrates how prediction markets can overcome group pressures and decrease bias. The goal of this particular CPM was to predict the launch date of a certain software product. After trading started, the market instantly predicted that the product would not be finished by the set deadline. In this case, the market revealed information that no member of the development team previously had the stomach to communicate directly to the project manager – evidently, a strong example of group pressures. Management trusted the market forecast and cut some features from the software that were thought to be slowing down the development process. This decision was again immediately reflected by the market, which now indicated higher probabilities that the product might still be finished on time. When customers demanded that the deleted features be restored, the market again predicted that the product would be finished late – and was ultimately right. In sum, taskand project-completion forecasts appear to be well suited for prediction markets: 1. Large projects can be cut down into smaller tasks, each task having a CPM to forecast its completion date. 2. Information on the completion dates for certain tasks is widely dispersed among project members and teams. 3. The time horizons of smaller tasks can be kept reasonably short – this way, interest in participating in the markets remains high. www.forecasters.org/foresight FORESIGHT

45

SUMMARY

conditions where they reliably improve upon alternate forecasting methods. The road ahead still may be a long one: in their 2010 report on the “Hype Cycle for Social Software,” Gartner estimated that mainstream adoption of corporate prediction markets would take another 5-10 years.

Nevertheless, it is a mistake to ignore the potential of CPMs as a valuable tool for business forecasting. Future research should delve more deeply into the relative accuracy of prediction markets, in order to identify

REFERENCES

Over the past five years, it would seem that interest in prediction markets has generally fallen off. Within the field of business forecasting, there is still a lack of evidence on the accuracy of prediction markets relative to other forecasting methods.

Andreas Graefe is 'PSFTJHIU’s

Prediction Market Editor. He recently began a position at a German pay-TV company, where he is forecasting call-center data. When he is not working on forecasting topics, Andreas enjoys the outdoors, particularly hiking and skiing the Bavarian Alps. [email protected]

Cherry, S. (2007). Bet on it! Can a stock market of ideas help companies predict the future?, IEEE Spectrum, 44, 48-53. Graefe, A. (2011). Prediction market accuracy for business forecasting. In: Vaughan Williams, L. (Ed.), Prediction Markets, New York: Routledge, 87-95. Graefe, A. (2008). Prediction markets: Defining events and motivating participation, Foresight, Issue 9 (Spring), 30-32. Hanson, R. (2003). Combinatorial information market design, Information Systems Frontiers, 5, 107-119. Pennock, D.M. (2004). A dynamic pari-mutuel market for hedging, wagering, and information aggregation, Paper presented at the ACM Conference on Electronic Commerce, New York, USA. Rieg & Schoder (2011). Corporate prediction markets: pitfalls and barriers, Foresight, Issue 21 (Spring), 35-40.

Prediction Markets and the “Trough of Disillusionment”: Reply by Robert Rieg and Ramona Schoder CORPORATE PREDICTION MARKETS: STILL A LONG WAY TO PLUG AND PLAY We agree with Andreas Graefe on these major points: tUIBU UIFSF BSF SFBM CFOFĕUT BT XFMM BT considerable potential for corporate prediction markets (CPM); tUIBUUIFSFBSFNFDIBOJTNTGPSPWFSDPNJOH the problem of “thin” markets; and tUIBU QSFEJDUJPO NBSLFUT TIPVME CF considered as a supplement to – rather than a replacement for – existing forecasting methods.

46

FORESIGHT Fall 2011

Nevertheless, challenges remain to incorporating a CPM into an organization’s culture, and these should not be taken lightly. Companies need to acquire pertinent evidence on the preconditions for successful implementation of a CPM.

The Issue of Thin Participation (Low Liquidity)

Andreas indicates that automated marketmaker mechanisms can improve performance in “thin” markets by matching orders that previously do not overlap, or by giving some traders more weight than others (Hanson

& Oprea, 2009). Another means to deal with the problem could be the dynamic pari-mutuel market, a hybrid between a continuous double auction and a pari-mutuel market. It combines the advantages of both mechanisms by providing infinite buy-in liquidity at no risk for the market institution and by aggregating information dynamically (Pennock, 2004). But perhaps we should ask why there’s a thin market in the first place. It could be that the participants do not understand how to trade, or don’t possess relevant information, or the items for prediction are not clearly defined. Such problems need to be solved by organizational intervention: better training, clearer explanation, tactics to attract additional traders, and so on. Greater participation means additional liquidity to the market and, just as important, provides more diverse information.

Prediction Markets as Supplements to Existing Forecasting Methods We agree with Andreas that we shouldn’t expect prediction markets to replace existing forecasting methods, but rather to supplement them. The way companies use prediction markets, however, requires careful consideration. Businesses must be reasonably sure that a prediction market will be beneficial in a particular application, and this means a need for robust evidence about the benefits and preconditions of using such a tool. There are many still-unanswered questions, and this is why we warn against using prediction markets without careful consideration.

Using Prediction Markets in Project Management We also see the use of prediction markets for project management as a promising area of application. But even here, people have to be convinced and motivated to participate in a prediction market, and still have to be integrated into the organizational culture of the company.

Conclusion: Not Yet a Ready-Made Tool While we broadly agree with Andreas’ points, we think the usage of prediction markets as a forecasting instrument for businesses is not fully understood. Further research is necessary before companies can safely decide about incorporating CPMs into their forecasting processes. The main message of our article in Foresight is that companies need a better understanding of the challenges in successful implementation of a CPM. REFERENCES Hanson, R. & Oprea, R. (2009). A manipulator can aid prediction market accuracy, Economica, 76(302), 304-314. Pennock, D.M. (2004). A dynamic pari-mutuel market for hedging, wagering, and information aggregation, Paper presented at the ACM Conference on Electronic Commerce, New York. Rieg, R. & Schoder, R. (2011). Corporate prediction markets: Pitfalls and barriers. Foresight, Issue 21 (Spring), 35-40.

Robert Rieg is a Professor of Management Accounting and Control and currently Vice Dean for Research of the Faculty of Business at Aalen University, Germany. He is also head of a research project on corporate prediction markets funded by the German Ministry of Education and Research. His interests include forecasting for planning and budgeting [email protected]

Ramona Schoder is a research assistant for the project on corporate prediction markets at Aalen University.

www.forecasters.org/foresight FORESIGHT

47