Percentage Forecasting Error: A New Forecasting Performance Measure— How Accurate Is It? Ronald K. Klimberg, Haub School of Business, Saint Joseph’s University, Philadelphia, PA, 19131,
[email protected]. George P. Sillup, Haub School of Business, Saint Joseph’s University, Philadelphia, PA, 19131,
[email protected] Kevin Boyle, Saint Joseph’s University, Haub School of Business, Philadelphia, PA, 19131,
[email protected] ABSTRACT The accuracy of forecasts has a critical impact on an organization. A new, practical and meaningful forecast performance measure, percentage forecast error (PFE), was introduced by the authors in an earlier publication. In this paper, we examined the accuracy of the PFE under several different scenarios and found the results to indicate that PFE offers forecasters an accurate and practical alternative to assess forecast accuracy. INTRODUCTION One of the most important, if not the most important, number for an organization to operate successfully is its forecast. The importance of an accurate forecast is time-tested wisdom and methods to ensure forecasting accuracy have been documented in the literature over the past decades (Barnett, 1988). Yet, organizations frequently use undisciplined approaches, approaches that are based on intuitive impressions by forecasters and often not supported by documented assumptions, to determine their forecasts. While some organizations have evolved to use more quantitative methods to assess forecast accuracy, such as Mean Absolute Percent Error (MAPE) or Mean Absolute Deviation (MAD), they are still left with discerning how accurate their forecast is. A new forecasting alternative, percentage forecasting error (PFE) offers a practical and accurate alternative that can give forecasters confidence.
LITERATURE REVIEW In response to today’s fast-changing market conditions and intensive competition in today’s struggling economy, companies are recognizing the need for increased forecasting accuracy and connecting that increased accuracy with quantitative methods (Pilinkieu, 2008). While this is encouraging, surveys of those who are involved with forecasting decisions, MBA students, 1
indicate that, although there is greater familiarity with forecasting techniques than there was over 25 years ago, there is still room for greater recognition of the benefit of using forecasting techniques (Rahmlow and Klimberg, 2002).
Need for Forecasting Accuracy across All Industries Reliance on quantitative methods is not for lack of quantitative methods. Several methods to improve forecast accuracy are being implemented within different industries. For example, the cost of forecast error (CFE) forecasting technique has been evaluated as more accurate than commonly used statistical approaches (e.g., regression) in enterprise resource planning (ERP) as part of SAP® software (Barbour, Robert and Robb, 2008). Customer Lifetime Value (CLV) modeling offers a method to evaluate forecasting error by assessing customer retention and to measuring actual variables to marketing efforts (Calciu 2009). Gallucci (2007) suggests that statistically based, hybrid forecasting should discard inputs that do not create value.
Interestingly, some of these forecasting methods that show models outperform judgment are being applied within a number of industries in the U.S. and in international markets. In the U.S., Phillips and Lopez (2009) applied Root Mean Square Error and Theil’s Inequality Coefficient to generate a forecast using real-time performance vs. out-of-sample analysis for the Western Blue Chip Economic Forecast. Its results were closest to actual results and beyond random chance; most favorable results were for the Dallas Fed Texas Model, which measured changes in in the Texas Leading Index that predicts changes in the economy.
Sichel (2009) used point of sales (POS) data from the Monet Group Inc. to forecast demand for costume jewelry fashion and found it to be an accurate indicator of consumer demand for costume jewelry. The pertrochemcials industry employed the Business Monitor International Ltd (BMI) methodology for generating industry forecasts, particularly for plant capacity, and found it to be an excellent comparator for companies’ forecast methods (USA Petrochemicals, 2011). Syntetos (2010) found that demand forecasting performance as measured by standard forecasting accuracy measures failed to anticipate consequences for stock control using pharmaceutical industry data. Syntetos’ findings about a regulated industry are consistent with other findings that 2
identified the need to use quantitative forecasting methods, such as Monte Carlo Analysis, to adjust for the uncertainty imposed by technical, regulatory and structural problems of regulated industries (Kiely, 2004).
Internationally, Kapetanios and Yates (2007) found that measuring recent economic events is less reliable than older data to develop an accurate forecast using U.K. aggregate expenditure data. Kerkkanen, Korpela and Huiskonen (2009) identified that inaccurate sales forecasts impact the supply chain in a large process industry company seeking to improve control over inventory policy decisions for its different sales divisions. The need for improved accuracy is heightened when a company’s product requires smooth coordination of international outsources as was the case for Sun Microsystems (Yellan, Kim and Stratulate, 2010).
Need for Quantitative Forecasting Techniques to Improve Accuracy Clearly, the need for greater forecasting accuracy is established in the literature. Furthermore, today’s harsh economic environment combined with shortened product life cycles and competitive pressure drastically increase the penalty of an inaccurate forecast, e.g., finished goods inventory for mobile phones (Reiner, Natter and Dreschler, 2009). Bearing that in mind, numerous approaches are being made to improve forecasting accuracy. These approaches include methods that range from changes to calculations to actual methods derived within companies to improve forecasting accuracy. For example, some adjustments are to use actual data as a divisor when comparing percentage error so it can be easily determined how much a forecast deviates from the actual data Katti (2008).
Other methods include improving accuracy by automating the process using Excel. Radovisky (2008) introduced a new integrated method of forecast accuracy, Mean Prediction Criterion (MPC), which combines mean absolute deviation (MAD) and mean square error (MSE) to identify the best forecasting method for Excel. Castle, Fawcett and Henry (2009) developed and tested a method in Europe, Nowcasting, that adjusts for interruptions in forecast periods and disaggregate monthly indicators built by automatic methods. To ensure flexible and robust supply chain forecasting under changing market demands (from slow to intermittent demand), 3
Wallstrom and Segerstedt (2010) evaluated the Croston Forecasting Technique (CFT) and single exponential smoothing and identified a way to equate the number of shortages with a cumulative forecast error.
The literature also addresses significant efforts to improve the well-documented forecasting method, MAPE. Ren and Glasure (2009) assessed their Revised MAPE (RMAPE) and compared it to different Moving Average Methods (MAM) for independent time series. MAM was more accurate than MAPE or RMAPE when data are from independent time series. Coleman and Swanson (2007) assessed Mean Absolute Percent Error-Rescaled (MAPE-R) and published that it overcame the shortcomings of MAPE as well as met the National Research Council’s major criteria as a summary measure of accuracy. These forecasting methods assessments are all viable but invite a question about the way the problem situation context relates to the accuracy of the model and suggest that PFE, a model with this capability, can atone for the shortcoming of other forecasting models. BACKGROUND Klimberg and Ratick (2000) introduced a new forecasting performance measurement called the percentage forecasting error (PFE): PFE
2 * se *100% ˆ Y t 1
ˆ is the forecasted value for the next time period, t+1 where se is the standard error and Y t 1
(Lawrence, K., Klimberg and Lawrence, S., 2008). More recently, Klimberg, Sillup, Boyle and Tavva (2010) reemphasized the practical benefit of PFE.
Motivation for the development of this new forecasting performance measurement is that the current set of forecasting performance measurements, e.g., MAPE, RMSE, MSE and MAD, although, in general the smaller the better, do not provide any problem situation context as to the accuracy of the model. That is, a MAD (or any other measure used), of 10 provides no understanding about how accurate the forecast could be, except that it is better than 20. Conversely, the PFE estimates with a high level of certainty, (95%), the percentage error of the 4
next period forecast. Klimberg, Sillup, Boyle and Tavva (2010) ran numerous simulations using various statistical distributions and component variations and expected that the forecasting performance measurements should follow the magnitude of the coefficient of variation (CV). The PFE (and MAPE) values consistently followed the expected patterns. The purpose of this paper is to test the accuracy of the PFE in estimating the percentage error.
EXPERIMENT The data source to conduct this experiment is from the M3-Competition (Makridakis and Hibon, 2000). The M-Competitions have been used as an opportunity to study the accuracy of various time series forecasting methods. M3-Competition data contain 3003 time series data sets from various business and economic categories and time intervals. Thirty (30) data sets were “semi” randomly selected from each time interval and category as shown in Table 1. “Semi” randomly selected means that attempts were made to maximize the number of data sets with the same number of observations within a particular category/time interval. As shown in Table 1, 11 of the 15 category/time intervals had all the data sets of the same size, e.g. the 30 Macro/Quarterly data sets each had 52 observations. On the other hand, for the 30 Finance/Quarterly data sets, three (3) had 35 observations, 16 had 38 observations and 11 had 71 observations. Table 1 Sample Size for Each Category/time Interval: Sample Size (Number of Samples) Time Interval
Category Type of Time Series Data Micro
Industry
Yearly
20
47
Quarterly
44
Monthly
69
Macro 23
Finance
Demographic
32(7); 35(3); 47(20)
25
59(10); 64(14); 52 68(6)
35(3); 38(16); 71(11)
44(10); 48(12); 70(8)
144
134
135
134
For each of 450 data sets (5 categories x 3 time intervals x 30), the last time period observation was withheld and Crystal Ball’s Predictor was used to find the best forecasting model (best based 5
on the root mean square error) to predict this last period. Based upon the forecasted value, the PFE was calculated for each data set and compared it to the actual value. Overall 91%, or 409 of the 450 data sets, the percentage error: Percentage error
Actual value - Forecasted value Forecasted Value
was less than the PFE, i.e., 91% of the time the percentage error was less than what was predicted by the PFE and shown in Table 2. Table 2 Percentage Errors Less than PFE Category Type of Time Series Data Time Interval
Micro
Industry
Macro
Finance
Demographic
Yearly
0.90
0.93
0.97
0.70
0.87
Quarterly
0.90
1.00
0.83
0.97
0.90
Monthly
0.93
0.87
1.00
0.97
0.90
If the percentage error is less than the PFE, then the forecast is less than two (2) standard deviates away from the actual value. The spread of errors of the 409 data sets were examined further with a percentage error less than the PFE as depicted in Table 3 where it can be observed that:
288 of the data sets (70% of the 409 or 60% of 450) were less than a .10 standard deviate away from the forecasted value
another 100 data sets were between .10 and .60 standard deviates away (for a total of 388 less than .6 or 95% of the 409 or 86% of the 450)
only 5% of the data sets (21) were more than .6 standard deviates away
6
Table 3 Distribution of Percentage Errors of 409 Data Sets # of std deviates 0.01 0.05 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00 1.50 2.00
# of series 87 132 69 55 17 17 8 3 7 3 2 3 4 2
% 0.21 0.32 0.17 0.13 0.04 0.04 0.02 0.01 0.02 0.01 0.00 0.01 0.01 0.00
cum. % 0.21 0.54 0.70 0.84 0.88 0.92 0.94 0.95 0.97 0.97 0.98 0.99 1.00 1.00
Note that nearly all of the forecasts for these 409 were less than one (1) standard deviate away from the actual value. On the other hand, 41 of the 450 data sets (9%) had a percentage error greater than the PFE or more than two standard deviates away. To measure the magnitude of these 41 data sets whose forecast was more than two (2) standard deviates away we calculated the percent difference: Percent Difference Actual % error - PFE
Table 4 lists the percent difference for each of these 41 data sets. Of these 41 data sets, 35 have a percent difference less than 10% (that is 85% of the 41 data sets or 7.8% of the 450), while only six (6) data sets had percent differences greater than 10% (15% of the 41 or 1% of the 450). One data set has a percent difference as great as 80%.
7
Table 4 Percent Difference for the 41 Data Sets
Yearly Micro 0.01 0.01 0.05
Industry Macro 0.00 0.00 0.04
Quarterly Micro Industry Macro 0.03 0.00 0.03 0.00 0.03 0.00 0.00 0.01 Monthly Micro Industry Macro 0.01 0.01 0.05 0.02 0.04 0.14
Finance Demographic 0.00 0.00 0.01 0.01 0.08 0.01 0.10 0.02 0.10 0.12 0.49 0.53 0.80
Finance Demographic 0.08 0.02 0.10 0.12
Finance Demographic 0.02 0.00 0.00 0.00
DISCUSSION Convincingly, the experiment using the M3-Competition data demonstrated the accuracy of the PFE in estimating percentage error. Virtually, all of the forecasts for 409 M3-Competition were less than one (1) standard deviate away from the actual value with 70 percent of these 409 less than a .10 standard deviate away from the forecasted value. These results provide forecasters a context to establish a confidence to the accuracy of the forecasting model to the related problem situation. 8
The results also help forecasters understand that key assumptions about the past, which undergird many forecast techniques, do not equal assumptions that are appropriate for future forecasts. In dynamic business circumstances, such as a branded pharmaceutical losing its exclusive patent status, the accuracy of managers’ qualitative judgment can be improved by the perspective PFE enables. At a minimum, the results should motivate forecasters to know how the “black box” forecasting methodology they are currently using works to understand the basis of their forecast.
CONCLUSIONS The results suggest that forecasters who have advanced from intuitive guessing to using a quantitative modeling technique should consider PFE as a measure of model accuracy. Furthermore, if a company is vested in the use of a proprietary model and/or quantitative forecasting technique, the results strongly suggest using PFE in parallel with that other method so results can be compared. When an organization is running its business based on its forecast, confidence in the forecast’s accuracy is a clear determinant of the forecaster’s credibility and the organization’s success. REFERENCES Barbour, P.M., Robert, H. and Robb, D.J. (2008). “Assessing Forecasting Model Performance in an ERP Environment.” Industrial Management & Data Systems, Vol. 108, Iss. 5, pp. 677679. Barnett, W.F. (1988). “Four Steps to Forecast Total Market Demand.” Harvard Business Review, Vol. 66, Iss. 4, pp. 28-38. Calciu, M. (2009). “Deterministic and Stochastic Customer Lifetime Value Models Evaluating the Impact of Ignored Heterogeneity in Non-Contractual Contexts.” Journal of Targeting, Measurement and Analysis for Marketing, Vol. 17, Iss. 4, pp. 257-271. Castle, J.L., Fawcett, N.W.P. and Henry, D. (2009). “Nowcasting Is Not Just Contemporaneous Forecasting.” National Institute Economic Review, Vol. 210, Iss. 71, DOI: 10.1177/0027950109354412. Coleman, C.D. and Swanson, D.A. (2007). “On MAPE-R as a Measure of Cross-Sectional Estimation and Forecast Accuracy.” Journal of Economic and Social Measurement, Vol. 32, pp. 219-233. 9
Gallucci, J.A. (2007). “How to Improve Forecasts with Hybrid Functions.” The Journal of Business Forecasting, Fall, pp. 14-17. Kapetanios, G. and Yates, T. (2010). “Estimating Time Variation in Measurement Error from Data Revisions: An Application to Backcasting and Forecasting Models.” Journal of Applied Econometrics, Vol. 25, pp. 869-893. Katti, R. (2008). “Some Observations on the Measurement of Forecast Error and Accuracy.” The Journal of Business Forecasting, Summer, pp. 33-35. Kerkkanen, A., Korpela, K. and Huiskonen, J. (2009). “Demand Forecasting Errors in Industrial Context; Measurement and Impacts.” Journal of Production Economics, Vol. 118, Iss. 1, pp. 43-48. Kiely, D. (2004). “The State of Pharmaceutical Industry Supply Planning and Demand Forecasting.” Journal of Business Forecasting Methods & Systems, Vol. 23, Iss. 3, pp. 2022. Klimberg, R.K. and Ratick, S. (2000). “A New Measure of Relative Forecast Error,” INFORMS Fall Meeting, San Antonio, Nov. Klimberg, R.K., Sillup, G., Boyle, K. and V. Tavva, “Forecasting Performance Measures—What Are Their Practical Meaning?”, in Lawrence, K. and Klimberg, R. (eds). Advances in Business and Management Forecasting, Vol. 7, Emerald Group Publishing Limited, 2010, pp. 137-148. Lawrence, K., Klimberg, R., and S. Lawrence (2008), Fundamentals of Forecasting Using Excel, Industrial Press, Nov. Makridakis, S. and M. Hibon (2000), “The M3-Competition: Results, Conclusions and Implications”, International Journal of Forecasting, Vol. 16, pp. 451-476. Phillips, K.R. and Lopez, J. (2009). “An Evaluation of real-time Forecasting Performance across 10 Western U.S. States.” Journal of Economic and Social Measurement, Vol. 34, pp. 119 -132. Pilinkieu, V. (2008). “Market Demand Forecasting Models and their Elements in the Context of Competitive Market.” Engineering Economics, Vol. 60, Iss. 5, pp. 24-31. Radovilsky, Z. (2008). “Improving Accuracy and Automating Forecasting with Spreadsheets.” International Journal of business Strategy, Vol. 8. No. 2, pp. 142. Rahmlow, H. and Klimberg, R. (2002). “Forecasting Practices of MBA’s.” Advances in Business and Management Forecasting, Vol. 3, pp. 113-123. 10
Reiner, G., Natter, M. and Dreschler, W. (2009). “Life Cycle Profit – Reducing Supply Risks by Integrated Demand Management.” Technology Analysis & Strategic Management, Vol. 21, Iss. 5, pp. 653-664. Ren, L and Glasure, Y. (2009). “Applicability of the revised Mean Absolute Percentage Errors (MAPE) Approach to Some Popular Normal and Non-Normal Independent Time Series.” International Advances in Economic Research, Vol. 15, pp. 409-420. Sichel, B. (2009). “Forecasting Demand with Point of Sales Data-A Case Study of Fashion Products.” Journal of Business Forecasting, Vol. 27, Iss. 4. Pp. 15-16. Syntetos, A.A. (2010). “Judging the Judges through Accuracy-Implication metrics: The Case of Inventory Forecasting.” International Journal of Forecasting, Vol. 26. Iss.1, pp. 134-143. USA Petrochemicals (2011). “BMI Technology.” USA Petrochemicals Report, Q2, pp. 87-91. Wallstrom, P., and Segerstedt, A. (2010). “Evaluation of Forecasting Error Measurements and Techniques for Intermittent Demand.” Journal of Production Economics, Vol. 128, Iss. 2, pp. 625-636. Yellan, P., Kim, S. and Stratulate, R. (2010). “A Bayesian Model for Sales Forecasting at Sun Microsystems.” Interfaces, Vol. 40, Iss. 2, pp. 118-129.
11
