Institut Europ6en d'Administration des Affaires, (INSEAD). France. (Received May 1980; in revised forra October 1980). Statistical forecasting is based on the ...
O M E G A The Int. JI of Mgmt Sci. Vol. 9. No. 3, pp. 307 to 31 I. 1981
0305-0483/81/030307-05502.00/0 Copyright © 1981 Pergamon Press Lid
Printed in Great Britain. All rights reserved
Forecasting Accuracy and the Assumption of Constancy SPYROS MAKRIDAKIS Institut Europ6en d'Administration des Affaires, (INSEAD). France
(Received May 1980; in revised forra October 1980) Statistical forecasting is based on the assumption of eoastancy, or structural ability in the data. This paper argues that such an assumption might not always be realistic in real life forecasting situations. Unfortunately, however, statisticians and forecasters tend to forget, or at least not make explicit this important assumption, thus creating false expectations which cannot be realistically fulfilled. The paper proposes a wider role for forecasting and makes specific suggestions to overcome the problems arising when the assumption of constancy does not hold.
INTRODUCTION STATISTICAL F O R E C A S T I N G is based on a misleading premise: the assumption of constancy. This assumption of constancy of patterns and/or relationships, or structural stability in the data, is either ignored outright, brushed aside as unimportant, overlooked as useless, or at least is not made explicit. The practical implications are that false expectations arise which cannot be fulfilled. The accuracy of statistical forecasting cannot, after all, exceed the informational content of the data. The assumption of constancy may adequately describe data from physical, natural, and most engineering related applications, but it fails to capture the essence of business and economic data which changes continually and is inherently unstable. Thus, approaches which are successful in the hard sciences cannot be automatically transplanted to those that are social. Furthermore, forecasts made in natural, physical or engineering science can rarely influence outcomes. This is not the case, however, with most business and economic forecasts where self-fulfilling prophecies and selfdefeating prognoses are the rules rather than the exceptions; people are quite capable of influencing the future course of events. 307
The main influence on present attitudes and expectations is that statistical forecasting was applied on a large scale during the 1960's, which was an exceedingly stable period where conditions resembled the data stability found in hard sciences. The "turbulent' 1970's, however, brought about innumerable changes, discontinuities and unforeseen events, causing the accuracy of statistical forecasting to weaken during this period. The following is a quotation by a chief executive: "The unforseen event will occur again, just as the oil embargo of 1973 and the worldwide recession of 1975... And no econometric model or economist has the power to accurately forecast such uncertainties.., despite our use of increasingly sophisticated tools, it's still a lot like rolling dice." I-3, p. 38].
In the opinion of this. author, certain fundamental changes need to be made in statistical forecasting, in order that it becomes relevant and applicable for business and economic uses. Central to these changes is the fact that data need not be structurally stable, and that forecasting accuracy must be mainly concerned with future, post-sample, predictions, rather than with what has already happened--that is, the accuracy of fitting a model to existing data. Finally, forecasting should not be judged on the simple accuracy criterion but its role
308
Makridakis--Forecastin# Accuracy and the Assumption of Constancy
should be enlarged and be concerned with its ability to improve decision making within organizations.
THE NEED AND LIMITATIONS OF STATISTICAL FORECASTING Today there is little doubt that there are many disappointed and frustrated forecasting users, and others who have discarded the use of formal forecasting methods as irrelevant [4]. At the same time the forecasting industry is flourishing and high interest is maintained, as shown by the number of recent books (see [7]) and articles published, the number of people attending conferences, and the opportunities for consulting in business and government. In stable economic and environmental conditions, forecasting involves continuations of established patterns/relationships in which case forecasting is usually accurate. In turbulent environments, as during the 1970's, forecasting errors can be serious [8]. How could high inflation rates have been predicted during the 1974/75 recession if such a phenomenon had n e v e r happened before? Moreover, how could the major 1974/75 recession have been foreseen on economic grounds when a major cause of it was political, the result of the Arab-Israeli War and the oil embargo? Models are not expected to possess prophetic powers. Paradoxically, there is little interest in forecasting when stable conditions prevail and high interest when changes are frequent and uncertainty great. But this is precisely when forecasting accuracy is at its lowest level, inevitably inducing dissatisfaction. Unfortunately, however, forecasters have not only failed to communicate this point to users but also to make explicit the alternative to statistical forecasting. All empirical evidence shows that human forecasters do not necessarily produce more accurate results than models (see below). Today, with interest in this field growing rapidly, forecasting is plagued by many problems. The challenge, however, is not to attempt to blame who or what has been wrong but rather to enlarge the role of forecasting in order to be capable of dealing with real life data which often involves non random changes from established patterns/relationships.
A WIDER ROLE OF FORECASTING The traditional role of statistical forecasting has been 'extrapolative'. That is, some model based on past data has been developed, which subsequently has been used to 'project' the past patterns/relationships beyond the sample data, thus, providing forecasts for the future. This usually works well as long as the established patterns/relationships do not change. However, if changes do occur, statistical forecasting cannot deal with this situation because the assumption of constancy will not hold. The resulting errors do not have to follow previous patterns: they can be non-random, their variance can be wider, and/or they can be nonsymmetric. If an enlarged role of forecasting is accepted (see Fig. 1) then the problem becomes how to continue forecasting when systematic changes from established patterns/relationships are involved. Unfortunately, from a pure statistical point of view, very little can be expected. Systematic changes from established patterns/relationships (e.g. oil embargo) are non-repetitive, or if they are (e.g. recessions) the length between two successive occurrences is not necessarily constant. Furthermore, even when repetitive events are involved, the effect of change cannot always be quantified because of the multitude of additional factors involved making the isolation and measurement of a single factor's influence impossible. Forecasters and end users must accept the inevitable, this being the inability to forecast statistically when the assumption of constancy does not hold. If this is accepted, there is but one choice: to devise ways of knowing as soon as possible when systematic changes from past patterns/relationships are taking place. This is referred to as monitoring and should be integrated into any forecasting system. If monitoring indicates some systematic change and the errors become non-random, this is when action must be taken in the form of adjustments to the quantitative forecasts. The need for adjustment, however, requires a knowledge, an understanding of how things relate and influence each other. Without such an understanding no meaningful adjustment is possible when systematic changes from established patterns/relationships occur.
Orneya, Vol. 9, No. 3
309
FORECASTING i_ Understanding
I~
J
Monitoring ]~
J Extrapolative predictions I
_I Strategic 1 r j consideration _I Planning and ~J other objectives
I
1
J
I
FIG. 1. Major functions in forecasting.
The dichotomy of forecasting continuations versus systematic changes in established patterns/relationships is critical on several grounds. Firstly, there is ample empirical evidence suggesting that where predictive accuracy is the issue, statistical models are superior to human judgement. Interestingly, it is not statisticians but psychologists who have been researching this area and advocating such a view. The conclusions are indisputable in the light of empirical evidence, available both in the psychological and forecasting literatures. The human predictive ability has been found to be inferior to that of models in all cases, with one exception. However, both this case and a subsequent one by Libby [5] have been disputed by Goldberg [2] who reversed the exceptional findings by simply transforming the data. In a review article, Dawes [1] states that he knows of no other findings that have been reported in the literature showing the superiority of clinical judgement over a quantitative model. Even though it is emotionally difficult to accept the idea that a model--and in some cases a model of the decision maker himself-can outperform the decision maker, it is obvious that the consequences of this evidence should be considered. At the very least, those involved in forecasting must be made aware of these findings and the consequences involved (see Hogarth and Makridakis [4]). Complaining, for instance, about the pure predictive ability of statistical forecasting makes little sense when the alternative, i.e. human judgement, can be even worse and is definitely more costly. On the other hand, stat-
istical models are not adequate when systematic changes from established patterns/relationships occur. In this case, the only alternative is adjustments by humans who must, however, understand the nature of the system to which the forecasts refer. A procedure in order to avoid inconsistencies and to initiate prompt and effective action is needed. More important is to understand the various factors affecting forecasting and the role of uncertainty while predicting the future.
Selecting appropriate forecasting methods Table 1 is an attempt to classify the major forecasting methods presently available in terms of the four forecasting functions shown in Fig. 1. It is important that a balanced portfolio of methods be employed, as some methods are very effective in extrapolative forecasting but inadequate for monitoring purposes, or for understanding. Furthermore, there is the question of cost and effort needed to utilize each method, which varies considerably and which must be taken into account. For instance, methods which are cheap and easy to use are the only viable alternatives when large numbers of frequently forecasted items are involved. For important series/products more personalized attention and possibly the use of more than one forecasting method should be contemplated.
Forecasting accuracy There must be a significant change of attitudes concerning the accuracy of forecasts.
310
Makridakis--Forecastin# Accuracy and the Assumption of Constancy TABLE 1. MAJOR TASKS OF FORECASTING IN GENERAL AND CORRESPONDING METHODS M a j o r forecasting t a s k s Forecasting methods
i[-
Decomposition methods Smoothing methods Trend extrapolations Leading indicators Naive approaches A R M A m e t h o d s a n d filters
Forecasting
Monitoring
Understanding
* ***** *****
***** ****
*
** *****
Adjustments in forecasting
***** *****
M u l t i p l e regression Econometric models Multivariate ARMA
** ** **
***** ***** *****
M u l t i p l e regression Econometric models I n p u t / o u t p u t tables
** ***** *****
* * *
A n t i c i p a t o r y surveys Juries of executive o p i n i o n s Technological methods: exploratory Technological methods: normative Sales force e s t i m a t e s
* ** ***** * *****
tlJ
Firstly, accuracy must be concerned with postsampling periods and how well a statistical model can do in the future, beyond the available data. Unfortunately, most of the models are presently concerned with the past and the ability of a model to be fitted to existing data. Secondly, whatever the accuracy, or inaccuracy, of statistical forecasting, a comparison should be made to other alternatives, notably informal, subjective predictions, before any judgements about its usefulness and effectiveness are made. Thirdly, sophisticated statistical methods should be always compared with naive approaches (e.g. random walk models, equal weighting schemes, or simply deseasonalizing the data) to know if the former is an improvement on the latter and by how much. This is the only way to decide on the cost/benefit o f simple versus sophisticated models. Fourthly, there is a great deal to be done if increases in accuracy are _the__major objective without having to necessarily apply more sophisticated methods. Cleaning the data, for instance, of outlayers, adjusting .the data for working or trading days, allocating large orders to the periods where they apply, modifying the data for accounting or organizational changes, and above all being consistent on what goes into the data can result in as high an improvement in accuracy as could be
*****
*****
the result of using more sophisticated methodologies.
Understanding forecasting errors Forecasting errors will inevitably occur in any prediction of the future but their existence must be understood. Furthermore, it means learning, on the part of the forecasting user-with the consequence of forthcoming improvements in accuracy. Forecasting in this respect, becomes a kind of a normative course to be followed, whose every deviation is measured and reasoned about. Under such an approach, errors become as important in predicting the future as the initial forecasts. They provide invaluable information as to where the organization is presently and how the direction it has been following is changing. In the final analysis, it is the analysis of the errors which determines systematic changes in past patterns/relationships.
Understanding and accepting uncertainty Forecasting errors fluctuate in a manner that can be adequately postulated, if continuation of existing patterns/relationships is assumed. Consequently, the extent of uncertainty can be measured and incorporated in decision making. Apart from the short term where inventory models are used to deal with such uncertainty not much is done to include uncertainty in
Omega, Vol. 9, No. 3
actual planning applications. Being prepared to face uncertainty involves costs which are not always well understood. There is always the possibility of never being caught without adequate supplies, for instance, but this would mean keeping higher inventories which cost money to maintain, and requires tying down considerable financial resources. One must therefore: (a) establish the criteria of the cost and benefits of various alternatives of under or over-estimation for continuations of existing patterns/relationships, and (b) be capable of quick reactions once it has been decided that a systematic change has taken place in a consistent and systematized manner. In the final analysis, however, there is no way of eliminating uncertainty which is a fact forecasting users have to accept. The only thing to be done is to understand it and be able to consider the implications involved.
Bringing closer the user and the preparer of forecasts It has been argued that the biggest advantage of formal forecasting methods is not the predictions themselves but, more important, the process of arriving at them, the way they should be interpreted and how they should be used. This necessitates the direct involvement of the forecasting users to the technical aspects of forecasting. They should become proficient in using the methods, grasping their advantages/limitations, being able to understand and explain the existence of the various forecasting errors, and the extent and nature of uncertainties involved. The present separation of preparers and users of forecasting has been a major obstacle in making forecasting more relevant and applicable to actual organizations. It is true that no planner, decision, or policy maker will utilize a set of forecasts that he or she has not been personally involved in developing, especially regarding important decisions. The only solution, therefore, is to inw)lve the forecasting users directly in the process of forecasting, even if this means much simpler and thus, easily understood methods.
311
CONCLUSIONS It has been advocated in this paper that a change in attitude is necessary to make forecasting more relevant and applicable. This is centred around the fact that forecasting continuations of established patterns/relationships is completely different to forecasting systematic changes from such patterns/relationships. It has therefore been suggested that monitoring methods be used so that systematic changes could be infered judgementally, or signalled automatically. Furthermore, it was advocated that an understanding of the forecasting environment is needed to be able to make adjustments when systematic changes are occurring. Finally, several suggestions were made as to how forecasting emphasis will have to be shifted from being simply concerned with accuracy to determining the reasons and causes behind forecasting errors, the need to examine the various forecasting alternatives available, and their cost and benefits, and the direct involvement of forecasting users in preparing the forecasts.
REFERENCES I. DAWES RM (1977) In Shallow Psychology (Eds CARROL JS & PAYNE JW). Erlbaum, Hillsdale, New Jersey, USA. 2. GOLDBERG LR 0976) Man versus model of man: jus! how conflicting is that evidence? Orgnl Behav. & Hum. Perf. 16, 13-22. 3. HALL W K (1971) Changing perspective of the capital investment process. Long Range Plann. 29-34. 4. HOGARTH RM & MAKRIDAKIS S (1981) Forecasting and planning: an assessment and some suggestions. Mgmt Sci. F e b r u a r y . 5. LIaaV R (1976) Man versus model of man: some conflicting evidence. Orgnl Behav. & Hum. Perf. 16, 1-12. 6. MAKRIDAKISS (1981) Forecasting, Planning and Strategy: Some New Directions. Long Range Plann. To be published. 7. MAKRIDAKISS & WHEELWRIGHT S (Eds) (1979) Forecasting. TIMS Studies in Management Science, 12. 8. MCNEES SK (1979) Lessons from the track record of macroeconomic forecasts in the 1970's. In Forecasting (Eds MAKRIDAKIS S & WHEELWRIGHT S). TIMS Studies in Management Science, 12. ADDRESS FOR CORRESPONDENCE: Professor Spyros Makri-
dakis, European Institute of Business Administration, Boulevard de Constance, F-77305 Fontainebleau Cedex, France.
