Expert Systems with Applications PERGAMON
Expert Systems with Applications 21 (2001) 31±36
www.elsevier.com/locate/eswa
Neural networks as a tool for developing and validating business heuristics Steven Walczak* University of Colorado at Denver, College of Business and Administration, Campus Box 165, PO Box 173364, Denver, CO 80217-3364, USA
Abstract The increasing availability of information via the Internet and world-wide-web has made business decision-making more complex. Heuristic rules and methods are frequently used in complex domains to facilitate the decision-making process by reducing the amount of information that is required for decision-making. Heuristic rules may `go out of date' as new information and new business methods are developed. Neural networks provide a fast and ef®cient means for evaluating the utility of existing heuristics. Two case studies are presented that demonstrate the use of neural networks for developing new heuristic rules or for refuting existing heuristic rules. Validation or adaptation of non-valid heuristics improves the quality of resulting business decisions. q 2001 Published by Elsevier Science Ltd. Keywords: Neural networks; Heuristics; Model selection; Empirical evaluation
1. Introduction As the world-wide-web continues to grow and more and more information is made available to managers, the decision-making process is becoming more complex. Solutions to business problems, especially in complex non-linear domains, are often approximated through the use of heuristics (Barr & Feigenbaum, 1981). Heuristic methods serve to reduce the quantity of information that must be evaluated in order to reach a near-optimal solution. Although heuristic decision-making methods do not guarantee an optimal solution, they do approximate the optimal solution and the tradeoff between optimal and near-optimal solutions is often desirable due to the time and information cost savings afforded through the use of heuristics. Since heuristics are typically based on induction, two problems face business managers utilizing heuristic decision strategies. First is the validity of the heuristic. Induction uses empirical evidence to formulate the heuristic rules. The exponential growth of the availability of information implies that new evidence may refute a heuristic. However, selecting the right information from the mountain of available data is problematic. Second, is the possibility that even though the heuristic may have been valid at one point, the dynamic state of our global economy may produce needed changes to the heuristic methods (e.g., changing from a hierarchical management structure to a matrix organization). Arti®cial intelligence has provided a number of meth* Tel.: 11-303-556-6777. E-mail address:
[email protected] (S. Walczak). 0957-4174/01/$ - see front matter q 2001 Published by Elsevier Science Ltd. PII: S 0957-417 4(01)00024-0
odologies for addressing complex problems and utilizing heuristic methods to produce solutions to these problems. Among these methodologies are: neural networks, expert systems, genetic programming, fuzzy systems, and others (e.g., arti®cial life). Current rule-based expert systems may implement hundreds or even thousands of heuristic rules. Validation of expert systems is a problematic issue and requires signi®cant development time to perform proper validation (Medsker & Liebowitz, 1994). Additionally, expert systems tend to be static in their use of heuristics and therefore do not facilitate the adaptation or refutation of current heuristics. Neural networks and genetic programming however, utilize automatic system adaptation (learning) to new empirical values and hence can promote the evaluation and adaptation of heuristic decision rules. This article discusses the utility of using neural networks for evaluating business decision heuristics. A very brief background on neural networks and the model selection research methodology is presented. Next, the means for using neural networks for developing and validating existing or new heuristics is described. Then, two case studies are presented that demonstrate how neural networks may be used to evaluate existing heuristics and determine the validity or appropriateness of new heuristics. Finally, the case studies are generalized to provide a methodology for evaluating heuristic decision rules with neural networks. 2. Background Neural networks and in particular multi-layer perceptrons have been shown to be universal approximators (Hornik,
32
S. Walczak / Expert Systems with Applications 21 (2001) 31±36
1991; Hornik, Stinchcombe & White, 1989; White, 1990). This means that neural networks can learn arbitrarily complex (non-linear) mappings between input and output variables making them an ideal candidate for evaluating heuristic rule effectiveness in complex domains. Neural networks are a non-parametric modeling technique. This means that the input data are not required to ®t a speci®c distribution and relationships and correlations between variables do not have to be pre-speci®ed, unlike many traditional statistical modeling techniques (e.g., linear and logistic regression or discriminant analysis) that require speci®c distributions for valid application of the statistical modeling technique. Additionally, neural networks are able to handle noise in both the training (model-building) set and test sets (Fausett, 1994). Various research results have demonstrated that neural networks consistently outperform commonly used parametric statistical models for a wide variety of domains (Bansal, Kauffman & Weitz, 1993; Dybowski & Gant, 1995; Piramuthu, Shaw & Gentry, 1994). How can the validity of heuristics be measured by neural networks? Swanson and White (1995) propose the use of a model selection perspective as an alternative to standard hypothesis testing. Model selection uses out-of-sample prediction accuracy as the measurement of model validity. Heuristics use one or more assumptions about the domain state. The model selection approach implies that the component elements of a heuristic rule may be tested with neural networks by evaluating relative change in categorization or forecasting performance (the output). Details on the method for accomplishing the heuristic variable evaluations are given in the next section. Because basic descriptions of neural networks and their associated learning methods have been widely published, such a description is omitted here, but interested readers are referred to (Fausett, 1994; Hornik, 1991; Hornik et al., 1989; Medsker & Liebowitz, 1994; Smith, 1993; White, 1990). 3. Neural network evaluation of heuristics (method) Heuristics are dependent on the domain state at the time of decision, which means that the values of state variables effect the outcome of the decision heuristic. The premise of the research in this article is that many business decision heuristics exist that contain either too many domain state variables corresponding to an overly constrained domain view or contain too few domain state variables producing an overly relaxed domain view. Evaluation of existing heuristics and modi®cation or development of new heuristics requires that multiple neural network models be developed, where each model is different in the set of input variables used to de®ne the problem. Alternative heuristic validation methods may also be attempted by varying other external variables, such as the quantity of data used to produce the neural network model. Input variable selection is a critical factor for the success of
neural network models (Pakath & Zaveri, 1995; Smith, 1993; Tahai, Walczak & Rigsby, 1998; Walczak and Cerpa, 1999) and is the key to evaluating heuristic models with neural networks. The importance of input variable selection is highlighted by Nath, Rajagopalan and Ryker (1997) who attempt to calculate the saliency of neural network input variables. Neural network shell tools, such as Aspen Technology's Professional II Plusw and also their Predictw tool have made advances in addressing the importance of identifying relevant input variables by providing functions that calculate a variable's contribution to the neural network's output value. These functional techniques for attempting to identify variable effect can assist researchers in proposing new heuristic models to be evaluated against existing heuristics. The basic model for a heuristic decision rule is pre-speci®ed and so the primary neural network model to evaluate heuristic effectiveness is pre-de®ned with respect to the input variables. Additions or deletions from the input variable set will produce alternative heuristics for evaluation using the model selection approach. The multiple neural network models, each of which has the same hidden node architecture and the same learning coef®cient and momentum values, are trained for an identical quantity of training epochs and the corresponding outputs of the neural networks are compared. Since the only difference in the neural network models of the heuristic decision rule and the alternate rule is in the presence or absence of variable values, any difference in the output of the various neural network models is directly attributable to the variables (or lack thereof). If the modi®ed heuristic neural network outperforms the original heuristic model, a few additional tests on different data sets (possibly using bootstrapping or jack-kni®ng depending on data availability) should be performed. The additional test will increase the con®dence interval on the new heuristic, since the original heuristic did not guarantee an optimal solution merely an approximation to the optimal solution. If a new heuristic rule fails to outperform the original heuristic, then this lends empirical support for the validity of the original heuristic. Additional derivations of the input variable set may be tried as alternative heuristics. Walczak and Cerpa (1999) claim that traditional knowledge acquisition with domain experts is the preferred method for attempting to identify alternative heuristic decision rules. 4. Two cases of neural network heuristic evaluation The methodology for using neural networks to evaluate the quality of competing heuristics described in the previous section is shown in Fig. 1. In this section, two cases are used to demonstrate the depicted method. After the best performing heuristic is determined, a brief discussion of the implication of using the selected heuristic is presented.
S. Walczak / Expert Systems with Applications 21 (2001) 31±36
33
Fig. 1. Neural network methodology for evaluating business heuristics.
4.1. Invalidating an existing heuristic The ®rst case deals with heuristic methods for determining hospital resource allocation problems, speci®cally bed utilization within the hospital. Cost effective patient management is dependent on accurate assessment of individual patient outcome and resource utilization (Buchman, Kubos, Seidler & Siegforth, 1994). Evaluating length of stay information is a challenging task (Weissman, 1997), but is essential for the operational success of a hospital. Pofahl, Walczak, Rhone and Izenberg (1997) report that two different heuristic methods are currently in use in the medical ®eld for determining outcomes for acute pancreatitis patients, the Apache II score and the Ranson score. A comparison of neural networks against the two other heuristic methods indicated that neural networks were better at identifying the length of stay for critically ill acute pancreatitis patients (staying . 7 days), but performed slightly worse when evaluating less ill acute pancreatitis patients (staying # 7 days). The question of whether the Ranson score (Ranson, 1982) is in fact an effective heuristic for predicting the length of stay and correspondingly the illness severity of acute pancreatitis patients may be evaluated using neural networks. Four different neural network models are designed in consultation with a pancreatitis domain expert. Two of the neural network models, one with 44 input variables and one with 23 input variables, contained the abbreviated Ranson score as an input value. The other two neural network models used the same variables, but excluded the Ranson score value, producing input vectors of 43 and 22 variables respectively. All four networks were trained on the same data set with regard to the patients and for an identical number of iterations.
The 22 variable input set (without the Ranson score) produced a classi®cation speci®city that was 5% higher than the 23 variable set. The 43 variable input set (also without the Ranson score) produced a classi®cation speci®city that was 2% higher than the 44 variable set. Each of the Ranson-less networks also produced a smaller mean difference error for the total length of stay for each patient. The two different size networks, 22 and 43 variables, provide the two or more trials needed in Fig. 1 to cause the rejection of the original Ranson-based heuristic rule. These results indicate that the Ranson score, when used in combination with the other selected variables, reduces the overall accuracy of the prediction heuristic, which implies that the Ranson score is not the best heuristic measure of pancreatitis patient outcome. For the reported research, a difference of 2±5% equates to accurate resource predictions for an additional two to ®ve patients (out of every 100 patients) above the traditional heuristic method. This knowledge will enable hospital administrators as well as physicians to better manage critical hospital resources, subsequently reducing the hospital's costs and subsequent charges to the patient. Furthermore, accurate identi®cation of the illness severity promotes higher quality care for the patients by identifying patients that may require additional tests and more intensive care, thus improving the patient's quality of life. Future research will examine other heuristic medical predictors of injury criticality, such as the Apache II, Glasgow Comma Score, and Pediatric Trauma Score. 4.2. Choosing between heuristics The second case is concerned with selecting between two different heuristic methods for determining trading positions
34
S. Walczak / Expert Systems with Applications 21 (2001) 31±36
for ®nancial time series, speci®cally currency exchange rates but this may also be extended to other ®nancial instruments. Two different heuristic methods are currently used for evaluating ®nancial time series. Technical analysis is concerned with the ®nancial time series itself and values directly derivable from the time series (e.g. trends and averages), while fundamental analysis incorporates information external to the time series. The majority of neural network research models of ®nancial time series currently use the technical analysis heuristic (Hu, Zhang, Jiang & Patuwo, 1999; Tahai et al., 1998; Walczak, 2000). Two different neural network models are developed with one corresponding to a technical analysis and one corresponding to a fundamental analysis. Differences between the two model types are minimized. The technical analysis neural network uses multiple lags of the time series itself and the fundamental analysis neural network includes the same set of multiple lags used in the technical analysis model and adds the rate of return on international bonds for the same time lags (Walczak, 1995). The rate of return on international bonds attempts to capture differences in interest rates, which is thought to in¯uence currency exchange rates (Mishkin, 1992). A 12% difference in the forecasting accuracy rates occurs between the technical neural network model, performing close to 54% forecasting accuracy (Walczak, 1995), and the fundamental model, performing at 66.7% forecasting accuracy (Walczak, Tahai & Khondar, 1998). Similar results where the fundamental heuristic neural network model outperformed a technical heuristic neural network model occurs for three different currency exchange models: the dollar/pound sterling, dollar/mark, and dollar/yen. The three different currency models provide the multiple tests speci®ed in Fig. 1 for acceptance of a new heuristic, even though neither of the two competing heuristics is essentially new. A further evaluation is performed by simulating currency trades predicted using the two neural network heuristic models. The simulation assumes that suf®cient cash remains in the account to permit one contract purchase each day (if indicated by the neural network heuristic model). The net difference in pro®t between the two accounts is almost $1969.00 in favor of the fundamental analysis neural network heuristic model. Higher pro®ts result from higher quality heuristic decisions using the heuristic model determined with the neural network analysis of the different heuristics. 4.3. Evaluating `new' heuristics An additional analysis of the use of interest rates as a `better' heuristic for performing exchange rate forecasting is accomplished by increasing the input variable set to include the S&P 500 index and CRSP index, which are not commonly thought to signi®cantly effect currency exchange rates. This `new' heuristic model attempts to
test a general heuristic: that more series-external information will provide a better solution. The second fundamental heuristic neural network model, with the additional index variables, had an average performance of 51% forecasting accuracy. The 51% forecasting accuracy is statistically similar to the heuristic technical analysis neural network model, but signi®cantly worse than the expert proscribed heuristic fundamental analysis neural network model. The above results indicate that for non-linear modeling of neural networks, a fundamental analysis approach will outperform a technical analysis approach in forecasting currency exchange rates. The non-parametric and non-linear nature of neural networks provides a facile tool for rapidly evaluating heuristic models that may have complex data patterns. Other ®nancial modeling heuristics may be evaluated in a slightly different manner. For example, a current modeling heuristic used in developing business expert systems and ®nancial time series (neural network and otherwise) is: greater quantities of data produce more accurate models. This may be tested similar to the method shown in Fig. 1, but substituting training data set size for heuristic variable selection. Multiple models that only vary in the quantity of data being used to train (build) the model are then evaluated. Current research is testing the data quantity heuristic by investigating the effect of different quantities of data on neural networks for forecasting ®nancial time series (Hu et al., 1999; Walczak, 2000, 2001). Preliminary results have provided empirical evidence that a maximum of 2 years of data will produce the optimal heuristic training set size for developing most ®nancial time series models. The Chicago Mercantile Exchange reports that historic data sets are sold at a cost of $100 per year per commodity. Using two years of data instead of ten would result in an immediate savings of $800 in data costs. Additionally, if smaller data sets can be used, then the neural network training can be accomplished in less time resulting in additional development cost saving. 5. Testing results Neural networks provide an ef®cient method for validating the selection of new heuristics. Unlike expert systems, which necessitate a lengthy re-validation of the entire system and all possible interactions between multiple heuristic rules (Medsker & Liebowitz, 1994), the development process of neural networks involves the construction of training (model building) data sets and test or validation data sets and hence the validation of the heuristic rules is built in to the development process. The model selection research method (Swanson & White, 1995) provides the means for comparing two similar neural networks: one that incorporates a heuristic variable and one that does not utilize the heuristic being evaluated. A direct comparison of the forecasting or categorization performance enables
S. Walczak / Expert Systems with Applications 21 (2001) 31±36
selection of the best heuristic rules amongst those being evaluated. Statistical signi®cance of performance differences between these similar neural network heuristic models may be determined using traditional t-test, Z-test, or R 2 statistical tests depending on the type of data being evaluated. Since the neural network models will all have the same output, dependent variable(s), and also will have highly similar input/independent variables, then each of these statistical test for signi®cance should be applicable. An example of the validation process is the evaluation of technical analysis trading rules versus fundamental analysis trading rules for ®nancial time series modeling that was described in Section 4.2. The input vector for the technical model contains multiple lag (difference between the value of the stock, currency, or other ®nancial time series) values for the US dollar/British pound exchange rate only, while the fundamental model's input vector contains the same currency lag values and also lag values for the rates of return on British international bonds (to gauge the interest rate differences between the two countries). Both the technical analysis and fundamental analysis neural network models are trained on data values from the same time period (and the same values for the currency lags). A baseline for comparison of the two neural network models is established by building ARIMA (auto-regression with moving averages) models on both the technical data and the fundamental data. ARIMA models are a typical trading model developed for ®nancial time series trading. Addition of the ARIMA model is still valid under the model selection research method and will serve to validate the `new' fundamental neural network model against an existing statistical model. The best ARIMA model produced a 1day future forecasting performance (percentage of correct predictions) of 48.6% (indicating that the currency time series may follow a random walk). A random walk model, yt11 xt 1 e, is also developed and produces a forecasting performance of 51.4%, which is statistically similar to the ARIMA model. Both of the neural network models outperformed the ARIMA baseline model, but the purpose of utilizing the multiple neural network models is to evaluate and validate which heuristic trading model performs the best. The technical neural network model produced a 1-day future forecasting performance of 54.86, which using a Z-test is not signi®cantly different from the hypothesized random walk behavior derived from the ARIMA model (Walczak, 1995). However, the fundamental neural network model produced a 66.67 forecasting performance and is statistically signi®cant (also with a Z-test) with a p-value of less than 0.01 (Walczak et al., 1998). As required by the method displayed in Fig. 1, an additional test/evaluation data set is acquired by using values that occur later in time than the original test data set. The reevaluation produces a forecasting performance of 60.1%, which is still signi®cantly different from the other three models with p , 0.01, using a Z-test again. Hence on two
35
different evaluation data sets, the neural network's fundamental (heuristic) model outperforms both the traditional ARIMA model (and random walk model) as well as a technical analysis neural network model. The new heuristic of utilizing a fundamental model for optimal prediction of currency exchange rates is further validated by building similar neural network models to forecast the 1-day future exchange rate of the US dollar/German mark. Again, ARIMA and random walk models are constructed and produce a similar (50%) forecasting performance to the dollar/pound models. The neural network heuristic technical model for the dollar/mark produced a 52.63 forecasting accuracy, while the neural network heuristic fundamental model achieved a 57.14 forecast accuracy. While not quite as signi®cant as the dollar/pound results, the neural network fundamental heuristic model is still statistically signi®cantly better than all three other models at p , 0.05. This process continues by validating the necessary components of the fundamental model: expected change in the time series and interest rate differences (Mishkin, 1992). As discussed brie¯y in Section 4.3, other fundamental variables including various stock indexes are added into the validated neural network fundamental model, but each of the new external values causes the performance of the fundamental neural network model to approach the statistical random walk model. Hence these newer fundamental models are discarded in favor of the original fundamental model. 6. Summary A method for using neural networks to evaluate heuristic business rules has been presented. The technique uses the model selection methodology for determining optimal models. A case study demonstrating that current heuristic rules may no longer be reliable was presented. Another case study showed how neural networks may be used to evaluate competing heuristic decision rules and determine the validity of new heuristic rules. Neural networks provide a fast and reliable tool for performing heuristic rule evaluation and are not constrained by required data distributions or variable dependencies. Identi®cation of unreliable heuristics or the determination of an optimal heuristic between competing heuristics provides several bene®ts. The primary bene®t is economic in that better quality heuristics will either reduce costs or increase revenues. A secondary bene®t is in the quality of the decision-making process. In the presented case that attempted to determine the level of illness being suffered by medical patients, the more accurate heuristic method used in place of the existing heuristic can lead to better quality of care fore the patients and ultimately a better quality of life. The performance of rule-based expert systems is dependent
36
S. Walczak / Expert Systems with Applications 21 (2001) 31±36
on identifying accurate and appropriate heuristics from domain experts. Neural networks provide a means for selecting between competing heuristics that can be acquired when multiple experts are interviewed during the knowledge acquisition process. Rapid identi®cation of the optimal heuristic set will decrease overall development costs and reduce error testing and system validation time. References Bansal, A., Kauffman, R. J., & Weitz, R. R. (1993). Comparing the modeling performance of regression and neural networks as data quality varies: a business value approach. Journal of Management Information Systems, 10 (1), 11±32. Barr, A., & Feigenbaum, E. A. (1981). The handbook of arti®cial intelligence volume 1, Reading, MA: Addison-Wesley. Buchman, T. G., Kubos, K. L., Seidler, A. J., & Siegforth, M. J. (1994). A comparison of statistical and connectionist models for the prediction of chronicity in a surgical intensive care unit. Critical Care Medicine, 22 (5), 750±762. Dybowski, R., & Gant, V. (1995). Arti®cial neural networks in pathology and medical laboratories. The Lancet, 346, 1203±1207. Fausett, L. (1994). Fundamentals of neural networks, Englewood Cliffs, NJ: Prentice Hall. Hornik, K. (1991). Approximation capabilities of multilayer feedforward networks. Neural Networks, 4, 251±257. Hornik, K., Stinchcombe, M., & White, H. (1989). Multilayer feedforward networks are universal approximators. Neural Networks, 2 (5), 359±366. Hu, M. Y., Zhang, G., Jiang, C. X., & Patuwo, B. E. (1999). A crossvalidation analysis of neural network out-of-sample performance in exchange rate forecasting. Decision Sciences, 30 (1), 197±215. Medsker, L., & Liebowitz, J. (1994). Design and development of expert systems and neural networks, New York: Macmillan. Mishkin, F. S. (1992). The economics of money, banking, and ®nancial markets, New York: Harper Collins. Nath, R., Rajagopalan, B., & Ryker, R. (1997). Determining the saliency of input variables in neural network classi®ers. Journal of Computers and Operations Research, 24 (8), 767±773. Pakath, R., & Zaveri, J. S. (1995). Specifying critical inputs in a genetic algorithm-driven decision support system: An automated facility. Decision Sciences, 26 (6), 749±779.
Pofahl, W. E., Walczak, S. M., Rhone, E., & Izenberg, S. D. (1998). Use of an arti®cial neural network to predict length of stay in acute pancreatitis. American Surgeon, 64 (9), 868±872. Piramuthu, S., Shaw, M., & Gentry, J. (1994). A classi®cation approach using multi-layered neural networks. Decision Support Systems, 11 (5), 509±525. Ranson, J. H. C. (1982). Etiological and prognostic factors in human acute pancreatitis: a review. American Journal of Gastroenterology, 77, 633± 638. Smith, M. (1993). Neural networks for statistical modeling, New York: Van Nostrand Reinhold. Swanson, N. R., & White, H. (1995). A Model-Selection approach to assessing the information in the term structure using linear models and arti®cial neural networks. Journal of Business and Economic Statistics, 13 (3), 265±275. Tahai, A., Walczak, S., & Rigsby, J. T. (1998). Improving arti®cial neural network performance through input variable selection. In P. Siegel, K. Omer, A. deKorvin & A. Zebda, Applications of fuzzy sets and the theory of evidence to accounting II (pp. 277±292). Stamford, Connecticut: JAI Press. Walczak, S. (1995). Developing neural nets for currency trading. Arti®cial Intelligence in Finance, 2 (1), 27±34. Walczak, S. (2000). Information effects on the accuracy of neural network ®nancial forecasting. In S. H. Zanakis, G. Doukidis & C. Zopounidis, Decision-making: Recent developments and worldwide applications (pp. 169±178). Dordrecht, Netherlands: Kluwer Academic. Walczak, S. (2001). An empirical analysis of data requirements for ®nancial forecasting with neural networks. Journal of Management Information Systems, 17 (4), 203±222. Walczak, S., & Cerpa, N. (1999). Heuristic principles for the design of arti®cial neural networks. Information and Software Technology, 41 (2), 109±119. Walczak, S., Tahai, A., & Khondar, K. (1998). Improved cash ¯ows using neural network models for forecasting foreign exchange rates. In P. Siegel, K. Omer, A. deKorvin & A. Zebda, Applications of fuzzy sets and the theory of evidence to accounting II (pp. 293±310). Stamford, Connecticut: JAI Press. Weissman, C. (1997). Analyzing intensive care unit length of stay data: Problems and possible solutions. Critical Care Medicine, 25 (9), 1594± 1599. White, H. (1990). Connectionist nonparametric regression: Multilayer feedforward networks can learn arbitrary mappings. Neural Networks, 3 (5), 535±549.
