Hybridization Model of Linear and Nonlinear Time ... - IEEE Xplore

Second Asia International Conference on Modelling & Simulation

Hybridization Model of Linear and Nonlinear Time Series Data for Forecasting Roselina Sallehuddin, Siti Mariyam Shamsuddin, Siti Zaiton Mohd Hashim Soft Computing Research Group Faculty of Computer Science and Information System, University Technology Malaysia, 81300 Skudai, Johor roselina, mariyam, sitizaiton @utm.my

series problems are often affected by irregular and infrequent events, which make time series forecasting become complicated and difficult. Thus, a single model isnÑt the best way for forecasting. Although both ANN and ARIMA models have achieved success in their own linear and nonlinear domains, neither ANN nor ARIMA can adequately model and predict time series since the linear model cannot deal with nonlinear relationships while ANN model alone is not able to handle linear and nonlinear patterns equally well. As a result, several researchers have proposed hybridizing ARIMA and ANN model, since different forecasting models can complement each other in capturing patterns of data set. Both theoretical and empirical studies have revealed that a hybridization forecast outperforms individual forecasting models [1, 5]. The merging of this structure can help the researchers in modeling complex autocorrelation structures in time series data more efficiently. Furthermore, by using ARIMA and ANN which is contradicting with each other significantly can assist in generating lower generalization variance or error. However, several researchers have argued that predictive performance improves when using hybrid models [2, 3]. For example, [3] showed that individual model outperformed five of nine data sets used. These inconsistent results indicate the need for further research on how to obtain a good forecasting result from hybrid linear and nonlinear model. It is observed that there are three weaknesses in the previous studies such as type of data used, redundancy factors and implementation of hybridization sequence. Hence, in this study, a new hybrid approach for combining nonlinear model and linear model is proposed to overcome the drawbacks of previous studies by including more additional features; these include multivariate time series, feature selection in removing and selecting significant input data and altering the sequence of combination execution. Since

Abstract The aim of this paper is to propose a novel approach in hybridizing linear and nonlinear model by incorporating several new features. The intended features are multivariate information, hybridization succession alteration, and cooperative feature selection. To assess the performance of the proposed hybrid model allegedly known as Grey Relational Artificial Neural Network(GRANN_ARIMA), extensive comparisons are done with individual model (Artificial Neural Network(ANN), Autoregressive integrated Moving Average(ARIMA) and Multiple Linear Regression(MR)) and conventional hybrid model (ARIMA_ANN) with Root Mean Square Error(RMSE), Mean Absolute Deviation(MAD), Mean Absolute Percentage Error (MAPE) and Mean Square error( MSE ). The experiments have shown that the proposed hybrid model has outperformed other models with 99.5% forecasting accuracy for small-scale data and 99.84% for large-scale data. The obtained empirical results have also proved that the GRANN-ARIMA is more accurate and robust due to its promising performance and capability in handling small and large scale time series data. In addition, the implementation of cooperative feature selection has assisted the forecaster to automatically determine the optimum number of input factor amid with its important ness and consequence on the generated output. Keywords: Forecasting, cooperative feature selection, Hybrid, GRANN_ARIMA, ARIMA_ANN

1. Introduction The real-world time series problems arenÑt absolutely linear or nonlinear; they often contain both linear and nonlinear patterns. Furthermore, real time

978-0-7695-3136-6/08 $25.00 © 2008 IEEE DOI 10.1109/AMS.2008.142

597

time series forecasting is complex, difficult and is affected by many factors, more related information such as human factors, environmental factors and government policies need to be considered in modeling process. Thus, multivariate time series (MTS) forecasting is needed. In this study, cooperative feature selection is utilized where grey relational analysis is integrated with ANN to obtain optimal significant inputs based on the level of relevancy of each input. Grey relational analysis is employed due to its adaptability in dealing with small or large data sets [4]. The primary objective of this study is to probe the effectiveness of our proposed hybrid model as follows: • Could the proposed model outperformed the well-accepted individual model? • Could the proposed model outperformed the conventional hybrid linear and nonlinear model, ARIMA_ANN? • Does the proposed model exhibit robustness compared to the individual and conventional hybrid model? The remainder of this paper is organized as follows. In section 2, description on experimental data is illustrated. Section 3 describes the difference between the conventional hybrid method and the proposed hybrid approach. Detailed discussion on the execution of the proposed hybrid method is given in section 4. The results are presented in section 5, and finally, section 6 is conclusion and future works.

consumer Index, construction index, gold index, finance index, product index, mesdaq index, mining index, plantation index, property index, syariah index, technology index, trading/service index, composite index and industrial index. Essentially, each data set is divided into two parts: in-of-sample and out-of-sample data. In-of-sample data refers to the training data set and is used exclusively for model development. While out-ofsample refers to the test data and is used for evaluation of the unseen data. In other words, test data is used for an independent measure of how the model might be expected to perform on untrained data. However, in ANN, training data usually are divided further into training and validation set. Validation set is used to monitor network performance during training with the intention that early stopping criteria will be met if the network attempts to over fit the training data.

3. The difference between conventional hybrid method and the proposed hybrid method. Figure 1 demonstrates the framework for existing hybrid linear and nonlinear method for univariate time series model, which have been employed by previous researchers. Thus, we refer this framework as conventional hybrid approach.

2. Experimental data To assist us in producing the analysis for benchmarking of the proposed model, two different datasets are used. The first dataset is a small scale dataset which consists of 13 observations of annual China gross grain crop yields from 1990 to 2003 with ten affecting factors; total power of agricultural, electricity consumed in rural areas, irrigation area, consumption of chemical fertilizer, areas affected by natural disaster, budgetary expenditure for agriculture, sown area of grain crops, consumption of pesticide, consumption of agricultural film, and agriculture laborers[4]. The second dataset are real data and obtained from Kuala Lumpur Stock Exchange (KLSE). It contains 200 observations of daily KLSE close price from 4th January 2005 till 21st October 2005, and it represents large scale data. These data set draw on few parameters, and these include close price for KLSE,

Figure1. Conventional hybrid approach (ARIMA_ANN)

Generally, the conventional approach consists of two steps. The first step involves the usage of an ARIMA model to analyze the linear part of the problem. In the

598

second step, a neural network structure is developed to model the residuals from the ARIMA. In this hybridization, ARIMA model is assumed to capture the linearity part of the problem. Thus, the residual yield from ARIMA forecast is presumed to represent the nonlinearity part of the problems. No linearity test is conducted to ensure this assumption. At this juncture, the results yield from ANN is regarded as error term prediction of ARIMA model. To obtain the ultimate result, forecasting values generate by ARIMA and ANN is integrated. Subsequently, the performance between the hybrid model and the individual model is executed. Figure 2 presents the framework of our proposed hybrid model. Unlike conventional hybrid method (Figure 1), the proposed hybrid model is differing from each other in terms of data type and the sequence of hybridization. Other steps in our proposed model are feature selection and linearity test.

The aim of this phase is to reduce the features so as only important and relevant features are being considered. The candidate features are selected based on their abilities to upgrade the forecasting accuracy and speed up the learning process. In this study, grey relational analysis (GRA) and ANN are collaborated as feature selection techniques (GRANN1) to represent filter and wrapper approach accordingly. In this model, GRA is used to select the significant inputs and to rank them based on the priority. Input with less priority will be excluded from the series. In that case, ANN is used to find the optimal significant input prior to ANN forecasting. Subsequently, ANN is trained using BP classifier. As oppose to conventional method; the proposed method ANN with BP classifier is initially applied, and followed by linear model, ARIMA. McLeod and Li test is conducted to verify the linearity of the residual data, before it is fed to linear ARIMA model. This tread is not incorporated in conventional methods. In the following section, a detail explanation on how the proposed framework is implemented will be stipulated.

4. Implementation of the proposed hybrid model There are three major steps in accomplishing the construction of the proposed hybrid model. These include cooperative feature selection, a design of proposed integrated hybrid approach and forecasting performance comparison.

4.1. Cooperative Feature selection (CFS): GRANN The first step is to implement cooperative features selection that combined two different approaches; filter and wrapper method with backward elimination. Output from this phase is the best optimum set of input or predictor that will be used as input for developing the proposed hybrid nonlinear and linear model. Two phases are involved in CFS: GRA analyzer and ANN optimizer. Figure 2. The proposed hybrid approach

4.1.1. GRA analyzer. GRA analyzer is used as preprocessing step and it is self-governing. GRA analyzer will examine the relevancy level between each predictor and the dependent variables. Finally, GRA will rank each predictor according to their importance or priority. For instance, in KLSE close

(GRANN_ARIMA)

In the proposed hybrid method, feature selection or more precisely input selection will be implemented using cooperative feature selection, i.e., the combination of two different approaches; filter and wrapper method with backward elimination strategy.

1

599

GRANN: grey relational artificial neural network

price, 14 affecting factors are initially assumed as predictors for determining the KLSE close price. The GRA analyzer will examine the relationship between each predictor and dependent variable; KLSE close price; and subsequently ranked each predictor based on its priority. Predictor with less priority will be disqualified from the list of predictors. Finally, these selected predictors are fed into ANN optimizer as input.

component, {Eát } , a series of linear components is obtained, which is assume to be {et } , hence et = Yt − Eá t

To ensure that {et } represents the linear component of {Yt } , Mc Leod and Li Test will be exploited. b)

4.1.2. Neural network optimizer. At this point, feature selection is carried out using wrapper method with backward elimination. Selected output from GRA analyzer will be the input for ANN optimizer. There are three iterative processes involve; testing the ANN accuracy, judging the data and deducing from the data input factors set. At each iteration, the input factor that produces the minimum drop off in predictive ANN performance is eliminated. The output from this phase is the optimum number of predictors or optimum input factors, which is ranked based on their precedence. These optimum input factors are utilized as input nodes to build the ANN forecaster.

In this step, ARIMA model is used to model linear component by using generated linear series, {et } . Then it is used to generate a series of forecasts of linear components, defined by{lt } . This step can be said as a process of error generation of time series prediction [5]. In this case, ARIMA is used as error correction of multivariate time series forecasting for ANN model.

c)

At this stage, the final forecasting results can be obtained by integrating the above two components Yá = Eá + lá (2) t

4.2. The design of the proposed integrated hybrid approach

t

t

In order to verify the effectiveness the proposed approach (GRANN_ARIMA), a comparison with conventional sequence will be conducted.

As shown in Figure 2, the proposed hybrid model consists of two main procedures, which integrate ANN with learning BP and ARIMA model in sequential manner. Prior to the development of ARIMA model, the linearity test is conducted to examine the linearity level of time series data. The motivation of using an integrated forecasting approach is: to create a synergy effect that will improve the forecasting power. Below is the procedure for developing the integrated forecasting approach for the proposed hybrid model: a)

(1)

4.3. Comparative analysis of forecasting performance Four statistical tests are employed to evaluate the performance of both individual models and hybrid models. These tests utilize error functions such as Root Mean Square Error (RMSE), Mean Square Error (MSE), Mean Absolute Percentage Error (MAPE) and Mean Absolute Deviation (MAD).

Firstly, ANN is used to fit nonlinear components of multivariate time series (MTS), which assumes to be {Yt , t = 1,2,3....} . Generate a series of forecast, which is defined as {Eát } . Since multivariate time series often contain complicated patterns (linear and nonlinear patterns), it is insufficient to fit only nonlinear component for a multivariate time series. Thus, to improve forecasting accuracy, we need to discover the linear relationship of multivariate time series. By comparing the actual value, {Yt , t = 1,2,3....} and the forecasted value of the nonlinear

1 n

RMSE =

MSE

=

1 n

=

( 3)

2

− pt )

(4)

t =1

∑

ot − p t

∑

ot − p t 100 X ot n

t =1

600

2

− pt )

t =1

∑ (o t

t =1 n

MAPE =

∑ (o t

n

n

MAD

n

n

(5) (6)

It is shown from Table 1 and 2 that the proposed hybrid models always give better result compared to individual model; ANN, MR and ARIMA and conventional hybrid model; ARIMA_ANN. In addition, the precision performance of GRANN_ARIMA is always better than MR, ARIMA, and ANN model. This result is not surprising because ARIMA and MR is a linear model whereas ANN model is a nonlinear model. Due to this reason, they cannot handle both linear and nonlinear data simultaneously. Table 1 depicts that the comparison with individual linear ARIMA and ARIMA_ANN cannot be implemented for the crop yield data. Due to small sample data set being used and the non-stationary time series data, ARIMA model cannot be developed. Theoretically, the minimum of data that need to apply ARIMA model is about 40 to 50 periods of data. This model needs more data for tracking the pattern or component in time series data prior to the modeling process. Before model estimation can be done, time series data must be in a stationary form. Otherwise, the differencing process needs to be employed and it will reduce the size of the data. In this study, the data is an annual data with approximately 11 periods, and non- stationary. These data need to be transformed into a stationary form. From the result, it shows that the propose GRANN_ARIMA model can also perform well in non stationary and small size of time series data Table 2 show the statistical test results and accuracy percentage obtained from each individual and hybrid model used in modeling forecasting model to predict daily close price for KLSE.

where, n is the number of forecasting periods, ot is the actual time series values and pt is the forecasting time series values. GRANN-ARIMA is the best alternative model for forecasting multivariate time series data if it gives the lowest values for RMSE, MSE, MAD and MAPE compared to other models. The RMSE, MSE, MAD and MAPE are calculated based on the out-of-sample data.

5.0 Experimental result and discussion The outcomes from cooperative feature selection indicate that only six factors are selected as the inputs to ANN to predict the grain crop yield (ranked according to their precedence); consumption of pesticide, electricity consumed in rural areas, consumption of chemical fertilizer, irrigation area, consumption of agricultural film, and total power of agricultural. This result is similar to the previous study by [4]. While, for KLSE daily close price, out of 14 affecting factors being observed, only four factors are identified as the most influential factors; syariah index, trading/service index, composite index and industrial index. Hence, these four factors are used as inputs to ANN to predict the next day close price for KLSE. Therefore, the most influential factors for crop yield is pesticide consumption, while syariah index is the most important factor need to be addressed in determining the movement of KLSE close price. Table 1 and Table 2 depict the errors generated by the individual model, conventional hybrid model and the proposed hybrid model. MR and ARIMA are chosen as individual benchmark model since they are linear statistical model that frequently used in univariate and multivariate time series framework accordingly as a sophisticated benchmark for evaluating alternative proposal. While ANN is used as benchmark that represent nonlinear model.

Table 2. Comparative performance for KLSE

Table 1. Comparative performance for crop yield data Table 2 shows that errors from RMSE, MSE, MAPE, MAD for GRANN_ARIMA are lowest compared to the ones obtained by the individual and conventional models. The results from this study also depicted that forecasting performance yield from individual model (ARIMA) outperformed the conventional hybrid (ARIMA_ANN), and it conforms with the previous study. This discrepancy may be due to insufficient data information, over fitting problems in linear model and

na : not applicable, model cannot be developed

601

redundancy information ARIMA_ANN.

while

modeling

it will relieve the model designer from the burden of attaining professional domain knowledge. Hence, we conclude that there are three factors that need to be considered when developing hybrid model. The first factor is the feature selection. As shown in Table 1 and Table 2, the performance of the GRANN_ARIMA model that performed cooperative feature selection is better than conventional hybrid model, ARIMA_ANN. The second factor is the sequence of the implementation for hybridization. From the experiment, it shows that by altering the conventional sequence of hybridization, the forecasting error is decreasing and simultaneously, the forecasting accuracy is increasing. The third factor is the type of data being used; the forecasting performance is increasing when multivariate data is used in modeling time series data. In conclusion, the proposed approach for hybridizing linear and nonlinear model, GRANN_ARIMA can be exploited as an alternative tool for forecasting time series data for better accuracy. Prior studies concealed that ANN learning algorithm was time consuming and tended to trap into local minima solution. Therefore, more studies will be conducted on a new concept of ANN learning algorithm, i.e., a biologically inspired algorithm to speed up the learning time and the accuracy of our proposed hybrid model.

This result indicates the importance of feature selection to identify the most influential factors as the input factors; to avoid the instability that will affect the forecasting accuracy. As well, the over fitting problems that occurred in ARIMA_ANN can be reduced by changing the sequence of hybridization. These discrepancies can be solved by using our proposed hybrid model approach, GRANN_ARIMA. The parameter settings are done as in Table 3: Table 3: Parameter setting for each model Model ARIMA ANN GRANN_ARIMA

ARIMA_ANN

Crop Yield Not applicable (I-H-O)=10-9-1 (α , β ) = (0.5,0.9 ) GRANN : (I-H-O)=6-12-1 (α , β ) = (0.5,0.9 ) ARIMA (p,d,q) = ARIMA (1,0,1) Not applicable

KLSE close Price ARIMA(p,d,q) =ARIMA (2,1,1) I-H-O)=14-29-1 (α , β ) = (0.5,0.9 ) GRANN : (I-H-O)=4-9-1 (α , β ) = (0.5,0.9 ) ARIMA(p,d,q) = ARIMA(0,1,3) ARIMA (p,d,q) =ARIMA (2,1,1) ANN (I-H-O)= (1,2,1) (α , β ) = (0.5,0.9 )

(I-H-O)= (input,hidden,output) ; α= learning rate, β= momemtum; p = parameter of AR; q = parameter of MA; d = the number of differencing

Acknowledgement This work is supported by Universiti Teknologi Malaysia, Skudai Johor Bahru MALAYSIA. Authors would like to thank Soft Computing Research Group, Faculty of Computer Science and Information Systems for their support in making this study a success.

6. Conclusions and future works In this study, GRANN_ARIMA is proposed as a new approach for hybridizing linear and nonlinear model. Unlike conventional hybrid model, the proposed model has a few integrated features such as engaged with multivariate time series data, cooperative feature selection to remove irrelevant input data and altering the sequence of hybridization. To verify the effectiveness of proposed hybrid model, several comparisons have been conducted. Results from the experiments show that the proposed model, GRANN_ARIMA is better than individual model and conventional hybrid model in terms of accuracy and robustness since it produce small forecasting errors and can work well in both small and large scale data. Besides improving the effectiveness of forecasting accuracy, the proposed forecasting hybrid model could also automatically find and choose the optimum number of input factor to be used and the significant of each input factors against the output. Consequently,

References [1]. A. Jain, & A.M. Kumar, Hybrid Neural Network Models For Hydrologic Time Series Forecasting, Applied Soft Computing, Vol. 7, Issue 2, March 2007, pp 585-592 [2]. T. Taskaya-Termizel & K. Ahmad, Are ARIMA Neural Network Hybrids Better Than Single Model? Proc. of Inter. Joint Conf. on NNS (IJCN 2005), July 31-Aug. 4, Montreal, Canada. [3]. T.Taskaya-Termizel, & M.C. Casey, A Comparative Study Of Autoregressive Neural Network Hybrids, Neural Networks, Vol.18, Issues 5-6, July-Aug. 2005, pp 781-789 [4]. Y. Zhang. &.Y. He, Study of Prediction Model on Grey Relational BP Neural Network Based on Rough Set Proc. of The 4th Inter. Conf. On Machine Learning and Cybernetics, Guangzhou, Aug. 2005, pp 18-21. [5] G.P. Zhang, Time Series Forecasting Using a Hybrid ARIMA And Neural Network Model, Neurocomputing 50, 2003, pp. 159-175.

602