Neural Networks Approach to Biocomposites Processing Joel-Ahmed M. Mondol1, Satyanarayan Panigrahi, and Madan M. Gupta1, 1Member, IEEE Abstract— Biological neural networks mathematical counterpart artificial neural networks (or neural networks: NN) have contributed to the evolution of a distinct parallel information processing methodology for computational sciences. Problems such as biocomposites modeling or prediction are complicated to model with traditional statistical and mathematical tools due to the inherent noise in data. NN’s efficient parallel processing capability for pattern recognition, forecasting, system analysis, controls and modeling can aid fast prediction, characterization and modeling of novel biocomposites, provided a good knowledge base is available. For the large knowledge base creation, samples with varying flax fiber (0% - 35% with 5% interval) load are created with 2 different operating pressures 1 psi and 1.6 psi (variable operating parameters) to produce compression molded biocomposite boards. These boards go through destructive sampling process to contribute to tensile, impact, hardness, flexural and density data. Using this data a number of neural networks using Matlab® were evaluated to find the optimal neural network architecture. The multilayer feed forward with backpropagation learning (FFBPNN, L1: 10, L2:10, L3: 2) provided best results. It was then further trained with 5 separate training algorithms. Finally the FFBPNN trained with TRAINLM was selected to generate prediction results that were optimal. The trained NN is capable of providing required composition and pressure based on desired mechanical property. Index Terms—Neural networks, Biocomposites.
Campus Drive, Saskatoon, SK, S7N 3R2, Canada (corresponding author phone: 306-261-5029; fax: 306-966-5407; e-mail:
[email protected]).
A mathematical representation of a single neuron, “Linear Neural Unit” (LNU) is provided in Figure 1. In comparison to a human neuron, the mathematical representation of the neuron neural unit can take 2 inputs x1 and x2. The synaptic operation or the connection of the dendrites with neighboring relevant neurons are represented
Fig.1. Biological neural network in their mathematical representation
with weights w1 and w2. Nonlinear mapping or the somatic operation generates the output. The output for the LNU is represented by y. Mathematical representation of a generalized neural unit with n inputs can be formulated with the following equations: 1. 2. 3. 4. 5.
Neural inputs: x1 …..xn Neural ouputs: yN Every input has corresponding weight: w1…wn Bias x0 and corresponding weight w0 Synaptic operation: n
va(k)=wi(k)xi(k) I. INTRODUCTION
i=0
EURAL network can be trained to provide intelligent decisions using its knowledge base [7]. In creating novel natural fiber materials such as biocomposites, researchers find that fiber properties are varying, non-homogeneous and inconsistent [2]. The consistency required for quality control in industrial products is dismayingly absent. Based on learning, adaptation, neural structure as well as the problem domain definition neural networks can aid in the prototype development for novel bio materials and generate decisions that are otherwise reserved for human expert.
N
Manuscript received June 15, 2011. J. M. Mondol is with the Department of Electrical and Computer Engineering, College of Engineering, University of Saskatchewan., 57
742
va(k)= w0(k)x0(k) + w1(k)x1(k) + …..+ wn(k)xn(k) va(k)=waTxa 6. waT= [w0(k) w1(k) ... wn(k)] Rn+1 7. xa = [x0(k) x1(k) … xn(k)]T Rn+1 [where bias x0 = 1] NN have already shown promising results towards coherently modeling materials property in different fields of metallurgy as well as synthetic fiber composites. Research results show that with the aid of an appropriate NN and a good research knowledge base – fast prediction, characterization and modeling of novel composites and materials can be possible. Due to its inherent structure neural networks can be used as an effective tool towards biocomposites materials prediction. Neural networks can aid the appropriate modeling of biocomposites by mapping real 978-1-4577-0253-2/11/$26.00 ©2011 IEEE
world data to develop an expert system within the precise domain. II.EXPERIMENTAL DESIGN A. Creating Training Data Sets from Biocomposite Boards 1. Oilseed Flax fiber L. usitatissimum, was collected (Source of the flax fiber for this experiment was from Biofiber Industries, Canora, Saskatchewan). 2. Contaminants (shive, dirt etc.) were removed by combing the fiber. 3. Pretreatment with 5% NaOH by soaking fiber in solution for 3 - 4 hrs and continuous rinsing in distilled water for 3 h. (Alkaline treatment) [7]. 4. Water was removed by leaving the fiber in a container with porous bottom. Extra water was removed by extensively ringing the water out of the fiber. 5. Treated fiber was dried in a drying cabinet for 3 days until fiber moisture content was lowered to 2% - 3%. Moisture content was calculated by repeatedly taking samples from the dryer and further drying it in an oven overnight to study the loss of moisture. 6. Dried fiber was ground to 2 mm size using a bench top grinder. The ground fiber once fine grounded was kept in sealed plastic bags. 7. Injection grade HDPE pellets (Exxon Mobil) was ground using bench top grinder. 8. Ground HDPE and flax fiber were mixed with 5% - 35% fiber loading with 5% interval. 9. Ground HDPE and ground flax fiber (oil seed flax) mix was extruded using a lab scale extruder with temperatures of 130°C in the 1st barrel zone, 143°C in the 2nd barrel zone, 150°C at the third barrel zone and 150°C at the die zone, the screw was rotating at 20 rpm. The extrudates were water cooled and dried in the drier for 48 hours. 10. Ground extrudates were compressed at 1 MPa and 1.6 MPa (variable operating parameters) with a compression molding unit at 150°C (top and bottom plate). 11. 5 different types of samples (tensile, impact, hardness, bending, density) were created. 12. After testing, the data was stored in Microsoft Excel. B. Creating Neural Network from Matlab for Biocomopsites Modeling 1. Stored individual data (input data), was transferred from Excel to Matlab. They were given the variable names T tensile force, D - density, I - impact force, H - hardness, B bending. 2. Corresponding pressure and fiber % data was transferred from Excel to Matlab. They were given the variable names P pressure, F - fiber %. 3. Matlab was used as the application for developing the neural network. NNTOOL application was loaded in Matlab. 4. 5 different types of neural networks were selected to investigate their best fit towards the biocomposite domain (cascade forward, feedforward backpropagation, feedforward time delay, neural unit (perceptron), nonlinear autoregressive exogenous model (NARX)). Based on training results and
743
fastest to achieve the least error within shortest time frame, the NN model (feedforward backpropagation) was selected. 5. Selected NN was tested for five different training algorithms. 6. Based on selected NN and selected training algorithm (feedforward backpropagation neural network with TRAINSCG (scaled conjugate gradient backpropagation)) – training was done to reach desired performance. 7. Data validation was done using Leave one out, [3, 6] algorithm. 8. The training that provides the best validation result was accepted with the NN model that best satisfies the biocomposites domain.
Fig. 2. Flow chart displaying different research components and activities
III. TRAINING DATA SIZE At first out of the 126 datasets 60% was allocated for training 20% for training and 20% for the prediction. Upon doing the initial training we saw that the error was 0.16. At this point test data was increased to 70% and training data to 20% and 10% for the prediction. In this case there was an increase in overall performance and it came to 0.10. Upon increasing the dataset to 80% and training data to 10% and 10% for the prediction we saw that there was further increase in the performance 0.06. This was still further away from the optimal performance of 0.01 that could be achieved. In this view a single data set from all of the 7 fibre loading and two pressures (7x2) were taken for the prediction and finally 112 dataset were taken as the training data set. For testing the LOO approach was taken as is described in the following. IV. TESTING APPROACH The general testing approach for neural network has been to use 70% for training and 30% for testing. However this becomes a problem when all of the dataset are required for the training. For this research work, in total 126 input datasets with 5 data points containing the tensile, impact, hardness, bending and density data were created. There were also 126 output datasets with 2 data points containing fibre % and pressure was created. Due to the small number of datasets and the large number of weights for the neural network training all maximum number of datasets possible are required. However for validation purposes to observe that the neural network is
capable of predicting biocomposites fibre loading and operational pressure 26 data sets were kept separate. In a scenario as such where there are a small number of training data sets there are a number of algorithms such as cross-validation [10], leave-one-out / jackknife [8] and bootstrapping techniques [4, 5] available for testing models. For this research work LOO (Leave one out) [3, 6] was used. LOO is a special case of k-fold partitioning [10]. In this type of testing methodology a single case is taken as the testing case and all other n-1 cases are used as the training set. In the LOO process for n number of data sets – n number of accuracy errors are calculated. Therefore at any point of time 125 number of dataset are used training sets and the possible level of performance that is considered reasonable is 5% error or less. V.TEST RESULTS Table I provides the five neural networks that were initially studied to find the right neural network best suited for this research work. Both NARX and Neural Unit NN required a large amount of time towards the training phase. The NARX training time requirement was due to the feedback mechanism within individual layers. However due to failure to reach comparable performance with respect to the feed forward back propagation NN – the NARX NN, Neural unit and feed forward time delay was not further investigated for this research work. Overall the feed forward back propagation NN showed best performance and lowest time requirement. Based on these two considerations – FF BPNN was selected as the neural network of choice for this experiment.
A. Neural network selection for FF BP neural networks The feed forward neural network was further trained with 6 different training algorithms. As with evaluation technique used for different neural networks – the training algorithms were evaluated for time required and performance recorded for the same weights and biases in the different layers of the FFBPNN. B. Training algorithms Following are the training algorithms that were investigated: 1. TRAINBFG, 2. TRAINCGB, 3. TRAINLM, 4. TRAINOSS, 5. TRAINRP, 6. TRAINSCG. This was to find the most efficient training algorithm that can effectively predict the biocomposite materials composition and pressure. This selection was based on a study performed by Vladimir Vacic [12] where enhanced performance was based on training, validation, testing and time. For this experiment – the neural architecture was created to have 10 neurons, in the first hidden layer, 10 neurons in the second hidden layer and 2 neurons in the third hidden layer. All of the neurons were exposed to the same amount weight and bias also. C. Training algorithm selection In regards to training algorithm selection for the Feed forward back propagation neural network – the TRAINLM (Levenberg – Marquardt) algorithm was found to be optimal for the biocomposites domain. It shows a performance of 0.981245 after 578 iterations or epochs were complete. The TRAINBFG, TRAINCGB and TRAINOSS training algorithms were not suitable for the FF – BPNN. All three of the algorithms – when used towards training failed to provide any output and exited upon training. TRAINSCG showed good performance but it was inferior to the TRAINLM algorithm. Hence – the TRAINLM training algorithm was accepted as the most suitable training algorithm for the FF BPNN. For all the training algorithms the weights and biases were all maintained the same. Table II provides the results observed from the different training algorithms for the Feed Forward Back Propagation Neural Network. TABLE I COMPARISON OF NEURAL NETWORKS USED IN RESEARCH Number Name of Performance Time neural network (MSE) required (min) 1 Cascade 5.70599 16 min 36 s forward NN 2 Feed forward 2.35769 18 min 47 s BPNN 3 Feed forward 3.98453 38 min 19 s time delay NN 4 Neural unit 9.625 5 min 12 s 5 NARX 17.208 20 min 41 s
Fig. 3. Performance vs Epoch graph for five different neural networks – Cascade forward backpropagation, feed forward back propagation, feedforward time delay perceptron and NARX
744
Epochs
41950 47025 70875 2625 64350
D. Optimized neural network Five different neural networks were used to find the optimal neural network. The feed forward back propagation neural network came out as the decisive optimal neural network. Although commonly used it has the capacity to model multiple domains and was flexible and easy in use. Finally the
technique was used to test and train the neural network. All of the data sets contained 5 different mechanical properties that corresponded to pressure and fiber loading. First run – there were 125 datasets used for training, the 126th dataset was used for testing. The performance was monitored. In this case the performance was 0.419. So 1 – 124 dataset, 126th dataset was used for training and 125th dataset was used towards testing. This process of LOO was continued until 104th dataset was used for testing. For this dataset – it was realized that NN was trained optimal. In this case the training result was the goal of 0.05 and hence the training and testing regimen was terminated. Results are given in Table III.
Fig.5. Feedforward backpropagation neural networks with TRAINSCG Fig. 4. Performance vs. Epoch graph for Feedforward back propagation neural network with 6 different types of training algorithms.
training algorithm was used to in the optimal neural network. A 10 neuron 1st hidden layer, 10 neuron 2nd hidden layer, and 2 neuron 3rd hidden layer was selected as the architecture for the NN. The first hidden layers learning algorithm was tansigmoidal. The 2nd hidden layers learning algorithm was purelin function. The higher numbers of neurons were seen as an improvement from an architecture that contained less neuron (5 neurons, 1st hidden layer, 5 neurons 2nd hidden layer and 2 neurons 3rd hidden layers) TABLE II TRAINING ALGORITHMS WITH REQUIRED TIME, EPOCHS AND PERFORMANCE Function name Algorithm Time Epochs Performa required nce TRAINBFG BFGS quasi-Newton 6s 6 50.0313 backpropagation TRAINCGB Powell –Beale 4s 3 54.5024 conjugate gradient BP TRAINLM Levenberg38 min 77997 0.99999 Marquardt 22 s 6 backpropagation TRAINOSS One step secant 10 s 10 50.0313 backpropagation TRAINRP Resilient 8s 42 50.0312 backpropagation TRAINSCG Scaled conjugate 29 min 115551 0.99999 gradient BP 12 s 4
E. Training and Testing results for the selected neural network architecture Once the appropriate neural network architecture was selected for the neural network we used a multi – layer feed – forward neural network with back propagation learning algorithm and TRAINLM training algorithm. The LOO
745
VI. BIOCOMPOSITES PREDICTION RESULTS Following error percentile formula was used for the calculation of error on every value. Following is the error calculation formula: (Empirical data – NN Result) x 100 Error % = Empirical data Fourteen experiment derived data have been selected for testing the NN for its prediction capability. The data was randomly selected using excel random number generator with 2D array. 1st array for the data type and 2nd array for the data cluster. TABLE III TRAINING AND TESTING PERFORMANCE RESULTS USING LOO TECHNIQUE Test number Testing data Performance (MSE) 1 126 0. 82 2 125 0. 139137 3 124 0. 248 4 123 0.3157273 5 122 0.6673 6 121 0.860526 7 120 0.3303 8 119 0.070409 9 118 0.224 10 117 0.556446 11 116 0.567149 12 115 0.2307 13 114 0.085 14 113 0.89143 15 112 0.718905 16 111 0.09065 17 110 0.46682 18 109 0.79297 19 108 0.24286 20 107 0.48149 21 106 0.66197 22 105 0.49162 23 104 0.0465501
For the unknown dataset, the different averages from the individual data clusters were selected. These data were not the data that the neural network was trained with. However they share properties of the data within the research domain. The error % would indicate the neural networks ability to generalize. 14 data (Representative of the research domain) have been selected for testing the NN for its generalization capability. Pressure average error calculated for known data types from Table 4 was: -4.% Fiber % average error calculated for known data types from Table 4 was: 0.33% Pressure average error calculated for relevant data types from Table 7 was: -2.45% Fiber % average error calculated for relevant data types from Table 7 was: -0.99% The variability of error in this case was simply due to the close proximity of the value of 1 MPa and 1.6 MPa. Hence the neural network that has been generated for the biocomposites domain has been clearly trained and tested and also finally it provides good prediction based on the provided input. VII. CONCLUSION Altinkok and Koker suggested that ANN is a successful analytical tool if properly used [1]. From the research on biocomposites it can be understood that within the biocomposites domain, NN can be an effective tool that assists the design, processing and analysis of biocomposites. The errors that were received are nominal and do not greatly impact the biocomposite materials design and or quality. The prediction that has been provided by the neural network are extremely close and accurate. ACKNOWLEDGEMENT Special thanks to Agriculture Saskatchewan, SAF Chair program, NSERC.
[1]
[2]
[3]
[4] [5] [6]
[7]
[8]
REFERENCES Altinkok, N., R. Koker. 2004. Neural network approach to prediction of bending strength and hardening behavior of particulate reinforced (Al-Si-Mg)-aluminium matrix composites. Materials and Design. 25: 595-602. Bauer., P. J. and J. R. Frederick. 2005. Tillage effects on canopy position specific cotton fiber properties on two soils. Crop Sci 45: 698-703. Capen, D.E., J.W. Fenwick, D.B. Inkley and, A.C. Boynton. 1986. Multivariate models of songbird habitat in New England forests. In: Wildlife 2000: Modelling Habitat Relationships of Terrestrial Vertebrates, ed. J.A. Verner, M.L. Morrison and C.J. Ralph, pp. 171– 75. Madison, WI, USA: University of Wisconsin Press. Efron, B., R. Tibshirani. 1993. An introduction to the Bootstrap. Chapman and Hall, New York. Guisan, A. and F.E. Harrell. 2000. Ordinal response regression models in ecology. J. Veg. Sci 11: 617 – 626. Osborne, P.E. and B.J. Tigar. 1992. Interpreting bird atlas data using logistic models: an example from Lesotho, South Africa. Journal of Applied Ecology 29: 55–62. Schocken, S. and G. Ariav. 1994. Neural networks for decision support: problems and opportunities. Decision Support Systems 11 (5): 393 – 414 Spitz, F., S. Lek. 1999. Environmental impact prediction using neural network modelling. An example in wildlife damage. J. Appl Ecol 746
[10]
36,317–326 Van Houwelingen, J.C. and Le Cessie. 1990. Predictive value of statistical models. Stat Med. 9:1303–1325.
Joel-Ahmed M. Mondol (M’2010) became an IEEE member in the year 2010. He is a PhD candidate in the Dept. of Electrical and Computer Engineering, College of Engineering, University of Saskatchewan. His research interests are in Cloud Computing, Computer Architecture, Service Oriented Architecture and Computer Security. His current research work involves data security, data management and novel service oriented architecture within the cloud computing environment. Mondol completed his M.Sc. (2009) from the Dept. of Agricultural and Bioresource Engineering, University of Saskatchewan and investigated the possible use of computational neural networks for biomaterials and biosystems modeling. He earned a B.Sc. (2006) in Computer Science with a focus in Software Engineering from the Dept. of Computer Science, University of Saskatchewan. Mondol has over 10 years of work experience in the area of ICT, project management and enterprise security systems.
