Articial Metaplasticity MLP applied to Image Classication Alexis Marcano-Cedeño, Antonio Álvarez-Vellisco and Diego Andina Technical University of Madrid. Group for Automation in Signals and Communications. Madrid, Spain. Email:
[email protected],
[email protected],
[email protected]
Abstract—In this paper we apply Articial Metaplasticity to a Multilayer Perceptron (MLP) for image classication. Articial Metaplasticity is a novel Articial Neural Network (ANN) training algorithm that gives more relevance to less frequent training patterns and subtracts relevance to the frequent ones during training phase, achieving a much more efcient training, while at least maintaining the MLP performance. In this paper, we test Metaplasticity MLP (MMLP) algorithm on an image standard data set: the Wisconsin Breast Cancer Database (WBCD). WBCD is a well-used database in Machine Learning, ANN and Signal Processing. Experimental results show that MMLPs reach better accuracy than any other recent results.
I. I NTRODUCTION The correct image classication is a very important problem in the real world of the Medical industry, Telecommunications, Manufacturing, Aerospace and many others [1] [2]. In this study we will apply a novel method to classify Breast cancer image. Breast cancer is one of the main causes of death in women and early diagnosis is important to reduce the mortality rate. A major class of problems in medical science involves the diagnosis of disease, based upon various tests performed upon the patient. When several tests are involved, the ultimate diagnosis may be difcult to obtain even for a medical expert. [3] Different methods have been used to classify patterns in medical images, such as wavelets, fractal theory, statistical methods, fuzzy theory, Markov models, data mining, neural networks, etc, most of them used features extraction using image-processing techniques [4] [5]. Articial neural networks (ANNs) have been used in different medical diagnoses and the results were compared with physicians’ diagnoses and existing classication methods [6] [7] [8]. The objective of these classication methods is to assign patients to either a “benign” group or a “malignant” group. There has been a lot of research done in medical diagnosis and classication of breast cancer with WBCD database c 2009 IEEE 978-1-4244-3760-3/09/$25.00
using NNs. Übeyli in [9] was presented a comparison of accuracies of different classiers, reported an accuracy of 99.54 % WBCD. In [8], Karabatak and Cevdet presented an automatic diagnosis system for detecting breast cancer based on association rules (AR) and NNs, and obtained a classication accuracy of 97.4% over the entire WBCD. Guijarro-Berdiñas et al. [10] presented a learning algorithm that applies linear-least-squares. They obtained a classication accuracy result of 96.0% over the entire WBCD. The main objective of the proposed work is to classify the lesions as benign or malignant by using MMLP based classier. This method consists in simulating the biological property of the metaplasticity on MLP with Backpropagation. We modeled this interpretation in the NNs training phase. Our MMLP algorithm has been compared with a Classical Backpropagation algorithm as well as with recently proposed algorithms applied on the WBCD database. Our results, prove the MMLP to be superior or at least an interesting alternative. The paper is organized as follows: In Section II the WBCD database is presented. In Section III we present an introduction to neuronal plasticity, to allow the understanding of the biological metaplasticity. In Section IV we introduce the NNs computational model with embedded neuronal plasticity properties. In Section V we present and briey discuss the results of the experimental analysis. In Section VI, we give the conclusions.
II. W ISCONSIN B REAST C ANCER DATA BASE This breast cancer database was obtained from the University of Wisconsin Hospital. It contains 699 examples, where 16 samples have missing values which are discarded in a pre-processing step, so only 683 were used. Each sample has one of 2 possible classes: benign or malignant. The Benign dataset contains 444 samples (65%) and Malignant contains 239 samples (35%). Each record in the database has nine attributes, which are shown in Table I [11]. 650
TABLE I W ISCONSIN BREAST CANCER DATA DESCRIPTION OF ATTRIBUTES Attrib. Attribute Numbers Description 1 Clump thickness 2 Uniformity of cell size 3 Uniformity of cell shape 4 Marginal adhesion 5 Single epithelial cell size 6 Bare nuclei 7 Bland chromatin 8 Normal nucleoli 9 Mitoses
Values Attribute 1-10 1-10 1-10 1-10 1-10 1-10 1-10 1-10 1-10
Mean 4.44 3.15 3.22
Standard Deviation 2.82 3.07 2.99
2.83 2.23
2.86 2.22
3.54 3.45 2.87 1.60
3.64 2.45 3.05 1.73
III. M ETAPLASTICITY The Metaplasticity is dened as the induction of synaptic changes and is also associated to the prior synaptic activity [12] [13]. Metaplasticity is due, at least partially, to variations in the level of postsynaptic depolarization for inducing synaptic changes: These variations facilitate synaptic potentiation and inhibit synaptic depression in depressed synapses and vice versa in potentiated synapses. The direction and the degree of the synaptic change are a function of postsynaptic depolarization during synaptic activation. Long-term potentiation (LTP) is obtained following low levels of postsynaptic depolarization whereas long-term depression (LTD)is produced by stronger depolarizations. On the other hand the metaplasticity indicate a higher level of plasticity, expressed as a change or transformation in the way synaptic efcacy is modied. An understanding of metaplasticity might yield new insights into how the modication of synapses is regulated and how information is stored by synapses in the brain. For a correct understanding of these mechanisms we will start with an introduction to synaptic plasticity [14]. IV. MMLP N EURAL N ETWORK The Multilayer Perceptron Neural Network (MLP) has been used for the solution of many classication problems in pattern recognition applications [15]. The functionality of the topology of the MLP is determined by a learning algorithm. The Backpropagation (BP), based on the method of steepest descent [15] in the process of upgrading the connection weights, is the most commonly used algorithm by the scientic community. The BP algorithm showed some limitations and problems during the training of MLP [16]. Many researchers have centered their research in improving and developing combinations of algorithms with the objective of reducing the complexity of the classiers and, simultaneously, to increase their advantages in terms of effectiveness of the classication [16] [17]. We propose a method to train MMLP. The Metaplasticity as a biological concept is widely known in the eld Biol-
ogy, Medical Computer Science, Neuroscience, Physiology, Neurology and others [18] [19] [20] [21] [22] . Articial metaplasticity is modeled as the ability to change the efciency of articial plasticity giving more relevance to the less frequent patterns and less relevance to the frequent ones [23]. In order to training the MLP we used the following weight function ∗ (x) = fX
A N
B
(2π) .e
N i=1
(1) x2i
where N is the number of components of input vector X that feeds rst hidden layer (for the second hidden layer, X is substituted by rst hidden layer output vector, and so on) and A, B parameters that have to be empirically found (A, B ∈ R). This weight function corresponds to the assumption that probabilities of the input patterns follow a Gaussian distribution. Note that although the algorithm is robust to divergences in this assumption [23], if this diverges much from reality, the training is degraded and can even not converge. A. Network Structure Selection Initially, in order to determine the network structure and metaplasticity parameters, we used the same network parameters applied in recent research [22] [23] [24] (you can see the order of A and B in Tables II). We applied two different criterions to decide for the better network structure and metaplasticity parameters such as: 1) Metaplasticity parameters: xing a number of neurons in the hidden layer sufciently high to presume that the ANN has sufcient processing units to perform the classication, begin to vary the metaplasticity parameters starting with A and nally with parameter B, until we achieve mentioned value (MSE ≈ 0.01) in the minimum number of iterations. 2) Number of neurons in hidden layers: We vary the number of neurons in hidden layers until we achieve the Mean Squared Error (MSE) of approximately 0.01 (metaplasticity parameters are not changed) with the minimum number of neurons without degrading nal performance. For example, in the rst experiment presented in the next section, Table II shows results obtained for different network structures and metaplasticity parameters.
V. RESULTS AND DISCUSSION The MMLP proposed as a classier for detection of the c (software breast cancer was implemented in MATLAB MATLAB version 7.4, R2007a) and computer Pentium IV of
2009 7th IEEE International Conference on Industrial Informatics (INDIN 2009)
651
TABLE II MMLP
RESULTS OBTAINED FOR DIFFERENT NETWORK STRUCTURES AND PARAMETERS OF METAPLASTICITY ALGORITHM
Network Structure I HL O 9 8 1 9 8 1 9 7 1
Metaplasticity Parameters A B 39 0.5 41 0.25 39 0.25
Mean Squared Error 0.01 0.01 0.01
Clustering Accuracy (% ) Training Testing 99.99 % 99.14 % 98.89 % 98.71 % 99.11 % 98.71 %
3.4 GHz with 2 GB of RAM. The nine attributes detailed in Table 1 were used as the inputs of the ANNs. The experiments results show that, the highest classication accuracy is obtained for the MMLP with 9 input neurons, 8 hidden layers neurons and 1 output neurons produce the highest accuracy, determined empirically. Table III, shows the network structure, metaplasticity parameters, epochs, mean square error (MSE) and numbers of patterns used in training and testing, phase. Figure 1 represents the architecture of the NNs developed in this paper. It is composed by one input layer with nine neurons, which maps input data into eight hidden layer and one output neuron. TABLE III N ETWORK PARAMETERS USING IN THIS RESEARCH Types Classiers MMLPs BPNNs
Network Structure I HL O 9 8 1 9 8 1
Metaplasti. MSE Epochs Parameters A B 0.01 2000 39 0.5 0.01 2000 NA2 NA2
Numbers Patterns Training Testing 410 273 410 273
3
2
4
3 4
5 6 7 8 9
5 6
1
Output
Input
1
7 8 Hidden Layers
Fig. 1. MMLP network architecture using in this research, 9 input neurons, 8 hidden layers neurons and 1 output neurons.
The activation function is sigmoidal with scalar output in the range (0,1) and it is the same for all the neurons. To comparatively evaluate the performance of the classiers, all the classiers presented in this study were trained with the 2 NA:
652
Not Apply
For the experiments, we generated 100 MMLPs with different weights whose values were random with normal distribution (mean 0 and variance 1). In each experiment 100 networks were trained in order to achieve an average result that does not depend on the initial random value of the weights of the ANN. Two different criterions were applied to stop the training: in one case it was stopped when the error reached 0.01 (the error reduces but cannot converge to 0) and in the other the training was conducted with a xed number of 2000 epochs. Two different types of experiments were performed. One to determine the degree of accuracy of the MMLP algorithm (considering the specicity, sensitivity and total classication accuracy) trying with several structures of network, varying with metaplasticity parameters A and B, least until the most efcient structure was obtained, with the criteria being the smallest number of patterns for the training and the shortest time of convergence of algorithm. The other experiment was used to compare our algorithm with Classical Backpropagation training. Classication results of the classiers were displayed by a confusion matrix. In a confusion matrix, each cell contains the raw number of exemplars classied for the corresponding combination of desired and actual network outputs. The confusion matrices showing the classication results of the classiers implemented for detection of breast cancer are given in Table IV [9].
1 2
same training data set and tested with same the evaluation data set. The network was trained with 60% of data, 410 samples, of which 144 malignant and 266 benign records. The testing set, remaining 40% of data, consisted of 233 samples of which 95 malignant and 178 benign records. Table 3, denes the network parameters implemented in this research, compare our MMLP algorithm with a Classical Backpropagation and show experimental results obtained.
TABLE IV C ONFUSION MATRICES OF C LASSIFIERS USED FOR D ETECTION OF BREAST C ANCER Type Classiers MMLPs BPNNs
Desired Result Benign records Malignant records Benign records Malignant records
Output Results Benign Malignant 176 2 1 94 175 3 12 83
Usually, to determine the performance of the classiers the specicity, sensitivity and total classication accuracy are calculated. For a correct understanding, the previously mentioned are here dened: Specicity: number of correctly classied benign records / total number of benign records. Sensitivity: number of correctly classied malignant records
2009 7th IEEE International Conference on Industrial Informatics (INDIN 2009)
/ total number of malignant records. Total classication accuracy: number of correctly classied records / total number of records. The performance of the two classiers detection of the breast cancer is presented in Table V. TABLE V T HE C LASSIFICATION ACCURACIES OF C LASSIFIERS USED FOR D ETECTION OF BREAST C ANCER Type Classication Accuracies (%) Classier Specicity Sensitivity Total Classication Accuracy MMLPs 98.95% 98.88% 98.90% BPNNs 98.31% 87.37% 94.51%
VI. C ONCLUSION The goal of this research was to compare the accuracy of two types of classiers: the proposed MMLP and the Classical MLP with Backpropagation, applied to the Wisconsin Breast Cancer Database. The classication results indicate that the MMLP achieved considerable success in image classication. The MMLP classier shows a great performance obtaining the following results average for 100 networks: 98.95% in specicity, 98.85% in sensitivity and the total classication accuracy of 98.90%. Our MMLP, proved to be equal or superior to the state-of-the-art algorithms applied to the WBCD database, and shows that it can be an interesting alternative for the medical industry, among others.
ACKNOWLEDGMENT This research has been partially supported by National (MICINN) and Madrid (CAM) Spanish institutions under the following projects: PTFNN (MCINN ref: AGL200612689/AGR). The author wishes to thank to The National Foundation of Science Technology and Innovation (FONACIT) of the Bolivariana Republic of Venezuela for its contribution in the development of his doctoral studies
[7] Orozco-Monteagudo M, Taboada-Crispí A, and Del Toro-Almenares A. “Training of multilayer perceptron neural networks by using cellular genetic algorithms”. Lecture notes in computer science, Springer. Vol. 4225/2006, pp. 389-398, 2006. DOI 10.1007/11892755. [8] Karabatak M and Cevdet-Ince M.“An expert system for detection of breast cancer based on association rules and neural network”. Expert Systems with Applications, vol. 36, pp. 3465-3469, 2009. [9] Übeyli ED.“Implementing automated diagnostic systems for breast cancer detection”. Expert Systems with Applications, vol. 33(4), pp. 10541062, 2007. [10] Guijarro-Berdiñas B, Fontenla-Romero O, Perez-Sanchez B, and Fraguela P.“A linear learning method for multilayer perceptrons using least-squares”. Lecture Notes in Computer Science, Springer, pp. 365374, 2007. DOI - 10.1007/978-3-540-77226-2. [11] http://archive.ics.uci.edu/ml/datasets.html. [12] Abraham WC and Tate WP.“Metaplasticity: a new vista across the eld of synaptic plasticity”. Progress in Neurobiology, Vol. 52, pp. 303-323, 1997. [13] Abraham WC and Bear MF.“Metaplasticity: the plasticity of synaptic plasticity”. Trends in Neuroscience, Vol. 19(4), pp. 126-130, 1996. [14] Peréz-Otaño I and Ehlers MD.“Homeostatic plasticity and nmda receptor trafcking”. Trends in Neuroscience, Vol. 28, pp. 229-238, 2005. [15] Hagan MT, Demuth HB, and Beale M. Neural network design. PWS Pub Co, Boston, USA, 1996. [16] Leung H and Haykin S.“The complex backpropagation algorithm”. Signal Processing, IEEE Transactions on, Vol. 39, pp. 2101-2104, 1991. [17] Man KF, Tang KS, and Kwong S. Genetic algorithms: Concepts and designs. Springer, London, 1999. [18] Kandel ER, Schwartz JH, and Jessell TM. Principles of neural science. McGraw-Hill, 2000. [19] Jedlicka P.“Synaptic plasticity, metaplasticidad and BCM theory”. Institute of Pathophysiology, Medical Faculty. Comenius University, Vol. 103(4-5), pp. 137-143, Bratislava. Slovakia. 2002. [20] Kinto E, Del-Moral-Hernandez E, Marcano A, and Ropero-Pelaez J. “A preliminary neural model for movement direction recognition based on biologically plausible plasticity rules”. Lecture Notes in Computer Science, Springer. Vol. 4528/2007, pp. 628-636, 2007. DOI - 10.1007/9783-540-73055. [21] Ropero-Pelaez J and Piqueira JR. “Biological clues for up-to-date articial neurons”. In Computational Intelligence for Engineering and Manufacturing, Andina D and Pham D.T. (Eds), Springer-Verlag, The Nederlands. 2007. [22] Andina D, Jevti´c A, Marcano A, and M. Barrón-Adame MJ. “Error weighting in articial neural networks learning interpreted as a metaplasticity model” Lecture Notes in Computer Science, Springer, Vol. 4527/2007, pp. 244-252, 2007. DOI - 10.1007/978-3-540-73053-8. [23] Andina D, Alvarez-Vellisco Antonio, Jevti´c A, and Fombellida J. “Articial metaplasticity can improve articial neural network learning”. In Intelligent Automation and Soft Computing, Special Issue in Signal Processing and Soft Computing. Guest Editor D. Andina. Vol. 15, No. 4, pp. 681-694. TSI Press, USA, 2009. ISSN: 1079-8587. [24] Andina D, Fombellida J. "Metaplasticity Articial Neural Networks Model Application to Radar Detection", Journal of Systemics, Cybernetics and Informatics, Vol. V, Num. 6 pp 91-96. January 2008. ISSN: 1690-4524.
R EFERENCES [1] Salahova S.“Remote sensing and GIS application for earth observation on the base of the neural networks in aerospace image classication”. Recent Advances in Space Technologies, RAST ’07, pp. 275-278, 2007. [2] Mroczek T, Paja W, Piatek L, and Wrzesie M.“Classication and synthesis of medical images in the domain of melanocytic skin lesions”. Conference on Human System Interactions, pp. 705-709, 2008. [3] Subashini TS, Ramalingam V, and Palanivel S. “Breast mass classication based on cytological patterns using rbfnn and svm”. Expert Systems with Applications, Vol. 36, pp. 5284-5290, 2009. [4] Chao L, Xue-Wei L, and Hong-Bo P.“Aplifcation of extension neural network for classication with incomplete survey data”. Cognitive Informatics, ICCI 2006, pp. 1-3, 2006. [5] Misra BB, Biswal BN, Dash PK, and Panda G.“Simplied polynomial neural network for classication task in data mining”. Evolutionary Computation, CEC 2007, pp. 721-728, 2007. [6] Übeyli ED.“Modied mixture of experts for diabetes diagnosis”. J Med Syst, Springer, pp.1-7, 2008. DOI - 10.1007/s10916-008-9191-3.
2009 7th IEEE International Conference on Industrial Informatics (INDIN 2009)
653
