An Improved Odor Recognition System Using Learning Vector Quantization with a New Discriminant Analysis Turgay Temel, Electronics Engineering Department, Engineering Faculty, Fatih University, Istanbul, Turkey
[email protected]
Bekir Karlik Computer Engineering Department, Engineering Faculty, Fatih University, Istanbul, Turkey
[email protected]
ABSTRACT A high-performance biologically-inspired odor identification system is described. As a means of odor recognition, learning vector quantization (LVQ) algorithm is employed. Performance improvement is obtained with the use of a preprocessing with discriminant analysis of input samples. Due to sample-based decision, the system can be reliably operated as a real-time electronic nose. Key Words: Olfactory System, Classification, Odour Recognition, Learning Vector Quantization
system, and the cerebral cortex. The subconscious memory evoke of odor can be accounted for by the connection to the hippocampus. Conscious perception of the odor and how to act on the odor takes place in the cerebral cortex [11]. The mammalian olfactory system uses numerous chemical sensors, olfactory receptors, combined with signal processing in the olfactory bulb and automated pattern recognition in the olfactory cortex of the brain. The smell sensory system has not been well understood yet. However, it is known that formation of smell is first subject to excitation which needs to reach the olfactory region with an air flow. In normal respiration, main air flow does not reach this region. The smell is sensed with particular diffusion. During sniff, which may be intentional or spontaneous response to odory stimuli, and changes on vestibule, the air flow becomes stronger and it is directed at olfactory region, [8]. Nervous system, as it does to other sensations, operates on coded spike trains. The spatio-temporal codes generated by sensing cells are converted to neural pulses, which will then be imaged to certain cortical areas in brain.
1. Introduction The objective of an artificial olfactory system is to devise a biologically plausible means to classify smells. There have been numerous mathematical models toward developing such systems and mostly they are of simulative neurobiological information processing forms [11], [6]. Sensing an odor begins at nasal mucosa composed of olfactory and respiratory regions. Olfactory region is corresponds to upper nasal cavity, superior on turbinal region and almost occupy 1/3 of the upper nasal septum. A human has almost 50 million receptive olfactory cells. Olfactory cells are bipolar resident cells amongst supportive cells. Each cell has a sensorial receptor and periphery extension neuron. On the surface of an olfactory cell are cilia with lipid contents. Hence, substances which are concentrated at extension of odory cells are better sensed by reaching bulbus olfactorius passing through the lamina cribrosa forming olfactorius. Fig.1 illustrates the functional units of the brain entitled to information processing. The olfactory cortex performs pattern classification and recognition of the odors sensed. Identified odor information is conveyed to hippocampus, limbic 1075
Fig. 2: Prototype of an electronic nose system Cerebral Cortex Olfactory Olfactory Bulb Tract Taste Sensory Cortex Olfactory Receptor
In this study, we introduce a learning vector quantization scheme to implement the neural machine. In order to reduce within-class separation while increasing between-class separation we will preprocess patterns with linear discrimination method.
Olfactory Cortex Taste Receptor
3. Artificial Neural Network and Learning Vector Quantization
Fig .1: The major processes of the olfactory system.
2. Electronic/Artificial Noses Electronic/artificial nose have been developed to automatically sense/detect and classify odors, vapors, and gases [5], [8] and [9]. The two important operations performed by an electronic nose are the sensing odors and automated recognition of pattern ensembles. The sensing system can be an array of several sensing elements, e.g. chemical sensors where each element measures a different property of the sensed odor, or it can be a combination. Each odor presented to the sensor array produces a signature or pattern characteristic of the odor, hence producing a database of signatures is built up. This database of labeled odor signatures is used to train the pattern recognition system. The goal of this training process is to configure the recognition system to produce unique mappings of each odor so that an automated identification can be performed [7], [10]. The prototype electronic nose, shown in Fig. 2, identifies odors from several common household chemicals [6]. Chemical Vapor
Chemical Sensor Array
Identified Chemical
Neural Network
1076
An artificial neural network (ANN) is an information processing paradigm that was inspired by the way biological nervous systems, such as the brain, process information. The key element of this paradigm is the novel structure of the information processing system. The basic unit of an ANN is the neuron. Each neuron receives sums the weighted inputs, and passes the sum through a transfer function which can be a sigmoid. An ANN is an interconnected network of neurons. The input layer has one neuron for each of the each of the sensor signals, while the output layer has one neuron for each of the different sample properties that should be predicted. Generally, one hidden layer neuron with a variable number of neurons is places between the input and output layer. During the ANN training phase, the weights and transfer function parameters are updated such that the calculated output values for a set of input values are as close as possible to the known true values of the sample properties. Due to its simplicity, flexibility, and efficiency, learning vector quantization (LVQ), as an ANN method, introduced by Kohonen has been widely used in a variety of areas, including real time applications like speech recognition, image compression etc. Several modifications of basic LVQ have been proposed which aim at a larger flexibility, faster convergence, more flexible metrics, or better adaptation to Bayesian decision boundaries, [4].
multi-class case, as a generalization to Fisher discriminant, consider the transformation on input vector x
LVQ procedures are easy to implement and intuitively clear. The classification of data is based on a comparison with a number of so-called prototype or codebook vectors representing the output layer neurons. The similarity is frequently measured in terms of Euclidean distance in feature space. Prototypes are determined in a training phase from labeled examples and can be interpreted in a straightforward way as they directly represent typical data in the same space. This is in contrast with, say, adaptive weights in feedforward neural networks or support vector machines which do not allow for immediate interpretation as easily. Among the most attractive features of LVQ is the natural way in which it can be applied to multi-class problems. A plausible training prescription updates the prototypes in a competitive learning manner as follows: When an input pattern, x, is input to the network, the neuron with the closest (Euclidean norm) weight vector is declared to be the winner, wwinner . Training procedure utilizes the rule
y = Ax
(2)
label of x neural network y preprocessing x Fig. 3: A neural-network with input preprocessing. Optimum class separation can be obtained by maximizing the objective function, [10], J ( A) = Tr{( AΔ w AT )−1( AΔ B AT )}
new = wold old wwinner winner ± η ( x − wwinner ) (1)
(3)
where ‘Tr’ denotes the trace of a matrix, ‘T’ is the transpose, ΔW and ΔB are the total within-class and the between-class covariance matrices, respectively. It should be noted that those definitions are valid in original x space. The complete solution to above optimization problem involves a utilization of spectral (eigen) decomposition, which is a very tedious task for real-time applications. Moreover, the challenge becomes more severe as new classes are added to the original database since predecessor class covariance matrices should be separately inserted into the equation. In order to remedy these shortcomings, we propose transformation given by
where if the winner is the correct class, ‘+’ is taken otherwise it is ‘-‘. It is seen that LVQ makes use of the proximity between the winner neuron and the input pattern as a training rule. The parameter η is an adaptation parameter.
4. Pre-processing with Linear Discriminant Analysis In order to increase the performance of the pattern recognition, input samples can be further pre-processed, Fig. 3. Linear discriminant analysis (LDA) has been widely used for enhancing the data separation ability as well as feature extraction, [2], Fig. 3. This method takes into account not only the within-class scatter but also between-class scatter, leading to a highly effective solution to many pattern classification problems. In a
⎧⎪Δ Δ−1x y=⎨ T i ⎪⎩ ΔT x
1077
for traini ng for post - training
(4)
where at training phase x∈Ci and Δi and ΔT are the covariance matrix of class Ci and overall dataset respectively.
5. Experiments and Results The experimental set-up used in experiments is shown in Fig. 4. It was composed of two chemical sensors (OMXGR) operating in real-time sampling mode. The samples were delivered to a preprocessing and LVQ-based ANN. In case of no-pre-processing, the block does not contain the respective stage.
Fig. 5: Class separation performance improvement with pre-processing.
Fig. 4: A prototype of a real-time odor sensing system.
LVQ algorithm was trained and tested with raw and pre-processed samples. Samples were populated by using boot-strapping, [1] to obtain a 10 times larger dataset organized in 10 subgroups. Training was carried out with 9 subgroups while the remaining subgroup set of data was used for testing. Experiments are performed 100 times to reach a performance distribution. Fig. 6 shows performance histograms of the experimental results as successful identification of odors for η0=0.9. The proposed algorithm was also performed for η0=0.3, 0.5 and 0.7. Table 1 illustrates some statistical values such as average (Av.), standard deviation (std), of successful decisions and average number of iterations (ait) required for preprocessing based algorithm to converge with respect to η0 values employed. It is seen that pre-processing brings out a superior classification performance to LVQ compared to operation with raw data. Another advantage stems from the use of larger range of adaptation parameter, which simply implies that the proposed scheme is very robust to range of variation in adaptation parameter.
The system was trained to identify odors of 20 different perfumes with 32 samples for each. LVQ weights were updated with an adaptation parameter governed by η (k ) = η0k
(5)
where η0 (for training/post-training) and k stands for the iteration number. The advantage of choosing η as in (5) is that there is no need to a priori knowledge of the total iteration number. In fact due to preprocessing applied, the value of η can be chosen in a wide range within (0,1) satisfying the convergence condition [12]. Proposed pre-processing algorithm is investigated with regard to separability of odor classes. Fig. 5 illustrates the class separation performance with preprocessing in Eqn. (4) for 5 odor classes where (x1,x2) and (y1-y2) refer to raw and pre-processed sensor samples, respectively. A clear improvement in class separation is observed with chosen scheme.
1078
The only disadvantage of the proposed algorithm is the need of storing class covariance matrices. Manipulation of a new data involves storage and retrieval of class covariance matrices, which in fact is a minor expense compared to bulky processing with other well-known methods.
References:
[1] G. Gong, ”Cross-validation, the jackknife, and the bootstrap: Excess error estimation in forward logistic regression,” Journal of the American Statistical Association, vol. 81(393), pp. 108-113, 1986. [2] G.J. McLachlan, “Discriminant Analysis and statistical Pattern Recognition, Wiley, New York, 1992. [3] K. Fukunaga, Introduction to Statistical Pattern Recognition, 2nd Ed. Academic Press, San Diego, 990. [4] M. Biehl, et al.,”Learning vector quantization: The dynamics of winner-takes-all algorithms”, Neurocomputing, Elsevier, vol. 69, pp. 660-670, 2006. [5] P. Érdi and G. Barna, “Neurodynamic Approach to Odor Processing,” Proc. Int. Joint Conf. on Neural Networks, IEEE, IJCNN'91, pp. 653-656, 1991. [6] P.E. Keller, et al., “Three Neural Network Based Sensor Systems for Environmental Monitoring,” Proc. IEEE Electro 94 Conf., pp. 377-382, 1994. [7] R.E. Baby, et al.,”Electronic nose: a useful tool for monitoring environmental contamination,” Sensors and Actuators, vol B 69 (3), pp. 214218, 2000. [8] T. Nakamoto et al., “Odor recorder using active odor sensing system,“ Sensors and Actuators, vol. B 76, pp. 465-469, 2001. [9] T. Nakamoto, et al., “Gas/Odor Identification by Semiconductor Gas Sensor Array and an Analog Artificial Neural Network Circuit,” Proc. Int. Conf. Microelectronics, IEEE, MIEL92, pp. 1-9, 1992. [10] W. Bourgeois and R.M. Stuetz, ‘Measuring wastewater quality using a sensor array: prospects for real-time monitoring’, Water SCI Techno. vol. 41 (12), pp. 107-112, 2000. [11] Z. Li and J.J. Hopfield, ‘Modeling the Olfactory Bulb and its Neural Oscillatory Processing’, Biological
Fig. 6: Classification performance distribution of LVQ with (a) raw data, (b) pre-processed data. Table 1: Some statistical values of LVQ with pre-processing η0=0.3 Av.: 96.9 std.: 1.76 ait.: 5.3
η0=0.5 Av.: 94.3 std.:1.95 ait.:6.6
η0=0.7 Av.: 95.1 std.:2.15 ait.:7.9
η0=0.9 Av.:94.8 std.:1.86 ait.: 9.3
6. Conclusions A real-time odor recognition system employing a learning vector quantization as a classifier is described. The classifier performance is augmented with use of a novel pre-processing scheme in a discriminative nature. The pre-processing leads to an improved separation of samples. It contains two-phase for training and posttraining phases. Training phase aims at localizing samples in their respective classes. It was shown that odors are identified very reliably and fast with LVQ using discriminant analysis.
1079
