FEATURE EXTRACTION USING MUTUAL INFORMATION BASED ON PARZEN WINDOW Nojun Kwak
Chong-Ho Choi
Telecommunication R&D Center Samsung Electronics, Suwon, Korea
[email protected]
School of Electrical Eng. & Computer Sci. Seoul National University, Seoul, Korea
[email protected]
ABSTRACT In this paper, feature extraction for classification problems are dealt with. The proposed algorithm searches for a set of linear combinations of original features that maximizes the mutual information between the extracted features and the output class. The difficulties in the calculating mutual information between the extracted features and output class are resolved using Parzen window density estimate. Greedy algorithm with gradient ascent method is used to find the new feature. The computational load is proportional to the square of the number of the given samples. We have applied the proposed method to a simple classification problems and have observed that the proposed method gives better or compatible performance than the conventional feature extraction methods. 1. INTRODUCTION For many pattern recognition problems, it is desirable to reduce the dimension of feature space via feature extraction because there may be irrelevant or redundant features that complicate the learning process, thus lead to erroneous results. Even when the features presented contain enough information about the problem, the result may be erroneous because the dimension of feature space can be so large that it may require numerous instances to obtain a generalized result. Though mutual information is widely accepted as a good measure in the feature extraction problems, the computational complexity makes it difficult to use mutual information as a measure of extracting features. For this reason, in [1] Torkkola extracted output relevant features based on mutual information maximization using Renyi’s entropy measure instead of that of Shannon. Recently we have developed an effective way of calculating mutual information between an output class and continuous input features and applied it to feature selection This work is partly supported by the Brain Neuroinformatics Research Program of Korean government.
problems [2]. In this paper, this method is applied to feature extraction problems by maximizing mutual information. In calculating the mutual information between the input features and the output class, instead of discretizing the input space, we use the Parzen window method to estimate the input distribution. With this method, more accurate mutual information is calculated and the projection direction which produces maximum mutual information is searched for. In the following section, the method of calculating mutual information by Parzen window is presented. In Section III, we propose a new feature extraction method and in Section IV, the proposed algorithm is applied to a simple classification problem to show its effectiveness. And finally, conclusions follow in Section V.
2. CALCULATION OF MUTUAL INFORMATION WITH PARZEN WINDOW In this section, the method of estimating the conditional entropy and the mutual information by the Parzen window in [2] is presented. To calculate the mutual information between the input features and the output class, we need to know the probability density functions (pdf s) of the inputs and the output. The Parzen window density estimate can be used to x) of a vector of conapproximate the probability density p(x tinuous random variables X [3]. It involves the superposition of a normalized window function centered on a set of samples. Given a set of n d-dimensional samples D = x1 , x2 , · · · , xn }, the pdf estimate by the Parzen window is {x given by n 1X x) = x − x i , h), pˆ(x φ(x (1) n i=1 where φ(·) is the window function and h is the window x) converges to the width parameter. Parzen showed that pˆ(x true density if φ(·) and h are selected properly [3]. The window function is required to be a finite-valued non-negative
the conditional probability p(c|ff ) is
density function where Z φ(yy , h)dyy = 1,
(2)
and the width parameter is required to be a function of n such that lim h(n) = 0, (3) n→∞
and
lim nhd (n) = ∞.
n→∞
(4)
For window functions, the rectangular and the Gaussian window functions are commonly used. The Gaussian window function is given by φ(zz , h) =
1 z T Σ−1z exp(− ), 2h2 (2π)d/2 hd |Σ|1/2
F) = − H(C|F
p(ff ) F
N X
p(c|ff ) log p(c|ff )dff ,
(6)
(7)
c=1
where N is the number of classes, is hard to get because it is not easy to estimate p(c|ff ). By the Bayesian rule, the conditional probability p(c|ff ) can be written as p(ff |c)p(c) p(c|ff ) = . p(ff )
(8)
If there are Nc classes, we get the estimate of the conditional pdf pˆ(ff |c) of each class using the Parzen window method as 1 X pˆ(ff |c) = φ(ff − f i , h), (9) nc c i∈I
where c = 1, · · · , Nc ; nc is the number of the examples belonging to class c; and I c is the set of indices of the training examples belonging to class c. Because the sum of the conditional probability eqauls one, i.e., Nc X k=1
p(k|ff ) = 1,
where hc and hk are window width parameters corresponding to class c and k. Using the Gaussian window function (5) with the same window width parameter h and the same covariance matrix ΣF for each class, (10) becomes P i∈I c
In this equation, because the class is a discrete variable, the entropy of the class variable H(C) can be easily calculated. But the conditional entropy Z
The second equality follows from the Bayesian rule (8). Using (9), the estimate of the conditional probability becomes P f − f i , hc ) c φ(f pˆ(c|ff ) = PNc i∈I , (10) P f − f i , hk ) k=1 i∈I k φ(f
(5)
where Σ is a covariance matrix of a d-dimensional vector of random variables Z . In classification problems, the class has discrete values while the input features are usually continuous variables. In this case, the mutual information between the input features F and the class C can be represented as follows: F ; C) = H(C) − H(C|F F ). I(F
p(c)p(ff |c) p(c|ff ) p(c|ff ) = PNc = PNc . f) f |k) k=1 p(k|f k=1 p(k)p(f
pˆ(c|ff ) = P Nc P k=1
−1 f −f f i )T ΣF f −f f i) (f (f ) 2h2 . −1 T f −f f i ) ΣF (f f −f f i) (f exp(− ) 2 2h
exp(−
i∈I k
(11)
Now in the calculation of the conditional entropy (7) with n samples, if we replace the integration with the summation of sample points and suppose that each sample has the same probability, then we get ˆ F) = − H(C|F
Nc n X 1X pˆ(c|ff j ) log pˆ(c|ff j ), n c=1 j=1
(12)
where f j is the jth sample. With (6) and (11), the estimate of the mutual information is obtained as follows: ˆ F ; C) = − I(F
Nc X
Nc n X 1X pˆ(c) log pˆ(c)+ pˆ(c|ff j ) log pˆ(c|ff j ), n c=1 c=1 j=1
(13) where pˆ(c) and pˆ(c|ff ) can be replaced with 1/nc and (12) respectively. 3. FEATURE EXTRACTION BY MAXIMIZING MUTUAL INFORMATION Because the covariance matrix ΣF i has to be inverted in the F i ), it would be better if ΣF i takes a calculation of H(C|F special form. To this end, the original N dimensional feature vector X is transformed into N 0 (≤ N ) dimensional vector Y = W TpcaX using PCA. Note that the rank N 0 of W pca becomes N if the covariance matrix ΣX of the original features X is nonsingular. Note also that by the data Y ; C) = I(X X ; C) if W pca has processing inequality [4], I(Y rank N . After PCA, the covariance matrix of Y becomes N 0 dimensional identity matrix, i.e., ΣY = IN 0 . Instead of using X , if we use Y , the feature extraction problem we are 0 going to study is to find v ∗i ∈
