Flexible Mixture Modeling with the Polynomial Gaussian Cluster ...

Flexible Mixture Modeling with the Polynomial Gaussian Cluster-Weighted Model

arXiv:1207.0939v1 [stat.ME] 4 Jul 2012

Antonio Punzo∗

Abstract In the mixture modeling frame, this paper presents the polynomial Gaussian cluster-weighted model (CWM). It extends the linear Gaussian CWM, for bivariate data, in a twofold way. Firstly, it allows for possible nonlinear dependencies in the mixture components by considering a polynomial regression. Secondly, it is not restricted to be used for model-based clustering only being contextualized in the most general model-based classification framework. Maximum likelihood parameter estimates are derived using the EM algorithm and model selection is carried out using the Bayesian information criterion (BIC) and the integrated completed likelihood (ICL). The paper also investigates the conditions under which the posterior probabilities of component-membership from a polynomial Gaussian CWM coincide with those of other well-established mixture-models which are related to it. With respect to these models, the polynomial Gaussian CWM has shown to give excellent clustering and classification results when applied to the artificial and real data considered in the paper.

Key words: Mixture of distributions, Mixture of regressions, Polynomial regression, Model-based clustering, Model-based classification, Cluster-weighted models.

1

Introduction

Finite mixture models are commonly employed in statistical modeling with two different purposes (Titterington et al., 1985, pp. 2–3). In indirect applications, they are used as semiparametric competitors of nonparametric density estimation techniques (see Titterington et al. 1985, pp. 28–29, McLachlan and Peel 2000, p. 8 and Escobar and West 1995). On the other hand, in direct applications, finite mixture models are considered as a powerful device for clustering and classification by assuming that each mixture-component represents a group (or cluster) in the original data (see Fraley and Raftery 1998 and McLachlan and Basford 1988). The areas of application of mixture models range from biology and ∗ Dipartimento

di Economia e Impresa - Universit` a di Catania (Italy), e.mail: [email protected]

1

medicine (see Schlattmann, 2009) to economics and marketing (see Wedel and Kamakura, 2001). An overview on mixture models is given in McLachlan and Peel (2000) and Fr¨ uhwirth-Schnatter (2006). This paper focuses both on direct and indirect applications. The context of interest is represented by data arising from a real-valued bivariate random vector (X, Y )′ in which a functional dependence of Y on x is assumed for each mixture-component. Note that, hereafter, all vectors are considered to be column vectors. The linear cluster-weighted model (CWM; Gershenfeld 1997) constitutes a natural choice, in the mixtures frame, when this functional relationship is supposed to be linear. The (linear) CWM factorizes ′

the joint density of (X, Y ) , in each mixture-component, into the product of the conditional density of Y |x and the marginal density of X. A Gaussian distribution is usually used for both of them leading to the (linear) Gaussian CWM (see Ingrassia et al., 2012a,b for details on this model and for an extension to the t distribution). Generally, regardless from the density shape, CWMs take simultaneously into account the potential of finite mixtures of regressions (see Fr¨ uhwirth-Schnatter, 2006, Chapter 8) and of finite mixtures of distributions (see, Titterington et al. 1985 and McLachlan and Peel 2000); the idea of the former approach is adopted to model the conditional density of Y |x, while the principle of the latter ′

is used to model both the joint density of (X, Y ) and the marginal density of X. Unfortunately, if the component-dependence structure of Y on x is different from the linear one, the linear CWM is not able to capture it. To solve this problem, the present paper illustrates the polynomial Gaussian CWM. It generalizes the linear Gaussian CWM by using a polynomial model for the dependence of Y on x in each mixturecomponent. Regarding the comparison between a polynomial Gaussian CWM and a finite mixture of polynomial Gaussian regressions, two related aspects need to be preliminarily highlighted: from an indirect point of view, in fact, they are not comparable since the latter is not conceived to model the joint ′

density of (X, Y ) , but only the conditional density of Y |x, and this difference, from a direct point of view, implicitly affects the criterion (vertical distance for finite mixture of regressions) used for clustering and classification. With respect to the latter point, in the paper we will show both artificial and real data on which the polynomial Gaussian CWM outperforms finite mixtures of polynomial Gaussian regressions. Also, most of the existing literature and applications for bivariate mixture models are based on Gaussian components. Mixtures of t distributions have been considered as robust alternatives (see Peel and McLachlan 2000 and Greselin and Ingrassia 2010). These two models have been widely used for model-based clustering (Celeux and Govaert 1995, Banfield and Raftery 1993, and McLachlan and Peel 1998) and model-based classification (Dean et al. 2006 and Andrews et al. 2011). However, Gaussian and t models imply that the groups are elliptically contoured, which is rather restrictive. Furthermore, it is well known that non-elliptical subpopulations can be approximated quite well by a mixture of several basic densities like the Gaussian one (Fraley and Raftery 1998, Dasgupta and Raftery 1998, 2

McLachlan and Peel 2000 p. 1 and Titterington et al. 1985 p. 24). While this can be very helpful for modeling purposes, it can be misleading when dealing with clustering applications since one group may be represented by more than one component just because it has, in fact, a non-elliptical density. A common approach to treat non-elliptical subpopulations consists in considering transformations so as to make the components as elliptical as possible and then fitting symmetric (usually Gaussian) mixtures (Gutierrez et al. 1995 and Lo et al. 2008). Although such a treatment is very convenient to use, the achievement of joint ellipticality is rarely satisfied and the transformed variables become more difficult to interpret. Instead of applying transformations, there has been a growing interest in proposing finite mixture models where the component densities are non-elliptical (above all in terms of skewness) so as to represent correctly the non-elliptical subpopulations (Karlis and Santourian 2009 and Lin 2009, 2010). While such models can be used to create finite mixture models and provide alternative shapes for the derived clusters, they have certain computational difficulties. On the contrary the polynomial Gaussian CWM, which allows the bivariate density components to be flexible enough by allowing the polynomial degree to increase, is easily applicable, has much simpler expressions for estimation purposes, and generates easily interpreted clusters. The paper is organized as follows. In Section 2 the polynomial Gaussian CWM is presented and, in Section 3, its use is contextualized in the presence of possible labeled observations among the available ones. In Section 4, maximum likelihood estimation of the model parameters is approached by considering, and detailing, the EM-algorithm. With this regard, computational details are given in Section 5 while methods to select the number of components and the polynomial degree are given in Section 6. Some theoretical notes about the relation, from a direct point of view, with the classification provided by other models in the mixture frame, are given in Section 7.1. Artificial and real data are considered in Section 7.2 and Section 7.3, and the paper closes, with discussion and suggestions for further work, in Section 8.

2

Model Definition

Let p (x, y; ψ) =

k X

πj f (x, y; ϑj )

(1)

j=1

be the finite mixture of distributions, with k components, used to estimate the joint density of (X, Y )′ . In (1), f (·; ϑj ) is the parametric (with respect to the vector ϑj ) density associated to the jth component,
′ P πj is the weight of the jth component, with πj > 0 and kj=1 πj = 1, and ψ = π′ , ϑ′ , with π = ′

′

(π1 , . . . , πk−1 ) and ϑ = (ϑ1 , . . . , ϑk ) , contains all the unknown parameters in the mixture. As usual,

3

model (1) implicitly assumes that the component densities should all belong to the same parametric family. Now, suppose that for each j the functional dependence of Y on x can be modeled as

Y = µj (x) + εj ,

(2)

where µj (x) = E (Y |X = x, j) is the regression function and εj is the error variable having a Gaussian dis  tribution with zero mean and a finite constant variance σε2j , hereafter simply denoted by εj ∼ N 0, σε2j .   Thus, for each j, Y |x ∼ N µj (x) , σε2j . In the parametric paradigm, the polynomial regression function r
X µj (x) = µr x; β j = βlj xl = β ′j x

(3)

l=0

represents a very flexible way to model the functional dependence in each component. In (3), β j = (β0j , β1j , . . . , βrj )′ is the (r + 1)-dimensional vector of real parameters, x is the (r + 1)-dimensional Vandermonde vector associated to x, while r is an integer representing the polynomial order (degree) which is assumed to be fixed with respect to j. The mixture model (1) becomes a polynomial Gaussian CWM when the jth component joint density is factorized as

   
2 f (x, y; ϑj ) = φ y x; µr x; β j , σε2j φ x; µX|j , σX|j .

(4)

  2 . Summarizing, In (4), φ (·) denotes a Gaussian density; this means that, for each j, X ∼ N µX|j , σX|j the polynomial Gaussian CWM has equation

p (x, y; ψ) =

k X j=1

   
2 . πj φ y x; µr x; β j , σε2j φ x; µX|j , σX|j

(5)

Note that the number of free parameters in (5) is η = kr + 4k − 1. Moreover, if r = 1 in equation (3), model (5) corresponds to the linear Gaussian CWM widely analyzed in Ingrassia et al. (2012a).

3

Modeling framework

As said in Section 1, the polynomial Gaussian CWM, being a mixture model, can be also used for direct applications, where the aim is to clusterize/classify observations which have unknown component memberships (the so-called unlabeled observations). To embrace both clustering and classification purposes, we have chosen a very general scenario where there are n observations (x1 , y1 )′ , . . . , (xn , yn )′ , m of which are labeled. As a special case, if m = 0, we obtain the clustering scenario. Within the model-based

4

classification framework, we use all the n observations to estimate the parameters in (5); the fitted mixture model is so adopted to classify each of the n − m unlabeled observations through the corresponding maximum a posteriori probability (MAP). Drawing on Hosmer Jr. (1973), Titterington et al. (1985, Section 4.3.3) pointed out that knowing the label of just a small proportion of observations a priori can lead to improved clustering performance. Notationally, let z i be the k-dimensional component-label vector in which the jth element zij is defined to be one or zero according to whether the mixture-component of origin of (xi , yi )′ is equal ei = (e to j or not, j = 1, . . . , k. If the ith observation is labeled, denote with z zi1 , . . . , zeik ) its com-

ponent membership indicator. Then, arranging the data so that the first m observations are labeled, n
′ o
′ e′m e′1 , . . . , xm , ym , z the observed sample can be denoted by S = {Sl , Su }, where Sl = x1 , y1 , z  ′ ′ is the sample of labeled observations while Su = (xm+1 , ym+1 ) , . . . , (xn , yn ) is the sample of un-

labeled observations. The completed-data sample can be so indicated by Sc = {Sl , Su∗ }, where Su∗ = o n
′ ′ xm+1 , ym+1 , z ′m+1 , . . . , (xn , yn , z ′n ) . Hence, the observed-data log-likelihood for the polynomial Gaussian CWM, when both k and r are supposed to be pre-assigned, can be written as

l (ψ) =

m X k X i=1 j=1

zeij [ln πj + ln f (xi , yi ; ϑj )] +

n X

i=m+1



ln 

k X j=1



πj f (xi , yi ; ϑj ) ,

(6)

while the complete-data log-likelihood is

lc (ψ) =

m X k X i=1 j=1

=

zeij [ln πj + ln f (xi , yi ; ϑj )] +

l1c (π) + l2c (κ) + l3c (ξ) ,

n k X X

zij [ln πj + ln f (xi , yi ; ϑj )]

i=m+1 j=1

(7)

′ ′  
′ ′ 2 , ξ = ξ′1 , . . . , ξ ′k with ξ j = βj , σε2j , and where κ = (κ′1 , . . . , κ′k ) with κj = µX|j , σX|j l1c (π) =

m X k X i=1 j=1

l2c (κ) =

l3c (ξ) =

zeij ln πj +

n k X X

zij ln πj

(8)

i=m+1 j=1

"
2 # m k   yi − β′j xi 1 XX 2 + zeij − ln (2π) − ln σεj − 2 i=1 j=1 σε2j "
2 # n k ′   y − β x 1 X X i i j + zij − ln (2π) − ln σε2j − 2 i=m+1 j=1 σε2j "
2 # m k   x − µ 1 XX i X|j 2 − zeij − ln (2π) − ln σX|j + 2 2 i=1 j=1 σX|j "
2 # n k   x − µ 1 X X i X|j 2 − zij − ln (2π) − ln σX|j . + 2 2 i=m+1 j=1 σX|j

5

(9)

(10)

4

The EM algorithm for maximum likelihood estimation

The EM algorithm (Dempster et al., 1977) can be used to maximize l (ψ) in order to find maximum likelihood (ML) estimates for the unknown parameters of the polynomial Gaussian CWM. When both labeled and unlabeled data are used, the E and M steps of the algorithm can be detailed as follows.

4.1

E-step

The E-step, on the (q + 1)th iteration, requires the calculation of   Q ψ; ψ (q) = Eψ (q) [lc (ψ) |Su∗ ] .

(11)

As lc (ψ) is linear in the unobservable data zij , the E-step – on the (q + 1)th iteration – simply requires the calculation of the current conditional expectation of Zij given the observed sample, where Zij is the random variable corresponding to zij . In particular, for i = m + 1, . . . , n and j = 1, . . . , k, it follows that Eψ(q) (Zij |Su∗ )

(q)

= zij

  (q) (q) πj f xi , yi ; ϑj  ,  p xi , yi ; ψ (q)

=

(12)

′

which corresponds to the posterior probability that the unlabeled observation (xi , yi ) belongs to the jth component of the mixture, using the current fit ψ (q) for ψ. By substituting the values zij in (7) with (q)

the values zij obtained in (12), we have         Q ψ; ψ (q) = Q1 π; ψ (q) + Q2 κ; ψ (q) + Q3 ξ; ψ (q) ,

6

(13)

where   = Q1 π; ψ (q) 

Q2 κ; ψ

(q)



m X k X i=1 j=1

n k X X

(q)

zij ln πj

(14)

i=m+1 j=1

"
2 # m k   yi − β′j xi 1 XX 2 zeij − ln (2π) − ln σεj − + 2 i=1 j=1 σε2j "
2 # n k   yi − β ′j xi 1 X X (q) 2 + − ln (2π) − ln σεj − z 2 i=m+1 j=1 ij σε2j "
2 # m k   xi − µX|j 1 XX 2 + zeij − ln (2π) − ln σX|j − 2 2 i=1 j=1 σX|j "
2 # k n   x − µ 1 X X (q) i X|j 2 + − − ln (2π) − ln σX|j z . 2 2 i=m+1 j=1 ij σX|j

=

  = Q3 ξ; ψ (q)

4.2

zeij ln πj +

(15)

(16)

M-step

On the M-step, at the (q + 1)th iteration, it follows from (13) that π (q+1) , κ(q+1) and ξ(q+1) can be computed independently of each other, by separate maximization of (14), (15) and (16), respectively. Here, it is important to note that the solutions exist in closed form.   Regarding the mixture weights, maximization of Q1 π; ψ (q) with respect to π, subject to the constraints on those parameters, is obtained by maximizing the augmented function m X k X i=1 j=1

zeij ln πj +

n k X X

i=m+1 j=1



(q)

zij ln πj − λ 

k X j=1



πj − 1 ,

(17)

where λ is a Lagrangian multiplier. Setting the derivative of equation (17) with respect to πj equal to zero and solving for πj yields to (q+1)

πj

=

m X i=1

zeij +

n X

i=m+1

n

(q)

zij

.

(18)

  With reference to the updated estimates of κj , j = 1, . . . , k, maximization of Q2 κ; ψ (q) leads to

(q+1)

µX|j

=

m X

zeij xi +

i=1 m X i=1

(q+1)

σX|j

zeij +

n X

i=m+1 n X

(q)

zij xi (q)

zij

i=m+1

n 2 1/2  2  X (q+1) (q) (q+1) zij xi − µX|j + zeij xi − µX|j     i=1 i=m+1  .  =  n m  X X   (q) zij zeij +



m X

i=m+1

i=1

7

  Finally, regarding the update estimates of ξj , j = 1, . . . , k, maximization of Q3 ξ; ψ (q) , after some

algebra, yields to

(q+1) βlj

=



m X

n X

′

(q) zij xi xi ′

zeij xi xi +   i=1 ˜ ˜ ˜ ˜ i=m+1  − n m  X (q) X  zij zeij + 

i=1

m X

(q) zeij yi xi

i=m+1 n X

(q) zij yi xi

+   i=1 ˜ i=m+1  · n m X X  (q) zij zeij +

(q+1)

β0j

=

zeij yi +

i=1 m X i=1

σε(q+1) j

=



zeij +

−

i=m+1

i=1

m X

˜

n X

i=m+1 n X

(q)

zij yi

− (q)

zij

r X l=1

i=m+1

m X

zeij xi ˜ i=1 m X

m X

zeij yi i=1 m X i=1 m X

+

zeij +

i=1

(q) zij xi

i=m+1 n X

˜

(q) zij

i=m+1 n X

zeij +

i=m+1 n X

′

m X

−1 (q) ′ zij xi 

n X

zeij xi + ˜  ˜ i=m+1  n m X (q)  X  zij zeij +

i=1

i=m+1

i=1

(q) zij yi

i=m+1 n X

m X

(q) zij

i=m+1 n X

zeij xli +

(q+1) i=1 m X

βlj

+

zeij +

i=1

n X

zeij xi ˜ i=1 m X i=1

(q)

zij xli

+

zeij +

n X

i=m+1 n X

i=m+1



(q) zij xi  (q) zij

˜ ,  

·

l = 1, . . . , r

(q)

zij

i=m+1

n  2  2 1/2 X (q) ′ (q+1) ′ (q+1) zeij yi − xi βj + zij yi − xi βj    i=1  i=m+1   , n m   X X   (q) zij zeij + m X

i=1

i=m+1

where the r-dimensional vector xi is obtained from the Vandermonde vector xi by deleting its first ˜ element.

4.3

Some considerations

In the following, the estimates obtained with the EM algorithm will be indicated with a hat. Thus, for b and zbij will denote, respectively, the estimates of ψ and zij , i = m+1, . . . , n and j = 1, . . . , k. example, ψ As said before, the fitted mixture model can be used to classify the n − m unlabeled observations via

the MAP classification induced by    1 MAP (b zij ) =   0

if maxh {b zih } occurs at component j

(19)

otherwise

i = m + 1, . . . , n and j = 1, . . . , k. Note that the MAP classification is used in the analyses of Section 7. As an alternative to the EM algorithm, one could adopt the well-known Classification EM (CEM; Celeux and Govaert, 1992) algorithm, although it maximizes lc (ψ). The CEM algorithm is almost identical to the EM algorithm, except for a further step, named C step, considered between the standard   (q) (q) E and M ones. In particular, in the C step the zij in (12) are substituted with MAP zij , and the 8

obtained partition is used in the M-step.

5

Computational issues

Code for the EM algorithm (as well as its CEM variant) described in Section 4 was written in the R computing environment (R Development Core Team, 2011).

5.1

EM initialization

Before running the EM algorithm, the choice of the starting values constitutes an important issue. The standard initialization consists in selecting a value for ψ (0) . An alternative approach (see McLachlan and Peel, 2000, p. 54), more natural in the modeling frame described in Section 3, is to perform the first E-step by (0)

specifying, in equation (12), the values of z i , i = m + 1, . . . , n, for the unlabeled observations. Among the possible initialization strategies (see Biernacki et al. 2003 and Karlis and Xekalaki 2003 for details) – according to the R-package flexmix (Leisch 2004 and Gr¨ un and Leisch 2008) which allows to estimate finite mixtures of polynomial Gaussian regressions – a random initialization is repeated t times from different random positions and the solution maximizing the observed-data log-likelihood among these t (0)

runs is selected. In each run, the n − m vectors z i

are randomly drawn from a multinomial distribution

with probabilities (1/k, . . . , 1/k).

5.2

Convergence criterion

The Aitken acceleration procedure (Aitken, 1926) is used to estimate the asymptotic maximum of the log-likelihood at each iteration of the EM algorithm. Based on this estimate, a decision can be made regarding whether or not the algorithm has reached convergence; that is, whether or not the log-likelihood is sufficiently close to its estimated asymptotic value. The Aitken acceleration at iteration k is given by a(k) =

l(k+1) − l(k) , l(k) − l(k−1)

where l(k+1) , l(k) , and l(k−1) are the log-likelihood values from iterations k + 1, k, and k − 1, respectively. Then, the asymptotic estimate of the log-likelihood at iteration k + 1 (B¨ohning et al., 1994) is given by (k+1) l∞ = l(k) +

  1 (k+1) (k) . l − l 1 − a(k) (k+1)

In the analyses in Section 7, we follow McNicholas (2010) and stop our algorithms when l∞ with ǫ = 0.05.

9

− l(k) < ǫ,

5.3

Standard errors of the estimates

b is determined by Once the EM algorithm is run, the covariance matrix of the estimated parameters ψ using the inverted negative Hessian matrix, as computed by the general purpose optimizer optim in the

R-package stats, on the observed-data log-likelihood. In particular, optim is initialized in the solution provided by the EM algorithm. The optimization method of Byrd et al. (1995) is considered among the possible options of the optim command. As underlined by Louis (1982) and Boldea and Magnus (2009), among others, the complete-data log-likelihood could be considered, instead of the observed-data log-likelihood, in order to simplify the computations because of its form as a sum of logarithms rather than a logarithm of a sum.

6

Model selection and performance evaluation

The polynomial Gaussian CWM, in addition to ψ, is also characterized by the polynomial degree (r) and by the number of components k. So far, these quantities have been treated as a priori fixed. Nevertheless, for practical purposes, choosing a relevant model needs their choice.

6.1

Bayesian information criterion and integrated completed likelihood

A common way to select r and k consists in computing a convenient (likelihood-based) model selection criterion across a reasonable range of values for the couple (r, k) and then choosing the couple associated to the best value of the adopted criterion. Among the existing model selection criteria, the Bayesian information criterion (BIC; Schwarz, 1978) and the integrated completed likelihood (ICL; Biernacki et al., 2000) constitute the reference choices in the recent literature on mixture models. The BIC is commonly used in model-based clustering and classifications applications involving a family of mixture models (Fraley and Raftery 2002 and McNicholas and Murphy 2008). The use of the BIC in mixture model selection was proposed by Dasgupta and Raftery (1998), based on an approximation to Bayes factors (Kass and Raftery, 1995). In our context the BIC is given by   b − η ln n. BIC = 2l ψ

(20)

Leroux (1992) and Keribin (2000) present theoretical results that, under certain regulatory conditions, support the use of the BIC for the estimation of the number of components in a mixture model. One potential problem with using the BIC for model selection in model-based classification or clustering applications is that a mixture component does not necessarily correspond to a true cluster. For example, a cluster might be represented by two mixture components. In an attempt to focus model 10

selection on clusters rather than mixture components, Biernacki et al. (2000) introduced the ICL. The ICL, or the approximate ICL to be precise, is just the BIC penalized for estimated mean entropy and, in the classification framework used herein, it is given by

ICL ≈ BIC +

n k X X

i=m+1 j=1

where

Pn

i=m+1

Pk

j=1

MAP (b zij ) ln zbij

(21)

MAP (b zij ) ln zbij is the estimated mean entropy which reflects the uncertainty in the

classification of observation i into component j. Therefore, the ICL should be less likely, compared to the BIC, to split one cluster into two mixture components, for example. Biernacki et al. (2000, p. 724), based on numerical experiments, suggest to adopt the BIC and the ICL for indirect and direct applications, respectively.

6.2

Adjusted Rand index

Although the data analyses of Section 7 are mainly conducted as clustering examples, the true classifications are actually known for these data. In these examples, the Adjusted Rand Index (ARI; Hubert and Arabie, 1985) is used to measure class agreement. The original Rand Index (RI; Rand, 1971) is based on pairwise comparisons and is obtained by dividing the number of pair agreements (observations that should be in the same group and are, plus those that should not be in the same group and are not) by the total number of pairs. RI assumes values on [0, 1], where 0 indicates no pairwise agreements between the MAP classification and true group membership and 1 indicates perfect agreement. One criticism of RI is that its expected value is greater than 0, making smaller values difficult to interpret. ARI corrects RI for chance by allowing for the possibility that classification performed randomly should correctly classify some observations. Thus, ARI has an expected value of 0 and perfect classification would result in a value of 1.

7

Illustrative examples and considerations

This section begins showing some relations between polynomial Gaussian CWM and related models. The section continues by looking at two applications on artificial and real data. Parameters estimation for finite mixtures of polynomial Gaussian regressions is carried out via the flexmix function of the R-package flexmix which, among other, allows to perform EM and CEM algorithms. If not otherwise stated, the number of repetitions, for the random initialization, will be fixed at t = 10 for all the considered models.

11

7.1

Preliminary notes

Before to illustrate the applications of the polynomial Gaussian CWM on real and artificial data sets, it is useful to show some limit cases in a direct application of this model. Let p (y |x; π, κ ) =

k X j=1

 
πj φ y x; µr x; β j , σε2j

(22)

be the (conditional) density of a finite mixture of polynomial Gaussian regressions of Y on x. Moreover, let p (x; π, ξ) =

k X j=1

  2 πj φ x; µX|j , σX|j

(23)

be the (marginal) density of a finite mixture of Gaussian distributions for X. The following propositions gives two sufficient conditions under which the posterior probabilities of component-membership, arising from models (22) and (23), respectively, coincide with those from the polynomial Gaussian CWM. Similar results for the linear Gaussian and the linear t CWM are given in Ingrassia et al. (2012a) and Ingrassia et al. (2012b), respectively. Proposition 1. Given k, r, π and κ, if µX|1 = · · · = µX|k = µX and σX|1 = · · · = σX|k = σX , then model (5) generates the same posterior probabilities of component-membership of model (22). Proof. Given k, r, π and κ, if the component marginal densities of X do not depend from j, that is if µX|1 = · · · = µX|k = µX and σX|1 = · · · = σX|k = σX , then the posterior probabilities of componentmembership for the CWM in (5) can be written as

P ( Zij = 1| k, r, π, κ, µX , σX )

=

 

2 πj φ yi xi ; µr xi ; βj , σε2j φ xi ; µX , σX

k X j=1

=

 

2 πj φ yi xi ; µr xi ; βj , σε2j φ xi ; µX , σX

 

✭✭ 2✭ φ✭ xi✭ ; µ✭ πj φ yi xi ; µr xi ; βj , σε2j ✭ X , σX

k  

X ✭✭ 2✭ πj φ yi xi ; µr xi ; βj , σε2j φ✭ xi✭ ; µ✭ X , σX ✭ j=1

=

 
πj φ yi xi ; µr xi ; βj , σε2j

k X j=1

 
πj φ yi xi ; µr xi ; βj , σε2j

,

i = 1, . . . , n, j = 1, . . . , k,

which coincide with the posterior probabilities associated to the model in (22). Proposition 2. Given k, r, π and ξ, if β1 = · · · = βk = β and σε1 = · · · = σεk = σε , then model (5) generates the same posterior probabilities of component-membership of model (23).

12

Proof. Given k, r, π and ξ, if the component regression models does not depend from j, that is if β 1 = · · · = β k = β and σε1 = · · · = σεk = σε , then the posterior probabilities of component-membership for model (5) can be written as

P ( Zij = 1| k, r, π, ξ, β, σε )

=


 2 πj φ yi xi ; µr (xi ; β) , σε2 φ xi ; µX|j , σX|j

k X j=1

=

=


 2 πj φ yi xi ; µr (xi ; β) , σε2 φ xi ; µX|j , σX|j

 ✭
 ✭✭✭ ✭; β) 2 2 x✭ ✭ ✭ φ x ; µ , σ ; µ (x , σ yi✭ πj φ ✭ i i r i X|j ε X|j ✭ k   ✭
X ✭✭✭ 2 2 x✭ πj φ xi ; µX|j , σX|j µr ✭ (x✭ φ✭ yi✭ i; ✭ i ; β) , σε ✭ j=1   2 πj φ xi ; µX|j , σX|j , i = 1, . . . , n, j = 1, . . . , k, k   X 2 πj φ xi ; µX|j , σX|j j=1

which coincide with the posterior probabilities of the model in (23).

7.2

Artificial data

An artificial data set is here generated by a polynomial Gaussian CWM. One of the aims is to highlight a situation in which a finite mixture of polynomial Gaussian regressions provides a wrong classification although the underlying groups are both well-separated and characterized by a polynomial Gaussian relationship of Y on x. The data consist of n = 700 bivariate observations randomly generated from a cubic (r = 3) Gaussian CWM with k = 2 groups having sizes n1 = 400 and n2 = 300. Table 1 reports the parameters of the generating model. Simulated data are displayed in Figure 1 by what, from now on, will be simply named Table 1: Parameters of the cubic Gaussian CWM (k = 2) used to generate the data. (a) Regression parameters

β0j β1j β2j β3j σεj

(b) Other parameters

Component j = 1

Component j = 2

0.000 -1.000 0.000 0.100 1.600

-8.000 0.100 -0.100 0.150 2.300

Component j = 1

Component j = 2

0.571 -2.000 1.000

0.429 3.800 0.700

πj µX|j σX|j

′

as CW-plot. It allows to visualize the joint distribution of (X, Y ) , the regression of Y on x, and the marginal distribution of X; as stressed from the beginning, these represent the key elements on which a CWM is based. A gray scale, and different line types (solid and dashed in this case), allow to distinguish

13

0.20 0.10 0.00

density of X

20

−6

−4

−2

0

2

4

6 2 2

2 2 2 222 2 222222 222 22 22222 2222 22 2 2 2 22222 2 222 2 22 22 2 222222222 2 2 22222 2 2 2 2222 2 2 22 2 22 22 2222 2 2222222222 2 2 2222 2 2 2 222222222 222222222222 2 22 222 22 2 2 22 2222 22222 2 222222 2 222222 2 2 2 2 2 2 2 2 2 2 2 222 22 222222 2222222 2 2 2 2 22222222 2222 22 2 222222 22 22222 2 2 2 2 2 22 222 2 2 20 222222 2 .0 2 212 2 2 0. 22 00 2224 2 22 2 2 2 2 622 .002 2 02 2 22 0.002

2

4

0. 01

10

15

22

−15

−10

01

0.

16

0.0

6

18

0.02

0.0

22

0.04

0.0 24

0. 02

2 1

0.0

22

0.0

0. 00 8

−5

0

0.0 42 0.032

Y

5

1 1 1 111 1 1 11 0.006 1 1 1 4 1 1 1 111 111 0.0118 1 11 1 1 1 1 11101.011 1111 1111 11111111111 11110.036 1 1 101.021111 111 11 11 111 .1014111111111111 11 11111 1 1111 111 1 11111111111111 11 111 0111111 1 11111 1 1 1 11111111111111 11 11 1 1 11 1 111 11 11111111111111111111 11 1 1 11 1 1 1 1 1 1 1 1 1111 111 1111 1 1 11 11111111110.038 11111 11111111 11 11 1 1111 1 1 11 11 11 111 1111 1 10.024 1 111 111 111 1111 10.016 1 11 111 0.01 1 111 111111 10.0 1 08 1 1 11 1 111 0.0112 0.004 1 0.002 11 1 11 1 1 1 1 1 1 11 1 111 1 1 1

2 1

−6

−4

−2

0

2

4

6

X

Figure 1: CW-plot of the simulated data from a cubic Gaussian CWM (n = 700 and k = 2). the underlying groups in Figure 1. The top of Figure 1 is dedicated to the marginal distribution of X. Here, we have an histogram of the simulated data on which are superimposed the component univariate Gaussian densities, multiplied by the corresponding weights π1 and π2 , and the resulting mixture. The scatter plot of the data is displayed at the bottom of Figure 1. The observations of the two groups are here differentiated by the labels 1 and 2 and by the different gray scales. The true cubic Gaussian regressions are also separately represented. The underlying (generating) joint density is also visualized via isodensities; its 3D representation is displayed in Figure 2. ei , i = 1, . . . , n, and we will evaluate the Now, we will suppose to forget the true classifications z

performance of a finite mixture of cubic Gaussian regressions on these data. To make flexmix in the best conditions, we have used the true classification as starting point in the EM algorithm and we have considered the true values of k and r. Nevertheless, without going into details about the estimated parameters, the clustering results are very bad, as confirmed by the scatter plot in Figure 3 and by a very low value of the adjusted Rand Index (ARI = 0.088). On the contrary, as we shall see in a short time, the results obtained with the polynomial Gaussian CWM are optimal. In particular, differently from the previous model, in this case we preliminarily 14

joint density

y x

Figure 2: Underlying joint density of the simulated data. estimate the number of mixture components k and the polynomial degree r according to what suggested   b , BIC, and ICL, obtained by using the EM algorithm, in Section 6.1. Table 2 reports the values of 2l ψ

for a large enough number of couples (k, r), k = 1, . . . , 5 and r = 1, . . . , 5. Bold numbers highlight the

best model according to each model selection criterion. It is interesting to note as BIC and ICL select the same (true) model characterized by k = 2 and r = 3. For this model, Table 3 shows the ML estimated parameters (and their standard errors in round brackets) while Figure 4 displays the resulting scatter plot. Visibly, optimal results in terms of fit and clustering are obtained, as also corroborated by the value ARI = 1. Other experiments, whose results are not reported here for brevity’s sake, have shown that a finite mixture of polynomial Gaussian regressions is not able to find the underlying group-structure when it also affects the marginal distribution of X. These results generalize to the case r > 1 the considerations that Ingrassia et al. (2012a) make with reference to the linear Gaussian CWM in comparison with finite mixtures of linear Gaussian regressions.

15

20

2 2

15

22 1

10

2 2

−5

0

Y

5

2 2 2 22 22 2 22 2 2 2 2 2 22 2 2 2 22222222 2222222 22 2 2 2 2 2 2 1 2 22 2 22222222 22222 222 2222222222222 222 2 2222222 1 1 2 22 22222222 222222222 22222222 21 1 1 2222222 2 2 22222 2222222 222 222 22 222222222222222 22222222 2 2222 22 2 1 1 222222 222 2 2 2 222222222222 2222 222 22 2 2 2 2 2 2 222222222222 2 2 22 222222 2 22 222 22 2 2 2 2 22 222222 22 1 1 222 2 22 22 22222 22 222 22 2 2 221 22 2 2 2 1 222 2 2 2 2 22 2 2 22 1 2 2 2 22 2 2 1 22 2 22 2 2 1 2 2 2 1 2 2 11 2 2

−10

2 2

1 22 2 2 1 22 2 12 2 1 12 1 11 1 12 2 1 2 111 2 1 1 22 11 11 2 22 2 2 11 1 11 1112222 111 1 1 12222 2 22 1 1111 22 2 2 1 1 11 12 22 2 1 1 1 22222 1 1 1 1 1 22222 2 1 111111111 22222 2 2 222222 1 11 22 2 22 11 11 1222222 11 2222222 222222 22 1 2 2 2 2 2 2 1 2 2 2 1 2 2 22 2 11 222222 2222222 2 2 1 2 2222222 22 2 22 2 2 2 222 2 2 2 2 22 2 2 2 22 22 2 2222 2 2 2 222222 2 2 2 2 2 22 2 2 22 2 222 222 2 2 2 2 22 2 2 2 2

−15

1

−6

−4

−2

0

2

4

6

X

Figure 3: Scatter plot of the artificial data with labels, and curves, arising from the ML-estimation (with the EM algorithm) of a finite mixture of k = 2 cubic Gaussian regressions. Plotting symbol and color for each observation is determined by the component with the maximum a posteriori probability.

7.3

Real data

The “places” data from the Places Rated Almanac (Savageau and Loftus, 1985) are a collection of nine composite variables constructed for n = 329 metropolitan areas of the United States in order to measure the quality of life. There are k = 2 groups of places, small (group 1) and large (group 2), with the first of size 303. For the current purpose, we only use the two variables X =“health care and environment” and Y =“arts and cultural facilities” measured so that the higher the score, the better. These are the two variables having the highest correlation (Kopalle and Hoffman, 1992). Figure 5 displays the scatter plot of X versus Y in both groups. In view of making clustering, the situation seems more complicated than the previous one due to a prominent overlapping between groups; this is an aspect that have to be taken into account in evaluating the quality of the clustering results. Furthermore, it is possible to see a clear parabolic functional relationship of Y on x in group 2. Regarding group 1, Table 4 shows the summary results of a polynomial regression (r = 5) as provided by the lm function of the R-package stats. At a (common) nominal level of 0.05, the only significant parameters appear to be those related to the parabolic function. Now, we will suppose to forget the true classification in small and large places and we will try to 16

  b , BIC and ICL for a polynomial Gaussian CWM. Different number of mixture Table 2: Values of 2l ψ components (k = 1, . . . , 5) and of polynomial degrees (r = 1, . . . , 5) are considered. Bold numbers highlight the best values for BIC and ICL.   b (a) 2l ψ

❍❍ r ❍❍ k ❍

1

2

3

4

5

1 2 3 4 5

-7356.441 -5855.160 -5713.095 -5672.787 -5661.108

-7317.690 -5653.833 -5634.769 -5618.283 -5616.798

-6582.602 -5636.396 -5629.208 -5599.250 -5579.644

-6552.363 -5636.224 -5584.800 -5549.372 -5548.913

-6546.713 -5633.650 -5584.004 -5549.227 -5546.247

(b) BIC

❍ ❍❍ r ❍❍ k 1 2 3 4 5

1

2

3

4

5

-7382.645 -5914.120 -5804.810 -5797.258 -5818.334

-7350.446 -5725.895 -5746.137 -5768.958 -5806.779

-6621.908 -5721.560 -5760.229 -5776.129 -5802.380

-6598.220 -5734.490 -5735.475 -5752.455 -5804.405

-6599.122 -5745.019 -5754.332 -5778.515 -5834.495

(c) ICL

❍ ❍❍ r ❍❍ k 1 2 3 4 5

1

2

3

4

5

-7382.645 -5914.120 -5847.224 -5906.636 -5923.268

-7350.446 -5726.044 -5747.712 -5914.615 -5967.415

-6621.908 -5721.563 -5873.706 -5783.383 -5851.145

-6598.220 -5734.492 -5740.202 -5759.843 -5965.444

-6599.122 -5745.198 -5754.593 -5790.514 -5876.014

Table 3: Maximum likelihood estimated parameters, obtained with the EM algorithm, for a cubic Gaussian CWM (k = 2). Standard Errors are displayed in round brackets. (a) Regression parameters

Component j = 1 βb0j

βb1j

βb2j

βb3j σ bεj

(b) Other parameters

Component j = 2

0.030 (0.270)

-8.042 (10.391)

-1.004 (0.398)

-0.090 (8.558)

0.041 (0.194)

-0.038 (2.304)

0.114 (0.028)

0.147 (0.202)

1.586 (0.056)

2.502 (0.102)

Component j = 1 π bj

µ bX|j σ bX|j

17

Component j = 2

0.571 (0.028)

0.429 (0.024)

-2.046 (0.051)

3.814 (0.039)

1.022 (0.036)

0.682 (0.027)

20

2 2

15

22 2 2 2

10

2 22 2 2 2 22 2 22 2 2 22 2 22 2 22 2 2 2 222 2 2 2 22 22 22 2 22 2 2 22 2 22 2222222 222 2 2 22222 2 22 2 2222 22 2 2 2 2 22 22 22 2 2 2 2 22222 2 2 2 2 2 22222 2 2 222222222 22222 2 2 222222 2 22 22 2 22 22 22 2222222 22 2222222 222222 22 2 2 2 2 2 2 2 2 2 2 2 2 2 2 22 2 22 222222 2222222 2 2 2 2 2222222 22 2 22 2 2 2 222 2 2 2 2 22 2 2 2 22 22 2 2222 2 2 2 222222 2 2 2 2 2 22 2 2 22 2 222 222 2 2 2 2 22 2 2 2

−5

0

Y

5

1 1 1 11 11 1 11 1 1 1 1 1 11 1 1 1 11111111 1111111 11 1 1 1 1 1 1 1 1 11 1 11111111 11111 111 1111111111111 111 1 1111111 1 1 1 11 11111111 111111111 11111111 11 1 1 1111111 1 1 111111 1111111 111 111 11 111111111111111 11111111 1 1111 11 1 1 1 111111 111 1 1 1 111111111111 1111 111 11 1 1 1 1 1 1 111111111111 1 1 11 111111 1 11 111 11 1 1 1 1 11 111111 11 1 1 111 1 11 11 11111 11 111 11 1 1 111 11 1 1 1 1 111 1 1 1 1 11 1 1 11 1 1 1 1 11 1 1 1 11 1 11 1 1 1 1 1 1 1 1 1 11 1 1

−10

2 1

2

−15

1

−6

−4

−2

0

2

4

6

X

Figure 4: Scatter plot of the artificial data with labels, and curves, arising from the ML-estimation (with the EM algorithm) of a cubic Gaussian CWM with k = 2. Table 4: Estimated parameters, and corresponding summary statistics, of a polynomial regression (of degree r = 5) fitted on group 1 via the R-function lm. β01 β11 β21 β31 β41 β51

Estimate

Std. error

t-value

p-value

0.256 4.024 -7.460 7.580 -2.762 0.328

0.705 3.215 5.033 3.373 0.993 0.104

0.363 1.251 -1.482 2.247 -2.782 3.141

0.71687 0.21176 0.13937 0.02537 0.00574 0.00185

  b , BIC, and ICL estimate it by directly considering the case k = 2. Figure 6 displays the values of 2l ψ

for the polynomial Gaussian CWM in correspondence of r ranging from 1 to 8. The best polynomial degree, according to the BIC (-1874.148) and the ICL (-1894.629), is r = 2. This result corroborates

the above considerations. Table 5 summarizes the parameter estimates, and the corresponding standard errors, for the quadratic Gaussian CWM with k = 2. The CW-plot is displayed in Figure 7. The ARI results to be 0.208; the corresponding value for a finite mixture of k = 2 quadratic Gaussian regressions is 0.146. Note that, the ARI increases up to 0.235 if the quadratic Gaussian CWM is fitted via the CEM algorithm.

18

Y

30

40

50

2

2

2 2 20

2 2

2

0

10

2 2 2 2 2 1 2 21211 1 2 1 2 1 1 2 1 11 11 111 1 1 1 12 1 1 11 21 1 111 1 1 1111 11 2111 211 2 1 1 111 21 1 11 1 1 1 1 1 11 11 1 1 1 1 1 1 1 1 1 1 111111 111111111 11 111111 1 1 1 1 1 11111 1111111111 111 1 11 11 111 1111 1111 1 1111111 11 11111 11111 11 1111 1 1 1 1111 11 1111111 11 1 1111111 11 11 11 111111 111 1 111 11 0

2

1 2

2

1

4

6

8

X

Figure 5: Scatter plot of X =“health care and environment” and Y =“arts and cultural facilities” for 303 small (denoted with 1) and 26 large (denoted with 2) metropolitan areas of the United States. Data from Savageau and Loftus (1985). −1800

3

4

5

6

7

8

2

2*loglik BIC ICL

likelihood−based functions

−1850

2

3 4

2

−1900

5

3 1

6

4

7

5

8

6 7

−1950

8

1

1

1

2

3

4

5

6

7

8

polynomial degree − r

  b , BIC and ICL, in correspondence of r = 1, . . . , 8, in the polynomial Gaussian Figure 6: Values of 2l ψ CWM.

19

Table 5: Parameters of a quadratic Gaussian CWM estimated with the EM algorithm (k = 2). Standard errors are displayed in round brackets. (a) Regression parameters

Component j = 1

(b) Other parameters

Component j = 2

0.677 (0.349)

4.503 (0.877)

0.191 (1.081)

-1.821 (0.592)

βb2j

0.892 (0.787)

1.019 (0.084)

0.866 (0.058)

2.220 (0.159)

σ bX|j

0.338 (0.039)

0.699 (0.024)

2.138 (0.136)

0.294 (0.019)

1.194 (0.082)

0.0

density of X

σ bεj

µ bX|j

0.662 (0.050)

0.4

βb1j

π bj

Component j = 2

0.8

βb0j

Component j = 1

0

2

4

6

8

30

Y

40

50

2

2

2 20

2

2 2

10

2

2

0.008

2 0.01 2 2 2 2 22 0.20132 2 2

242222 2 0.005 2 0.0029 0.0 212 0.004 2 2 2015 22222 22 2222 22 2 222 2 8 2 2 22 2012222220.2 0. 0.01 2 1 2 2 2 2 2 01 2 0. 2 7 1 1 22 1 1111 1 22 201 2 0.

0.002

0

2

22 111 112 2 2 2 11111111 11 16111 111 01.011 11 1 11 111111 11111111111 41 111 111 220.007 1111111 11 111 1611 122 22 11111 .1 11 1 111 1111 11 1111111 1111101.01112 0.003 11101111 11 11111111 111 11111 0.001

2

2 2

2

2

2

0.006

0

2

4

6

8

X

Figure 7: CW-plot of the quadratic Gaussian CWM fitted, via the EM algorithm, on the “places” data (k = 2). Plotting symbol and color for each observation is determined by the component with the maximum a posteriori probability. 7.3.1

Classification evaluation

Now, suppose to be interested in evaluating the impact of possible m labeled data, with m < n, on the classification of the remaining n − m unlabeled observations. With this aim, using the same data, we

20

have performed a simple simulation study with the following scheme. For each m ranging from 1 to 250 (250/329=0.760), we have randomly generated 500 vectors, of size m, with elements indicating the observations to consider as labeled (using the true labels for them). The discrete uniform distribution, taking values on the set {1, . . . , n}, was used as generating model. In each of the 500 replications, the quadratic Gaussian CWM (with k = 2) was fitted, with the EM algorithm, to classify the n−m unlabeled observations. Figure 8 shows the average ARI values, computed across the 500 replications only on the

0.0

0.2

0.4

ARI

0.6

0.8

1.0

n−m unlabeled observations, for each considered value of m. To give an idea of the conditional variability

0

50

100

150

200

250

m

Figure 8: Average ARI values, with respect to 500 replications, for each considered value of m (central line with the highest width). The simulated quantiles of probability 0.025, 0.05, 0.95 and 0.975, are also superimposed. of the obtained results, quantiles of probability 0.025, 0.05, 0.95 and 0.975, are also superimposed on the plot. As expected, the knowledge of the true labels for m observations tends (on average, as measured by the ARI) to improve the quality of the classification for the remaining observations when m increases.

8

Discussion and future work

An extension of the linear Gaussian CWM has been introduced which allows to model possible nonlinear relationships in each mixture component through a polynomial function. This model, named polynomial Gaussian CWM, was justified on the ground of density estimation, model-based clustering, and modelbased classification. Parameter estimation was carried out within the EM algorithm framework, and the BIC and the ICL were used for model selection. Theoretical arguments were also given showing as the posterior probabilities of component-membership arising from some well-known mixture models can be obtained by conveniently constrained the parameters of the polynomial Gaussian CWM. The proposed 21

model was then applied on a real and an artificial data set. Here, excellent clustering/classification performance was achieved when compared with existing model-based clustering techniques. Future work will follow several avenues. To begin, extension of the polynomial Gaussian CWM, to more than two variables, will be considered. This extension could initially concern only the X variable and then involve also the Y one. Afterwards, the polynomial t CWM could be introduced by simply substituting the Gaussian density with a Student-t distribution providing more robust inference for data characterized by noise and outliers (Lange et al., 1989). Finally, it could be interesting to evaluate, theoretically or by simulation, the most convenient model selection criteria for the proposed model. Note that this search, in line with Biernacki et al. (2000, p. 724), could be separately conducted according to the type of application, direct or indirect, of the model.

Acknowledgements The author sincerely thanks Salvatore Ingrassia for helpful comments and suggestions.

References Aitken, A. (1926). On Bernoulli’s numerical solution of algebraic equations. In Proceedings of the Royal Society of Edinburgh, volume 46, pages 289–305. Andrews, J., McNicholas, P., and Subedi, S. (2011). Model-based classification via mixtures of multivariate t-distributions. Computational Statistics and Data Analysis, 55(1), 520–529. Banfield, J. D. and Raftery, A. E. (1993). Model-based Gaussian and non-Gaussian clustering. Biometrics, 49(3), 803–821. Biernacki, C., Celeux, G., and Govaert, G. (2000). Assessing a mixture model for clustering with the integrated completed likelihood. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 22(7), 719–725. Biernacki, C., Celeux, G., and Govaert, G. (2003). Choosing starting values for the EM algorithm for getting the highest likelihood in multivariate Gaussian mixture models. Computational Statistics & Data Analysis, 41(3-4), 561–575. B¨ohning, D., Dietz, E., Schaub, R., Schlattmann, P., and Lindsay, B. (1994). The distribution of the likelihood ratio for mixtures of densities from the one-parameter exponential family. Annals of the Institute of Statistical Mathematics, 46(2), 373–388.

22

Boldea, O. and Magnus, J. (2009). Maximum likelihood estimation of the multivariate normal mixture model. Journal of the American Statistical Association, 104(488), 1539–1549. Byrd, R., Lu, P., Nocedal, J., and Zhu, C. (1995). A limited memory algorithm for bound constrained optimization. SIAM Journal on Scientific Computing, 16(5), 1190–1208. Celeux, G. and Govaert, G. (1992). A classification EM algorithm for clustering and two stochastic versions. Computational Statistics & Data Analysis, 14(3), 315–332. Celeux, G. and Govaert, G. (1995). Gaussian parsimonious clustering models. Pattern Recognition, 28(5), 781–793. Dasgupta, A. and Raftery, A. (1998). Detecting features in spatial point processes with clutter via model-based clustering. Journal of the American Statistical Association, 93(441), 294–302. Dean, N., Murphy, T., and Downey, G. (2006). Using unlabelled data to update classification rules with applications in food authenticity studies. Journal of the Royal Statistical Society: Series C (Applied Statistics), 55(1), 1–14. Dempster, A., Laird, N., and Rubin, D. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society. Series B (Methodological), 39(1), 1–38. Escobar, M. and West, M. (1995). Bayesian density estimation and inference using mixtures. Journal of the American Statistical Association, 90(430), 577–588. Fraley, C. and Raftery, A. (2002). Model-based clustering, discriminant analysis, and density estimation. Journal of the American Statistical Association, 97(458), 611–631. Fraley, C. and Raftery, A. E. (1998). How many clusters? which clustering method? answers via model-based cluster analysis. Computer Journal, 41(8), 578–588. Fr¨ uhwirth-Schnatter, S. (2006). Finite mixture and Markov switching models. Springer, New York. Gershenfeld, N. (1997). Nonlinear inference and cluster-weighted modeling. Annals of the New York Academy of Sciences, 808(1), 18–24. Greselin, F. and Ingrassia, S. (2010). Constrained monotone EM algorithms for mixtures of multivariate t distributions. Statistics and computing, 20(1), 9–22. Gr¨ un, B. and Leisch, F. (2008). Flexmix version 2: Finite mixtures with concomitant variables and varying and constant parameters. Journal of Statistical Software, 28(4), 1–35.

23

Gutierrez, R., Carroll, R., Wang, N., Lee, G., and Taylor, B. (1995). Analysis of tomato root initiation using a normal mixture distribution. Biometrics, 51(4), 1461–1468. Hosmer Jr., D. (1973). A comparison of iterative maximum likelihood estimates of the parameters of a mixture of two normal distributions under three different types of sample. Biometrics, 29(4), 761–770. Hubert, L. and Arabie, P. (1985). Comparing partitions. Journal of Classification, 2(1), 193–218. Ingrassia, S., Minotti, S. C., and Vittadini, G. (2012a). Local statistical modeling via the cluster-weighted approach with elliptical distributions. Journal of Classification, 29(3), ..–.. Ingrassia, S., Minotti, S. C., and Punzo, A. (2012b). Model-based clustering via linear cluster-weighted models. arXiv.org e-print 1206.3974, http://arxiv.org/abs/1206.3974. Karlis, D. and Santourian, A. (2009). Model-based clustering with non-elliptically contoured distributions. Statistics and Computing, 19(1), 73–83. Karlis, D. and Xekalaki, E. (2003). Choosing initial values for the EM algorithm for finite mixtures. Computational Statistics & Data Analysis, 41(3–4), 577–590. Kass, R. and Raftery, A. (1995). Bayes factors. Journal of the American Statistical Association, 90(430), 773–795. Keribin, C. (2000). Consistent estimation of the order of mixture models. Sankhy¯ a: The Indian Journal of Statistics, Series A, 62(1), 49–66. Kopalle, P. and Hoffman, D. (1992). Generalizing the sensitivity conditions in an overall index of product quality. Journal of Consumer Research, 18(4), 530–535. Lange, K. L., Little, R. J. A., and Taylor, J. M. G. (1989). Robust statistical modeling using the t distribution. Journal of the American Statistical Association, 84(408), 881–896. Leisch, F. (2004). Flexmix: A general framework for finite mixture models and latent class regression in R. Journal of Statistical Software, 11(8), 1–18. Leroux, B. (1992). Consistent estimation of a mixing distribution. The Annals of Statistics, 20(3), 1350–1360. Lin, T. (2009). Maximum likelihood estimation for multivariate skew normal mixture models. Journal of Multivariate Analysis, 100(2), 257–265. Lin, T. (2010). Robust mixture modeling using multivariate skew t distributions. Statistics and Computing, 20(3), 343–356. 24

Lo, K., Brinkman, R., and Gottardo, R. (2008). Automated gating of flow cytometry data via robust model-based clustering. Cytometry Part A, 73(4), 321–332. Louis, T. (1982). Finding the observed information matrix when using the EM algorithm. Journal of the Royal Statistical Society. Series B (Methodological), 44(2), 226–233. McLachlan, G. and Peel, D. (1998). Robust cluster analysis via mixtures of multivariate t-distributions. In A. Amin, D. Dori, P. Pudil, and H. Freeman, editors, Advances in Pattern Recognition, volume 1451 of Lecture Notes in Computer Science, pages 658–666. Springer Berlin - Heidelberg. McLachlan, G. J. and Basford, K. E. (1988). Mixture models: Inference and Applications to clustering. Marcel Dekker, New York. McLachlan, G. J. and Peel, D. (2000). Finite Mixture Models. John Wiley & Sons, New York. McNicholas, P. (2010). Model-based classification using latent gaussian mixture models. Journal of Statistical Planning and Inference, 140(5), 1175–1181. McNicholas, P. and Murphy, T. (2008). Parsimonious Gaussian mixture models. Statistics and Computing, 18(3), 285–296. Peel, D. and McLachlan, G. (2000). Robust mixture modelling using the t distribution. Statistics and Computing, 10(4), 339–348. Rand, W. (1971). Objective criteria for the evaluation of clustering methods. Journal of the American Statistical Association, 66(336), 846–850. R Development Core Team (2011). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. Savageau, D. and Loftus, G. (1985). Places Rated Almanac: your guide to finding the best places to live in America. Rand McNally & Company, Chicago. Schlattmann, P. (2009). Medical Applications of Finite Mixture Models. Springer-Verlag. Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics, 6(2), 461–464. Titterington, D. M., Smith, A. F. M., and Makov, U. E. (1985). Statistical Analysis of Finite Mixture Distributions. John Wiley & Sons, New York. Wedel, M. and Kamakura, W. (2001). Market segmentation: Conceptual and methodological foundations (2nd edition). Kluwer Academic Publishers, Boston, MA, USA.

25