1
Learning Topic Models by Belief Propagation
arXiv:1109.3437v3 [cs.LG] 3 Oct 2011
Jia Zeng, William K. Cheung and Jiming Liu Abstract—Latent Dirichlet allocation (LDA) is an important class of hierarchical Bayesian models for probabilistic topic modeling, which attracts worldwide interests and touches many important applications in text mining, computer vision and computational biology. This paper proposes a novel factor graph representation for LDA within the Markov random field (MRF) framework, which enables the classic loopy belief propagation (BP) algorithm for approximate inference and parameter estimation. Although two commonly-used approximate inference methods, such as variational Bayes (VB) and collapsed Gibbs sampling (GS), have gained great successes in learning LDA, the proposed BP is competitive in both speed and accuracy validated by encouraging experimental results on four large-scale document data sets. Furthermore, the BP algorithm has the potential to become a generic learning scheme for variants of LDA-based topic models. To this end, we show how to learn two typical variants of LDA-based topic models, such as author-topic models (ATM) and relational topic models (RTM), using belief propagation based on the factor graph representation. Index Terms—Latent Dirichlet allocation, topic models, belief propagation, message passing, factor graph, Bayesian networks, Markov random fields, hierarchical Bayesian models, Gibbs sampling, variational Bayes.
✦
1
I NTRODUCTION
Latent Dirichlet allocation (LDA) [1] is a three-level hierarchical Bayesian model (HBM) that is able to infer probabilistic word clusters called topics from a copus of documents. Fig. 1 shows the generative graphical representation of LDA. It uses the document-specific topic proportion θd (k) to generates a topic label index zn = k, 1 ≤ k ≤ K, which generates each observed word index 1 ≤ wn ≤ W in the document d based on the topic-specific multinomial distribution φk (w) over the vocabulary words. Both multinomial parameters θd (k) and φk (w) are generated by two Dirichlet distributions with hyperparameters α and β, respectively. The plates indicate replications. For example, the document repeats D times in the corpus, the word repeats Nd times in the document d, the vocabulary size is W , and there are K topics. Table 1 summarizes important notations. Through allocating the topic labels z = {zn } to the observed word indices w = {wn } in the corpus, LDA solves the topic modeling problem by partitioning all word indices w into K probabilistic topics according to three intrinsic assumptions: 1) Different word indices within the same document tend to be associated with the same topic labels. 2) The same word indices in different documents are likely be associated with the same topic labels. 3) All word indices do not tend to associate with the same topic labels. • J. Zeng is with the School of Computer Science and Technology, Soochow University, Suzhou 215006, China. He is also with the Shanghai Key Laboratory of Intelligent Information Processing, China, and the Center for Bioinformatics and Computational Biology, University of Maryland, College Park, MD 20742, USA. To whom correspondence should be addressed. E-mail:
[email protected]. • W. K. Cheung and J. Liu are with the Department of Computer Science, Hong Kong Baptist University, Kowloon Tong, Hong Kong.
α
θd
zn
wn
φk
β
N D
K
Fig. 1. Generative graphical representation of LDA [1]. TABLE 1 Notations 1≤d≤D 1≤w≤W 1≤k≤K 1≤a≤A 1≤c≤C w = {w, d} z = {zw,d } z−w,d zw,−d ad µ(z·,d ) µ(zw,· ) θd φw ηc f (·) α, β
Document index Word index in vocabulary Topic index Author index Link index Bag of words Topic labels for words Labels for d excluding w Labels for w excluding d Coauthors of the document d P Pw µ(zw,d ) d µ(zw,d ) Factor of the document d Factor of the word w Factor of the link c Factor functions Dirichlet hyperparameters
The first assumption is determined by the documentspecific topic proportion θd (k), where it is more likely to assign a topic label zn = k to the word index wn if the topic k is more frequently assigned to other word indices in the document d. Similarly, the second assumption is based on the topic-specific multinomial distribution φk (w). Given the observed word index w in the document d, the parameter φk (w) implies a higher likelihood to associate the word index w with the topic label zn = k if k is more frequently assigned to the same word index w in other documents except d. The third
2
assumption avoids grouping all word indices w into one topic through normalizing φk (w) in terms of all word indices. If most word indices are associated with the topic k, the multinomial parameter φk will become too small to further allocate the topic k to these word indices. Indeed, the first two assumptions of topic modeling encode the labeling smoothness prior, i.e., the topic labels for different word indices within the same document or for the same word indices in different documents tend to be the same. Such prior as the smoothness has been widely used in image analysis and computer vision within the Markov random field (MRF) [2] framework. LDA has no exact inference methods because it contains cycles or loops in Fig. 1. Recently, variational Bayes (VB) [1] and collapsed Gibbs sampling (GS) [3] have been two commonly-used approximate inference methods for LDA as well as variants of LDA-based topic models. Other approximate inference methods include expectation-propagation (EP) [4] and collapsed VB inference (CVB) [5]. The connections and empirical comparisons among these approximate inference methods for learning LDA can be found in [6]. Alternatively, this paper casts the topic modeling as a labeling problem, in which the objective is to assign a set of semantic topic labels z = {zw,d } to explain the observed bag-of-word (BOW) indices w = {w, d}. As an undirected graphical model, MRF solves the labeling problem existing widely in image analysis and computer vision by two important concepts: neighborhood systems and clique potentials [2], [7], [8]. As far as topic modeling is concerned, MRF assigns the best topic labels according to the maximum a posteriori (MAP) estimation through maximizing the posterior probability p(z|w), which is in nature a prohibited combinatorial optimization problem in the discrete topic space. However, MRF often employs the smoothness prior [2] over neighboring topic labels to reduce the complexity by constraining the total number of possible labeling configurations. From the novel MRF perspective, we transform the generative graphical representation of LDA into a novel factor graph [7], [9], and present the classic loopy belief propagation (BP) algorithm for approximate inference and parameter estimation. First, we define the neighborhood system of the topic label zw,d as z−w,d and zw,−d , where the notation z−w,d denotes a set of topic labels associated with all word indices in the document d except w, and the notation zw,−d denotes a set of topic labels associated with the word indices w in all documents except d. Second, through designing proper factor functions or clique potentials, we encourage or penalize different local labeling configurations in the neighborhood system in order to realize three intrinsic assumptions of topic modeling. For example, we may encourage the topic labeling smoothness among neighbors {zw,d, z−w,d , zw,−d }. Indeed, the factor graph is able to represent a generative model. For example, the directed factor graph [10] enhances the visual language to represent LDA. So, the factor graph may be a more generic graphical repre-
sentation of various generative models in real-world applications. Because the BP algorithm operates well on the factor graph [7], it has the potential to become a generic learning scheme for learning variants of LDAbased topic models. Without loss of generality, we extend the proposed BP algorithm to learn two typical variants of LDA-based topic models: author-topic models (ATM) [11] and relational topic models (RTM) [12]. These extensions include novel factor graph representations and purpose-driven factor functions. Recently, HBMs have found many important applications in text mining and computer vision, for example, tracking historical topics from time-stamped documents [13] and activity perception in crowded and complicated scenes [14]. The factor graph representation of LDA reveals an underlying equivalence between HBM and MRF, where the former emphasizes more on the conditional dependencies, but the latter stresses more on the joint dependencies between connected hidden variables. Indeed, HBM can be formulated as a directed graphical model within the Bayesian network framework [15], which relates to the undirected graphical model (MRF) but with causal dependencies between hidden variables. Although learning HBM often has difficulty in estimating parameters and inferring hidden variables due to the causal coupling effects, the alternative factor graph representation as well as the BP-based learning scheme may shed more light on faster and more accurate learning algorithms for HBM. The rest of this paper is organized as follows. Section 2 introduces the novel factor graph representation for LDA, and derives the loopy BP algorithm for approximate inference and parameter estimation. Moreover, we discuss the intrinsic relations between BP and other state-of-the-art approximate inference algorithms. Sections 3 and 4 present how to learn ATM and RTM using the BP algorithm. Section 5 validates the BP algorithm on four large-scale document data sets. Finally, Section 6 draws conclusions and envisions future work.
2
B ELIEF P ROPAGATION
FOR
LDA
In this section we present the factor graph representation of LDA and develop the BP-based learning algorithm. Detailed discussions are provided on relations between BP and other state-of-the-art approximate inference methods. From the MRF perspective, we transform the generative graphical representation of LDA in Fig. 1 to the factor graph in Fig. 2. The factors θd and φw are denoted by squares, and their connected variables zw,d are denoted by circles. The factors θd and φw define the same neighborhood system as Fig. 1. The factor θd connects topic labels on different word indices {zw,d, z−w,d } within the same document, while the factor φw connects topic labels on the same word indices {zw,d, zw,−d } but in different documents. Note that we absorb the observed word w as the index of φw , which is similar to absorbing
3
µθd →zw,d α
θd
zw,d
µzw,d ←φw φw
µ(z−w,d )
β µ(zw,−d )
z−w,d
zw,−d
Fig. 2. Factor graph and message passing of LDA.
the observed document d as the index of θd in Fig. 1. Because the factors can be parameterized functions [7], both θd and φw can be the same multinomial parameters in LDA. The hyperparameters α and β connected with the factor nodes can be viewed as pseudo-counts [15] of topic labels regularizing the topic configuration. This resembles the collapsed GS [3] that integrates out parameter variables θ and φ and treats hyperparameters α and β as pseudo topic counts in order to perform inference on the collapsed hidden variable space z. Figs. 1 and 2 reflect two facets of LDA, in which the former focuses more on generating the observed words w, while the latter emphasizes more on the topic labeling configuration z. As far as topic modeling is concerned, Fig. 2 is equivalent to Fig. 1 with two main reasons. First, both Figs. 1 and 2 have the same neighborhood system because the connection of hidden variables remains the same. Second, in the next subsection, we shall design specific factor functions to realize three intrinsic assumptions of topic modeling as discussed in Section 1. Note that both Figs. 1 and 2 contain loops so that the exact inference method is intractable. The conversion from Fig. 1 to 2 has also revealed the intrinsic relation between HBM and MRF. HBM explicitly represents the conditional dependencies of observed and hidden variables, which makes the posterior probability of hidden variables difficult to be factorized. By contrast, MRF can factorize the joint distribution of hidden variables into the product of factors according to the Hammersley-Clifford theorem [16], which facilitates the loopy BP algorithm for approximate inference and parameter estimation. Although the convergence of loopy BP is not ensured theoretically, it often converges and works well for general graphs in real-world applications. For the shorthand statement, we will simply call the “BP” algorithm and omit the word “loopy” in the rest of the paper. Fig. 2 assumes that all word tokens at index {w, d} have the same topic label zw,d with different probabilities. For example, in the document d, the word “apple” occurs twice, and we assume that these two tokens have the same topic label but with different probabilities. This assumption contradicts LDA in Fig. 1 because LDA assumes two “apple” tokens in the same document may have two different topic labels. However, under the BOW representation, LDA cannot distinguish the topic
labels for two “apple” tokens in the same document because there is no contextual information. For example, LDA does not know which “apple” token belongs to the topic “Fruit” or the topic “Apple Computers” in the same document. Hence, the effect of different topic labels for all “apple” tokens equals that of the same topic label for the index “apple” but with different probabilities multiplied by the number of word counts. Experiments show that this assumption is reasonable and does not decrease the topic modeling performance. Note that allocating topic labels to each word index {w, d} rather than each word token will significantly reduce the computational cost, because the average number of word indices Wd is often much smaller than the total number of word tokens Nd per document due to the word sparsity, i.e., Wd ≪ Nd . 2.1 Inference and Parameter Estimation The BP [7, Chapter 8] algorithms such as the sumproduct and the max-sum algorithms [9] provide exact solutions for inference problems in tree-structured factor graphs but approximate solutions in factor graphs with loops. Rather than directly computing the posterior joint probability p(z|w), we turn to the posterior marginal probability p(zw,d ), referred to as message µ(zw,d ), which can be normalized using a local computation. Messages are passed from variables to factors, and in turn from factors to variables until convergence or the maximum number of iterations. To implement the BP algorithm, we have to make two decisions [17]. First, we must choose either the sum-product or the max-sum algorithm to compute the marginal probabilities. Second, we must choose either the synchronous or the asynchronous message update schedule to propagate messages within the neighborhood system. In this paper we adopt the sum-product algorithm to infer µ(zw,d ). Fig. 2 shows the message passing from two factors θd and φw to the variable zw,d . The message µ(zw,d ) is proportional to the product of both incoming messages from factors, µ(zw,d ) ∝ µθd →zw,d (zw,d ) × µφw →zw,d (zw,d ),
(1)
where we use the arrows to denote the message passing directions. The messages from factors to variables are sum of all incoming messages from the neighboring variables as follows, X Y µθd →zw,d (zw,d ) = fθd µ(z−w,d )α, (2) −w
z−w,d
µφw →zw,d (zw,d ) =
X
zw,−d
f φw
Y
µ(zw,−d )β,
(3)
−d
where α and β can be viewed as the pseudo-messages, −w and −d denote all word indices except w and all document indices except d, the notations z−w,d and zw,−d represent all possible neighboring labeling configurations, and fθd and fφw are the factor functions that encourage or penalize the incoming messages.
4
Based on the smoothness of topic structural dependencies, we encourage only K smooth topic configurations without summing up all possible configurations z−w,d and zw,−d in (2) and (3), which can hence be further simplified as Y µθd →zw,d (zw,d = k) = fθd µ(z−w,d = k)α, (4)
input : w, K, T, α, β. output : θd , φw . µ1w,d (k) ← initialization and normalization; for t ← 1 to T do P µt+1 w,d (k) ∝
µt−w,d(k) + α t k [µ−w,d (k)
+ α]
×P
µtw,−d (k) + β t w [µw,−d (k)
+ β]
;
−w
µφw →zw,d (zw,d = k) = fφw
Y
µ(zw,−d = k)β.
(5)
−d
In practice, however, Eqs. (4) and (5) often cause the product of multiple incoming messages close to zero [8]. To avoid arithmetic underflow, we use the sum operation rather than the product operation of incoming messages because when the product value increases the sum value also increases, Y X µ(z−w,d )α ∝ µ(z−w,d ) + α, (6) −w
Y
−w
µ(zw,−d )β ∝
−d
X
µ(zw,−d ) + β.
(7)
−d
Such approximations as (6) and (7) transform the sumproduct to the sum-sum algorithm, which resembles the relaxation labeling algorithm for learning MRF with good performance [8]. For convenience, we use the P shorthand notations µ(z ) = µ(z ), µ(zw,· ) = ·,d w,d w P P µ(z ), and µ(z µ(z ), µ(z ) = −w,d w,−d ) = w,d −w,d −w Pd µ(z ) in the subsequent formulas. w,−d −d In the MRF model, we can design the factor functions arbitrarily to encourage or penalize local topic labeling configurations based on our prior knowledge. Based on the topic smoothness prior, we design fθd and fφw as 1 , [µ(z −w,d = k) + α] k 1 = P . w [µ(zw,−d = k) + β]
fθd = P f φw
(8) (9)
Eq. (8) normalizes the incoming messages by the total number of messages for all topics associated with the document d in order to make outgoing messages comparable across documents. Eq. (9) normalizes the incoming messages by the total number of messages for all words in the vocabulary in order to make outgoing messages comparable across vocabulary words. Note that (4) and (5) realize the first two assumptions, and (9) encodes the third assumption of topic modeling discussed in Section 1. Combining (1) to (9) yields the complete message update equation,1 µ(z−w,d = k) + α µ(zw,d = k) ∝ P k [µ(z−w,d = k) + α] µ(zw,−d = k) + β , ×P w [µ(zw,−d = k) + β]
(10)
where the updated P message is normalized locally in terms of k, i.e., k µ(zw,d = k) = 1. In practice, this 1. For short,
P
zw,d =k
µ(zw,d = k) =
P
k
µ(zw,d = k).
end P θd (k) ← [µ·,d (k) + α]/ Pk [µ·,d (k) + α]; φw (k) ← [µw,· (k) + β]/ w [µw,· (k) + β];
Fig. 3. The synchronous BP for LDA. updated message will converge on the factor graph in Fig. 2 after finite iterations. The message µ(zw,d ) is multiplied by the number of word counts xw,d during propagation, i.e., µ(zw,d )xw,d . Thus, xw,d can be viewed as the weight of µ(zw,d ) during propagation, where the bigger xw,d corresponds to the larger influence of its message to those of its neighbors. In this way, the topics may be distorted by those documents with greater word counts. To avoid this problem, we may choose word weights like term frequency (TF) or term frequency-inverse document frequency (TF-IDF) for weighted message passing or belief propagation. Fig. 3 shows the synchronous message update schedule.2 At each iteration t, each variable uses the incoming messages in the previous iteration t − 1 to calculate the current message. Once every variable computes its message, the message is passed to the neighboring variables and used to compute messages in the next iteration t + 1. An alternative is the asynchronous message update schedule. It updates the message of each variable immediately. The updated message is immediately used to compute other neighboring messages at each iteration t. The asynchronous update schedule often passes messages faster across variables, which causes the BP algorithm converge faster than the synchronous update schedule. Another termination condition is that the change of the training perplexity [1] is less than a predefined threshold. To estimate parameters θ and φ, we use (4) and (5) by the product of the factor functions and the incoming messages from the variables including µ(zw,d ), µ(z·,d = k) + α θd (k) = P , k [µ(z·,d = k) + α] µ(zw,· = k) + β φw (k) = P , w [µ(zw,· = k) + β]
(11) (12)
which are actually the joint marginal distributions of the set of variables belonging to factors θd and φw . Alternatively, we can derive (11) and (12) by the expectationmaximization (EM) algorithm [18]. The E-step infers the normalized marginal posterior probability µ(zw,d). Employing the Dirichlet-Multinomial conjugacy and Bayes’ 2. For short, µ(zw,d = k) = µw,d (k).
5
rule [15], the marginal Dirichlet distributions on parameters are p(θd ) = Dir(θd |µ(z·,d ) + α),
(13)
p(φw ) = Dir(φw |µ(zw,· ) + β),
(14)
The M-step maximizes (13) and (14) with respect to θd and φw , resulting in the same point estimates of parameters (11) and (12). In real-world applications, the asymmetric Dirichlet prior α over θ may have substantial advantages over the symmetric prior [19], while the asymmetric prior β over φ may not. Based on the inferred messages, we may estimate the asymmetric unconstrained K-tuple hyperparameter αk using a fixed-point iteration schedule for an ML estimation [15], [20], P αtk [ d Ψ[µ(z·,d = k) + αtk ] − DΨ(αtk )] t+1 P t , P P αk ← t k [µ(z·,d = k) + αk ]) − DΨ( k αk ) d Ψ( where Ψ(·) is the digamma function. After initialization of αk , we iterate this equation until αk converges. The estimation of W -tuple hyperparameter βw is analogous to that of αk . More details on estimating the symmetric hyperparameters can be found in [15], [20]. Alternatively, we can also use the Newton-Raphson method [1] to estimate these hyperparameters. Generally, the hyperparameters determine the sparseness of multinomial parameters θ and φ that influence the topic modeling performance. However, for simplicity, we assume that the hyperparameters are symmetric and are provided by users as prior knowledge in Fig. 3. In many cases, we often use the smoothed LDA with fixed symmetric Dirichlet hyperparameters α = 50/K and β = 0.01 [3] in order to avoid the complex full Bayesian inference of α and β, respectively. 2.2 The Simplified BP (siBP) Algorithm We may simplify the message update equation in Fig. 3. Substituting (11) and (12) into (10) gives the approximate message update equation, µ(zw,d = k) ∝ θd (k) × φw (k),
(15)
which includes the current message µ(zw,d = k) in both numerator and denominator in (10). In many real-world topic modeling problems, a document often contains many different word indices, and the same word index appears in many different documents. So, at each iteration, Eq. (15) deviates slightly from (10) after adding the current message to both numerator and denominator. Such slight difference may be enlarged after many iterations in Fig. 3 due to accumulation effects, leading to different estimated parameters. Intuitively, Eq. (15) implies that if the topic k has a higher proportion in the document d, and it has the a higher likelihood to generate the word index w, it is more likely to allocate the topic k to the observed word {w, d}. This allocation
function [phi, theta] = siBP(X, K, T, ALPHA, BETA) % % % % % % % % % % % % % %
X is a W*D sparse matrix. W is the vocabulary size. D is the number of documents. The element of X is the word count 'xi'. 'wi' and 'di' are word and document indices. K is the number of topics. T is the number of iterations. mu is a matrix with K rows for topic messages. phi is a K*W matrix. theta is a K*D matrix. ALPHA and BETA are hyperparameters. normalize(A,dim) returns the normalized values (sum to one) of the elements along different dimensions of an array.
[wi,di,xi] = find(X); % random initialization mu = normalize(rand(K,nnz(X)),1); % simplified belief propagation for t = 1:T for k = 1:K md(k,:) = accumarray(di,xi'.*mu(k,:)); mw(k,:) = accumarray(wi,xi'.*mu(k,:)); end theta = normalize(md+ALPHA,1); %Eq.(11) phi = normalize(mw+BETA,2); %Eq.(12) mu = normalize(theta(:,di).*phi(:,wi),1); %Eq.(15) end return
Fig. 4. The MATLAB code for siBP.
scheme in principle follows three intrinsic topic modeling assumptions in Section 1. Fig. 4 shows the MATLAB code for the simplified BP (siBP). Alternatively, we derive (15) by the Bayes’ rule. To best generate the observed words {w, d}, the objective of MRF is to maximize the joint probability conditioned on a set of parameters {θ, φ, α, β}, p(w, d, z|θ, φ, α, β) = p(w|z, φ, β)p(z|d, θ, α).
(16)
If we absorb the observed word {w, d} into the index of hidden variable z, the joint probability (16) becomes the message µ(zw,d ). Again, employing the Bayes’ rule, we represent the first term in (16) as p(z|w, φ, β)p(w|φ, β) p(w|z, φ, β) = P w p(z|w, φ, β)p(w|φ, β) p(z|w, φ, β) , ∝P w p(z|w, φ, β)
(17)
because p(w|φ, β) is a constant in terms of z. According to the topic smoothness prior, combining (16) and (17) obtains the following message update equation, p(z = k|φw , β) µ(zw,d = k) ∝ p(z = k|θd , α) P , w p(z = k|φw , β)
(18)
which equals (15) because the first term equals (11), the conditional probability to generate the topic label z given the document-specific proportion θd and the hyperparameter α, and the second term equals (12), the conditional probability to generate the topic label z given the word-specific topic proportion φw and the hyperparameter β normalized by the sum of all such probabilities
6
across vocabulary words. Since (18) contains all probabilistic terms, Eqs. (8) and (9) are reasonably designed as the normalization factors of incoming messages. 2.3 Discussions We discuss some intrinsic relations between BP with three state-of-the-art LDA learning algorithms such as VB [1], GS [3], and zero-order approximation CVB (CVB0) [5], [6] within the unified message passing framework. The message update scheme is an instantiation of the E-step of the EM algorithm [18], which has been widely used to infer the marginal probabilities of hidden variables in various graphical models according to the maximum-likelihood estimation [7] (e.g., the E-step inference for Gaussian mixture models (GMM) [21], the forward-backward algorithm for hidden Markov models (HMM) [22], and the probabilistic relaxation labeling algorithm for MRF [23]). After the E-step, we can identify the optimal parameters based on the updated messages and observations at the M-step of the EM algorithm. VB is a variational message passing method [24] that uses a set of factorized variational distributions to approximate the posterior distribution by minimizing the Kullback-Leibler (KL) divergence between them. Employing the Jensen’s inequality makes the approximate variational distribution an adjustable lower bound on the posterior distribution, so that maximizing the posterior probability is equivalent to maximizing the lower bound by tuning a set of variational parameters. Hence, we can exploit standard nonlinear optimization techniques to determine the optimal values for the variational parameters. Because there is always a gap between the approximate variational distribution and the true posterior distribution, VB intrinsically introduces bias when learning LDA. The variational message update equation resembles (15) but with two major differences. First, VB updates variational messages based on complicated digamma functions, which introduce bias in updated variational messages but may be corrected by proper settings of hyperparameters [6]. Second, VB uses a different variational EM update schedule. At the E-step, it simultaneously updates both variational messages and parameter of θ until convergence, holding the variational parameter of φ fixed. At the M-step, VB updates the variational parameter of φ given the variational messages. Eq. (15) also reveals that siBP is in nature a probabilistic matrix factorization algorithm that factorizes the word count matrix, X = [xw,d ]W ×D , into a matrix of document-specific topic proportions, Θ = [θd (k)]K×D , and a matrix of vocabulary word-specific topic proportions, Φ = [φw (k)]K×W , i.e., X ∼ ΦT Θ. From this point of view, the multinomial principle component analysis (PCA) [25] interprets intrinsic relations among LDA, probabilistic latent semantic analysis (PLSA) [26], and non-negative matrix factorization (NMF) [27]. In particular, Eq. (15) is the same as the E-step update for PLSA except that the parameters θ and φ are smoothed by
the hyperparameters α and β [6] to prevent overfitting problems. GS resembles the asynchronous BP implementation but with two subtle differences. First, it randomly samples the current topic label zn = k conditioned on other neighboring topic labels from the message p(zn |z−n ), which truncates all K-tuple message values to zeros except the sampled topic label k with p(zn = k|z−n ) = 1. Such information loss introduces bias when learning LDA. Second, GS must sample a topic label for each word token instead of index, which significantly restricts its applicability to large-scale document repositories containing billions of word tokens. CVB0 is exactly equivalent to our asynchronous BP implementation but based on word tokens. Previous empirical comparisons [6] advocated the CVB0 algorithm for LDA but without detailed theoretical explanations. In this paper we explain that the superior performance of CVB0 has been largely attributed to its asynchronous BP implementation from the MRF perspective. Our experiments also support that the message passing over word indices instead of tokens will produce comparable or even better topic modeling performance but with significantly smaller computational costs. At each iteration, VB, BP and siBP have the computational complexity O(KDWd ), but GS and CVB0 require O(KDNd ), where Wd is the average vocabulary size and Nd is the average number of word tokens per document.
3
B ELIEF P ROPAGATION
FOR
ATM
Author-topic models (ATM) [11] depict each author of the document as a mixture of probabilistic topics, and have found important applications in matching papers with reviewers [28]. Fig. 5A shows the generative graphical representation for ATM, which first uses a documentspecific uniform distribution ud to generate an author index a, 1 ≤ a ≤ A, and then uses the author-specific topic proportions θa to generate a topic label zn = k for the word index wn in the document d. The plate on θ indicates that there are A unique authors in the corpus. The document often has multiple coauthors. ATM randomly assigns one of the observed author indices to each word in the document based on the documentspecific uniform distribution ud . However, it is more reasonable that each word {w, d} is associated with an author index a ∈ ad from the multinomial rather than uniform distribution, where ad is a set of author indices of the document d. As a result, each word label takes two variables zw,d = {a, k}, a ∈ ad , where a is the author index and k is the topic index attached to the word. We transform Fig. 5A to the factor graph representation of ATM in Fig. 5B. Similar to Fig. 2, we absorb the observed author index a ∈ ad of the document d as the index of the factor θa∈ad . The notation z−w,· denotes all labels connected with the authors a ∈ ad except those for the word index w. The only difference between ATM and LDA is that the author a ∈ ad instead of
7
ud
D
a
µθa →zw,d
α A
Nd zn
α
θa∈ad
zw,d
φw
β
θ µ(z−w,· )
wn
z−w,·
φk
zw,−d
K (B)
(A)
Fig. 5. (A) Generative graphical representation [11] and (B) Factor graph and message passing of ATM.
input : w, ad , K, T, α, β. output : θa , φw . µ1w,d (a, k), a ∈ ad ← initialization and normalization; for t ← 1 to T do µt+1 w,d (a, k) ∝ P
µtw,−d (k) + β µt−w,· (a, k) + α P × ; t t k [µ−w,· (a, k) + α] w [µw,−d (k) + β]
end P θa (k) ← [µ·,· (a, k) + α]/P k [µ·,· (a, k) + α]; φw (k) ← [µw,· (k) + β]/ w [µw,· (k) + β];
Fig. 6. The synchronous BP for ATM.
the document d connects the labels zw,d and z−w,· . As a result, ATM encourages topic smoothness among labels zw,d = {a, k} attached to the same author a instead of the same document d. 3.1 Inference and Parameter Estimation Unlike passing the K-tuple message µ(zw,d = k) in Fig. 3, the BP algorithm for learning ATM passes the |ad | × Ktuple message vectors µ(zw,d = {a, k}), a ∈ ad through the factor θa∈ad in Fig. 5B, where |ad | is the number of authors in the document d. Nevertheless, we can still obtain the K-tuple word topic message µ(zw,d = k) by marginalizing the message µ(zw,d = {a, k}) in terms of the author variable a ∈ ad as follows, X µ(zw,d = k) = µ(zw,d = {a, k}). (19) a∈ad
Since Figs. 2 and 5B have the same right half part, the message passing equation from the factor φw to the variable zw,d and the parameter estimation equation for φw in Fig. 5B remain the same as (10) and (12) based on the marginalized word topic message in (19). Thus, we only need to derive the message passing equation from the factor θa∈ad to the variable zw,d in Fig. 5B. Because of the topic smoothness prior, we design the factor function
as follows, fθa = P
1 , [µ(z = {a, k}) + α] −w,· k
(20)
where µ(z−w,· = {a, k}) denotes the sum of all incoming messages attached to the author index a and the topic index k excluding µ(zw,d ). Likewise, Eq. (20) normalizes the incoming messages attached the author index a in terms of the topic index k in order to make outgoing messages comparable for different authors a ∈ ad . Similar to (4), we derive the message passing µfθa →zw,d through adding all incoming messages evaluated by the factor function (20). Multiplying two messages from factors θa∈ad and φw yields the message update equation as follows, µ(z−w,· = {a, k}) + α µ(zw,d = {a, k}) ∝ P k [µ(z−w,· = {a, k}) + α] µ(zw,−d = k) + β . ×P w [µ(zw,−d = k) + β]
(21)
Note that the |ad | × K-tuple message µ(zw,d = {a, k}) is normalized in terms of all combinations of {a, k}, a ∈ ad . Based on the normalized messages, the author-specific topic proportion θa (k) can be estimated from the sum of all incoming messages including µ(zw,d ) evaluated by the factor function fθa as follows, θa (k) = P
µ(z·,· = {a, k}) + α . k [µ(z·,· = {a, k}) + α]
(22)
As a summary, Fig. 6 shows the synchronous BP algorithm for learning ATM.3 The difference between Fig. 3 and Fig. 6 is that Fig. 3 considers the author index a as the label for each word. At each iteration, the computational complexity is O(KDWd Ad ), where Ad is the average number of authors per document. 3. For short, µ(zw,d = {a, k}) = µw,d (a, k).
8
z·,d′
α D
µz·,d′ →ηc
D θd
ηc
θd ′
η Nd
µηc →zw,d
Nd′
zd,n
c
wd,n
φk
zd′ ,n
α
wd′ ,n K
(A)
θd
zw,d
z−w,d
β
φw
zw,−d (B)
Fig. 7. (A) Generative graphical representation [12] and (B) Factor graph and message passing of RTM.
4
B ELIEF P ROPAGATION
FOR
RTM
Network data, such as citation and coauthor networks of documents [28], [29], tag networks of documents and images [30], hyperlinked networks of web pages, and social networks of friends, exist pervasively in data mining and machine learning. The probabilistic relational topic modeling of network data can provide both useful predictive models and descriptive statistics [12]. In Fig. 7A, relational topic models (RTM) [12] represent entire document topics by the mean value of the document topic proportions, and use Hadamard product of mean values z d ◦z d′ from two linked documents {d, d′ } as link features, which are learned by the generalized linear model (GLM) η to generate the observed binary citation link variable c = 1. Besides, all other parts in RTM remain the same as LDA. We transform Fig. 7A to the factor graph Fig. 7B by absorbing the observed link index c ∈ c, 1 ≤ c ≤ C as the index of the factor ηc . Each link index connects a document pair {d, d′ }, and the factor ηc connects word topic labels zw,d and z·,d′ of the document pair. Besides encoding the topic smoothness prior, RTM explicitly describes the topic structural dependencies between the pair of linked documents {d, d′ } using the factor function fηc (·). 4.1 Inference and Parameter Estimation In Fig. 7, the messages from the factors θd and φw to the variable zw,d are the same as LDA in (4) and (5). Thus, we only need to derive the message passing equation from the factor ηc to the variable zw,d. We design the factor function fηc (·) for linked documents as follows, P ′ {d,d′ } µ(z·,d = k)µ(z·,d′ = k ) ′ , (23) fηc (k|k ) = P ′ {d,d′ },k′ µ(z·,d = k)µ(z·,d′ = k ) which depicts the likelihood of topic label k assigned to the document d when its linked document d′ is associated with the topic label k ′ . Note that the designed factor function does not follow the GLM for link
input : w, c, ξ, K, T, α, β. output : θa , φw . µ1w,d (k) ← initialization and normalization; for t ← 1 to T do µt+1 w,d (k) ∝ [(1 − ξ)µtθd →zw,d (k) + ξµtηc →zw,d (k)] × µtφw →zw,d (k); end P θd (k) ← [µ·,d (k) + α]/ Pk [µ·,d (k) + α]; φw (k) ← [µw,· (k) + β]/ w [µw,· (k) + β];
Fig. 8. The synchronous BP for RTM.
modeling in the original RTM [12] because the GLM makes inference slightly more complicated. However, similar to the GLM, Eq. (23) is also able to capture the topic interactions between two linked documents {d, d′ } in document networks. Instead of smoothness prior encoded by factor functions (8) and (9), it describes arbitrary topic dependencies {k, k ′ } of linked documents {d, d′ }. Based on the factor function (23), we resort to the sumproduct algorithm to calculate the message, µηc →zw,d (zw,d = k) =
XX d′
fηc (k|k ′ )µ(z·,d′ = k ′ ), (24)
k′
where we use the sum rather than the product of messages from all linked documents d′ in order to avoid arithmetic underflow. The standard sum-product algorithm requires the product of all messages from factors to variables. However, in practice, the direct product operation cannot balance the messages from different sources. For example, the message µθd →zw,d is from the neighboring words within the same document d, while the message µηc →zw,d is from all linked documents d′ . If we pass the product of these two types of messages, we cannot distinguish which one influences more on the topic label zw,d . Hence, we use the weighted sum of two types of messages and obtain the following update
9
TABLE 2 Summarization of four document data sets Data sets CORA MEDL NIPS BLOG
D 2410 2317 1740 5177
A 2480 8906 2037 −
W 2961 8918 13649 33574
C 8651 1168 − 1549
Nd 57 104 1323 217
Wd 43 66 536 149
equation, µ(zw,d = k) ∝ [(1 − ξ)µθd →zw,d (zw,d = k) + ξµηc →zw,d (zw,d = k)] × µφw →zw,d (zw,d = k),
(25)
where ξ ∈ [0, 1] is the weight to balance two messages µθd →zw,d and µηc →zw,d . When there are no link information ξ = 0, Eq. (25) reduces to (10) so that RTM reduces to LDA. Fig. 8 shows the synchronous BP algorithm for learning RTM.4 Given the inferred messages, the parameter estimation equations remain the same as (13) and (14). The computational complexity at each iteration is O(K 2 CDWd ), where C is the total number of links in the document network.
5
E XPERIMENTS
In this section we evaluate the proposed BP algorithm for learning LDA, ATM and RTM. We use four large-scale document data sets: 1) CORA [31] contains abstracts from the CORA research paper search engine in machine learning area, where the documents can be classified into 7 major categories. 2) MEDL [32] contains abstracts from the MEDLINE biomedical paper search engine, where the documents fall broadly into 4 categories. 3) NIPS [33] includes papers from the conference “Neural Information Processing Systems”, where all papers are grouped into 13 categories. NIPS has no citation link information. 4) BLOG [34] contains a collection of political blogs on the subject of American politics in the year 2008. where all blogs can be broadly classified into 6 categories. BLOG has no author information. Table 2 summarizes the statistics of four data sets, where D is the total number of documents, A is the total number of authors, W is the vocabulary size, C is the total number of links between documents, Nd is the average number of words per document, and Wd is the average vocabulary size per document. 5.1 BP for LDA We compare BP with two commonly-used LDA learning algorithms like VB [1]5 and GS [3]6 under the same 500 4. For short, µθd →zw,d (zw,d = k) = µθd →zw,d (k). 5. http://chasen.org/∼daiti-m/dist/lda/ 6. http://psiexp.ss.uci.edu/research/programs data/toolbox.htm
training iterations and fixed hyperparameters α = β = 0.01. To estimate θ for the test set, we fix φ learned from the training set and use 500 learning iterations on the test set. We use MATLAB C/C++ MEX-implementations for all these algorithms, and carry out experiments on a common PC with CPU 2.4GHz and RAM 4G. Fig. 9 shows the perplexity [1] (average ± standard deviation) from five-fold cross-validation on the test set for different topics, where the lower perplexity indicates the better generalization ability for the unseen test set. Consistently, BP has the lowest perplexity for different topics on four data sets, which confirms its effectiveness for learning LDA. On average, BP lowers around 11% than VB and 6% than GS in perplexity. In addition, BP is faster than both VB and GS as shown in Fig. 10. We show only 0.3 times of the real learning time of VB because of the time-consuming digamma functions. BP is faster than GS because it computes messages for word indices. The speed difference is largest on the NIPS set due to its largest ratio Nd /Wd = 2.47 in Table 2. To examine the convergence rate of BP, we use the entire data as the training set, and calculate the training perplexity at every 10 iterations in the total of 1000 training iterations. Fig. 11 shows that the training perplexity of BP generally decreases rapidly with the number of training iterations increases. In our experiments, BP on average converges with the number of training iterations T ≈ 170 when the difference of training perplexity between two successive iterations is less than one. Although this paper does not theoretically prove that BP will converge on all data sets in real-world applications, the resemblance among VB, GS and BP in the subsection 2.3 implies that there should be the similar underlying principle that ensures BP to converge on general sparse word vector space. Further analysis reveals that BP on average converges slower than VB (T ≈ 100) but much faster than GS (T ≈ 300) on the four data sets. Although VB converges rapidly partly attributed to digamma functions, it often consumes triple more training time shown in Fig. 10. As a result, BP on average enjoys the highest efficiency for learning LDA with regard to the balance of convergence rate and training time. The fast convergence rate is a desirable property as far as the online [35] and distributed [36] topic modeling for large-scale corpus are concerned. LDA can be used to reduce dimensionality of the word vector space for document classification [1] based on support vector machines (SVM) [37]. Fig. 12 shows the average accuracy of document classification using the five-fold cross-validation. BP increases slightly 2% accuracy over VB and GS for all data sets. Although BP achieves lower perplexity than VB and GS, it does not show significant advantage in document classification. For example, VB is more accurate than BP when the number of topics is small on MEDL in Fig. 12. The inconsistency between Fig. 9 and Fig. 12 implies that lower perplexity does not always correspond to better dimensionality reduction for document classification.
10
BP GS VB
1100 Perplexity
2400
850
1200
BP GS VB
800 750
1000
2300
700 650
900
600
800
550 700 600
0
20
40
60
450
0
MEDL 20 40
BP GS VB
3400
2200
3200
2100
3000
2000
2800
1900
2600
1800
500 CORA
3600 BP GS VB
2400
1700 60 0 20 Number of Topics
NIPS 40
60
2200
0
BLOG 20 40
60
Fig. 9. Perplexity as a function of number of topics on CORA, MEDL, NIPS and BLOG. 300
Learning time (seconds)
200 BP GS VB
160
1600 BP GS VB
250
0.3x
0.3x
150 80
BP GS VB
1200
0.3x
800
800
100
40 0
0.3x
1200
200
120
BP GS VB
1600
CORA 0
20
40
60
0
400
400
50 0
20
MEDL 40
60
0
NIPS 0
20
40
0
60
0
BLOG 20 40
60
Number of Topics
Fig. 10. Training time as a function of number of topics on CORA, MEDL, NIPS and BLOG. For VB, it shows 0.3 times of the real learning time denoted by 0.3x. 650
Perplexity
600
CORA
BP GS VB
480
BP GS VB
440
550 500
400
450
360
0
MEDL
200 400 600 800 1000
0
1700
NIPS
1650
BP GS VB
2200 2100
1600
2000
1550
1900
BP GS VB
1800
1500
1700
1450 1400 200 400 600 800 1000 0
BLOG
1600 200 400 600 800 1000
0
200 400 600 800 1000
Number of Iterations
Fig. 11. Training perplexity as a function of number of iterations when K = 50 on CORA, MEDL, NIPS and BLOG. We compare six BP implementations such as siBP, BP and CVB0 [6] using both synchronous and asynchronous update schedules. We name three synchronous implementations as s-BP, s-siBP and s-CVB0, and three asynchronous implementations as a-BP, a-siBP and aCVB0. Because these six belief propagation implementations produce comparable perplexity, we show the relative perplexity that subtracts the mean value of six implementations in Fig. 13. Overall, the asynchronous schedule gives slightly lower perplexity than synchronous schedule because it passes messages faster and more efficiently. Except on CORA set, siBP generally provides the highest perplexity because it introduces subtle biases in computing messages at each iteration. The biased message will be propagated and accumulated leading to the inaccurate parameter estimation. Although the proposed BP achieves
lower perplexity than CVB0 on NIPS set, both of them work comparably well on other sets. But BP is much faster because it computes messages over word indices. The comparable results also confirm our assumption that topic modeling can be efficiently performed on word indices instead of tokens. To measure the interpretability of a topic model, the word intrusion and topic intrusion are proposed to involve subjective judgements [38]. The basic idea is to ask volunteer subjects to identify the number of word intruders in the topic as well as the topic intruders in the document, where intruders are defined as inconsistent words or topics based on prior knowledge of subjects. Fig. 14 shows the top ten words of K = 10 topics inferred by VB, GS and BP algorithms on NIPS set. We find no obvious difference with respect to word intrusions in each topic. Most topics share the similar top ten words
11
Accuracy (%)
36
47
CORA
45
BP GS VB
34
27
MEDL BP GS VB
NIPS
BLOG 29
26.6
28
26.2
43 32
25.8
41 30
27 BP GS VB
25.4 0
20
40
60
39 0
20
40
BP GS VB
60 0 20 Number of Topics
40
60
26 0
20
40
60
Fig. 12. Classification accuracy as a function of number of topics on CORA, MEDL, NIPS and BLOG.
Relative Perplexity
15 10 5
a-siBP s-siBP a-BP s-BP a-CVB0 s-CVB0
10
40
80
8
30
60
6
20
4 2
0
CORA −15
10 20 30 40 50
−4 −6
0
−10
−2 −10
20
0
0
−5
40
10
−20
−20 MEDL 10 20 30 40 50
−30
NIPS 10 20 30 40 50
−40
BLOG 10 20 30 40 50
Number of Topics
Fig. 13. Relative predictive perplexity as a function of number of topics on CORA, MEDL, NIPS and BLOG. 3000 L-LDA LDA
2500 Perplexity
but with different ranking orders. Despite significant perplexity difference, the topics extracted by three algorithms remains almost the same interpretability at least for the top ten words. This result coincides with [38] that the lower perplexity may not enhance interpretability of inferred topics. Such phenomenon has also been observed in MRFbased image labeling problems [17]. Different MRF inference algorithms such as graph cuts and BP often yield comparable results. Although one inference method may find more optimal MRF solutions, it does not necessarily translate into better performance compared to the ground-truth. The underlying hypothesis is that the ground-truth labeling configuration is often less optimal than solutions produced by inference algorithms. For example, if we manually label the topics for a corpus, the final perplexity is often higher than that of solutions returned by VB, GS and BP. For each document, LDA provides the equal number of topics K but the groundtruth often uses the unequal number of topics to explain the observed words, which may be another reason why the overall perplexity of learned LDA is often lower than that of the ground-truth. To test this hypothesis, we compare the perplexity of labeled LDA (L-LDA) [39] with LDA in Fig. 15. L-LDA is a supervised LDA that restricts the hidden topics as the observed class labels of each document. When a document has multiple class labels, L-LDA automatically assigns one of the class labels to each word index. In this way, L-LDA resembles the process of manual topic
2000 1500 1000 500 0
CORA
MEDL
NIPS
BLOG
Fig. 15. Perplexity of L-LDA and LDA on four data sets.
labeling by human, and its solution can be viewed as close to the ground-truth. For a fair comparison, we set the number of topics K = 7, 4, 13, 6 of LDA for CORA, MEDL, NIPS and BLOG according to the number of document categories in each set. Both L-LDA and LDA are trained by BP using 500 iterations. Fig. 15 confirms that L-LDA produces higher perplexity than LDA, which partly supports that the ground-truth often gives the higher perplexity than the optimal solutions of LDA inferred by BP. Under this situation, improving the formulation of topic models such as LDA is better than improving inference algorithms in order to enhance the performance significantly. Indeed, the proper settings of hyperparam-
12
Topic 1
Topic 2
Topic 3
Topic 4
Topic 5
Topic 6
Topic 7
Topic 8
Topic 9
Topic 10
recognition image system images network training speech figure model set image images figure object feature features recognition space visual distance recognition image images training system speech set feature features word network networks units input neural learning hidden output training unit network networks neural input units output learning hidden layer weights network units networks input output neural hidden learning unit layer analog chip circuit figure neural input output time system network analog circuit chip output figure signal input neural time system analog neural circuit chip figure output input time system signal time model neurons neuron spike synaptic activity input firing information model neurons cells neuron cell visual activity response input stimulus neurons time model neuron synaptic spike cell activity input firing model visual figure cells motion direction input spatial field orientation time noise dynamics order results point model system values figure model visual figure motion field direction spatial cells image orientation function functions algorithm linear neural matrix learning space networks data function functions number set algorithm theorem tree bound learning class function functions algorithm set theorem linear number vector case space learning network error neural training networks time function weight model function learning error algorithm training linear vector data set space learning error neural network function weight training networks time gradient data training set error algorithm learning function class examples classification training data set performance classification recognition test class error speech training data set error learning performance test neural number classification model data models distribution probability parameters gaussian algorithm likelihood mixture model data models distribution gaussian probability parameters likelihood mixture algorithm model data distribution models gaussian algorithm probability parameters likelihood mixture learning state time control function policy reinforcement action algorithm optimal learning state control time model policy action reinforcement system states learning state time control policy function action reinforcement algorithm model
Fig. 14. Top ten words of K = 10 topics of VB (first line), GS (second line), and BP (third line) on NIPS.
Fig. 17. Community structures inferred by GS and BP.
eters can make the predictive perplexity comparable for all state-of-the-art approximate inference algorithms [6]. However, we still advocate BP with two main reasons. First, BP is more accurate because it passes messages without loss of information. Second, BP is much faster than both VB and GS, even if they all can provide comparable perplexity and interpretability under proper settings of hyperparameters. 5.2 BP for ATM Fig. 16 shows the perplexity from five-fold crossvalidation. On average, BP lowers 12% perplexity than GS, which is consistent with Fig. 9. Another possible reason for such improvements may be our assumption that all coauthors of the document account for the word topic label using multinomial instead of uniform probabilities.
The GS algorithm for learning ATM is implemented in the MATLAB topic modeling toolbox.7 We compare BP and GS for learning ATM based on 500 iterations on training data. ATM describes the expertise of each author as a mixture of topics encoded by the parameter θa . We examine whether the inferred topic-based expertise θa can group all authors into different communities. If θa has a good discriminative power to assign authors into different communities, it will enhance the specificity of matching papers with reviewers for a given topic. Fig. 17 compares the two-dimensional embedding visualization (using the t-SNE stochastic neighborhood embedding [40]) of community structures based on K = 10 topic-based author expertise θa inferred by GS and BP with 500 iterations on the whole NIPS data set, respectively. Different colors represent different communities, where the community label of each author is determined by the document category. NIPS contains a total of 13 communities in Table 2. For simplicity, we remove authors within multiple communities. Fig. 17 shows that ATM learned by BP provides a slightly better grouping and separation of the authors in different communities. In contrast, ATM leaned by GS does not produce a well separated embedding, and authors in different categories tend to mix together. This result is also consistent with the document classification performance of LDA learned 7. http://psiexp.ss.uci.edu/research/programs data/toolbox.htm
13
2600
1100 BP GS
Perplexity
1000
BP GS
BP GS
2400
750
900
2200
650
800 700
850
2000
550 10
20
CORA 30 40
50
450
10
20
MEDL 30 40
50
1800
10
20
NIPS 30 40
50
Number of Topics
Fig. 16. Perplexity as a function of number of topics for ATM on CORA, MEDL and NIPS.
by GS and BP in Fig. 12, where BP often gives slightly better discriminative topic representations than GS. 5.3 BP for RTM The GS algorithm for learning RTM is implemented in the R package.8 We compare BP with GS for learning RTM using the same 500 iterations on training data set. Based on the training perplexity, we manually set the weight ξ = 0.15 in Fig. 8 to achieve the overall superior performance on four data sets. Fig. 18 shows average perplexity on five-fold crossvalidation. On average, BP lowers 6% perplexity than GS. Because the original RTM learned by GS is inflexible to balance information from different sources, it has slightly higher perplexity than LDA (Fig. 9). To circumvent this problem, we introduce the weight ξ in (25) to balance two types of messages, so that the learned RTM gains lower perplexity than LDA. Future work will estimate the balancing weight ξ based on the feature selection or MRF structure learning techniques. We also examine the link prediction performance of RTM. We define the link prediction as a binary classification problem. Similar to [12], we use the Hadmard product of a pair of document topic proportions as the link feature, and train an SVM [37] to decide if there is a link between them. Note that the original RTM [12] learned by the GS algorithm uses the GLM to predict links. Fig. 19 compares the average F-measure of link prediction on five-fold cross-validation. Encouragingly, BP provides significantly 15% higher F-measure over GS on average. These results confirm the effectiveness of BP for capturing accurate topic structural dependencies in document networks.
6
C ONCLUSIONS
First, this paper has presented the novel factor graph representation of LDA within the MRF framework. Not only does MRF solve topic modeling as a labeling problem, but also facilitate classic loopy BP algorithms for approximate inference and parameter estimation. We summarize this BP-based learning scheme on factor graphs as three steps: 8. http://cran.r-project.org/web/packages/lda/
1) First, we absorb the observed variables {w, d} as indices of factors, which connect hidden variables such as topic labels in the neighborhood system. 2) Second, we design the proper factor functions to encourage or penalize different local topic labeling configurations in the neighborhood system. 3) Third, we develop the approximate inference and parameter estimation algorithms within the message passing framework. The BP algorithm is easy to implement, rapidly convergent, computationally efficient, and more accurate than other two approximate inference methods like VB [1] and GS [3] in several text mining tasks of broad interests. Furthermore, the superior performance of BP algorithm for learning ATM [11] and RTM [12] confirms its potential effectiveness in learning all variants of LDA-based topic models. Second, as the main finding of this paper, the proper definition of neighborhood systems as well as the design of factor functions can constrain MRF to be equivalent to LDA in the sense that they can achieve the same topic modeling goal. However, how to constrain MRF to represent the generic HBM still remains to be studied in our future work. Since the topic modeling is essentially a word clustering paradigm, the opened MRF perspective may inspire us to extend several classic MRF or graph-based image segmentation or data clustering algorithms, for example, normalized graph cuts [41], random walks [42], and affinity propagation [43], to learn variants of LDA-based topic models. Finally, the scalability of BP will be an important issue in our future work. Similar to VB and GS, the BP algorithm has a linear complexity with the total number documents D and the total number of topics K. On the one hand, we may extend the proposed BP algorithm for online [35] and distributed [36] learning for LDA, where the former incrementally learns parts of D documents in data streams and the latter learns parts of D documents on distributed computing units. On the other hand, since the K-tuple message is often sparse [44], we may pass or update only salient parts of the K-tuple messages at each learning iteration in order to speed up the whole message passing process.
14
850
Perplexity
1000
BP GS
900
BP GS
750
BP GS
3400 3000
800
650
700
550
600
10
20
CORA 30 40
50
450
2600
10
20
MEDL 30 40
50
2200
10
20
BLOG 30 40
50
Number of Topics
Fig. 18. Perplexity as a function of number of topics for RTM on CORA, MEDL and NIPS. 1 F-measure
BP GS
1
CORA
BP GS
0.8
0.8
MEDL
BP GS
BLOG
0.72
0.6
0.6
0.76
0.68
0.4 0.64
0.4 10
20
30
40
50
0.2
10
20 30 40 50 Number of Topics
10
20
30
40
50
Fig. 19. F-measure of link prediction as a function of number of topics on CORA, MEDL and BLOG.
ACKNOWLEDGEMENTS This work is supported by NSFC (Grant No. 61003154) and the Shanghai Key Laboratory of Intelligent Information Processing, China (Grant No. IIPL-2010-009).
R EFERENCES [1] [2] [3] [4] [5] [6] [7] [8]
[9] [10] [11] [12] [13]
D. M. Blei, A. Y. Ng, and M. I. Jordan, “Latent Dirichlet allocation,” J. Mach. Learn. Res., vol. 3, pp. 993–1022, 2003. S. Z. Li, Markov Random Field Modeling in Image Analysis. New York: Springer-Verlag, 2001. T. L. Griffiths and M. Steyvers, “Finding scientific topics,” Proc. Natl. Acad. Sci., vol. 101, pp. 5228–5235, 2004. T. Minka and J. Lafferty, “Expectation-propagation for the generative aspect model,” in UAI, 2002, pp. 352–359. Y. W. Teh, D. Newman, and M. Welling, “A collapsed variational Bayesian inference algorithm for latent Dirichlet allocation,” in NIPS, 2007, pp. 1353–1360. A. Asuncion, M. Welling, P. Smyth, and Y. W. Teh, “On smoothing and inference for topic models,” in UAI, 2009, pp. 27–34. C. M. Bishop, Pattern recognition and machine learning. Springer, 2006. J. Zeng and Z.-Q. Liu, “Markov random field-based statistical character structure modeling for handwritten Chinese character recognition,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 30, no. 5, pp. 767–780, 2008. F. R. Kschischang, B. J. Frey, and H.-A. Loeliger, “Factor graphs and the sum-product algorithm,” IEEE Transactions on Inform. Theory, vol. 47, no. 2, pp. 498–519, 2001. L. Dietz, “Directed factor graph notation for generative models,” Max Planck Institute for Informatics, Tech. Rep., 2010. M. Rosen-Zvi, T. Griffiths, M. Steyvers, and P. Smyth, “The author-topic model for authors and documents,” in UAI, 2004, pp. 487–494. J. Chang and D. M. Blei, “Hierarchical relational models for document networks,” Annals of Applied Statistics, vol. 4, no. 1, pp. 124–150, 2010. I. Pruteanu-Malinici, L. Ren, J. Paisley, E. Wang, and L. Carin, “Hierarchical Bayesian modeling of topics in time-stamped documents,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 6, pp. 996–1011, 2010.
[14] X. G. Wang, X. X. Ma, and W. E. L. Grimson, “Unsupervised activity perception in crowded and complicated scenes using hierarchical Bayesian models,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 3, pp. 539–555, 2009. [15] G. Heinrich, “Parameter estimation for text analysis,” University of Leipzig, Tech. Rep., 2008. [16] J. M. Hammersley and P. Clifford, “Markov field on finite graphs and lattices,” unpublished, 1971. [17] M. F. Tappen and W. T. Freeman, “Comparison of graph cuts with belief propagation for stereo, using identical MRF parameters,” in ICCV, 2003, pp. 900–907. [18] A. P. Dempster, N. M. Laird, and D. B. Rubin, “Maximum likelihood from incomplete data via the EM algorithm,” Journal of the Royal Statistical Society, Series B, vol. 39, pp. 1–38, 1977. [19] H. Wallach, D. Mimno, and A. McCallum, “Rethinking LDA: Why priors matter,” in NIPS, 2009, pp. 1973–1981. [20] T. P. Minka, “Estimating a Dirichlet distribution,” Microsoft Research, Tech. Rep., 2000. [21] J. Zeng, L. Xie, and Z.-Q. Liu, “Type-2 fuzzy Gaussian mixture models,” Pattern Recognition, vol. 41, no. 12, pp. 3636–3643, 2008. [22] J. Zeng and Z.-Q. Liu, “Type-2 fuzzy hidden Markov models and their application to speech recognition,” IEEE Trans. Fuzzy Syst., vol. 14, no. 3, pp. 454–467, 2006. [23] J. Zeng and Z. Q. Liu, “Type-2 fuzzy Markov random fields and their application to handwritten Chinese character recognition,” IEEE Trans. Fuzzy Syst., vol. 16, no. 3, pp. 747–760, 2008. [24] J. Winn and C. M. Bishop, “Variational message passing,” J. Mach. Learn. Res., vol. 6, pp. 661–694, 2005. [25] W. L. Buntine, “Variational extensions to EM and multinomial PCA,” in ECML, 2002, pp. 23–34. [26] T. Hofmann, “Unsupervised learning by probabilistic latent semantic analysis,” Machine Learning, vol. 42, pp. 177–196, 2001. [27] D. D. Lee and H. S. Seung, “Learning the parts of objects by nonnegative matrix factorization,” Nature, vol. 401, pp. 788–791, 1999. [28] J. Zeng, W. K. Cheung, C.-H. Li, and J. Liu, “Coauthor network topic models with application to expert finding,” in IEEE/WIC/ACM WI-IAT, 2010, pp. 366–373. [29] J. Zeng, W. K.-W. Cheung, C.-H. Li, and J. Liu, “Multirelational topic models,” in ICDM, 2009, pp. 1070–1075. [30] J. Zeng, W. Feng, W. K. Cheung, and C.-H. Li, “Higher-order Markov tag-topic models for tagged documents and images,” arXiv:1109.5370v1 [cs.CV], 2011.
15
[31] A. K. McCallum, K. Nigam, J. Rennie, and K. Seymore, “Automating the construction of internet portals with machine learning,” Information Retrieval, vol. 3, no. 2, pp. 127–163, 2000. [32] S. Zhu, J. Zeng, and H. Mamitsuka, “Enhancing MEDLINE document clustering by incorporating MeSH semantic similarity,” Bioinformatics, vol. 25, no. 15, pp. 1944–1951, 2009. [33] A. Globerson, G. Chechik, F. Pereira, and N. Tishby, “Euclidean embedding of co-occurrence data,” J. Mach. Learn. Res., vol. 8, pp. 2265–2295, 2007. [34] J. Eisenstein and E. Xing, “The CMU 2008 political blog corpus,” Carnegie Mellon University, Tech. Rep., 2010. [35] M. Hoffman, D. Blei, and F. Bach, “Online learning for latent Dirichlet allocation,” in NIPS, 2010, pp. 856–864. [36] K. Zhai, J. Boyd-Graber, and N. Asadi, “Using variational inference and MapReduce to scale topic modeling,” arXiv:1107.3765v1 [cs.AI], 2011. [37] C.-C. Chang and C.-J. Lin, “LIBSVM: A library for support vector machines,” ACM Transactions on Intelligent Systems and Technology, vol. 2, pp. 27:1–27:27, 2011. [38] J. Chang, J. Boyd-Graber, S. Gerris, C. Wang, and D. Blei, “Reading tea leaves: How humans interpret topic models,” in NIPS, 2009, pp. 288–296. [39] D. Ramage, D. Hall, R. Nallapati, and C. D. Manning, “Labeled LDA: A supervised topic model for credit attribution in multilabeled corpora,” in Empirical Methods in Natural Language Processing, 2009, pp. 248–256. [40] L. J. P. van der Maaten and G. E. Hinton, “Visualizing highdimensional data using t-SNE,” J. Mach. Learn. Res., vol. 9, pp. 2579–2605, 2008. [41] J. Shi and J. Malik, “Normalized cuts and image segmentation,” IEEE Trans. Pattern Anal. Machine Intell., vol. 22, no. 8, pp. 888–905, 2000. [42] L. Grady, “Random walks for image segmentation,” IEEE Trans. Pattern Anal. Machine Intell., vol. 28, no. 11, pp. 1768–1783, 2006. [43] B. J. Frey and D. Dueck, “Clustering by passing messages between data points,” Science, vol. 315, no. 5814, pp. 972–976, 2007. [44] I. Porteous, D. Newman, A. Ihler, A. Asuncion, P. Smyth, and M. W. U, “Fast collapsed Gibbs sampling for latent Dirichlet allocation,” in KDD, 2009, pp. 569–577.
