An Improved Fuzzy System for Representing Web

4.2 FCC: a previous Web Page Representation Based on Fuzzy Logic 113

tion of a normal distribution. All the variables except emphasis are defined by means of sets of equal size symmetrically distributed along the possible input values. These sets were defined regardless of concrete datasets, aiming to divide the input space in sets of equal size considering the possible input values, that is, frequency values from 0 to 1, since the input values are normalized to the maximum. Nevertheless, emphasis is considered separately because when the maximum frequency value for emphasized words in a document is small, the normalization could mislead us about the importance of other emphasized terms. For example, using symmetrical sets and having a maximum of 4 would lead to consider the importance of terms emphasized once as low. However, emphasis is used by authors to stress some words they consider important over the rest. In the case of our example we could want to give more importance to this kind of terms. For this reason, the sets for emphasis were asymmetrically defined. This way, frequencies that would be strictly low can be also considered as medium, since we can expect small maximum values in emphasis. Figure 4.3c shows how FCC takes this issue into account by defining different sets for emphasis than for the rest of the criteria, with bigger medium and high sets and a smaller low set, i.e., the low set for emphasis will only include terms with small emphasized frequency that also belong to documents with high maximums.

(a) Input linguistic variable: Term Position.

(b) Output linguistic variable: Position.

Figure 4.2: Data base for the Auxiliary Fuzzy System: input and output linguistic variables. The other part of the knowledge base is a set of IF-THEN rules that combine the variables in order to define the behavior of the system. The aim of the rules is to combine one or more input fuzzy sets (antecedents or premises) and to associate them with an output fuzzy set (consequent). Once the consequents of each rule have been calculated, and after an aggregation stage, the final set is obtained.

114

Fuzzy Combinations of Criteria: An Application to Web Page Representation for Clustering

(a) Frequency sets

(b) Title sets

(c) Emphasis sets

(d) Position sets

Figure 4.3: Data base for FCC: input linguistic variables.

Figure 4.4: Data base for FCC: output linguistic variable Importance.

First, we show the rule base for the Auxiliary Fuzzy system in Table 4.1. This is a small set of rules aiming to establish the position value of a term in a document depending on where that term appears more frequently within the document. The rule base assigns preferential position to terms appearing mostly

4.2 FCC: a previous Web Page Representation Based on Fuzzy Logic 115

in the first or the last part of the document, and standard position to the rest. To do this, the document is divided in three sections that were called: introduction, body and conclusion (see Figure 4.2a). It is worth noting that this is just a naming scheme to split document contents. However, the contents of each of these three parts could have nothing to do with an introduction or conclusions (the combination of criteria should deal with this fact). Then, each time a term appears in the document, its position—normalized to the total number of terms in the document and previously referred to as term position—is used as input for the Auxiliary Fuzzy System. The total number of terms is used for normalization, because the different renderings of the same page depending on the browser, window size, etc., make very difficult to use other references, as the number of lines in the document, in a solid way. Once all the positions of the same term have been evaluated by the auxiliary system, the result is the global position of that term, called here position to simplify the naming. More details about how the evaluation process is performed are given below, after explaining the General Fuzzy System. The idea behind the auxiliary system is to assign different Table 4.1: Rule base for the Auxiliary Fuzzy System. IF

Term Position Introduction Body Conclusion

THEN

⇒ ⇒ ⇒

Position Preferential Standard Preferential

degrees of membership of a given word to preferential or standard sets depending on whether that word appears more frequently at the beginning and at the end of the document, or in the middle. The general system has a bigger set of rules as can be seen in Table 4.2. As these rules are based on expert knowledge, it is advisable to understand the ideas behind the rules before trying to read the rules themselves. Concretely, FCC rule base relies on the following considerations: 1. If a word appears in the title or the word is emphasized, that word should also appear in one of the other criteria in order to be considered important. This aims to alleviate the problem of rhetoric titles or non-informative highlighting, problems that were described in Section 1.2. 2. Words appearing in the beginning or at the end of a document may be more important than the rest of the words, because documents sometimes contain overviews and summaries in order to attract the interest of the reader. Besides, FCC is a general representation, not oriented to a particular type of document, which means that some documents could have introduction and conclusions and others do not. By means of the combination of criteria

116

Fuzzy Combinations of Criteria: An Application to Web Page Representation for Clustering

we can try to detect which case we are dealing with. When the words in preferential position do not also appear in the title or emphasized, then we could assume that the document does not follow the mentioned structure and we could reduce the importance value of that word. 3. It is possible that there are no emphasized words in a document. In the same way, it is possible that a document does not have a title, or the title does not contain important words. In these cases we have to take care of the penalization it could cause to the combination. We could not always expect to find the most important terms by combining title and emphasis, other criteria also count, particularly in these cases. 4. If the previous criteria were not able to choose the most important words, the frequency of the words in the whole document may help to find them. Different from the others, frequency criterion is always available, so it gives a last chance to establish word importance when the rest of the criteria fail. Keeping these ideas in mind, now we can concentrate on the rule base for FCC, which is shown in Table 4.2. In this table, each row is a rule that contains the values of different criteria and the resulting output (importance). Yet, essentially, the way of reading the propositions does not vary too much. First, blanks in the table correspond with premises that are not present in a concrete rule. That is, the values corresponding to that criterion do not affect the rule. We have also separated rules containing blanks from the rest, to ease table reading. To give an example, the first row below the title header should be read as: IF Title IS High AND Frequency IS High AND Emphasis IS High THEN Importance IS Very High Going into more detail, the rule set must be complete, that is, given any possible input, at least one rule must be fired. It is important to realize than more than one rule could be fired at the same time. In those cases, the inference engine evaluates all the fired rules on the basis of the Center Of Mass4 (COM) algorithm, that weights the output of every fired rule, taking into account the truth degree of their antecedents. The COM algorithm takes the balance point or centroid of all the scaled membership functions taken together for that variable (Hopgood, 2011). Lastly, as we mentioned above, the output is a linguistic label (e.g., low, medium, very high) with an associated number related to the importance of a word in the document. Concretely, the output—for each term input to the system—is calculated by scaling the membership functions by product and combining them 4 It

is the most common method for defuzzification and it is also known as the centroid, center of gravity or center of area method.

4.2 FCC: a previous Web Page Representation Based on Fuzzy Logic 117

Table 4.2: Rule base for FCC. Inputs are related to normalized term frequencies. IF Title AND Frequency AND Emphasis AND Position High High High High Low

High Medium High Medium Low

High High Medium Medium Low

High High High High High High High High High High Low Low Low Low Low Low Low Low Low Low Low Low Low Low Low Low

High High Medium Medium Low Low Low Low Low Low High High High High High High Medium Medium Medium Medium Medium Medium Low Low Low Low

Low Low Low Low High High Medium Medium Low Low High High Medium Medium Low Low High High Medium Medium Low Low High High Medium Medium

Preferential Standard Preferential Standard Preferential Standard Preferential Standard Preferential Standard Preferential Standard Preferential Standard Preferential Standard Preferential Standard Preferential Standard Preferential Standard Preferential Standard Preferential Standard

THEN Importance

⇒ ⇒ ⇒ ⇒ ⇒ ⇒ ⇒ ⇒ ⇒ ⇒ ⇒ ⇒ ⇒ ⇒ ⇒ ⇒ ⇒ ⇒ ⇒ ⇒ ⇒ ⇒ ⇒ ⇒ ⇒ ⇒ ⇒ ⇒ ⇒ ⇒ ⇒

Very High Very High Very High High No Very High High Medium Low Very High High High Medium Medium Low Very High High High Medium Medium Low Very High High Medium Low Low No High Medium Medium Low

118

Fuzzy Combinations of Criteria: An Application to Web Page Representation for Clustering

by summation. These kind of systems are called additive and their main advantage is the computing efficiency (Kosko, 1998). A more detailed explanation of the fuzzy system can be found in Ribeiro et al. (2003) and Fresno (2006).

4.2.2

PF Method for Dimension Reduction

This method is oriented to term weighting functions that assigns weights that are directly proportional to the estimated importance for a feature in a document. Other lightweight reduction methods are based on frequencies or probabilities calculated from these frequencies. Thus, commonly used reductions like DF do not take into account all the factors that FCC combines, but only frequencies within the corpus. For this reason, together with FCC, a different reduction method called PF was proposed, aiming to avoid penalizing functions that consider other aspects than frequency, like FCC. PF is based in the term weighting function itself, that is employed also as feature reduction function. The first step of this method is ranking the terms corresponding to each page on the basis of their weight. Then, for each document, the first n ranked terms are selected to compose the reduced vocabulary. Then, the maximum number of terms in the reduced vocabulary will be n × | D |, where | D | is the number of documents in the collection. Actually, the number will be smaller because we can find the same term in different rankings (each ranking corresponds to a different document). Besides, for different term weighting functions applied on the same collection, the term rankings for each document will be different and the reduced vocabularies obtained by using PF could be of different size and contain different terms. Reducing by DF, the reduced vocabularies would be the same, regardless the different weighting functions applied. It is important to apply methods like PF to term weighting functions that do assign weights directly proportional to the estimated importance for a feature in a document. For instance, by using PF over term weights calculated with TF, the result will be to select the terms with highest frequencies on each page, that could result in a vocabulary composed of common use terms (that should not be the most important ones, following the hypothesis of H.P. Luhn exposed in Section 1.2). Moreover, binary weighting functions could lead to a random selection, as the number of possible values for each term is limited and it is not possible to create suitable term rankings.

4.3 Experimental Settings

4.3

119

Experimental Settings

In this section we describe the common experimental settings we use in all the experiments of this chapter. First, in preprocessing, a list consisting of 621 stop words was used to remove common words. Punctuation marks were also removed. Terms were stemmed using a standard implementation of the Porter’s algorithm for English5 . After this preprocessing the vocabulary sizes are: • Banksearch: 210, 785 terms. • WebKB: 40, 258 terms. • SODP: 625, 137 terms Regarding the clustering process, we chose Cluto rbr (k-way repeated bisections globally optimized) as a state of the art algorithm Karypis (2003). The number of desired clusters (K) was fixed to the number of categories in the dataset in order to make the evaluation process more intuitive. Thus, it is possible to find a perfect clustering and the algorithm does not introduce additional bias to the representation step. That is, when we modify the representation, we can be sure that the variation on the clustering results will come from that modification. The rest of the algorithm parameters were set by default. More details regarding the algorithm and its parameters can be found in Section 2.3 on page 71. As all the datasets we deal with in this thesis have associated their ideal solution, we decided to employ an external evaluation measure to evaluate the clustering quality. These solutions can be used as evaluation benchmarks or gold standards. Then, we need a measure which evaluates how well the answer of our clustering system matches that gold standard (Manning et al., 2008). Typically the F-measure (Van Rijsbergen, 1974) (Equation 4.6) is the most used for this task which is equal to the harmonic mean of recall (Equation 4.4) and precision (Equation 4.5): nij (4.4) Recall (i, j) = nj Precision(i, j) =

F (i, j) =

nij ni

2 · Recall (i, j) · Precision(i, j) Recall (i, j) + Precision(i, j)

(4.5)

(4.6)

where i is the cluster, j the category, ni the number of documents in the cluster i and n j the number of documents in the category j. Both terms, cluster and category were defined in Section 3.1. In clustering, recall is the fraction of documents in the category j that are also in the cluster i. Precision refers to the fraction of 5 http://www.tartarus.org/~martin/PorterStemmer

120

Fuzzy Combinations of Criteria: An Application to Web Page Representation for Clustering

documents in the cluster i that are also in the category j (Crabtree et al., 2007). The overall F-measure is the weighted average of the F-measure for each category (Equation 4.7): F=

∑ j

nj · max{ F (i, j)} n i

(4.7)

where n is the total number of documents we want to group, n j is the number of documents in a category j, and the maximum function is applied over the whole set of clusters. The F-measure values are in the interval [0,1] and larger values correspond to higher clustering quality. Other measures like entropy, purity or inverse purity have been also used in the literature to evaluate clustering quality. Recently, the BCubed precision and recall metrics have been also applied to clustering evaluation, as they satisfy some constraints that other measures do not (Amigó et al., 2009). However, the most widely used is F-measure based on precision and recall, that is utilized in this thesis and allow us to keep backwards compatibility of our results with previous works.

4.4

Dimension Reduction

To compare different term weighting functions we need to reduce the initial vocabulary to smaller sizes. Using different reduced vocabulary sizes also allows us to analyze the behavior of the different weighting functions when the vocabulary size changes. This is an important factor, as it could allow to establish a compromise between the reduction and the results. Basically, in terms of computational cost, it is interesting to use vocabulary sizes as small as possible without strongly penalizing clustering results. However, as we saw before, PF reduction does not allow to select the exact number of features in the reduced vocabulary. With PF, vocabulary size will depend on the number of unique terms on the first n positions of each document ranking. To compare different reduction methods we should use the same vocabulary sizes for all of them. There are different dimension reduction techniques that are able to overcome this limitation of PF. In this section we test some of them first and them we propose a new method based on the idea of PF, but overcoming its limitation. We evaluate the effect of different dimension reduction techniques in document representation. Our goal is understanding how dimension reduction affects clustering results, taking into account how this process benefits or not the different term weighting functions.

4.4 Dimension Reduction

4.4.1

121

Initial Tests

First, we analyze the effect of using different dimension reduction techniques over two different term weighting functions that we will use as baselines in this chapter. On the one hand, FCC weighting function, because we are using the same framework and we think it has to be used as a baseline in this thesis. On the other hand, TF-IDF has been used as a standard term weighting function in clustering. This function exploits only plain text, so it is another good baseline to compare against functions that combine different criteria together with plain text. It is worth noting that both functions, FCC and TF-IDF were evaluated against other functions, like ACC (see 1.2 on page 28), binary, binary-IDF, TF, TF-IDF or weighted IDF, in previous works (Fresno, 2006; Pérez García-Plaza et al., 2008; Pérez García-Plaza et al., 2009). In most cases FCC and TF-IDF outperformed the rest in clustering tasks on Banksearch and WebKB collections (described in sections 3.2 and 3.3 respectively). In order to reduce the vocabulary size, in this thesis we employ different techniques (these and other techniques were described in Section 2.2.3, here we describe some details related with how we use them in this thesis): • Document Frequency (DF): the size of the reduced vocabulary is obtained by means of two thresholds. These thresholds are maximum and minimum values for DF, in such a way we select those terms whose DF is between both thresholds. They are usually defined as percentages of the total number of documents. For example, selecting the upper threshold as 60% would mean selecting only terms appearing in less than the 60% of collection documents. Thus, the thresholds are used to decide the final size of the feature set, although with this technique it is necessary to test different possible values to get an approximate number of features in the final vocabulary. It is also possible to obtain two different vocabularies from different thresholds but with approximately the same size, as different thresholds could lead to the same reduction in terms of size, but with different terms. We decided to include it with TF-IDF in this thesis because they are commonly used together in the literature achieving reasonably good results, as we saw in Section 2.2.3. • Latent Semantic Indexing (LSI) (Landauer et al., 1998): in this thesis, as suggested by Tang et al. (2005), LSI was applied after a previous reducing step to alleviate its computational complexity. We reduced vector dimension on the basis of the corresponding term weighting function by selecting the highest ranked terms for each document from the original size—210, 785 terms for Banksearch, 40, 258 for WebKB—to 5, 000 features before applying LSI. To compute LSI with vectors having more than 5, 000 dimensions requires too much time for the computation (to compute the algorithm to

122

Fuzzy Combinations of Criteria: An Application to Web Page Representation for Clustering

reduce to 2, 000 features the elapsed time is about an hour, in addition to the initial reduction to 5, 000 features needed to make possible the computation of LSI). • Random Projection (RP) (Bingham and Mannila, 2001): as Tang et al. (2005) stated, the effectiveness of RP in text clustering is still not clear (as a substitute of LSI), reason why we decided to test this technique by ourselves in this thesis instead of assuming that can be used as a lightweight alternative to LSI. Precisely, as RP is used as an alternative method to LSI, the same previous reducing step was performed in both cases. The number of features per vector, that is, the vocabulary size, was not fixed to a unique value. We wanted to discover how different each term weighting function behaves depending on the vocabulary size. Thus, in our research for this thesis, vector sizes were reduced to 100, 500, 1,000, 2,000, and 5,000 dimensions. The maximum of 5, 000 was selected because with this size, the computational cost of the clustering begins to be considerable. Moreover, the size of the initial vocabulary corresponding to the smallest dataset, WebKB, is 40, 258 terms, so reducing to 5, 000 features constitutes a reduction of approximately one order of magnitude. We consider this scale reasonable for the reduction process. Table 4.3 shows the F-measure results for the combinations of weighting functions and dimension reduction techniques mentioned above. These experiments were carried out over two datasets: Banksearch and WebKB (see Chapter 3, sections 3.2 and 3.3 for more information on these collections). Each table row contains F-measure values corresponding to the clustering solution obtained by using the representation specified in the first column with the number of features per vector detailed on top of the remaining columns, being Avg. and S.D. the average and the standard deviation for that row. The average value and the standard deviation give an idea of the global behavior of the corresponding representation method. However, they do not take into account that, for clustering tasks, it is more interesting to obtain good results with as few features as possible. This way, we will reduce the computational cost of the clustering algorithm. On the other hand, as we do not know the perfect number of features to represent a document, a good balance among the results of all the vocabulary sizes is also desirable. So, what we are looking for is a representation with a good average value combined with good results with vocabularies below 1, 000 features. This way, reducing to smaller vector sizes would guarantee good clustering quality at the same time that the computational cost remains reasonable. Besides, the S.D. gives us an idea about the stability of the results among different vocabulary sizes. Looking at the results, RP shows no real advantage over the rest of reductions in all cases. Although it is computationally less expensive than LSI, its results are worse than the rest in most of the cases, especially when we apply higher dimensionality reductions. It is worth mentioning that in WebKB, FCC RP seems

4.4 Dimension Reduction

123

Table 4.3: F-measure results for dimension reduction experiments Rep.\Dim.

5,000

Avg.

0.748

0.749

0.659

0.149

100

500

1,000

2,000

0.398

0.674

0.723

S.D.

Banksearch TF-IDF DF TF-IDF RP

0.480

0.685

0.707

0.730

0.741

0.669

0.108

TF-IDF LSI

0.750

0.755

0.756

0.757

0.763

0.756

0.005

FCC RP

0.484

0.739

0.740

0.746

0.758

0.693

0.117

FCC LSI

0.775

0.763

0.785

0.763

0.758

0.769

0.011

WebKB TF-IDF DF

0.469

0.521

0.530

0.534

0.532

0.517

0.027

TF-IDF RP

0.313

0.423

0.492

0.499

0.516

0.449

0.084

TF-IDF LSI

0.516

0.507

0.505

0.506

0.501

0.507

0.006

FCC RP

0.446

0.477

0.528

0.470

0.470

0.478

0.030

FCC LSI

0.449

0.460

0.473

0.474

0.475

0.466

0.011

to progressively improve its clustering results from 100 to 1, 000 features, where it achieves its best result. However, with larger vector sizes its results fall again. Then, choosing RP for reducing dimensionality it is not a good option as the probability of obtaining bad clustering results depending on the vector size is high. The case of DF is particularly interesting, because it is the worst reduction method in Banksearch, but at the same time it can be considered the best one in WebKB. We strongly believe that the good balance of documents per category hinders DF in Banksearch due to the thematic divergence among categories (see Section 3.2.1). This way, when reducing to small vector sizes, useful terms to represent concrete categories would be removed. Reducing by DF has a clear drawback: the way of establishing the thresholds. There is no automatic way of selecting them, so the performance of the system could be affected by changing them, even in the case of reducing the vocabulary to the same size (but by using different thresholds values). In our case we select the upper threshold about a 60% to remove too common terms, and we change the lower threshold to get the desired number of features. The upper threshold is used to make minor adjustments, as the number of terms with high DF is relatively small, while the lower threshold is used to make major changes in the number of features, as there is a very large number of terms with low DF. Looking at the row of TF-IDF DF for Banksearch in Table 4.3, when we apply aggressive reductions, like below 1, 000 features per vector, we could probably be removing category related terminology, particularly with the lower DF threshold. It should be taken into account, that the lower threshold needed to get only 100 features is established with values that are bigger than category size (in

124

Fuzzy Combinations of Criteria: An Application to Web Page Representation for Clustering

Banksearch all the categories have the same size, a 10% of the size of the collection). When we relax the thresholds, this terminology enters to our vocabulary, leading to an improvement in clustering results, as occurs from 1, 000 to 5, 000 features. However, the bad result obtained with 100 features strongly penalizes this row, because it implies that choosing a wrong number of features (or the wrong thresholds) would be harmful for the clustering results. By choosing a different lower threshold (below category size), we could probably improve the clustering. However, to do this selection we had needed to know, at least, the size of the categories we wanted to find. Then, in absence of this kind of information, reducing by DF can lead to bad results with aggressive reductions, since it does not provide a clear way of estimating the least important terms. Nevertheless, the case of WebKB and TF-IDF DF is just the opposite, because WebKB consists of pages from Universities, most of them from only four Universities. In Section 3.3.1 we saw that the difficulty of clustering this collection comes mainly from the heterogeneity of documents within categories, that are also unbalanced in terms of documents per category. In this sense, DF seems to help removing terms shared among different categories. Moreover, four out of six WebKB categories are also bigger than Banksearch ones, with respect to the total size of the dataset. Thus, the lower threshold would have a softer effect, and probably in at least three categories (that are bigger than a 20% of collection size) would have even a positive effect, by removing the least common terms in those categories. Besides, the WebKB has an initial vocabulary smaller than Banksearch (40, 258 terms WebKB, 210, 785 terms Banksearch), so the reduction is less aggressive in the case of the former. For all the reasons exposed above, it can be said that the DF method applied here fits better the problem of filtering too common or too rare terms for WebKB than for Banksearch. Looking at the performance of TF-IDF LSI in Banksearch we find very good results. They show also good stability regardless the vocabulary size. Its results are only outperformed by FCC LSI in Banksearch. Both, term weighting functions and the dimension reduction technique works well in Banksearch. However, focusing on WebKB, results are different. First, TF-IDF LSI obtained good results, but DF outperformed LSI in 4 out of 5 cases. When using the smallest vector size LSI performs better than DF. The former transform the features to independent components and this approach seems to work well to maintain the results of the weighting function regardless the vector size. This stability could be explained because by finding the independent components, the fundamental information is preserved through the reduction process. The latter seems to remove important features when reducing to 100 features, but moving to 500 features outperforms the rest of the combinations of term weighting functions and dimension reduction techniques for WebKB. As we commented before, we believe that DF fits better the problem of removing the least representative terms for each category. In ad-

4.4 Dimension Reduction

125

dition to this fact, DF does not depend on the term weighting function values assigned to vocabulary terms, while LSI is applied over the 5, 000 terms having highest weights. Then LSI can only find independent components among these 5, 000 terms, so if the weighting function left some representative terms out of them, LSI can not do anything to solve it (it can be said that LSI is limited by the previous reduction to make its application computationally feasible). Nevertheless, DF is applied over the whole initial vocabulary without using the term weighting function. We believe that DF in WebKB helps TF-IDF to select better the most representative terms for differentiating the categories that the weighting function. Probably the use of IDF hinders the function by penalizing terms in the bigger categories and boosting those belonging to the smaller ones, since in this case high frequencies in the biggest categories are also high frequencies in the collection. On the other hand, when applied on FCC, LSI achieve very bad results in WebKB, even worse than FCC RP. As LSI search form independent components, we believe these bad results are due to the bad performance of the weighting function in this collection, where FCC is not able to find the most important terms for each document. Later, in Section 4.5 we will analyze this bad performance in more detail. Summarizing, from this initial experiments we discard RP reduction because of its bad results in almost all cases. Among the rest, LSI is the best option in Banksearch regardless the weighting function. Nevertheless, in WebKB, DF reduction lead to results even a 6% better than LSI (about 0.03 in terms of Fmeasure). Both of them will be used as baselines in this Chapter for WebKB, given the very good result of TF-IDF LSI with 100 features. However, DF achieves higher F-measure values in the rest of the cases, showing also higher maximum values. Globally, we recommend the use of DF together with TF-IDF for collections with heterogeneous documents within categories, that have also different sizes, and gathered from a small number of web domains. In these cases, DF can help removing too common terms and web domain related terminology spread throughout the collection (we talked about it in Section 3.3). Lastly, there is no clear candidate combination of weighting function and reduction method to be used regardless of the collection, being TF-IDF LSI the most stable option, but at the same time, being improved by others in both collections: FCC LSI in Banksearch, TF-IDF DF in WebKB.

4.4.2

MFT: a Proposal for Dimension Reduction

On the one hand, as we saw in Section 4.2.2, PF reduction method does not allow to reduce the initial vocabulary to concrete sizes. In particular, the smallest reduction it could achieve depends on the number of unique terms in the first po-

126

Fuzzy Combinations of Criteria: An Application to Web Page Representation for Clustering

sition of each document ranking. This usually leads to reduced vocabulary sizes larger than 1, 000 features, depending on the collection. This is a problem for our analysis because of the good results that can be achieved with only 100 features, shown with some combinations of term weighting functions and dimension reduction techniques in the previous Section 4.4.1. On the other hand, a reduction like PF is especially appropriate for functions like FCC that aims to establish term importance from document contents. It also allows to directly analyze the correlation between the estimated importance given to terms and clustering results, since by selecting the most important terms we should achieve better clustering results. Then, this kind of reductions would allow a direct analysis of the weighting function, which is quite interesting for the goals of this thesis. For these reasons, we presented a reduction method called Most Frequent Terms (MFT) (Pérez García-Plaza et al., 2008) that, as in the case of PF, is based on term importance estimated by means of a term weighting function. It can be seen as a method to select a subset of features composed of the n features with highest estimated importance. The algorithm also grants to get always the same output when reducing to a concrete vocabulary size, in contrast with other methods like DF. MFT method works as follows: 1. The terms in each document are ranked based on the term weighting function values. 2. Then, terms on the first position in the document rankings are put in order according to how many times they have appeared in the rankings. If two or more terms appear the same number of times in different rankings, we put them in order based on the maximum weight found for each of them. 3. Next we take the terms appearing in the second position in the rankings, and so forth. 4. The process stops when the desired number of terms is reached. Notice that by following this algorithm the resulting list may be larger than the required size, because there are as many rankings as documents in the dataset (this is exactly the same issue we have commented about PF). Nevertheless, as we put the list in order, we can get the exact number of terms just taking the first n terms of the list. This method was applied to FCC and TF-IDF. Table 4.4 shows the results of MFT reduction compared with the experiments of the previous section. The results corresponding to RP were removed for clarity, since it achieved the worst results.

4.4 Dimension Reduction

127

Table 4.4: F-measure results for dimension reduction experiments Rep.\Dim.

5,000

Avg.

0.748

0.749

0.659

0.149

100

500

1,000

2,000

0.398

0.674

0.723

S.D.

Banksearch TF-IDF DF TF-IDF LSI

0.750

0.755

0.756

0.757

0.763

0.756

0.005

TF-IDF MFT

0.703

0.737

0.768

0.772

0.758

0.748

0.028

FCC LSI

0.775

0.763

0.785

0.763

0.758

0.769

0.011

FCC MFT

0.723

0.757

0.768

0.765

0.768

0.756

0.019

TF-IDF DF

0.469

0.521

0.530

0.534

0.532

0.517

0.027

TF-IDF LSI

0.516

0.507

0.505

0.506

0.501

0.507

0.006

TF-IDF MFT

0.385

0.438

0.466

0.498

0.513

0.460

0.051

FCC LSI

0.449

0.460

0.473

0.474

0.475

0.466

0.011

FCC MFT

0.453

0.472

0.475

0.468

0.475

0.469

0.009

WebKB

Looking at the results of TF-IDF MFT, it is clear that this reduction does not work well for TF-IDF when reducing to smaller vocabulary sizes: • Compared with LSI and DF in Banksearch, MFT outperformed the bad results of DF, while LSI improved MFT with small vector sizes (100 and 500 features). However, with 1, 000 and 2, 000 features MFT obtained higher results, and with 5, 000 the difference is less than a 0, 01, that corresponds to an improvement of only 0.66% of LSI over MFT. It seems that LSI helps TF-IDF to find independent components that were not discovered by the weighting function itself (as MFT reduction can be seen like obtaining the n terms with highest estimated importance from the initial vocabulary). • In WebKB occurs something similar, but MFT only improved LSI with the largest vocabulary size. As MFT strongly depends on the term weighting function to select the most important terms, an improvement of LSI over MFT implies that the weighting function is not working as well as it could. In this sense, the use of IDF could be strongly penalizing results in WebKB, in a similar way as we commented for LSI after our initial tests in the previous section. Then, terms appearing in most of the documents in the biggest category (which has a 36% of WebKB documents, see Section 3.3.1) will be removed when reducing by means of MFT to smallest vector sizes, because their weights were penalized by their high document frequency. The same could occur with other two categories that are also bigger than a 20% of collection size. For this reason, frequency based methods as DF work better for TF-IDF in this collection, since they do not remove these terms unless they have higher DF than the upper threshold.

128

Fuzzy Combinations of Criteria: An Application to Web Page Representation for Clustering

Focus our attention in FCC MFT, its results compared to FCC LSI show a similar relation: • In Banksearch, FCC MFT only outperforms FCC LSI in the case with larger vocabulary size. So again, it seems like LSI is able to find independent components that are hidden for FCC weighting function, i.e., the weighting function is not finding the most representative terms, as LSI is able to find a better reduction than MFT. • In WebKB, FCC MFT results are almost as bad as when using LSI. Sometimes MFT is slightly better, sometimes it is slightly worse. Then it seems that the problem is not in the reduction side, but in the term weighting function side. Comparing both weighting functions again, the conclusions coincide with Section 4.4.1: while the combination of FCC and LSI outperforms TF-IDF in Banksearch, in WebKB the best results correspond to TF-IDF helped by DF and LSI. After reviewing all these results we find three open issues that would be interesting to analyze: • The first one has to do with FCC. After showing very good results in Banksearch, it achieved the worst ones in WebKB. Even after searching for independent components by means of LSI, its results were bad. Therefore, we believe that the problem is the weighting function itself. A deeper analysis on FCC could help find the reason of this bad performance. • The second one, also commented in Section 4.4.1, has to do with the different performance of the same combinations of weighting function and reduction method in each collection. Therefore, there is no clear candidate combination that can be used regardless of the collection. As we said before and it is also seen in Table 4.4, TF-IDF LSI is the most stable option, but at the same time, it is improved by others in both collections: FCC LSI in Banksearch, TF-IDF DF in WebKB. • Finally, apart from discovering which combinations of term weighting functions and reduction techniques lead to better clustering results, a second issue emerges: the difference between using MFT or LSI with the same term weighting function. The bad performance of MFT with small vector sizes points out that the weighting functions are not estimating well the importance of the words. Our hypothesis is: if a weighting function is able to assign the highest term weights to the most representative features of a document, MFT should work similar to, or even better than LSI.

4.5 Analysis of the Combination of Criteria

4.5

129

Analysis of the Combination of Criteria

Section 4.4.2 left three open issues: (1) To study the bad performance of FCC in WebKB dataset. (2) To find a combination of weighting function and reduction technique with a more stable performance regardless the dataset than the previously tested combinations. (3) To test whether we are able to find a term weighting function that used with MFT achieves a similar performance than used with LSI. All of them are clearly related through a common need: to analyze the weighting function. For this reason, in this section we perform a study about how to improve the fuzzy combination of criteria performed by FCC for web page clustering. We will use the framework offered by FCC to further investigate about how information extracted from HTML documents can be used to enrich document representation and to improve the results obtained in clustering tasks. We divide the section in two main parts: • We perform an analysis on the combination of criteria, aiming to discover the influence of each individual criteria in the combination. • After analyzing the results of those experiments, we present and compare two new combination proposals aiming to alleviate the problems we detected.

4.5.1

Study of Individual Criteria

The first step is to analyze the contribution of each criterion in order to find some clues about why the combination does not perform in WebKB as well as in Banksearch. To do this, we repeat the clustering process modifying the combination of criteria proposed by FCC. We did four variations of this function, one for each criterion, in such a way that the output of the system will correspond only to one criterion at a time. For example, Table 4.5 shows the rule base for the system that utilizes only emphasis criterion to determine the output. It is worth mentioning that the data base of the system remains untouched, as described in Section 4.2.1. We used MFT reduction because it does not transform features and selects those with higher weight, allowing us to study the effectiveness of each alternative to give more importance to the most representative terms. Table 4.6 shows the results of each individual criterion compared to the combination, i.e., FCC. Focusing on Banksearch results, values corresponding to FCC

130

Fuzzy Combinations of Criteria: An Application to Web Page Representation for Clustering

Table 4.5: Rule base for the system based on emphasis criterion. Inputs are related to normalized term frequencies. IF Title AND Frequency AND Emphasis AND Position

THEN Importance

⇒ ⇒ ⇒

High Medium Low

Very High Medium No

Table 4.6: F-measure results for criteria analysis experiments Rep.\Dim.

100

500

FCC MFT

0.723

0.757

title MFT

0.626

0.646

emphasis MFT

0.586

0.671

frequency MFT

0.689

0.715

2,000

5,000

Avg.

S.D.

0.768

0.765

0.768

0.756

0.019

0.632

0.634

0.639

0.635

0.007

0.674

0.685

0.693

0.662

0.043

0.720

0.724

0.731

0.716

0.016

0.516

0.121

0.469

0.009

1,000

Banksearch

position MFT

0.310

0.525

0.538

0.599

0.608

0.453

0.472

0.475

0.468

0.475

WebKB FCC MFT title MFT

0.432

0.433

0.404

0.488

0.479

0.447

0.035

emphasis MFT

0.415

0.431

0.433

0.465

0.489

0.447

0.030

frequency MFT

0.441

0.460

0.460

0.468

0.446

0.455

0.011

position MFT

0.301

0.283

0.317

0.281

0.286

0.294

0.015

4.5 Analysis of the Combination of Criteria

131

are always higher than individual ones. This means that the combination contributes to improve the results over individual criteria in all cases. Besides, frequency obtains the best values, while position obtains the worst ones. We believe that frequency helps the good results of the combination more than the rest of criteria. The homogeneity within category documents affects positively to this criterion and allow to achieve good results in the combination. WebKB results are quite different. On one hand, frequency is not always the best among individual criteria and, on the other hand, FCC does not always outperform individual criteria, concretely title, emphasis or frequency have equal or higher F-measure values in some cases when vector dimension is reduced to 2, 000 and 5, 000 features. In this collection, the frequency distribution of emphasized terms shows a more restricted use of emphasis, as it was explained (see Section 3.3.2 on page 92). It could be due to the limited number of web domains and the similarity among web page contents that only come from Universities. These factors could limit the number of different writing styles, fact that would be reflected in a less scattered distribution of emphasized term frequencies. Until now, we did not have any other information to confirm our belief, but the good results of emphasis with the largest vocabulary sizes lead us to confirm that emphasized terms in WebKB are particularly meaningful. The same consideration about the restrictions on the creation of WebKB can explain the good results achieved by title criterion. We can expect that authors use titles in a similar way than emphasis within the collection, as both resources are employed to highlight important words. Besides, it seems that frequency and position strongly affect FCC results, and going further, when title and emphasis could lead to a better clustering, their combination with frequency and position makes results worse. In particular, WebKB documents within categories can be much more heterogeneous than in Banksearch, factor that negatively affects frequency criterion. In this case, the combination should help to correct this issue, but it does not. Thus, it seems that frequency and position are hindering the combination. Therefore, in general, while frequency gets higher results than the other criteria, the combination works fine, but when title or emphasis outperform frequency, the combination does not work as good as it could. Thus, frequency is very important for a good grouping, as well as title and emphasis, and all of them should be very important in the combination. However, position is the criterion with the worst results in all cases, so we have to take care using it to establish the importance of a term. This bad performance of position could come from its definition, since its heuristics were based on written texts and not in web pages, where document parts could have different importance level. Although one could expect to find an introduction at the beginning and brief summaries at the end of a web page, there are different types

132

Fuzzy Combinations of Criteria: An Application to Web Page Representation for Clustering

of pages and this could not happen so often. For this reason, position in a web page should be taken into account carefully to establish term importance.

4.5.2

Improving the Fuzzy Combination of Criteria

The first step to try to improve the combination is understanding the bad performance of FCC in WebKB. We know that the problem comes from the combination, as we showed in Section 4.5.1. In FCC rules (Table 4.2), when frequency is low output can be very high (the maximum) depending on position, if title and emphasis are high. As we saw before, frequency contributes to a good clustering much more than position, so the output should reflect that fact. But, in this case, frequency is totally ignored. This occurs again when title is low and frequency medium. Both criteria are important for a good grouping, but the output is very high based on position, the same as the previous case. In these cases we are clearly underestimating the discrimination power of frequency and title. The same happens when frequency is medium, being title and emphasis low: position decides again that importance can be the minimum or not, but frequency should count more than position, as we saw before in Section 4.5.1. Summarizing, FCC overestimates the contribution of position, underestimating at the same time the discriminative power of title, emphasis and frequency. When high frequencies favored the clustering, as in Banksearch where frequency got always better values than the rest of the criteria, the combination does not suffer the effect of this problem so much. However, when frequency is not so determinant in the clustering, as in WebKB, it seems that the problem has a greater effect on the results. On the other hand, the high number of rules in FCC makes the possible combinations more difficult to understand. As the fuzzy system is able to combine the conclusions of the rules, another possibility for the knowledge base is the use of a set of single-input rules for each criterion. Thus, the system would calculate the output by combining the different outputs of the fired rules. In this thesis we call this approach AddFCC, which rule base is shown in Table 4.7. This approach reduces the number of cases that is needed to specify to the minimum. The data base of the fuzzy system remains untouched as it was described in Section 4.2.1. Nevertheless, if we are looking for very specific definitions for each criterion, we may miss part of the knowledge expressed in the FCC system, especially when dealing with dependencies among criteria and not all of them contribute equally to the combination, as occurs in our case. We strongly believe that an approach like AddFCC would reduce the expressiveness of the system, fact that, in some cases, could lead to mistakes as a consequence of a bad specification of the heuristic knowledge.

4.5 Analysis of the Combination of Criteria

133

Table 4.7: Rule base for AddFCC. Inputs are related to normalized term frequencies. IF Title AND Frequency AND Emphasis AND Position High Low High Medium Low High Medium Low Preferential Standard

THEN Importance

⇒ ⇒ ⇒ ⇒ ⇒ ⇒ ⇒ ⇒ ⇒ ⇒

Very High No Very High Medium No Very High Medium No Very High No

In order to avoid this problem, an intermediate approach is proposed. We refer to it as Extended Fuzzy Combination of Criteria (EFCC). Its rule base combines some criteria explicitly and for others lets the combination to the fuzzy engine (see Table 4.8). The main idea is to have two sets of rules: one for frequency and another for the rest of the criteria, in such a way that we have always at least one rule of each set fired by the system, which will combine the outputs. Thus, we simplify the problem of underestimating frequency, because both subsets are always evaluated and combined. We have also reduced the discriminative power of position criterion, that in EFCC is considered the least important. It is important to say that these rules are based on the same expert knowledge as FCC, but taking into account the issues discovered in Section 4.5.1 to obtain a better definition—in terms of a Fuzzy Rule Based System—of that knowledge. First, in Table 4.9 we show the clustering results for the two new alternatives compared to FCC. Looking at these results, EFCC improves clustering results in WebKB, while in Banksearch AddFCC outperforms the rest. Thus, EFCC improves the bad performance of FCC in WebKB in all cases, but AddFCC does not. Moreover, EFCC also achieved good results in Banksearch. It is worth highlighting the bad result of EFCC in WebKB with 5, 000 features, which is much lower than the rest of its results. EFCC weights terms more aggressively than FCC and TF-IDF, in the sense that its rules highly benefit terms appearing in titles and emphasized. As the number of these terms is limited, when we increase the number of features to select, the probability of selecting terms that appear neither in titles nor emphasized increases. When we reduce the initial vocabulary to 5, 000 features, we have started to introduce this kind of terms in the representation, and therefore the results get worse, or better said,

Fuzzy Combinations of Criteria: An Application to Web Page Representation for Clustering

134

Table 4.8: Rule base for EFCC. Inputs are related to normalized term frequencies. IF Title AND Frequency AND Emphasis AND Position High

High

High High High High Low Low Low Low Low Low

Medium Medium Low Low High High Medium Medium Low Low

Preferential Standard Preferential Standard Preferential Standard Preferential Standard Preferential Standard

High Medium Low

THEN Importance

⇒ ⇒ ⇒ ⇒ ⇒ ⇒ ⇒ ⇒ ⇒ ⇒ ⇒ ⇒ ⇒ ⇒

Very High High Medium Medium Low High Medium Medium Low Low No Very High Medium No

Table 4.9: Comparison of the fuzzy logic-based alternatives in terms of Fmeasure. Rep.\Dim.

100

500

1,000

2,000

5,000

Avg.

S.D.

FCC MFT

0.723

0.757

0.768

0.765

0.768

0.756

0.019

EFCC MFT

0.768

0.778

0.758

0.740

0.759

0.760

0.014

AddFCC MFT

0.775

0.788

0.777

0.784

0.779

0.781

0.005

Banksearch

WebKB FCC MFT

0.453

0.472

0.475

0.468

0.475

0.469

0.009

EFCC MFT

0.516

0.546

0.545

0.566

0.484

0.532

0.032

AddFCC MFT

0.459

0.493

0.494

0.491

0.471

0.482

0.016

4.5 Analysis of the Combination of Criteria

135

EFCC results approximate the results of the other representations. Besides, AddFCC obtains the best results in Banksearch in all cases. However, it leads to worse results than EFCC in WebKB. In AddFCC all the criteria contributes equally to the combination. This way, its problem in WebKB is the same we detected for FCC: position is overestimated in the combination. Thus, despite its good performance in Banksearch, it does not solve the problem of FCC in WebKB. This fact supports our belief in the need for a system where not all the criteria contribute the same to the combination. At this point, we decided to select EFCC as an alternative to FCC for our next experiments, since it solves the bad results of the other fuzzy logic based approaches in WebKB and it also achieves good results in Banksearch. In our next experiment we apply LSI in combination with EFCC to test our hypothesis about the improvement obtained by LSI over MFT (that is based on the term weighting function). We also compare EFCC results with the best results achieved by TF-IDF and FCC for each dataset in Section 4.4.2. Table 4.10 shows the comparison. Table 4.10: Comparison of EFCC with previous alternatives in terms of Fmeasure. Rep.\Dim.

100

500

TF-IDF LSI

0.750

0.755

FCC LSI

0.775

0.763

2,000

5,000

Avg.

S.D.

0.756

0.757

0.763

0.756

0.005

0.785

0.763

0.758

0.769

0.011

0.760

0.014

1,000

Banksearch

EFCC MFT

0.768

0.778

0.758

0.740

0.759

EFCC LSI

0.780

0.756

0.744

0.755

0.757

0.758

0.013

TF-IDF DF

0.469

0.521

0.530

0.534

0.532

0.517

0.027

TF-IDF LSI

0.516

0.507

0.505

0.506

0.501

0.507

0.006

FCC MFT

0.453

0.472

0.475

0.468

0.475

0.469

0.009

EFCC MFT

0.516

0.546

0.545

0.566

0.484

0.532

0.032

EFCC LSI

0.483

0.483

0.483

0.483

0.484

0.483

0.000

WebKB

Looking at EFCC, these experiments corroborates our hypothesis: with EFCC we have improved our weighting function and, as a result, MFT has achieved clustering results as good as, or even better than LSI in most of the cases, with a much lower computational cost. Besides, in Banksearch EFCC performs slightly worse than FCC, but better than TF-IDF for vocabulary sizes from 100 to 1000. In WebKB, EFCC MFT achieves the best results in all cases except one, corresponding to 5, 000 features. The case of EFCC LSI in WebKB is also interesting. When reducing to 100 features by means of LSI, the result is approximately the same than for the rest of

136

Fuzzy Combinations of Criteria: An Application to Web Page Representation for Clustering

vector sizes. Then LSI is finding almost the same independent components for all vector sizes. It is as if LSI had reached an upper bound and the number of vocabulary terms does not help to find more representative independent components. In this case, MFT reduction is able to find more representative features than LSI. Globally, EFCC MFT offers the most stable results among collections, though in Banksearch it is not always the best alternative. Thinking in applying the representation to a new collection, EFCC MFT would be the best option. Our results shows that EFCC MFT is the best combination of weighting function and reduction method among collections. TF-IDF LSI is the only alternative, but at the price of a much higher computational cost and worse results in most of the cases when reducing to 1, 000 features or less. Another idea appears looking at the results of EFCC. It seems like dealing with collections with homogeneous documents within categories that are also well balanced, EFCC MFT needs less terms to get to its maximum. In the case of collections composed of more heterogeneous documents within categories, that are also more unbalanced, the number of terms needed to obtain the best results with EFCC MFT is greater. This makes sense, because in a corpus with heterogeneous documents within categories, to find the most representative terms to differentiate the categories should be harder. Furthermore, the additive properties of the fuzzy system make possible to reduce the number of rules needed to specify the knowledge base of EFCC and therefore, the system is easier to understand. Throughout this section we have tried to answer the three issues we stated at the beginning of Section 4.5. First, EFCC MFT is able to achieve good performance in both collections. This has been possible after analyzing the worst performance of FCC in WebKB, by improving the combination on the basis of our findings. Finally, we have shown that a lightweight dimension reduction technique as MFT is able to obtain similar or even better results than LSI when used with the proper term weighting function. In fact, the good behavior of MFT depends on the term weighting function applied before. Because of this, we believe that the use of light dimension reduction techniques is a good alternative, at the price of selecting a proper term weighting function for the clustering problem we want to solve.

4.6

Criteria Beyond the Document Itself

Until now, we have used web pages content to represent them, but there exists other information that could be useful. This is the case of IDF and anchor texts, two widely used resources in clustering, as we described in Chapter 2. In this section we explore both alternatives, trying to improve EFCC by adding contextual information, i.e., information from sources external to the document itself.

4.6 Criteria Beyond the Document Itself

4.6.1

137

Inverse Document Frequency

By adding IDF we try to include information coming from the whole collection to our representation proposal. In order to add IDF function (Equation 2.5) to EFCC we use a linear combination of both: EFCC-IDF(t, d, D ) = EFCC(t, d) × IDF(t, D )

(4.8)

where t is a term, d a document, D the whole corpus. The experimental settings are the same as in the previous sections. In this case we use EFCC for the comparison because it was the most stable fuzzy logic based alternative among collections in Section 4.5.2. Table 4.11: F-measure results for efcc-idf experiments Rep.\Dim.

100

500

1,000 2,000 5,000

Avg.

S.D.

Banksearch EFCC MFT EFCC-IDF MFT

0.768 0.778 0.758 0.740 0.759 0.522 0.773 0.799 0.825 0.827

0.760 0.014 0.749 0.129

0.516 0.546 0.545 0.566 0.484 0.383 0.346 0.291 0.282 0.451

0.532 0.032 0.350 0.070

WebKB EFCC MFT EFCC-IDF MFT

Looking at the Table 4.11, EFCC-IDF works really well above 500 features in Banksearch, but really bad with 100, probably due to the penalization that IDF causes to the most common terms. In Banksearch, categories are well defined in terms of homogeneity among category documents and document sources, as we showed in our category analysis in Section 3.2.2. Then, we can assume that most important terms will be those appearing in most of the documents of a single category. Some of these terms are probably receiving high EFCC scores, but at the same time, they are penalized by IDF. Thus, by reducing to only 100 features, the possibility of losing those terms increases, which affects the representation. As we incorporate more terms to the vocabulary, the results improve, because the representative terms that were penalized by IDF begin to appear step by step. This behavior is similar to the case of TF-IDF with MFT in Section 4.4.2, but much more accentuated by the combination of EFCC and IDF. As we said before, EFCC weights terms more aggressively than FCC and TF-IDF, since its rules highly benefit terms appearing in titles and emphasized. As the number of these terms is limited, the difference between these terms and the rest should be clear in terms of importance. Thus, if IDF penalizes some of the terms that appear in titles or emphasis, when we apply MFT they will fall from the top of the document rankings to intermediate positions, and when we reduce the initial vocabulary to 100 features, we will lose them.

138

Fuzzy Combinations of Criteria: An Application to Web Page Representation for Clustering

However, the use of IDF strongly penalizes results in WebKB. In previous experiments with TF-IDF and MFT (see Section 4.4.2) we saw a penalization similar to the one we have analyzed in Banksearch. But here, the representation directly does not work. Only the case of biggest vocabulary size achieves reasonable clustering results. As we saw in Section 4.4.2, IDF strongly affects the weighting function in a negative sense for the case of WebKB. The reason can be found in the size of WebKB categories (see Section 3.3.1). The biggest one, having a 36% of total collection documents could suffer the effect of IDF on its most common terms. As mentioned above in the case of Banksearch, this situation is similar than the one analyzed before for TF-IDF MFT in WebKB (see Section 4.4.2), but much more accentuated, due to the more aggressive term weighting performed by EFCC in combination with MFT. EFCC assigns well differentiated weights to terms appearing in titles and emphasis with respect to the rest. Nevertheless, at the same time, some of these terms could be penalized by IDF due to their high document frequency as a consequence of the effect of the biggest collection categories. This effect is more accentuated in WebKB because most of its categories are bigger than Banksearch ones. Then, these terms would fall from the first positions in the rankings and we would need to introduce more features in the vocabulary to recover them, like in the case of 5, 000 features, where clustering results get better. Summarizing, the combination of EFCC and IDF does not help to find a good clustering solution in all the cases. Although it can lead to very good results, it also can drive us to very bad ones. Furthermore, in unsupervised environments where we do not have any category information, a bad selection of the vocabulary size could have a terrible effect. Particularly bad is the case of WebKB, where most of the results are far away from the rest of representations. From a general point of view, IDF introduces a penalization for common terms. However, in clustering problems we search for common terms to group documents. The compromise of both things is not easy to find in unsupervised environments. As we have seen, there are cases where IDF could not help the clustering, but the opposite.

4.6.2

Anchor Texts

For this experiment we needed to employ a recently crawled collection, in such a way that it was easy to find other web pages with hyperlinks to the collection documents. We decided to use the SODP dataset—described in Section 3.4— composed of 12, 616 documents retrieved from social bookmarking sites and classified by extracting the category for each URL from the first classification level of Open Directory Project. Thus, the entire collection is divided in 17 unbalanced categories, having from 39 to 3,289 documents each. In addition to the documents themselves, we collected the anchor texts corresponding to a maximum

4.6 Criteria Beyond the Document Itself

139

of 300 unique inlinks per each document in the collection (2,704 web pages have less than 50 inlinks, 4,717 have less than 100, so the rest, approximately 60%, have more than 100 inlinks). There are a number of ways of adding anchor texts to document representation methods. We are interested in elucidating whether anchor texts could help improve web page representation in clustering or not, but at the same time, we want to investigate different alternatives for the combination. Therefore, we decided to combine anchor texts with EFCC in two different ways: (a) In addition to each document textual content, as other document terms, i.e., they are added to frequency criterion. The idea is to analyze whether anchor texts terms are similar to document content terms and, therefore, they contribute in the same way to the combination in order to estimate term importance. Other works like Wang and Kitsuregawa (2002) and Huang et al. (2006) followed a similar approximation. (b) In addition to each document title, i.e., giving them the same importance than title terms in the combination. We aim to discover whether anchor texts are better suited than document content terms for clustering tasks. The format of anchor texts is also similar to document titles, since they usually are short texts. Besides, we did three experiments for each case: (1) Just adding anchor texts. (2) Adding anchor texts and removing text corresponding to outlinks (links from a dataset page to other pages). (3) Removing a set of stop words based on a study over collection anchor text terms. We created a list of terms appearing in anchor texts and looked at their document frequencies in the collection to find non-informative but frequent terms. This set contains words like click, link or homepage. The whole list can be found in Appendix D. This way, our experiments are oriented to find out not only whether anchor texts are useful for web page representation in clustering tasks, but also to compare different manners to include this information in our combination proposal. We perform the experimentation using the same settings as before, except the dataset, since with Banksearch and WebKB was not possible to recover a sufficient number of inlinks pointing at collection pages. As we introduce here a new collection, we decided to add FCC as baseline to validate our previous EFCC results. We also include AddFCC in these experiments due to its good performance in Banksearch (Section 4.5.2). The main aim of this decision was to ensure that by adding anchor texts to EFCC, we were adding them to the best of our fuzzy combinations for this collection. Hence, we employ SODP not only to test

140

Fuzzy Combinations of Criteria: An Application to Web Page Representation for Clustering

the usefulness of anchor texts in our representation proposal, but also to validate the results of EFCC against the other fuzzy logic-based representations in a new dataset. Table 4.12: F-measure results for anchor text experiments Rep.\Dim.

100

500

1,000

2,000

5,000

Avg.

S.D.

FCC MFT

0.195

0.237

0.254

0.256

0.266

0.242

0.028

AddFCC MFT

0.208

0.267

0.276

0.279

0.282

0.262

0.031

EFCC MFT

0.233

0.273

0.287

0.283

0.296

0.275

0.025

EFCC a-1 MFT

0.225

0.262

0.279

0.286

0.290

0.268

0.027

EFCC a-2 MFT

0.245

0.246

0.285

0.289

0.269

0.267

0.024

EFCC a-3 MFT

0.248

0.260

0.285

0.294

0.293

0.276

0.022

EFCC b-1 MFT

0.254

0.287

0.275

0.282

0.285

0.277

0.015

EFCC b-2 MFT

0.254

0.249

0.276

0.279

0.291

0.270

0.016

EFCC b-3 MFT

0.249

0.261

0.263

0.278

0.285

0.267

0.012

SODP

Table 4.12 shows the results of the different alternatives. Each one is followed by a letter and a number, corresponding to the way in which the anchor texts were added to the representation. The letters and numbers were defined above. The first three table rows show that EFCC based approaches outperform FCC and AddFCC in almost all cases. Particularly, EFCC always outperforms FCC and AddFCC in SODP. This corroborates our findings about the drawbacks of FCC, stated in Section 4.5, and confirms our belief in the need of a system where not all the criteria contribute the same to the combination, in contrast to AddFCC. Regarding the contribution of anchor texts, there is no clear alternative to improve EFCC in all cases. Looking at the averages, there are only slight differences among the different EFCC alternatives. Focusing our attention on concrete vocabulary sizes, anchor texts help improve clustering results with small vector sizes, particularly when anchor texts terms are considered as page titles (b alternative). However, when we increase vector size, they seem to introduce noise, because clustering results get worse. About using anchor texts as titles, the best option is just adding anchor texts as title terms (named b-1). It is interesting to have found an improvement for smaller vector sizes. In the best case, this improvement is about 0.02 in terms of F-measure, which corresponds to about a 9% over the result of EFCC with the same vocabulary size. However, in some scenarios this improvement does not compensate for all the process needed to collect anchor texts. A possible explanation for these results might be a poor link density or bad anchor text quality, or just the nature of clustering problems, where the aim is to capture the aboutness of documents. This conclusion coincide with other

4.7 Empirical Evaluation of the Proposed Modifications

141

works like Eiron and McCurley (2003); Noll and Meinel (2008) (see Chapter 2, Section 2.2.1.3 on page 49), where authors conclude that anchor text terms are more similar to terms used in search queries. Also they experimentally found that these terms are not often in web page contents, and therefore this information is not so good for capturing the aboutness of web documents.. Finally, it is important to highlight that, in general, all the results are very bad in this collection. SODP is composed of 17 unbalanced categories and there is also a clear bias towards Computers category that contains a 26% of documents. Besides, the other 74% of documents is divided in 16 categories with different number of documents each. Terminology from bigger categories will favor the division of documents belonging those categories instead of finding smaller ones. Thus, finding a clustering for SODP collection that corresponds with the categories in the gold standard is very difficult.

4.7

Empirical Evaluation of the Proposed Modifications

In this section we perform a robust evaluation of EFCC to be sure about whether or not exists a real improvement over FCC. As we are using a deterministic algorithm, we want to avoid the possible bias introduced by feeding the algorithm with a single set of vectors for each dataset. The solution presented here involves dividing each dataset in 100 different sub-datasets 50% smaller than the original, where the categories are in proportion to the original ones. We performed 100 experiments per each vector size and each sub-dataset, resulting a total of 3,000 different clustering experiments. Due to computational reasons, we chose MFT reduction for all the experiments. This decision was also made to compare both term weighting functions in the exactly same conditions. Besides, MFT does not transform features, but selects those with higher weight, allowing us to study the effectiveness of each alternative to give more importance to the most representative terms. Basically, we want to ensure that EFCC and FCC lead to different results. Nevertheless, the average value of a set of experiments it is not enough to ensure that one of them is better than the other. To make such a statement we decided to employ statistical significance analysis over the samples of both methods. In our case we use statistical hypothesis testing for paired samples, because our data come from applying both methods over the same datasets. In particular we employ a paired two-tailed t-test over the results obtained by both representations for each concrete vector size in the 100 sub-datasets. This test assumes that the error follows the normal distribution, but it often performs well even when this assumption is violated. The t-test is relatively robust to many violations of nor-

142

Fuzzy Combinations of Criteria: An Application to Web Page Representation for Clustering

mality and only heavy skewness6 or large outliers7 will seriously compromise its validity (Hull, 1993). The t-test is a statistical hypothesis test where the null hypothesis is that the values we want to compare are drawn from the same population. The alternative hypothesis depends on the type of t-test we apply. For two-tailed tests, the alternative hypothesis is that the values we want to compare come from different populations, that is, their means are different from each other. More formally, for two populations X and Y that we want to compare and paired samples ( x1 , y1 ), ..., ( xk , yk ) the hypothesis for the t-test would be: • Null hypothesis: µ X − µY = ∆µ • Alternative hypothesis: µ X − µY 6= ∆µ where µ X is the mean of X population, µY the mean of Y population and ∆µ is the difference between both means. Then, the t statistic is calculated as: T=

D − ∆µ √ SD / n

(4.9)

where D is the sample mean of the differences, i.e., the differences between all pairs (Equation 4.10), ∆µ is the difference between the populations means (Equation 4.11) and SD is the standard deviation of the above mentioned differences (Equation 4.12): 1 k D = X − Y = ∑ xi − yi (4.10) k i =1 ∆ µ = µ X − µY v u u k 2 u u ∑ (( xi − yi ) − D ) t i =1 SD = k−1

(4.11)

(4.12)

By using the Student’s T distribution with k − 1 degrees of freedom, the value of T is converted in a probability value P( T ) (also noted as p-value): the probability of obtaining a difference like the observed D, if the real difference between the means of the populations is ∆µ . If confidence in the hypothesis (reported as p-value) is lower than a significance level α, then it is common to assume that values are from different populations with likelihood greater than 1 − α to reject the null hypothesis (Sanderson and Zobel, 2005). It is usual to set al pha to 0.05 or 0.01 as upper bound. Other works like Zhao and Karypis (2004); Higa and Tozzi (2008); Farahat and Kamel (2010) took a similar approach. 6 Lack

of symmetry in the distribution. that are numerically distant from the rest of the data.

7 Observations

4.8 Conclusion

143

Table 4.13: F-measure results for EFCC/FCC t-test experiments Rep.\Dim.

100

500

1,000

2,000

5,000

0.764 0.718 0.047 0.000

0.774 0.760 0.014 0.000

0.770 0.765 0.006 0.002

0.760 0.768 -0.008 0.000

0.753 0.768 -0.015 0.000

0.487 0.446 0.041 0.000

0.514 0.462 0.051 0.000

0.528 0.470 0.059 0.000

0.534 0.485 0.049 0.000

0.483 0.490 -0.007 0.016

0.230 0.200 0.030 0.000

0.271 0.233 0.037 0.000

0.279 0.246 0.033 0.000

0.282 0.251 0.031 0.000

0.289 0.266 0.023 0.000

Banksearch EFCC MFT FCC MFT Difference p-value WebKB EFCC MFT FCC MFT Difference p-value SODP EFCC MFT FCC MFT Difference p-value

In Table 4.13, for each vector size and representation we show the average F-measure values corresponding to the 100 clustering experiments (one per each sub-dataset), the difference between the corresponding averages, and the p-value resulting of applying the statistical t-test between the samples corresponding to both representations. Attending to p-values, in all cases except one, we can say that values are from different populations with likelihood greater than 99%. In those cases the p-values are highlighted in bold font. Besides, looking at the averages, in most of the cases EFCC outperforms FCC. Regarding differences between representations, just in three cases FCC performs better than EFCC, being the difference lower than 0.01 in two cases and lower than 0.02, that corresponds to an improvement of a 1.9% of FCC over EFCC, in the other. In the rest of the experiments EFCC gets an improvement over FCC, higher than 0.03 in SODP (corresponding to more than a 12%), and greater than 0.04 in WebKB (corresponding to about a 10% in all cases), and also with the smallest vector size in Banksearch (corresponding to about a 6%).

4.8

Conclusion

Throughout this chapter we have studied how to combine different criteria extracted from the information contained in HTML web pages to represent them for document clustering. We have explored the possibilities of fuzzy systems to help apply expert knowledge and combine these criteria, and tested our findings

144

Fuzzy Combinations of Criteria: An Application to Web Page Representation for Clustering

with three different datasets. Parts of the research in this chapter have been published in Pérez García-Plaza et al. (2008) and Pérez García-Plaza et al. (2012a). We believe that fuzzy combinations of criteria, as FCC, fits better the problem of establishing term importance, because there are dependencies among criteria that should be taken into account in order to deal with authors’ writing style, automatism in web page creation (as titles automatically generated by HTML editors), etc. In general, to deal with heuristic knowledge about how to establish term importance by means of combining different criteria. These facts encourages us to explore the way on which criteria can be combined by means of a fuzzy system. Our work is oriented to propose a web page representation method that works reasonably well regardless the collection of documents. In this sense, achieving the best results is not the most important condition to fulfill. In our research, we consider more important to obtain stability among collections. That is to say, the same representation model (weighting function and dimension reduction technique) should achieve good results in different collections. We studied different dimension reduction techniques in order to apply some methods that had not been used in the context of FCC. We have shown that with a good weighting function it is feasible to employ lightweight dimension reduction techniques, as the proposed MFT, instead of using other more complex techniques like LSI, which implies an important reduction in the computational cost. Regarding LSI, MFT and DF, as the first one is computationally more expensive and it transforms the features—losing the reference between the original features and the final vector values—, we find MFT and DF more appropriate in order to study the weighting functions, as they keep the features as they are, allowing the direct analysis of the functions. Another interesting issue is that DF showed to be useful in concrete circumstances, particularly when dealing with collections composed of heterogeneous web pages coming from a small number of web domains. In these cases, DF can help removing too common terms and web domain related terminology spread throughout the collection. This is the case of collections like WebKB, described in Section 3.3, where the probability of sharing terminology among categories is high, given the heterogeneity of documents within collection categories. It is worth mentioning that RP method was also tested as a lightweight alternative to LSI, but our experimental results showed that it is not a good alternative. Moreover, the experimental results of our initial tests on dimension reduction techniques showed the bad performance of FCC in WebKB collection. For that reason, we analyzed FCC, finding some issues that could cause its bad perfor-

4.8 Conclusion

145

mance in WebKB dataset. We detected that some individual criteria performed better isolated than the FCC combination. After identifying the aspects that, in our opinion, hinder FCC—the overestimation of position, underestimating at the same time the rest of the criteria as detailed in Section 4.5.2—, we proposed two alternative ways of combining criteria, AddFCC and EFCC, within the same fuzzy logic framework. Our experiments showed that EFCC worked better than FCC by means of a different way of combining criteria, where term frequency is considered as discriminant as title and emphasis, and position is taken into account as the least important criterion. This approach makes also possible to reduce the number of rules needed to specify the knowledge base taking advantage of the additive properties of the fuzzy system. Thus, it makes the system easier to understand. In contrast, AddFCC obtained very good results in Banksearch, but its clustering results in WebKB were bad. The problem of AddFCC comes from the way of combining criteria, where all the criteria contribute the same to the combination. This fact supports our belief in the need for a system where not all the criteria contribute the same to the combination. In order to continue exploring new criteria for the combination, we considered to employ collection information and anchor texts to represent documents. First, we evaluated the possibility of a linear combination between EFCC and IDF, but we rejected this alternative based on the bad experimental results we found in some cases, particularly in WebKB. Second, we also used anchor texts to enrich document representation with contextual information. Although results were not bad, they were not clearly better than the ones obtained by EFCC. Besides, the cost of preprocessing anchor texts and their dependence on link density limit the applicability of this alternative. For these reasons, we believe that it could be an interesting option when a collection fulfills these requirements and time complexity is not a problem, but in most of the cases this will not happen and we will have to carry out document representation only with document contents. Finally we performed statistical significance tests to ensure that the application of our findings in our representation proposal has a real effect compared to FCC, as both representations relies on the same fuzzy engine. We conclude that our proposed approach, EFCC, improves the results of FCC in most cases. It is particularly interesting the good results obtained with vocabulary sizes below 1, 000 features, since using smaller vocabularies allows to reduce the computational cost of the clustering algorithm.

146

Fuzzy Combinations of Criteria: An Application to Web Page Representation for Clustering

5 Fitting Document Representation to Specific Datasets by Adjusting Membership Functions

Fuzzy ruled-based systems have been successfully used to represent web documents by means of heuristic combinations of criteria. In these systems, rules were established based on the way humans have a quick look at documents in order to extract the essential information, as we have analyzed in previous chapters. However, membership functions parameters were fixed by default, assuming that any document would follow similar patterns regardless of the rest of documents in the collection. In the current chapter we analyze to what extent collection information could be used to adjust the membership functions in order to improve document representation and, therefore, clustering results. We compare our proposal to the previous systems we have been working with in this thesis, particularly with the fuzzy ones in which it is based. Results show that adjusting document representation parameters to concrete collections leads to better clustering results when collections present particular characteristics.

148

Fitting Document Representation to Specific Datasets by Adjusting Membership Functions The chapter is organized as follows. First, in Section 5.1 we briefly introduced the main ideas of the chapter. Next, in Section 5.2 we review works related with fuzzy rule-based system tuning. Then, in Section 5.3 on page 151 we describe the problem we study in this chapter, analyzing every aspect in detail and presenting our approach to deal with it. Next, we evaluate our proposal in Section 5.4 on page 160 and finally we present our findings to conclude the chapter in Section 5.5 on page 167.

5.1

Introduction

In previous chapters we have shown that, although using term frequencies is a common approach to represent documents, Fuzzy Rule-Based Systems (FRBSs) have become an effective alternative in order to exploit other additional information that HTML tags provide. Moreover, FRBSs can be tuned to fit a concrete problem, aiming to improve their results. This fact is especially interesting when we consider the problem of document representation, where each document collection may differ from the rest. For example, Banksearch (see Section 3.2.2 on page 86) and WebKB (see Section 3.3.2 on page 92) show clear differences between their term distributions, particularly for emphasized terms. Therefore, our initial hypothesis in this chapter is that some dataset characteristics could have an influence on the way of defining the FRBS for representing the documents belonging to that concrete collection. As we will see later in this chapter, these characteristics has to do with term frequencies, since they could allow to better capture the information related to each criterion. Fuzzy modeling (FM) is problem modeling by means of FRBSs. One of the most important fields within FM is linguistic FM. As stated in Casillas et al. (2005): “the two main requirements in FM are interpretability, capability to express the behavior of the real system in a comprehensible way, and accuracy, capability to faithfully represent the real system”. Throughout this chapter we analyze the possibility of adjusting a FRBS developed for web page representation to different datasets. We evaluate our results to elucidate whether this adjustment can improve clustering results or not. In addition we perform a statistical significance test to verify our results compared to those obtained by other FRBSs whose parameters are fixed by default.

5.2

Tuning of Fuzzy Rule-Based Systems

In this work we are interested in adapting a FRBS to improve web page representation for clustering tasks. Before going into further details it is worth to remember that the knowledge base of a fuzzy system is composed of the rule

5.2 Tuning of Fuzzy Rule-Based Systems

149

base, that is the set of rules, and the data base, which contains the membership functions associated to the linguistic variables. Regarding FRBSs tuning aiming to extract a suitable set of fuzzy rules from numerical data, in Casillas et al. (2005) the authors presented a genetic tuning process for knowledge base refinement. They focus their work in maintaining a good trade-off between accuracy and interpretability by reducing the rule set in a first step and tuning the resulting system next. They conclude that this order is crucial to obtain good results in terms of efficiency and accuracy. They applied their method to two real world problems as benchmarks: the rice (Nozaki et al., 1997) and electrical (Cordón et al., 1999) problems, concluding that tuning and reduction processes can significantly improve the accuracy of a fuzzy model. A similar approach focused in obtaining more compact models was presented in Gacto et al. (2008). To do so, they used Multi-Objective Evolutionary Algorithms (MOEAs) as a tool to get an improved solution with respect to a classic single objective approach. They also evaluated their method using two datasets about real world problems: the electrical problem, the same as the previously commented work, and the Abalone dataset (Waugh, 1995). Their results showed that an appropriate use of MOEAs can help obtain more accurate and simpler linguistic models than those obtained by only considering performance measures. Again, in Li et al. (2010) one of the main tasks is to learn fuzzy rules from examples of the problems, but the aim of the authors was to avoid local convergence as a result of the increasing complexity and dimensionality in classification problems. They used a fitness sharing method based on the similarity level of each rule and its neighbors rules. Their method was studied for sonar signal classification (Ishibuchi et al., 2005) and hand movement recognition (Dias et al., 2009) problems. Their experimental results show an improvement over two classical genetic machine learning approaches, that are widely used in the construction of fuzzy rule based classification systems: Pittsburgh approach (Venturini, 1993), where the whole rule set is handled as a genetic individual, and Michigan approach (Booker et al., 1989), where each rule is considered an individual. Another recent work exploiting MOEAs to generate FRBSs proposed to adopt partition integrity (interpretability of fuzzy partitions) as an objective of the evolutionary process (Antonelli et al., 2011). The authors introduced a three-objective evolutionary algorithm which generates a set of FRBSs with different trade-offs between complexity, accuracy and partition integrity by concurrently learning the rule base and the membership functions parameters. They experimented on six real-world regression problems, where their approach improved results of applying the same MOEA, but with accuracy and complexity as objectives only. They also compare their work with a similar approach proposed by Gacto et al. (2010), showing that their solutions were characterized by a better trade-off between complexity and accuracy.

150

Fitting Document Representation to Specific Datasets by Adjusting Membership Functions

A more complete review on the most representative genetic fuzzy systems relying on Mamdani-type FRBSs to obtain interpretable linguistic fuzzy models can be found in Cordón (2011). There are two important differences between the above mentioned works and the problem we deal with in this chapter. First, there are no works oriented to web page representation among those oriented to FRBSs tuning. Second, all the above mentioned works employ real data samples in the learning process, that is, they employ an approach corresponding to the supervised classification field, since they use training information to adjust the system. In our case, we deal with datasets composed of web pages and we represent them from an unsupervised point of view, by utilizing only their own contents. Thus, although the previous approaches are not oriented to document representation for clustering, but to classification tasks, it is interesting for this dissertation to briefly summarize how the fuzzy systems are modified to fit concrete problems. Different from the above mentioned works, in document clustering we do not have category information. However, it is possible to analyze the documents we want to cluster to find common patterns or particular features that can help improve the way of capturing criteria information. This work focuses on adapting membership functions to dataset concrete features, which could enhance document representation, leading to improve clustering results. Tuning of Membership Functions In general, there are two main ways of tuning membership functions: • Changing basic parameters (Casillas et al., 2005; Gacto et al., 2008; Li et al., 2010) (see the left side of figure 5.1). It involves varying the shape of the fuzzy set associated to the membership function by modifying these parameters (a, b, c and d in the figure 5.1). • Using non-linear scaling factors Liu et al. (2001); Casillas et al. (2005) (see the right side of figure 5.1). This approach is based on changing the membership function in a nonlinear fashion, but does not modify the basic parameters and, when dealing with symmetrical fuzzy sets, their centers of mass do not change. Some works like Li et al. (2010) used the most similar triangular shapes to replace the curve shapes (this work was reviewed in the previous section 5.2). It is worth mentioning that both tuning alternatives are not exclusive but complementary. In this chapter our goal is to automatically adjust a FRBS to better capture the information of the documents we want to represent and cluster. The main

5.3 Problem Analysis

151

Figure 5.1: Left: Example of changing basic membership function parameters (a, b, c and d in this case); Right: Example of tuning by using non-scaling factors. Dashed lines show possible results of each type of tuning. difference with previous works is that this adjustment must be done before the clustering process, so the information available is that contained in dataset documents themselves. In this scenario, our hypothesis is that fuzzy sets can be adjusted to better fit input data, in order to improve the process of information capture. The effect of this improvement should be reflected in document representation and, therefore, in clustering results. As a mean to do that, we explore the effect of modifying fuzzy sets basic parameters taking into account the particular characteristics of each dataset. We analyze statistical data of each criterion within every dataset in order to find these particular characteristics and adjust the system. The use of non-linear scaling factors is out of the scope of this thesis, since it requires an objective function to maximize while performing the adjustment, and these functions are usually based on category information employed for training.

5.3

Problem Analysis

As we see in Chapter 4, both systems, FCC and EFCC, use the same membership functions (see figure 4.3), where the input frequencies for each criterion are normalized using the maximum number of appearances of a term in the corresponding criterion within the document (since we want to grant independence to the rules regarding the document size). In order to study the problem of adapting document representation to concrete datasets, we first analyze three different document collections to find differences and common patterns and then we study the original membership functions to discover if they could fit better each dataset, proposing an automatic way of adjusting them.

5.3.1

Datasets

In the previous chapter we performed several experiments with our fuzzy systems on three different datasets: Banksearch, WebKB and SODP. Our first step here is analyzing them to check whether they follow similar patterns or not. In

152

Fitting Document Representation to Specific Datasets by Adjusting Membership Functions

Chapter 3 we analyzed the datasets one by one, showing their main features and giving a general idea of their differences and similarities. Nevertheless, we did not do a direct comparison among them. In this section we come back to the term distribution analysis in order to fill this gap. As we tackle the problem of web page representation from an unsupervised point of view, we are interested in term distribution analysis. Category analysis is based on supervised information, so it remains out of the scope for adjusting the system. In this thesis, this kind of information—supervised information—is used only with evaluation or result analysis purposes.

(a) Banksearch term frequencies

(b) WebKB term frequencies

(c) SODP term frequencies

Figure 5.2: Comparison of normalized term frequencies for frequency criterion in Banksearch, WebKB and SODP. Figure 5.2 shows normalized term frequencies—corresponding to frequency criterion—in the three collections. Each bar represents the amount of terms

5.3 Problem Analysis

153

having a concrete normalized frequency. All of them show a tail of terms with high frequency, but WebKB seems to have a different distribution under 0.5 value. In WebKB there are more terms with intermediate frequencies, that is, not as far from document maximum values than in Banksearch or SODP. Thus, the long tail we found in Banksearch and SODP from frequencies above 0.2 is not clearly visible in WebKB, since the bars in the distribution do not always decrease with respect to the immediately previous one. However, if we divide the space between 0.2 and 1 in quartiles, we found almost the exact same values for Banksearch, SODP and WebKB. Then, the number of terms between 0.2 and 1 follow a similar proportion among quartiles in all the collections. In the case of WebKB the tail could be not seen as clearly as in the other cases due to smaller maximum values per document, that lead to a smaller set of possible values after normalization. Having less possible values, we can expect to find the term frequencies more concentrated in concrete points, like in Figure 5.2b. This way, tails look different in the figures, yet splitting them in quartiles shows that they are not so different at certain degree of detail. Due to the scale imposed by the normalization process, terms having low frequency values should be considered as noise because all of them are far away from document maximum value and there is no way to difference among their representativeness due to the high term density in that area. At the same time, in Banksearch and SODP there are few terms with high frequencies, probably because document maximum values are far away from the rest. Figure 5.3 shows normalized frequencies for emphasized terms—emphasis criterion—in the three collections. In this case the Banksearch and SODP datasets show very similar distributions, but they are totally different compared with WebKB distribution. In our opinion, this fact implies that emphasis have been used by web page authors in a different manner. Emphasized terms have higher frequencies than in other collections. Besides, the bar corresponding to a frequency of 0.5 is clearly the larger one. This indicates that there are a small set of possible values within each document, that makes the distribution tend towards the maximum. This reflects a more restricted use of emphasis in WebKB, where less terms are emphasized and that emphasis is probably more meaningful. Finally, figure 5.4 shows normalized frequencies for title terms, inputs for title criterion, in the three datasets. In this case, as titles are usually short text strings, there are not much different possible values. In most cases, the whole set of title terms of a document will appear just once, so the whole set will have the maximum frequency. The second most probable case will be when the term with maximum frequency appears twice, and then there will be two possible values: 1 and 1/2. Even with these extreme conditions, Banksearch and SODP look more similar between them than compared with WebKB.

154

Fitting Document Representation to Specific Datasets by Adjusting Membership Functions

(a) Banksearch term frequencies for emphasis

(b) WebKB term frequencies for emphasis

(c) SODP term frequencies for emphasis

Figure 5.3: Comparison of normalized term frequencies for emphasis criterion in Banksearch, WebKB and SODP.

Summarizing, these datasets allow us to verify that different document collections can have different features that should be taken into account when representing their documents to establish the importance of their terms in each criterion. Banksearch and SODP presents different features than WebKB, which is the most different one. Banksearch and SODP tend more clearly to exponential distributions for frequency and emphasis. But WebKB, in particular for emphasis, tends more to a uniform distribution.

5.3 Problem Analysis

155

(a) Banksearch term frequencies for titles

(b) WebKB term frequencies for titles

(c) SODP term frequencies for titles

Figure 5.4: Comparison of normalized term frequencies for title criterion in Banksearch, WebKB and SODP.

How to interpret these distributions is a key factor for adjusting the membership functions: • Exponential distributions tell us that most of the terms are indistinguishable in terms of frequency, since they are too far from document maximum. In this case the first goal is to separate these terms from the rest, that will be the really representative ones. Then, we could establish different importance levels for the rest by dividing them in a relative manner with respect to the maximum.

156

Fitting Document Representation to Specific Datasets by Adjusting Membership Functions

• A more uniform distribution, like the case of emphasis in WebKB, points out in the opposite direction, i.e, there are more representative terms, since their term frequencies are higher compared to the maximum. As we commented above, this kind of distribution corresponds to a small set of possible values within each document (probably due to smaller maximums). This small set of possible values in turn reflects a more restricted use of emphasis in WebKB, where less terms are emphasized and that emphasis is probably more meaningful. In contrast, for titles there is a small set of possible values and then frequency distributions look very similar in all cases.

5.3.2

Analysis of the Membership Functions

One question that needs to be answered is whether considering the same partitions in membership functions regardless the dataset is the best option when dealing with term frequencies in a document. Moreover, these sets were defined taking into account the possible input values, i.e., frequencies from 0 to 1, and not the concrete input values corresponding to the term frequencies for each criteria in a given collection. We saw in section 5.3.1 that different datasets could have different frequency distributions for each criterion. Terms are commonly distributed in such a way that most of them have very low frequency and, as the frequency increases, the number of words having those frequencies in the document decreases. Besides, the effect of normalizing frequencies—to the maximum frequency of a term in the document for each concrete criterion—is that low values are compressed, making impossible in practice to make any distinction among their representativeness, further than all of them are far from the maximum on each corresponding document. Notice that this compression effect would be even worse if the normalization process had been performed by using the total maximum in the collection or the sum of all the term frequencies for a criterion in a document. Looking at the original fuzzy sets defined for FCC and EFCC (Figure 4.3 on page 114) and comparing them with the frequency distributions extracted from Banksearch, WebKB and SODP (Figures 5.2, 5.3 and 5.4), it seems that those sets does not fit the tails as much as they could, in the sense that they do not take into account the number of terms in the collection belonging to those sets. For instance, the long tail in Banksearch and SODP term frequency distributions could lead to consider in the high fuzzy set only the terms with maximum frequency value in each document. The fuzzy sets for FCC and EFCC were defined as symmetrical, except for emphasis. In fact, symmetrical sets are also defined as the initial state of most of the FRBSs tuning processes. Thus, some of the fuzzy sets defined for FCC and

5.3 Problem Analysis

157

EFCC coincide with this initial state. As we saw in Section 4.2.1, emphasis is considered separately because when the maximum frequency value for emphasized words in a document is small, the normalization could mislead us about the importance of other emphasized terms. For this reason, the sets for emphasis were asymmetrically defined (see Figure 4.3c), with bigger medium and high sets and a smaller low set, i.e., the low set for emphasis will only include terms with small emphasized frequency that also correspond to documents with high maximums. Moreover, we believe that what we call high or low are not absolute values, but relative values. This is the main point to capture criteria information in a better way. A term is not very important because its normalized frequency is 1, it is important because its normalized frequency is higher than most of the rest of term frequencies. This way, what we consider high, medium or low should depend on the concrete frequency distribution of the dataset. In an ideal case, all term frequencies would be uniformly distributed between 0 and 1, and then we could configure the fuzzy set basic parameters using the original heuristic information, e.g., with equal size sets for frequency (see Figure 4.3a on page 114), because of the relative difference among frequencies, which would be uniformly distributed. Nevertheless, we are working with texts, so in most of the real cases their term frequency distributions will tend to follow Zipf’s law, that establishes an inversely proportional relationship between the frequency of a term and its ranking in the frequency table. Thus, the frequency for the term with maximum number of occurrences in a document will be approximately the double of the frequency corresponding to the second most frequent term, and so forth (Manning and Schütze, 1999). Actually, we could expect to find different term distributions depending on different aspects like thematic divergence among categories, web page author’s style, web domain of pages in the dataset, etc. Some of these aspects were analyzed in detail Chapter 3 and reviewed in the previous section. As we saw, though tending to Zipf’s law in most cases, we could also find distributions in some point between the ideal case, exposed above, and a power law distribution like stated in Zipf’s law. At this point, it is clear that the question that opened this section needs to be clarified. Each particular dataset will have its own features and membership function tuning could be a useful way to improve the information capture process. As the input data patterns are not always the same, we strongly believe that the way of capturing criteria information can not be always the same. More than that, even if the original fuzzy sets defined for FCC and EFCC were valid for all the cases, it would be interesting to be able to obtain them in a more automated way.

158

Fitting Document Representation to Specific Datasets by Adjusting Membership Functions

5.3.3

Tuning of the membership functions

In previous sections we talked about the input criteria and the shape of their frequency distributions. Now we focus our attention in how to adapt our fuzzy system to these distributions in order to capture the input information in a better way. In order to automatically adjust the membership functions basic parameters, we assume two base cases in our hypothesis: 1. Text in web pages follows the Zipf’s law. Figure 5.2 shows that a long queue appears in the three collections we are working with, with most of their terms falling between 0 and 0.2 term frequency values. 2. Web page authors sometimes want to stress concrete terms they consider important to understand the document. This is the case of emphasized terms, whose frequency distribution can be totally different depending on how the author use them. Looking at figure 5.3 we can see the case of Banksearch and SODP, with emphasized term frequencies following a distribution more similar to a power law, and the case of WebKB, with a totally different distribution, due to, in our opinion, a more restricted and meaningful use of emphasis. The first base case corresponds to frequency criterion. We consider we have a distribution tending to a power law when the majority of terms, i.e., more than a half of them (55%) have normalized frequencies below 0.2. Depending on whether this condition is fulfilled or not, we set the membership functions as follows: • When the precondition is fulfilled, we assume a distribution tending to a power law, with low frequencies for the most of the terms. As we need 5 intervals to build three sets (low, medium and high and two intersection areas between them, see Figure 4.3a), our worst case would be to have only one possible value for each interval, that is, a maximum frequency of 5. With a maximum of 5, there would be just one possible value for each interval, as we would have only 5 possible values. Thus, to guarantee at least one possible value for the low set in that case, we chose the first interval from 0 to 1/5. This aims to separate noise from important terms. Finally, the rest of the intervals are selected using equidistant percentiles for term frequencies from 1/5 to 1, because in this way we consider high frequencies relative to intermediate ones, excluding terms with low representativeness, which we call here noise and would correspond to term frequencies below 1/5. Of course, it would be possible to have a maximum value smaller than 5, but taking our precondition into account, we are sure than more than 55% of terms in the collection belong to documents with maximum above 4,

5.3 Problem Analysis

159

because their normalized frequency is smaller than 0.2. Moreover, small maximum values will correspond to short documents. Our system will give more importance to terms belonging to short documents, since the frequency of these terms will be closer to the maximum than in longer documents. Therefore, we can expect the behavior of our system will be reasonable even in those cases. • When our precondition is not fulfilled, then we assume the distribution tends to be more uniform, so we can establish the fuzzy sets with the original heuristic, that is, all of them will have equal size. We use the corresponding percentiles to fit the distribution slightly better than using exact values (0.2, 0.4, 0.6, 0.8, see Figure 4.3a on page 114). Notice that in case of a uniform distribution, the adjustment corresponding to the first case—distribution tending to Zipf’s law—would lead to these exact values too, because as the distribution moves towards a uniform distribution, the percentile 0.2 will approximate to 1/5 and the rest of the parameters correspond to equidistant percentiles relative to this initial value in both cases. In those cases, the fuzzy sets would be symmetrical, that it is not only the case of the original sets in FCC and EFCC, but also the initial case used by most of the FRBS tuning methods we reviewed previously. About the emphasis criterion, we follow the same precondition as with frequency to determine whether or not the distribution tends to a power law, but modifying the fitting rules due to the different meaning of emphasis. It is worth mentioning that the original intervals used for the emphasis sets were not of equal size. This decision was made because of the meaning of emphasis, which is used by authors to stress terms, so its use will be more restricted. Again, we have two alternatives for emphasis: • When the distribution tends to a power law, we set the first interval as in the frequency case, and the rest with decreasing percentiles, each one a half of the previous. The reason is that in the original heuristic-based fuzzy sets, the medium set is the biggest one, and we want to preserve the original heuristic knowledge, but always taking into account the relative difference between the number of elements in each set instead of absolute exact values, as we explained in Section 5.3.2. • If the collection do not fulfills our precondition, we assume the distribution tends to be more uniform, so we can establish the membership functions basic parameters by using the original heuristic rules but, as in the case of frequency, we use the percentiles instead of the exact values to fit slightly better the distribution (in this case the values were 0.05, 0.15, 0.55, 0.75, see Figure 4.3c on page 114).

160

Fitting Document Representation to Specific Datasets by Adjusting Membership Functions

The special case of titles was also mentioned in section 5.3.1, and actually there are not much possibilities to adjust their membership functions. We just try to do it taking into account the worst case in which we have a possible value for each interval. As this is an extreme case, we use the lowest value of the distribution to set the first interval, dividing the rest of the space in equidistant percentiles. The idea here is to guarantee that the low set will have at least one value despite of the small number of possible values. Finally, it must be noted that we do not adjust the auxiliary system because word positions on a page do not depend on anything else than the number of words in the document. Summarizing the ideas behind our proposal, we believe that membership functions should change depending on the inputs in order to keep their correct meaning, because the distribution of term frequencies could be different on each dataset. As the meaning of each criterion is different, we use different approaches for each of them. We have seen that this kind of distributions raises two problems, specially when they tend to a power law. First, words with low frequencies are usually not representative enough for the document theme, so establishing the low fuzzy set is crucial to remove noise in the final representation. Second, and supposing we are able to remove noise with the low set, we have to configure the rest of the intervals. Our method sets the basic parameters of membership functions to fit distributions that tends to follow the Zipf’s law. At the same time, as these distributions tend to have more uniformly distributed frequency values, our system progressively adapts its membership functions towards the case where all the frequencies are uniformly distributed.

5.4

Empirical Analysis

The experimental settings we use in order to evaluate our proposal are the same as in the previous chapter. As we present a modification over the fuzzy logic based approaches, it is natural to use FCC and EFCC as baselines. In addition, TF-IDF will be also used. Thus, we start our research using these three weighting functions in our experiments in addition to our new proposal, called here Abstract Fuzzy Combination of Criteria (AFCC). We apply MFT dimension reduction technique in all cases in order to compare the weighting functions in the same conditions. We also take into account the conclusions of the previous chapter about the fact that MFT reduction is suitable in order to study the weighting functions, as it keeps the features as they are, selecting those with higher weights. This allows the direct analysis of the weighting functions. Table 5.1 shows F-measure results for these representations, with the best ones for each vector size in bold font. Averages and standard deviations for each method are also shown in the table.

5.4 Empirical Analysis

161

Table 5.1: F-measure results for membership functions experiments Rep.\Dim.

100

500

1000 2000

5000

Avg.

S.D.

Banksearch TF-IDF MFT FCC MFT EFCC MFT AFCC MFT

0.703 0.737 0.768 0.772 0.758 0.723 0.757 0.768 0.765 0.768

0.748 0.028 0.756 0.019

0.768 0.778 0.758 0.740 0.759 0.767 0.785 0.787 0.757 0.753

0.760 0.014 0.770 0.016

0.385 0.438 0.466 0.498 0.513 0.453 0.472 0.475 0.468 0.475

0.460 0.051 0.469 0.009

0.516 0.546 0.545 0.566 0.484 0.528 0.580 0.579 0.589 0.549

0.532 0.032 0.565 0.025

0.244 0.300 0.293 0.307 0.323 0.195 0.237 0.254 0.256 0.266

0.293 0.030 0.242 0.028

0.233 0.273 0.287 0.283 0.296 0.233 0.269 0.292 0.284 0.282

0.275 0.024 0.272 0.023

WebKB TF-IDF MFT FCC MFT EFCC MFT AFCC MFT SODP TF-IDF MFT FCC MFT EFCC MFT AFCC MFT

On the one hand, looking at the results, among the fuzzy logic based representations, AFCC outperforms the rest in WebKB in all cases, while in Banksearch got better results than the others with 2 out of 5 vector sizes, having also a higher average F-measure. This different performance between collections could be due to the fact that Banksearch frequency distributions tends to power law, and in those cases, once the less important terms have been assigned to the low fuzzy set, there are not so much terms to assign to medium and high sets, so the difference with EFCC and FCC fixed sets is small. Comparing AFCC with EFCC, it gets better results in Banksearch in 3 out of 5 cases, while with 100 features both performs similarly, and with 500 and 5, 000 features the difference between them is smaller than 0.01, that would correspond to an improvement smaller than a 1% in both cases. The same occurs in SODP, where EFCC and AFCC get similar results, though FCC performs worse, probably due to its underestimation of frequency (see Section 4.5.2 on page 132). However, when term frequency distribution is more uniform, as in the case of WebKB, adjusting the fuzzy sets has a much bigger effect in clustering results, because there are much more terms to be distributed between medium and high fuzzy sets, and small variations of the membership functions basic parameters will have a much bigger effect. More than that, it is important to adjust this kind of distributions, because they reflect a different use of terms and other resources to highlight them, so the way of capturing this information must be changed too.

162

Fitting Document Representation to Specific Datasets by Adjusting Membership Functions

On the other hand, TF-IDF obtained surprisingly good results in SODP compared to the results of the same function with Banksearch and WebKB datasets. In general, all the representations get bad results in SODP, due to the special difficulties of this collection. The differences among category sizes are a key factor for clustering this dataset (see Figure 3.11 on page 95). As we saw in the previous chapter, there is a clear bias towards Computers category, that contains a 26% of collection documents, while from the other 16 categories, 15 contain less than a 10% of collection documents each, and 11 contain less than a 5% of documents each. We believe that the use of IDF could help improving results of TF-IDF because it would alleviate the effect of the bigger categories, whose terms would be penalized giving more representativeness to those belonging to smaller categories. This fact would slightly reduce the bias introduced by the bigger categories, allowing to cluster the smaller ones slightly better. This improvement in the clustering of smaller categories could cause the consequent improvement in the overall clustering results of TF-IDF. Looking at AFCC and EFCC in Banksearch with 2, 000 features and in WebKB with 5, 000, AFCC improves the results of EFCC. In these cases EFCC shows a fall on its performance. A reason for this behavior could be that EFCC is more sensible to noise when the vocabulary size increases. The different information capture process of AFCC seems to overcome this fall. In general, adjusting the membership functions to the dataset seems to be useful not only to add more automatism to the document representation process, but also because this automation allows to the system a better adaptation to datasets with specific characteristics. The proposed method is able to achieve similar results to EFCC when dealing with exponential distributions. Moreover, when the distribution shape changes, the adjustment helps improve clustering results, as is the case of WebKB.

5.4.1

Statistical significance

To sum up the previous section, AFCC performed well in two collections, Banksearch and WebKB—obtaining particularly good results with WebKB dataset— and in the third one, SODP, it obtained similar results than EFCC. As our proposal is a modification of a previous representation method (EFCC, described in Chapter 4), we decided to perform a robust evaluation of AFCC to be sure about whether or not exists a real improvement over EFCC. In other words, we analyze in depth the difference between using membership function tuning as described in section 5.3.3 and the original representation with fixed fuzzy sets. This analysis aims to check the conclusions stated at the end of previous section about when our tuning method contributes to improve the web page representation.

5.4 Empirical Analysis

163

Besides, we also include FCC in the comparison. It was utilized in the previous chapter (Section 4.7) for a similar analysis together with EFCC. At this point, it is interesting to see the global benefits or drawbacks of the new proposal, AFCC, with respect to the original baseline. To this end, we employ a similar approach than in the previous chapter. We performed 100 experiments per each vector size corresponding to each subdataset, resulting a total of 4,500 different clustering experiments. We calculated the statistical significance between F-measure results of each pair of representations (AFCC-FCC, EFCC-FCC, AFCC-EFCC), utilizing the same procedure described in Section 4.7 on page 141, that is, basically, a paired two-tailed t-test for each concrete vector size. In Table 5.2, for each vector size and representation we show the average Fmeasure values corresponding to the 100 clustering experiments (one per each sub-dataset), the difference between the corresponding averages, and the p-value resulting of applying the statistical t-test between each pair of representations. First, attending to p-values, in all cases except 5 (out of 15), we can say that F-measure results of AFCC and EFCC are from different populations with likelihood greater than 99% (we use bold font to highlight p-values confirming this fact). Besides, looking at the averages of the F-measure, in most of the cases AFCC outperforms EFCC. Regarding differences between representations, just in one case EFCC performs better than AFCC, being the difference lower than 0.01, which would correspond to an improvement smaller than a 1% over AFCC. In particular, AFCC gets an improvement over EFCC in the case of WebKB. We commented the same fact in section 5.4, so detecting the same behavior in this more exhaustive evaluation confirms our previous conclusions. Therefore, the difference between term frequency distributions of the datasets, in combination with all of these results allow us to conclude that membership function tuning helps capture criteria information in a better way, which improves clustering results. About the comparison between EFCC and FCC, it was previously carried out in Section 4.7. With respect to the comparison between AFCC and FCC, the conclusions exposed above are also applicable, because of the improvements of AFCC over EFCC are also translated as improvements over FCC, as it results logical due to the improvement that EFCC obtained over FCC. It is interesting to highlight that this improvement of AFCC over FCC in WebKB means, in 3 out of 5 cases, a difference above 0.07 in terms of F-measure, that leads to improvements higher than a 15% of AFCC over FCC. In fact, the case with the smallest difference between AFCC and FCC in WebKB, AFCC gets an improvement about a 10%. There is a strange case in WebKB for AFCC with 5, 000 features. The result for this case is very bad compared to the rest of AFCC results in WebKB. Nevertheless, the sub-datasets are a 50% smaller than the original one. Then, their

164

Fitting Document Representation to Specific Datasets by Adjusting Membership Functions

Table 5.2: Results for AFCC/EFCC/FCC t-test experiments 100

500

1000

2000

5000

Banksearch F-measure

AFCC MFT EFCC MFT FCC MFT

0.759 0.764 0.718

0.776 0.774 0.760

0.776 0.770 0.765

0.765 0.760 0.768

0.760 0.753 0.768

Difference

AFCC-FCC EFCC-FCC AFCC-EFCC

0.041 0.047 -0.005

0.016 0.014 0.003

0.011 0.006 0.005

-0.003 -0.008 0.005

-0.007 -0.015 0.008

p-value

AFCC-FCC EFCC-FCC AFCC-EFCC

0.000 0.000 0.000

0.000 0.000 0.092

0.000 0.002 0.003

0.039 0.000 0.001

0.000 0.000 0.001

F-measure

AFCC MFT EFCC MFT FCC MFT

0.489 0.487 0.446

0.538 0.514 0.462

0.540 0.528 0.470

0.572 0.534 0.485

0.485 0.483 0.490

Difference

AFCC-FCC EFCC-FCC AFCC-EFCC

0.043 0.041 0.002

0.076 0.051 0.025

0.070 0.059 0.011

0.087 0.049 0.038

-0.004 -0.007 0.002

p-value

AFCC-FCC EFCC-FCC AFCC-EFCC

0.000 0.000 0.512

0.000 0.000 0.000

0.000 0.000 0.001

0.000 0.000 0.000

0.122 0.016 0.002

F-measure

AFCC MFT EFCC MFT FCC MFT

0.235 0.230 0.200

0.274 0.271 0.233

0.280 0.279 0.246

0.284 0.282 0.251

0.289 0.289 0.266

Difference

AFCC-FCC EFCC-FCC AFCC-EFCC

0.035 0.030 0.005

0.040 0.037 0.003

0.033 0.033 0.000

0.033 0.031 0.001

0.023 0.023 0.000

p-value

AFCC-FCC EFCC-FCC AFCC-EFCC

0.000 0.000 0.001

0.000 0.000 0.002

0.000 0.000 0.668

0.000 0.000 0.059

0.000 0.000 0.586

WebKB

SODP

5.4 Empirical Analysis

165

initial vocabularies should be also considerably smaller. As the size of the initial vocabulary in WebKB is 40, 258 terms, the initial vocabulary of its sub-datasets could be about 20, 000 and 30, 000 terms. By reducing from those sizes to 5, 000 is probably not enough to remove noise. For this reason, AFCC (and also EFCC) could suffer a performance drop. So far, we know that AFCC is able to obtain statistically significant better results than EFCC in most cases, concretely in 9 of 15, while in 5 cases both alternatives perform the same, and just in 1 case EFCC outperforms AFCC. However, averages are often confusing, because they do not show the distribution of values that they come from. For this reason, in Table 5.3, we compare all the individual cases (1, 500 per each pair of representations, 4, 500 comparisons in total) we employed to calculate the statistical significance among FCC, EFCC and AFCC. The first column indicates the difference between F-measure results of each pair of representations for the data on a given row. Then we show three blocks, one for each pair of representations in the comparison: EFCC vs FCC, AFCC vs FCC and AFCC vs EFCC. Each block is divided in two different columns. The first one, entitled “# cases” shows the number of cases in which we find the difference corresponding to a given row. The second column, “% cases” shows the percentage that these cases represents over the total number of cases. For example, if we look at the first row, labeled “< −0.09”, the column “# cases” indicates that there were only 3 cases in which the difference between EFCC and FCC was lower than −0.09, that is, FCC outperformed EFCC with a difference greater than 0.09 in 3 cases out of 1, 500, corresponding to a 0.2% of the total number of cases, as shown in the next column of the same row, that is “% cases” within the block EFCC vs FCC. Finally, the table has a horizontal line to separate the negative and the positive cases. When the difference is below 0, it means that the second representations of the corresponding pair outperforms the first. When the difference is greater than 0, then the first one outperforms the second. We assume that results with differences lower than 0.01 in terms of F-measure could be considered as equal results, because they imply a very small difference between the corresponding representations. Therefore, following this assumption and looking at the Table 5.3 from left to right, in most of the cases EFCC works as well as (20.7%) or better (68.9%) than FCC. Between AFCC and FCC, the first one works as well as the second in a 15.5% of cases, and better in a 76.1% of cases. In the last block, AFCC works as well as EFCC in a 58% of cases, and AFCC outperforms EFCC in a 28.5% of cases. Focusing our attention on the comparison of AFCC and EFCC, the difference is favorable to AFCC and greater than 0.02 in a 18.8% of total cases, and greater than 0.03 in a 12.1% of cases. On the other side, EFCC improves AFCC in more than 0.02 only in a 5.5% of the cases and in more than 0.03 in a 2.7%.

166

Fitting Document Representation to Specific Datasets by Adjusting Membership Functions

Table 5.3: Results summary for sub-datasets experiments. Difference

< −0.09 < −0.08 < −0.07 < −0.06 < −0.05 < −0.04 < −0.03 < −0.02 < −0.01 < 0 and > −0.01 0 > 0 and < 0.01 >= 0.01 >= 0.02 >= 0.03 >= 0.04 >= 0.05 >= 0.06 >= 0.07 >= 0.08 >= 0.09 >= 0.10 >= 0.12

EFCC vs FCC # cases % cases

AFCC vs FCC # cases % cases

AFCC vs EFCC # cases % cases

3 6 14 19 25 29 38 67 155 155 310

0.2 0.4 0.9 1.3 1.7 1.9 2.5 4.5 10.3 10.3 20.7

1 2 5 8 10 13 19 38 126 120 246

0.1 0.1 0.3 0.5 0.7 0.9 1.3 2.5 8.4 8.0 16.4

1 1 3 8 17 23 40 83 202 438 640

0.1 0.1 0.2 0.5 1.1 1.5 2.7 5.5 13.5 29.2 42.7

1190 156 1034 882 690 437 266 170 117 92 70 44 11

79.3 10.4 68.9 58.8 46.0 29.1 17.7 11.3 7.8 6.1 4.7 2.9 0.7

1254 112 1142 978 756 517 337 249 206 172 134 88 23

83.6 7.5 76.1 65.2 50.4 34.5 22.5 16.6 13.7 11.5 8.9 5.9 1.5

860 432 428 282 182 126 96 75 54 34 17 6 1

57.3 28.8 28.5 18.8 12.1 8.4 6.4 5.0 3.6 2.3 1.1 0.4 0.1

5.5 Conclusion

167

Summarizing, adjusting the membership functions to a dataset leads to results as good or better than FCC in a 91.6% of cases, and as good or better than EFCC in a 86.5% of the cases. EFCC and AFCC outperform FCC in most of the cases, and between them, AFCC allows to improve the results of EFCC in a 28.5% of cases.

5.5

Conclusion

In this chapter we aimed to study the possibility of automatically fitting a document representation, based on fuzzy logic, to different datasets. Our main concern was not to use any other information than that included in the datasets themselves. Keeping this in mind, in this chapter we have analyzed whether or not clustering results obtained by the fuzzy approaches analyzed in Chapter 4 can be improved using information from the datasets to perform the adjustment. Parts of the research in this chapter have been published in Pérez García-Plaza et al. (2012b). First, our analysis of the datasets showed clear differences among them. We compared them on the basis of their term frequency distributions. We showed the frequency distributions for terms within each criterion used in FCC and EFCC. Banksearch and SODP distributions showed a clear tendency towards exponential distributions. However, in WebKB we found a slightly different distribution for frequency criterion and a clearly different distribution for emphasis criterion. Based on this analysis, we proposed a way of automatically establishing the basic membership function parameters, taking into account term frequency distributions and the original heuristics. This way, our proposal, called AFCC, is able to automatically adjust its membership functions aiming to better capture the information corresponding to each criterion. The proposed method does not adjust position criterion, because word positions on a page do not depend on anything else than the number of words in the document. Then, we compared the resulting web page representation, called AFCC, with the previous ones in which it is based (FCC and EFCC), and with TF-IDF as standard term weighting function. Our evaluation showed that adjusting the representation to concrete collections can help improve clustering results. Besides, we also saw the bad performance of all the representations with SODP collection. This dataset is characterized by a strong imbalance of the number of documents within each category. This could introduce a bias towards the biggest categories, causing that the clustering algorithm favors the division of the documents belonging to those categories. It is also worth noting the good performance of TD-IDF compared to our proposals in the case of SODP. We believe that

168

Fitting Document Representation to Specific Datasets by Adjusting Membership Functions

the use of IDF could be a key factor to explain this good behavior, because it could alleviate the effect of the biggest categories by giving more representativeness to terms coming from smaller categories. For this reason, we believe that TF-IDF can be useful to deal with collections where most of the documents belong to a small number of categories, while most of the categories contain a much smaller number of documents. In these cases, TF-IDF could allow to improve the overall clustering results by improving the clustering related to the small categories. We found that representations not tuned to fit concrete collections could perform reasonably well in most cases, but with our automated proposal it is possible to maintain their results in common cases, i.e., those following Zipf’s law, at the same time that the representation is able to deal with not so common cases. Moreover, identifying those cases is not a computationally expensive task. Thus, we conclude that tuning the system to fit the dataset is a feasible way of improving web page representation for clustering tasks.

6 Test Scenario: Hierarchical Clustering Applied to Learn a Taxonomy from a Set of Web Pages in Two Languages

After analyzing different ways of combining criteria we presented two proposals to improve web page representation for clustering tasks (EFCC and AFCC). Nevertheless, these representations have been only tested in a plain clustering environment, with documents written in English. To validate our previous conclusions in a different environment, in this chapter we propose to apply that representations in a totally different framework. We introduce here the problem of taxonomy learning and a possible solution by means of web page hierarchical clustering. We also perform the experiments for two different languages: English and Spanish. The chapter is organized as follows. First, in Section 6.1 on the following page we present the motivation of our research in this chapter. Next, in Section 6.2 on the next page we review previous work on the field of taxonomy learning to situate the scenario of our research. Then, in Section 6.3 on page 174 we describe the clustering method utilized in the experiments of this chapter. The evaluation methodology for taxonomy learning that we apply in this thesis is detailed in Section 6.4 on page 175, followed by a description of the experimental settings in Section 6.5 on page 178. Then, we show our experimental results in Section 6.6 on page 180 and finally we present our findings to conclude the chapter in Section 6.7 on page 184:

Test Scenario: Hierarchical Clustering Applied to Learn a Taxonomy 170 from a Set of Web Pages in Two Languages

6.1

Introduction

In previous chapters we have proposed different ways of representing web pages for clustering tasks by means of fuzzy combinations of criteria. However, the whole set of experiments was performed in an environment of plain clustering. Besides, these experiments were performed for web pages written in English. In this chapter we extend this experimental framework to test the appropriateness of the proposed representations for hierarchical clustering tasks. We use a clustering process based on the SOM algorithm, with the aim of validating our results in a totally different environment. Finally, this experimentation is performed with a comparable corpus (WAD dataset, see Section 3.5), composed of web pages written in English and Spanish. This corpus contains Wikipedia documents that describe concepts. Each concept is described in a separate web page, and then the hierarchical clustering over that documents aims to create a taxonomy. In this work we consider a taxonomy as a simplification of an ontology. Ontologies are a useful tool to represent relations between concepts. In fact, they are usually defined as formal representations of knowledge by means of a set of concepts and their relations. Of course, they are focused on concrete domains that are, by extension, described by the ontologies. We refer to a taxonomy as a simplified ontology, because of the relations between concepts are limited to parent-child relations only. In this way, we can see a taxonomy as a set of concepts hierarchically arranged. Taxonomies and ontologies have a broad field of application that includes several natural language processing tasks like recommender systems (Yang and Hsu, 2010), IR (Pruski et al., 2011), text clustering and classification (Bloehdorn et al., 2006; Sridevi and Nagaveni, 2011), among others. Due to the cost of the manual creation of ontologies, more automated methods based on machine learning have emerged. Along these lines, we presented an unsupervised method for creating a taxonomy of concepts from a set of web pages (Paukkeri et al., 2010, 2012) that is used in this thesis to complete the validation and evaluation of our representation proposals. The experimental framework of this method will be explained later in this chapter, but the main idea is organizing a set of web pages in a hierarchical structure like shown in the Figure 6.1.

6.2

Works on Taxonomy Learning

This chapter presents a different scenario to test our web page representation proposals. For this reason, we briefly describe here some previous works related with taxonomy learning, the field of application of our proposals in this chapter. First of all, as this chapter deals with taxonomy learning from a set of web

6.2 Works on Taxonomy Learning

171

Figure 6.1: Web pages are hierarchically clustered to build a taxonomy; from Paukkeri et al. (2012).

pages, and a taxonomy can be considered a simplified ontology, this application of our representation proposals could be seen as a first step towards a most general problem of ontology learning. On the one hand, inducing a taxonomy of concepts, that is, a concept hierarchy, implies that each deeper hierarchy level has associated an increase in the specificity of the concepts it contains. Then, there is only one type of relation between concepts, that can be seen as hypernyms and hyponyms through the hierarchy levels. On the other hand, when dealing with non-taxonomic structures, the possible types of relations increase, appearing others like meronymy, synonymy or antonymy. Without the intention of writing an exhaustive review on the state-of-theart about taxonomy learning, we present here a brief review on the four main paradigms for inducing taxonomies: • The first paradigm comprises methods that apply lexico-syntactic patterns to detect hyponymy relations. It is based on the possibility of finding explicit knowledge in some kind of texts. For instance, handbooks, textbooks, dictionaries, or even other popular resources as Wikipedia or social tagging systems can contain definitions such as “a red-tailed hawk is a bird” or “mammals such as moose, bat or mouse”. One of the most relevant approaches was introduced by Hearst (1992). Her method identify the patterns that indicate hyponymy relations. For example, the following lexicosyntactic pattern corresponds to the hyponymy relation: If (NP0 such as NP1 , NP2 . . . , (and | or) NPn ) For all NPi , 1 ≤ i ≤ n, hyponym(NPi , NP0 ) where NP means Noun Phrase. Nevertheless, this approach suffers some drawbacks. First, it is not easy to find sufficient lexico-syntactic patterns. Second, it is also difficult to find examples enough including the terms of interest (Brewster and Wilks, 2004). Thus, this kind of approaches achieve reasonable precision values, but in exchange, their recall is very low. Brewster showed the extremely low probability of finding sufficient examples

Test Scenario: Hierarchical Clustering Applied to Learn a Taxonomy 172 from a Set of Web Pages in Two Languages

of any given lexico-syntactic pattern to be able to get reliable results. Another approach exploits the internal structure of noun phrases in order to derive taxonomic relations between classes. These classes are represented by the head of the noun phrase and its subclasses, that can be derived from a combination of the head and its modifiers (Buitelaar, 2000). A similar approach was presented in Pennacchiotti and Pantel (2006), where a bootstrapping method was applied to learn lexical patterns from raw text, given a small set of seed instances for a particular relation. Again, their results showed good precision values, but low recall. Natural language tools have also been used to extract semantic relations from text. In particular, Specia and Motta (2006) uses lemmatizer, syntactic parser, part-of-speech tagger, pattern-based classification and word sense disambiguation models together with resources such as domain ontology and lexical databases. The work of Ciaramita et al. (2005) presents an unsupervised model for learning arbitrary relations between concepts. This approach utilizes, in addition to syntactic patterns, a corpus of manually tagged named entities that correspond to ontology concepts. • The second paradigm is based on the distributional hypothesis stated by Harris (Harris, 1954): words that occur in the same contexts tend to have similar meanings. Most of the approaches following this paradigm exploit hierarchical clustering algorithms to automatically derive term hierarchies from text, e.g., Grefenstette (1994), Faure and Nédellec (1998) and Cimiano et al. (2005). Hierarchical and non-hierarchical clustering, similarity measures and different linking schemes, among other statistical methods for extraction of taxonomic relations, were explored in Maedche et al. (2002). Besides, Staab (2005) introduced a guided agglomerative hierarchical clustering algorithm to create concept hierarchies from text collections by employing a hypernym oracle. This oracle uses hypernyms from WordNet and the above mentioned lexico-syntactic patterns proposed by Hearst, matched in a corpus or the Internet. Another approach based on clustering was proposed by Snow et al. (2006). The authors incorporates evidence from multiple classifiers to optimize the entire structure of the taxonomy. Our work has to do with this paradigm, as it is common to find hierarchical clustering approaches relying on the VSM. Our proposals are based on the VSM, so they can be directly applied. Besides, our work presented in Paukkeri et al. (2012) proposes an unsupervised method to hierarchically cluster a set of web pages to build a taxonomy. We rely on this method to perform our experiments in this chapter, since this method only needs a set of web pages represented in the VSM as input. • The third paradigm comes from the IR field. It basically relies on the use of documents retrieved in response to queries. With this approach, Sander-

6.2 Works on Taxonomy Learning

173

son and Croft (1999) derived concept hierarchies from text documents by using a type of co-occurrence known as subsumption. Given a query and a document collection, they use Local Context Analysis (Xu and Croft, 1996) to expand the query with additional terms and retrieve the top 500 ranked documents to apply two means of term selection over them. One of them is based on the co-occurrences of words and phrases in the best matching passages of the top ranked documents and the other on a comparison of a term’s frequency of occurrence in the retrieved documents with its occurrence in the collection. Finally, every selected term was compared to every other term to test for subsumption relationships. In Sanchez and Moreno (2005), another automatic and unsupervised methodology to obtain taxonomies of terms from the Web was presented. It is based on an intensive use of web search engines to retrieve domain suitable resources to extract knowledge and to obtain web scale statistics from which infer knowledge relevancy. They employ a domain defined by an initial keyword and linguistic patterns involving that keyword to find candidate concepts for the domain. The result is a hierarchical organization of the available knowledge and web resources for the given domain. The main drawback comes from the use of a keyword, because as the authors stated: “if several ways of expressing the same concept exist (e.g. synonyms, lexicalizations or even different morphological forms), a considerable amount of potentially interesting web resources will be omitted”. In Riloff and Kozareva (2009) a supervised bootstrapping algorithm is presented for reading web texts and learning taxonomy terms. The algorithms starts with two seed words and progressively learns hyponym and hypernym terms using Google search engine to obtain documents to apply the patterns. • The fourth paradigm is based on social media. Concretely, social tagging systems have a flat and uncontrolled underlying resource organization paradigm. This fact has encouraged the development of new approaches for taxonomy learning based on folksonomies, like the work of Benz and Hotho (2007). Moreover, taxonomies have been learned also from Wikipedia by using its categories as concepts in a semantic network, and finding relations between concepts on the basis of the connectivity of the network. In a recent work, Wong (2009) introduces the Tree-Traversing Ants (TTA) clustering technique for learning taxonomic relations. TTA is based on dynamic tree structures and it adopts a two-pass approach for term clustering. During the first pass, nodes are recursively broken into sub-nodes using Normalized Google Distance. The second pass is a refinement phase where terms are relocated according to the n-degree of Wikipedia measure, that uses information from Wikipedia categories.

Test Scenario: Hierarchical Clustering Applied to Learn a Taxonomy 174 from a Set of Web Pages in Two Languages

Our decision on using the method presented in Paukkeri et al. (2012) was made because of our interest in a method that relies only in the dataset documents themselves, as the web page representations proposed in this thesis are also oriented to this kind of unsupervised environment. We chose this test scenario as an interesting application of hierarchical clustering to test our representation proposals. We do not aim to improve taxonomy learning techniques, but check whether or not our representation can be successfully applied to that problem.

6.3

Clustering Method

In this case, we want to cluster concepts. These concepts correspond to the documents in the dataset, that is, each document is considered a concept definition. In this way, it is assumed that each concept has a hypernym, that corresponds with a parent node in the hierarchical clustering tree. Thus, the algorithm is feed by a set of vectors that represents the documents in the VSM. In this chapter, as each document corresponds to a concept definition, by extension we will refer to document vectors also as concept vectors. Intuitively, the approach used to create the taxonomy is top-down, starting from the zero-level, that corresponds with the root node. This node will contain the whole collection. In the next step, the collection will be divided in separated clusters. Thus, the first level will consist of different clusters, each one containing a different part of the collection. Every cluster on the first level will be split again, resulting in a new level of the hierarchy. The process will continue until finding the desired hierarchy. In particular, the clustering process applied on each level is based on the SOM algorithm. A more detailed description of this algorithm can be found in Section 2.3 on page 71. The SOM algorithm does not produce proper clusters, but a set of neurons arranged in such a way that they represent a topological ordering of the input data. So, proximity in the map will imply similarity between the input vectors. To explain the process, Figure 6.2 show an example of how the hierarchy is generated by splitting the SOMs in clusters. We start the process in the zero level, with the whole set of input vectors, that corresponds to collection documents. The SOM is trained with these vectors and the vectors are then mapped onto the SOM. The next step is splitting the map in proper clusters, because as we said above, the SOM does not generate proper clusters. However, several adjacent neurons— and the document vectors mapped onto them—could be considered a cluster, as proximity in the map implies similarity. To define the clusters and their limits on the map, i.e., to split the SOM in separate clusters, an agglomerative hierarchical binary algorithm is applied. These clusters will constitute the first level of the taxonomy, that is, the level just below the root. It is worth noting that each first

6.4 Evaluation Methodology

175

level cluster will contain part of the initial set of document vectors. Then, on the first level we have a set of clusters coming from the zero level (resulting of splitting the SOM that contained the whole set of document vectors). For each cluster on the first level, a new SOM is trained with the vectors belonging to that concrete cluster, that are mapped onto the SOM after the training stage. Then the agglomerative algorithm mentioned above is applied over each SOM to split them into clusters again. The same clustering procedure is repeated for each level in the desired taxonomy, or until finding a cluster containing only one concept.

Zero Level – a SOM with all the document vectors

…

…

…

First Level – a SOM for each zero-level cluster

Second Level – a SOM for each first-level cluster

…

…

…

…

…

Figure 6.2: Example of the SOM based hierarchy.

6.4

Evaluation Methodology

Ontologies have been evaluated in previous works using four different approaches (Brank et al., 2005): • By comparison to a gold standard, that could be directly a reference ontology. • Evaluating the ontology as a part of a concrete application. • By comparison to source data about the domain. • Evaluation made by humans.

Test Scenario: Hierarchical Clustering Applied to Learn a Taxonomy 176 from a Set of Web Pages in Two Languages

In this thesis we employ the first method, that is, we compare the taxonomies created by means of the hierarchical clustering method against a manually constructed reference ontology. To do the comparison, we rely in the method proposed by Dellschaft and Staab (2006). The authors propose a set of taxonomic measures oriented to perform a gold standard based evaluation of ontology learning. Concretely, three measures were introduced: taxonomic precision, taxonomic recall and taxonomic F-measure. Other works like Brewster et al. (2003) and Brewster and Wilks (2004) applied these metrics to evaluate their results by using a gold standard. However, the experimentation of those works was oriented to find single concepts and their semantic relations, that is a different approach than ours, where the goal is to create a taxonomy. For calculating the global taxonomic precision (TPcsc ) of two ontologies, they rely on the common semantic cotopy (csc). The semantic cotopy of a concept is defined as the set of all its super- and subconcepts. The common semantic cotopy excludes all concepts which are not also available in the set of concepts of the other ontology: csc(c, O1 , O2 ) = {ci |ci ∈ C1 ∩ C2 ∧ (ci