Evaluating Answer Quality across Knowledge Domains: Using Textual ...

Evaluating Answer Quality across Knowledge Domains: Using Textual and Non-textual Features in Social Q&A Hengyi Fu School of Information Florida State University [email protected]

Shuheng Wu Graduate School of Library and Information Studies Queens College, City University of New York [email protected]

ABSTRACT

As an increasing important source of information and knowledge, social questioning and answering sites (social Q&A) have attracted significant attention from industry and academia, as they address the challenge of evaluating and predicting the quality of answers on such sites. However, few previous studies examined the answer quality by considering knowledge domains or topics as a potential factor. To fill this gap, a model consisting of 24 textual and non-textual features of answers was developed in this study to evaluate and predict answer quality for social Q&A, and the model was applied to identify and compare useful features for predicting high-quality answers across four knowledge domains, including science, technology, art, and recreation. The findings indicate that review and user features are the most powerful indicators of high-quality answers regardless of knowledge domains, while the usefulness of textual features (length, structure, and writing style) varies across different knowledge domains. In the future, the findings could be applied to automatically assessing answer quality and quality control in social Q&A. Keywords

Social question and answer sites, answer quality, quality assessment. INTRODUCTION

Social questioning and answering sites (social Q&A) have become an increasingly important online platform for knowledge building and exchange. Social Q&A allows individuals to post questions, provide answers and comments, and evaluate questions and answers among the peers. The popularity of social Q&A has led to a growing body of literature regarding assessing and predicting answer quality. Most of the prior work has focused on this issue from two perspectives – users and data. The first group investigated what criteria users applied to evaluate answer {This is the space reserved for copyright notices.] ASIST 2015,November 6-10, 2015, St. Louis, MO, USA. ©2015 Hengyi Fu, Shuheng Wu, and Sanghee Oh.

Sanghee Oh School of Information Florida State University [email protected]

quality (Kim & Oh, 2009; Shah & Pomerantz, 2010; Zhu, Bernhard, & Gurevych, 2009). Zhu et al. (2009) developed a multi-dimensional model for assessing answer quality from user perspectives, including informativeness, completeness, readability, conciseness, truthfulness, level of detail, originality, objectivity, novelty, usefulness, and expertise. This model was later used to evaluate answer qualities of Answerbag and Yahoo! Answers based on human judgement (Shah & Pomerantz, 2010; Zhu et al., 2009). The second group focused on extracting data features from questions and/or answers to automatically predict answer quality (Dalip, Gonçalves, Cristo, & Calado, 2013; Shah & Pomerantz, 2010), but the selection of features varies across studies and depends on the researchers’ perceptions of quality and the availability of different features. Despite an ongoing concern pertaining to the answer quality in social Q&A, there is a lack of research and understanding of knowledge domain (topic) as a potential factor in assessing and predicting answer quality from both user and data perspectives. The criteria that a user applies to evaluate answers to science-related questions may be different from those of art-related questions. The usefulness of a particular quality criterion in predicting answer quality may also vary across knowledge domains. For example, the number of reference resources an answer includes may be highly correlated to the answer quality of science-related questions, but may not be very useful in predicting the answer quality of art-related questions. Harper, Raban, Rafaeli, and Konstan (2008) concluded that topics had a small and marginally significant effect on answer quality, but did not go further to study how it could influence quality assessment. Therefore, the purpose of this study is to fill this gap by (a) developing a model to evaluate and predict answer quality for social Q&A and (b) applying this model to identify and compare useful features of highquality answers across different knowledge domains. RELATED WORK

Answer quality in social Q&A can be captured either directly through textual features, such as content length or structure, or indirectly through non-textual features, such as review histories or author reputation. Textual features can be further divided into four groups: length, structure, style, and readability of texts. Non-textual features can be data

related to users, reviews, and networks in social Q&A (Dalip et al., 2013).

Overflow, and concluded that these features were not very indicative of answer quality.

Textual features

Non-textual features

Length features include the count of characters, words, or sentences, which give hints about whether the text is comprehensive. The intuition behind them is that a mature text of good quality should be neither too short, which may indicate an incomplete coverage, nor excessively long, which may indicate verbose contents. Length features have been proven to be one of the most powerful indicators of quality in both Wikipedia (Blumenstock, 2008; Hu et al., 2007) and social Q&A (Agichtein et al., 2008; Harper et al., 2008).

User features reflect users’ activities and expertise levels in a social Q&A site. The intuition behind user features is that content quality can be inferred by examining the author who provides it. User features, such as account history (e.g., member since), community role, achievement, and reputation, have been used and proven to be significant quality predictors in different knowledge creation communities (Shah & Pomerantz, 2010; Stvilia, Twidale, Smith, & Gasser, 2005; Tausczik & Pennebaker, 2011). In social Q&A, user features extracted from answerers’ profiles and the points that they earned were the most significant features for predicting the best quality answers (Shah & Pomerantz, 2010).

Structure features are indicators of how well the content of texts is organized. Previous studies on the quality of Wikipedia articles highlighted the significance of features that reflect organizational aspects of the article, such as the number of sections, images, external links, and citations (Hasan, André Gonçalves, Cristo, & Calado, 2009). Although the content structure of answers in social Q&A is not exactly the same as that of Wikipedia articles, some structure features can still be used to predict answer quality. For example, the presence of references and external resources are known as influential features of high-quality answers in social Q&A (Gazan, 2006). Style features are intended to capture the author’s writing style. The intuition behind them is that high-quality content should present some distinguishable characteristics related to word usage, such as the use of auxiliary verbs, pronouns, conjunctions, prepositions, and short sentences. Style features have been proven as effective in predicting the quality of user-generated content in Wikipedia articles (Lipka & Stein, 2010) and answers in social Q&A (Hoang, Lee, Song, & Rim, 2008; Dalip et al., 2013), but the selection of style features varies among studies. Readability features are intended to estimate the minimal age group necessary to comprehend a text. The intuition behind them is that high-quality content should be well written, understandable, and free from unnecessary complexity. Several measures of readability have been proposed, including the Gunning-Fog Index, the FleschKincaid Formula, and SMOG Grading. These measures combine the number of syllables, words and sentences in the text, and were used to predict content quality in Wikipedia (Rassbach, Pincock, & Mingus, 2007) and social Q&A (Dalip et al., 2013), but resulted in low effectiveness in both settings. Another set of textual features proposed specifically for online question answering is relevance features, identifying the similarity between the answer and its associated question, such as the number of words or sentences shared by the questions and answers. Dalip et al. (2013) tested relevance features using question-answer pairs from Stack

Review features are those extracted from the review history or comments on certain content. These features are useful for estimating the maturity and stability of content. It can be expected that a content receiving many edits or comments has likely been improved over time. In general, the more the content was reviewed, the better its quality. Positive correlation exists between content quality and review features, such as the number of edits/revisions, discussions, and comments (Lih, 2004; Stvilia et al., 2005; Wilkinson & Huberman, 2007). Network features are those extracted from the connectivity network inherent within a community. The motivation for using these features to evaluate content quality is that citations between contents can provide evidence about their importance. For example, although a high-quality Wikipedia article should be stable, such stability may result from poor quality, and thus no one is interested in it. It is therefore important to take into account the popularity of content during quality evaluation, which can be estimated by measures taken from its connectivity network. Network features such as PageRank, link count, and translation count, have been tested in Dalip et al.’s (2011) study on Wikipedia article quality, but did not show high effectiveness. RRESEARCH QUESTIONS

This poster addresses the following research questions: RQ1: What are the textual or non-textual features of social Q&A that can be useful to assess quality of answers? RQ2: How do the features of high quality answers in social Q&A vary across different knowledge domains, especially in science, technology, art, and recreation? DATA COLLECTION AND RESEARCH METHOD

To answer the first research question, this study selected a set of 24 quality features, which have been reported in literature review as useful for predicting the quality of contents in Wikipedia and social Q&A. Table 1 shows the

five feature groups of the 24 features. The usefulness of these quality features was tested and reported in this poster. Feature group

Features

Length

Counts of words (l-wc) and sentences (l-sc)

Structure

Counts of paragraphs (s_pc), images (s_ic), external links (s_elc), internal links (to other question/answers in the site) (s_ilc)

Style

Counts of auxiliary verbs (t_avc), “to-be” verb (t_bvc), pronoun (t_pc), conjunction (t_cc), interrogative pronoun(t_ipc), subordinating conjunction(t_scc), preposition(t_prc), nominalization(t_nc), and passive voice sentence(t_pvc)

User

Counts of user points (u_point), merit badges (u_mbc), questions posted by a user (u_qc), answers posted by a user (u_ac), comments posted by a user (u_cc), suggested edits (u_ec) made by a user

Review

Counts of revision (r_rc), and comments an answer received (r_cc), answer age/day (r_aage)

Table 1. Quality features of five different groups

To answer the second research question, this study collected data from StackExchange1, a group of popular social Q&A sites, currently with 133 different topical Stack Exchange websites hosting almost 3.8 million users. As of May 2015, 3.1 million questions have been posted, eliciting 4.5 million answers. The 133 social Q&A sites are grouped into six topic areas, including science, technology, business, professional, culture/recreation, and life/arts. This study selected four to represent different knowledge domains: science, technology, art, and recreation; and uses one site from each area based on their similarities in site age, membership, and active levels, namely Mathematics, Ask Ubuntu (Q&A for Ubuntu developers), Photography, and Arqade (Q&A for passionate video gamers), to compare the features associated with high quality answers in these sites. High quality answers were identified using StackExchange’s Answer Score, which is calculated by the number of upvotes minus the number of downvotes that an answer received. With the help of StackExchange API2, the top 1,000 answers with the highest scores were extracted to represent high-quality answers from each of the four sites. One of the researchers developed C++ code to collect data related to structure features of answer contents. Length and style features of answers were calculated by the GNU Style 1

http://stackexchange.com/sites?view=list#traffic

2

https://api.stackexchange.com/docs/

and Diction software program3. Data related to user and review features associated with the 1,000 answers were extracted from the four sites directly using StackExchange API. Exploratory factor analysis (EFA) was conducted to identify and compare the features in the five groups (factors) among the four sites using SPSS. EFA is a method aimed at extracting maximum variance from the dataset within each factor. For each dataset, all 24 features were tested using Maximum Likelihood as the extraction method and varimax rotation; factors having eigenvalues above 1 were retained. A cut-off of 0.6 was selected for analyzing factor loading since factor loading above this value is generally considered high (Field, 2005). FINDINGS AND DISCUSSION

The results of EFA for each site are given in Table 2. Factor 1 for Mathematics contains two structure features: image count and external link count. Factor 2 is dominated by review features, including revision count and comment count. The count of comments that the author made is significantly related to this factor as well. All features in Factor 3 are related to user/author, including the count of merit badges, answers, comments, and suggested edits. Factor 4 is a combination of length (sentence count), structure (internal link count), and style (passive voice sentences count). This factor can be interpreted as content or text related. For Ask Ubuntu, factor 1 is mainly about the user/author, including points, numbers of merit badges, comments, and suggested edits. Review features play an important role in factor 2. Similar to Mathematics, factor 3 has a good mix of length (sentence count), structure (external link count and internal link count), and style (preposition count). Factor 1 for Photography contains three features: image count, revision count, and comment count. Factor 2 is mainly related to user features (merit badge count, answer count, and comment count a user made, and the answer age). Factor 3 includes two length features (counts of words and sentences) and one structure feature (external link count). Factor 4 is dominated by style features. For Arqade, Factor 1 has three features related to the review (revision count and comment count) and user (user points) groups. Factor 2 contains only one length feature (word count). Factor 3 contains three features corresponding to features of the user group (merit badge count, answer count, and comment count a user made). Factor 4 is dominated by structure features (paragraph count, image count, and external link count). Factor 5 is mainly related to style features (pronoun count and subordinating conjunction count).

3

http://www.gnu.org/software/diction/

Sites

Factor

Features

Mathematics (Science)

1

s_ic(.816),s_elc(.960)

2

u_cc(.627),r_rc(.904),r_cc(.967)

3

u_mbc(.997),u_ac(.747), u_ec(.926)

4

l_sc(.815),s_ilc(.741),t_pvc(.604)

1

u_point(.873),u_mbc(.868),u_cc(.652),u _ec(.742), r_aage(.606)

2

l_wc(.634),r_rc(.858),r_cc(.914)

3

l_sc(.763),s_ilc(.714),s_elc(.623),t_prc(. 667)

1

s_ic(.704),r_rc(.823),r_cc(.876)

2

u_mbc(.872),u_ac(.695),u_cc(.740),r_a age(.800)

Ask Ubuntu (Technology)

Photography (Art)

Arqade (Recreation)

3

l_wc(.918),l_wc(.755),s_elc(.793)

4

t_scc(.721),t_pc(.738),t_prc(.817)

1

u_point(.810),r_rc(.726),r_cc(.872)

2

l_wc(.804)

3

u_mbc(.903),u_ac(.765),u_cc(.737)

4

s_pc(.689),s_ic(.842),s_elc(.709),t_prc(. 641)

5

t_pc(.870),t_scc(.616),t_nc(.722)

Table 2. A summary of quality groupings (factors) in social Q&A sites

The results show that the review and user features are the most useful groups, which are both non-textual features. Review features, especially revision count and comment count, are indicators of high-quality answers across the four sites. These features, measuring user engagement in an answer, could be the most effective indicators of answer quality. User features, such as the count of merit badges, answers, and comments a user made also present usefulness in identifying high-quality answers, regardless of the knowledge domains in the sites. This highlights the importance of user profiles and the history of a user in assessing answer quality. The findings also indicate that the importance of textual features (length, structure, and writing style) vary across different knowledge domains. For example, structure features, such as image count and external link count, are the most essential features of high-quality answers in

Mathematics Q&A, indicating user’s recognition of using equations (usually presented in image format) and references in science related answers. Similarly, image count is a powerful indicator of answer quality in Photography Q&A, suggesting that attaching real photos in answering these kinds of questions is essential. The length feature plays an important role in assessing answer quality in Photography and Arqade. This implies that for social Q&A addressing more “subjective” topics (e.g., art, recreation), people tend to consider that the answer length has a positive relation to answer quality. Style features, such as pronoun count, subordinating conjunction count, preposition count, nominalization count, and passive voice sentence count, are useful for identifying high-quality answers in art and recreation social Q&A sites, but not that important for those addressing science and technology questions. While review and user features are the most important feature groups across different knowledge domains, text features are also useful, and more importantly, less difficult to generate in terms of the preprocessing required. IMPLICATIONS AND CONCLUSION

This study develops a model consisting of 24 quality features for assessing answer quality in social Q&A, and then uses the model to identify and compare useful features of high-quality answers in four different knowledge domains. This study has a few limitations in that StackExchange is one of the social Q&A sites; it may not represent all other social Q&A sites. Also, the four sites that have been randomly selected may not represent all other sites of each knowledge domain. The proposed model of predicting answers with the textual and non-textual features from social Q&A can be applied for automatic assessment of answer quality and quality control in social Q&A. The model provides a comprehensive but critical set of quality features, and it can be used to build classifiers to select high-quality answers across knowledge domains. Future research can validate or expand this model by testing it in social Q&A sites from other knowledge domains. This study also reveals that highquality answers in different knowledge domains are associated with a variety of quality features. Social Q&A designers can customize the reward and promotion system of different knowledge domains to encourage contributions that contain the most useful quality features for a particular domain. For example, since review features are strongly associated with quality across knowledge domains, more points could be awarded to those who submit comments or edit suggestions to answers provided by others. For science or technology, more points and merit badges can be granted to those who provide equations, source code, and references in their answers in social Q&A. REFERENCES

Agichtein, E., Castillo, C., Donato, D., Gionis, A., & Mishne, G. (2008). Finding high-quality content in social

media. In Proceedings of the 2008 International Conference on Web Search and Data Mining (pp. 183194). ACM. Blumenstock, J. E. (2008). Size matters: Word count as a measure of quality on Wikipedia. In Proceedings of the 17th International Conference on World Wide Web (pp. 1095-1096). ACM. Dalip, D. H., Gonçalves, M. A., Cristo, M., & Calado, P. (2011). Automatic assessment of document quality in web collaborative digital libraries. Journal of Data and Information Quality, 2(3), 14. Dalip, D. H., Gonçalves, M. A., Cristo, M., & Calado, P. (2013). Exploiting user feedback to learn to rank answers in Q&A forums: A case study with Stack Overflow. In Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 543-552). ACM. Field, A. (2009). Discovering statistics using SPSS. Sage. Gazan, R. (2006). Specialists and synthesists in a question answering community. In Proceedings of the American Society for Information Science and Technology, 43(1), 1-10. Hasan Dalip, D., André Gonçalves, M., Cristo, M., & Calado, P. (2009). Automatic quality assessment of content created collaboratively by web communities: A case study of Wikipedia. In Proceedings of the 9th ACM/IEEE-CS Joint Conference on Digital libraries (pp. 295-304). ACM. Harper, F. M., Raban, D., Rafaeli, S., & Konstan, J. A. (2008). Predictors of answer quality in online Q&A sites. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 865-874). ACM. Hoang, L., Lee, J. T., Song, Y. I., & Rim, H. C. (2008). A model for evaluating the quality of user-created documents. In Information Retrieval Technology (pp. 496-501). Springer Berlin Heidelberg. Hu, M., Lim, E. P., Sun, A., Lauw, H. W., & Vuong, B. Q. (2007). Measuring article quality in Wikipedia: Models and evaluation. In Proceedings of the sixteenth ACM Conference on Information and Knowledge Management (pp. 243-252). ACM.

Kim, S., & Oh, S. (2009). Users' relevance criteria for evaluating answers in a social Q&A site. Journal of the American Society for Information Science and Technology, 60(4), 716-727. Lih, A. (2004). Wikipedia as participatory journalism: Reliable sources? Metrics for evaluating collaborative media as a news resource. In Proceedings of the 5th International Symposium on Online Journalism, Austin, USA. Lipka, N., & Stein, B. (2010). Identifying featured articles in Wikipedia: Writing style matters. In Proceedings of the 19th International Conference on World Wide Web (pp. 1147-1148). ACM. Rassbach, L., Pincock, T., & Mingus, B. (2007). Exploring the feasibility of automatically rating online article quality. In Proceedings of the 2007 International Wikimedia Conference, Taipei, Taiwan (p. 66). Shah, C., & Pomerantz, J. (2010). Evaluating and predicting answer quality in community Q&A. In Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 411-418). ACM. Stvilia, B., Twidale, M. B., Smith, L. C., & Gasser, L. (2005). Assessing information quality of a communitybased encyclopedia. In Proceedings of the International Conference on Information Quality (pp.442-454). Cambridge, MA: MITIQ. Tausczik, Y. R., & Pennebaker, J. W. (2011). Predicting the perceived quality of online mathematics contributions from users' reputations. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 1885-1888). ACM. Wilkinson, D. M., & Huberman, B. A. (2007). Cooperation and quality in Wikipedia. In Proceedings of the 2007 International Symposium on Wikis (pp. 157-164). ACM. Zhu, Z., Bernhard, D., & Gurevych, I. (2009). A multidimensional model for assessing the quality of answers in social Q&A sites. Technical Report TUD-CS2009-0158. Technische Universität Darmstad.