H6 - 1
2011 International Conference on Electrical Engineering and Informatics 17-19 July 2011, Bandung, Indonesia
Measuring User Engagement Levels in Social Networking Application Firdaus Banhawi, Nazlena Mohamad Ali, Hairuliza Mohd Judi School of Information Science, Faculty of Information Science and Technology, Universiti Kebangsaan Malaysia 43650 Bangi, Selangor, Malaysia
[email protected] Abstract— This paper describes our work in performing an disriminant analysis (DA) to determine the relationship between engagement levels and Facebook activities. From our previous work, we found that there are four attributes of engagement while interacting with social network application namely; Focus Attention, Novelty Endurabilty, Perceived Usability and Aesthetics. In this paper we adapted the scale from other previous work to link up with engagement levels by using discriminant analysis. A number of 103 Facebook users responded to the administered questionnaires in two weeks duration. From the experiment, we found that Social Connection is the most engaging activity, followed by Photographs, Status Updates, Social Investigation, Social Network Surfing, and Contents. We also provided discriminant function to test the engagement level of Facebook activities for future work. Keywords— user engagement, social networking application, discriminant analysis.
I. INTRODUCTION Social networking application serves a number of functions in our life that provides social and emotional support information resources [23]. It is an approach for representing relation between individuals, groups or organizations [14]. This study has an aim to define engagement attributes in the perspective of social networking application (i.e. Facebook). Facebook has been categorized as one of an engaging [12]; fascinating, an interactive application, and image laden directory featuring groups that share lifestyles or attitudes [2]. Social networking application like Facebook is expected to have its own engagement attributes that might be different from other application domains. Previous work has indicated that a multidimensional scale may be used to test the engagement in other software applications [17]. It also identified six attributes of engagement namely; Perceived Usability, Aesthetics, Focused Attention, Felt Involvement, Novelty and Endurability by using the Reliability Analysis and Exploratory Factor Analysis in an online shopping environment. The user engagement scale is conceivable that the instrument could also be generalized to other environment or application such as the web digital libraries or task-specific applications [17].
978-1-4577-0751-3/11/$26.00 ©2011 IEEE
In this work we chose social networking application as our domain in order to find out the user engagement attributes while interacting with the application. There are subtle differences among shoppers, gamers, learners and searchers regarding the manifestation of some of the engagement attributes, particularly among those designed for individual versus collaborative use [16]. It might be possible to have different sets of engagement attributes because format influences the engagement factor [10]. It is believed that survey instrument is the most appropriate method for collecting user’s perception of their level of engagement [17] as a variety of researchers have developed survey instruments to evaluate engagement (e.g. [22], [9], [15]). The user engagement scale that has been used in an online shopping environment is being adapted into Facebook social network application environments without changing the meaning of each item. The objective of this study is to determine the relationship between engagement levels and Facebook activities. Joinson [11] has suggested further research on a wider group of participants, or attempt to identify pattern of usage. This paper was organized in the following sections. Section II elaborates briefly on the methodology and followed by the results of the evaluation in Section III. Section IV discussed the findings and finally the conclusions are reported in Section V. II. METHODOLOGY Target participants of this survey are registered Facebook users of any ages, gender and occupation. A total of 103 Facebook users responded to the administered questionnaires. Participant’s data are gathered using two methods which are online survey and direct approached survey. The survey form is created by using Joomla Survey. Direct approached survey was conducted by finding Facebook users randomly from university colleges and offices. An online survey was posted for two weeks (March 24 – April 7, 2011) and direct approach survey was carried out as ongoing during this period. Participants were recruited through different methods; posting to the ‘wall’ of two accounts on the Facebook; and by spreading at the Facebook forum. Direct approach recruitment involved visiting
undergraduate classes, libraries and offices. The survey comprised a series of basic demographics questions, with some measures of Facebook usage, the five-point Likert scale for engagement attributes while using Facebook in Study I and the five-point Likert scale for Facebook activities in Study II. One of the common ways to measure attitude is using Likert scale [18]. It was chosen because it is fit with the data and supplied the ability to provide summed ratings. The scale options address the intensity of users’ attitude about the application; the five-point Likert scale was “strongly disagree”, “disagree”, “neutral”, “agree” and “strongly agree” for Study I. While in Study II, the five-point Likert scale was “never”, “rarely”, “sometimes”, “often”, and “very often”. A social networking application, Facebook was selected as domain of study based on the phenomenon of ‘Facebooking’ which has claimed by many students as addictive [2]; in order to determine engagement level for each activity in Facebook. Thus, Facebook seems an appropriate and novel domain measuring the engagement attributes and levels. The survey was pre-tested with four people who responding to the survey. The reaction, suggestion and questions were noted during this exercise and verbal comments were gathered after they had completed. The second pre-test was then conducted with two different individuals and further incorporated into improving the presentation and understanding of the survey. Overall, the two pre-test reduced the scale to 23 items (from 28) divided into seven components of activities (i.e. Social Connection, Shared Identities, Photographs, Contents, Social Investigation, Social Network Surfing, and Status Updates) as stated in Appendix 1. It was the final version used in this study.
and between one and three hours (30.1%) A relatively small proportion of users claimed to spend either between three and five hours (12.6%) or more than five hours (13.6%). 88.3% of participants have internet access in their home residence. For internet usage activities among the respondents, social networking (Facebook) indicates the highest score with 21.68%; followed by emailing (20.42%) and learning purposes (20.05%). These findings help us to reaffirm the objective to measure user engagement on social networking application. The findings also indicate the preference usage among different platform devices with Non-mobile (i.e. computer, laptop) access devices (n=57, 55.3%); both mobile and non-mobile (n=42, 40.8%); and mobile device (n=9, 3.9%). 85.4% of participants agree that mobile devices make users feels more engaged while interacting with Facebook application as compared to non-mobile devices (14.6%).
A. Study I
The analysis of the results includes performing a factor analysis (FA) to assess construct validity and the nature of the factors. The sample of 103 is adequate to proceed with data analysis as recommended is at least 100 [13]. Exploratory factor analysis (EFA) was selected in order to examine the construct validity and multidimensionality of the instrument. The Kaiser Meyer-Olkin (KMO) Measure of Sampling Adequacy (KMO=0.819) indicated that factor analysis should result in distinct, reliable factors [8] and the Barlett’s Test of Sphericity verified that relationships existed among the items (x2 = 2686.88, df = 496, sig. = .000). The significant value is lower than 0.05; therefore the variables in the population correlation matrix are uncorrelated. As a result, it is necessary to process the factor analysis for the data due to the strength between the variables which is strong [5]. III. RESULTS Principle component extraction was used to maximize the There were 53 females (51.5%) and 50 males (48.5%) variance extracted and because an outcome of this analysis participating this study. Participants age ranged in age from was to identify the most parsimonious set of items [19]. 18-24 (n=20, 19.4%); 25-34 (n=67, 65.0%); 35-44 (n=14, Varimax rotation, the most common of the rotational 13.6%); and over than 45 (n=2, 1.9%). Out of these 103 techniques, was used to simplify the factors with Kaizer participants who stated their occupation, only 17.5% were Normalization. The cut-off value of 0.50 was selected to be students; unemployed were 4.9% and the remainder were conservative. Eleven iterations of factor analysis were employed in various fields (77.7%). For education level of the converged. During each iteration, items that loaded on participants; 71.8% were under-graduate (diploma and degree); multiple factors were eliminated [19]. post-graduate (masters, PhD) stated 8.7%; high-school and Factors were interpreted based on their make-up and other certifications were 9.7% each. labelled accordingly. The four factors, Focused Attention, Participants had friends linked to their Facebook profile Novelty-Endurability, Perceived Usability and Aestatics are between 251 and 500 friends (32.0%), more than 500 friends described according to the amount of variance explained by (30.1%), between 101 and 250 friends (28.2), and less than each factor; alpha values, the resulting number of items and 100 friends (9.7%). Participants who had been registered on item loadings. The obtained alpha score is 0.8676 which the application between one and two years (45.6%) indicate indicates that the scale is high in internal consistency. slightly higher than those who have registered for more than The first factor, Focused Attention accounted for 20.07% of two years (38.8%); meanwhile the remaining just registered the variance and consisted of nine items. These items related for less than six months (2.9%); and between six months and to user’s perceptions of time passing and their degree of almost one year (12.6%). The majority of participants visited awareness about what was taking place outside of their the site almost daily with 63.1%; one or two days (13.6%); interaction with Facebook. The remaining items pertained to three or four days (9.7%); and five or six days (13.6%). user’s ability to become absorbed while socializing with Among all respondents, the most common responses for the Facebook. The second factor, Novelty-Endurability was time spent on the site each day were almost one hour (43.7%) defined by nine items and accounted for 15.982% of the total
variance. Based on the previous findings of six engagement attributes [17], this new factor is a combination of items from Novelty and Endurability. Although it also included two items from each Aestatics and Felt Involvement, the name of factor Novelty-Endurability was taken. The third factor, Perceived Usability consisted 10.761% of the variance and comprised of four items. Items for this factor pertained to emotions experienced by respondents when completing their shopping task, i.e., “confusing”, “frustrated”, “demoralized”, and “annoyed” [17]. The fourth factor, Aestatics comprised of four items and accounted for 10.152% of the variance. This set of items pertained to specific feature of the interface, such as graphics/images and screen layout, and to respondents’ overall aesthetic impressions of the Facebook’s attractiveness and sensory appeal. The fifth factor consisted of one item with 6.799% of variance; the sixth factor consisted of two items with 6.529% of variance and; the seventh factor consisted of one item with 3.993% of variance were eliminated from the scale. Although most of item loadings are high and moderate for loading condition [20], it is considered as weak and unstable factors as the result of fewer than three items [3]. More findings regarding Study I are reported in [1].
B. Study II The Facebook activities’ instrument was adapted from previous work [11]. The obtained alpha score is 0.8876 which indicates that the scale is high in internal consistency. Each component of activities reliability index was also found to be relatively high; which are Social Connection (0.7264), Shared Identities (0.7978), Photographs (0.7991), Contents (0.7033), Social Investigation (0.7713), Social Network Surfing (0.8273), and Status Updates (0.8383). Thus, the scale of Facebook activities is consistence and reliable to be used in this study as mentioned in Appendix 1. Discrminant analysis is a parametric technique used to determine the weights of the best predictors for distinguishing two or more groups [7]. It is used to answer the question of how engagement measurement can be located into levels based on seven components of Facebook activities. We selected items that has been contributed for the four factors of engagement and combined it into the engagement variable. 60 54
Percentage (%)
50
40
34
30
20
10
12
0 slightly engaged
moderately engaged
highly engaged
Engagement Level
Figure 1: Engagement Level towards Facebook activities.
We classified engagement into three levels which are slightly engaged (less than 2.67), moderately engage (2.68 to 3.44) and highly engaged (more than 3.45). The sample size is large enough to enable the normal distribution assumptions to be fulfilled according to the central Limit Theorem. The second assumption related to the discriminant analysis of variance was tested using a Box’s M Homogeneity statistics. The results as follow: Box’s M = 72.607; F = 1.070; p-value > 0.05). Homogeneity of variance assumption was fulfilled. There are three levels of engagement which has divided as slightly engaged, moderately engaged and highly engaged. Moderately engaged (54%), followed by highly engaged (34%) and slightly engaged (12%) in Figure 1. Based on the three levels of engagement, the comparison of the mean and standard deviation for each activity is presented in Table I. The result shows the engagement level with highly engaged has the highest scores in all activities, moderately engaged provide the middle scores, while engagement level with slightly engaged indicate the lowest. The Facebook activities indicate engagement intensity from the highest to the lowest which descending sorted by defining the mean scores as follows; Social Connection (3.5777), Photographs (3.484), Status Updates (3.3139), Shared Identities (2.9385), Social Investigation (2.8252), Social Network Surfing (2.8026), and Contents (2.4628). Table II provides the results of mean equality test for the seven components of Facebook activity. Results shows there are significant differences in activities components for each level of engagement except Shared Identities. Thus, only six components of attitude will be compared in the subsequent analysis. Table III shows the comparison of eigenvalues of the variance between groups to variance within groups. Eigenvalue is one statistics for evaluating the magnitude of a discriminant analysis. A large eigenvalue is associated with a strong function. The output shows that the first discriminant functions indicate greater effect than the second function. The first discriminate function explains 83.7% of the total of variance of engagement level towards Facebook activities and 16.3% for the second function. Since both functions contribute for the total 100% of variance of engagement, function 1 through function 2 must be perform together in function test. Table IV shows the significance of discriminant function based on Wilks Lambda value. Wilks Lambda indicates how good the discriminating power of the model is. Both function 1 and function 2 are significant if being perform as a unit. However, if only function 2 is accounted for (by removing the first discriminant function), the second function is not significant because its significance value over than 0.05. The first two columns of Table V describe the component that make up each discriminant function. It shows that the seven components of Social Connection, Shared Identities, Photographs, Contents, Social Investigation, Social Network Surfing, and Status Updates, have form the first discriminate function. Four of the last columns in Table V show the correlation between each variable with each discriminant function. The value of non-standard coefficient is used to
create the discriminate function equation. The equations are as follows: Discriminant function I = -6.238 + .211 (Social Connection) + .253 (Photographs) + .287 (Contents) + .203 (Social Investigation) + .172 (Social Network Surfing) + .771 (Status Updates)
Discriminant function II = -1.211 + 1.145 (Social Connection) + -.963 (Photographs) + .624 (Contents) + .563 (Social Investigation) + -.304 (Social Network Surfing) + .701(Status Updates)
TABLE I MEAN AND STANDARD DEVIATION SCORE
Engagement Level Activities Social Connection Shared Identities Photographs Contents Social Investigation Social Network Surfing Status Updates
Slightly Engaged Mean Std.dev. 3.3958 0.69461 2.6667 1.09175 2.9167 0.50377 2.0278 0.83434 2.5000 0.50252 2.4167 0.69812 2.5833 0.57075
Moderately Engaged Mean Std.dev. 3.4732 0.50829 2.8214 0.94059 3.4420 0.68256 2.3155 0.74514 2.7024 0.83769 2.7083 0.80670 3.2440 0.74533
Highly Engaged Mean Std.dev. 3.8071 0.69949 3.2190 0.96319 3.7000 0.66088 2.8476 0.90150 3.1333 0.69640 3.0857 0.71557 3.6762 0.72077
Mean 3.5777 2.9385 3.4684 2.4628 2.8252 2.8026 3.3139
Total Std.dev. 0.61838 0.97880 0.69193 0.85461 0.78777 0.78997 0.78636
TABLE II TEST OF EQUALITY OF GROUP MEANS
Factor Social Connection Shared Identities Photographs Contents Social Investigation Social Network Surfing Status Updates
Wilks Lambda 0.927 0.955 0.886 0.884 0.914 0.920 0.821
F 3.943 2.362 6.436 6.585 4.698 4.346 10.880
df1 2 2 2 2 2 2 2
df2 100 100 100 100 100 100 100
Sig. 0.022 0.099 0.002 0.002 0.011 0.015 0.000
TABLE III EIGENVALUE FOR DISCRIMINANT FUNCTION
Function 1 2
Eigenvalue 0.285 0.055
% of Variance 83.7 16.3
Cumulative % 83.7 100.0
Canonical Correlation 0.471 0.229
TABLE IV SIGNIFICANCE FOR DISCRIMINANT FUNCTION
Function Test 1 through 2 2
Wilks Lambda 0.737 0.948
Chi-Square 29.565 5.230
df 14 6
Sig. 0.009 0.515
TABLE V STRUCTURE METRIC AND CANONICAL COEFFICIENT
Factor Function Social Connection Photographs Contents Social Investigation Social Network Surfing Status Updates Constant
Structure Metric 1 .864 .651 .557 .549 .489 .390 -
2 -.286 -.381 .317 .130 .440 .263 -
IV. DISCUSSION The engagement attributes which were identified from the previous research and an exploratory study formed the
Standard Coefficient 2 1 .127 .688 .166 -.633 .233 .506 .154 .428 .131 -.233 .555 -.504 -
Non-standard Coefficient 1 2 .211 1.145 .253 -.963 .287 .624 .203 .563 .172 -.304 .771 -.701 -6.238 -1.211
multidimensional scale to measure engaging user experiences with a technology [17]. We have evaluated the instrument’s reliability and validity which was adapted from the user
engagement scale in an online shopping environment to social network environment. The outcome is a reliable and valid scale comprised of four distinct factors: Focused Attention, Novelty-Endurability, Perceived Usability and Aestatics [1]. From these findings, we identified the engagement attributes in social networking are slightly difference than the attributes of engagement that have been found in the previous work. From the four distinct attributes of engagement, we recoded it into engagement levels which represented by slightly engaged, moderately engaged, and highly engaged. The results shows 54% of Facebook users are moderately engaged, 34 % of them are highly engaged and slightly engaged (12%). All Facebook activities in highly engaged level scored the highest, followed by moderately engaged level, and slightly engaged is the lowest score. One of activities which has been proposed by previous research, Shared Identities was eliminated from the function because its value showed no significant different for level of engagement. The results indicate that Social Connection is the most engaging activity, followed by Photographs, Status Updates, Social Investigation, Social Network Surfing, and Contents as the least engaging activity. All the activities expect the least engaging activity, Contents are associated with the social capital building gratification, where Facebook is used to build, invest in and maintain ties with distant friends and contacts ([4], [6]). There is evidence that Facebook profiles serve an important self-presentation tool [21] which keeps the users engage with it. Contents which contained instruments (i.e “use applications within Facebook”, “play Facebook games”, and “do Facebook quizzes”) is considered as less engaging element in Facebook usage and the mean scores of Contents were relatively low in previous research [11]. We also found the fact that majority of users agree that mobile devices make them more engage to Facebook as compared to non-mobile devices. V. CONCLUSIONS In this paper, we have recode four engagement attributes into engagement levels (i.e. slightly engaged, moderately engaged, & highly engaged). We conducted discriminant analysis (DA) to determine the level of engagement for each Facebook activities which has been done in our previous work. By conducting this work, we can have generalization ideas on user engagement attributes towards interacting with social networking application. The study also provides discriminate function that can be used to predict a level of engagement by using the scores of Facebook activities which are Social Connection, Photographs, Contents, Social Investigation, Social Network Surfing and Status Updates. Future work may be test discriminate function in a focus group (e.g mobile users and non-mobile user). ACKNOWLEDGMENT The research was supported by the university research grant (UKM-TT-03-FRGS0135-2010).
REFERENCES [1] [2] [3]
[4]
[5] [6] [7] [8] [9] [10] [11] [12] [13] [14]
[15] [16] [17] [18] [19] [20] [21]
[22] [23]
Banhawi, F., & Mohamad Ali, N., Measuring User Engagement Attributes In Social Networking Application, STAIR 2011, Malaysia. Bugeja, M.J., Facing the Facebook. Section: Chronicle Careers, Volume 52, Issue 21, 2006, Page C1.Retrieved March 23, 2011 from http://vpss.ku.edu/pdf/PSDC%20Facing%20the%20Facebook.pdf. Costello, A. B., & Osborne, J. W., Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical Assessment Research & Evaluation, 10(7), 2005. Retrieved April 17, 2011 from http://pareonline.net/pdf/v10n7a.pdf. Ellison, N., Steinfield, C., Lampe, C. Spatially Bounded Online Social Networks and Social Capital: the Role of Facebook. Paper presented at the annual meeting of the International Communication Association, Dresden, June 2006. Foulger, D. & other participants. Statistical Methods. MediaSpaceWiki, [Online]. Retrieved April 19, 2011 from http://evolutionarymedia.com/wiki.htm?StatisticalMethods. Golder, S.A., Wilkinson, D., & huberman, B.A, Rhythm of Social Interaction: Messaging within a Massive Online Network 3rd Conference on Communities and Technologies, (2007). Hair, J.F., Black, W.C., Babin, B.J., Anderson, R.E., & Tatham, R.L. , Multivariate Data Analysis. 6th Edition, New Jersey: Pearson, 2006. Hutcheson, G.D., & Sofroniou, N., The multivariate social scientist: Introductory statistics using generalized linear models. Landon: Sage, 1999. Jasques, R.D., The nature of engagement and its role in hypermedia evaluation and design. Unpublished doctoral dissertation, South Bank University, London, 1996. Jacques, R., Preece, J., & Carey, T., Engagement as a design concept for multimedia. Canadian Journal of Education Communication, 24(1), 1995, 49-59. Joinson, A.N., ‘Looking at’, ‘Looking up’, or ‘Keeping up with’ People? Motives & Uses of Facebook, Copyright 2008, ACM 978-160558-011-1/08/04. Lampe, C., Ellison, N., and Steinfield, C., A Face(book) in the Crowd: Social Searching vs. Social Browsing. CSCW’06, Banff, Canada, 2006. MacCallum, R. C., Widaman, K. F., Zhang, S., & Hong S., Sample size in factor analysis. Psychological Methods, 4, 1999, 84-99. Mahyuddin, K.M.N, & Noah, S.A., Superficial Method for Extracting Social Network for Academics Web Snippets. Rough Set and Knowledge technology Lecture Notes in Computer Science, Volume 6401/2010, 483-490, DOI: 10.1007/978-3-642-16248-0_68, 2010. M. Mohd, et al., “Design of an Interface for Interactive Topic Detection and Tracking,” Lecture Notes in Artificial Intelligence 5822, 2009, pp. 227-238. O’Brien H.L, & Toms, E.G., What is user engagement? A conceptual framework for defining user engagement with technology. Journal of the American Society for Information Science and Technology, 2008. O’Brien, H.L., & Toms, E.G., the Development and Evaluation of a Survey to Measure User Engagement, Journal of the American Society for Information Science and Technology, 61(1):50-69, 2010. Singleton, R.A., Straits, B.C, & Straits, M.M., Approaches to social sciences research. New York: ACM Press, 1993. Tabachnick, B.G., & Fidell, L.S., Using multivariate statistics (fourth ed.). needham heights, MA: Allyn & Bacon, 2001. Velicer, W. F., & Fava, J. L., Effects of variable and subject sampling on factor pattern recovery. Psychological Methods, 3, 1998, 231-251. Walther, J.B., Van Der Heide, B., Kim, S-Y., Westerman, D., Tong, S.T., The Role of Friends’ Appearance and Behavior on Evaluation of Individuals on Facebook: Are we known by the company we keep? Human Communication Research, 2008. Webster, J., & Ho, H., Audience engagement in multimedia presentations. The DATA BASE for Advances in Information Systems, 28(2), 1997, 63-67. Wellman, B., & Gulia, M., Net Surfers Don’t Ride Alone: Virtual Communities as Communities, Communities in cyberspace. Chapter 7, 1999, 167-179.
Appendix 1. Facebook Activities Factor Social Connection [1] [2] [3] [4] Shared Identities [5] [6] [7] Photographs [8] [9] [10] [11] Contents [12] [13] [14] Social Investigation [15] [16] [17] Social Network Surfing [18] [19] [20] Status Updates [21] [22] [23]
Items I find out what my old friends are doing now. I reconnect with people whom I have lost touch with. I keep in touch with friends/acquaintances to prevent lost of contact. I approve or reject friend requests if there is any. I organize or join events by using Facebook (e.g. party, wedding invitation) I join my favourite groups' pages (e.g musics, politics, football teams & etc.) I expand my social network with people of the same interest. I view photos of other friends/acquaintance. I am being tagged in photos. I tag my friends in photos when there is a purpose to do so. I share and post photos on Facebook. I use applications within Facebook (e.g chatbox,messenging & etc). I play Facebook games. I do Facebook quizzes. I observe my friends communicating with each other. I search new friends with specific interest by using advanced search. I get to know someone better through their Facebook interactions with others. I look at other people's Facebook page which has caught my attention. I view strangers' friends list to check out interesting people. I view my friends' friends list to check out interesting person/people I might know. I keep my status updated when I have something to share with. I check out news feed to know my friends' activities on Facebook. I check out what others put as their status and its comments.
