ARTICLE IN PRESS
International Journal of Industrial Ergonomics 35 (2005) 697–711 www.elsevier.com/locate/ergon
Evaluating the industrial ergonomics of service quality for online recruitment websites Joanna P.C. Tonga, Vincent G. Duffyb,, Ginger W. Crossc, Fugee Tsunga, Benjamin P.C. Yend a
Department of Industrial Engineering and Engineering Management, The Hong Kong University of Science and Technology, Hong Kong b Department of Industrial Engineering, Mississippi State University, P.O. Box 9542, Mississippi State, MS 39762-9542, USA c Department of Psychology, Mississippi State University, MS, USA d Faculty of Business and Economics, School of Business, The Hong Kong University, Hong Kong Received 18 February 2004; received in revised form 7 February 2005; accepted 8 February 2005 Available online 21 April 2005
Abstract In this current economic climate, many people are searching for jobs with Internet-based (online) recruitment service providers. In order to serve as a viable source of help for these people and ultimately maintain the satisfaction of industrial employers—their paying customers, online recruitment websites should strive to achieve a high level of service quality for the job seekers. The current research (1) proposes an evaluation tool to assist online recruitment websites in quantifying service quality and (2) examines the relationship between service quality and measures of mental workload and performance time. Results suggest a central role for performance time and mental workload on perceptions of service quality. Key elements of service quality for Internet-based job recruitment are provided based on the results and are shown in contrast with those suggested as traditional measures of service quality. Relevance to industry: Time and mental workload required to complete job recruitment tasks with Internet-based service providers can significantly impact overall service quality evaluations. Thus, they should be monitored and benchmarked when possible, as poor service quality perceptions can ultimately impact employers’ willingness to subscribe. General service quality, accuracy/efficiency and interface are defined. r 2005 Elsevier B.V. All rights reserved. Keywords: Service quality; Recruitment website; Mental workload; Information technology
1. Introduction 1.1. Unemployment statistics Corresponding author. Tel.: +1 662 325 1677;
fax: +1 662 325 7618. E-mail address:
[email protected] (V.G. Duffy).
According to the United States Department of Labor’s Bureau of Labor Statistics (2003)
0169-8141/$ - see front matter r 2005 Elsevier B.V. All rights reserved. doi:10.1016/j.ergon.2005.02.004
ARTICLE IN PRESS 698
J.P.C. Tong et al. / International Journal of Industrial Ergonomics 35 (2005) 697–711
Employment Situation Summary, 8.8 million people in the United States were unemployed in April 2003, which was an increase from 5.8 to 6.0 percent of the population. In addition, only 62.4% of Americans had total employment and 4.8 million were working part time, when in fact they would prefer full-time positions. Particular industries have recently experienced significant job losses. In April 2003, more than 95,000 job losses were reported in the manufacturing industry alone. This figure more than doubles the average monthly decline reported over the past 12 months for that industry. Because of these recent job losses, many Americans are currently searching for jobs. 1.2. Using the Internet to find a job When looking for a job, the Internet may be a source of help for some, as websites such as monster.com and hotjobs.com offer free job searches and resume posting. In recent years, job recruitment websites have undergone considerable growth and the number of job seekers who conduct job searching over the Internet has increased. This increase may be due to the added convenience provided by online recruitment websites (e.g., job seekers are able to view multiple openings and post their resumes for businesses to view without leaving the comfort of their homes). Online recruitment websites are designed as a powerful medium that brings employers and job seekers together and allows them to interact in a fast, efficient, and effective way. For job seekers, an online recruitment website provides a unique opportunity to explore constantly updated employment opportunities through an extensive database of jobs. Job seekers can then post their resumes online and announce their availability to potential employers. According to Kuhn and Skuterud (2000), 15% of unemployed people who were searching for jobs, even in December 1998, took advantage of Internet sources. In addition, they found that more people used the Internet to search for a job than other, more traditional means (e.g., private employment agencies, union registers, or friends and relatives). For employers, online recruitment delivers a conveni-
ent, low cost, and efficient solution by providing direct access to a continuously expanding database of resumes. In general, the recruitment service provided on these websites is free for the job seekers, who can enjoy the service once they have become members of that website. In contrast, employers typically have to pay a subscription fee in order to take advantage of the recruitment service. Because employers are required to pay for the service, their perceptions of the level of service are typically the concern of most recruitment websites and the recruitment service quality level for the job seekers is typically ignored. However, this is a concern because, if job seekers are unhappy with the service provided by the recruitment website, they may look for other websites that fulfill their expectations for service quality. If this continues to happen, the websites with low service quality for the job seekers may experience a decrease in the number of active job seekers. This, in turn, may lead to a reduction in the number of employers who are willing to subscribe to the recruitment service. Therefore, the current research examines job seekers’ perceptions of service quality level and how these perceptions relate to previously defined dimensions of service quality as well as measures of performance time and mental workload. If websites are able to determine the underlying dimensions of service quality that are relevant to online recruitment, they may be better able to design their websites to provide users with the highest service quality possible. In addition, if they have a tool for measuring a user’s perception of their service quality, they may be able to adjust their service to retain current potential employees (or job seekers) and motivate new employees to use their site. Thus, one purpose of this research is to propose an evaluation tool to assist online recruitment websites in quantifying their service quality level. Without face-to-face human interaction in the web environment, web user interface becomes the sole source of interaction between company and customer. As a result, numerous website evaluation studies have focused on web user interface design (Gehrke and Turban, 1999; Zhang et al., 1999; Zhang et al., 2000; Zhang and von Dran,
ARTICLE IN PRESS J.P.C. Tong et al. / International Journal of Industrial Ergonomics 35 (2005) 697–711
2001). Lohse and Spiller (1998) claimed that designing online stores with effective customer interfaces has a critical influence on traffic sales. Gehrke and Turban (1999) found that a user interface is the major factor that determines buyers’ willingness to make purchases on the Internet. In addition, Liang and Lai (2000) pointed out that design quality has a significant impact on consumer choice of electronic stores and consumers are more likely to repeatedly visit stores with well-designed sites. However, these studies do not typically mention much about the mental effort or amount of time required when using these interfaces. As a result, an additional objective of this research is to investigate the amount of time and mental effort required to perform tasks on the websites and how they relate to evaluations of electronic service quality. 1.3. Previous research on service quality Traditional research on service quality has measured service quality for exchanges that are more interpersonal in nature. That is, most of these studies have focused on brick and mortar businesses, not those that are on the Internet, where the interpersonal aspect of transactions has been removed. However, recent studies have focused on evaluating service quality for online businesses (Janda et al., 2002). According to Sterne (2000), Internet customers are pickier, more demanding, and have higher expectations. Therefore, excellent electronic service (e-service) should be provided to these customers in order to fulfill their expectations. In the web environment, the web (user interface) can be regarded as a service and users are consumers (Zhang and von Dran, 2001). Rust and Lemon (2001) claimed that an understanding of e-service is critically important and confirmed the idea that today’s Internet customers are less willing to put up with poor service. Therefore, it is essential for businesses to evaluate and improve e-service quality in order to keep existing customers. According to Zeithaml et al. (2000), services differ from goods in terms of how they are produced, consumed, and evaluated. Based on the definitions from Parasuraman et al. (1985), the
699
nature of services is probably intangible, heterogeneous, and inseparable. Parasuraman et al. (1988) pointed out that service quality is a global judgment, or attitude, relating to the superiority of a service. The empirical research conducted by Parasuraman et al. (1988) indicated that there are five principle dimensions of service quality: Tangibles, Reliability, Responsiveness, Assurance, and Empathy. Zeithaml et al. (2000) pointed out that perceived electronic Service Quality (e-SQ) can be represented by eleven dimensions: Reliability, Responsiveness, Access, Flexibility, Ease of Navigation, Efficiency, Assurance/Trust, Security/Privacy, Price Knowledge, Site Aesthetics, and Customization/Personalization. Previous research on service quality, like that mentioned above, tends to focus on retail service quality and has not typically examined service quality dimensions for online recruitment websites where the type of service expected may be distinct from the traditional retail setting. As previously indicated, online recruitment sites typically offer free job searches and resume postings for job seekers, but charge a small fee to employers. In order to retain potential employees for their paying customers, it is important to determine what factors influence a job seekers’ perception of overall service quality. A recent study by Liljander et al. (2002) examined different dimensions of service quality in the context of a specific online recruitment portal. Job applicants who were registered with the company were invited to complete an online survey that included questions on perceived service quality. The authors came up with 4 dimensions of service quality that they deemed relevant to the particular online recruitment portal. These included site design and content, trust, empathy, and security/privacy. 1.4. Purpose of the current research The current research attempts to extend previous research by relating service quality evaluations to cognitive measures such as mental workload and objective measures such as overall performance time required to complete tasks on an online recruitment website. In addition, the current research attempts to
ARTICLE IN PRESS 700
J.P.C. Tong et al. / International Journal of Industrial Ergonomics 35 (2005) 697–711
determine the underlying dimensions of service quality that most influence a person’s perception of overall service quality. 1.5. Hypotheses 1.5.1. Hypothesis 1: Performance time and overall service quality According to a survey conducted by Hamilton (1997), slow speed is the number one complaint of web users. Kalakota and Robinson (1999) claim that as the time quotas of the customers shrink, they probably look for companies whose sites can serve them faster. According to Xie and Salvendy (2000), performance time is the total time that a person spends accomplishing a task. Therefore, with respect to the current research, the recruitment companies should pay attention to the time spent on every step of the online job seeking process on their websites. In the current research, we hypothesize that overall service quality will be significantly related to measures of performance time for completing job search and application tasks. That is, we believe that longer performance times will result in lower service quality ratings. 1.5.2. Hypothesis 2: Mental workload and overall service quality Mental workload has long been recognized as an important factor in human performance in complex systems (Hancock and Meshkati, 1988). However, the concept of mental workload is incompletely defined and is multifaceted (Meshkati, 1988; Xie and Salvendy, 2000), and, according to Xie and Salvendy (2000), mental workload cannot be detected directly, but can be measured indirectly through subjective ratings, performance, or physiological variables, which are thought to correlate highly with mental workload. Some of the common subjective methods are SWAT (Subjective Workload Assessment Technique; Reid and Nygren, 1988), NASA-TLX (NASA-Task Load Index; Hart and Staveland, 1988), modified Cooper-Harper Scale (Wierwille and Casali, 1983), and Subjective Workload Assessment (Vidulich and Tsang, 1987). In the current research, we hypothesize that overall service quality will be significantly related to measures of both overall
(measured with the NASA-TLX) and task-specific mental workload (measured with the modified Cooper-Harper Scale). That is, we expect that higher measured mental workload will result in lower perceived service quality.
2. Method 2.1. Selecting example high and low service quality websites In order to measure performance time and mental workload while completing tasks on websites that offer low service quality versus high service quality, it was necessary to obtain examples of each. Thus, a preliminary study was conducted to select two recruitment websites that represented the two service quality levels (Low service quality, or LSQ, and High service quality, or HSQ). An email questionnaire was sent to 100 Hong Kong Chinese. In this questionnaire, the participants were asked to imagine a scenario in which they would like to find and apply for a job over the Internet. The participants were given a list of ten recruitment websites, which were selected based on the results of a search engine. The task of respondents was to indicate their three most preferred and three least preferred sites (in order of priority) based on nine of the dimensions of eSQ proposed by Zeithaml et al., (2000). Specifically, they were asked to evaluate the sites based on reliability, responsiveness, flexibility, ease of navigation, efficiency, assurance/trust, security/ privacy, aesthetics, and customization/personalization. Of the original 11 dimensions, ‘‘access’’ (which was related to a person’s ability to get to the site quickly and contact the company if necessary) was modified to apply to the current task and combined with ‘‘reliability’’ (see Table 1). In fact, Zeithaml et al. (2000) mentioned that some dimensions could be combined into more global dimensions. In addition, ‘‘price knowledge’’ was removed because the job recruitment service was free for job seekers. In total, 83 responses were received (response rate ¼ 83%). The site that received the highest number of ‘‘first most preferred’’ votes, served as
ARTICLE IN PRESS J.P.C. Tong et al. / International Journal of Industrial Ergonomics 35 (2005) 697–711
701
Table 1 Original and modified definitions of the nine electronic service quality dimensions proposed by Zeithaml et al. (2000) Dimension
Original definition
Modified definition
Reliability
Reliability involves the correct technical functioning of the site and the accuracy of service promises (having items in stock, delivering when promised), billing and product information
Responsiveness
Responsiveness means quick response and the ability to reach the company when needed
Flexibility
Flexibility involves choice of ways to pay, ship, buy, search for and return items
Ease of navigation
Ease of navigation means that a site contains functions that help customers find what they need without difficulty, possesses a good search engine, and allows the customer to maneuver easily and quickly back and forth through the pages
Efficiency
Efficiency means that a site is simple to use, structured properly, and requires a minimum of information to be input by the customer Assurance/trust involves the confidence the customer feels in dealing with the site
Reliability involves the correct technical functioning of the site and the accuracy of service promises (having adequate employment), employment description, and contact information of the recruiters Responsiveness means quick response and the ability to get help if there is a problem or question from the job seekers Flexibility involves choice of different platform/browsers, language, and ways to search for the ideal jobs Ease of navigation means that a site contains functions that help job seekers find what they need without difficulty, possesses a good search engine, and allows the job seekers to maneuver easily and quickly back and forth through the pages Efficiency means that a site is simple to use, structured properly, and requires a minimum of information to be input by the job seekers Assurance/trust involves the confidence the job seeker feels in dealing with the site and is due to the site reputation, and the well-known employers in the website Security/privacy involves the degree to which the customer believes the site is safe from intrusion and personal information is protected Site aesthetics refers to the visual appearance of the site Customization/personalization is how much and how easily the site can be tailored to individual job seekers’ job preferences, applied job history, ways of job seeking
Assurance/trust
Security/privacy
Site aesthetics Customization/ personalization
Security/privacy involves the degree to which the customer believes the site is safe from intrusion and personal information is protected Site aesthetics relates to the appearance of the site Customization/personalization is how much and how easily the site can be tailored to individual customers’ preferences, histories, and ways of shopping
the HSQ site, while the one that received the highest number of ‘‘first least preferred’’ votes served as the LSQ. Thus, a website called JobsDB (http:// www.jobsdb.com) served as the HSQ site (39 out of 83 participants indicated it was their most preferred) and a website called Strongnet (http:// www.strongnet.com) served as the LSQ site (30 out of 83 indicated it was their least preferred). Both sites were in English and were evaluated in 2002. 2.2. Participants and apparatus used in the study 100 students from the Hong Kong University of Science and Technology (HKUST) were recruited
and paid HKD75 ($9.62 US) for their participation in the 1-h experiment. The age of participants ranged from 18 to 25 years old (M ¼ 20.98, SD ¼ 1.05). Half of the participants were male and half were female. All participants had basic Internet navigating experience, and were considered potential users of the online recruitment websites. In fact, 83% indicated they had previously used the Internet for job searches, 98% indicated they would use free online recruitment services, and 82% indicated they preferred online application procedures to traditional application procedures. Participants were asked to complete four job application tasks on their assigned
ARTICLE IN PRESS 702
J.P.C. Tong et al. / International Journal of Industrial Ergonomics 35 (2005) 697–711
website using a personal computer with a Pentium III, 750 MHz processor, and a 17’’ SVGA monitor. Participants used standard input devices such as the mouse and keyboard to complete the tasks. Fifty participants were assigned to the LSQ website and 50 were assigned to the HSQ website. After completing the 4 tasks, participants were asked to complete the Website Service Quality Questionnaire (Appendix A) that asked them to rate their respective website on particular service quality dimensions. In addition, between tasks and after completing all tasks, participants were asked to complete different measures of mental workload (described in more detail in the following sections). The four job-application tasks and the Website Service Quality Questionnaire are described below.
involved in the job seeking process (in minutes). We hypothesized that longer performance times would result in lower perceived service quality. For both websites, the stopwatch was started when the participant clicked on the link that allowed them to start creating an account (i.e., after clicking ‘‘Join Us’’ on JobsDB or ‘‘New User’’ on Strongnet). The stopwatch was stopped between tasks to allow participants to complete the taskspecific mental workload measure (described below). The clock was stopped for the last time when participants clicked on the link that completed the job application (i.e., after clicking ‘‘Confirm’’ on JobsDB or clicking ‘‘Send’’ on the email composition box for Strongnet).
2.3. Tasks completed using the online recruitment websites
2.5. Measuring mental workload
Participants were asked to complete an online job application (made up of 4 tasks) using their assigned recruitment website. Participants assigned to the LSQ condition used Strongnet, while participants assigned to the HSQ used JobsDB. Specifically, participants were given information to insert into a resume regarding their educational background, project experience, skills, expected salary, and availability and were told that their desired position was an engineering or programming job in the field of information technology. Using this information, participants were asked to (1) create an account, (2) create a resume, (3) conduct a job search, and (4) complete a job application. Before completing the 4 tasks, participants were shown a slide show overview of Hong Kong job recruitment websites. The slide show included a description of the four general job application procedures that they would complete and a definition of the nine electronic service quality dimensions that were used to create the Website Service Quality Questionnaire. 2.4. Measuring performance time A stopwatch was used to capture the overall performance time required to complete all 4 tasks
Moray (1988) claimed that optimizing the allocation of mental workload to operators could reduce human error, improve system safety, and increase productivity and operator satisfaction. For the purpose of the current research, we were interested in the relationship between mental workload and perceived overall service quality. We hypothesized that higher mental workload would result in lower perceived service quality. We measured overall mental workload with the NASA-TLX (NASA-Task Load Index; Hart and Staveland, 1988) and task-specific mental workload with the Modified Cooper–Harper Scale (Wierwille and Casali, 1983).
2.5.1. Overall mental workload After completing the entire job application process (all 4 tasks), participants were asked to complete the NASA-TLX (NASA-Task Load Index; Hart and Staveland, 1988).
2.5.2. Task-specific mental workload After performing each of the four job application tasks, participants were asked to complete the Modified Cooper–Harper Scale (Wierwille and Casali, 1983).
ARTICLE IN PRESS J.P.C. Tong et al. / International Journal of Industrial Ergonomics 35 (2005) 697–711
703
2.6. Creating the service quality evaluation tool
2.7. Summary of experimental design
After completing the 4 tasks previously described, participants were also asked to complete a Website Service Quality Questionnaire (see Appendix A), which asked them to rate the performance of their assigned website on different questions pertaining to dimensions of service quality, using a 7-point Likert scale ranging from 1—‘‘Very strongly disagree’’ to 7—‘‘Very strongly agree’’. The first 27 questions were derived by a review of relevant research on website service quality evaluation in the context of online recruitment websites as well as by modifying nine of the eleven service quality dimensions proposed by Zeithaml et al. (2000) to apply to online recruitment. Thus, the service quality dimensions that were used to create the evaluation tool were validated by Zeithaml et al. (2000) and we modified the dimensions to fit the context of online recruitment. Our interest was in determining how well these dimensions would apply to online recruitment sites. Our goal was to use the data from the participants in this study to classify the items and characterize the dimensions relevant to job recruitment service quality evaluations. Specific judgments regarding question selection were made based on professional experience gathered during studies of product development effectiveness in more than 100 companies as reported in Duffy and Salvendy (1998, 1999a, b) and Duffy et al. (1995). Table 1 shows the original and modified definitions of the nine electronic service quality dimensions. The last question on the questionnaire (number 28) asked participants to give an overall service quality rating for the website. This scale is a uni-dimensional bipolar scale that consists of a horizontal line divided into 20 slots. ‘‘Low’’ is marked on the left end of the scale and ‘‘High’’ was marked on the right end. Participants indicated their overall service quality rating by putting an ‘‘x’’ in one of the 20 slots. The purpose of this additional question was to determine which dimensions of the questionnaire would be correlated with a person’s perception of overall service quality. That is, particular dimensions may have weighed more on a person’s perception of overall service quality.
In summary, 100 participants were assigned to one of two chosen websites that represented different service quality levels (HSQ and LSQ). Each participant completed four job application tasks on their assigned website. After each task, they completed the Modified Cooper–Harper Scale (used to measure task-specific mental workload). At the end of all tasks, they completed the NASA TLX (used to measure overall mental workload). In addition, at the end of all tasks, participants used the Website Service Quality Questionnaire to rate their assigned website on different dimensions of service quality and then give an overall rating for service quality. We were interested in (1) the differences between the assigned LSQ and HSQ websites with regard to performance time and mental workload, (2) the relationship between these measures and overall service quality rating, and (3) the dimensions of service quality that correlate with participants’ perceptions of overall service quality.
3. Results Much of the data in this experiment does not follow a normal distribution. Therefore, both parametric and distribution-free tests were conducted. 3.1. Validating LSQ and HSQ websites used in experiment At the end of the questionnaire (question 28), participants were asked to give an overall service quality rating for their assigned website. The means and standard deviations for the LSQ and HSQ websites used in this experiment are shown in Table 2. Based on the results of a Two-Sample ttest and Mann–Whitney test, the mean overall service quality rating for the HSQ website was significantly higher (Two-Sample t-test, po0.01; Mann–Whitney test, po0.01) than that for the LSQ website. Thus, it appears that JobsDB may be used to represent the HSQ level and Strongnet may be used to represent the LSQ level website.
ARTICLE IN PRESS J.P.C. Tong et al. / International Journal of Industrial Ergonomics 35 (2005) 697–711
704
Table 2 Means and standard deviations of overall service quality ratings
HSQ LSQ
Mean
Standard deviation
77.40 28.20
12.26 14.70
Table 3 Means and standard deviations of overall performance time
HSQ LSQ
Mean
Standard deviation
15.28 19.97
2.94 3.23
Table 4 ANOVA table of overall performance time in HSQ and LSQ websites Source
df
SS
MS
F
P value
Factor Error Total
1 98 99
549.90 933.81 1483.71
549.90 9.53
57.71
o0.01
3.2. Relationship between performance time and overall service quality We hypothesized that overall service quality would be significantly related to the amount of time required to complete the job application procedure (all 4 tasks). The means and standard deviations of the overall performance times are shown in Table 3. The results of an ANOVA (see results in Table 4) and a Mann–Whitney test showed that the amount of time required to accomplish the four job seeking tasks was significantly longer for the low service quality website than for the high service quality website (ANOVA: F ¼ 57.71, po0.01; Mann–Whitney test: po0.01). In addition, there was a significant negative correlation between performance time and overall service quality. (r ¼ .58, po0.01). Thus, longer performance times were correlated with lower overall service quality ratings.
The significant difference in overall performance time between the HSQ and LSQ websites may be explained by the difference in the interface design of the two recruitment websites. In order to complete task 2 (Create a resume), Strongnet required participants to type in the entire resume in a provided text area. In task 3 (Conduct job search), Strongnet (LSQ) did not provide a search engine, so participants had to search for a desired job by manually clicking on the job titles one by one. In task 4 (Complete a job application), participants using Strongnet (LSQ) had to send an email to inform the Webmaster of Strongnet that they would like to apply for the job vacancy. As a result, the total performance time required to complete an online job application was much longer in Strongnet (LSQ). To conclude, overall performance time required for conducting the job application process was significantly longer for Strongnet (LSQ) than for JobsDB (HSQ) and there was a significant negative correlation between performance time and overall service quality rating. As a result, our first hypothesis was supported. Specifically, overall performance time required for conducting the whole job seeking process was higher for the LSQ website and showed a significant relationship with the perception of electronic service quality of a recruitment website. 3.3. Relationship between mental workload and overall service quality We next hypothesized that overall and taskspecific mental workload would be higher for the Low Service Quality websites and that this would result in lower perceived electronic service quality. That is, we hypothesized that there would be a negative correlation between mental workload measures and perceived service quality. 3.3.1. Overall mental workload The means and standard deviations for overall mental workload are shown in Table 5. According to the ANOVA results (see Table 6), overall mental workload measured for those using the LSQ recruitment website was significantly higher than that for those using the HSQ recruitment
ARTICLE IN PRESS J.P.C. Tong et al. / International Journal of Industrial Ergonomics 35 (2005) 697–711 Table 5 Means and standard deviations of overall mental workload
HSQ LSQ
Mean
Standard deviation
39.81 75.49
18.51 12.78
Table 6 ANOVA table of overall mental workload in HSQ and LSQ Source
df
SS
MS
F
P value
Factor Error Total
1 98 99
31816 24792 56608
31816 253
125.76
o0.01
website (F ¼ 125.76, po0.01). That is, the overall mental effort required to complete the job seeking process was significantly higher for participants using Strongnet (LSQ) than for participants using JobsDB (HSQ). In addition, there was a significant negative correlation between overall mental workload and overall service quality rating (r ¼ .78, p o.01). The difference in measured overall mental workload in different service quality recruitment websites may be explained by the difference in the interface design for the two websites. In Strongnet (LSQ), participants were required to enter their whole resume in the provided text area. In addition, Strongnet did not provide a search engine, so subjects had to search for jobs manually by clicking on the job titles one by one. After identifying the desired job vacancy, they had to send an email to the Webmaster of Strongnet and request Strongnet to submit their resume to the corresponding company for job application. In short, as the web interface provided in Strongnet was more difficult to use, more mental effort was required to finish the job application process. In contrast, subjects using JobsDB (HSQ) created their electronic resume by filling out a form. They were also able to use a search engine provided on the website to search for job. Once they identified the desired position, they could
705
apply for the job with only three mouse clicks. As the web interface provided on JobsDB was very user-friendly, little mental effort was required to accomplish the job application process. In conclusion, higher overall mental effort was required for the low service quality recruitment website (Strongnet) than for the high service quality website (JobsDB). Therefore, our second hypothesis was supported. That is, there was a significant difference in the overall mental workload required to complete tasks in the LSQ and HSQ sites and a significant negative relationship (r ¼ .78, po.01) between mental workload and the perception of overall service quality of a recruitment website. 3.3.2. Task-specific mental workload The means and standard deviations for taskspecific mental workload measured for low and high service quality websites are shown in Table 7. Normal probability plots showed that the taskspecific mental workload measured after task 1 (p40.15) and task 2 (p ¼ 0.10) was approximately normally distributed. While for tasks 3 and 4, normal probability plots show that the taskspecific mental workload measured was not normally distributed. 3.3.2.1. Task 1: create an account. According to the results of an ANOVA (see Table 8) and a Mann–Whitney test, there was no significant difference between LSQ and HSQ (ANOVA: F ¼ 0.11, p ¼ 0.74; Mann–Whitney test: p ¼ 0.55) with respect to the measured mental workload for task 1 (create an account). In addition, Table 7 Means and standard deviations of task-specific mental workload for tasks 1–4 Task 1
Task 2
Task 3
Task 4
LSQ Mean Standard deviation
2.15 0.66
5.53 2.13
4.94 2.02
4.49 1.91
HSQ Mean Standard deviation
2.18 1.10
4.02 1.74
2.62 1.05
1.70 0.95
ARTICLE IN PRESS J.P.C. Tong et al. / International Journal of Industrial Ergonomics 35 (2005) 697–711
706
Table 8 ANOVA table of task-specific mental workload for task 1 (create an account) in HSQ and LSQ websites
Table 10 ANOVA table of task-specific mental workload for task 3 (conduct a job search) in HSQ and LSQ websites
Source
df
SS
MS
F
P value
Source
df
SS
MS
F
P value
Factor Error Total
1 98 99
0.090 80.660 80.750
0.090 0.823
0.11
o0.742
Factor Error Total
1 98 99
547.56 254.28 801.84
547.56 2.59
211.03
o0.01
Table 9 ANOVA table of task-specific mental workload for task 2 (create a resume) in HSQ and LSQ websites Source
df
SS
MS
F
P value
Factor Error Total
1 98 99
234.09 370.66 604.75
234.09 3.78
61.89
o0.01
there was no significant correlation between task 1 mental workload and overall service quality rating, r ¼ .033, po0.75. Therefore, it can be concluded that for this task, there is not a significant relationship between service quality level and mental workload for task 1. This may be due to the fact that, for this task, the web user interfaces for LSQ and HSQ were quite similar. That is, both websites required participants to input a small amount of information to complete the task of creating an account. 3.3.2.2. Task 2: create a resume. The results of an ANOVA (see Table 9) and a Mann–Whitney Test indicate that measured mental workload for task 2 (create a resume) in the LSQ website was significantly higher than that for the HSQ website (ANOVA: F ¼ 61.89, po0.01; Mann–Whitney test: po0.01). This finding implies that participants found it more difficult to accomplish task 2 in Strongnet and hence more mental effort was required to complete the task. This may be because Strongnet required participants to enter the whole resume in a provided text area. While in JobsDB, participants created their resume by simply completing a web-based form. In addition, a significant correlation between task 2 mental workload and
Table 11 ANOVA table of task-specific mental workload for task 4 (complete a job application) in HSQ and LSQ websites Source
df
SS
MS
F
P value
Factor Error Total
1 98 99
795.24 223.72 1018.96
795.24 2.28
348.35
o0.01
overall service quality rating was found, r ¼ .62, po.001. 3.3.2.3. Task 3: conduct a job search. According to the results from an ANOVA (see Table 10) and a Mann–Whitney Test, the measured mental workload for task 3 (conduct a job search) in the LSQ website was significantly higher than that for the HSQ website (ANOVA: F ¼ 211.03, po0.01; Mann–Whitney test: po0.01). Therefore, more mental effort was required to complete task 3 in Strongnet (LSQ) than in JobsDB (HSQ). This may be because there was no search engine provided in Strongnet (LSQ) and participants needed to search for jobs manually. While in JobsDB, (HSQ) participants searched for the jobs by using the search engine. As a result, it may be concluded that the amount of mental workload necessary for task 3 was significantly different for the LSQ and HSQ websites and was significantly correlated with overall service quality ratings, r ¼ .83, po .001. 3.3.2.4. Task 4: complete a job application. Finally, the results of an ANOVA (see Table 11) and a Mann–Whitney Test show that the measured mental workload for task 4 (complete a job
ARTICLE IN PRESS J.P.C. Tong et al. / International Journal of Industrial Ergonomics 35 (2005) 697–711
707
Table 12 Component loadings Loading Component 1: general service quality Question 1: Running smoothly Question 3: Adequate job employment Question 4: Confirmation of job application Question 5: Online help functions Question 7: Customized search functions Question 8: Un-subscription of promotional materials Question 10: Different language versions Question 12: Ability to maneuver through the site Question 14: High speed of online application Question 17: Structured properly Question 18: Well known Question 19: Well known recruiters Question 20: Secure Question 21: Privacy policy Question 22: Eye-catching design Question 24: Job alert Question 25: Shows the applied job history Question 26: Customized information presentation Question 27: Provides different ways of job seeking
.711 .709 .829 .561 .849 .761 -.455 .557 .762 .565 .855 .826 .650 .758 .444 .867 .812 .824 .806
Component 2: accuracy and efficiency Question 2: Accurate information Question 15: Simple to use Question 16: Minimum input of information
.660 .696 .541
Component 3: interface Question 11: Provides navigation aids Question 23: Simple background colors and texture
.607 .785
Component 4: maneuvering speed Question 13: Provides high speed of maneuvering
.874
Component 5: additional support Question 6: Frequently asked question (FAQ) section Question 9: Support for different platforms and/or browsers
.892 .424
application) in the HSQ website was significantly lower than that for the LSQ website (ANOVA: F ¼ 348.35, po0.01; Mann–Whitney test: po 0.01). Therefore, more mental effort was required to accomplish task 4 in Strongnet (LSQ) than in JobsDB (HSQ). This may be because participants could not apply for the job directly on Strongnet. Instead, they had to send an email to the Webmaster of Strongnet and request Strongnet to submit the electronic resume for them. While in JobsDB, participants could complete a job application with three mouse clicks. As a result, it may
be concluded that the amount of mental effort required to complete task 4 differed as a function of service quality level and was correlated with perceptions of overall service quality, r ¼ .88, po.001. 3.4. Dimensions related to overall service quality To ensure that all questions in the Website Service Quality Questionnaire were internally consistent, reliability assessment was conducted using Cronbach’s alpha. The alpha value
ARTICLE IN PRESS 708
J.P.C. Tong et al. / International Journal of Industrial Ergonomics 35 (2005) 697–711
was 0.94, which is above 0.70 and is acceptable (Nunnally, 1978). Therefore, it can be concluded that the questionnaire was internally consistent. A factor analysis was conducted to determine the underlying components of the first 27 items on the Website Service Quality Questionnaire. Principal components analysis was conducted with varimax rotation and five components produced eigenvalues greater than 1. After rotation, the first component accounted for 40.04% of the total variance, the second component accounted for 11.58%, the third 6.52%, the fourth 6.15%, and the fifth 4.99%, for a combined 69.28% of variance explained by all five components. Table 12 presents the loadings for each component. Component number 1, referred to as general service quality, consisted of 19 of the 27 questions. 18 of the questions had positive loadings while 1 (support for different languages) had a negative loading. The second component consisted of 3 of the 27 questions. These questions all had positive loadings and addressed accuracy and efficiency. The third component consisted of 2 of the 27 questions. These questions all had positive loadings and addressed the interface. The fourth component consisted of 1 of the 27 questions. This question had a positive loading and addressed maneuvering speed. The fifth component consisted of 2 of the 27 questions. These questions all had positive loadings and addressed additional support. Table 13 shows the mean factor scores for the LSQ and HSQ websites along with the correlation between each component and overall service quality rating. Thus, only 3 of the components Table 13 Mean factor scores for HSQ and LSQ websites and correlation of factor score with overall service quality rating Component
Mean Mean r for HSQ for LSQ
P
General service quality Accuracy & efficiency Interface Maneuvering speed Additional support
.841 .346 .063 .036 .156
o.001 o.001 o.037 o.214 o.158
.841 .346 .063 .036 .156
.814 .374 .209 .125 .142
(general service quality, accuracy/efficiency, and interface) showed significant relationships with perceived overall service quality rating. In addition, it appears that for all components except additional support that the HSQ website received higher scores. Thus for general service quality, accuracy/efficiency, interface, and maneuvering speed, the HSQ website received higher scores, but for additional support, the LSQ website received a higher score, but the LSQ still received significantly lower overall service quality ratings.
4. Discussion 4.1. Performance time Based on the analytical results, it is believed that the total time spent on the whole job seeking process can impact the perceived electronic service quality of a recruitment website. The total time spent on a recruitment website includes both system response time and interface manipulation time. As a result, recruitment companies are recommended to keep track of web traffic. In addition, they should also pay attention to the time spent on every procedure of the online job seeking process when using the web user interface. According to Kalakota and Robinson (1999), customers count speed of service as a key reason for doing business with certain companies. This may also be true for job seekers. Inconvenience and poor electronic service make job seekers impatient. As a result, they may look for other recruitment websites. 4.2. Mental workload Based on the analytical results, it is believed that overall and task-specific mental effort required to complete tasks on a recruitment website show a significant relationship with the perceived overall service quality. As a result, in order to maintain a high level of service quality, recruitment websites should identify problems with and redesign web interfaces that demand much mental effort from job seekers. It is important for recruitment
ARTICLE IN PRESS J.P.C. Tong et al. / International Journal of Industrial Ergonomics 35 (2005) 697–711
companies to understand from their customers’ perspectives the most frustrating aspects of their relationships with the companies (Taylor and Terhune, 2001). Certainly, there is one exception where low mental workload does not mean high service quality. For example, some recruitment websites do not provide as many services as others. Therefore, less mental workload is required when manipulating the user interface on the website. As a result, low mental workload can mean low recruitment service quality. In conclusion, recruitment websites are recommended to provide web user interfaces that are simple to use as long as these interfaces can assist job seekers in the job seeking process. 4.3. Relationship between performance time and overall mental workload In the present study, a significant relationship was found to exist between the amount of time taken to complete all 4 tasks and the overall mental workload required to complete all tasks (r ¼ .62, po0.01). According to Xie and Salvendy (2000), performance time can be used as a performance measure for overall mental workload. Because we used a different measure of mental workload than the one reported by Xie and Salvendy (2000) and because performance time may be influenced by other factors (e.g., internet connection speed), it is unclear if one can be substituted for the other. 4.4. Dimensions of overall service quality The results of this study provide evidence that particular dimensions of service quality are significantly correlated with a person’s perception of overall service quality. However, the results also indicate that particular questions on the questionnaire may not be relevant for online recruitment websites when they attempt to evaluate their overall service quality. That is, maneuvering speed and additional support did not significantly correlate with overall service quality ratings. Thus, it may be more important for online recruitment companies to focus
709
their attention on the dimensions that were most highly correlated with overall service quality ratings, general service quality, accuracy/efficiency, and interface. 4.5. Limitations and future work The findings from Tasks 1–4 in this research may only be applicable to the four job seeking tasks performed in JobsDB and Strongnet because the experimental data was collected based on the web user interface design in both JobsDB and Strongnet. As each recruitment website has its own web user interface design, the result of this research may not be applicable to all job recruitment websites. As a result, additional research is necessary to verify the conclusions of this research for other websites and other e-service domains. In addition, the present research may be somewhat limited in that the usage of online recruitment sites may not be as time-critical as the usage of business-to-business sites. However, we still believe that this research may be very important for the numerous websites that offer a service which links its viewers to a particular business (e.g., recruitment sites as the ones evaluated in this research and shopping sites that may refer customers to particular businesses). If these sites are more difficult to use (i.e., require more time and mental effort) and are thus perceived to offer low service quality, viewers may be less likely to return to the site or refer their friends and family, which will lead to a diminished market for paying customers (e.g., employers or site sponsors).
Acknowledgements The authors would like to thank Miss Szeto Yee Man, Mr. Ng Pun Wai, and Mr. Or Ka Lun for their insightful comments and suggestions. Funding for this research was provided by the Direct Allocation Grant, 2001, DAG, HKUST, Hong Kong. F. Tsung’s work was supported by RGC Competitive Earmarked Research Grants HKUST6183/03E and HKUST6232/04E.
ARTICLE IN PRESS 710
Appendix A
J.P.C. Tong et al. / International Journal of Industrial Ergonomics 35 (2005) 697–711
ARTICLE IN PRESS J.P.C. Tong et al. / International Journal of Industrial Ergonomics 35 (2005) 697–711
References Duffy, V.G., Salvendy, G., 1998. Concurrent engineering diagnostic model integrating people, organization and technology. International Journal of Computer-Integrated Manufacturing 11 (5), 461–474. Duffy, V.G., Salvendy, G., 1999a. The impact of organizational ergonomics on work effectiveness with special reference to concurrent engineering in manufacturing industries. Ergonomics 42 (4), 614–637. Duffy, V.G., Salvendy, G., 1999b. Relating company performance to staff perceptions: the impact of concurrent engineering on time to market. International Journal of Production Research 37 (4), 821–834. Duffy, V., Danek, A., Salvendy, G., 1995. A predictive model for the successful integration of concurrent engineering with people and organizational factors: Based on data of 25 companies. International Journal of Human Factors in Manufacturing 5 (4), 429–445. Gehrke, D., Turban, E., 1999. Determinants of successful website design: Relative importance and recommendations for effectiveness. Proceedings of the 32nd Annual Hawaii International Conference on System Sciences. Hamilton, A., 1997. Avoid the #1 website sin: Slow loading pages. http://www4.zdnet.com/anchordesk/story/story_1244. html. Hancock, P.A., Meshkati, N., 1988. Human Mental Workload. Elsevier, Amsterdam. Hart, S.G., Staveland, L.E., 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In: Hancock, P.A., Meshkati, N. (Eds.), Human Mental Workload. Elsevier, Amsterdam, pp. 139–178. Janda, S., Trocchia, P., Gwinner, K., 2002. Consumer perceptions of Internet retail service quality. International Journal of Service Industry Management 13, 412–431. Kalakota, R., Robinson, M., 1999. E-Business: Roadmap for Success. Addison-Wesley, Reading, MA. Kuhn, P., Skuterud, M., 2000. Job search methods: Internet versus traditional. Monthly Labor Review 123 (10), 3–11. Liang, T.P., Lai, H.J., 2000. Electronic store design and consumer choice: An empirical study. Proceedings of the 33rd Annual Hawaii International Conference on System Sciences, pp. 2060–2069. Liljander, V., van Riel, A., Pura, M., 2002. Customer satisfaction with E-services—An online recruitment portal. In: Bruhn, M., Stauss, B. (Eds.), Yearbook of Services Management 2002—E-Services. Gabler Verlag. Lohse, G.L., Spiller, P., 1998. Electronic shopping. Communications of the ACM 41 (7), 81–87. Meshkati, N., 1988. Toward development of a cohesive model of workload. In: Hancock, P.A., Meshkati, N. (Eds.), Human Mental Workload. Elsevier, Amsterdam, pp. 305–310. Moray, N., 1988. Mental workload since 1979. International Review of Ergonomics 2, 123–150.
711
Nunnally, J.C., 1978. Psychometric theory, 2nd ed. McGrawHill, New York. Parasuraman, A., Zeithaml, V.A., Berry, L.L., 1985. A conceptual model of service quality and its implications for future research. Journal of Marketing 49 (Fall), 41–50. Parasuraman, A., Zeithaml, V.A., Berry, L.L., 1988. SERVQUAL: a multiple-item scale for measuring consumer perceptions of service quality. Journal of Retailing 64 (1), 12–40. Reid, G.B., Nygren, T.E., 1988. The Subjective Workload Assessment Technique: A scaling procedure for measuring mental workload. In: Hancock, P.A., Meshkati, N. (Eds.), Human Mental Workload. Elsevier, Amsterdam, pp. 185–214. Rust, R.T., Lemon, K.N., 2001. E-Service and the consumer. International Journal of Electronic Commerce 5 (3), 85–101. Sterne, J., 2000. Customer Service on the Internet: Building Relationships, Increasing Loyalty, and Staying Competitive. Wiley, New York. Taylor, D., Terhune, A.D., 2001. Doing E-Business: Strategies for Thriving in an Electronic Marketplace. Wiley, New York. United States Department of Labor Bureau of Labor Statistics, 2003. Employment situation summary. Retrieved May 8, 2003, from http://www.bls.gov/news.release/empsit. nr0.htm. Vidulich, M.A., Tsang, P.S., 1987. Absolute magnitude estimation and relative judgment approaches to subjective workload assessment. Proceedings of the Human Factors Society 31, 1057–1061. Wierwille, W.W., Casali, J.G., 1983. A validated rating scale for global mental workload measurement application. Proceedings of the Human Factors Society 27, 129–133. Xie, B., Salvendy, G., 2000. Prediction of mental workload in single and multiple tasks environments. International Journal of Cognitive Ergonomics 4 (3), 213–242. Zeithaml, V.A., Parasuraman, A., Malhotra, A., 2000. A conceptual framework for understanding e-service quality: Implications for future research and managerial practice, Marketing Science Institute. Report Summary No. 00–115. Zhang, P., von Dran, G., 2001. Expectations and rankings of website quality features: Results of two studies on user perceptions. Proceedings of the 34th Annual Hawaii International Conference on System Sciences, pp. 2323–2332. Zhang, P., von Dran, G.M., Small, R.V., Barcellos, S., 1999. Websites that satisfy users: A theoretical framework for web user interface design and evaluation. Proceedings of the 32nd Annual Hawaii International Conference on System Sciences. Zhang, P., von Dran, G.M., Small, R.V., Barcellos, S., 2000. A two factor theory for website design. Proceedings of the 33rd Annual Hawaii International Conference on System Sciences, pp. 1925–1934.
