506787
research-article2013
NVS43610.1177/0899764013506787Nonprofit and Voluntary Sector QuarterlyHuang et al.
Article
Training Effectiveness and Trainee Performance in a Voluntary Training Program: Are Trainees Really Motivated?
Nonprofit and Voluntary Sector Quarterly 2014, Vol. 43(6) 1095–1110 © The Author(s) 2013 Reprints and permissions: sagepub.com/journalsPermissions.nav DOI: 10.1177/0899764013506787 nvsq.sagepub.com
Yunchen Huang1, Lesley Strawderman1, Kari BabskiReeves1, Shaheen Ahmed1, and Arash Salehi1
Abstract Effective training programs are critical for successful employee performance. The same can be said for volunteer programs. Volunteers need to have the knowledge and skills necessary to fulfill the mission of the organization. In this study, we examine the impact of training methods and trainees’ demographic factors on the effectiveness of a training program for a voluntary program. Survey results (N = 5,727) indicate a very low percentage of participation (16.46%) in the voluntary program after training. Responses also indicate that content knowledge recall after training is affected by training year (the year receiving training), industrial sector, calling history, and training delivery method. Participants’ desires for retraining are also impacted by the type of training received, the industry sector, as well as past participation in the program. Recommendations for voluntary training programs based on study results and future research directions are presented. Keywords voluntary training program, training delivery method, training effectiveness, trainee performance, trainee demographics
Introduction A marked increase in volunteerism has been observed in recent years in the United States. According to the Bureau of Labor Statistics, 64.3 million people performed 1Mississippi
State University, USA
Corresponding Author: Yunchen Huang, Department of Industrial and Systems Engineering, Mississippi State University, P.O. Box 9542, MS 39762, USA. Email:
[email protected]
Downloaded from nvs.sagepub.com at MISSISSIPPI STATE UNIV LIBRARIES on December 18, 2014
1096
Nonprofit and Voluntary Sector Quarterly 43(6)
volunteer work in 2011, compared with 60.8 million people in 2007 (U.S. Department of Labor, 2011). The issue of providing effective and efficient training for the large population of volunteers is critical. The safety of the volunteers and the success of the voluntary program are closely bound with the implementation of appropriate volunteer training. Therefore, it is important that researchers and practitioners have a better understanding of the relationship between the design of a voluntary training program and the effectiveness of the training program. Training program effectiveness is dependent on training delivery method (Anderson et al., 1996; Boyle, Anderson, & Newlands, 1994; Doherty-Sneddon et al., 1997; Hale, 1998; Meline, 1976; Raphael & Wagner, 1974; Veinott, Olson, Olson, & Fu, 1999) and trainee attributes and characteristics (Clark, 1983, 1994; Noe, 1986; Sitzmann, Kraiger, Stewart, & Wisher, 2006). Although many researchers have examined training efficiency and effectiveness, most studies were based on simple problem-solving tasks (Hale, 1998; Meline, 1976) or procedural work training within a profitable enterprise (Barrett & O’Connell, 2001). Few studies exist that focus on the effectiveness of nonprofit, voluntary training program effectiveness. Worthington (2008) studied a hospice volunteer training program that stressed the importance of communication skills training. Gelkopf, Ryan, Cotton, and Berger (2008) assessed a voluntary training program that trained volunteers to help Sri Lankan tsunami-survivor children, while Meier, Singletary, and Hill (2012) investigated the effectiveness of a community development training program in developing volunteers’ leadership skills. Dailey (1986) also stated that the effectiveness of nonprofit voluntary program depends on organizational commitment among volunteers. This article describes a study in which trainees’ participation and performance in an existing voluntary training program were evaluated, emphasizing the procedural knowledge gained by trainees. Effects of training delivery method, industry sector, and participation history were investigated.
Training Delivery Methods Many studies have been conducted to investigate the efficacy of various training delivery methods (e.g., Arthur, Bennett, Edens, & Bell, 2003; Buch & Bartley, 2002; Salehi, Strawderman, Huang, Ahmed, & Babski-Reeves, 2009; Wexley & Latham, 2002). Common delivery methods include classroom (face-to-face or lecture style), online, video, and audio training. Traditional classroom training is considered an effective delivery method because it allows interaction between the trainees and trainer. Studies have shown that a critical factor influencing skill transferability between training and the job is the extent to which trainees receive the opportunity for practice and constructive feedback (Goldstein, 1993; Latham & Saari, 1979; Wexley & Latham, 1991). In classroom training, interactive activities are often used to engage trainees and enable real-time feedback for the trainees and trainer. These activities are considered critical for a quality learning experience (Wagner, 1998). Buch and Bartley (2002) also observed that most trainees preferred the traditional classroom training to other training delivery methods. Regardless, classroom training has multiple disadvantages. For example, Huang (1997) concluded that in corporate training situations, classroom training will bring
Downloaded from nvs.sagepub.com at MISSISSIPPI STATE UNIV LIBRARIES on December 18, 2014
1097
Huang et al.
about additional costs associated with transportation and accommodations and lost wages during travel time. In addition, classroom trainees may lose the chance of accessing resourceful online materials and multimedia information (Taylor, 2002). In recent years, with the advance of computer technology and Internet coverage, online training has increasingly been adopted as an effective training delivery method. Advantages for online training include consistency in training, the potential for real-time training delivery, reduced information overload, and the ability to tailor the training program based on individual preferences (Goldstein & Ford, 2002; Welsh, Wanberg, Brown, & Simmering, 2003). Taylor (2002) summarized the advantages for online training as follows: eases instructor customization, enhances virtual teamwork, eases access of documents or materials, and removes the obstacle of time and space. Taylor also presents multiple disadvantages of online training, including limited ability to provide immediate response and high effort from the instructor to monitor each trainee’s progress. Video-based training has been shown to be useful in sports contexts (Scott, Scott, & Howe, 1998; Walker & Fisk, 1995). It may also be used either to provide feedback to the trainee or to provide a model of the correct or desired performance (Franks & Miller, 1991). However, video training is regarded as a passive, rather than active, training method. Without appropriate connection with text material and class participation, video training has been found to result in decreased learning and retention (Mykityshyn, Fisk, & Rogers, 2002) and has many of the same drawbacks as classroom training. A review of the literature revealed inconsistencies in study findings for the “best” or most effective training delivery method. For example, when comparing audio training and audio combined with video training, some studies found that audio combined with video training can lead to superior trainee performance (Boyle et al., 1994; McLaughlin, Rogers, & Fisk, 2002; Veinott et al., 1999). However, other studies showed that audio training alone does not yield significant difference in trainees’ performance compared with audio combined with video training. Hale (1998) and Meline (1976) even found that audio combined with video training actually resulted in inferior trainee performance compared with audio training alone. When comparing online and traditional classroom training, Schmeeckle (2003) found both methods to be equally effective; though online training is more efficient than classroom training. In addition, no difference in students’ learning, motivation, and attitude were found across these two delivery methods. Conversely, Batcheller, Brennan, Braslow, Urrutia, and Kaye (2000) and Todd et al. (1998) found video training to be superior to traditional classroom training in both studies of cardiopulmonary resuscitation training. The inconsistency in the judgment of superiority for various training delivery methods opens the door for further evaluation. Few studies, if any, have compared the four common training delivery methods.
Trainees’ Attributes The literature notes that trainees’ attributes and attitudes influence training effectiveness in nonvoluntary training scenarios (Clark, 1983, 1994; Noe, 1986; Sitzmann
Downloaded from nvs.sagepub.com at MISSISSIPPI STATE UNIV LIBRARIES on December 18, 2014
1098
Nonprofit and Voluntary Sector Quarterly 43(6)
et al., 2006). Trainees’ attributes include trainee’s characteristics, motivation, learning styles, and so on. However, it is unknown how trainee attributes and attitudes affect training effectiveness in voluntary training programs, which may pose many unique challenges ranging from participant recruitment to training development to training presentation. Wilson (2000) mentioned that the trainees’ attitudes and motivations may be a factor influencing volunteer training effectiveness. Education research and practice have demonstrated that learning can be enhanced when the delivery process accommodates the various trainee learning styles (Buch & Sena, 2001; Kolb, 1984). It is conceivable that voluntary program trainees require higher levels of motivation to perform well than in training programs that yield direct incentives to trainees (e.g., higher salary, chance of promotion, job opportunity). However, little empirical data are available to analyze the impact of trainees’ demographics on performance. Therefore, in this article, we quantified trainee demographics and training delivery method on training effectiveness.
Research Questions Training effectiveness can be measured by assessing several aspects within the training program, such as training design, trainee’s knowledge acquisition or retention, and changes in a trainee’s performance and behavior. In this article, we investigated participation in the Highway Watch (HWW®) program, which was originally administered by the American Truck Associations (ATA), and explore reasons behind participation or lack thereof. Specifically, we question whether training delivery method or trainee demographics impact participation. Acquisition represents trainee learning after the training process. For the voluntary training program evaluated here, the knowledge that is being transferred is mostly procedural. Therefore, trainees are supposed to gain knowledge of what to do as part of participating in the voluntary program. However, trainees’ procedural knowledge acquisition has not been quantified. Thus, the second research question is did trainees retain the information conveyed in training, and what factors impacted knowledge acquisition? Factors considered in this research were trainee demographics, training delivery method, and time since training. The voluntary training program under evaluation currently offers one-time training, though it is expected that participants will be actively involved for prolonged time periods. Research has shown that information is lost over time (McGeoch, 1932). Therefore, the third research question is do trainees foresee the need for refresher or additional training? These research questions correspond well with Kirkpatrick’s (1967) hierarchical model of training outcomes that incorporate the trainee’s reaction, knowledge acquisition, behavior change, and individual’s organizational improvement. Together, the answers to these research questions will provide insights into the effectiveness of the current voluntary training program and provide suggestive explanations of current training problems.
Downloaded from nvs.sagepub.com at MISSISSIPPI STATE UNIV LIBRARIES on December 18, 2014
1099
Huang et al.
Method The Training Program A nonprofit voluntary training program using four different training delivery methods was studied. The HWW® program was founded in 1998 as a motor carrier safety program for professional truck drivers. There were 798,269 trainees participating in the program at the end of 2007. Most of the trainees are truck drivers in transportation positions. The program goal was to improve the safety and security of U.S. roadways by training transportation professionals to be alert of suspicious or dangerous activities and conditions. Program participants are trained to report any suspicious activities or dangerous conditions to the program call center using a toll-free telephone number or 911 depending on the severity of the situation. Participants were recruited from transportation-related companies and organizations across the United States. Participants completed training to raise their awareness of highway-related public security and safety-related issues, as well as to learn how to use the program. Participants were instructed to submit reports to the toll-free hotline and identify themselves using their ID number, and the call center personnel recorded the incident information. If the incident required immediate attention of first responders, trainees were instructed to contact 911 prior to calling the program hotline. Those completing the training were issued a unique program identification number, an ID card, and a certificate of completion. Participation in the program was voluntary, and participants were not compensated for participating in training or for submitting reports to the call center. The four training delivery methods used in this program were video, online, audio, and face-to-face (classroom) training. Each person participated in one type of training. Video training was a recorded class session performed by a master trainer to a classroom of trainees. In the video training, short scenarios for clarification of complicated situations as well as short movies and PowerPoint slides were provided. Video training was available through VHS cassette or DVD and lasted approximately 70 min. Online training contained the recorded classroom training session used in video training and was accessed via the Internet. Therefore, program participants were required to be familiar with the Internet and have access to high-speed Internet connection to participate in online training. In audio training, a formal recorded lecture, lasting 40 min, was provided for the trainees. The audio training used a number of short conversations to simulate real-life situations for the trainees. The audio training was available on CD and cassette tape. Classroom training consisted of a certified master trainer who presented the training material in a formal classroom environment. PowerPoint slides and short movies (similar to those used in video training) were presented, though the trainer had the flexibility to tailor the training session to the needs of the trainees or to share their unique experiences to further illustrate and clarify concepts. Classroom training typically lasted 60 min.
Downloaded from nvs.sagepub.com at MISSISSIPPI STATE UNIV LIBRARIES on December 18, 2014
1100
Nonprofit and Voluntary Sector Quarterly 43(6)
Table 1. Member Survey Content. Section
Content
Number of questions
Demographics
Employment type, industry type, time of training, reason for participation, hours and miles driving in week, frequency of using Internet and e-mail, being part of union, and so on. Previous participation Previous time of participation with the program if any, participation in similar program, reasons for not participating, previous participation feedback Scenario Scenario-based questions spanning a range of complexity Retraining Perception of trainees about the need to receive refreshing training Training delivery method Choices of audio, video, live classroom, online training Other Adequacy of duration of calls, participation in program comparing with the time of training, having and remembering the program’s ID, suggestions for the program Total
15
17
4 1 1 5
43
Data Collection A 43-question survey was designed to gather information from program members regarding their demographics, participation information, understanding of different scenarios, perception toward retraining, and other basic training information. A summary of question content is shown in Table 1. At the time that the survey was created, 708,948 people had completed program training. As this study was part of a larger research project, it was decided that the survey would be distributed through e-mail, reducing the participant pool to the 242,581 individuals providing an e-mail address, of which 100,000 of these were randomly selected to receive the survey. Of the 100,000 surveys mailed out, 7,199 were returned. After cleaning the data, 5,727 out of 7,199 were possible to use for descriptive statistical analyses. However, all 5,727 participants did not respond to every single question; many questions were unanswered. The nonresponses to the survey questions and the questions used for specific hypotheses reduce the overall sample size for inferential statistical analyses to 2,199. Although the sample size was reduced, there was enough data to complete the analyses (Dillman, Smyth, & Christian, 2009).
Downloaded from nvs.sagepub.com at MISSISSIPPI STATE UNIV LIBRARIES on December 18, 2014
1101
Huang et al.
Dependent Variables Two dependent variables were analyzed: knowledge retention and retraining perception. Trainees’ performance score on the four scenario-based questions was used as a measurement of their retention of the training content. Participants were instructed to identify the appropriate actions according to written scenario descriptions. Possible responses included “take no action,” “call 911,” “call 911 then HWW®,” “call HWW®,” and “not sure.” Responses were coded “1” for correct responses and “0” for incorrect responses. Participants’ performance scores were obtained by averaging the four scenario-based questions scores. A question included in the survey was used to collect participants’ perceptions of the need for retraining. Responses were coded either “0” (don’t want retraining) or “1” (would like retraining).
Independent Variables Seven questions relating to trainee demographics, behaviors, and perceptions were the independent variables. Specifically, three levels of industrial sector (self-employed, company employed, and government employed), three levels of training delivery method (face-to-face training, web-based training, and video training), four levels of calling history (called HWW® only, called 911 only, called both, and called none), and two levels of desire for retraining (want retraining and do not want retraining) were evaluated. Six levels of training year were incorporated ranging from 2003 to 2008. Finally, seven levels of time/mileage on the road were examined ranging from 0 hr per week to more than 70 hr per week.
Data Analysis To investigate low program participation, we presented descriptive statistics obtained from the questionnaires. One-way ANOVA was conducted on the relationship between trainees’ performance scores and various independent variables identified above (Research Questions 1 and 2). A general linear regression line was fit for the significant factors identified in the ANOVA analysis. Finally, chi-square tests were conducted to investigate the relationship between trainees’ perception for retraining and various factors (Research Question 3).
Results Descriptive Statistics Participation in the program was generally low. Among all respondents, only 78 trainees (1.36%) reported calling HWW® even once. Only 785 trainees (13.7%) reported calling 911. When asked, “when was the last time you called Highway Watch® Call Center (HWW®, n = 78),” 5% (n = 4) of the respondents said that they called HWW® within last 2 months, 22% (n = 17) said that the last time they called HWW® was
Downloaded from nvs.sagepub.com at MISSISSIPPI STATE UNIV LIBRARIES on December 18, 2014
1102
Nonprofit and Voluntary Sector Quarterly 43(6)
between 2 and 6 months ago, and 71% (n = 55) said it was more than 6 months ago. The remaining 2% (n = 2) reported that they were not sure. When asked, “when was the last time you called 911 (n = 785),” 25% (n = 196) said that they called 911 within last 2 months, 36% (n = 283) said the last time they called 911 was between 2 and 6 months ago, and 35% (n = 275) said it was more than 6 months ago. The remaining 4% (n = 31) reported that they were not sure. When asked, “how often do you report an incident to HWW® (n = 110),” 4% (n = 4) report incidents around once per month, 67% (n = 74) report incidents around once per year, and 17% (n = 19) did not make calls. Another 12% (n = 13) reported that they were not sure. When asked, “how often do you report an incident to 911 (n = 779),” only 1% (n = 8) of respondents report around once per week, 7% (n = 54) report incidents around once per month, 77% (n = 600) report incidents around once per year, and 2% (n = 16) did not make calls. Another 13% (n = 101) reported that they were not sure. When asked, “how often do you report an incident to 911 and then follow up with another call to HWW®” (n = 43), which is stressed in training, 33% (n = 14) of respondents failed to follow up a call with HWW® after calling 911. Another 33% (n = 14) made these calls about once per year, 23% (n = 10) made these calls more than once per year, and 11% (n = 5) reported they were not sure. For those participants who did not make any calls (n = 14), 57% (n = 8) said they did not call the call center after calling 911 because they did not think they needed to, 21% (n = 3) said they are not sure whether they are supposed to do so, and 14% (n = 2) said they did not have the phone number. Another 8% (n = 1) reported that they were not sure. As this is contrary to training, this procedural knowledge was not retained by these program participants. Further questions were asked to investigate the reasons why participants never called HWW® or 911. Most respondents selected, “I did not see anything to report,” for HWW® (95%) and 911 (94%). Other minor reasons include “I am supposed to call my employer/dispatcher and they make the report,” “was not sure if incident was important enough to call,” “did not have HWW® number,” or other reasons (all less than 1% respondents each). When asked whether they had relayed life-threatening or suspicious events in other ways (n = 4,674), the majority of the respondents (82%) had relayed the incidents to their employer. Other responses included “my dispatcher” (8%), “I put the word out on my CB” (5%), “made an entry in my log book” (3%), and “talked to drivers at my next stop” (2%).
Inferential Statistics Altogether, 2,199 usable questionnaires were available for statistical analysis. Filtering criterion included data completeness, minor sample exclusion (less than 5% of the total population), and exclusion of “no comments” responses. Of the 2,199 responses, the average number of years since training was 2.78 years, with a standard deviation of 1.11 years. The average performance score (average of score for the four scenariobased questions) was 0.53 (or 53%, equating to two of four scenarios answered correctly), with a standard deviation of 0.24. The categorical breakdown of participants based on various demographic factors is provided in Table 2.
Downloaded from nvs.sagepub.com at MISSISSIPPI STATE UNIV LIBRARIES on December 18, 2014
1103
Huang et al. Table 2. Demographic Responses. Demographic series Industrial sector Self Company Government Total Calling history HWW® 911 Both None Total Training delivery method Face-to-face web-based Web-based Video Total Retraining wanted Yes Yes, but no time No Total
n
%
294 1,345 560 2,199
13.37 61.16 25.47 100
32 307 23 1,837 2,199
1.46 13.96 1.04 83.54 100
956 747 496 2,199
43.47 33.97 22.56 100
447 78 1,674 2,199
20.33 3.55 76.12 100
Note. HWW = Highway Watch.
Training year, F(5, 2193) = 2.31, p = .042; industrial sector, F(2, 2196) = 5.89, p = .0028; calling history, F(3, 2195) = 4.51, p = .0037; and training delivery method, F(2, 2196) = 3.29, p = .0374, were found to significantly affect performance on scenariobased questions. Post hoc analysis was unable to identify differences between training years due to large variances. Government employed trainees were found to have significantly lower performance scores than self-employed or company employed trainees. In addition, trainees with a history of calling 911 scored poorer than those who did not. Finally, trainees completing web-based training performed significantly better than those participating in face-to-face training. As employer support and training protocol is critical to the success of this training program, it is critical to investigate the mediating effect of industry section. Therefore, the interaction effect between industry sector and training year, calling history, and training delivery method is examined. Results show that only the interaction of industry sector and calling history have a marginally significant interaction effect, F(6, 2187) = 1.85, p = .0861. The remaining interaction effects were not significant. Post hoc comparison shows that for the trainees who never called, company trainees had significantly higher average performance scores than government employed trainees.
Downloaded from nvs.sagepub.com at MISSISSIPPI STATE UNIV LIBRARIES on December 18, 2014
1104
Nonprofit and Voluntary Sector Quarterly 43(6)
Table 3. Parameter Estimates for the Regression Model. Term Intercept x1: Training year x2: Industry (company) x3: Industry (government) x4: Training delivery method (live) x5: Training delivery method (video) x6: Calling history (911) x7: Calling history (both) x8: Calling history (HWW®)
Estimate
Standard error
t ratio
Probability > |t|
0.543 −0.009 0.013 −0.024 −0.005 −0.004 −0.036 −0.029 0.053
0.022 0.005 0.007 0.009 0.007 0.008 0.019 0.039 0.034
24.47 −1.89 1.75 −2.83 −0.71 −0.52 −1.85 −0.75 1.54
