Research in Food Science Education
Development of an Evaluation Tool for Online Food Safety Training Programs Jack A. Neal Jr., Cheryl A. Murphy, Philip G. Crandall, Corliss A. O’Bryan, Elizabeth Keifer, and Steven C. Ricke
Abstract: The objective of this study was to provide the person in charge and food safety instructors an assessment tool to help characterize, identify strengths and weaknesses, determine the completeness of the knowledge gained by the employee, and evaluate the level of content presentation and usability of current retail food safety training platforms. An existing evaluation tool was modified to assess online food safety training programs that contained 5 areas of emphasis: content and instruction, learners, job transfer, design and packaging, and operation. Participation included watching 3 food safety training modules and afterwards evaluate what degree the module content met objectives. More than 85% of the participants responded positively to the scale-based items of the course evaluation. Items addressing the content and instruction received positive responses. The assessment tool helped panelists to objectively evaluate an online food safety training program that provided numeric data for a valid comparison.
Introduction Active managerial control is the term used by the U.S. Food and Drug Administration (FDA) to describe the food industry’s responsibility for the development and implementation of food safety management systems to reduce the risk of foodborne illnesses. The goal of active management control is to incorporate specific actions and procedures by food service managers into their operations to attain control over foodborne illness risk factors (FDA 2006). With the implementation of active managerial control, more emphasis has been placed on training food service workers by the person in charge (PIC). A similar trend has been observed in the United Kingdom (Seaman and Eves 2010). The idea behind this emphasis is that, while many are responsible for food safety along the farm to fork continuum, the ultimate responsibility for food safety at the retail level lies with the retail and food service operators and their ability to develop and maintain effective food safety management systems (FDA 2006, 2010). Therefore, the FDA Food Code 2009 Chapter 2 states that the PIC must be able to demonstrate to the regulatory authority upon request knowledge of foodborne disease prevention, application of the Hazard Analysis Critical Control Points (HACCP) principles and the requirements of the food code (FDA 2010). There are many tools that retail food operators can use to develop their food safety management systems. Elements of an MS 20100853 Submitted 7/28/2010, Accepted 9/19/2010. Author Neal is with Conrad N. Hilton College of Hotel and Restaurant Management, Univ. of Houston, Houston, TX 77204, U.S.A. Authors Murphy and Keifer are with Dept. of Curriculum and Instruction, Univ. of Arkansas, Fayetteville, AR 72704, U.S.A. Authors Crandall, O’Bryan, and Ricke are with Dept. of Food Science and Center for Food Safety, Univ. of Arkansas, Fayetteville, AR 72704, U.S.A. Direct inquiries to author Neal (E-mail:
[email protected]).
c 2010 Institute of Food Technologists® doi: 10.1111/j.1541-4329.2010.00112.x
effective food safety system include certified food protection managers who have passed a test that is part of an accredited program, standardized operating procedures, standardized recipes that include the identification of critical control points (CCPs) as part of a HACCP plan, as well as monitoring and record keeping protocols, purchasing specifications, proper equipment and facility design, an employee health policy for restricting or excluding ill employees, and manager and employee training (FDA 2006). Creating a food safety management system takes time to develop and implement and there are several methods that the managers can take. Therefore, the PIC of food safety training must decide whether to make or buy the system. Many large food service operations have a training department and prefer to create their own food safety training material. One popular option is the use of e-learning technologies, which have been used to develop more effective training material with enhanced learning efficiency (Bosco 1986; Kulik and others 1994; Parlangeli and others 1999; O’Bryan and others 2010). U.S. Census data from 2003 indicated that approximately 83% of adult Americans age 18 y or older used the internet at home (U.S. Census Bureau 2010). Convenience, efficiency, and affordability are some of the main features attracting participants to use online education courses (Santerre 2005). While there are many benefits of using e-learning, e-learning designers must understand their target audience and how the learners will use the training tools created (Theofanos and Redish 2003). One key element to consider when creating online material is usability testing that evaluates the interactions between the user and the product being tested. O’Bryan and others (2010) described how user-centered design combined with usability testing to provide successful e-learning products and improving user satisfaction. Food service operations that are too small to have their own trainers or do not want to create their
Vol. 10, 2011 r Journal of Food Science Education 9
Development of an evaluation tool . . . own material may choose to purchase existing food safety training material. There are multiple online food safety training programs available to food service operators that include ServSafe, AIB Intl., Training Achievement Programs (TAPs Series), Alchemy systems, and FMI Super Safe Mark Program. Each one of these programs has unique features that may or may not meet the needs of the operation. Despite the popularity of food safety training, it does not always translate into positive food-handling practices (Riben 1994; Powell and others 1997; Clayton and others 2002; Green and others 2005). What is needed is a system that not only educates the employee but also changes their behavior. Therefore, before purchasing food safety software, food service managers need to identify what needs to be included in a food safety training program. Seaman (2010) suggests using the Food Hygiene Training Model, which includes evaluation stages, managerial components, and overall performance measures. These components consider the planning and training of the program, managerial support needed to determine the type of training and support needed to facilitate it, and the overall performance measures needed to insure that training transferred into changing behavior. One of the challenges to having multiple online food safety training programs available is to evaluate the software to determine if it meets the needs of the individual company and if it is instructionally sound. Betera (1990) identified the need for examination of service provision and program evaluation to ensure credibility and relevance in a study on the planning and implementation of workplace health promotion. The key to success is finding a tool that can be used to evaluate multiple food safety training programs consistently. Pisik (1997) recommends that the instrument should emphasize instructional soundness and include numeric data for comparison. While the evaluation tool should be thorough, it must also take into consideration the manager’s time to complete it. Once the data has been analyzed, objective information can be obtained and save time and money. The objective of this study was to provide the PIC and food safety instructors an assessment tool to help characterize, identify strengths and weaknesses, determine the completeness of the knowledge gained by the employee, evaluate the level of content presentation and usability of current retail food safety training platforms.
Methods Measures By modifying the online evaluation form developed by Pisik (1997), an online evaluation tool was created that contained 5 areas of emphasis: content and instruction, learners, job transfer, design and packaging, and operation. These 5 areas included 68 statements used to judge an online food safety training program. The evaluation tool contained 5 general demographic questions. The 1st measure, content and instruction, contained a series of statements on the objectives, content, information method, instructional strategy, graphics, animation, diagrams, audio and video quality, examples, practice exercises, level of feedback, lesson tests, and available help. The 2nd measure contained a series of statements that examined the training module’s ability to be easily used and transferred to the user’s work environment. The 3rd module contained a set of 8 statements that examined the design and packaging of the course material. The purpose for this series of questions is to help PIC compare multiple online training modules by assessing the layout and organization of the course 10 Journal of Food Science Education r Vol. 10, 2011
material. Design and packaging (the 4th measure) consisted of 6 statements concerning the layout, grammar, organization of the training material. Lastly, the 5th measure looked at the operation of the software such as ease of navigation and clarity of instructions. To ensure validity and relevance of the items included, the questionnaire was piloted (n = 10) prior to the commencement of data collection and a number of adjustments were made.
Participants The participants included 37 graduate students majoring in hotel and restaurant management (37% men and 63% women) at an urban university. The majority of the participants (63%) held a part- or full-time job in the hospitality industry. About 35% identified their ethnicity/race as Caucasian, 49% as Asian, and 16% as Hispanic. Procedure Participation included watching 3 food safety training modules available online at http://www.imalert.com.au/foodsafety/ southburnett/southburnett-food-safety.php. The “I’m Alert” online and interactive food safety training program developed by Environmental Health Australia was chosen based on the materials covered, its interactive capabilities, and price. The 3 modules selected included: health of the person who handles food, hygiene of food handlers, and temperature-measuring devises. These topics were chosen because these behaviors contribute the most to foodborne illness (FDA 2000, 2004). Respondents were asked to indicate what degree the module content met objectives (No, Somewhat, Yes, or Not Applicable).
Results Table 1 shows the demographic distribution of participants by gender, race/ethnicity, age, education, and work experience. The majority of participants were female ethnic minorities. More than 85% of the participants responded positively to the scale-based items of the course evaluation (Table 2). Items addressing the content and instruction received positive responses. For example, 90% of participants ranked the content and instruction as good or higher (excellent or very good). More specifically, 93% responded that the course objectives were clear and comprehensive and 95% felt that the examples were clear, logical, accurate, and relevant. Participants did not respond as positively to the learning and job transfer sections (mean positive response of 53% and 67%, respectively) mainly because they were not able to get a summary of their Table 1–Demographic distribution for participants (n = 37). Variable Gender Ethnicity/race Age
Education Years in profession 0 1 to 3 4 to 6 7 to 9 10+
Level Male Female Caucasian Hispanic Asian 19 to 28 29 to 35 36 to 45 46 to 55 Over 56 Bachelor’s degree Master’s degree
Frequency (n) 14 24 13 6 18 27 8 0 2 0 15 23
% 37 63 35 16 49 73 22 0 5 0 39.5 60.5
14 15 4 2 3
37 39.5 10.5 5 8
Available on-line through ift.org
Development of an evaluation tool . . . Table 2–Responses to food safety online training evaluation form (n = 30). Yes Scale-based item Content and instruction Introduction and/or course overview properly prepares learner for training. Course objectives are clear and comprehensive. Content is complete and thorough. The practice exercises and feedback are realistic and accurate. The instruction applies to different populations. Learners Learners are engaged in activities. Learners are able to get a summary of their performance. Job transfer The instruction facilitates the learner’s acquisition of the prescribed knowledge and skills. Information is presented in a format/manner that can be easily used and transferred to the learner’s work environment. Design and packaging The layout (screen/paper-based) is clear, clean, and well organized. Course materials are well organized for ease of use. Operation The program is easy to load and instruction is clear. On screen instructions are simple, clear, and concise. The system will function without the need for an instructor.
N
%
N
25
83.3
4
28 22 16 19
93.3 73.3 53.3 63.3
24 13
No
N/A
N
%
N
%
13.3
1
3
0
0
2 6 12 11
6.7 20 40 36.7
0 2 1 0
0 6.7 3.3 0
0 0 1 0
0 0 3.3 0
80 43.3
5 4
16.7 13.3
1 8
3.3 26.7
0 5
0 16.7
27
90
3
10
0
0
0
0
23
76.7
5
16.7
2
6.7
0
0
26 27
86.7 90
2 2
6.7 6.7
2 0
6.7 0
0 1
0 3.3
28 29 25
93.3 96.7 83.3
1 1 5
33 3.3 16.7
1 0 0
3.3 0 0
0 0 0
0 0 0
performance. In addition, participants categorized the design and packaging as well as operation positively (mean positive response of 97% and 90%, respectively). Mean time spent completing the survey tool was 15 min. In response to the open-ended questions on the evaluation, written responses from participants indicated that the evaluation tool statements helped them to identify key strengths and weaknesses of the modules. For example, panelists stated that the modules did not distinguish between the unit and lesson objectives. Participants liked the overall effectiveness of the modules but many expressed that they thought the actual modules were not long enough and that more material should be covered within the allotted subject. For the Learner component of the survey, the participants stated that they did not have a chance to bypass a unit of a lesson via a test that accurately measured their knowledge of the subject. Likewise, concerning job transfer, panelists affirmed that learners would clearly be able to use the information learned to perform at a minimum level when back on the job. Technology issues mentioned focused on not being able to skip onscreen instructions if the information was already known and the inability to preview previous frames of information.
Discussion and Conclusions The evaluation of the assessment tool to help identify strengths and weaknesses and determine the completeness of the knowledge gained by the employee offers useful information to the area of food safety training and indicates that online food safety training programs are an effective method for delivering training material for food service managers seeking to educate their employees. Because there is an increasing emphasis on active managerial control, there is a clear need for food safety training programs that managers can use in their operations. Given the time constrains of food service operators, managers, and PICs, online content delivery has proven to be effective. An online evaluation tool could potentially help managers compare multiple food safety programs. Ideally, multiple decision makers within an organization could evaluate the online training programs and produce similar results giving them useful information to identify differences between
Available on-line through ift.org
Somewhat %
programs. Therefore managers could make more informed decisions prior to purchasing the programs. The opportunity to take an online food safety training program online at the employee’s own pace is a benefit for companies to offer. Santerre (2005) stated that individuals who work full-time like the option of internet-based training. Another benefit for food service operations offering internet-based training to employees is the frequency in which the modules can be offered. With turnover rates estimated at 200% in some markets, being able to offer employees food safety training when needed can eliminate gaps in training and can also be offered as continuing education for existing employees. In our study, panelists were predominantly female ethnic minorities. What is important about this demographic group is that they represent the largest growing segment of the U.S. population. Despite language barriers, millions of non-English speaking immigrants are entering the U.S. labor force (Madera and others 2011). Currently, the foodservice industry is one of the largest employers of non-English speaking individuals in the United States (National Restaurant Association 2006). Data show that 26% of employees in the foodservice industry speak a language other than English at home (National Restaurant Association 2006). Future studies may include the evaluation of online food safety training programs offered in multiple languages. While this evaluation tool is a good start, there are several limitations with the first one being the variability within panelists. While many panelists had positive responses to items addressing the content and instruction, there were inconsistent responses to multiple statements within the survey. For example, while some panelists reported that the module was very effective while others commented that the modules were not comprehensive or were too basic. In addition, panelists mentioned that the exercises did not cover enough of the material presented within the modules. While this feedback is beneficial, it does indicate that this evaluation tool may not be strong enough or detailed enough to reduce the amount of variability among the panelists. Additional limitations of this study include the sample size and the fact that only 1 program was evaluated; however, of the 37 participants, 63% Vol. 10, 2011 r Journal of Food Science Education 11
Development of an evaluation tool . . . currently held jobs in the hospitality industry. Based on the feedback from the respondents, the researchers are planning to conduct a larger study comparing 4 programs and will include members of the foodservice industry. Ideally, an evaluation tool can be created so that multiple people within an organization can evaluate various online training programs and have similar results. Evaluating online food safety training programs to determine the usefulness and if it is instructionally sound are important parts of analyzing online training courses. Before investing time and money into a program, it is important to effectively and efficiently evaluate the training material to determine if it meets the needs of the company. Developing a tool that can be used to evaluating the multiple programs currently available is a preliminary step. The evaluation instrument should emphasize the content instruction, learners, job transfer, design and packaging, and overall operation of the modules and include numerical data for valid, objective comparisons. The instrument needs to be thorough while at the same time, consider the time commitment needed from the manager to review all of the programs. Once the data have been properly analyzed, the food service operator should be able to gain the necessary information to meet their needs while saving time.
References Betera RL. 1990. Planning and implementing health promotion in the workplace: a cast study of the Du Pont company experience. Health Edu Behav 17:307–27. Bosco J. 1986. An analysis of evaluation of interactive video. Edu Technol 25:7–16. Clayton DA, Griffith CJ, Price P, Peters AC. 2002. Food handlers’ beliefs and self-reported practices. Int J Environ Health Res 12:25–39. Green LR, Selman C. 2005. Factors impacting food workers’ and managers’ safe food preparation practices: a qualitative study. Food Prot Trends 25:981–90. Kulik CC, Kulik JA, Shwalb BJ. 1994. The effectiveness of computer applications: a meta-analysis. J Res Comput Edu 27:48–61. Madera JM, Neal JA, Dawson M. 2011. Strategies for diversity training: focusing on empathy in the workplace. J Hospitality Tourism Res. Forthcoming. National Restaurant Association. 2006. Restaurant industry facts. Available from: http://www.restaurant.org/research/ind_glance.cfm. Accessed Jul 13, 2010. O’Bryan CA, Johnson DM, Shores-Ellis KD, Crandall PG, Marcy JA, Seideman SC, Ricke SC. 2010. Designing an affordable usability test for e-learning modules. J Food Sci Ed 9:6–10.
12 Journal of Food Science Education r Vol. 10, 2011
Parlangeli O, Marchigiani E, Bagnara S. 1999. Multimedia systems in distance education: effects of usability on learning. Interact Comput 12:37–49. Pisik GB. 1997. Is this course instructionally sound? A guide to evaluating online training courses. Ed Tech 7:50–9. Powell SC, Attwell RW, Massey SJ. 1997. The impact of training on knowledge and standards of food hygiene—a pilot study. Int J Environ Health Res 7:329–34. Rubin J. 1994. Handbook of usability testing: how to plan, design, and conduct effective tests. New York: John Wiley & Sons Inc. 330 p. Santerre CR. 2005. X-train: teaching professionals remotely. J Nutr 135:1248–52. Seaman P. 2010. Food hygiene training: introducing the food hygiene training model. Food Control 21:381–7. Seaman P, Eves A. 2010. Efficacy of the theory of planned behavior model in predicting safe food handling practices. Food Control 21:983–7. Theofanos MF, Redish J. 2003. Bridging the gap between accessibility and usability. Interactions 10:36–51. U.S. Census Bureau. 2010. Current population survey report. Available from: http://www.census.gov/population/www/socdemo/computer/ 2003.html. Accessed Jul 8, 2010. [FDA] U.S. Food and Drug Administration. 2010. FDA Food Code 2009: Chapter 2 – Management and Personnel. Available from: http://www.fda. gov/Food/FoodSafety/RetailFoodProtection/FoodCode/FoodCode2009/ ucm181242.htm. Accessed Oct 21, 2010. [FDA] U.S. Food and Drug Administration. 2006. Managing food safety: a regulator’s manual for applying HACCP principles to risk-based retail and food service inspections and evaluating voluntary food safety management systems. Available from: http://www.fda.gov/Food/FoodSafety/ RetailFoodProtection/ManagingFoodSafetyHACCPPrinciples/ Regulators/ucm078165.htm. Accessed Sept 7, 2010. [FDA] U.S. Food and Drug Administration. 2004. Report of the FDA retail food program database of foodborne illness risk factors. Available from: http://vm.cfsan.fda.gov/∼dms/retrsk.html. Accessed Sept 7, 2010. [FDA] U.S. Food and Drug Administration. 2000. FDA report on the occurrence of foodborne illness risk factors in selected institutional foodservice, restaurant, and retail food store facility types. Available from: http://www.fda.gov/Food/FoodSafety/RetailFoodProtection/ FoodborneIllnessandRiskFactorReduction/RetailFoodRisk FactorStudies/ucm089696.htm. Accessed Sept 7, 2010.
Supporting Information The following supporting information is available for this article. Food Safety Online Training Evaluation Tool Please note: Wiley-Blackwell is not responsible for the content or functionality of any supporting materials supplied by the authors. Any queries (other than missing material) should be directed to the corresponding author for the article.
Available on-line through ift.org
