research design and methodology

CHAPTER 3 RESEARCH DESIGN AND METHODOLOGY Study Description: Sample Population The study group for this research project was a group of 85 graduate nursing students enrolled in a graduate course in pathophysiology. The students participated in a four-phase project. The first part of the project was be to survey their attitudes about course objectives, self-efficacy, and time-budget constraints with regard to their ability to perform in a science course. The second part of the study was to compare two types of performance-related feedback, simple corrective feedback and elaborate focused feedback. The third part included a comparison of computer exam scores to paper and pencil exam scores. Finally, retention of content was assessed with a comprehensive final exam. All students in this study held a bachelor’s degree in nursing and were continuing their education into an advanced practice role. These students had at least one year of nursing experience. They have all had undergraduate courses in anatomy, physiology and pathophysiology. Since all students were admitted to the graduate nursing program and practicing in a professional nursing role in their work, it was assumed they had the innate cognitive ability to be competitive in the course content. The majority of students were enrolled in classes in the Family Nurse Practitioner Program while the other students were studying to become Pediatric Nurse Practitioners or Mental Health Nurse Practitioners. All students in the course were undertaking a program of study that required practice skills in diagnosing disease and prescribing treatment regimes.

The course content in pathophysiology was a survey of the effects of disease on physiology of the human body. Several hundred diseases were covered in the onesemester course. Topics include signs and symptoms of a disease, progression of a disease, and potential outcomes of treatment regimes. Specific treatment regimes were discussed in general terms such as what type of regime is used in treatment such as surgery, medication, etc. However, specific treatment regimes were usually not discussed as they are introduced in later courses in the program. Age specific and developmental alterations were correlated with clinical diagnosis and management. Students typically take pathophysiology within the first two semesters of entering their graduate program of study and prior to any clinical experiences. Graduate pathophysiology is considered to be a pivotal course in the program of study for nurse practitioner programs and is typically mandated by state boards of nursing and by the credentialing agencies for nurse practitioner programs (Program, 2003). All students were admitted to the class and the study without regard to gender, race, nationality, or religious affiliation. Although the majority of the students admitted to this nursing program were Caucasian; some students were African American, a few students were Asian, and some of Middle Eastern descent. No differentiation was made according to racial background. An IRB # 2004-11-074 EX (Appendix A, Appendix D) was obtained from both the University of Nebraska-Lincoln and from the participating university. All students were given consent forms, which were signed and returned.

Research Plan The major components of this research study were developed using the three major steps of Zimmerman’s Model of Self-Regulated Learning (Zimmerman, 2001, 2002). The components are labeled forethought phase, performance phase and review phase to evaluate retention of concepts. Table 1 outlines how each phase was conducted. Table 1. Phases of the Study Phase

Design Element

Forethought

Survey

Item Background Information Time Budget Demands Course Attitudes Content Objectives

Performance

Pretest – Posttest

Exam Questions

Paper & Pencil Exams

Exams 1 – 5

Computer Exams

Immune, Fluids,

Renal & Endocrine Retention

Final Exam

Review Questions

Forethought Phase The Forethought Phase includes goal setting, strategic planning, and the selfmotivation beliefs such as self-efficacy, outcome expectations and intrinsic motivation. Many sources cite nursing student attitudes with regard to science content or attitudes towards an entire course (Gresty & Cotton, 2003; Nicoll & Butler, 1996; Wynne et al., 1997). Nursing students have often indicated lack of applicability to practice as a

reason for poor attitudes and poor performance in science courses. Chapman and Sorge (1999) found that many studies in education were not effective in determining student’s ability to use data to achieve specific learning outcomes. They proposed that learning simulations and research should be linked to specific course objectives because they increased the student’s involvement with the material and increased their interest. No survey data could be found that addressed specific science course objectives with regard to the student attitudes or belief system about the use of the content in those objectives in practice. The survey in this Dissertation is unique in assessing attitudes about how the course objectives had been used previously in their professional practice roles and were compared to their beliefs about the use of specific course objectives in their future practice. All students have had experience in a pathophysiology course at the undergraduate level and no experience with the course at the graduate level. Students were surveyed about attitudes on course objectives with course objectives targeted at the undergraduate level (knowledge-based objectives) and at the graduate level (applicationbased objectives). A comparison was made to determine if student attitudes were different between the leveled objectives as rated by importance to their professional practice. This survey evaluated the student’s background with regard to knowledge base and factors that may limit the amount of time the student spends studying. Questions on this survey related to the number of years in practice and the particular areas of practice. For example, a student who had experience in renal or intensive care may have had a

very different set of skills in the course than a student who worked as a school nurse or as a community health nurse. Students were asked their beliefs about their ability in science courses. Students were asked to rate past performance in science courses and attitudes about difficulty of the course and applicability to practice. Since the tutorials were computerized, students were asked to rate their confidence with computer technology. Additional questions on the survey included time constraints including the number of courses the student was taking, whether they had children living at home, and the number of hours a week they were working? Additional correlations were made with regard to distance traveled to class since many graduate nursing students commuted to class. These were referred to as time budget constraints or time budget items. Performance Phase The procedure was modeled to take advantage of positive results of many prior research studies on feedback as reported in a synopsis by Bangert-Drowns et al. (1991). The first step of their model includes characterization of the student’s self-efficacy and prior relevant knowledge. These authors presumed that richer elaborations activated search and retrieval processes. The learner was stimulated to search and retrieval processes by a question. The next step was to have learner respond to the question. Each student received feedback. Although the authors found feedback to each question to have a greater positive effect than end-of-test feedback, both types of feedback had a positive effect. The final stage of their model allowed students to adjust their relevant knowledge. This study was operationalized using the same steps. Self-efficacy was assessed in the initial course survey and prior relevant knowledge was determined through

pretesting. To provide the rich elaborations, each question in this portion of the study included a paragraph illustrating the concept being covered in the question. The computer system provided differential feedback to the students based upon their assigned groups. The computer system used in this simulation could not provide feedback until the end of tutorial, but would be available for the student immediately upon completion of the tutorial. Buzhardt and Semb (2002) reported this method to have increased merit if frustration and anxiety were factors in the testing. To allow the students time to adjust their relevant knowledge, students were given one week to practice with the tutorial prior to the exam. During the course of study, course content was presented in a lecture format. Each student received an outline of the lecture prior to the lecture. Following the lecture, students received a worksheet of open-ended study questions to guide them through the essential content to review for the exam. No feedback was given for these worksheets. However, each question on the worksheet related directly to an exam question as in the tutorial. This content was tested using primarily multiple-choice questions and a few true-false questions. Four course units were selected for development of computer tutorials in place of the worksheet. A Blackboard course development system was used for tutorial development. Students were randomly divided into six research groups for this portion of the study using a random number generator available on the Internet. The groups were given tutorials for the purpose of studying for the exams over this material. Prior to issuing the tutorials to the students, 15 of the 25 tutorial questions were selected for pretesting using the random number generator. These pretests were used to assess prior

knowledge of the material. A comparison of the scores, pretested and non-pretested, was made. Two tutorials had feedback of correct/incorrect (simple feedback). Two of the tutorials had reflective feedback to focus the student on the correct answer in the form of an explanation as to why the answer was correct or given rationale why the incorrect answers would not answer the questions (elaborate feedback). Each group of students was rotated so that each group received two tutorials with the simple feedback and two tutorials with elaborate feedback. The tutorials could be retaken as many times as the student wished. The tutorials did not count towards the course grade, but the subsequent exam over the material in the tutorial contributed to the course grade. All questions from the tutorial were reworded or redesigned so that memorization of answers did not provide a correct answer, but that the concept from each question was covered. Each tutorial had password protection that was electronically mailed to each student. This allowed them access to the correct tutorial. The tutorials were followed by a computer-based examination of the material from the tutorials. Although the tutorials were available for students to use at home or anywhere access to the Internet was possible, the computer exams were given as in-class proctored exams. The average scores on each exam were compared to determine if the elaborate feedback affected the outcome or scores on the exams. The scores on the computer-based tests were compared to the student averages on the previous written exams to determine if exam scores were significantly changed by the introduction of computer exams.

Each student was assigned a coded number for protection of student confidential data. Students were not given information regarding which study group they were assigned. Students were encouraged not to share materials or passwords from other tutorials with each other. Review Phase Self-reflection is a chance for students to reflect on performance and to make adjustments to improve that performance (Schunk & Zimmerman, 1998). The students were given a list of worksheet (non-computer concepts) and tutorial questions (computer concepts) to study and prepare for the final exam. A comparison was made of the noncomputer items and the computer tutorial items that are contained on the final exam. The average scores on individual questions were used to evaluate student retention of content for both non-computer concepts and concepts covered in computer tutorials. Data Analysis The survey and assessment data was entered into the Statistical Package for the Social Sciences, version 12 (SPSS) for data analysis. Table 2 below outlines how each component was assessed.

Table 2. Statistical Tests Data

Comparison

Test

Nature of Variable

Reliability

Objectives

Split-Half

Ordinal

Reliability Validity

All Objectives &

Factor Analysis

Ordinal

ANCOVA

Nominal

Subscales Demographic & time

Scores on exams

data Confidence of course content

Learning objectives

to Ratio Average of learning

t-test

Ordinal

Paired samples

Ordinal

objectives

Past Practice vs. Future

t-test Pretested questions

Nonpretested Questions

t-test

Ratio

Computer (elaborate

Computer (simple feedback)

t-test

Ratio

Computer (feedback)

t–test

Ratio

Unit Exam content

t-test

Ratio

feedback) Paper & Pencil (nonfeedback) Final Exam

questions

Course Survey Data Analysis Time Budget Many variables may influence a graduate nursing student’s ability to spend significant amounts of time studying for a content heavy science course. These include having children at home, working full or part-time and taking additional classes beyond the class studied. Although this course was not offered through distance education technology, students who drive to campus from a distance can have additional factors influencing their performance including distance traveled and the ability to have a study partner or someone to assist with driving. These data were collected in the initial course survey (See Appendix C). Degrees completed, children (yes or no), work demands and school demands were coded as nominal data. Work hours were coded as an average number of hours each student works per week. Courses enrolled were coded as none, one or two additional courses. None of the graduate students should have been enrolled in more than three courses per semester. Recall of course content can be influenced by the time the student has been away from undergraduate courses and the type of work history since that time. These data were categorized into three major groups: acute care, psych-mental health, and clinic. The data collected on all participants were analyzed using an analysis of covariance (ANCOVA). Since all of the variables are obtained from the same population and because an interaction of time constraints may create additional differences in data, a multivariate analysis of covariance (MANCOVA) was used to determine if the interaction of these time-budget variables has an effect on the sample. For example, it

may be that students were simply overextended in all time demands (Polit & Hungler, 1999). Course Confidence Information Nursing student perceptions about science course content and their beliefs about applicability of course content have been found to have a strong influence in their academic performance (Andrew & Vialle, 1998). The survey data on confidence of course content were used as a measure of overall student beliefs for importance of this content to practice. Common complaints of nursing students enrolled in a science course are that the course content was “to in-depth” and not applicable to practice (Thorton, 1997). However, students rate the need for science courses as high, therefore, a comparison of the item on the need of the course information for practice was compared to the mean score the student gave for the individual objectives for advanced practice. Content Level Objectives The taxonomy proposed by Bloom, Engelhart, Furst, Hill and Krathwohl (1956) has been widely used in delineating a level of knowledge of course content depth (Bruning et al., 2004). As suggested by Chapman and Sorge (1999), course objectives were used for student evaluation to measure importance and depth of course content as it relates to practice. Thirty-two course objectives in the survey were developed using both graduate and undergraduate level pathophysiology texts. These texts were used as a determinant of the content reliability for the objectives in the survey. The leveling of the objectives between undergraduate and graduate was indicated by the texts as being appropriate and the taxonomy developed by (Bloom et al., 1956) was used to verify leveling.

The objectives were paired. Each pair includes a lower knowledge-level objective and a more complex objective requiring application or analysis-level information. The participants rated the importance of each objective to practice both past and the belief about future practice. The objectives were rearranged to reduce the ability to recognize the pairings. A comparison was made of the undergraduate and graduate level objectives including importance to past practice and to future advanced practice using paired t-tests. Reliability and Validity of Course Survey The reliability of an instrument is the extent to which the results of using the instrument would be similar during repeated uses of the instrument (Gillis & Jackson, 2002). Reliability can be measured as a norm-referenced procedure such as test-retest or an internal consistency procedure. The alpha coefficient is a preferred index of internal consistency because it represents a single value for the data set and is equal in value to all of the possible split-half coefficients for the data set (Waltz, Strickland, & Lenz, 2005). The survey objectives were measured for internal consistency using a split-half coefficient expressed as a Spearman-Brown correct correlation (Green, Salkind, & Akey, 2000). The halves were split into undergraduate and graduate and then assigned to a group based upon their order in the survey instrument. Three types of validity are commonly defined. Content validity for educational material is frequently measured through standardized textbooks, curriculum guides, and syllabi from courses across the country (Waltz et al., 2005). Several undergraduate and graduate textbooks were used as reference materials to develop the course objectives measured in the survey.

Criterion validity requires the use of a readily available performance measure (Waltz et al., 2005). Since no validated factor was found to use in this study, criterion validity was not established for this survey. Construct validity is a measure to determine consistency of items with the theory and concepts as designed. Factor analysis can be used to measure construct validity of a document when a Likert-type scale is being used in the research study (Munro, 2001). A factor analysis was completed on the objectives developed for the student survey. Exam Data Analysis Computer Tutorials with or without Feedback The computer tutorials were assessed on the basis of elaborate versus simple feedback. The computer tutorial had information leading to a question. Each question had four answers. Students were asked to select the best answer. Approximately onehalf of the students received feedback that was designed to promote evaluation of the question and to elaborate on why one answer was correct and other answers were not. The other students received the same questions and answers. However, this group received feedback that stated correct/incorrect depending upon the student’s response. Since all students in the study group had completed an undergraduate course in pathophysiology and were employed as a registered nurse for a minimum of one year, prior knowledge of content was assessed. A representative group of 30 students was selected to receive a pretest for the four computer exams prior to any content being presented on the topics. To reduce the pretest effect of having students access the answers, only 15 questions were used in the pretest. The posttest for each computer exam had 25 questions. The pretest and posttest scores were analyzed using repeated

measures t-tests. Since students could enter the tutorials as many times as they wished and could retake several times, scores on student tutorials were neither entered as data nor used in the data analysis. Computer Exams Compared to Paper and Pencil Exams Following each lecture in the course, students received a study guide. For five exams, the student received a paper worksheet with questions that related to the topic for paper and pencil exams. The student did not receive any feedback on this information from the instructor. Students were allowed to post questions on the class web page to ask for help from other students. The instructor only answered these questions when the student response to a question was incorrect or misleading. The students took five exams on paper and pencil using primarily a multiple choice question format. A few questions were true/false. No other types of questions were used for the examinations. The same types of questions were used on the computer exams. The paper and pencil exam scores were compared to the scores on computer exams to determine potential effects of feedback versus non-feedback using a repeated measures t-test. Retention A measure of student retention of concepts was the performance on a final exam given at least one week following the completion of the other exams. The four computer tutorial exam scores were coded using a percentage value for each. The final exam score was coded as three values. The first value was the total score for the exam. The second value was the percentage score on the questions relating to the paper and pencil exams. The third value was the percentage score on the questions relating to the computer exams. These values were compared to the scores on the previous exam questions. The paper

and pencil question retention was compared to computer tutorial retention. Repeated measures t-tests were used to analyze any differences in these exam scores.