Defining Quantifiable Primary Verification Metrics of Program ... - iNEER

Session M3F

Defining Quantifiable Primary Verification Metrics of Program Outcomes in Biomedical Engineering at Florida International University Anthony J. McGoron1 and Richard T. Schoephoerster2 Abstract - The BME BS degree at FIU is a 128-hour program without specific tracks. It is interdisciplinary and weaves a strong chemistry and biology foundation with multidisciplinary engineering fundamentals, towards an emphasis of engineering living systems at the cellular and molecular level. It is directed at two audiences. The first audience includes students intending to enter industry or pursue advanced study in tissue engineering. The second includes students who intend to enter medical school following the BS degree. A Continuous Improvement Process for the program has been implemented that models the Design Control Process as prescribed by the Food and Drug Administration for the development of medical devices. This cascade process begins with User Needs (Objectives), which are converted into measurable Design Input (Outcomes). The Design Input drives the Design Process (Curriculum) to produce a specific set of Design Output (Student Capabilities) that are then used to produce a Medical Device (Graduate on the Job). A Verification Loop verifies that the Design Output (Student Capabilities) matches the Design Input (Outcomes). A Validation Loop validates that the Medical Device (Graduate on the Job) fulfills the User Needs (Objectives).

instruments for outcomes assessment have proven to be difficult to implement, are not well received by students, and not expected to provide all information required for the full evaluation of a program’s effectiveness [1]. Since the EC 2000 criteria were introduced by ABET in 1999 for accrediting the outcomes of engineering programs, advice was published to assist programs with the transition [2, 3]. Outcomes attributes modified from Bloom’s taxonomy [4] provide a more objective assessment of outcomes to allow attributes to be converted to variables and therefore, make them more quantifiable, and to provide specificity for evaluating outcomes by ensuring evaluators use identical criteria for scoring their assessments. The BME undergraduate program at FIU was designed after EC2000 was established, allowing the faculty to use EC2000 criteria as a foundation. This paper describes the program, its assessment, and verification. 1. Program Educational Objectives 1.1 Description of Educational Objectives The program’s educational objectives describe the expected career and professional accomplishments of our graduates three to five years after graduation from the University. The specific objectives of the Baccalaureate Programs in Biomedical Engineering at FIU are the following:

Index Terms – ABET, EC 2000, Outcomes Assessment INTRODUCTION Demonstrating that academic programs have a self-regulating process in place for assessing objectives and outcomes, and that they are consistent with the Universities mission is important as legislatures and the public demand high quality and accountability. Governing bodies require that the selfregulating processes be based on outcomes assessment and be focused on continuous improvement. The Florida Board of Governors has well defined guidelines for the review and continuous improvement process for programs within the State University System. These expectations are consistent with the requirements of the Accreditation Board for Engineering and Technology (ABET) and the Southern Association for Colleges and Schools (SACS), the primary outside accrediting bodies for the BME program. Establishing quantitative outcomes assessments is challenging. Standardized exams as 1 2

1. To provide an education that is at the interface of engineering and biology, with an emphasis on engineering living systems down to the cellular and molecular levels, and adequately prepares graduates for a wide range of career opportunities, including: a. Advanced study in engineering, medicine, or the sciences b. Professional practice as a biomedical engineer in industry c. Entrepreneurial endeavors in the biomedical industry 2. To produce graduates whose careers demonstrate proficiency in one or more of the following: a. Clinical application of biomedical engineering tools b. Product development, manufacturing, and commercialization in the biomedical industry c. Participation in diverse teams d. Biomedical engineering research e. Entrepreneurship

Associate Professor and Undergraduate Program Director, Florida International University. Miami, FL, 33199, [email protected] Professor and Chair, Florida International University. Miami, FL, 33199, [email protected]

San Juan, PR

July 23 – 28, 2006 9th International Conference on Engineering Education M3F-8

Session M3F 3. To produce graduates who have proper sense of professional responsibilities, service to community, leadership, ethics, and means of effective communication. 1.2 Evaluation Process: Overall Description of the Program Continuous Improvement Process The FIU BME Undergraduate Program Continuous Improvement Process is modeled after the medical device design process as prescribed by the Food and Drug Administration under the Design Controls section of the Quality Systems Regulations (Figure 1). The FDA describes the design process as a waterfall process that begins with establishment of user needs, using these needs to create design input of the device, using the design input during the design process to determine design output corresponding to the design input, and then using the design output as specifications for the final product, the medical device. The FDA also delineates two distinct evaluation loops: verification and validation. Verification assures that the design output matches the design input, and validation occurs when the finished medical device is used and fulfills the user needs. We use an analogous system for the continuous improvement of our undergraduate program, and in the evaluation and assessment of the program educational objectives and outcomes. The user needs are analogous to the constituent needs, these are then coalesced into a rigorous, coherent set of educational objectives. These program educational objectives are then converted into a specific set of program outcome measures (design input). Based on the desired program outcome measures, a curriculum is designed such that, at the time of graduation, a student who has successfully completed the entire curriculum requirements possesses the capabilities (design output) that are specified in the program outcomes. Finally, once the student (medical device) has been on the job (or in postgraduate studies) for several years, he/she should be fulfilling the user needs (objectives). In the spirit of this analogy, we use the term Verification to define the process used to assess the program outcomes and the use of that assessment to improve the program (“closing the loop” on outcomes), and the term Validation to define the process used to evaluate the program educational objectives and the use of that evaluation data to also improve the program (“closing the loop” on objectives).

Figure 1 The BME Program Continuous Improvement Process

1. Ability to apply knowledge of mathematics (including differential equations and statistics), physical and life sciences, and engineering to carry out analysis and design to solve problems at the interface of engineering and biology; 2. Ability to design and conduct experiments, as well as to measure, analyze and interpret data from living systems; 3. Ability to design a system, component, or process to meet desired needs, including systems that involve the interaction between living and non-living materials, within realistic constraints such as economic, environmental, social, political, ethical, health and safety, manufacturability, and sustainability; 4. Ability to identify, formulate, and adapt engineering solutions to unmet biological needs, 5. Ability to use the techniques, skills, and modern engineering tools necessary for engineering practice, including the ability to model and analyze biological systems as engineering systems; 6. Ability to function on multi-disciplinary teams; 7. Ability to communicate effectively; 8. Awareness of the characteristics of responsible professional engineering practice, including ethical conduct, consideration of the impact of engineering solutions on society in a global and contemporary context, and the value of life-long learning.

2.2 Relationship to Objectives The Program Outcomes were derived from the constituent needs formalized into the Program Educational Objectives. 2.1 Description Objective 1 describes the overall theme of the educational Based on the needs of the constituents, coalesced into a program and environment and type of practice (advanced coherent set of Program Educational Objectives as described study, professional practice as an engineer in a large or in the previous section, a set of Program Outcomes was entrepreneurial company), and Outcomes 1 through 5 describe developed simultaneously with the Objectives. These Program the skill set derived from the thematic focus of the program Outcomes describe the knowledge and skill set our graduates that are necessary to function in those environments and will have at the time of graduation from the program, and are varieties of professional practice. Objective 3 describes the thus analogous to the Design Input as depicted in Figure 1. attributes our graduates will display in their professional The FIU Biomedical Engineering Program Outcomes are the practice or advanced studies, and Outcomes 6 through 8 following. Graduates of the program will possess the describe the skills and knowledge necessary to display those following skills and knowledge set: attributes. Finally, Objective 2 describes the various specific San Juan, PR July 23 – 28, 2006 9th International Conference on Engineering Education M3F-9 2. PROGRAM OUTCOMES AND ASSESSMENT

Session M3F activities we expect our graduates to partake in during the first few years after graduation. All 8 outcomes contribute specific skills and knowledge for all of these activities. 3 ASSESSMENT PROCESS 3.1 Description A. Establishment of Outcomes The FIU BME Program Outcomes were established through the identical process used to establish the Program Educational Objectives. An Undergraduate Education Committee of the FIU BME Advisory Board was established and first convened in Fall 2003. Based on their initial input, a draft set of Program Educational Objectives and Outcomes were fully developed at the BME Annual Retreat in December 2003. A consultant was retained in Spring 2004 to assess our progress towards accreditation. This consultant met with our Undergraduate Education Committee, and the draft objectives and outcomes were discussed. Major modifications were suggested. Over the summer the objectives and outcomes were modified according to these suggestions, and the current set of objectives and outcomes were approved by the BME faculty in August 2004. B. Assessment plan The Verification process, as depicted in Figure 2, is used to assure that the capabilities our students posses at the time of graduation match the Program Outcomes. The review loop is designed as a yearly process. Outcomes measure data are collected for a calendar year, and this data is reviewed at our annual Faculty Retreat. At each Retreat, we first evaluate the appropriateness of the measure and measurement tool, and then determine if the outcome was met. If a modification to the measure or measurement tool is required, data are collected for only one term before assessing whether the outcome is still being met. At any point an outcome is not met, modifications to the curriculum are required. The measurement tools, which are inherently derived from the curriculum, are then modified as needed and data collected for another year. The assessment plan includes two sets of Outcomes measures: a Primary set, and a Secondary set. The Primary set of outcome measures has prescribed acceptance criteria. If the acceptance criteria are not met, action must follow (typically modification of the curriculum, if the measure and measurement tool are appropriate). The Secondary set of outcome measures does not contain specific acceptance criteria. These data sets are used to help understand the possible reasons why the Primary outcome measures were not met, if applicable, and point to possible remediation measures. For example, one Primary outcome measure is a rating of student performance on the senior design project by a panel of judges representing members of our Advisory Board. If, over the course of one year of student design projects, the rating for Outcome # 7 (ability to communicate effectively) does not meet the acceptance criteria, then the secondary measures would be used to determine where, for example, in the

curriculum is Outcome #7 not being covered adequately. The Secondary outcomes measures are, for the most part, tied to specific courses. Each of the courses, in turn, has course outcomes that relate to program outcomes. We would then look at the courses that contribute to Outcome # 7 and mine the secondary outcome measures to determine where changes in the program may result in improvement of Program Outcome # 7. Each of the 8 program outcomes is evaluated similarly.

Figure 2 The BME Verification Process

Three of the Primary Outcome Measures are derived from student performance on the Senior Design Project and BME Lab courses. The FIU BME curriculum is designed such that all knowledge gained through the curriculum is put into practice via these set of courses. Therefore, each of these courses encompasses the entire set of Program Outcomes. The Primary Program Outcome Measures are as follows: 1. Senior Design Project External Evaluation At the end of each Fall and Spring term, student teams completing their Senior Design Projects present their projects in the BME Technology Expo and Competition. This event provides an external evaluation of student performance. A group of external judges, representing organizations from our Advisory Board, rate the student teams on their ability to demonstrate, through activities required to complete the design project, all FIU BME Program Outcomes and determine the award winners. Each judge receives a detailed

San Juan, PR

July 23 – 28, 2006 9th International Conference on Engineering Education M3F-10

Session M3F executive summary of the project in advance of the competition. On the day of the competition, the students give a 15 minute presentation of the project followed by extensive questioning. Based on this input, the judges rate, on a quantitative scale of 4-0, the student team’s ability to demonstrate each of the 8 Program Outcomes. The taxonomy used in the External Evaluation was developed to define specific performance expectations and to provide specific and consistent criteria among judges. The taxonomy is consistent with that used for the Faculty Assessment and the BME Lab Course Assessment. This results in a direct objective measure of each program outcome based on specific performance criteria on the senior design project and BME Lab courses. 2. Senior Design Project Faculty Assessment Each Senior Design Project team has a faculty advisor. The faculty advisor is involved during both semesters of the design course sequence. The faculty advisor uses an Assessment tool to assess the quality of the project and assign a grade. Each item on the assessment tool is assigned a grade of A-F that is then converted to a quantitative score from 4-0. Each of these items in turn contributes to a score for each program outcome, also on a 4.0 scale. This results in a direct measure of each program outcome based on specific performance on the senior design project. Program Outcome #6 (ability to function on multidisciplinary teams) was assessed through a separate student Self & Peer Evaluation tool. This assessment tool is given to each student team member. The score from outcome #6 was calculated from both the instructors assessment of the students ability to function on multidisciplinary teams, as well as the students own assessments of themselves and their peers. 3. BME Lab Course Assessment Each BME student is required to complete two BME lab courses (BME 4050L and 4051L). As in the Senior Design Project, the lab courses are designed to encompass the entire range of program outcomes. The Laboratory Instructor uses the Lab Course Assessment tool to assess the quality of the laboratory work and assign a grade. Each item on the assessment tool is assigned a grade of A-F that is then converted to a quantitative score from 4-0. Each of these items in turn contributes to a score for each program outcome, also on a 4.0 scale. This results in a direct measure of each program outcome based on specific performance in the BME Lab courses. 4. Student Exit Survey Each graduating senior is required to complete the Exit Survey. This survey asks each graduate to assess their own capabilities and rate their level of achievement of the Program Outcomes, in their opinion, after completing the entire curriculum. Every engineering course syllabus the students receive contain course specific outcomes and their corresponding program outcomes. Therefore, throughout the program, students are aware of the expected outcomes. The scale is 1-4 with the following definitions: 1 –very dissatisfied, 2 –dissatisfied, 3 –satisfied, and 4 –very satisfied.

The Secondary Program Outcome Measures are as follows: 1. Course Outcomes Each of the required BME courses contains a specific set of course outcomes that describe the capabilities that each student should posses by the end of the course. Each of these course outcomes corresponds to at least one program outcome. Thus each course partially contributes to the program outcomes. Each program outcome is covered in at least five different required BME courses. Each course outcome has an associated student output (homework or exam question, part of written report, etc.) that is used to assess the student’s ability to master that capability. The course instructor uses this data as a discussion item during the course review at the Annual Retreat. 2. Transcripts Since each course has outcomes that contribute to the overall program outcomes, the grades each student earns in the required BME courses is also a measure of their ability to master the expected capabilities. 3. Course Outcomes Surveys At the end of each course, students are surveyed as to their opinion of their own mastery of the outcomes stipulated for each course. These data are quantified (achieved – 1.0, partially achieved – 0.5, and not achieved – 0.0) and used as another discussion item for each course review at the BME Annual Faculty Retreat. 4. Student Resumes At the time of graduation from the program, each student is expected to provide a professional resume to the Department. Their resumes contain extracurricular activities that contribute to the demonstration of their experiences and capabilities, particularly for Program Outcome #8. Examples of such activities are participation in student organizations, working in research laboratories, internships or coop experience, etc. 3.2 Metric Goals for Each Outcome For each of the primary outcome measures listed above, the Department has determined specific, quantifiable metric values that we expect the BME students to achieve. For each primary outcome measure, we expect a certain percentage of students to achieve a particular minimum score (Table 1). For example, for Program Outcome #1, one of the four primary measures is from the Senior Design Project External Evaluation tool. The target metric for this outcome measure is for 80% or more of the students to achieve a score of greater than or equal to 2.5. We expect that all students that graduate with a BS in Biomedical Engineering from FIU have achieved the program outcomes and possess, at a minimal level, the skill set that those outcomes describe. The target metrics shown above represent the kind of quality we hope to achieve from our graduates. The rational for each outcome measure is as follows.

San Juan, PR

July 23 – 28, 2006 9th International Conference on Engineering Education M3F-11

Session M3F A. Senior Design Project External Evaluation The Senior Design Project External Evaluation score ranges from 0 at the lowest to 4 (essentially corresponding to the 4.0 grade point scale). We expect that at least 80% of all student senior design project teams will receive a score of 2.5 or greater on each program outcome from the judges at the Biomedical Engineering Technology Expo and Competition. A score of 2.5 indicates that the student design team demonstrated above satisfactory achievement on a particular program outcome. B. Senior Design Project Faculty Assessment The Senior Design Project Faculty Assessment score ranges from 0 to 4. We expect that at least 80% of all student senior design project teams will receive a score of 2.5 or greater on each outcome from the team faculty advisor. A score of 2.5 indicates that the student design team demonstrated above satisfactory achievement for a particular program outcome. C. BME Lab Course Assessment The BME Lab Course Assessment score ranges from 0 to 4. We expect that at least 80% of all students will receive a score of 2.5 or greater on each outcome from the laboratory instructor. A score of 2.5 indicates that the student demonstrated above satisfactory achievement on a particular program outcome. The exception to this target metric is on Outcomes #8. For Outcome #8, we require all students to complete online training on the following: NIH course on Human Subjects in Research which addresses the ethical conduct of human research, ethical use of animals in research (administered through FIU’s Institutional Animal Care and Use Committee, IACUC), and in radiation safety, biohazard safety, and safe laboratory practices (administered through FIU Environmental Health & Safety & Risk Management Department). If the student completes each of these online training modules, they receive a score of 4.0. D. Student Exit Survey The Student Exit Survey score ranges from 1-4. Since this is a self-assessment of overall outcomes achievement, we expect that 90% of students will score themselves at a level of 3 or greater on each outcome. A score of 3 indicates the student is satisfied they have the abilities represented by each outcome. 3.3 Primary Verification Loop Data and Analysis The Senior Design data shown in Table 1 are for the 30 BME students that have completed the senior design project. The Exit Survey data are from the 10 students that have graduated from the program by Summer 2005. Exit Survey data from 9 students that graduated Spring 2006 are being compiled. The BME lab course data are from 45 students from Spring 2005Spring 2006. The table displays, for each outcome and their four measures, the average score achieved, the target metric, and the percentage of students that achieved equal to or greater than the target metric. In general, these data indicate that the students are meeting most metric targets for the

External Design Project Evaluation and for the Student Exit Survey. However, the data also indicate that the students are currently under the target metrics for some Senior Design Project Faculty and BME Lab Course Assessment measures. The scores for Outcome #1, #2, #3 and #5 from the Senior Design Faculty Assessment are low. This is partly due to two very poor senior design projects in Spring 2005 that skewed these data. The average values were above the 2.5 target, and in fact, a number of groups were at approximately 2.4, but below the target of 2.5. The faculty assessment is extensive and attempts to be comprehensive and very critical. Nevertheless, the faculty continue to develop a strategy to improve outcomes for the Senior Design by clearly articulating expectations during both semesters of the senior design course sequence. Faculty expectations are very high because the senior design experience represents the students overall achievement of all program outcomes, and demonstrates their preparedness for engineering practice. The Lab Course scores for Outcomes #1-#3 were below target, though all were at least 75% grater than or equal to target. Our Lab Courses are structured such that each faculty designs an exercise based on their own research, utilizing equipment in their laboratories and with the aid of their graduate students. Therefore, the lab exercises tend to be lengthy and complicated. We continue to refine and improve these lab exercises. In addition, the lab courses do not contain a strong design element. We are in the process of evaluating our Lab Courses to determine how to incorporate an element of design in which students must identify a problem and design a laboratory experiment to find a solution. The score for Outcome #4 from the Lab Course Assessment tool is also low. The laboratory instructor is now placing a larger emphasis on expectations within the lab report for the student to frame the laboratory exercise within the context of the biomedical engineering field. In other words, the student will be expected to use the knowledge gained from previous course work to show understanding of the context and use of the laboratory equipment and exercises within the field of biomedical engineering. The score for Outcome #5 from the Lab Course Assessment tool is also low. We are currently adding new laboratory experiments to the BME lab courses that expose the students to modern engineering tools. We have also modified our Modeling and Simulation, and Engineering Analysis of Biological Systems I and II courses to further emphasize the analysis and modeling of biological systems to prepare the students for the BME lab courses. Finally, Outcome #7 is low in the Senior Design Project External Evaluation and BME Lab course assessments, and just meets the target for the Senior Design Faculty Evaluation. We will look for ways to continue to improve the communication skills of the students. This is an ongoing issue at FIU since a significant portion of our students do not have English as a first language in their homes. For the Lab Course, it is worth noting that this score reflects the student’s performance in the course as a whole. A closer look at the data from which this score was derived shows a clear improvement

San Juan, PR

July 23 – 28, 2006 9th International Conference on Engineering Education M3F-12

Session M3F by the end of the semester for every student. But this remains an area to be emphasized throughout the entire curriculum. 4.

DISCUSSION

Continuous program improvement is critical for accreditation and for providing students with the skills necessary to become competent engineers. EC2000 adds the emphasis in the development of professional skills, such as solving unstructured problems, effective communication, team work, and ethical considerations, to the standards emphasizing mathematical, scientific, and technological knowledge [5]. Many of these skills are difficult to assess quantitatively. However, quantitative assessment lends itself better to objective continuous improvement. Since we designed the BME undergraduate program after EC2000 was established, we benefited from being able to develop the program specifically to meet the criteria. Accurate and useful quantitative assessments are highly dependent and the design of the measurement tools. The tools must be consistent, unbiased, and clearly define performance expectation. The

Outcome 1.

2.

3.

4.

5.

6.

7.

8.

tools must be “calibrated” to ensure that they accurately and directly measure the appropriate outcomes. They should be based on multiple inputs so that they are not inappropriately weighted by only a few inputs. We believe that our assessment tools meet these criteria, and that our assessment and verification process could be a model for other programs. REFERENCES: [1]

Banta, T., W. “Methods for Comparing Outcomes Assessments”, Research in Higher Education, 30(5), 1999, 455-469.

[2]

Rogers, G., "Outcomes Assessment. Opportunity on the Wings of Danger", Chemical Engineering Education, 33(2), 1999, 106-115.

[3]

Felder, R.M. and Brent, R. “Designing and Teaching Courses to Satisfy Engineering Criteria 2000”.

[4]

Bloom, B.S. and Krathwohl, D.R. “Taxonomy of Educational Objectives.” Handbook 1. Cognitive Domain. New York., AddisonWesley.

[5]

www.abet.org

Table 1 Outcome Measures, Target Metrics, and Outcome Data for 2004-2005 Measure Metric Target Avg. Score % ≥ Metric Senior Design Project External Evaluation 80% achieve ≥ 2.5/4.0 3.42 95 Senior Design Project Faculty Assessment 80% achieve ≥ 2.5/4.0 2.80 57 BME Lab Course Assessment 80% achieve ≥ 2.5/4.0 2.79 75 Student Exit Survey 90% achieve ≥ 3.0/4.0 3.42 100 Senior Design Project External Evaluation 80% achieve ≥ 2.5/4.0 3.42 90 Senior Design Project Faculty Assessment 80% achieve ≥ 2.5/4.0 2.68 43 BME Lab Course Assessment 80% achieve ≥ 2.5/4.0 2.82 75 Student Exit Survey 90% achieve ≥ 3.0/4.0 3.50 100 Senior Design Project External Evaluation 80% achieve ≥ 2.5/4.0 3.33 95 Senior Design Project Faculty Assessment 80% achieve ≥ 2.5/4.0 2.70 70 BME Lab Course Assessment 80% achieve ≥ 2.5/4.0 2.88 76 Student Exit Survey 90% achieve ≥ 3.0/4.0 3.67 100 Senior Design Project External Evaluation 80% achieve ≥ 2.5/4.0 3.51 95 Senior Design Project Faculty Assessment 80% achieve ≥ 2.5/4.0 2.99 83 BME Lab Course Assessment 80% achieve ≥ 2.5/4.0 2.66 60 Student Exit Survey 90% achieve ≥ 3.0/4.0 3.67 100 Senior Design Project External Evaluation 80% achieve ≥ 2.5/4.0 3.44 90 Senior Design Project Faculty Assessment 80% achieve ≥ 2.5/4.0 2.79 57 BME Lab Course Assessment 80% achieve ≥ 2.5/4.0 2.70 66 Student Exit Survey 90% achieve ≥ 3.0/4.0 3.29 83 Senior Design Project External Evaluation 80% achieve ≥ 2.5/4.0 3.12 85 Senior Design Project Faculty Assessment 80% achieve ≥ 2.5/4.0 3.28 85 BME Lab Course Assessment 80% achieve 2.5/4.0 3.82 93 Student Exit Survey 90% achieve ≥ 3.0/4.0 3.67 100 Senior Design Project External Evaluation 80% achieve ≥ 2.5/4.0 3.10 68 Senior Design Project Faculty Assessment 80% achieve ≥ 2.5/4.0 3.07 80 BME Lab Course Assessment 80% achieve ≥ 2.5/4.0 2.88 66 Student Exit Survey 90% achieve ≥ 3.0/4.0 3.83 100 Senior Design Project External Evaluation 80% achieve ≥ 2.5/4.0 3.13 82 Senior Design Project Faculty Assessment 80% achieve ≥ 2.5/4.0 2.84 83 BME Lab Course Assessment 100% achieve 4.0/4.0 3.27 98 Student Exit Survey 90% achieve ≥ 3.0/4.0 3.73 100

San Juan, PR

July 23 – 28, 2006 9th International Conference on Engineering Education M3F-13