A Framework for Evaluation of a Large Scale Technology Innovation ...

Framework for Evaluation 1

A Framework for Evaluation of a Large Scale Technology Innovation Project: Lessons Learned in a Higher Education Initiative Jeffrey D. Kromrey Ha Phan Melinda R. Hess Chitra Subramaniam Lou M. Carey Kristine Y. Hogarty Shauna Schullo Ann E. Barron Gianna Rendina-Gobioff Gwen Smith Melissa Venable University of South Florida

Paper presented at the annual meeting of the Eastern Educational Research Association, Clearwater, Florida, February 11 – 14, 2004


A Framework for Evaluation of a Large Scale Technology Innovation Project: Lessons Learned in a Higher Education Initiative

Introduction In recent decades, American education has undergone a rapid evolution designed to meet the needs of the emerging information society. The use of technology in education and the development of distance learning courses and programs have increased at a brisk pace (Eaton, 2001). Initiatives for the retooling of higher education (necessary processes for the enhancement of the infrastructure and development of faculty) have generated a variety of programs in which technology plays integral roles. The effective development and implementation of such programs requires carefully planned and executed evaluation strategies. The purpose of this paper is to disseminate the evaluation framework, methods and instrumentation that were developed for the evaluation of a 5-year, federally funded technology project at a large, metropolitan research university. The major goals of the project (titled Innovations in Technology and Teaching or ITT) to be evaluated include development of innovative learning models (by combining technology and pedagogy) and examination of the impact of these models with respect to cost, effectiveness, access, and implications for underserved populations. Participants include a central project administration group; a large planning, development, and implementation team; an evaluation team; and many faculty members and graduate students who are teaching and learning in the new online programs.

Overview and Objectives The specific activities designed to fulfill the purposes of the project fall into three categories: (a) enhancing the infrastructure to support the new models, (b) development of courses and programs that implement novel strategies supporting the delivery of innovative courses and programs to underserved populations, and (c) the initiation and support of a coherent program of research related to innovations in teaching, technology and learning. For each of the planned activities, specific evaluation questions, instrumentation, and data collection strategies were developed. Consistent with the perspective that evaluation serves an important role in gathering and communicating information to facilitate successful project


completion, the evaluation activities sought to (a) provide timely, detailed information to project directors about the progress of the major grant activities, while providing opportunities to modify the activities and timelines on an as-needed basis; and (b) provide an independent analysis of the impacts and outcomes that are attributable to project activities. This paper has three main sections. First, we provide a logical, detailed description of the framework developed to facilitate the project evaluation. The framework includes the questions leading the evaluation, the primary data sources that inform our efforts, the schedule of data collection activities, and the communication network required to support our evaluation efforts. Secondly, we describe the instrumentation developed for the project, together with evidence of reliability and validity of the scores resulting from these instruments. Finally, we discuss our initial evaluation results, delineating the utility of the information gathered and planned enhancements to the framework. Program Evaluation Standards and Models The best approach to developing an evaluation plan for a given project is to consider the nature of the program to be evaluated; to examine standards, theories, and models that are appropriate for the nature and context of the program; and then to create an amalgam model or framework that fits the program purpose. The various program evaluation models in the literature are collectively useful for focusing program and course formative evaluations. They are especially helpful for identifying what evaluation questions to ask, who to ask, and when and how to ask them. A set of educational program evaluation standards (the Joint Committee Standards for Educational Program Evaluation) and two evaluation models (Stufflebeam’s CIPP project management model and Dick and Carey’s systems-based instructional design and development model) were used to plan the framework for the evaluation of these courses, technologies, and programs. These standards and models are compared for similarities across the phases, and optional data gathering strategies are listed for each of the models’ phases. Joint Committee Educational Program Evaluation Standards. These professional standards were developed and are reviewed and updated by a committee of representatives from sixteen professional organizations including, among others, the American Association of School Administrators (AASA), American Counseling Association (ACA), American Educational Research Association (AERA), American Evaluation Association (AEA), Association for


Supervision and Curriculum Development (ASCD), National Council on Measurement in Education (NCME), and the National Education Association (NEA). The purpose for considering the Joint Committee Standards when developing an evaluation model is to focus the project evaluation team on critical criteria to be considered in the design, conduct, and dissemination of information. The four main areas of evaluation standards include utility, feasibility, accuracy, and propriety. In Table 1, the standards are identified and defined in the left column, and criteria areas for planning and assessing an evaluation are included in the right column. Information in quotation marks is taken directly from information presented on the website (http://www.wmich.edu/evalctr/jc/), which provides details related to the complete set of standards. The website also includes standards for personnel and student evaluation that are applicable for guiding the development and conduct of educational program evaluations. Program Evaluation Models for Management. There are several forms of evaluation models, many based on the systems theory approach of input, process, and output procedure. Activities within each phase of the model produce outcomes or products that are the inputs for the next phase of the model. One of the originals was the widely used research and development model (R & D) based on “scientific method” that included four phases (see Figure 1).

Design

Develop

Implement

Document impact

Figure 1. The four phases of the research and development model

With this basic R & D model, the notions of formative feedback and revision were included through a feedback loop, meaning that following the work of any one phase, the preceding phase can be refined in light of new information.


Table 1 Brief Summary of Joint Committee Standards for Educational Program Evaluation

Standard Group “The utility standards help to ensure that an evaluation serves the information needs of intended users”

“The feasibility standards are intended to ensure that an evaluation will be realistic, prudent, diplomatic, and frugal.”

“The propriety standards are intended to ensure that an evaluation will be conducted legally, ethically, and with due regard for the welfare of those involved in the evaluation, as well as those affected by its results.”

Criteria Areas • • • •

Ensure evaluator credibility Identify all stakeholders Plan for appropriate scope of information collected and disseminated Make explicit the standards and values used to interpret data

• • • • •

Use practical non-disruptive data gathering procedures Ensure political viability for group cooperation Avoid bias and misinterpretation Ensure cost effectiveness Ensure evaluation report clarity, timeliness, and dissemination

• • • • • •

Proposal is written agreement Ensure a service orientation Respect and protect the rights and welfare of human subjects Respect human dignity and worth (do no harm) Fairly represent program strengths and weaknesses in records Ensure findings, including limitations, are accessible to persons affected Deal with any conflict of interest openly and honestly Accountably allocate and expend resources Ensure expenditures are prudent and ethical

• • • • • •

“The accuracy standards are intended to ensure that an evaluation will reveal and convey technically adequate information about the features that determine worth or merit of the program being evaluated.”

• • • • • • • •

Describe program purposes and procedures for identification and assessment Examine program context so that likely influences can be identified Guide evaluation so that, on completion, stakeholders can examine its strengths and weaknesses Use defensible information sources Ensure information is valid and reliable for intended use Correct any errors in information using systematic review Analyze systematically quantitative and qualitative information to answer questions effectively Describe and document the program clearly and accurately Fairly reflect findings in evaluation reports Explicitly justify conclusions for stakeholders’ assessment of them Formatively and summatively evaluate the evaluation against standards


Many program evaluation models are available to guide the design and conduct of program evaluation, and one of the most widely used management models is Stufflebeam’s original systems-based CIPP model. There appear to be several similarities between the original CIPP model and the basic R & D model in that they each include four phases of activity and provisions for feedback and refinement. The initials in the CIPP model stand for the four phases of activity illustrated in Figure 2.

Context

Input

Process

Product

Figure 2. The four initial phases of Stufflebeam’s CIPP model The definitions of these four phases are included in Table 2. The definitions and questions to be answered during each phase are located within quotation marks, and they represent actual quotes from Western Michigan University’s website (Stufflebeam, 2002). Readers are encouraged to visit the website (http://www.wmich.edu/evalctr/checklists/.htm) for a complete description of the issues and activities related to each phase of the model. Table 2 Stufflebeam’s CIPP Educational Program Evaluation Model Context Evaluation “Context evaluation assesses needs, assets, and problems within a defined environment. This phase of the evaluation answers the question, What needs to be done?”

Input Evaluation “Input evaluation assesses competing strategies and the work plans and budgets of the selected approach. The question answered is: How should it be done?”

Process Evaluation

Product Evaluation

“Process evaluations monitor, document, and assess program activities. The question answered is: Is it being done?”

“Product evaluation assesses a program's reach to the target audience. The product evaluation part is divided into impact, effectiveness, sustainability, and transportability evaluations. The question answered is: Did it succeed?”

In the fifth installment of his evaluation model, Stufflebeam (2002) added more phases to the initial four. Two of the new phases include contractual agreements at the beginning (prior


to context evaluation) and metaevaluation at the end (following product evaluation). Although they are located at the beginning and end of the CIPP phases, they influence activity throughout all phases. These new phases are summarized in Table 3. Table 3 Evaluation Phases Added in the Fifth Installment of Stufflebeam’s Program Evaluation Model. Model Phase

Questions and Activities

“Contractual agreements: CIPP evaluations should be grounded in explicit advance agreements with the client, and these should be updated as needed throughout the evaluation.”

This initial phase sets the ground rules for the evaluation and includes the following sample activities. Ensure written agreements for: • Obtaining information and channels for communication • Budgets, payments and dates • Analyses needed and activity timelines • Timing for interim and final reports and agreements to the integrity of the reports • Provisions for reviewing, amending, or canceling the evaluation.

“Metaevaluation is an assessment of an evaluation's adherence to pertinent standards of sound evaluation.”

This phase helps to ensure the quality of the evaluation and includes the following sample activities: • Apply Joint Committee Program Evaluation Standards and other professional standards as appropriate • Document evaluation process and findings to enable independent evaluation of the evaluation • Systematically use metaevaluation to improve the evaluation • Document the extent to which standards were used in the evaluation plan, process, and reports.

Instructional Design Models. Other models to consider when designing evaluations of projects that require the development and implementation of instruction, regardless of its format, are those gleaned from instructional design literature. There are several models of instructional design available, and most models have similar components to those within the systems-based Dick and Carey Model (Dick, Carey & Carey, 2001). The model illustrated in Figure 3 is converted from the original design and development model to include only formative evaluation activities, and the broken lines indicate an evaluation feedback and revision loop.


Conduct Formative Evaluation 8

Evaluate Instructional Analysis 2 Evaluate Instructional Goals 1

Evaluate Analysis of Learners and Contexts 3

Evaluate Performance Objectives 4

Evaluate Assessments 5

Evaluate Instructional Strategy 6

Evaluate Instruction 7

Figure 3. Dick and Carey Instructional Design Model Converted for External Formative Evaluation Evaluation criteria for each step in the model are derived from the learning theories and principles used in the initial design and development of instruction. Using these criteria, rubrics can be created by external evaluators to judge existing instruction. As an example, the partial checklist included in Table 4 is based on only Step 6, Evaluate Instructional Strategy. This particular checklist is an observation form to be used by evaluators; however, the design criteria within it can be easily converted to a questionnaire or interview with students, faculty, or instructional designers. In the left column of the table, all the areas for review are listed; however, the criteria for review within each area are included for illustration purposes for only the first few areas. Specific criteria can be included or excluded on evaluation instruments based on the nature of particular instruction. The formative evaluation model and criteria for each step in the model provide an important framework for focusing instructional evaluations and for developing evaluation instruments and procedures.


Table 4 Partial Checklist of Criteria for Evaluating an Instructional Strategy (Dick, Carey & Carey, 2005) I. Evaluate Instructional Strategy A. Plan for content sequence is: 1. Appropriate for type of learning? 2. Logical (e.g., chronological, simple to complex, concept to rule to principle)? 3. Follows main steps? 4. Covers all skills/information within a main step before moving forward to next step? B. Plan for content clusters (chunks) is appropriate for: 1. Skill complexity? 2. Learners’ age and ability? 3. Type of learning? 4. Content compatibility? 5. Time available (hour, day, week, and semester)? 6. Delivery format (self-paced, instructor-led, televised, web-based, combination, etc.)? 7. Time required for all instructional events per cluster? C. Plan for preinstructional activities: 1. Is appropriate for learners’ characteristics? 2. Is likely to be motivational for learners (gain attention, demonstrate relevance)? 3. Will inform learners of objectives and purpose for instruction? 4. Will cause learners to recall prerequisite knowledge and skills? 5. Will inform learners of any inputs needed to complete any tasks required? D. Plan for presentation materials includes: 1. Materials appropriate for the type of learning? 2. Clear examples and clear non-examples for learners’ experience? 3. Appropriate materials such as explanations, illustrations, diagrams, demonstrations, model solutions, sample performances? 4. Learner guidance through the presentation materials? 5. Aids for linking new content and skills to prerequisites? 6. Progression from the familiar to the unfamiliar? 7. Organization? E. Plan for learner participation is: F. Plan for feedback appears it will be: G. Plan for assessments is appropriate for: H. Plan follow through activities that are likely to: II. Plan for Student Groupings. Are groupings appropriate for: A. Learning requirements: B. Learning context: III. Plan for Media and Delivery system. Are plans appropriate for: A. Learning type B. Practical constraints

Not Relevant

No

Yes

Not Relevant

No

Yes


Integrating Models from Program Evaluation and Instructional Design. While there are some overlapping evaluation activities in these two types of evaluation models, the focus of each is on very different activities. Within the program evaluation model, the focus is on administration and management of the evaluation project; within the instructional design model, the focus is on evaluation of the design, development, and implementation of instruction. To illustrate these different foci, Table 5 includes the main phases of the CIPP model as column headings, and evaluation activities for related phases from the CIPP and instructional design models are fitted beneath each main phase. It is important to note in developing models and frameworks for unique evaluations of instructional programs that the Joint Committee Educational Evaluation standards are applicable throughout all phases of both the management and instructional design models (as indicated by the arrows at the bottom of the table). Project Description Evaluation goals and activities for the Innovations in Technology and Teaching Project were designed to be accomplished in two phases. During the first phase, evaluation activities were planned to provide timely information about the progress of course/program development and supported research activities. The purpose of this phase was to provide useful information to project directors about the progress of the major grant activities, while providing opportunities to modify the activities and timelines on an as-needed basis. During the second phase, the evaluation focus will expand to include an analysis of impacts and outcomes. Evaluation Questions A set of questions was delineated to guide the evaluation activities over the course of the project (Table 6). Specific questions were developed for each of the major categories of project activities.


Table 5 Integrating Models: Evaluation Activities in Stufflebeam’s Management Model and Dick and Carey’s Instructional Design Model Program Evaluation Phases Context: Identify and analyze: • Background information (e.g., program purpose, goals and directions) • Needs and problems • Clients’ and stakeholders perspectives • Viability of goals and directions considering political environment, policies, contracts, resources, and contexts, etc.

Input: • Compare plan with model programs, R & D literature, and alternative plans • Assess plan viability to achieve goals, alleviate needs, solve problems • Assess budget for work plan, timelines, resources, personnel, facilities and equipment • Assess congruence of materials and procedures developed with goals • Assess beneficiaries’ needs, • Assess communication paths.

Process: • Examine and document actual program events (advertising, orientations, time schedules, materials and procedures, facilities, equipment, operational procedures, and personnel) • Assess perceptions of progress by persons involved in program.

Product: Impact Assess whether intended target groups were served(effectiveness) • Assess effects on target groups’ immediate, mid-range, and long range achievement and attitudes Sustainability • Assess program cost effectiveness, replicability across groups • Assess stakeholders’ perceptions of outcomes • Identify goal-free positive and negative intended and unintended outcomes • Assess comparative cost effectiveness • Review plans, budgets, and staff assignments for likelihood of sustaining program • Analyze transportability • Assess potential adopters’ perceptions of viability and utility for transfer of program to new setting and groups.


Table 5 (Cont’d) Integrating Models: Evaluation Activities in Stufflebeam’s Management Model and Dick and Carey’s Instructional Design Model Instructional Design Phases Conduct front-end analysis including: • Identifying problem • Assessing needs • Analyzing resources • Analyzing context • Evaluating selected goals

Evaluate design including: • Instructional analysis • Learners’ characteristics • Learning and performance contexts • Performance objectives Evaluate materials including: • Assessment instruments • Instructional strategy • Instructional materials Evaluate materials mediation and delivery systems

Use formative evaluation strategies (one-to-one, small group, and field trial assessments) to: • Identify strengths and problems in materials and procedures • Summarize data by objectives and goals • Refine instruction and procedures based on data

Use summative evaluation to examine the strengths and problems with existing materials including: • Expert review and analysis • Learner perceptions and attitudes • Learner achievment • Learners’ ability to transfer new capabilities to workplace • Return on investment relative to organization needs, costs, transportability, and problems in workplace • Evaluate comparability with like instruction

Sample Data Gathering Strategies for Each Phase Document analysis Expert reviews Focus groups Delphi procedures Questionnaires and surveys Interviews Observations

Document analysis Expert reviews Focus groups Questionnaires and surveys Interviews Observations

Document analysis Expert reviews Questionnaires and surveys Interviews Observations Achievement assessments Attendance reports Events and time requirements Compatibility with ongoing program Equipment fidelity and viability (web system)

Document analysis Expert reviews Questionnaires and surveys Interviews Observations Achievement assessments Retention rates Graduation rates Employment rates Alternative program comparisons Cost analyses

Joint Committee Standards for Educational Evaluation Utility Feasibility Proprietary Accuracy


Table 6 Questions Guiding the Evaluation. Infrastructure Enhancement Equipment • What are the goals of the equipment acquisitions (i.e., what needs were to be met)? • What are the nature, structure, purpose and cost of the intended innovation? • To what extent does the enhanced infrastructure meet the identified needs and fulfill the intended purpose? • How can the equipment infrastructure and its use be improved? Faculty Development Initiatives • What are the goals of the faculty and staff development activities, and how were they determined? • To what extent have the development activities been effective? • What is the impact of the development activities on the staff providing the activities? • What faculty development needs remain, and how can development activities be improved? Course and Program Development • • • • • • •

To what extent are the funded development projects innovative (to the field as a whole and to the faculty members)? What are faculty conceptions of innovative learning models? To what extent are faculty prepared for the development and delivery of the funded courses and programs? What supports are available to faculty for development and delivery? What obstacles are encountered during development and delivery? How are these obstacles overcome? To what extent are the programs and courses effective? How can course development and delivery be improved?

Technology Innovation Research • • • •

To what extent do funded research projects reflect the depth and breadth of potential innovations in technology, teaching and learning? To what extent are deliverables provided and results disseminated? To what extent do these funded projects lead to additional external support? How can the support of research in technology and teaching innovations be improved?


Research Methods and Data Sources A variety of data sources were used to address the evaluation questions (Table 7). Both qualitative data (in the form of interviews, focus groups and document analyses) and quantitative data (e.g., rating scales, usage logs and measurable faculty and student outcomes) were gathered. Table 7 Data Sources for the Project Evaluation. Grant Activities

Data Sources

Collection Times

Enhancement of Infrastructure Equipment

Faculty Development Initiatives

Course and Program Development

Technology Innovation Research

Document Analysis

Summer 2003

Interviews

Summer/Fall 2003

Usage logs

Quarterly

User questionnaire

Quarterly

User interviews

Annually

Needs Assessments

Annually

Training records

Quarterly

Learner evaluations

Ongoing

Outcome measures

Ongoing

Document Analysis

Summer 2003

Innovation Scale

Summer 2003

Faculty Questionnaire

Quarterly

Focus Groups

Annually

Assistance logs

Ongoing

Student enrollment

Ongoing

Student evaluations

Ongoing

Usability assessment

Ongoing

Student outcome measures

Ongoing

Faculty time logs

Ongoing

Document Analysis

Summer 2003

PI progress reports

Quarterly

PI final reports

Ongoing


Three major components of evaluation were included: an analysis of the planned project activities, evaluation of the processes and progress of each component of the project, and evaluation of impacts and outcomes. Each of these components is described in detail. Planned Project Activities. This component consisted of an analysis of documents and interviews with major participants in the projects. The evaluation of previous needs assessments and individual project proposals provided a comprehensive overview of the scope of work that was funded under the grant, the proposal timelines for individual projects, and the extent of planned data collection and analysis activities. Fundamental to this aspect of the evaluation is the question of the extent to which the funded projects represent the depth and breadth of ‘innovative models.’ Data from these analyses were used formatively to inform the potential revision of project activities. Evaluation of Processes. Information was collected from the Principal Investigators (PI) and participants in the individual funded projects. Information provided in usage logs, problem reporting forms, and training records were continuously monitored and reports of progress were produced. In addition, data were gathered each semester in the form of faculty questionnaires and PI progress reports. These reports were supplemented by conducting annual focus groups in order to share ideas and identify potential problems that may be common across projects. Finally, data from student enrollment, perceptions of usability and outcomes assessments were used to inform potential modifications of the projects. Evaluation of Impacts and Outcomes. During the second year of the project, information on impact and outcomes will be collected and analyzed. These data will consist of student achievement and perception information, interviews and questionnaires for users of the enhanced infrastructure, and products of the funded research projects. The focus of this aspect of the evaluation is an assessment of the extent to which the purposes of the project have been fulfilled and the goals of the project have been met. In addition, evaluation data will be gathered to inform recommendations for improvement in subsequent developmental activities related to innovative teaching and technology. A mapping of the evaluation questions and activities into the framework described earlier provides an indication of the comprehensiveness of the planned evaluation (Table 8).

Reporting timelines

Funding

Communication lines

Questions to address

Contractual Agreements

How were goals determined?

Scope of funded research projects

Obstacles encountered during development

Supports available to faculty during course development

Faculty preparation for development and delivery of funded courses

Monitor use

Faculty conceptions of innovative learning models

Goals of faculty and staff training

Deliverables provided and results disseminated

Obstacles encountered during delivery

Supports available to faculty during course delivery

Innovation in funded development projects

Process

Nature and cost of intended innovation

Inputs

Goals of equipment acquisitions

Context

Table 8 Project Evaluation Questions and Activities Mapped into Stufflebeam’s CIPP Model.

Improved support of research in technology and teaching innovations

Additional external funding

Sustainability of courses in subsequent semesters

Course development and delivery improvements

Course and program effectiveness

Training improvements

Training needs remaining

Impact of training on trainers

Training effectiveness

Remaining improvements in equipment infrastructure and use

Enhanced infrastructure meets identified needs and fulfills purpose

Product

Areas where evaluation design and methods can be improved

Usability of interim reports

Effectiveness of evaluation design and data collection instruments and procedures

Effectiveness of communication lines

Meta-evaluation



Discussion of Findings Overview of Initial Evaluation Activities and Processes The activities and processes implemented during this first phase of the evaluation process included an extensive literature review to help identify appropriate measurement methods and tools for standards and objectives, acquisition of materials to guide the assessment, and identification of appropriate data sources. Sources of information used to guide the process included the National Educational Technology Standards (ISTE, 2000), institutional guidance, as well as additional information about other technology initiatives (e.g., the national PT3 grant). The team independently reviewed the wide variety of documentation available and then collectively designed a guiding framework to aid the development of the assessment and evaluation system of the ITT project. This framework was designed to allow flexibility during the development and implementation of the evaluation and assessment process while ensuring compliance with the overall purpose and specific objectives. Assessment Initiatives Implemented All three components of this long-term evaluation effort have been implemented to varying degrees. A multi-method approach was undertaken across these three elements of the project, incorporating qualitative and quantitative techniques to ensure appropriate methods were used for the various components of both this initial study as well as for the long-term development and validation of this assessment process. Enhancement of Infrastructure. The evaluation of the Enhancement of Infrastructure component of this project addresses a myriad of elements that provide the necessary support for implementation of the teaching initiatives called for under this project. In order to support distance learning aspects of the ITT grant including upgrades and uses of distance learning studios and associated equipment, items such as a video conference bridge, satellite receivers, and digital routers, among other necessities, had to be purchased. Likewise, other projects funded through the grant also required the purchase of equipment and resources to support implementation and delivery of instruction. The costs associated with these elements of the project were substantial. For example, the costs associated with purchasing a vehicle, retrofitting the vehicle, and acquiring the equipment necessary to do field experiments was approximately $200,000. As such, it was necessary to include evaluation of the effectiveness and use of both the equipment and training


acquired in support of the initiative undertaken through the ITT award. These efforts included such actions as conducting thorough reviews of equipment and training records, developing tracking and use logs, as well as discussions with the individuals responsible for overseeing these critical elements of a project. Records of equipment that was purchased under the project were reviewed with project and equipment managers and users in order to determine the appropriate types of data to be collected (e.g., location of equipment, intended purpose, users of the equipment) as well as identification of the appropriate office or individual responsible for tracking, monitoring, and reporting equipment use. Tracking documents were designed and constructed to facilitate monitoring of equipment use in order to be able to gauge cost effectiveness and utility of the equipment relative to the intended purpose. Data collection for this part of the infrastructure portion of the evaluation is in the initial stages. A review of the process and utility of the data will occur in subsequent phases of the evaluation process and, if necessary, processes and instruments will be revised. Enhance Outreach Efforts. One of the innovative initiatives supported by the ITT project proposes to enhance outreach efforts in the area of Environmental Science. To this end, a large van was retrofitted to allow installation and transportation of technical equipment to the field in order to conduct experiments and analyses. This van (affectionately named the ‘Enviro-van’) is intended to provide K-12 students in the surrounding schools with unique opportunities to conduct realistic experiments using sophisticated equipment in field situations. Additionally, the use of the van is to be maximized by allowing university researchers, including faculty and students, to engage use of the van in support of their research efforts. Evaluation efforts of this portion of the project are still in the initial stages. Meetings with project personnel have provided insight into intended uses of the van, including training efforts for teachers to use the van. A ‘Use Log’ has also been designed to track who uses the van for how long and for what purposes, in order to gather data on the utility of the van. Faculty Development Activities. Faculty development and training activities were required to provide instructors and other faculty with the skills and knowledge necessary for delivering effective instruction using innovative methods and equipment. Thus far, these activities have consisted of a 12-hour workshop provided to faculty in one of the colleges participating in the online-course delivery portion of the project. In discussions with training


personnel and project managers, it was discovered that this workshop was designed as a result of a needs assessment conducted by the training team approximately six months in advance of delivery. Members of the evaluation team attended each of the four 3-hour training sessions and gathered data regarding whether the workshops followed a best practices model of instruction in order to make recommendations for future workshops. The team was able to identify numerous commendable aspects of the workshops (e.g., timely technical assistance, positive engagement by participants, good question and answer sessions) with only a few recommendations for improvements. Additionally, the recommendations were rather minor regarding the pedagogical aspect of the workshops and tended to address more of the supporting elements, e.g., availability of ‘creature comforts’ such as drinks over a three-hour time span, technical problems, and materials distribution methods. At least one more workshop series is planned for faculty in another participating college although there are no plans for a similar series of workshops for the third college participating in the project due, at least in part, by the perceived lack of need by the faculty of that college. It was suggested that, due to the changing needs of faculty, a follow-up needs assessment be conducted to identify a possible change in the faculty members’ needs over time. Course and Program Development. A large part of the ITT initiative is focused on innovative delivery of instruction using technology. Both the Enhancement of Infrastructure and Technology Innovation Research components of the project support, at least in part, this component of the initiative. This part of the evaluation effort primarily includes formative evaluation of implementation of online delivery of Master’s degree programs in three colleges. The evaluation of the online delivery of the three full Master’s programs has begun in much greater depth and breadth compared to some of the other initiatives, primarily because of the larger scope and impact of this project component. Mixed methods have been employed to gather data from a wide variety of sources. Qualitative methods for this initial study consisted of content analyses of course documents, including course syllabi, Scope of Work (SOW) documents, Instructional Design Plans (IDPs), and on-line course materials, as well as interviews with instructors and instructional designers. The content analyses process is still in the early stages of evaluation. Instrumentation to conduct the review is under development, as is acquisition of the multitude of documentation required for the review. A small sample of syllabi, SOWs and IDPs has been obtained from specific courses to help guide this instrument


development. Additionally, guidance on requirements and intent of these various documents is also being gathered to assist with development of instruments and review processes. Interviews with faculty and instructional designers have been conducted on a fairly limited scale. An interview protocol was developed for each group and was implemented for the first time with the initial group of faculty delivering online courses (six instructors). The instructors were interviewed individually by a team of two evaluators. The instructional designers (IDs) participated in a two-part, focus group activity with a team of four evaluators. The content of the discussions are currently being analyzed to identify topics and themes discussed by the designers, including what worked well and the nature of problems that occurred during initial course development efforts. Their experiences, as well as those reported by the faculty will help inform not only the project, but also the process. Technology Innovation Research. The third component of this project encompasses three research projects awarded under the grant. These projects, although not directly tied to the technology and teaching initiatives undertaken within the scope of the project, do have a relationship to the purposes and goals of the ITT initiative and were, according to the project proposal (April 16, 2003), “selected in an effort to help support university policy decisions related to distance learning.” A brief description of the three research projects is provided in Table 9.

Table 9 Description of Research Projects Short Title

Description

Copyright and Fair Use Issues Relative to On-Line Instruction

Develop faculty awareness of copyright and fair use generalized other intellectual property issues associated with on-line teaching.

Effective Use of Synchronous Methods in Distance Learning

Investigate optimal structures for synchronous web-based courses

Evaluation of Courses Delivered On-Line

Develop and recommend a plan for administration of appropriate and effective course evaluation in a distance learning environment.


To varying degrees, each of these projects will be part of the evaluation effort. For example, the third research endeavor listed in Table 10 addresses the development of an appropriate and effective evaluation structure and process for online courses. As noted in the second component of the overall project, Course and Program Development, the evaluation of the effectiveness of online courses delivered under this grant is a major part of the evaluation effort. As such, the formative evaluation of the courses delivered under ITT serves to inform development of a more generalizable evaluation process for courses delivered online. Process Evaluation Activities (Implementation) During the first semester, six courses were delivered in three separate colleges. To monitor the implementation process, three student surveys and a faculty survey were developed. These were administered at various and distinct stages of course delivery to gain a holistic and in-depth evaluation of online courses. The initial phase of this study required a careful delineation of expectations of the evaluation as well as identification of appropriate domains for each step of data collection. Once the team had determined the general focus for each of the three student surveys as well as the faculty survey, subgroups worked on development of the instruments. An initial version of each instrument was constructed, and the entire team reviewed each for purpose, consistency of focus, wording and uniqueness. The instruments were then refined and finally sent for review by the instructors of six online courses for content, layout, comprehensiveness and applicability. Table 10 contains a summary of the instruments that were developed as a result of this process. This process resulted in a three-phase approach to gathering data from students enrolled in online courses. Initially, five domains were identified based on an extensive literature review and then further collapsed into three after extensive review and mapping of items to domains by the research team (see Smith, Helmick, Kromrey, Carey, Lang, Ducher, Rendina-Gobioff, Hess & Hogarty, 2003 for further details). The three resulting domains were identified as: (1) Course Delivery, (2) System, and (3) Design. Additionally, a fourth domain (Satisfaction) was added for the purposes of overall evaluation. Depending on the nature of the items and the domain, response scales varied from ‘yes/no’ to Likert-type scales, to open response items. A more thorough discussion of these scales and examples can be found in Smith et al. (2003). Once the items and scales had been reviewed and refined by the research team, the six faculty members, who initially delivered the online courses, reviewed each instrument. Their input


guided further refinement and adjustments to the instruments. For example, one of the instructors suggested adding an item asking students about the source and processes needed to acquire materials necessary for the course (e.g., did they buy books in person at the school bookstore or online). Table 10. Description of Surveys for Online Courses

Survey

Purpose of the Survey

Expected Administration

Student Entry

To gather data on the reasons students choose to take an online course and to determine whether underserved students were included

Within the first two weeks of the semester

Student Process

To gather data on students’ perceptions of the usability and effectiveness of the various technology elements of the online course (e.g., navigation, online resources, etc.)

At approximately the mid-semester point

To gather data regarding student satisfaction with the course. This survey is somewhat analogous to the typical end-of-course survey administered to in-class students

Within the last two weeks of the semester

To gather data regarding instructor’s satisfaction with, and perception of, online course delivery, including support, resources, and pedagogical effectiveness.

At the end of the semester of initial course delivery

Student Satisfaction (Product) Instructor Survey (Product)

The faculty survey, administered at the end of the semester, gathers data intended to inform about the preparation, delivery and effectiveness of their courses in an online environment. This instrument consisted of both Likert response items as well as open-ended items to obtain information in both a structured and flexible manner. Initial validation efforts were taken through cyclic reviews by both the team and the faculty involved in the online courses. Meta-Evaluation of the Framework Implementation In focusing the meta-evaluation, it is important to recognize that the stages of a project, context, input, process, and output, are also useful for examining the phases of the evaluation. After all, there are agreements and understandings to be made, materials and resources to gather, evaluations to get underway, and products of the evaluation. Table 11 contains the four management phases redirected to focus on the evaluation activities themselves and the types of


activities within each phase. The descriptions following the table are the successes and lessons learned for activities within each phase of the evaluation. As with any evaluation effort, regardless of breadth and depth, these preliminary efforts in development and implementation of an evaluation framework have met with certain successes as well as valuable lessons learned. Acknowledgements of the major elements that worked favorably in the process are important to not only inform future evaluation efforts of similar scope and purpose, but also to aid those involved in the current project maintain those practices. Alternatively, conscientious review of problems or issues encountered during the process will, hopefully, help alleviate future problems of similar ilk, or provide the evidence and information needed to help the evaluation team develop alternative plans and processes.

Table 11 Structure for Examining Evaluation Activities Context for Evaluation Project • Goals of and purpose for the evaluation • Agreements • Lines of Communication • Access to program personnel

Input for Evaluation Project • Link project goals to evaluation goals • Design and develop data gathering instruments • Design and develop data gathering procedures • Examine the psychometric characteristics of instruments and refine if needed

Process for Evaluation Project • Communication with those who need to provide data • Schedule evaluation activities (surveys, interviews, focus groups • Deliver instruments • Obtain data • Organize and summarize data • Interpret data

Products of Evaluation Project • Write and distribute evaluation reports • Discuss findings with key personnel • Adjust materials, procedures, and reports as needed

Context for Evaluation Project Communication. Although a fair number of individuals were very responsive and helpful, the evaluation team has realized that this is an area that requires more focus if adequate data are to be available for a fully informed evaluation. The development and maintenance of adequate communication venues tended to be problematic with individuals implementing various parts of the ITT efforts. At times, faculty or other ITT project personnel were unresponsive to requests to meet and discuss the project through either formal interviews or informational meetings. At other times, issues with obtaining necessary data that did not


require face-to-face meetings was problematic (e.g., providing copies of their syllabi, completing surveys, completing and returning problem or use logs). Inputs for Evaluation Project Development of evaluation materials. The thirteen evaluators involved in this effort evolved into a cohesive team through regularly scheduled large group meetings every two weeks as well as clear delineations of individual responsibilities through memberships in various subgroups focused on specific parts of the evaluation. This purposeful full-team approach facilitated the team working together as a whole and also provided team members with opportunities to assume individual responsibility for specific elements of the evaluation. Additionally, the composition of the team was well orchestrated, including experts with varying degrees of experience and backgrounds in evaluation, instructional technology, and statistical analysis. As such, the skills and abilities of team members complemented each other without being redundant. Team cohesion, camaraderie, and composition on an evaluation of this magnitude should not be overlooked as critical and vital elements in ensuring an effective and informative evaluation effort. Additionally, the iterative and sometimes painstaking processes undertaken in these early stages of the evaluation effort have resulted in creating instrumentation and defining data collection processes that are, by early indications, providing results with relatively strong evidence of validity and reliability. All instruments, whether they were surveys, problem logs, document review instruments, etc. underwent numerous stages of construction and review. Initial construction of an individual instrument typically was undertaken by an individual or small group using theory and previous studies to guide development. After this initial construction of instruments, the entire team reviewed all instruments, followed by other key personnel (e.g., instructors, designers, students). Although time-consuming, this process facilitated development of quality instrumentation and processes and also helped all team members keep informed of the various components of the evaluation effort. Processes for Evaluation Project Student Surveys. One challenge encountered during the process of data gathering was, ironically, technology-based. Although development and parts of initial validation procedures for the student survey instruments were accomplished, some portions of actual data collection had to be delayed due to problems encountered beyond the control of the research team.


Once surveys were developed, the team encountered issues with the server, which was maintained by a different department. The first problem was that the survey files were “mysteriously” being over-written by older versions. Next, the permissions to the database were set incorrectly, causing an inability to capture survey data. Finally, the server backup was not operational, requiring manual backups of data. The server issues delayed administration of the instruments to the point that data from students in ITT courses could only be acquired from the third (final) survey. Although somewhat frustrating, it is a reality that must be acknowledged and accepted. As technology continues to become an integral part of the education realm, our dependence on it also continues to grow. This dependence can be problematic and even detrimental. The wide variety of issues with technology, such as different platforms (e.g., Macintosh vs. PC), different web editors (e.g., FrontPage vs. Dream Weaver), different management systems (e.g., BlackBoard vs. WebCT), different browsers (e.g., Explorer vs. Netscape), all add to the complexity of effective and efficient use of technology. This complexity is likely to increase, not decrease, and as a result, more challenges and issues will arise associated with technology use. For the purposes of this evaluative effort, the problems previously encountered have been overcome, however this was a valuable lesson to be learned for future evaluation efforts. Faculty Questionnaire. Another data gathering problem was realized for the faculty questionnaire. Few faculty members returned the questionnaire (two out of six), which has hindered the team’s ability to make informed decisions about both the instrumentation as well as the project itself. Further efforts to collect these data have been undertaken to enhance the minimal data currently on hand. Interviews. The protocol used with the instructional designers appears, based on preliminary feedback, to work well, whereas the team needs to revisit the protocol used for interviews with instructors. Although only three instructors were interviewed, it was evident that the protocol did not work well to support and facilitate the review process. Instructors provided information through informal discussion that was not addressed by the current instrument. Outcomes for the Evaluation Project In preparing interim reports, the team realized the need for careful tracking of events and processes. Due to the large scope of such an evaluation, efficient means of documenting


stages of development, implementation, and evaluation is critical. For example, it was decided early in the project to use a comprehensive matrix of teams and tasks to track progress and changes between the large group meetings, which occur every two weeks. This instrument has provided an excellent historical record of progress and issues without requiring meeting minutes in the traditional sense. Alternatively, documentation of revisions of individual instruments, whether in content or format, has not been as carefully maintained regarding decisions and justifications for alterations. As the project progresses, recording the developmental aspects of this framework will be critical to ensuring accurate and useful information for the ITT program personnel. In addition it will inform the evaluation team about adequate resources (e.g., personnel, time) needed when considering undertaking a similar evaluation effort. Educational Significance At this point in the project, the team recognizes the importance of integrating program management models, instructional design models, and professional standards for guiding their evaluation work and ensuring that they have the appropriate focus when addressing the myriad of questions that arise. As the proliferation of distance-learning opportunities continues in the era of educational accountability, educators, administrators and institutions must have tools and methods available to ensure that the courses and programs they offer continue to meet the requirements of accreditation, policy-making and funding agencies. This study provides a foundation upon which to build and develop necessary tools and plans that can be readily adapted by a host of individuals, groups, and key stakeholders involved in higher education.


References

Dick, W., Carey, L., and Carey, J. O. (2001). The systematic design of instruction. Boston, MA: Allyn & Bacon. Dick, W., Carey, L., and Carey, J. O. (2004). The systematic design of instruction (6th ed.). Boston, MA: Allyn & Bacon. Eaton, J. (2001). Maintaining balance: Distance learning, higher education accreditation, and the politics of self-regulation. Paper based on Distance Learning: Academic and Political Challenges. Council for Higher Education Accreditation Monograph Series, Number 1. International Society for Technology in Education. (2000). National educational technology standards for teachers [Electronic version]. Available: http://cnets.iste.org/teachers/t_stands.html Joint Committee on Standards for Educational Evaluation (1994). The Program Evaluation Standards. Thousand Oaks, CA: Sage Publications. http://www.wmich.edu/evalctr/jc/. Smith, G. F., Helmick, I., Kromrey, J. D., Carey, L., Lang, T., Ducher, J., Rendina-Gobioff, G., Hess, M. R. & Hogarty, K. Y. (2003, November). Assessing online course delivery: Development and validation of instrumentation. Paper presented at the annual meeting of the Florida Educational Research Association, Orlando. Stufflebeam, D. L. (2002). CIPP evaluation model checklist: fifth installment of the CIPP model. Kalamazoo, MI: The Evaluation Center. Available: http://www.wmich.edu/evalctr/checklists/cippchecklist.htm


Acknowledgements This work was supported, in part, by the University of South Florida and the Fund for the Improvement of Postsecondary Education, under Grant No. P339Z000006. The opinions expressed are those of the authors and do not reflect the views of the United States Department of Education or the University of South Florida.