This course provides an introduction to Institutional Research (IR). The goal of
the course is to ... (Eds.), The handbook of institutional research. San Francisco: ...
ID 602 – Institutional Research Fall 2013 Tuesday, 2:00-5:00 p.m. 200 West Quad
Dr. William E. Knight Office of Institutional Effectiveness 200 West Quad
[email protected] 765-285-2191
Course Goals and Objectives This course provides an introduction to Institutional Research (IR). The goal of the course is to expose students to the theory, field, and practice of institutional research, which is a unique combination of higher education policy, research methodology, data analysis, and applied research. In this class we will be examining the answers to the following four questions: 1. What is institutional research? 2. What do we know or need to know about our institutions? 3. How do we measure, determine, or judge our institutional quality? 4. How do we judge our effectiveness? Students will be able to: 1. Identify research needs and define research questions associated with current issues in higher education. 2. Describe historical, social, political, financial, ethical, and other issues pertaining to assessment and evaluation in higher education. 3. Discuss the roles of diverse constituencies in requesting, providing, and receiving information related to assessment and evaluation. 4. Identify formal and informal sources of data and information. 5. Conduct critical reviews of existing research and identify factors influencing the application of research findings to specific problems in higher education. 6. Prepare a research plan that includes well-defined questions, research methods, a timeline for implementation, resource requirements, and procedures for communicating results. 7. Gather, organize, analyze, and synthesize data and interpret information from multiple sources. 8. Effectively communicate research findings orally and in writing to diverse constituencies. Course Format Multiple class formats will be used, including lectures/discussions led by the instructor, students, and guest speakers. Some activities will be completed in class, while others will be completed individually or as part of groups outside of the class setting. Effective contributions to in-class discussions require that readings be completed before the class for which they are assigned. Texts (2) Hedrick, T. E., Bickman, L., & Rog, D. J. (1993). Applied research design: A practical guide. Newbury Park, CA: Sage Publications. Howard, R., McLaughlin, G., & Knight, W. (2012). (Eds.), The handbook of institutional research. San Francisco: Jossey-Bass. 1
Other course materials will be provided by the instructor or found on the Internet. Office Hours and Location My office is at 200 West Quad. I am normally there 8:00-5:00 every day, and you are welcome to phone or e-mail me at any time. Because I want to make sure that I am available when you come to see me, I ask that you please contact me ahead of time unless it is for something very brief. Don't hesitate to contact me! I am here for you! Requirements and Expectations I expect all students to attend class regularly and to be on time, to make substantial contributions to class discussions, to adhere to the Student Academic Ethics Policy1 and its principles for the ethical treatments of human subjects in research2. Please note that all written materials handed in are to be typewritten, double-spaced, and in APA format (latest edition). Group assignments will be graded upon a group basis; members of each presentation and research project group will receive the same grade on assignments. While students are encouraged to provide input and feedback about topics, methods, assignments, etc., the instructor holds the final responsibility for the syllabus, methods, and assignments and reserves the right to modify the syllabus as necessary. If you need course adaptations or accommodations because of a disability, please contact me as soon as possible. Ball State's Disabled Student Development office coordinates services for students with disabilities; documentation of a disability needs to be on file in that office before any accommodations can be provided. Disabled Student Development can be contacted at 765-2855293 or
[email protected]. 1 2
http://cms.bsu.edu/About/AdministrativeOffices/StudentRights/PoliciesandProcedures/StudentCode/VIIEthicsPolicy.aspx http://cms.bsu.edu/About/AdministrativeOffices/SPO/ResearchIntegrity/HumanSubjects.aspx
2
Grade Requirements Assignment
Percentage
Due date
Self-Assessment of Knowledge and Skills
5%
Sept. 3
Research Proposal Outline
5%
Sept. 10
10%
Sept. 17
Survey Critique
5%
Sept. 24
Survey Development Assignment
5%
Oct. 1
Reporting Assignment
5%
Oct. 8
Statistical Analysis Assignment
5%
Oct. 15
10%
Oct. 29
5%
Oct. 29
10%
Nov. 5
IR Report Critique 2
5%
Nov. 12
Assessment Consultation Presentations
5%
Nov. 19
Assessment Topics Presentation
5%
Nov. 26
Exam 1
Exam 2 IR Report Critique 1 Full Research Proposal
Exam 3
10%
Final Research Presentations and Papers Presentations: Papers Due:
10%
Dec. 3
Dec. 10 Dec. 12 (5:00 pm)
3
TENTATIVE COURSE OUTLINE Students are responsible for supplying their textbooks and items in the following list where a web site has been included in the reference. The instructor will provide handouts of all other reading assignments.
4
Class 1 (August 20): Class Overview Class 1. Introductions to the course and to each other 2. Discuss issues, concepts, and definitions 3. Discuss possible research projects Class 2 (August 27): Background of Institutional Research (1) Assignment 1. Read HBR [Chapter 1] Saupe, J. L. (1990). The functions of institutional research (2nd ed.). Tallahassee, FL: The Association for Institutional Research. http://www.airweb.org/page.asp?page=85 Terenzini, P. T. (1993). On the nature of institutional research and the knowledge and skills it requires. In J. F. Volkwein (Ed.), What is institutional research all about? A critical and comprehensive assessment of the profession. San Francisco: Jossey-Bass. [Pages 21-29] Volkwein, J. F., Liu, J., & Woodell, J. (2012). The structure and function of IR offices. In R. Howard, W. Knight, & G. McLaughlin (Eds.), The handbook of institutional research. San Francisco: Jossey-Bass. [Pages 22-39] 24 Linear Inches: The IR Bookshelf Class 1. Discuss readings 2. Discuss IR projects Class 3 (September 3): Background of Institutional Research (2) Assignment 1. Read HBR [Chapters 2-3]
5
Eimers, M., Ko, J., & Gardner, D. (2012). Practicing institutional research. In R. Howard, W. Knight, & G. McLaughlin (Eds.), The handbook of institutional research. San Francisco: Jossey-Bass. [Pages 40-56] Reichard, D. (2012). The history of institutional research. In R. Howard, W. Knight, & G. McLaughlin (Eds.), The handbook of institutional research. San Francisco: Jossey-Bass. [Pages 3-21] 2. Prepare a 3-5 page self-assessment paper concerning your strengths and weaknesses in the three areas of organizational intelligence identified by Terenzini and include your plan of action to address your areas for further development. Class 1. Discuss readings 2. Discuss internal resources for IR 3. Discuss class IR projects Class 4 (September 10): Primary Institutional Data Sources for IR Assignment 1. Read HBR [Chapters 4-6] American Statistical Association brochures about survey research at http://www.amstat.org/sections/SRMS/whatsurvey.html [Skim and use for future reference] Billups, F.D. (2012). Conducting focus groups with college students: Strategies to ensure success. AIR Professional File, 127. https://www.airweb.org/EducationAndEvents/Publications/Documents/PF127.pdf Cheng, D. X. (2005). Assessing students’ perceptions of campus community: A focus group approach. AIR Professional File, 95. http://airweb3.org/airpubs/95.pdf Dillman, D. (2000). Mail and Internet surveys: The tailored design method (2nd ed.). New York: Wiley. [Page 27] Molasso, W. R. (2005). Ten tangible and practical tips to improve student participation in web surveys. http://www.studentaffairs.com/ejournal/Fall_2005/StudentParticipationinWebSurveys.htm Noble, J., & Sawyer. R. (2012). Institutional research with published instruments. In R. Howard, W. Knight, & G. McLaughlin (Eds.), The handbook of institutional research. San Francisco: Jossey-Bass. [Pages 523-549]
6
Simone, S., Campbell, C., & Newhart, D. (2012). Measuring opinion and behavior. In R. Howard, W. Knight, & G. McLaughlin (Eds.), The handbook of institutional research. San Francisco: Jossey-Bass. [Pages 502-522] Suskie, L. A. (1992). Questionnaire survey research: What works. Tallahassee, FL: The Association for Institutional Research. [Optional; available to borrow from the instructor; skim the whole book] Ball State Living Learning Community (LLC) focus group script Copies of Institutional Research printed surveys (distributed by the instructor) and Web surveys: http://www.bgsu.edu/offices/ir/OnlineSurvey/NewStudentTransition.htm https://us2.qualtrics.com/SE/?SID=SV_9yRbfIYcp6KdZJi&Preview=Survey&BrandID=bsuoir https://us2.qualtrics.com/SE/?SID=SV_9mG0bf8kzv0ycoA&Preview=Survey&BrandID=bsuoir 2. Complete research proposal outline and submit electronically as an email attachment Class 1. 2. 3. 4.
Discuss HBR, Chapters 4-6 Discuss focus groups and survey research Brief demonstration of Qualtrics web survey software and MAP-Works Review for Exam 1 Class 5 (September 17): IR Secondary Data Sources, General IR Internal Reporting, Effective Reporting
Assignment 1. Prepare for Exam 1 2. Read Leimer, C. (2011). First, get their attention: Getting your results used. AIR Professional File, 122. Sanders, L., & Filkins, J. (2012). Effective reporting. In R. Howard, W. Knight, & G. McLaughlin (Eds.), The handbook of institutional research. San Francisco: Jossey-Bass. [Pages 594-610] Kot, F. C. (2013, May). Estimating the causal effect of centralized advising on academic outcomes. Paper presented at the annual conference of the Association for Institutional Research, Long Beach, CA. [Air Forum paper for ungraded critique] Class 1. Exam 1 2. Discuss external resources for IR 3. Discuss general IR internal reporting (Fact Books, survey results, analytic studies, …) 4. Discuss effective reporting 5. Group un-graded IR report critique 7
Class 6 (September 24): External Data Sources, External Reporting, Accountability Assignment 1. Read Fink, G., & Muntz, C. (2012). Federal reporting databases and tools. In R. Howard, W. Knight, & G. McLaughlin (Eds.), The handbook of institutional research. San Francisco: JosseyBass. [Pages 354-370] Fuller, C., Lebo, C., & Muffo, J. (2012). Challenges in meeting demands for accountability. In R. Howard, W. Knight, & G. McLaughlin (Eds.), The handbook of institutional research. San Francisco: Jossey-Bass. [Pages 299-309] Jankowski, N. A., Ikenberry, S. O., Kinzie, J., Kuh, G. D., Shenoy, G. F., & Baker, G. R. (2012). Transparency and accountability: An evaluation of the VSA college portrait pilot. Champaign, IL: National Institute for Learning Outcomes Assessment. Krist, P., Jones, E., & Thompson, K. (2012). Accreditation and the changing role of the institutional researcher. In R. Howard, W. Knight, & G. McLaughlin (Eds.), The handbook of institutional research. San Francisco: Jossey-Bass. [Pages 310-324] Ohio’s Higher Education Information web site. http://regents.ohio.gov/hei/index.php [Skim] Voluntary System of Accountability. http://www.voluntarysystem.org and http://www.bgsu.edu/downloads/finance/file55132.pdf [Skim] 2. Complete survey critique assignment Class 1. Discuss readings 2. Discuss external data sources 3. Discuss federal and state reporting, college guidebooks Class 7 (October 1): Quantitative and Qualitative Analysis of IR Data Assignment 1. Read Carson, C. (2004). The effective use of effect sizes in institutional research. Coughlin, M. A. (2005). Applications of intermediate/advanced statistics in institutional research. Tallahassee, FL: The Association for Institutional Research. [Optional; available to borrow from the instructor; skim the whole book] 8
Howard, R. D., & Borland, K. W. (2007). The role of mixed method approaches in institutional research. In R. D. Howard (Ed.). Using mixed methods in institutional research. Tallahassee, FL: The Association for Institutional Research. [Pages, 2-8] McLaughlin, G., Howard, R., & Jones-White, D. (2012). Analytic approaches to creating planning and decision support information. In R. Howard, W. Knight, & G. McLaughlin (Eds.), The handbook of institutional research. San Francisco: Jossey-Bass. [Pages 459-477] Milam, J., Porter, J., & Rome, J. (2012). Business intelligence and analytics: The IR vision for data administration, reporting, data marts, and data warehousing. In R. Howard, W. Knight, & G. McLaughlin (Eds.), The handbook of institutional research. San Francisco: JosseyBass. [Pages 434-453] National Survey of Student Engagement. (n.d.). Contextualizing NSSE effect sizes: Empirical analysis and interpretation of benchmark comparisons. http://nsse.iub.edu/pdf/effect_size_guide.pdf Statistical Decision Tree 2. Complete survey development assignment Class 1. Discuss readings 2. Discuss IR data analysis methods Class 8 (October 8): IR Support for Planning and Resource Management (1) Assignment 1. Read Carpenter-Hubin, J., Carr, R., & Hayes, R. (2012). Data Exchange Consortia: Characteristics, current examples, and developing a new exchange. In R. Howard, W. Knight, & G. McLaughlin (Eds.), The handbook of institutional research. San Francisco: Jossey-Bass. [Pages 420-433] Dorris, M., & Rackoff, J. (2012). Institutional planning and resource management. In R. Howard, W. Knight, & G. McLaughlin (Eds.), The handbook of institutional research. San Francisco: Jossey-Bass. [Pages 183-202] Purcell, J., Harrington, C., & King, B. (2012). Supporting institutional governance. In R. Howard, W. Knight, & G. McLaughlin (Eds.), The handbook of institutional research. San Francisco: Jossey-Bass. [Pages 133-144]
9
Teeter, D., & Brinkman, P. (2003). Peer institutions. In W. E. Knight (Ed.), The primer for institutional research. Tallahassee, FL: The Association for Institutional Research. [Pages 103-122] Examples of program evaluation provided by instructor, plus see: http://www.bgsu.edu/offices/ir/page19298.html 2. Complete reporting assignment Class 1. Discuss readings Class 9 (October 15): IR Support for Planning and Resource Management (2) Assignment 1. Read Cheslock, J., & Kroc, R. (2012). Managing college enrollments. In R. Howard, W. Knight, & G. McLaughlin (Eds.), The handbook of institutional research. San Francisco: Jossey-Bass. [Pages 221-236] Cuevas, N., & Matveev, A. (n.d.). Academic program review: Making the case for program quality and viability. Unpublished manuscript. Knight, W. E. (2001). Surviving and benefiting from an IR program review. AIR Professional File, 80. http://airweb3.org/airpubs/80.pdf Posey, J., & Pitter, G. (2012). Supporting the provost and academic vice president. In R. Howard, W. Knight, & G. McLaughlin (Eds.), The handbook of institutional research. San Francisco: Jossey-Bass. [Pages 145-164] Examples of enrollment management research provided by instructor, plus see: http://www.bgsu.edu/offices/ir/page16011.html http://www.bgsu.edu/downloads/finance/file19270.pdf http://www.bgsu.edu/downloads/finance/file26849.pdf 2. Complete statistical analysis assignment Class 1. Discuss readings 2. Review for Exam 2 [OCTOBER 22 NO CLASS]
10
Class 10 (October 29): IR Support for Planning and Resource Management (3) Assignment 1. Prepare for Exam 2 2. Read Kelly, H., Seybert, J., Rossol, P., & Walters, A. (2012). Measuring and evaluating faculty workload. In R. Howard, W. Knight, & G. McLaughlin (Eds.), The handbook of institutional research. San Francisco: Jossey-Bass. [Pages 550-572] McLaughlin, G., & Howard, R. (2003). Faculty salary analyses. In W. E. Knight (Ed.), The primer for institutional research. Tallahassee, FL: The Association for Institutional Research. [Pages 48-78] Teodorescu, D. (2012). Examining faculty recruitment, retention, promotion, and retirement. In R. Howard, W. Knight, & G. McLaughlin (Eds.), The handbook of institutional research. San Francisco: Jossey-Bass. [Pages 165-182] Examples of institutional research concerning financial issues 3. Complete IR Report Critique 1 Class 1. Exam 2 2. Discuss readings
Class 11 (November 5): Assessment of Student Learning (1) Assignment 1. Read Association of American Colleges and Universities. Board of Directors. (2008). Our students’ best work: A framework for accountability worthy of our mission (2nd ed.). Washington, DC: Author. [Skim] Borden, V. (2010, April 30). The accountability/improvement paradox. Inside Higher Education. Retrieved from http://www.insidehighered.com/views/2010/04/30/borden The Conference Board, Partnership for 21st Century Skills, Corporate Voices for Working Families, & The Society for Human Resource Management. (2006). Are they really ready to work?: Employers’ perspectives on the basic knowledge and applied skills of new entrants to the 21st century U.S. workforce. Retrieved December 13, 2006, from http://www.conference-board.org/pdf_free/BED-06-Workforce.pdf 11
Heart Research Associates. (2008). How should colleges assess and improve student Learning? Employers’ views on the accountability challenge. Washington, DC: Author. Jaschik, S. (2009, January 22). Assessing assessment. Inside Higher Education. Lederman, D. (2009, January 8). A call for assessment: Of the right kind. Inside Higher Education. Maki, P. L. (2004). Assessment for learning: Building a sustainable commitment across the institution. Sterling, VA: Stylus. [Chapters 1-2] New Leadership Alliance for Student Learning and Accountability. (2012). Committing to quality: Guidelines for assessment and accountability in higher education. Washington, DC: Author. Shavelson, R. J. (2007). A brief history of student learning assessment: How we got where we are and a proposal for where to go next. Washington, DC: Association of American Colleges and Universities. [Pages 5-13] Suskie, L. (2004). Assessing student learning: A common sense guide. Bolton, MA: Anker. [Chapters 1, 3] Walvoord, B. E. (2010). Assessment clear and simple: A practical guide for institutions, departments, and general education (2nd ed.). San Francisco: Jossey-Bass. [Pages 1-26] 2. Complete full research proposal Class 1. Discuss readings Class 12 (November 12): Assessment of Student Learning (2) Assignment 1. Read Astin, A. W. (2002). Assessment for excellence: The philosophy and practice of assessment and evaluation in higher education. Westport, CT: The Oryx Press. [Chapters 1-2] Astin, A. (2004, October 22). To use graduation rates to measure excellence you have to do your homework. The Chronicle of Higher Education, B20. Carey, K. (2006, September). College rankings reformed: The case for a new order in higher education. Washington, DC: Education Sector. Confessore, N. (2003, November). What makes a college good? Atlantic Monthly. Kuh, J. D., & Pascarella, E. T. (2004, September/October). What does institutional selectivity 12
tell us about educational quality? Change. [Pages 52-58] National Institute for Learning Outcomes Assessment. http://www.learningoutcomeassessment.org/ [Skim] 2. Complete IR Report Critique 2 Class 1. Discuss readings Class 13 (November 19): Assessment of Student Learning (3) Assignment 1. Read Delaney, A. M. (n.d.). Indirect evidence of learning from a campus perspective. Unpublished manuscript. Eames, A. (n.d.). Assessing the impact of curriculum-related interventions. Unpublished manuscript. Gonyea, R. M., BrckaLorenz, A., & Ribera, T. (n.d.). The assessment of student engagement in institutional research. Unpublished manuscript. 2. Prepare for assessment consultation presentations Class 1. Assessment consultation presentations 2. Discuss readings 3. Review for Exam 3 Class 14 (November 26): Class Presentations on Assessment Topics Assignment
1. Prepare for assessment presentation
Class 1. Assessment presentations Class 15 (December 3): Final Topics 13
Assignment 1. Prepare for Exam 3 2. Read The Association for Institutional Research. (2001). Code of ethics and suggestions for avoiding ethical dilemmas in the institutional research office. Tallahassee, FL: Author. Boon, R. D. (2012). Regulated ethics: Institutional research compliance with IRBs and FERPA. In R. Howard, W. Knight, & G. McLaughlin (Eds.), The handbook of institutional research. San Francisco: Jossey-Bass. [Pages 325-339] Leimer, C. (2011). The rise of institutional effectiveness: IR competitor, customer, collaborator, or replacement? AIR Professional File, 120. http://airweb3.org/airpubs/120.pdf Leimer, C. (n.d.). Ways to waste your IR capacity, or make the most of it. Unpublished manuscript. Luna, A. L. (2012). Data, discrimination, and the law: How two unrelated practices can work together. In R. Howard, W. Knight, & G. McLaughlin (Eds.), The handbook of institutional research. San Francisco: Jossey-Bass. [Pages 340-353] Class 1.
Exam 3
2. Discuss readings Final Project Presentations (December 10) Assignment 1. Prepare for IR project presentations 2. Final research project papers due Wednesday, December 12, at 5:00 pm Class 1. Final research project presentations
14
