Session F1E A WEB-TO-DATABASE SYSTEM FOR COLLECTING STUDENT DATA Ryan McFall1 , Mark Urban-Lurain 2 and Don Weinshank 3 Abstract Many instructors use the Web to deliver information to students. However, serious instructional development requires continuous fine-grained feedback from students. Instructors may want to use the Web to collect data from students, but integrating the Web with databases is difficult for those who are not database experts. We therefore created a general-purpose Web data collection and processing system. Questions and their responses (a "survey") are stored in a database, allowing instructors to easily analyze the data gathered with online form. To create a "survey" the instructor need only define the questions and possible responses, including free responses. For most surveys, no new code need be written. The system generates forms to collect data and stores the responses in a database. The system is very flexible since the data can include HTML, allowing the system to perform additional processing using JavaScript or other HTML techniques. The system also includes several tools to analyze the data. We have used this system for a variety of purposes including 1) gathering information about prior student experience at the start of the semester; 2) giving students the means to provide anonymous feedback of their comprehension of instruction; 3) creating on-line quizzes; 4) providing fill-in-the-blank forms that teaching assistants use to grade student projects; 5) course and instructor evaluations; and 6) to allow students to conduct student government voting on-line. This paper describes the system in detail, provides examples of how we have used the system and indicates how instructors who have database expertise may further extend the system. This software is freely available from the authors. Index Terms Web Software, Course Management
INTRODUCTION In many situations, instructors want to gather data from students, e.g., assessing background information, administering short quizzes, or evaluating the course and instructors. Unfortunately, designing Web forms and the code needed to handle the data they generate is a time consuming process. Given the time and resource constraints faced by instructors in typical college and university settings, creating such instruments is often infeasible. We originally developed this system to assess students' incoming computing experiences on the first day of class. 1 2 3
We wanted to gather the data, analyze it and display the results before the second class meeting. The course in question enrolls approximately 2000 students per semester, so a paper and pencil instrument was not an option. We therefore developed a Web-based system to perform this survey. In order to meet our needs, we identified the following design goals for the system • We wanted to tie the data to other student data in the system. Doing so allowed us to associate responses with individual students. We could use that data for instructional interventions or use the aggregate data as part of our Continuous Quality Improvement [1,2] of course instruction and TA performance. • We wanted to be able to collect a variety of types of data: multiple choice, free form responses and combinations of the two. • We wanted to dynamically generate the forms that present the questions and collect the answers, then deliver the forms to students via the Web. This approach allowed the instructors the greatest flexibility for creating the forms and gave the students maximum flexibility because they could complete the forms on any computing platform. • We wanted the ability to authenticate students who were answering the questions to ensure only authorized students submit responses. • We wanted to allow authenticated students to submit anonymous responses. We were already making extensive use of a relational database system to maintain records for the course. We therefore stored the survey data in the same database. Later, recognizing the frequent need for an on-line survey, we generalized the system, reusing much of the code and database design. Because of the flexibility of the system, we have been able to use it in ways we did not originally anticipate. This paper first describes related work that has been done in this area. We then describe the general architecture of our system, including a discussion of the ways in which the system is configurable and the way in which data are stored in the database. Next we describe several applications for the system we have used at both of our institutions, a Research I university and a small liberal arts college. Finally, we give instructions for obtaining the software and discuss future
Ryan McFall, Hope College, Dept. of Computer Science, 27 Graves Place, Holland, MI 49423,
[email protected] Mark Urban-Lurain, Michigan State University, Department of Computer Science, East Lansing, MI 48823,
[email protected] Don Weinshank, Michigan State Univ ersity, Department of Computer Science, East Lansing, MI 48823,
[email protected]
0-7803-7444-4/02/$17.00 © 2002 IEEE November 6 - 9, 2002, Boston, MA 32 nd ASEE/IEEE Frontiers in Education Conference F1E-14
Session F1E enhancements.
RELATED WORK
First, while use of the SurveySuite system has been free up until this point, the Web site states that henceforth there will be a yearly account fee. Second, the software is not open-source, meaning that users cannot modify the system to meet their special needs, whereas our system is open-source and can be modified at will. Third, SurveySuite does not provide any type of authenticated access to the system. The only way to identify the user who submits responses is to include a field in the survey asking the user for this information. Frequently, user authentication is crucial to maintain data integrity. For example, we have used our system to have teaching assistants grade student projects. For this application, it is critical that only TAs complete the surveys. Without authentication, anyone with knowledge of the survey's URL could complete the survey. Fourth, in many applications it is essential to allow a user to edit his/her responses to a survey, maintaining a single set of responses at all times. For example, our teaching assistants have used this feature to partially grade projects and come back to them later. Students find this capability useful since it allows them to partially fill in a course evaluation and complete it at a later time. This feature is crucial when the system is used to give out-of-class quizzes as well. Without an accurate authentication method, these applications would be difficult, if not impossible, to implement correctly. Finally, in SurveySuite some portions of the survey text entered by the user do not recognize HTML text, interpreting it instead as literal text. This reduces the flexibility of the software. For example, the survey author may wish to provide links to additional information as part of the survey. This is not always possible with SurveySuite, since in some parts of the web pages it generates, the HTML element used to describe a link is displayed literally, rather than using it to create a link. Our system solves this problem by allowing all portions of a survey system to contain embedded HTML content
Demand for CS courses has increased enrollments and course sizes at many institutions, requiring more complex logistics and management than smaller courses. Kay [3] discusses some of the logistic problems faculty must address as enrollment expands to hundreds of students per semester. He identifies three problems associated with increased course size: (a) increased faculty workload, which includes more time to prepare and evaluate assignments and to manage the greater diversity of student abilities; (b) student anonymity, leading to reduced student motivation and engagement and increased attrition; and (c) instructional inconsistency: if large courses are broken into multiple sections with Teaching Assistants, instruction and evaluation can vary across the sections. Kay discusses several strategies that faculty can employ to address these problems, including using information technology to manage the enormous amount of data these courses generate. Many faculty have created sui generis software to support their own course models. For example, Connelly, et al. [4] discuss the simulation software they created to model how compilers translate high-level programs into assembly language. Mason and Woit [5] describe how they created a secure environment for on-line programming quizzes using a combination of several Unix utilities. Canup and Shackelford [6] discuss several pieces of software they implemented to support the large service courses. They use software to help them manage groups of students, distribute, collect, grade and provide students with feedback on assignments, and provide some individualization for students in a large course. Similarly, Finkel and Wills [7] created several pieces of software to support peer learning, including software to form student groups, electronically turn in projects, administer surveys and allow students to check their grades electronically. The explosion in on-line courses has fueled the market for software to support these courses, such as WebCT (www.webct.com) and BlackBoard (www.blackboard.com). SYSTEM ARCHITECTURE AND DESIGN One of our institutions has licensed Blackboard for use by In order to meet the design goals outlined in the all faculty. However, the version of Blackboard in use at our institution is a closed environment; it does not allow users to introduction, we implemented our system as two components. The first is a relational database that holds the connect to their own databases to provide functionality not questions and response data. The second is a set of scripts native to Blackboard. Therefore, we wanted a general used to create the Web front end and to process and store the purpose solution that would allow us to connect to our data on the back end. existing database and integrate the results of our surveys Each survey is stored as a series of questions. Currently, with our other data. there are three types of questions: multiple-choice, free form SurveySuite [8] is another online survey system that response and a combination of the two. The question types allows users to create surveys that can be administered over are defined in the database, allowing the addition of other the Web. It also provides several useful tools that facilitate types of questions, such as check boxes, in the future. analysis of the data. Our system has several features that we In addition to the questions, the database stores survey consider important that are not present in the SurveySuite attributes such as package. 0-7803-7444-4/02/$17.00 © 2002 IEEE November 6 - 9, 2002, Boston, MA 32 nd ASEE/IEEE Frontiers in Education Conference F1E-15
Session F1E • •
•
A flag indicating whether or not the responses should be stored in the database anonymously. Flags that determine whether users are permitted to complete the survey more than one time and, if so, whether to store all responses or only the most recent set. An optional URL to a Cascading Style Sheets (CSS) stylesheet. If specified, the stylesheet is loaded when the survey is dis played in a Web browser, thus allowing different surveys to be formatted differently.
A separate database table holds a set of messages associated with each survey. One record in this table holds an introductory paragraph for the survey, which is displayed before the input fields corresponding to the survey's questions are generated. This paragraph might be used to give instructions on the survey, or perhaps to explain the reasons why the survey is being given. Similarly, another entry is provided to hold a closing paragraph, which is displayed at the end of the survey, before the responses have been submitted. All entries in this messages table can be stored as HTML, so entries can contain references to images or hyperlinks to other documents. Storing the messages and question data in HTML format also provides a great deal of flexibility to users who want to customize the appearance and functionality of the form. For example, special data validation rules could be implemented by storing Javascript code in the database along with the actual text of a question. Other messages are used to communicate various error conditions, such as • The message displayed when the user has previously completed the survey, but only one set of responses is allowed. • The message displayed when the user is not enrolled in the course for which the survey was created. • The text displayed on the authentication page, where the user enters a userid/password combination to verify his/her identity. Figure 1 shows a sample survey. In this example, the introductory paragraph includes an image displaying the course number and name together with the instructions for the survey. The formatting of the form is controlled by the stylesheet, including the background image and the font used to display the questions. Many other characteristics can be controlled using the stylesheet as well. The first two questions shown in figure 1 are free form response types. The textboxes for the user's responses are automatically generated by the system. The third question is a combination of multiple-choice and free-form response. The system generates the option buttons for the choices and the text box for the user's input. The system also generates the SUBMIT button that the user clicks after completing the form.
FIGURE 1: A SURVEY TO GATHER STU DENT DATA FOR AN OPERATING SYSTEMS COURSE Creating a survey
The first step in creating a survey is to give it a unique name and to associate it with a particular course in the database table that holds this information. The second step is to create the questions. For each question, the user must choose an appropriate question-type attribute and create a record in the appropriate table. After the questions have been defined, the set of possible choices for each question is defined in another database table. The last step is to specify the various messages associated with the survey. Storing the Responses User responses are stored in two database tables. Each student's responses are held in one table, while a second table records the unique student identifier and submission date. In an anonymous survey, the response table does not hold the student's actual identifying information. This schema supports the anonymous survey feature while restricting each student to a single submission if desired. Security and Authentication Any system that records sensitive data must be secure. Forms -based systems on the Web must not implement security by assuming that the user will traverse the pages in the order the designer presents them -- a savvy user can easily circumvent this type of security. For example, it is
0-7803-7444-4/02/$17.00 © 2002 IEEE November 6 - 9, 2002, Boston, MA 32 nd ASEE/IEEE Frontiers in Education Conference F1E-16
Session F1E possible to use the Telnet program to interact directly with a Web server. Our greatest concern was that a malicious user could impersonate a legitimate one. While we could have used authentication methods built into HTTP, these methods are easily spoofed. We therefore created a password checking mechanism. To authenticate a user for a survey, the database stores a unique identifier and password. To maintain security as a user navigates among the various forms, we use cookies to pass DES encrypted data that identifies the user. It could be possible for a malicious user to discover the DES key by repeatedly attempting to decrypt the cookie generated by properly authenticated student. However, this scheme significantly increases the knowledge, time, and computer resources needed to spoof the system. Furthermore, the system administrator can change the DES key at anytime. Another security problem can occur when a user begins a survey and then leaves the computer unattended. A malicious user could sit down at the machine and submit responses. We attempt to prevent this situation by adding a time-out to the forms. Once the survey is loaded, the user has a limited period of time to complete the survey. If the time-out interval has passed when a set of responses are submitted, the user must re-authenticate before submitting the responses.
SOME APPLICATIONS OF THE SYSTEM As noted, the software was originally designed to gather data on students' prior experience in a computer fluency course. We quickly adapted it to collect evaluations of the course and Teaching Assistants, giving short on-line quizzes and grading student projects. The first time we used the system to collect course evaluations, we were dismayed that the response rate was only 18 percent. We then modified the system so that, if students failed to submit a response -- even a blank one -- by a deadline, they were no longer able to look up the results of their exams on the Web. The response rate then rose to 85 percent, with most of these students choosing to complete the forms in considerable detail. Another application of the system has been to gather feedback from students on the quality of lectures and laboratory assignments. A separate survey was created for each instructional day, allowing students to answer questions such as • What is your overall rating of the effectiveness of this exercise (Very Helpful, Helpful, Somewhat effective, Not Effective)? • What (if any) aspects of this exercise were particularly effective in helping you learn? • What (if any) aspects of this exercise made it difficult for you to learn?
•
What questions has this exercise raised in your mind that you would like to see answered? While it would certainly be possible to gather this data through e-mail or traditional written means, summarizing and analyzing it would be time consuming and tedious. Use of our system allows the results to be quickly aggregated and presented to the instructor, facilitating rapid feedback to improve the next instructional unit. Recently, the system was used to support a course in database management systems. One of the difficulties in teaching such a course has always been finding large data sets students can u se to practice query design skills. A Web survey was created that allowed each student to submit approximately 20 records. In a class of 25 students, students generated a total of approximately 500 records. With this data stored in our database, it was then straightforward to transform the data into a format the students could use for practice and as part of their final project. Finally, we have also used the software to implement online elections for student government. Although the system was not designed with this use in mind, adapting it was straightforward. The elections were done by residence hall. Students were only allowed to vote for their own hall's representatives. We enforced this by using the system's enrollment checking feature. A “dummy ” course was created for each residence hall, and each resident was “enrolled” in that course. Storing survey completion records separately from the actual response data allowed votes to be anonymous, without allowing anyone to vote multiple times.
USE AND ANALYSIS OF THE DATA The system provides great flexibility for the use and analysis of the data. For example, by using ODBC drivers to connect to the database, we • Use report generation software to do ad hoc queries. • Use spreadsheets to graph the results and do simple data analysis. • Use statistical software to do more elaborate data analysis. • Send database-driven e-mail to students based upon the data they submit. In addition, scripts that allow data analysis in ‘real-time’ via the Web can be written in many languages. For example, we have written a PHP script that generates pie charts from the data in the database for a particular survey. Similarly, we have written Perl scripts that summarize the data and display the results in a text format as well. These scripts are provided as part of the distribution.
OBTAINING THE SOFTWARE The software is packaged for use at other institutions at the URL http://www.cs.hope.edu/surveys/. A complete set of scripts and documentation is provided to assist the user in
0-7803-7444-4/02/$17.00 © 2002 IEEE November 6 - 9, 2002, Boston, MA 32 nd ASEE/IEEE Frontiers in Education Conference F1E-17
Session F1E installing and configuring the software. In order to run the software, the following components are required • The Sybase database: the system makes use of several stored procedures as well as other Sybase specific features. Sybase is available at reasonable academic discounts. • A Web server supporting the CGI interface. We use the freely available Apache server, but any server should do. In order to support the graphing module described above, PHP must be supported by the Web server as well. • Perl must be installed on the same machine as the Web server. Several Perl modules must be installed as well. Perl and all modules are freely available.
FUTURE WORK Currently, the system lacks an interface to create or edit surveys. While using a front-end such as M icrosoft Access to enter data into the database can simplify these tasks, the person creating the survey still needs to know the structure of the database and work directly with the tables. This can be time consuming and occasionally error-prone. We are investigating a number of solutions to this problem. One approach would be to define an XML vocabulary for the surveys and use existing tools to validate the survey data and enter it into the database. Another approach would be to create a graphical front end, perhaps in Java or an MS-Access form, that guides the user through the process of survey creation. We plan to evaluate these possibilities and others, and then choose the one that is the most portable and easiest to use. A second possible improvement would be to add a check box type of question. This would allow the user to select multiple responses, for example, allowing the user to indicate with which operating systems he/she has had experience. Adding this question type would require changes to t he structure of the database, since the responses table only has a single column to hold the response for each question. It is currently possible for a multiple-choice question without a default choice to be left blank by the user. While this is desirable in many cases, it is also useful to be able to require a response without indicating a default choice. Implementing this change will require adding a new attribute to the table containing the survey questions. To further increase the security of the system, we are contemplating running on a Web server equipped to handle the Secure Sockets Layer (SSL). While doing so would further increase the difficulty of spoofing the system, this feature could be implemented without any changes to the system itself.
SUMMARY One of the best ways to improve instruction is by gathering reliable and fine-grained information about students' experience, attitudes and performance. Unfortunately, writing the software to collect, process and analyze the data is time-consuming. The instructor must be extremely careful that the data are secure if they are in any way sensitive. The limited amount of time available to faculty members and computing center support staff frequently prohibits the use of such data gathering activities at many institutions. The software described in this paper is designed to be a general-purpose Web-based data-gathering tool. Since the system is entirely database-driven, no code need be modified to create surveys. All survey questions and responses are stored in the database, simplifying the entire process. The system provides many desirable features, including several different types of questions, anonymous responses, multiple sets of responses by a student to the same survey, and a flexible means for user authorization for each survey. We have already used the system in many different contexts and we anticipate many more. The software is available for general use, and includes a set of setup scripts to facilitate installation.
REFERENCES [1]
Dill, D. D., “Quality by design: toward a framework for academic quality management”, Higher Education: Handbook of Theory and Research, vol. VIII, 1992, pp. 37-83
[2]
Kaufman, R., and Zahn, D. Quality management plus: the continuous improvement of education, 1993
[3]
Kay, D. “Large introductory computer science classes: strategies for effective course management”, Proceedings of the 1998 SIGCSE Technical Symposium on Computer Science Education, Vol. 30 No. 1, March 1998, pp. 131-134
[4]
Connelly, C., Biermann, A.W., Pennock, D., and Wu, P. “Homestudy software: flexible, interactive and distributed software for independent study”, Proceedings of the 1996 SIGCSE Technical Symposium on Computer Science Education, Vol. 28 No. 1, March 1996, pp. 63-67
[5]
Mason, D. V. and Woit, D.M. “Integrat ing technology into computer science examinations”, Proceedings of the 1998 SIGCSE Technical Symposium on Computer Science Education, Vol. 30 No. 1, March 1998, pp. 140-144.
[6]
Canup, M., and Shackelford, R. “Using software to solve problems in large computing courses”, Proceedings of the 1998 SIGCSE Technical Symposium on Computer Science Education, Vol. 30 No. 1, March 1998, pp. 135-139
[7]
Finkel, D. and Wills, C. “Computer supported peer learning in an introductory computer science course”, Proceedings of th e Sixth Annual Conference on Integrating Technology into Computer Science Education, Vol. 28 No. SI, June 1996, pp. 55-56
[8]
Internet Commerce Group, University of Virginia. “SurveySuite home page”, online at http://intercom.virginia.edu/SurveySuite, March 2002
0-7803-7444-4/02/$17.00 © 2002 IEEE November 6 - 9, 2002, Boston, MA 32 nd ASEE/IEEE Frontiers in Education Conference F1E-18
