DISCLAIMER. THIS SOFTWARE IS PROVIDED BY NORTHWESTERN UNIVERSITY ..... An input file containing item responses can be read into the program by.
FIRESTAR: Computerized Adaptive Testing (CAT) Simulation Program for Polytomous IRT models Version 1.2.2 Seung W. Choi, PhD. Department of Medical Social Sciences Northwestern University Feinberg School of Medicine October 2009
Funding Acknowledgment Development of Firestar was supported in part by the National Institute of Health Patient-Reported Outcomes Measurement Information System (PROMIS) Network (U-01 AR 052177-04, PI: David Cella).
DISCLAIMER THIS SOFTWARE IS PROVIDED BY NORTHWESTERN UNIVERSITY FEINBERG SCHOOL OF MEDICINE AT NO COST. IT IS PROVIDED “AS IS” WITH NO WARRANTIES EITHER EXPRESSED OR IMPLIED. Copyright © 2007-2009 Northwestern University Feinberg School of Medicine
TABLE OF CONTENTS 1. Overview ......................................................................................................................... 4 2. Installation Instructions ................................................................................................... 5 3. Configuring the CAT ...................................................................................................... 8 3.1 Launching the Program ......................................................................................... 8 3.2 CAT Configuration—Input Item Parameter File .................................................. 9 3.3 CAT Configuration—Input Item Response File ................................................. 10 3.4 CAT Configuration—Input External Theta File ................................................. 11 3.5 CAT Configuration—Number of Items in the Bank .......................................... 12 3.6 CAT Configuration—Maximum Number of Categories .................................... 12 3.7 CAT Configuration—Minimum/Maximum Theta and Increment ..................... 13 3.8 CAT Configuration—Stopping Criteria ............................................................. 13 3.9 CAT Configuration—Exposure Control ............................................................. 14 3.10 CAT Configuration—Prior Distribution ........................................................... 15 3.11 Interim Theta Estimators................................................................................... 16 3.12 First Item Selection ........................................................................................... 17 3.13 IRT Models ....................................................................................................... 18 3.14 Standard Error (SE) Calculation Methods ........................................................ 19 3.15 Scaling Constant ............................................................................................... 20 3.16 CAT Configuration—Simulating Theta and Item Responses .......................... 21 4. Output Control .............................................................................................................. 22 4.1 Output—Theta Audit Trail ................................................................................. 22 4.2 Output—Item Usage Information ....................................................................... 23 4.3 Output—Item Information .................................................................................. 24 4.4 Output—Other Theta Estimators ........................................................................ 25 4.5 Output—Plot Item Category Probabilities .......................................................... 26 5. Save Files ...................................................................................................................... 27 5.1 Save Files—Items Administered ........................................................................ 28 5.2 Save Files—Theta History .................................................................................. 29 5.3 Save Files—SE History ...................................................................................... 30 5.4 Save Files—Final Theta/SE ................................................................................ 31 5.5 Save Files—Other Theta Estimates .................................................................... 32 5.6 Save Files—Likelihood Distributions................................................................. 33 5.7 Save Files—Posterior Distributions.................................................................... 34 5.8 Save Files—Item Information Functions ............................................................ 35 5.9 Save Files—Full Length Theta Estimates........................................................... 36 5.10 Save Files—Selected Item Responses .............................................................. 37 6. Working with R............................................................................................................. 38 6.1 Generating R Code .............................................................................................. 38 6.2 Running R ........................................................................................................... 39 6.3 Basic Graphical Output ....................................................................................... 40 Appendix A. Starting Criteria ........................................................................................... 41 Appendix B. Item Selection Criteria................................................................................. 42 Appendix C. Interim Theta Estimation ............................................................................. 43
2
Appendix D. Exposure Control......................................................................................... 44 Appendix E. Stopping Criteria.......................................................................................... 45 Appendix F. Final Theta Estimation ................................................................................. 46 Acknowledgments............................................................................................................. 47
3
1. Overview FIRESTAR is a computer program for simulating computerized adaptive testing (CAT) with polytomous1 items. The item response theory (IRT) models supported by the program include Samejima’s graded response model (GRM) and Muraki’s generalized partial credit model (GPCM). Both Master’s partial credit model (PCM) and Andrich’s rating scale model are also supported as special cases of GPCM. The program provides various item selection techniques, stopping criteria, final theta estimators, and output files. The back-end engine of the program was developed in R (http://cran.r-project.org/) and the front-end user interface in C#. R is a system for statistical computation and graphics and is freely available under the GNU “copy-left” license agreement.
1
The program also supports dichotomous items. The two-parameter logistic model is supported as a special case of the graded response model (GRM) or the generalized partial credit model (GPCM) for two ordered categories. The Rasch model (one-parameter logistic) is also supported as a special case of the two-parameter logistic model with the slope parameter set to 1.0.
4
2. Installation Instructions2 To begin installing FIRESTAR double-click the file “setup.exe”. The installer will prompt you to specify a location for the program files and to select the scope of the installation. The setup program installs three sample input files (.csv) in the program directory you designate. If you wish to uninstall the program, this is accomplished through Add or Remove Programs located under Window’s Control Panel. The setup program will not install shortcuts automatically. You can create these manually by right-clicking the target file and moving the shortcut to the desired location. The following screenshots show the sequence of dialogs and prompts through which you will be guided as the program is installed. (1)
2
This program is designed to run under Windows XP®. It may also run under previous versions of Windows® with some DLLs installed. When attempted to run under previous versions, Windows® may prompt the user to install the necessary DLLs from their website.
5
(2)
(3)
6
(4)
Once the setup process is completed, the program directory should look similar to the following.
7
3. Configuring the CAT 3 .1 L a u n c h in g t h e P r o g r a m Double-clicking the program icon or its shortcut will launch the following splash window. Click the button labeled “Run R Command Generator” to open the CAT configuration (simulation settings) window.
8
3.2 CAT Configuration—Input Item Parameter File The only required input file is a file containing IRT item parameters. Clicking the rectangular box next to “Item Parameter File” will pop up a file open dialog window. The program expects a comma separated (.csv) file containing item parameters and no column headings. Values of the variables must appear in this exact order: a, cb1, cb2,…,cb(ncat-1), and ncat, where a is the discrimination parameter;cb1, cb2,…,cb(ncat-1) are the category parameters, and ncat is the number of response categories for each item. Items for a single simulation may have different numbers of response categories.
9
3.3 CAT Configuration—Input Item Response File An input file containing item responses can be read into the program by clicking the space next to “Item Response File.” The program expects a comma separated file (.csv) containing item responses and no column headings. The number and order of columns in the file must match the items in the required item parameter file. The lowest response category must be assigned a value of 1 (not 0). Missing values are allowed and should be represented as blanks3. You must either specify a file to import or tell the program to simulate item responses by checking the check box labeled “Simulate Theta / Data.” The response simulation function is described more fully in Section 3.11.
3
i.e., two consecutive commas (,,) with or without a blank space in between and dot (.) is not allowed.
10
3.4 CAT Configuration—Input External Theta File An input file containing external theta estimates can be specified by clicking the space next to “External Theta File.” The program expects a comma separated file (.csv) containing theta estimates in a single column with no column heading. If no theta estimate file is imported the user must tell the program to calculate theta estimates by checking the check box labeled “Estimate Full-Length Theta” (to calculate theta estimates using EAP based on all items). If item responses are to be simulated within the program as described in Section 3.3, the user must check the Simulate Theta/Data box to instruct the program to generate full-length (full-scale) theta estimates.
11
3.5 CAT Configuration—Number of Items in the Bank In the box labeled “Number of Items in Bank,” you must specify the number of items in the input item parameter file. An error message will be generated if you attempt to run (submit) the program without supplying this information.
3.6 CAT Configuration—Maximum Number of Categories In the box provided, specify the number of response categories in the item with the most response options. For example, if half of your items have 3 categories and half have 4 categories, you would change the default value of 5 to 4. An integer value greater than or equal to 2 is expected. The program can handle dichotomous items. They are treated as polytomous items with two response options.
12
3.7 CAT Configuration—Minimum/Maximum Theta and Increment You can specify the minimum, maximum, and increment of the vector of theta values to be generated by the program. The default values for the Minimum and Maximum Theta and the Increment are -4.0, 4.0, and 0.1, respectively. The Minimum and Maximum Theta define the range of the theta continuum. The Increment defines the granularity. The vector is used as the quadrature for theta estimation (e.g., EAP), and as domain values for prior, likelihood/posterior distributions, and information functions. The precision of measurement for the final theta estimators (MAP, MLE and WLE) is also affected by the granularity of the vector of theta values. This is the case because the final theta estimators (excepting EAP) implemented in this program utilize an exhaustive search algorithm by evaluating all possible function values along the vector of theta values. Setting the Increment value too small (e.g.,
