Fuzzy Signal Detection Theory (FSDT) has the potential of enhancing ... tasks in which precision of Signal Detection Theory (SDT) analysis is limited by discrete ...
PROCEEDINGS of the HUMAN FACTORS and ERGONOMICS SOCIETY 55th ANNUAL MEETING - 2011
1366
FUZZY SIGNAL DETECTION THEORY: A MONTE CARLO INVESTIGATION James L. Szalma & Maureen E. O'Connell University of Central Florida Fuzzy Signal Detection Theory (FSDT) has the potential of enhancing performance measures in detection tasks in which precision of Signal Detection Theory (SDT) analysis is limited by discrete mutually exclusive categorization of the state of the world and/or responses available to the observer. While there have been empirical efforts to demonstrate the benefits and tenability of FSDT, the question still remains whether traditional SDT computational procedures for measures of sensitivity and bias can be used with FSDT procedures. Through the use of Monte Carlo simulation and ROC analysis, the current study examined whether data analyzed by FSDT met the assumptions of traditional SDT on which sensitivity and bias measures are predicated. The results indicated that FSDT does in fact meet the normality and equal variance assumptions of SDT. However, the results also indicated that further theoretical elaboration of ‘fuzzy criterion setting’ is necessary.
Copyright 2011 by Human Factors and Ergonomics Society, Inc. All rights reserved DOI 10.1177/1071181311551284
INTRODUCTION Signal Detection Theory (SDT) has been applied successfully to a variety of detection and decision making tasks, and it is considered by some as one of the most important developments in psychology in the last fifty years (Hancock, Masalonis & Parasuraman, 2000; Estes, 2002). There are limitations to SDT, however. As suggested by Hancock et al. (2000), it is difficult to know the true state of the world because it often varies as a function of multiple factors. As such, signals and responses in operational environments are often not dichotomous. In these situations, in which SDT requires forced categorization of signals and non-signals, there may be a loss of information regarding the ‘true’ nature of the event. Parasuraman, Masalonis, & Hancock (2000) proposed Fuzzy Signal Detection Theory (FSDT) to address this problem. FSDT combines fuzzy set theory with traditional SDT, which allows for the non-mutually exclusive categorization of the signal and response variables, thereby eliminating the limitation regarding forced discrete categorization. Through the use of mapping functions that define fuzzy set membership as a function of one or more physical variables, mixed implication set functions are applied to relate stimulus presentation to response, which generates fuzzy membership in the sets hit, miss, false alarm, and correct rejection. By summing these outcomes over trials and dividing by either the summed signal set or noise set, rates can be calculated for each outcome. These rates are then used in the same way as rates derived from traditional SDT analysis to compute sensitivity and bias measures. However, the
traditional procedures for computing sensitivity and bias are predicated on a set of assumptions (MacMillan & Creelman, 2005). Parasuraman et al. (2000) assumed that use of standard SDT formulas for sensitivity and response bias were appropriate because the fuzzy properties of the stimulus and/or the response were incorporated into the fuzzy hit and false alarm rates. Subsequent studies have supported this assumption, providing evidence that SDT measures derived from FSDT procedures conform to the core assumptions of SDT (Murphy, Szalma, & Hancock, 2004; Szalma, Oron-Gilad, Saxton, & Hancock, 2006). However, this contention has yet to be fully tested. Purpose of Study The purpose of the present study was to test the assumptions of traditional SDT for Fuzzy Signal Detection Theory FSDT using Monte Carlo simulation methods. Simulated random variables were therefore used to generate a data set to test via ROC analysis whether FSDT procedures conform to the assumptions of SDT. METHOD Monte Carlo methodology allows for the simulation of mathematical and statistical systems through the use of pseudo random numbers. By using a large number of replications, stable parameter estimates can be computed. In the present study, random variables corresponding to aspects of the signal detection model
PROCEEDINGS of the HUMAN FACTORS and ERGONOMICS SOCIETY 55th ANNUAL MEETING - 2011
were generated in order to simulate numerous participants completing a large number of trails of a signal detection decision making task. The specific task simulated was the air traffic controller/aircraft distance example described by Parasuraman et al. (2000). The simulation was computed with normally distributed aircraft distances. Signal and response data were generated through five steps. These five steps were replicated 100 times to represent a hundred participants performing the task. In the first step of the simulation 500,000 random aircraft separation distance values were generated. In the second step this population of values was sampled 1,000 times representing a 1,000 presentation trails to a ‘participant.’ Step three involved introduction of ‘error’ variation into the sampled data, first by creating a 1% chance to miss a signal to simulate errors of omission, and secondly by introducing a new random variable, the perceived stimulus value by the participant. This former variable simulated observation omissions by the participant, i.e., the failure to make an observing response (Holland, 1958). This latter variable simulated the errors in perception that occur during observation, i.e., the difference between observer's perceived value of the stimulus and stimulus' true value. The perceived stimulus value was a normally distributed random variable with a mean equal to the sampled stimulus value and a standard deviation of 0.5. In step four, signal and noise trials were defined by determining whether the sampled data value was less than five, in which case it was defined as a signal. Values greater than five were defined as noise. In step five, two kinds of response data was created, a binary response for SDT and FSDT, and a categorical response for FSDT. In each case the general method for generating response data was to compare three randomly generated criteria values to the perceived stimulus values generated earlier. Each criterion differed in terms of its leniency, and how the
1367
criteria were generated differed between the binary and categorical cases. The three criteria for the binary case were generated using normally distributed random variables with means equal to 5.5, 6, and 6.5 and standard deviation of 1. The three criteria for the categorical case were a set of three criteria each indicating the border between categories, and were also normally distributed random variables. After generating the signals and the responses of the simulated participants, hit and false alarm rates were computed using both SDT and FSDT methodology for the binary case, and FSDT methodology for the multiple category case. ROC analysis was computed on both SDT and FSDT hit and false alarm rates, the former to verify whether the generated data was indeed analyzable by SDT methods, and the latter to determine whether FSDT data meets the assumptions of SDT. RESULTS Wickens' (2002) ROC analysis software (ROCFit) was used to derive the ROC functions. Because this software only permits entry of whole number frequencies for hits, false alarms, misses, and correct rejections, for FSDT analysis the frequencies were rounded to the nearest integer. Tables 1 and 2 report the goodness of fit (χ2) statistics as compared to the equal variance Gaussian Signal Detection model, and estimates for ROC function parameters (a and b) perceptual sensitivity (Az) and response bias (βlog). Only the parameters for the equal variance model are reported, as for all conditions the analysis fit this model. The resulting ROC analysis of the SDT data for revealed a linear ROC (in z-score form) with a slope of one, indicating that the data conformed to both the equal variance assumption of SDT as well as the assumption of normality (see Table 1).
PROCEEDINGS of the HUMAN FACTORS and ERGONOMICS SOCIETY 55th ANNUAL MEETING - 2011
1368
Table 1 Displays goodness of fit, sensitivity, and criterion bias calculated for both SDT and FSDT analysis of normally distributed data.
Analysis
χ2
A(z)
a
b
Conservative βlog
Unbiased βlog
Lenient βlog
SDT Binary
0.607
0.95
2.321
1
-0.130
0.544
1.165
FSDT Binary
0.061
0.765
1.023
1
-1.261
-1.023
-0.789
FSDTCategorical
0.316
0.948
2.295
1
-0.729
-1.304
1.836
The resulting ROC analysis of the FSDT data for the two response sets indicated that FSDT does meet the assumptions of the traditional SDT model. The zscore form ROCs for both the uniform and normal distributions were linear with a slope of 1, indicating that the normality and equal variance assumptions, respectively, were met. DISCUSSION In general, the results of the study demonstrated that the method of data generation using Monte Carlo simulation can be used to generate data for ROC analysis. Further, these results provide further evidence indicating that FSDT analysis meets the assumptions of traditional SDT. Hence, the results of this study supported the contention by Parasuraman et al. (2000) that sensitivity and response bias can be computed from fuzzy hit and false alarm rates using the formulas of traditional SDT. Although the results indicate that FSDT did meet the assumptions of SDT, FSDT analysis resulted in a lower perceptual sensitivity when compared to the sensitivity measure of SDT for the binary response sets. This may be due to a loss of information when binary response mappings are used, with a corresponding loss of sensitivity of the FSDT indices. When the response set was extended using a multiple category response set
there was an increase in the perceptual sensitivity of FSDT analysis. Consistent with previous empirical studies (Murphy et. al, 2004; Szalma et al. 2006), the three points generated by the simulation for estimating the ROC function were close to one another in value. This effect seems to be a consistent property of fuzzy ROCs, but the reason for such a pattern is unclear. It may be that it is difficult for observers to adopt a fuzzy criterion, causing the differences between points to be attenuated. Future empirical research should seek to develop payoff matrices or instructional sets to aid observers in adopting the desired criterion levels in FSDT tasks. In sum, the present simulation study confirmed the results of prior empirical research, indicating that the derivation of fuzzy hit and false alarm rates can be used to compute the traditional measures of sensitivity and response bias. Application of FSDT to complex signal detection and decision making tasks has the potential to more accurately model performance in cases where decisions can be non-binary (e.g., in determining the level of threat or risk). A challenge for application will be the development of ‘user friendly’ methods for applying the FSDT model to data.
PROCEEDINGS of the HUMAN FACTORS and ERGONOMICS SOCIETY 55th ANNUAL MEETING - 2011
REFERENCES Box, G.E.P, and Muller, M.E. (1958). A Note on the Generation of Random Normal Deviates. The Annals of Mathematical Statistics. 29, 2. 610–611. Estes, W. K. (2002). Traps in the route to models of memory and decision. Psychonomic Bulletin & Review, 9, 3–25 Hancock, P.A., Masalonis, A.J., & Parasuraman, R. (2000). On the theory of fuzzy signal detection: Theoretical and practical considerations. Theoretical Issues in Ergonomic Science, 1, 207-230. Holland, J.G., (1958) Human vigilance. Science, 128, 61-67. Kadlac, H. (1999). Statistical properties of d' and β estimates of signal detection theory. Psychological Methods, 4(1), 23-43. Macmillan, N.A., & Creelman, C.D. (2005). Detection theory: A user’s guide, 2nd ed. Mahwah, NJ: Erlbaum. Masalonis, A.J., Parasuraman, R. (2003) Fuzzy signal detection theory: Analysis of human and machine performance in air traffic control, and analytic considerations. Ergonomics, 46(11), 1045-1074. Murphy, L., Szalma, J.L., & Hancock, P.A. (2004). Comparison of fuzzy signal detection and traditional signal detection theory:Analysis of duration discrimination of brief light flashes. Proceedings of the Human Factors and Ergonomics Society, 48, 24942498. Parasuraman, R., Masalonis, A.J., & Hancock, P.A. (2000). Fuzzy signal detection theory: Basic postulates and formulas for analyzing human and machine performance. Human Factors, 42, 636-659. Stafford, S.C., Szalma, J.L., Hancock, P.A., & Mouloua, M. (2003). Application of fuzzy signal detection theory to vigilance: The effect of criterion shifts. Proceedings of the Human Factors and Ergonomics Society, 47, 1678-1682. Swets, J.A., Dawes, R.M., Monahan, J. (2000). Psychological science can improve diagnostic decisions. Psychology Science in the Public Interest, 1 (1), 1 -26. Szalma, J.L., Oron-Gilad, T., Saxton, B., & Hancock, P.A. (2006). Application of fuzzy signal detection theory to the discrimination of morphed tank images. Proceedings of the Human Factors and Ergonomics Society, 50, 1716-1720. Triesman, M., & Williams, T.C. (1984). A theory of criterion setting with an application to sequential dependencies. Psychological Review, 91 (1), 68-111. Wickens, T.D. (2002). Elementary Signal Detection Theory. Oxford University Press.
1369
