Applying fuzzy linear regression to understand ...

Applying fuzzy linear regression to understand metacognitive judgments in a human-in-the-loop simulation environment Jung Hyup Kim, Ling Rothrock, Senior Member, IEEE, Anand Tharanathan  Abstract—In this paper, a new approach to evaluate metacognitive activities is investigated using fuzzy linear regression analysis. Metacognition shows a broad picture of learning competencies that significantly influences learning processes such as confidence judgment and control of learning. However, it is hard to detect changes in metacognitive judgments because there is no direct way to evaluate metacognition while individuals are learning a new task. We investigated the internal relationship between an individual’s metacognitive judgments and task performance. Participants performed a radar monitoring task by playing the role of an Anti-Air Warfare Coordinator (AAWC) in a Human-In-The-Loop (HITL) simulation. In order to measure task performance, participants were given a situation awareness (SA) probe. To measure their metacognitive judgments, we administered a Retrospective Confidence Judgments (RCJ) probe. A fuzzy linear regression model was used to analyze the relationship between RCJ and SA. There were three groups in this experiment. The first group (SA+RCJ feedback) viewed their SA performance with the correct answers to all SA questionnaires and triangular graphs of both SA confidence and SA scores together. The second group (SA feedback) only watched their SA performance with the correct answers to all SA questionnaires. The third group was the control group and it did not observe any feedback. The results showed that the SA + RCJ feedback screen could significantly influence the participants’ mental state from over confidence (OC) to under confidence (UC) as well as SA accuracy. Using the outcomes of the experiment, we modeled mental state change in metacognitive judgments using fuzzy linear regression. Index Terms— Fuzzy Linear Regression, Metacognition, Over Confidence, Under Confidence, Situation Awareness, Human-In-The-Loop Simulation

I. INTRODUCTION etacognition is defined as thoughts about thoughts or cognition about cognitions [1]. It is a way that learners think about their cognition processes [2]. Metacognition has been a popular research topic since its inception by Flavell years ago [3]. However, most measurements of metacognitive activities pertain to educational applications, such as childhood learning [4] and academic performance [5]. Development of

M

Jung Hyup Kim is Department of Industrial and Manufacturing Systems Engineering, University of Missouri, Columbia, MO, 65211, USA, e-mail: [email protected]. Ling Rothrock is with Department of Industrial and Manufacturing Engineering, Pennsylvania State University, University Park, PA, 16802, USA, e-mail: [email protected]. Anand Tharanathan is with Accenture, Chicago, IL, 60601, USA, e-mail: [email protected].

monitoring mechanisms in metacognitive judgments has not received much attention within the metacognitive research community. Hence, we investigated a new approach to detecting changes in metacognitive activities by using Fuzzy Linear Regression (FLR) analysis. The main objective is to explore an FLR model between Retrospective Confidence Judgments (RCJ) and situation awareness (SA). A Human-In-The-Loop (HITL) simulation is used to collect the individual’s SA accuracy and his or her RCJ score regarding SA in real time. FLR is one of the techniques to analyze the linear relation between non-fuzzy variables and fuzzy variables. In this technique, the uncertainty of human decision making is modeled as fuzziness rather than random errors [6]. It is useful for understanding the vague relationship between dependent and independent variables. Because the subject confidence judgments of SA is a second order thought about SA, which is a first order thought, applying a fuzzy variable is better to represent the confidence judgments about the accuracy of SA. A fuzzy variable is defined as ̂ Y = (𝒚, 𝒆), where y is the trend of center value and e is the ambiguity in the variable (the trend of spread). In order to analyze between fuzzy data and non-fuzzy data by using the FLR technique, a fuzzy coefficient ̂ ) and input is used. The output variable (fuzzy number: Y variable (a set of non-fuzzy data points x ) are expressed by the form

̂𝒊 𝐱𝒊 𝐲̂𝒊 = 𝑪

(1)

Where 𝐲̂𝒊 is the fuzzy output variable, such as the trainees’ ̂𝒊 self-evaluation of their performance (second order thought), 𝑪 is the fuzzy coefficient, and 𝐱 𝒊 is the real-valued input variable, such as the trainees’ situation awareness performance (first order thought), with i = 1…n, and n = number of ̂ 𝒊 , is called the participants. Here, the fuzzy coefficient 𝑪 ̂ Self-Regulated Fuzzy index (SRF: 𝑪𝒊 = 〈𝐶𝑖 , 𝑆𝑖 〉). This can be converted to the problem of finding the vectors 𝐶𝑖 (the trend of center: presenting the individual’s current relationship between second order thought and first order thought), and 𝑆𝑖 (the trend of spread: presenting the individual’s current fuzziness between second order thought and first order thought), such that 𝐲̂𝒊 is given by equation (1).

A. Metacognition Researchers have found that problem-solving activities require self-regulated learning (relying on metacognition) to be aware of the problem itself and to manage one’s cognitive processes [7-9]. According to Newell, the first step of problem solving is encoding the given elements [10]. This process involves identifying the most informative features of a problem. Metacognitive activities are able to promote this encoding process because these activities are the mental representation which help to draw a “mental map” of the problem’s elements and the relationship between the elements when information from the problem is mentally inserted, deleted, interpreted, or held in memory [11]. To solve problems, a person should know how to use encoded data and the mental mappings to reach the desired goal. Therefore, people normally choose to update their plans based on previous success and what opportunities are still available. The speed of internal cognitive processing can be accelerated by metacognitive activities [12]. This means that metacognitive processes can guide goal-directed thinking when people are learning about or solving their problems. Accurate metacognitive monitoring of learning performance is important to improve a student’s academic performance, according to several classroom studies. Hacker, Bol, Horgam, and Rakow reported that the poorest performers were the most overconfident, and that this group had the largest deviation between predicted and actual scores [13]. According to their research, a negative relationship existed between the prediction errors regarding the test results and the actual test scores. They concluded that the main reason some students obtained bad results was that those who were in the poorest performers group were not only unskilled but also unaware. Poor performers’ lack of skill reduces not only their ability to detect the correct responses, but also their ability to surmise what they are doing wrong [14]. For this reason, students who are consistently overconfident or underconfident in learning a new task may use ineffective strategies, resulting in a large deviation between their understanding and actual performance. Although there are many extraneous factors to assess the influence of metacognitive activities, several studies have shown the effect of metacognitive activities on training. Teong’s study showed that the experimental students, who were trained with the ability to make metacognitive decisions, outperformed the control students with the ability to solve word problems [15]. Also, the results from Veenman’s research revealed that metacognitive skillfulness was positively related to learning behavior and to the scores on a qualitative test during inductive training with a complex computer simulation [16]. Several metacognitive models have been developed to understand and evaluate trainees’ performance on a learning task [17-19]. Among them, we focus on improving trainees’ knowledge seeking skills by using metacognitive judgment feedback in an HITL simulation environment. Feedback can prompt trainees to think about their previous decisions. This cognitive activity helps them to understand their previous mistakes and to reallocate tactical resources based on capabilities and limitations from their experience [20]. However, the need still exists to develop effective training

methodologies because there is no direct way to evaluate metacognition during dynamic control tasks. Nelson developed a general framework of metacognitive processes [21]. This framework is conceptualized by the interplay of two levels: the Meta-Level and Object-Level. The Object-Level represents process of ongoing cognitive activities, such as learning and problem solving. The Meta-Level represents the subject’s understanding of learning and problem solving which is related to the ongoing cognitive processes at the Object-Level. In the general framework, metacognitive monitoring and control are the main process activities that communicate between the two levels. Metacognitive monitoring activities update the model in the Meta-Level while metacognitive control processes influence the modification at the Object-Level. Based on this general framework, we asked participants in our study to answer SA questionnaires for the Object-Level. The SA questionnaires were developed based on Endsley’s model of situation awareness [22]. For the Meta-Level, RCJ scores were collected to measure the participants’ second order thoughts regarding their SA performance. The RCJ scores represent participants’ SA self-confidence level that was measured before they were aware of their SA performance. This SA performance came from metacognitive monitoring processes associated more directly with retrieval [23]. B. Fuzzy Linear Regression (FLR) Statistical linear regression has been used to explain the variation of a dependent variable based on the variation of explanatory factors. However, in operational contexts, human estimations are ambiguous which makes it difficult to find a strong relationship between human judgments and task performance. To compensate, we model the uncertainty of human decisions in a regression as a fuzziness factor. FLR is useful to understand the relationship between the fuzzy dependent and non-fuzzy independent variables [24]. A general form is ̃ Y = 𝑓(𝐴̃, 𝑥) = 𝐴̃0 + 𝐴̃1 𝑥1 + 𝐴̃2 𝑥2 + ⋯ + 𝐴̃𝑛 𝑥𝑛

(2)

̃ is a vector of fuzzy coefficient, x is n-dimensional set of A ̃ is the non-fuzzy input (control or independent) variables and Y fuzzy predictor of output (dependent or observed) variable. There are different shapes of fuzzy membership functions, such as symmetric triangular, asymmetric triangular, trapezoidal, and bell shaped membership functions. In this research, all of the fuzzy components are assumed to be a symmetrical triangular fuzzy numbers (STFN) to serve as a common starting point.

2) The sum of the spreads of all of the fuzzy coefficient should be minimized [29]. Individuals will make a better decision if they can minimize the sum of the trend of spread in the fuzzy coefficient II. SELF-REGULATED FUZZY INDEX

Fig. 1. Fuzzy Coefficient

There are two trends in a fuzzy coefficient: the trend of spread (S) and the trend of center (C). C represents the general linear relation between dependent and independent variables in the system. S represents the degree of fuzziness between fuzzy and non-fuzzy data. The membership function μ𝐴̃ for each of the coefficients is expressed as μ𝐴̃ (𝑎𝑖 ) = {

1− 0,

|𝐶𝑖 −𝑎𝑖 | 𝑆𝑖

, 𝐶𝑖 − 𝑆𝑖 ≤ 𝑥𝑖 ≤ 𝐶𝑖 + 𝑆𝑖 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒

(3)

Where 𝑎𝑖 is ith coefficient corresponding to a vector of values of real-valued non-fuzzy data with i = 1,…,n, and n = number of data points, μ𝐴̃ (𝑎𝑖 ) is the response fuzziness which is influenced by C. In other words, μ𝐴̃ (𝑎𝑖 ) represents the degree of the confidence in human estimated judgments regarding ̃= their cognitive activities. The fuzzy coefficient 𝑨 ̃ ̃ ̃ {𝑨𝟏 , … , 𝑨𝒏 } can be denoted in the vector form of 𝑨 = {𝑪, 𝑺}, where C = (c1, …, cn) and S = (s1, …, sn). Therefore, the output is the revised version of equation (2). ̃ Y = (𝑐1 , 𝑠1 )𝑥1 + (c2 , 𝑠2 )𝑥2 + ⋯ + (c𝑛 , 𝑠𝑛 )𝑥𝑛

(4)

In this study, metacognitive activities between the Meta-Level and Object-Level are measured by the fuzzy coefficient. The general fuzzy linear regression model is: 𝑌̃ = 𝐶̃ ∙ 𝑥

(5)

Where the control variable x is non-fuzzy data points of individual’s SA accuracy, 𝐶̃ is symmetric triangular fuzzy coefficient with trends of center and of spread, 𝑌̃ is a symmetric triangular fuzzy number corresponding to the second order thought of the control variable x. For the fuzzy coefficient 𝐶̃ , the criteria of goodness of fit are below: 1) The fuzzy set 𝑌̃ is based on the h-cut of each observed data. The h-cut is defined as a rectangular membership function that satisfies ℎ ⋀ 𝜇𝑌̃ℎ (𝑥). The membership function should be contained in the h-cut of the predictor 𝑌̃ where 0 ≤ ℎ < 1. The main difficulty for using this criterion is that we do not know how to choose h [25]. Based on previous work [26-28], we assume that h = 0.5. It means 𝑌̃ℎ = {𝑥|𝜇𝑌̃ (𝑥) > 0.5}

To evaluate the linear relation between the Meta-Level and the Object-Level, equation (6) shows a fuzzy linear regression model related to the fuzzy parameter and crisp data [6]. In this study, the fuzzy parameters 𝐶̂𝑖 is called Self-Regulated Fuzzy ̂ 𝒊 =< 𝐶𝑖 , 𝑆𝑖 > ). It can be converted to the index (SRF: 𝑪 problem of finding the vectors 𝐶𝑖 (the trend of center: presenting ith participant’s current self-regulated state) and 𝑆𝑖 (the trend of spread: presenting ith participant’s current fuzziness between the Meta-Level and Object-Level) such that ŷ𝑖 given by equation (1). The described fuzzy regression problem can be formulated in terms of the following classical linear programming problem [6] 𝑀𝑖𝑛 𝑠. 𝑡.

𝑆𝑖 𝑇 (1 − ℎ)𝑆𝑖𝑇 |𝑎𝑖𝑗 | − |𝑏𝑖𝑗 − 𝑎𝑖𝑗 𝐶𝑖 | ≥ 0

(6)

𝑆𝑖 , 𝐶𝑖 , 𝑎𝑖𝑗 , 𝑏𝑖𝑗 ≥ 0 𝑖 = 1, … , 𝑛; n = number of participants 𝑗 = 1, … , 𝑚; 𝑚 = 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑡𝑟𝑖𝑎𝑙𝑠 For each given datum , where aij is a vector of values of SA accuracy, bij is a vector of values of RCJ scores. The datum belongs to the corresponding x𝑖 = {𝑎𝑖1 , … , 𝑎𝑖𝑗 }, and ̂𝒊 = {< 𝑏𝑖1 , s𝑖 >, … , < 𝑏𝑖𝑗 , 𝑠𝑖 >} with a grade fuzzy output 𝒚 that is greater than or equal to some given value h ∈ [0,1]. s𝑖 is corresponding to the fuzziness for ith participant between second order thought (bij) and first order thought (aij). As h increase, the fuzziness of the output increases [30]. During the experiment, all participants performed a set of scenarios for two days. It means that we collected 𝐲̂𝒊 = {< 𝑏𝑖1 , 𝑠𝑖 >, < 𝑏𝑖2 , 𝑠𝑖 >, < 𝑏𝑖3 , 𝑠𝑖 >, < 𝑏𝑖4 , 𝑠𝑖 >} and x𝑖 = {𝑎𝑖1 , 𝑎𝑖2 , 𝑎𝑖3 , 𝑎𝑖4 } on Day 1 of the session, 𝐲̂𝒊 = {< 𝑏𝑖5 , 𝑠𝑖 >, < 𝑏𝑖6 , 𝑠𝑖 >, < 𝑏𝑖7 , 𝑠𝑖 >, < 𝑏𝑖8 , 𝑠𝑖 > } and x𝑖 = { 𝑎𝑖5 , 𝑎𝑖6 , 𝑎𝑖7 , 𝑎𝑖8 } on Day 2 of the session from every participant. Figure 2 shows the membership function for the fuzzy output. The fuzzy output set, which contains yi, is associated with a membership value greater than h. μ(y𝑗 ) ≥ ℎ, 𝑗 = 1, … , 𝑚

(7)

Based on equation (7), the fuzzy output should lie between A and B of Figure 2. In this model, 𝑆𝑖 represents the current fuzziness between ŷ𝑖 and 𝐱 𝒊 for ith participant. The variation of the individuals retrospective confidence judgments are significantly influenced by the vector of values of real-valued SA accuracy. Hence, the trend of the spreads ( 𝑆𝑖 ) in the SRF index indicates the consistency of the self-evaluation in their situation awareness related to the given tasks. The objective of this regression model is to determine the optimum fuzzy coefficient 𝐶̂𝑖 . For the case of the non-fuzzy

independent (SA accuracy) and the dependent (RCJ score) observed response variable, each datum is translated into the following constraints: 𝑙 ∑𝑚 (8) 𝑗=1 𝐶𝑖 𝑋𝑖𝑗 + (1 − ℎ) ∑𝑗=1 𝑆𝑖 |𝑋𝑖𝑗 | ≥ 𝑌𝑖 + (1 − ℎ)𝑒𝑖 𝑙 ∑𝑚 𝑗=1 𝐶𝑖 𝑋𝑖𝑗 − (1 − ℎ) ∑𝑗=1 𝑆𝑖 |𝑋𝑖𝑗 | ≤ 𝑌𝑖 − (1 − ℎ)𝑒𝑖

Other symbols represent ground objects. Figure 4 shows the simulation user interface.

(9)

𝑚 The center value ∑𝑚 𝑗=1 𝐶𝑖 𝑥𝑖𝑗 and the spread ∑𝑗=1 𝑆𝑖 |𝑥𝑖𝑗 | are obtained by considering the equation 8 and 9, where h is specified by the user.

Fig. 3. Visual representation of the perfect self-regulated (SR) state between RCJ and SA accuracy

TABLE I AIRCRAFT SYMBOLS IN AAWC SIMULATION Unknown

Fig. 2. Fuzzy Output Function

Based on previous work [26-28], we assume that the fuzzy parameters are symmetric triangular numbers with ℎ = 0.5. For the SRF index, the trend of center 𝐶𝑖 could represent the participant ith’s current self-regulated state between the Meta-Level and Object-Level. For example, if 𝐶3 is equal to 1, then the participant (i=3) can perfectly predict his or her SA accuracy before knowing the result. This is called the perfect self-regulated (SR) state. If the participant is in this perfect SR state, then he or she is an ideal self-regulated learner. If 𝐶3 is much larger than 1, then his or her current state is overconfident about his or her situation awareness. This is called over confident state. If 𝐶3 is much smaller than 1, then this means that the participant (i=3) is underconfident for his or her skill. This state is called under confident state (see Figure 3). III. METHODS A. Materials and Apparatus A radar-monitoring human-in-the-loop (HITL) training simulator was used to collect data. The simulator, called AAWC, required participants to take the role of an Anti-Air Warfare Coordinator. The AAWC is similar to an air traffic controller simulation within the context of military command and control. Participants were required to identify unknown objects and take appropriate actions according to specified Rules of Engagement (ROE). In other words, participants were asked to control their resources while performing the tasks during the simulation. This system is used to simulate a complex command-and-control task environment, which promotes decision-making under dynamic task conditions [31]. Table 1 shows the aircraft symbols of the AAWC simulation.

Hostile

Friendly

Own ship

Symbol of aircraft

The primary objective of AAWC is to protect one’s ship from hostile aircraft. If an unknown or hostile aircraft appears on the radar screen, the operators must perform appropriate actions according to specified ROE. The ROE consists of the following: 1) Identification Rule (Unknown aircraft only) - Make a primary identification of unknown aircrafts (i.e., friendly, hostile) - Make an AIR identification of air contact (i.e., Military aircraft, Commercial aircraft, Missile, Helicopter) 2) Warning Rule (Hostile or Unknown aircraft only) - Issue Level 1 Warning if the aircraft is between 40 – 50 nautical miles from one’s ship - Issue Level 2 Warning if the aircraft is between 30 – 40 nautical miles from one’s ship - Issue Level 3 Warning if the aircraft is less than 30 nautical miles from one’s ship 3) Assign Rule (Hostile or Unknown aircraft) - Activate the Engaged Status if the aircraft is less than 30 nautical miles away and after a Level 3 Warning has been issued. The first rule is the identification rule. If the operator perceives a symbol of the unknown aircraft on the radar screen, he or she must identify its primary identification and AIR identification. There are two categories of primary identification. One is “Friendly Aircraft”. Another one is “Hostile Aircraft”. The operator must choose one of them in order to execute this primary identification task. For the AIR identification, there are four categories: helicopter, commercial aircraft, military aircraft, and missile. Based on the information

Fig. 4. AAWC simulation interface

from the radar screen, the operator must select one of the categories. The second rule is the warning rule. The operator must give several warning signs to the unknown or hostile aircraft if they are closer than 50 nautical miles (NM) from his/her ship. If the distance between them is 40 – 50 NM, then the operator must issue a level 1 warning. If the distance is 30 – 40 NM, then the operator must issue a level 2 warning. The final warning (level 3 warning) must be issued when the unknown or hostile aircraft is closer than 30 NM. The last rule is the assign rule. If the operator observes an offensive movement from the aircraft and the operator already issued the level 3 warning, then the engaged status of this aircraft should be changed from OFF to ON. During the experimental sessions, the participants were asked specific SA questions regarding each test scenario. This technique is called Situational Awareness Global Assessment Technique (SAGAT). It is the most well-known measure of situation awareness and is designed for real-time human-in-the-loop simulations such as measuring driver’s situation awareness [32] or nurse’s decision-making [33]. There are three levels of Situation Awareness: Level 1 SA (L1 SA) represents perception of the elements in the environment within a volume of time and space; Level 2 SA (L2 SA) represents comprehension of the current situation; and Level 3 SA (L3 SA) points toward a projection of the future status. The followings are examples of SA questions:   

L1 SA Question #1 -- What was the location of TN6026? (Within 20NM, 30NM, 40NM, or 50NM) L2 SA Question #2 -- What was the primary identification of TN6023? (Friendly or Hostile) L3 SA Question #3 -- TN6021 will cross the Koraban border in the near future. (True, False)

In addition, participants also answered an RCJ probe. The RCJ score was scaled from 1 to 100. The score was collected as the metacognitive judgment of participants’ SA accuracy. A score of 1 represents a failure to answer SA probes while a score of 100 represents the perfect response to SA probes. During the practice session, participants learned how to respond to this

RCJ probe and experienced through multiple practice scenarios. The following is the probe for the RCJ rating: 

"How well do you think you are aware of the situation regarding the objects and events in your airspace?”

B. Participants A total of 48 students, 20 – 34 years of age, participated. The volunteers were screened for prior experience. So, any volunteers with radar monitoring tasks were excluded. A pilot study with 24 undergraduate and graduate student volunteers was conducted in order to identify possible problems regarding the interface, and to set the levels for the complexity factor discussed below. C. Procedure The experiment consisted of two sessions: a practice session and the actual experiment. Participants completed a screening questionnaire to assess their previous experience with radar monitoring environments. The initial training session (Day 0) lasted for 60 minutes. During this session, the participants learned: 1) the specific functions of the simulator, 2) the rules of engagement, 3) air track abbreviations, 4) how to identify a track, 5) how to use resources in track identification, and 6) how to respond to SA and RCJ probes. Following each lessons participants completed a 5-minute practice scenario, answering SA and RCJ questionnaires. At the end of each practice exercise, participants received instructor feedback regarding their ROE tasks, SA results, and RCJ scores. The total number of ROE events in the practice scenario was 12. Participants who completed three practice scenarios were considered ready to engage in the actual experiment. For the actual experiment, each participant was assigned to one feedback group. The two 90-minute experimental sessions were spread across 2 days (Day 1 and 2). On each day, each participant completed four scenarios. During each test scenario, the participants were

Fig. 5. Feedback screen for experimental groups

asked specific situation awareness questions after the simulation was frozen automatically at a random time between 10 and 15 minutes from the beginning of the scenario start time. Then, they were also asked to respond to the RCJ probe. After that, they observed different styles of feedback screens (SA+RCJ or SA) based on their assigned group. The control group, however, did not receive any feedback. D. Independent Variables 1) Feedback Two feedback types were tested: SA+RCJ feedback and SA feedback. The control group did not receive any feedback. For the SA+RCJ feedback group, the feedback screen contains the participant’s responses and the correct answers for all SA probes. In addition, the screen showed two symmetric triangles. One was related to the score of SA accuracy, and another one was related to the score of RCJ shown as feedback component (a), (b), and (c) in Figure 5. For the SA feedback group, the feedback screen contained the participants’ responses and the correct answer for each of the SA probes (feedback component (b) and (c) in Figure 5). 2) Task Complexity Each participant experienced two levels of task complexity (low versus high). Task complexity was defined as the total number of required ROE events that the operators performed within a given time period. The total number of required events in a high task complexity scenario was twice as large as a low task complexity scenario. Each participant performed eight scenarios for two consecutive experimental sessions (Day 1: 4 trials, Day 2: 4 trials). Among the eight scenarios, four scenarios were at the low complexity level, and four were at the high level. The low complexity scenarios contained 23 predefined scheduled ROE events. The high complexity scenarios contained 46 predefined scheduled ROE events.

3) Scenario and Day Sixteen scenarios (8 high complexity and 8 low complexity) were developed. In this experiment, we chose 4 low complexity scenarios and 4 high complexity scenarios based on the NASA-TLX average scores from the pilot test. NASA-TLX is a multidimensional, subjective workload rating technique, commonly used to measure an operator’s workload [34]. There was a significant effect of task complexity on NASA-TLX (high complexity >> low complexity). Each scenario was designed to run for 15 minutes. The scenario order was counterbalanced across subjects. Participants were randomly assigned to two scenario sequences. Each sequence had a different scenario order to reduce their chances of influencing the results. Twenty-four participants were assigned to each sequence. The first scenario sequence included Scenarios 1 to 8 in order and the other, Scenarios 8 to 1 in order. The even numbered scenarios were of low complexity and the odd ones of high complexity. E. Dependent Variables 1) Situation Awareness (SA) SA is defined as the awareness of the environment within time and space [35]. SA is related to understanding what is happening around an operator and what will happen in the future. For that reason, the concept of SA is usually applied to an operational situation such as driving a car or monitoring air traffic. The formal definition of SA is ‘the perception of the elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future’ [36]. The following equation calculates the accuracy of SA. 𝑆𝐴 𝑎𝑐𝑐𝑢𝑟𝑎𝑐𝑦 =

( 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑐𝑜𝑟𝑟𝑒𝑐𝑡 𝑟𝑒𝑠𝑝𝑜𝑛𝑠𝑒 ×100) 𝑇𝑜𝑡𝑎𝑙 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑆𝐴 𝑞𝑢𝑒𝑠𝑡𝑖𝑜𝑛𝑠

(10)

2) Retrospective Confidence Judgments (RCJ) The RCJ score is used for measuring a participant’s confidence

level regarding the SA accuracy. Self-rating RCJ scores were collected as metacognitive monitoring. During the practice session (Day 0), participants experienced how to respond to RCJ probes through multiple practice scenarios. For example, they learned score 10 represented 10% SA accuracy regarding the objects and events in their airspace. During the experimental sessions (Day 1 and 2), the participants responded the RCJ probe after they answered SA probes. F. Experimental Design and Data Analysis We designed a three-factor experiment on one factor where the between-subjects factor is the feedback type; within-subjects factors are the scenario complexity and day. A repeated measures ANOVA was conducted to analyze the data. To compare the difference between Fuzzy Linear Regression (FLR) and Linear Regression (LR) related to Meta and Object-Level, the means for the self-regulated fuzzy index ( 𝐶𝑓̅ = 〈𝐶̅ , 𝑆̅〉 ) from FLR and slope (𝑎̅) from LR are calculated by 𝐶𝑓̅ = 〈𝐶̅ = 𝑎̅ =

1 𝑛

∙

1 𝑛

∙ ∑𝑛𝑖=1 𝐶𝑖 , 𝑆̅ =

∑𝑛𝑖=1 𝑎𝑖

1 𝑛

∙ ∑𝑛𝑖=1 𝑆𝑖 〉

(11) (12)

General LR model (Y𝑖 = 𝑎𝑖 ∙ 𝑋𝑖 ) is used where Y is the RCJ score, X is SA accuracy, and i represents participants for i = 1,2,… 16. IV. RESULTS A. Feedback and Task Complexity For Retrospective Confidence Judgment (RCJ), the task complexity significantly influenced RCJ score [F(1,45) = 28.93,p < 0.001]. However, subjects’ confidence judgments regarding the SA were not affected by the feedback type. For SA accuracy, the feedback type significantly influenced SA accuracy [F(2,45) = 8.5, p