Gray Forecast Approach for Developing Distance Learning and ...

98

IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART C: APPLICATIONS AND REVIEWS, VOL. 37, NO. 1, JANUARY 2007

Gray Forecast Approach for Developing Distance Learning and Diagnostic Systems Gwo-Jen Hwang

Abstract—In recent years, researchers have attempted to apply network techniques to the development of computer-assisted learning systems. They have also attempted to develop more effective programs to test and improve the learning performance of students. However, the most conventional testing systems present only a score as a test result, which is not capable of providing advice to the students. To cope with this problem, we propose a gray forecast approach for modeling the relationships among the subject concepts and the test items. These relationships are then employed to diagnose the student learning problems. A testing and diagnostic system based on our approach has, therefore, been implemented. Some experimental results have demonstrated the feasibility of this approach in enhancing the students’ learning performance, making it highly promising for further study.

In this paper, we propose a gray forecast approach to cope with this problem. The relationships among subject concepts are modeled as a “prerequisite relationship” and “reciprocal relationship” based on the gray theory [4]. A learning diagnosis method is then used to detect student learning problems, based on those derived relationships. A testing and diagnostic system based on our approach has been implemented, which can analyze the answer sheets and offer learning suggestions to the students. To evaluate the efficacy of our novel approach, some experimental results on a computer course are provided, demonstrating how the new approach can help students improve their learning performance.

Index Terms—Adaptive hypermedia, computer-assisted learning, computer-assisted testing, gray theory, learning diagnosis.

I. INTRODUCTION URING the tutoring process, it is important to evaluate the status of each student, and tests are a typical evaluation method to gain this knowledge. In the past decades, researchers have shown that paper-administered and computer-administered tests are equivalent, in terms of testing quality, encouraging the development of computer-based testing systems and relevant techniques [14]. In conventional testing systems, students are assigned a score (or grade) as a test result to represent their learning status. This approach allows students to know their score (or grade) in reference to their learning status; it also means, however, that the students may be unable to improve their learning status without further guidance. Therefore, researchers have proposed a concept effect model to represent the prerequisite relationships among concepts in a course [13]. The model has been adopted to help teachers to detect student learning problems in several schools [15], [16]. However, in addition to the prerequisite relationships, there are other relationships among concepts that need to be taken into consideration. For example, in studying a computer engineering course “on data structures,” the concepts “queue” and “stack” do not have a prerequisite relationship, while the learning status of one of the concepts can significantly affect the learning of the other. Such relationships need to be modeled, to precisely diagnose student learning problems.

D

Manuscript received November 11, 2003; revised June 6, 2005. This work was supported in part by the National Science Council of the Republic of China under Contract NSC-94-2524-S-024-001. This paper was recommended by Editor V. Marik. The author is with the Department of Information and Learning Technology, National University of Tainan, Tainan City 70005, Taiwan, R.O.C. (e-mail: [email protected]). Digital Object Identifier 10.1109/TSMCC.2006.876062

II. RELEVANT RESEARCHES The issues of computer-assisted learning and testing have received lots of attention in the past decades. Earlier computerassisted learning systems focused on the provision of subject materials with multimedia presentation. In recent years, researchers have attempted to develop educational hypermedia systems which have the ability to adapt to the individual user’s needs [1], [11], [12], [21], [26]. Such systems improve basic hypermedia functionality through incorporation of intelligent tutoring techniques to enable personalization features [17]; that is, they are able to dynamically adapt the instructional sequence to the individual user knowledge level and learning goals, provide intelligent guidance, and support the users in acquiring knowledge [2]. In the following, we shall introduce several adaptive hypermedia systems and relevant techniques that support the adaptations of instructional contents. A. Adaptive Hypermedia Systems Snow and Farr [23] suggested that sound learning theories are incomplete or unrealistic if they do not include a wholeperson view, integrating both cognitive and affective aspects, which implies that no educational program can be successful without due attention to the personal learning needs of individual students. Brusilovsky [3] suggested using adaptive hypermedia to support individual learning. The idea of adaptive hypermedia is to adapt the course content for a particular learner based on the profile or records of the learner. Most of the adaptive hypermedia systems can adapt displayed information and dynamically support navigation through hypermedia material. For example, Vasandani and Govindaraj [24], [25] proposed an intelligent tutoring system that can assist operators in organizing their system knowledge and operational information to enhance operation performance. Gonzalez and Ingraham [6] developed an intelligent tutoring system which

1094-6977/$25.00 © 2007 IEEE

HWANG: GRAY FORECAST APPROACH FOR DEVELOPING DISTANCE LEARNING AND DIAGNOSTIC SYSTEMS

is capable of determining exercise progression and remediation automatically during a training session according to the students’ past performance. Harp et al. employed the technique of neural networks to model the behavior of students in the context of an intelligent tutoring system, and they used self-organizing feature maps to capture the possible states of student knowledge from an existing test database [8]. Ozdemir and Alpaslan presented an intelligent agent to guide students throughout the course material on the Internet. The agent can assist the students in learning concepts by allocating navigational support based on their knowledge levels [19]. Paolucci [20] addressed the importance of individualization in hypermedia that any strategy should be adaptive and personalized. To insure personalization, adaptive hypermedia systems should be capable of diagnosing and identifying each student’s misconceptions. Therefore, it becomes an important issue to identify student learning problems such that the adaptive hypermedia systems can assist the students in improving their learning performance accordingly. In the following, a model that is capable of diagnosing student learning problems and giving feedback to adaptive hypermedia systems has been introduced.

Fig. 1.

99

Illustrative example of the concept effect graph. TABLE I ILLUSTRATIVE EXAMPLE OF A TEST ITEM RELATIONSHIP TABLE

B. Concept Effect Model In conventional testing systems, for each student, a score is given as a result of the test and to represent the student’s learning status, which is not helpful in finding the learning problems of students. In 2003, a concept effect model was proposed to cope with this problem [13]. In the concept effect model, the subject materials are divided into concepts with some “learning order” relationships based on Salisbury’s [22] notation that learning information, including facts, names, labels, or paired associations, is often a perquisite to efficiently performing a more complex, higher level skill. For example, to learn the concept “multiplication,” one might need to learn “addition” first; before learning the concept “division,” one might need to learn “multiplication” and “subtraction.” Such learning orders or inclusion relationships can be represented as a concept effect graph as shown in Fig. 1. As the concept effect graph could be very complicated, researchers have attempted to identify possible concept effect relationships by applying a statistical approach to a series of test records and providing graphical user interfaces to assist the teacher in modifying the derived relationships [16]. 1) Constructing the Concept Effect Graph: Assume that a learning unit consists of ten concepts (C1 , C2 , C3 , . . . , C10 ), and a test sheet consisting of ten test items (Q1 , Q2 , Q3 , . . . , Q10 ) is given to a student; we have a test item relationship table (TIRT) as shown in Table I. Each value of TIRT(Qi , Cj ) represents the relationship between the test item Qi and the concept Cj , which is an integer ranging from 0 to 5 to represent no relationship, “very weak” relationship, “weak” relationship, “average” relationship, “strong” relationship, and “very strong” relationship, respectively. SUM(Cj ) represents the total strength of the concept Cj in the test sheet. For example, in Table I, only Q1 and Q6 are related to C1 ; therefore, SUM(C1 ) = TIRT(Q1 , C1 ) +TIRT(Q6 , C1 ) = 5 + 1 = 6. ERROR(Cj ) represents the to-

tal strength of incorrect answers relevant to Cj . CR(Cj ) = [SUM(Cj ) − ERROR(Cj )]/SUM(Cj ), representing the correctly answering ratio relevant to Cj . If the student fails to correctly answer Q3 , Q6 , and Q7 , we have ERROR(C1 ) = TIRT(Q3 , C1 )+TIRT(Q6 , C1 ) + TIRT(Q7 , C1 ) =0+1+0=1 CR(C1 ) = [SUM(C1 ) − ERROR(C1 )]/SUM(C1 ) = (6 − 1)/6 = 0.83. A TIRT could be generated by the system based on the concept–test item relationships given by the teachers and the test results of the students. Most of the previously proposed test systems provide a graphical interface to assist the teachers in defining the concept–test item relationships. Fig. 2 shows an illustrative example of a concept effect graph, in which each node represents a concept, each link represents a learning order or inclusion relationship between two concepts,

100

Fig. 2.

IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART C: APPLICATIONS AND REVIEWS, VOL. 37, NO. 1, JANUARY 2007

Illustrative example of a concept graph with CR values.

and the label on each node represents the corresponding CR value. 2) Finding the Enhanced Learning Paths: To diagnose the learning problems of students, we need to find out all of the possible learning paths. For example, in Fig. 2, there are five learning paths as follows: PATH1: C1 PATH2: C1 PATH3: C1 PATH4: C2 PATH5: C2

→ C3 → C3 → C4 → C4 → C5

→ C6 → C7 → C8 → C10 → C8 → C10 → C9 .

A threshold θ is used to represent the acceptable correctly answering ratio. If CR(Cj ) ≥ θ, it suggests that the student has achieved the standard of learning concept Cj ; otherwise, the assumption that the student has failed to learn the concept and, therefore, the concept is selected as a node of some to-beenhanced learning path. Assuming that θ is 0.7, we have CR(C3 ), CR(C6 ), and CR(C7 ) less than 0.7, and hence, the “to-be-enhanced” learning paths are as follows: PATH1: C3 → C6 PATH2: C3 → C7 . It can be observed that the key problem of learning the subject unit is because of the misunderstanding of concepts C3 , C6 , and C7 ; moreover, the student should learn concept C3 before learning C6 and C7 . The information can be transmitted to the adaptive hypermedia systems so that the subject materials presented to individual students can be adapted accordingly. III. GRAY FORECAST APPROACH FOR LEARNING DIAGNOSIS Although the concept effect model is able to offer learning suggestions in some application domains, we found that the relationships among the concepts are not only prerequisite (which need to be learned in some specific order) but also reciprocal (which affect the learning of each other without any specific order). If only the prerequisite relationship is taken into consideration, the learning diagnosis results could be imprecise. In the following, we propose a new model that takes both prerequisite and reciprocal relationships into consideration. A. Prerequisite and Reciprocal Relationships Among Concepts Taking the “Java programming” course as an example, the original course structure is a tree consisting of chapters, sections,

Fig. 3. course.

Prerequisite and reciprocal relationships in “Java programming”

and subsections. By applying our approach, the prerequisite and reciprocal relationships can be derived as shown in Fig. 3, where each block represents a concept to be learned and each solid directional line represents the “prerequisite relationship” and is represented as Ci → Cj (Ci should be learned before Cj ). For two related concepts without any obvious learning sequence, a bidirectional line is used to connect them, which is nominated the “reciprocal relationship” and is represented as Ci ↔ Cj . For example, it is difficult to decide the prerequisite relationship between concepts “JDBC” and “AWT”; instead, we observed that both concepts affect the learning of each other. That is, if a person wants to write a database application with the Java language, it is possible that he (or she) learns “JDBC” without “AWT” background. However, if he (or she) learns “AWT” as well, the skill of windows programming and the knowledge of object property will make the “JDBC” concept more applicable. Both “prerequisite relationship” and “reciprocal relationship” will affect the status of the students while they try to learn new concepts. To model the effects of these relationships among the concepts, two variables (α and β) are introduced. The α variable represents the effect degree of “prerequisite relationship,” and the β variable represents that of the “reciprocal relationship.” For an individual student, a pair of αk and βk values is used to denote the impacts of “prerequisite relationship” and “reciprocal relationship” on the learning performance of that student, respectively, based on a series of k tests [18]. The mathematical model is defined as

wis + βk wir (1) Wij (αk , βk ) = wij + αk cs →cj

cr ↔cj

where wij represents the relationship between the test item Qi and the concept Cj , which is an integer ranging from 0 to 5 to represent the strength of the relationship. Consider the example given in Table II, where C1 = “Java Basic,” C2 = “Native Code,” C3 = “JDBC,” C4 = “Event Model,” C5 = “Thread,” C6 = “RMI,” C7 = “AWT,” C8 = “Java Bean,” and C9 = “Reflection.” We have W11 = 3, W12 = 4, and W13 = 0. The marked rows represent the test items that the student fails to answer (i.e., Q3 , Q5 , and Q7 ).

HWANG: GRAY FORECAST APPROACH FOR DEVELOPING DISTANCE LEARNING AND DIAGNOSTIC SYSTEMS

TABLE II QUESTION-CONCEPT RELATIONSHIPS IN THE “JAVA PROGRAMMING” COURSE

101

In gray forecasting, the accumulated generating operation (AGO) and inverse accumulated generating operation (IAGO) are the main methods which provide a manageable approach to treating disorganized evidence [5]. The AGO equation is x(r) (k) = x(r−1) (1)+x(r−1) (2) + x(r−1) (3) + · · · + x(r−1) (k) = x(r) (k − 1) + x(r−1) (k).

(2)

The IAGO equation is x(r−1) (k) = x(r) (k) − x(r) (k − 1)

Since each Wij (α, β) takes both the prerequisite and reciprocal relationships into consideration, it can be used to replace wij to achieve more accurate results. In Sections III-B–D, a novel approach is proposed to derive α and β based on the gray forecasting model proposed by Deng [4].

where x(0) = {x(0) (1), x(0) (2), x(0) (3), . . . , x(0) (n)} is the original series and x(r) = {x(r) (1), x(r) (2), x(r) (3), . . . , x(r) (n)} is the r times accumulated series. The generated series can be used to build a GM, which is developed by applying the approximate exponential law [5]. GM(1, 1) is a one-variable and one-degree differential equation, which applies AGO, IAGO, and other equations to predict a series of values. The first-order differential equation of GM(1, 1) model is dx(1) + ax(1) = u dt

B. Gray Forecast Approach In gray theory, random variables are regarded as gray numbers, and a stochastic process is referred to as a gray process [10]. A gray system is defined as a system containing information presented as gray numbers, consisting of both known and unknown information [4]. For example, “the human body” system is a gray system, because some characteristics of the body such as height, weight, eyesight, etc. are known; and some physiological phenomena in the human body are unknown. A gray system can often be characterized by a (time) series, and a stochastic process (i.e., gray process) is defined as a family of time series random variables (i.e., gray numbers) [5]. Deng considers most of the existing systems to be a “generalized energy system,” and nonnegative smooth discrete functions can be transformed into a sequence with an approximate exponential law [4], [5]. The gray forecasting model (GM) is the core of the gray system theory, which treats all variables as a gray quantity within a certain range. The GM collects available data to obtain the internal regularity. It examines the nature of internal regularity in managing the disorganized primitive data and transfers the arranged sequence into a differential equation. There are five types of gray forecast: series forecast for predicting the quantity of characteristic data (e.g., the population forecast); disaster forecast for predicting a chronological element of unusual value; season disaster forecast for predicting the disaster within a special zone (e.g., rainy season begins on May 4); topology forecast for predicting waveform as a singular calculation, instead of predicting an approximate value of waveform change (e.g., the year of meteorological average rainfall curve, the sum of rainfall on several years); and system synthetic forecast for predicting the change of the entire system that depends on the variables of the system.

(3)

(4)

where t denotes the independent variables in the system, a represents the developed coefficient, and u is the gray controlled variable. From (2) and (4), coefficient a ˆ can be obtained based on the ordinary least square method   a T −1 T (5) a ˆ = (B B) B yN = u where 

− 12 (x(1) (1) + x(1) (2))

 1 (1)  − 2 (x (2) + x(1) (3)) B=  ···

yN

− 12 (x(1) (n − 1) + x(1) (n))  (0)  x (2)  (0)   x (3)   . =   ··· 

1



 1   1 1

x(0) (n) The approximate relationship can be derived by substituting a ˆ obtained in the differential equation and solving (4)

u  −ak u e (6) x ˆ(1) (k + 1) = x(0) (1) − + . a a C. Estimate αk and βk We apply the series forecast to estimate αk and βk . The series forecast of the gray theory initially requires an original series to predict the possible values of the next series. Its mathematical

102

IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART C: APPLICATIONS AND REVIEWS, VOL. 37, NO. 1, JANUARY 2007

TABLE III TEST ITEM AND CONCEPT RELATIONSHIPS WITH α = 0.7 AND β  = 0

Consider the following series: x(0) (k)

x(0) (1) x(0) (2) x(0) (3) x(0) (4) 0.75

0.64

0.80

x(0) (5)

0.86

0.71

In the following, we shall demonstrate how the GM(1, 1) model can be used to predict the value of αk . Step 1: Employing the AGO function By applying (2), we have x(1) (k)

x(1) (1) x(1) (2) x(1) (3) x(1) (4) 0.75

1.39

2.19

3.05

Step 2: Find matrix B and yN  − 1 (x(1) (1) + x(1) (2)) 2

 1 (1)  − 2 (x (2) + x(1) (3)) B=  1 (1)  − 2 (x (3) + x(1) (4)

equations are defined as follows:  


Wrj + α Wrj  Ak = 0.5 +

r

j



i

Bk = 0.5 −

j



i

Wij + α

j

 Wrj + β 


Cn ↔Cj

 Wij + β


Cn →Cj

j



r

Cn →Cj








 − 0.5 (7) Wij   Wrj   − 0.5 (8)

yN

Wij 

Cn ↔Cj

where i is the test item index, r represents the index of the test items that the student fails to correctly answer, and j represents the concept index. We record the answers of each student for a series of tests to produce Ak and Bk , and then utilize the series forecast to derive αk and βk . If Ak is large, we say that the student’s current status is deeply affected by his (or her) previous status. Initially, we assign 0 to β  and an arbitrary value to α to calculate Ak and Wij (α , 0). Similarly, we assign 0 to α and an arbitrary value to β  to calculate Bk and Wij (0, β  ). By repeatedly testing the students, series A = (A1 , A2 , . . . , Ak ) and B = (B1 , B2 , . . . , Bk ) are generated by applying (7) and (8). The precise values of αk and βk are then obtained by applying GM(1, 1) on these two series. Consider the “Java programming” example given in Table II. By assigning (α , β  ) with (0.7, 0), we have Wij (0.7, 0) as shown in Table III. From Table III, Ak can be obtained as follows: Ak = 0.5     1.0 + 1.0 + 0.7 + 0.7 + 6.7 + 0.7 + 0.7 + 11.7 + 8.2 − 0.5 + 7.0 + 7.0 + 13.9 + 9.9 + 10.9 + 12.9 + 14.6 + 15.2 = 0.62. After several tests, a series A consisting of Ak is obtained, and the gray model of the series forecast GM(1, 1) is then employed to estimate αk .

− 12 (x(1) (4) + x(1) (5))   −1.07 1  −1.79 1  =  −2.62 1 −3.41 1  (0)    x (2) 0.64  x(0) (3)   0.80  =  (0)  =  . x (4) 0.86 x(0) (5) 0.71

Step 3: Derive a ˆ T

−1



T

a ˆ = (B B) B yN

x(1) (5) 3.76

1  1   1 1

−0.03 = 0.69



Step 4: Build a forecast equation dx(1) − 0.03x(1) = 0.69 dt

u  −ak u e + x ˆ(1) (k + 1) = x(0) (1) − a a x(0) (1) = 0.75 0.69 u = = −23 a −0.03 x ˆ(1) (k + 1) = 23.75e0.03k − 23 Step 5: Find αk Based on the results of Step 4, αk can be predicted by computing the values of next series. Since k = 5 in this example, we have ∗

x ˆ(1) (5 + 1) = 23.75e0.03 5 − 23 = 4.59 and α5 = x ˆ(0) (6) = x ˆ(1) (6) − x ˆ(1) (5) = 4.59 − 3.76 = 0.83. Similarly, we can assign (α , β  ) with (0, 0.3) to obtain Wij (0, 0.3) and Bk , and then employ GM(1, 1) to derive β5 .

HWANG: GRAY FORECAST APPROACH FOR DEVELOPING DISTANCE LEARNING AND DIAGNOSTIC SYSTEMS

103

TABLE IV TEST ITEM AND CONCEPT RELATIONSHIPS WITH αk = 0.8 AND βk = 0.2

D. Finding Poorly Learned and Well-Learned Concepts

Fig. 4.

Structure of the intelligent testing, evaluation, and diagnosis system.

Fig. 5.

Workflow of the learning diagnosis module.

Taking test item and concept relationships of the “Java programming” course in Table II as an example, consider the prerequisite and reciprocal relationships in Fig. 3. By assuming the derived αk = 0.8 and βk = 0.2, we have W12 (0.8, 0.2) = w12 + 0.8∗ w11 + 0.2∗ (0) = 4 + 0.8∗ 3 + 0.2∗ 0 = 6.40 W13 (0.8, 0.2) = w13 + 0.8∗ w11 + 0.2∗ (0) = 0 + 0.8∗ 3 + 0.2∗ 0 = 2.40 ... W63 (0.8, 0.2) = w63 + 0.8∗ w11 + 0.2∗ (w67 + w68 ) = 0 + 0.8∗ 0 + 0.2∗ (0 + 5) = 1.00 ... Therefore, the Wij (0.8, 0.2) matrix can be derived as shown in Table IV. In Table IV, CR(Cj ) represents the ratio of correct answers relevant to the concept Cj . For example, there are three test items Q1 , Q3 , and Q8 concerning the concept C1 with weight 3, 1, and 3, respectively. Since Q1 and Q8 have been correctly answered, we have CR(C1 ) = (3 + 3)/(3 + 1 + 3) = 0.86. After constructing Table IV, the status of each student can be precisely evaluated. If CR(Cj ) is greater than the defined threshold, we say that the concept has been well learned; otherwise, the concept is said to be poorly learned. Consider the concept effect graph in Fig. 3, if the threshold is 0.7, we have four poorly learned concepts: “Event Model,” “AWT,” “Java Bean,” and “Reflection.” Since “Event Model” is the prerequisite of “AWT” and “Java Bean” and “Reflection” inherit “AWT,” we suggest that the student should enhance the study of the “Event Model” before learning other concepts. IV. DEVELOPMENT OF AN INTELLIGENT TESTING AND DIAGNOSTIC SYSTEM Fig. 4 shows the structure of a network-based intelligent testing and diagnostic system implemented and based on the novel approach. The system has been implemented with the Java language and a Microsoft SQL server. The concept relationship

database contains the prerequisite and reciprocal relationships among concepts of each course. Those relationships and the test results are then used by the learning diagnosis module to detect any student learning problem, from which learning suggestions can be generated by the learning guidance module for each student. The Java-based user interface allows the users to access the system through web browsers. Fig. 5 shows how the learning diagnosis module works to generate the learning guidance. It reads conceptual relationships with αk−1 and βk−1 values from the conceptual relationship database and output diagnosis results after receiving the test results. Meanwhile, it also generates Ak and Bk , which will be used to produce αk and βk values by the gray model. Fig. 6 shows the user interface for assisting the teachers in defining the relationship among the concepts, and Fig. 7 shows the test item management interface. Fig. 8 shows the student interface for taking an online selfassessment or a test assigned by the teacher. When a student finishes the test sheet and submits the answers, the system analyzes the student’s learning status immediately (see Fig. 9), and then presents the suggestions to the student accordingly (see Fig. 10). V. EVALUATION OF THE NOVEL APPROACH In this section, we try to evaluate the performance of the novel approach. First, we would like to depict how this approach

104

IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART C: APPLICATIONS AND REVIEWS, VOL. 37, NO. 1, JANUARY 2007

Fig. 6.

Conceptual relationship constructing interface.

Fig. 7.

Test item management interface.

HWANG: GRAY FORECAST APPROACH FOR DEVELOPING DISTANCE LEARNING AND DIAGNOSTIC SYSTEMS

Fig. 8.

Assessment interface for students.

Fig. 9.

Results generated by the learning diagnosis module.

improves some previously imprecise predictions and provides more precise results in the diagnosis process by employing both of the prerequisite and reciprocal relationships and the personalized learning factors. An example is given in Fig. 11, which

105

shows that some “inconsistent” results may occur if a imprecise model is used to diagnose student learning problems. For example, the student seemed to learn the “Java Bean” concept well (with CR = 1), but completely failed to learn the “AWT”

106

IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART C: APPLICATIONS AND REVIEWS, VOL. 37, NO. 1, JANUARY 2007

Fig. 10.

Learning guidance presented to the student.

Fig. 11.

Learning diagnosis results generated by different approaches.

concept (with CR = 0). Regardless of poorly designed test items and test sheets, there are several factors that may cause such imprecise results. 1) The omission of some relationships among concepts: Consider the example given in Fig. 11; the old models treat “AWT” as the only concept that benefits the learning of “Java Bean.” That is, a student cannot learn “Java Bean” well without learning “AWT.” However, in fact, the reciprocal relationship between “JDBC” and “Java Bean”

shows that a student who has learned “JDBC” might be able to learn part of “Java Bean” notation even without completely knowing “AWT.” 2) The use of imprecise models to evaluate student learning performance: The previously proposed models treat those concepts as independent targets while evaluating their CR values. For example, as the student fails to correctly answer all of the test items that are indicated to be related to “AWT,” the corresponding CR value is 0. Similarly, the

HWANG: GRAY FORECAST APPROACH FOR DEVELOPING DISTANCE LEARNING AND DIAGNOSTIC SYSTEMS

student correctly answers the entire test items related to “Java Bean,” and hence, the corresponding CR value is 1. Therefore, an imprecise result has been produced. In fact, it is possible that some test items concerning “Java Bean” are related to part of the “AWT” concept although such relevance has not been indicated apparently. In our new model, such possibly embedded relevance has been taken into consideration while calculating Wij (α, β). 3) The omission of personalized learning factor: The previously proposed algorithms simply evaluate the learning performance for different students based on the same set of concept effect relationships. However, researchers of pedagogical psychology have indicated that each person may have a personalized learning style [7]. That is, some students may be significantly influenced by their knowledge of “AWT” in learning “Java Bean” (with larger α value and smaller β value), while some may not (with smaller α value and larger β value). Consequently, some imprecise diagnosis results might be produced owing to the omission of those personalized learning factors by employing the previously proposed models. To sum up, those previously proposed approaches employ the models that ignore the reciprocal relationships and personalized learning factors and, hence, produce imprecise results. Such imprecise predictions can be significantly improved by employing our novel approach. For example, by replacing wij with Wij (α, β), we have CR (“AWT”) = 0.44 and CR (“Java Bean”) = 0.63, which provide more precise information, indicating that the student’s “JDBC” knowledge is helpful to the learning of the “Java Bean” concept. To evaluate the efficacy of the novel approach, an experiment has been conducted on a Computer Science course from February to May 2005. Seventy college students participated in the experiment. Those students were randomly separated into two groups designated V1 (control group) and V2 (experimental group), each of which contains 35 students. The learning guidance for each student in V1 was generated by applying the original concept effect model; that is, only the “prerequisite relationship” was taken into considerations. The students in V2 received learning suggestions generated by applying our novel approach; that is, both the “prerequisite relationship” and “reciprocal relationship” have been considered. A pretest and a post-test have been performed to compare the learning levels of the students in the two groups. In the followings, the experimental results are presented.

107

TABLE V t-TEST OF THE PRETEST RESULTS (FEBRUARY. 18, 2005)

TABLE VI t-TEST OF THE POST-TEST RESULTS (MAY 20, 2005)

the above, it was evident that the two groups of students have equivalent abilities in learning the computer course. B. Post-Test The post-test was intended to compare the learning achievements of the two groups of students after learning the computer course. Table VI lists the t-test values for the post-test results. Notably, the mean and standard deviation of the posttest were 77.0 and 14.83 for V1, and 93.57 and 6.89 for V2. From the mean of the post-test, V2 at first observation achieves better performance than V1. As the p-value = 0.000 < 0.05 and t = −5.996, we can conclude that V2 achieved significantly better performance than V1 after implementing the subject approach. VI. CONCLUSION In this paper, we propose a gray forecast approach to modeling the prerequisite and reciprocal relationships among concepts to be learned. A testing and diagnostic system based on the novel approach has been implemented on computer networks. The system not only diagnoses the problems of the students, but also generates a learning guidance procedure for each student. The novel approach has been successfully applied to the diagnosis of student learning problems for a computer course of a college. From the analysis results, it can be seen that our novel approach is helpful in improving the learning performance of students. In the near future, we plan to employ this approach to several courses, including a Natural Science course and a Mathematics course of a primary school, a Mathematics course of a junior high school, and several computer courses of a college. Moreover, a project is currently being conducted to recode the testing and diagnostic system so that it can work with more adaptive hypermedia systems via a predefined web service protocol. REFERENCES

A. Pretest The pretest aimed to ensure that both groups of students had the equivalent knowledge required for learning the course. Table V presents the t-test results of the pretest. Notably, the mean and standard deviation of the pretest were 55.86 and 11.62 for V1 (control group), and 57.14 and 10.77 for V2 (experimental group). As the p-value (significant level) = 0.633 > 0.05 and t = −0.48, we can infer that in the pretest, V1 and V2 do not significantly differ at a 95% confidence interval. From

[1] B. A. A. Antao, A. J. Brodersen, J. R. Bourne, and J. R. Cantwell, “Building intelligent tutorial systems for teaching simulation in engineering education,” IEEE Trans. Educ., vol. 35, no. 1, pp. 50–56, Feb. 1992. [2] P. Brusilovsky, “Adaptive hypermedia: An attempt to analyze and generalize,” in Multimedia, Hypermedia, and Virtual Reality, Berlin, Germany: Springer-Verlag, 1996, pp. 288–304. [3] “Methods and techniques of adaptive hypermedia,” in Adaptive Hypermedia and Hypermedia, A. Kobsa and J. Vassileva, Eds. London, U.K.: Kluwer, 1998, pp. 1–43. [4] J. L. Deng, “Grey linear programming,” presented at the Int. Conf. Information Processing Management Uncertainty Knowledge-Based System, Paris, France, 1986.

108

[5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19]

IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART C: APPLICATIONS AND REVIEWS, VOL. 37, NO. 1, JANUARY 2007

, “Introduction to grey system theory,” J. Grey Syst., vol. 1, no. 1, pp. 1–24, 1989. A. J. Gonzalez and L. R. Ingraham, “Automated exercise progression in simulation-based training,” IEEE Trans. Syst., Man, Cybern., vol. 24, no. 6, pp. 863–874, Jun. 1994. A. F. Gregorc and H. B. Ward, “Implications for learning and teaching: A new definition for individual,” NASSP Bull., no. 61, pp. 20–26, 1977. S. A. Harp, T. Samad, and M. Villano, “Modeling student knowledge with self-organizing feature maps,” IEEE Trans. Syst., Man, Cybern., vol. 25, no. 5, pp. 727–737, May 1995. S. Hopper, “Cooperative learning and computer-based instruction,” Educ. Technol. Res. Develop., vol. 40, no. 3, pp. 21–38, 1992. C. I. Hsu and Y. H. Wen, “Application of grey theory and multiobjective programming towards airline network design,” Eur. J. Oper. Res., vol. 127, no. 1, pp. 44–68, 2000. G. J. Hwang, “A tutoring strategy supporting system for distance learning on computer networks,” IEEE Trans. Educ., vol. 41, no. 4, p. 343, Nov. 1998. , “On the development of a cooperative tutoring environment on computer networks,” IEEE Trans. Syst., Man, Cybern. C, vol. 32, no. 3, pp. 272–278, Aug. 2002. , “A concept map model for developing intelligent tutoring systems,” Comput. Educ., vol. 40, no. 3, pp. 217–235, 2003. , “A test sheet generating algorithm for multiple assessment requirements,” IEEE Trans. Educ., vol. 46, no. 3, pp. 329–337, Aug. 2003. , “A data mining algorithm for diagnosing student learning problems in science courses,” Int. J. Distance Educ. Technol., vol. 3, no. 4, pp. 35– 50, 2005. G. J. Hwang, J.-L. Hsiao, and J. C. R. Tseng, “A computer-assisted approach for diagnosing student learning problems in engineering courses,” J. Inf. Sci. Eng., vol. 19, no. 2, pp. 229–248, 2003. A. Kavcic, “Fuzzy user modeling for adaptation in educational hypermedia,” IEEE Trans. Syst., Man, Cybern. C, vol. 34, no. 4, pp. 439–449, Nov. 2004. T. Larkin-Hein and D. D. Budny, “Research on learning style: Applications in the physics and engineering classrooms,” IEEE Trans. Educ., vol. 44, no. 3, pp. 276–281, Aug. 2001. B. Ozdemir and F. N. Alpaslan, “An intelligent tutoring system for student guidance in Web-based courses,” in Proc. 4th Int. Conf. Knowledge-Based Intell. Eng. Syst. Allied Technol., 2000, vol. 2, pp. 835–839.

[20] R. Paolucci, “Hypermedia and learning: The relationship of cognitive style and knowledge structure,” ED-MEDIA/ED-TELECOM 1998, Freiburg, Germany, Jun. 1998, presented at the. [21] J. B. Pugliesi and S. O. Rezende, “Intelligent hybrid system for a training and teaching environment,” in Proc. 3rd Int. Conf. Comput. Intell. Multimedia Appl., 1999, pp. 148–152. [22] D. F. Salisbury, “Effect drill and practice strategies,” in Instructional Designs for Microcomputer Courseware, D. H. Jonassen, Ed. Hillsdale, NJ: Lawrence Erlbaum, 1998, pp. 103–124. [23] R. Snow and M. Farr, “Cognitive-conative-affective processes in aptitude, learning, and instruction: An introduction,” in Conative and Affective Process Analysis, vol. 3, Hillsdale, NJ: Lawrence Erlbaum, 1998, pp. l–10. [24] V. Vasandani and T. Govindaraj, “Intelligent diagnostic problem solving tutor: An experimental evaluation,” in Proc. IEEE Int. Conf. Decision Aiding Complex Syst., Charlottesville, VA, Oct. 13–16,1991, pp. 1739– 1744. [25] , “Knowledge organization in intelligent tutoring systems for diagnostic problem solving in complex dynamic domains,” IEEE Trans. Syst., Man, Cybern., vol. 25, no. 7, pp. 1076–1096, Jul. 1995. [26] A. Yoshikawa, M. Shintani, and Y. Ohba, “Intelligent tutoring system for electric circuit exercising,” IEEE Trans. Educ., vol. 35, no. 3, pp. 222–225, Aug. 1992.

Gwo-Jen Hwang received the Ph.D. degree from the National Chiao Tung University, Hsinchu, Taiwan, R.O.C. He is currently a Professor in the Department of Information and Learning Technology and the Dean of the College of Science and Engineering, National University of Tainan, Tainan, Taiwan. His research interests include e-learning, computer-assisted testing, expert systems, and mobile computing. He has published nearly 200 papers.