Method and apparatus for improving performance on multiple-choice ...

US006514084B1

(12)

(54)

United States Patent

(10) Patent N0.:

Thomas

(45) Date of Patent:

METHOD AND APPARATUS FOR IMPROVING PERFORMANCE ON MULTIPLE-CHOICE EXAMS

Computer—based Testing (CBT) Program Supplement to the

patent is extended or adjusted under 35

U.S.C. 154(b) by 0 days.

1992—93 GRE Information Bulletin, Educational Testing Service, 1992, pp. 1 and 7—9.

Computer—based Testing (CBT) Program Supplement to the 1993—94 GRE Information Bulletin, Educational Testing Service, 1993, pp. 1, 9 and 11. The SMH Personal Tutor, SMH Bar Review, printed in

National Jurist, Feb/Mar. 1994. (Product Brochure). The Law in Flash Series (Product Catelog), Professional

Flash Cards, Ltd., 1993 (TestMaster Computer Software). SmartStudy software program, BAR/BRI Multistate Test

(21) Appl. N0.: 09/542,999 Apr. 4, 2000 (22) Filed:

ing: Bar Review, by Engineering Associates and Harcourt Brace Legal & Professional Publications, Inc., 1993, 16 pages.

Related US. Application Data Continuation of application No. 09/253,851, ?led on Feb. 22, 1999, now Pat. No. 6,086,382, which is a continuation of application No. 08/810,547, ?led on Mar. 3, 1997, now Pat. No. 5,885,087, which is a continuation of application No. 08/315,976, ?led on Sep. 30, 1994, now Pat. No.

“PassMaster: The Intermediate Accounting Supplement of Choice,” Harcourt Brace Jovanovich, 1992, pp. 1—12. Conviser Duffy and Logicat, PassMaster marketing mate rial, 1992, 6 pages.

5,618,182.

(51) (52)

(List continued on next page.)

Int. Cl.7 ................................................ .. G09B 7/00

US. Cl. ..................... .. 434/322; 434/118; 434/323;

434/362; 706/927

(58)

Feb. 4, 2003

OTHER PUBLICATIONS

(75) Inventor: C. Douglass Thomas, Campbell, CA (Us) (73) Assignee: Robolaw Corporation, Campbell, CA (Us) (*) Notice: Subject to any disclaimer, the term of this

(63)

US 6,514,084 B1

Field of Search ............................... .. 464/118, 156,

464/169, 307 R, 322, 323, 327, 332, 335, 336, 338, 350, 353, 359, 362, 365; 463/9, 19, 29, 40—42; 345/169; 702/182; 706/927; 707/1, 500

A A A A A A A

4,867,685 A

12/1970 6/1982 11/1982 6/1986 *

6/1986

A computerized learning approach that enables a user to improve their performance on multiple-choice exams is

disclosed. Although the learning approach includes various aspects and implementations, in general, the invention

approach designed to improve their performance. The inven tion provides important performance feedback to user’s, for

Koos et al. Hatta Yamamoto Fascenda et al. Graves

ABSTRACT

atic way forces the users to follow a proven problem-solving

U.S. PATENT DOCUMENTS

3,546,791 4,337,047 4,358,273 4,592,546 4,593,904 4,798,543 4,829,431

(57)

forces test-takers to practice their examination skills and subject matter knowledge in a systematic way. The system

References Cited

(56)

Primary Examiner—Joe H. Cheng

example, elapsed time per question and percentage of cor rectly answered questions. The invention can also pinpoint

........................ ..

463/9

1/1989 Spiece 5/1989 Ott et al. 9/1989 Brush et al.

for the user the substantive areas of the exam which the user

is either weak or strong. Further, the invention is able to assist the user in predicting his/her eventual score.

27 Claims, 10 Drawing Sheets

(List continued on next page.) (.0

mmmcmmimxmmh UMBER,

coiu ‘CT/INCORRECT ENDICATOR, ELAPSED FOR EA

cowurs k m: .AY mmcmon or PERCENTAGE olmsnom 1N SET ANSWERED commv AND AVERAGE

ELAPSED TIME For 'n-m SET

HISTORY

msvnw PERFORMANCE i-us‘mRY

/

US 6,514,084 B1 Page 2

US. PATENT DOCUMENTS 5,002,491 5,018,082 5,033,969 5,035,625

A A A A

3/1991 5/1991 7/1991 7/1991

5,178,545 A 5,204,813 5,211,563 5,218,537 5,228,859 5,257,185

1/1993 Thompson

A A A A A

4/1993 5/1993 6/1993 7/1993 10/1993

5,259,766 A

* 11/1993

5,267,865 A

5,302,132 5,306,154 5,310,349 5,316,485

* *

5,421,730 A 5,456,607 A

5,632,624 A

5,820,386 A 5,842,871 A

Sack et al. ................ .. 434/323

4/1994 4/1994 5/1994 5/1994

Corder Ujita et al. Daniels et al. Hirose

1/1995 5/1995

Derks ....................... .. 434/350 Mirando ...................... .. 463/9

6/1995 Lasker, III et al. * 10/1995

5,586,889 A 5,616,033 A

Samph et al. Haga et al. Hemphill et al. Rowe Farley et al.

12/1993 Lee et al.

A A A A

5,379,213 A 5,411,271 A

Abrahamson et al. Obata et al. Kamimura Munson et al.

Antoniak .................. .. 434/323

12/1996 Goodman *

4/1997

Kerwin ..................... .. 434/118

5/1997 Cameron et al.

10/1998 Sheppard, II * 12/1998 Cutler et al. .............. .. 434/335

OTHER PUBLICATIONS

Conviser Duffy and Logicat, “PassMaster CPA Review Software,” 1990, 5 pages. “Computer Test Prep Meets the New SAT,” PC Magazine,

Sep. 13, 1994, pp. 467—468. Sleeman, “Intelligent Tutoring Systems: A Review,” Pro ceedings of EdCompCon, IEEE Computer Society, Oct. 18—20, 1983, pp. 95—101. Jelovsek et al., “Application of Teaching and Learning

Hause et al., “Effective Computer Aided Instruction in

Biomedical Science,” IEEE, 1985, pp. 534—538. Ananda et al., “Design of an Intelligent On—line Examina

tion System,” Computer Educ., vol. 13, No. 1, 1989, pp. 45—52.

Lee—Sammons et al., “Computerized practice tests and effects on in—class exams,” Behavior Research Methods,

Instruments & Computers, vol. 21, No. 2, 1989, pp. 189—194.

Loose, “Finding the Appropriate Instructional Text from Question Performance Evaluation : A Prototype Interface,” 2”“ International Conference on Computer Aided Learning, 1989, pp. 336—345. Grabe et al., “The Impact of Computer Supported Study on Student Awareness of Examination Preparation and on

Examination Performance,” Journal of Computer Based Instruction, vol. 17, No. 4, 1990, pp. 113—119. Grabe et al., “An Evaluation of Computer Assisted Study in Controlled and Free Access Settings,” Journal of Computer Based Instruction, vol. 16, No. 3, 1989, pp. 110—116. Georgianis, “Cliffs Notes debuts series of apps for course

review,” Computer Reseller News, May 3, 1993, 2 pages. Scisco, “Cliffs Study Ware for the SAT,” PC MagaZine, vol. 12, No. 15, Sep. 14, 1993, 4 pages. Sides, “Study Ware fpr the SAT,” Compute, vol. 13, No. 10, Oct. 1991, 2 pages.

Yakal, “Study Ware for the GRE,”Compute, vol. 13, No. 9, Sep. 1991, 2 pages. Schneider, “Keeping up with the SATs,” Technology & Learning, vol. 15, No. 1, Sep. 1994, 3 pages. StudySmart for Contracts, BARBRI Bar Review, 1994, 10 pages.

Kaplan—MCAT, Kaplan Educational Center, 1994, 5 pages. Cliffs Study Ware for Chemistry, Cliffs Notes, Inc. 1993,

Principles to Computer—Aided Instruction,” Computing

pp.1—40.

M.D., vol. 6, No. 5, Sep./Oct. 1989, pp. 267—273. Jeffreys et al., “A Freely—Available Authoring System for the IBM PC,” Proceedings of the Ninth Annual Symposium on Computer Applications in Medial Care, IEEE Computer Society, Nov. 10—13, 1985, pp. 575—579.

Cliffs Study Ware for the SAT I, Cliff Notes, Inc. 1994, pp.

Cliffs Study Ware for the GRE, Cliffs Notes, Inc. 1993, pp. 1—48. 1—48. * cited by examiner

U.S. Patent

Feb. 4, 2003

Sheet 1 0f 10

DISPLAY SCREEN

US 6,514,084 B1

COMPUTER

I INPUT

DEVICE

.

‘

LEARNING

USERS

PROGRAM

PERF INFO.

ii

15 ~

8

'_F OPERATING

QUESTIONS

SYSTEM

ii

2 *

2N

MISC.

_2__Q N[EMORY

FIG. 1

U.S. Patent

Eb

Feb. 4, 2003

Sheet 2 0f 10

@

US 6,514,084 B1

4/22

DISPLAY NEXT QUESTION AND A PLURALITY OF ANSWER CHOICES

,24

+ START QUESTION TIMER

DISPLAY INDICATION OF ELAPSED TIL/1E

DISPLAY HINT TOWARDS THE CORRECT ANSWER CHOICE

FIG. 2A

26

/28

U.S. Patent

Feb. 4, 2003

Sheet 3 0f 10

US 6,514,084 B1

HAS

USER SELECTED ANOTHER ANSWER CHOICE. 7

SELECTED ANSWER

CHOICE EQUAL TO A PREDE'I'ERL/HN'ED CORRECT ANSWER CHOICE '1

42

\

DISPLAY SUBSTANTIVE

INFORMATION RELEVANT To THE QUESTION

@ DISPLAY EXPLAINATTON OF THE CORRECT ANSWER CHOICE

RESET QUESTION TIMER

FIG. 2B

‘~48

"44

U.S. Patent

Feb. 4, 2003

Sheet 4 0f 10

US 6,514,084 B1

49

/

( SUBSTANTIVE INFORMATION ROUTINE ) IDENTIFY SUBJECT & TOPIC FOR THE QUESTION

SEARCH SUB STANTIVE INFORMATION FOR

F 50

52

THE TOPIC

l DISPLAY A PORTION OF THE SUB STANTIVE INFORMATION PERTAINING TO THE TOPIC

CLOSE

54

56

REQUESTED OR

QUESTION

ANSWERED CORRECT LY t7

REMOVE THE DISPLAY OF THE SUBSTANTIVE INFORMATION

FIG. 3

\’ 53

U.S. Patent

Feb. 4, 2003

Sheet 5 0f 10

US 6,514,084 B1

60

C PERFORMANCE EVALUATION ROUTINE 3 / DISPLAY QUESTION_NUMBER, CORRECT/INCORRECT INDICATOR, ELAPSED TIME, SUBJECT AND TOPIC FOR EACH I 62

QUESTION IN QUESTION SET 7

COMPUTE & DISPLAY INDICATION OF PERCENTAGE OF QUESTIONS 1N SET ANSWERED CORRECTLY AND AVERAGE

.-,- 64

ELAPSED TIME FOR THE SET

ELIMINATE THE CORRECTLY ANSWERED QUESTIONS FROM THE MASTER

QUESTIONS SET (7

DISABLE QUESTIONS ANSWERED CORRECTLY FROM THE MASTER QUESTIONS SET

VIEW

70

PERFORMANCE HISTORY '2

DISPLAY PERFORMANCE HISTORY

END

FIG. 4

/ 72

U.S. Patent

Feb. 4, 2003

Sheet 6 0f 10

SET NUMBER

FIG. 5A

SET NUMBER

FIG. 5B

US 6,514,084 B1

U.S. Patent

Feb. 4, 2003

Sheet 7 0f 10

CO

R ECTN S

TGT:—