Experimental Evaluation of the Efficiency of a Case-based Organisational Memory Information System Used as a Decision Aid Burstein, F.V., Smith, H.G., Fung, S.M. Monash University, Melbourne, AUSTRALIA
[email protected]
Abstract An organisational memory information system (OMIS) is a computer-based information system that supports the capture, storage, and recall of organisational memory. A wide range of information technologies have been proposed as the means for organisational memory information systems. This paper describes a project to evaluate the quality of decision making using a case-base as the technology for an organisational memory aid. The paper reports specifically on the efficiency of the decision making process when using a case-based OMIS as a decision aid. Keywords: Decision Support, Organisational Memory, Case Based Reasoning, Evaluation of Decision Support Systems
1. Introduction Organisational memory is seen as the means by which knowledge from the past can be used to influence present organisational activities [1]. An organisational memory information system (OMIS) is a computer-based information system that supports the capture, storage, and recall of organisational memory. Research concerning empirical evaluation of OMIS has been highlighted as an important but complex area that needs considerable attention [2]. The lack of empirical studies is a principal shortcoming in the development of a theory of OMIS. Stein and Zwass [1] point to the need to evaluate the impact of OMIS on decision making and refer to a number of empirical studies in the DSS area. The design of any information systems evaluation study must identify the context of evaluation and the focus of the study. It usually involves applying a number of methods in order to provide the necessary measures to draw relevant conclusions. However, it should be noted that, even in the general area of information systems research, the issue of evaluation still generates a great deal of
discussion [3], [4]. Although there is no agreement about what procedures and measures should be applied in evaluation and what should be considered as the outcomes, it is accepted that the multiple viewpoints of the stakeholders should be considered during evaluation, and several evaluation criteria used [5], [6], [7], [8]. One significant factor in the effectiveness of OMIS is the technology used for memory storage and retrieval. A general architecture for an OMIS for decision support has been proposed [9]. This architecture comprises rulebased, database and case-based components to provide various levels of decision support depending on the “structuredness” of the decision situation. A prototype has been developed for the case-based component of the architecture using the case-based reasoning tool, CBR Express¥. This prototype incorporates information about past decisions and acts as organisational memory for the problem. Our current intention is to experimentally evaluate the suitability of a case-based approach to organisational memory information systems for intelligent decision support. This paper reports on an empirical study which evaluates the quality of decision making and the perceptions of the decision makers using an OMIS as a memory aid in a decision support situation; and present some findings from this experiment. The research design borrows from a variety of studies performed on experimental evaluations of information systems, decision support systems and organisational memory information systems. In this experiment, a prototype case-based OMIS was compared with a more traditional method of data storage, ie, a spreadsheet database. The results obtained from this experiment did not show a unequivocal improvement in quality of the decision process or outcome gained from the use of casebased OMIS. The small sample size prevents generalisation from most of these results, however some measures, ie efficiency and relevance of recall, were significant enough to indicate a generalisable difference in the quality of decisions depending on the tool used. The relevance of recall that was achieved using the case-
1060-3425/98 $10.00 (C) 1998 IEEE
based tool was significantly higher than with the other tools used and indicates that a case-based OMIS provides advantages in the recall of relevant material. The measure of efficiency of the decision making process using this tool, however, gave results that do not support the claim that a case-based OMIS is a tool for efficient decision making but did provide interesting material for discussion. In this paper we report the results of an evaluation of the efficiency of the decision making process using the case-based OMIS. This evaluation of efficiency was performed in two different ways: as an actual measure of the efficiency of the decision making task and as the perceptions of the experimental participants of the efficiency of the process. Section 2 presents some theoretical justification for the use of case-bases in OMIS. Section 3 describes the design of an OMIS to support decision making in a semistructured situation. Section 4 then describes a laboratory study which was intended to test the extent to which a case-based form of organisational memory is able to improve the performance of decision makers in a specific decision situation.
2. Theoretical Background Organisational memory information systems have a significant role in intelligent decision support. It is assumed that intelligent decision support should provide some memory aids for the decision maker and include “learning”, in some way, from the decision maker’s experience [10]. In addition, these systems must support associative retrieval and analogical search since search and retrieval are intrinsic mechanisms in human memory. A wide range of information technologies has been proposed as means to implement OMIS [1]. Some conventional information systems technologies, eg. databases and multimedia systems, provide convenient storage and retrieval of past documentary materials associated with organisational activities. The effectiveness and efficiency of such systems is entirely dependent on the data representation and indexing approach. The case-based reasoning formalism was proposed as a way of storing human experiences and manipulating reasoning by analogy [11]. A case based system reflects human knowledge by storing data about previous significant events as “cases” within a computerised system. Case-based reasoning is performed by retrieving stored cases which are similar to the current item through a process of analogical search. Case-based systems are claimed to “learn” through addition of further significant cases to the case-base and by forms of
abstraction which may then be applied to this collection of cases. The case representation includes richer representations of past situations than a simple database, since a case includes information about the context of the event as well as details of the event itself. Case bases for decision support are intended to help decision makers in aspects of the domain where expert knowledge is very limited [12]. These situations are considered to be “semi-structured” at best, that is, the decision maker is faced with a decision situation for which there is no established structured approach to reach a conclusion. Case-based systems provide the user with a set of past situations judged relevant by some analogical retrieval mechanism. To be effective, a case base should not store all previous events of a type but only those with particular significance for the supported decisions. Effective generalisation and abstraction then become important. It could be anticipated that, in some situations, the ability of analogical search to recall previous decisions which are in some way “similar” to the current decision situation may improve the performance of the decision maker. For example, knowledge developed in organisations may be more readily shared if it is stored in some form of case-based OMIS. The current project attempts to provide evidence of the effect of a case-base on the performance of the decision maker.
3. Design of an OMIS The proposed architecture for an OMIS to support decision making in a semi-structured situation has been implemented in a prototype. This system aims to support decision making in an academic administration situation where no clear rule or precedent exists [13]. The university attempts to recognise the prior learning of its students and to recognise this by offering “credit transfer” for previous study at other institutions. The decision situation is a repetitive one in which many applications for credit transfer are unique in detail while they may be similar at a more general level to others dealt with in the past. The university administration is concerned to ensure consistency in the decisions made, an aim that is in some conflict with the relatively high turnover of decision making staff. The information relevant to credit transfer decision making includes the university regulations and agreements made with other institutions. More particularly, precedents set in the past relating to credit granted are important in ensuring consistency of decisions and can be viewed as organisational memory. An organisational memory information system would enable this shared memory to be accessed by a succession
1060-3425/98 $10.00 (C) 1998 IEEE
of decision-makers in a systematic way reducing the loss of memory resulting from rapid change of staff. Currently the administrative staff use several computerised systems to assist in making credit transfer decisions, eg., spreadsheet, database, text files. These systems suffer as OMIS because they lack an efficient way of accessing the relevant past decisions for the current decision situation. The proposed OMIS architecture uses case-based technology to support the storage and retrieval of related but not matching decisions, in this situation, the previous decisions regarding credit. A case-based prototype OMIS has been created to capture the significant cases from the past decision making situations [14]. The prototype reflects the experience of an “expert” in the domain [13], capturing the decision-making experiences of this expert in order to distribute this knowledge more widely in the organisation. The resulting prototype system is not viewed as an expert system, however, since the role of the system is purely to advise the most similar past decisions, rather than propose the “correct” decision. The prototype can also be viewed as a decision support tool, to be used in semi-structured decision situations where no clear rule or precedent exists to justify a decision. The prototype has been tested in a laboratory study to evaluate the efficiency of such a tool for decision support. The next section discusses the rationale for using a casebase as an organisational memory component.
4. Laboratory Study Design The aims of the laboratory study were to evaluate the the extent to which a case-base, as an organisational memory, improves the quality of the decision making process and outcomes in a semistructured decision situation. The case-based OMIS (CB-OMIS) was compared with a conventional spreadsheet aid (spreadsheet) which already existed. The spreadsheet aid could also be viewed as an OMIS in the sense that it too contained data about past decisions. However, the spreadsheet included all past decisions, rather than just the significant ones, and did not offer any automated support for similarity matching beyond the conventional Find and Filter commands. So the two OMIS could be seen to differ in the representation of the knowledge (level of generalisation, modes of access available) and in the retrieval mechanism. A further difference between the case-based OMIS and the spreadsheet-based OMIS was in the user interface. This difference in the interfaces was seen as a factor which, on its own, could have a significant impact on the experiment, particularly on the perceptions of the
decision makers. To reduce the experimental impact of this interface, a third OMIS based on the spreadsheet version (with the same fundamental knowledge representation) was included in the experiment, but with an interface that emulated that of the case-based OMIS. This third OMIS is referred to in the discussion as the ‘macro spreadsheet’. All of the OMIS contained essentially the same information describing historical experiences about the decisions made in the area of credit transfer by expertdecision makers. Each of the systems contained over 100 past decisions relating to credit transfer. The past decision situations recorded in the two spreadsheet-based versions were identical while the case-based OMIS contained the same past decisions but stored in a more aggregated fashion, as suited to the case-based tool. The experimental study took the form of a laboratory experiment involving senior undergraduate computing students. The experimental subjects performed a standard set of decision making tasks with the assistance of one of the three OMIS decision aids. The decision making tasks were presented in a standard order to all participants in the experiment. The set of decision making tasks comprised a range of levels of difficulty. Some situations closely approximated past decisions while others were loosely related to past situations. The framework used for the laboratory experiment (see Figure 1) was based on Morrison and Weiser's [15] organisational memory causal research framework for OMIS evaluation. Their empirical study framework originated from the Computer Supported Cooperative Work (CSCW) research model and represents a contingency model of the processes involving an organisational memory information system [16]. Controlled Variables
Dependent Variables
Agent / Decision Maker
Task
OM Process for Decision Support
DM’s perspective on the use of OM
OM Outcomes
Independent Variables OM Technology
Task Outcomes
Figure 1. OMIS evaluation framework for the laboratory experiment Morrison and Weiser’s research model has been adapted to suit the current study by including Adelman’s
1060-3425/98 $10.00 (C) 1998 IEEE
[5] empirical (objective) and subjective evaluation facets. In addition, the current study differs from Morrison and Weiser's in concentrating on the technology used as an organisational memory aid and in placing the use of the organisational memory in the context of decision support.The measures used for evaluation include some adapted from past experimental studies investigating the effectiveness and impact of DSS aids [7], [16], [17]. A series of hypotheses were proposed addressing measures of the quality of the decision making process and outcomes. In this paper we discuss two of these measures relating to efficiency of the decision making process. The hypotheses for these measures were stated as: H1 There is no significant difference in the decision makers’ perceived efficiency of the decision making process when using the case-based OMIS compared with the other two decision aids; H2 There is no significant difference in the actual efficiency of the decision making process when using the case-based OMIS compared with the other two decision aids.
5. Design of the Experiment 5.1. Subjects Forty-eight senior undergraduate students were used as subjects for the experiment. These students had similar backgrounds in terms of age and experience with computers. Approximately half (23) of the students were female. Subjects were randomly allocated to one of three groups: case-based OMIS, spreadsheet OMIS and macro spreadsheet OMIS. Computing students were chosen as subjects for this experiment to reduce the variability expected in characteristics such as familiarisation with computers and spreadsheet software, age and other background features. Of the 48 students, 15 (approx 30%) had applied for credit transfer for their own studies. These participants can be considered to have prior experience in credit transfer. The remaining 33 (approx 70%) students had not requested credit and were therefore be assumed to have little or no experience in this task.
5.2. The Task At the commencement of each laboratory session, the participants were trained in the basic concepts of credit transfer as applied in the faculty, and in the specifics of the decision making process. The level of training was considered to emulate that expected to be given to new
decision makers in the faculty administration. The subjects were then given a set of ten simulated credit transfer requests for which they needed to make decisions within a limited time period, supported by the OMIS prototype. The simulated requests were developed by an “expert” in the domain to represent both fairly typical credit requests and also to pose situations for which there were no clear-cut precedents. The participants recorded the time taken for each decision as well as their satisfaction with the tool and their confidence in the outcome for each of the credit requests processed. After solving the problems, each participant completed a questionnaire designed to elicit the subjects' views of the decision making process according to the measures specified for the study (eg, satisfaction with the process, confidence in the outcomes, appropriateness of recommended solutions, etc.). Each question was measured on a 5-point Likert scale with 1 as the most positive response (strongly agree) and 5 as the most negative response (strongly disagree). Table 1 lists the questions relevant to the perceived efficiency of the decision making process. If a case-base is more appropriate as a technology for OMIS then it could be anticipated that the group using the case based tool would indicate that they had significantly better support for decision making than either of the groups using the spreadsheet-based OMIS and that the objective measures would show improved performance.
5.3. The Instrument The instrument used to gather the experimental data comprised two parts: a questionnaire to collect subjective measures (as mentioned above) and a response sheet to record the decisions made, the times taken to make these decisions and the information supplied by the tools and deemed by the decision makers to be relevant. The subjective measures obtained from the questionnaire were recoded as follows. a) to ensure that a higher score would have the same meaning for each question; b) to ensure the items representing the responses to the questions were unidimensional and. reliable; (questions that did not seem to measure the same concept and were not reliable were removed from the scale). The responses for each question were then combined to give each subject’s overall score (scale score) for each variable. A high scale score indicated a more positive response.
1060-3425/98 $10.00 (C) 1998 IEEE
Finally, the scale score was collapsed into an ordinal variable with three categories: low, moderate and high, since the variables being measured have relative rather than absolute values. In order to distribute the scores evenly, a cumulative frequency table was used so that the scores of the third of the sample with the lowest score was labelled low, scores of the next third were labelled moderate and scores of the upper third were labelled high. The response sheets were analysed for only the set of three credit transfer requests which had been processed by all subjects. These three decisions were then considered to be one experimental task for the purposes of analysis. Participants recorded the start time and finish time for each processed credit transfer application. The time required to process each application, in minutes, was calculated from these. These times were aggregated over the three decision tasks evaluated to produce the total amount of time spent on the decision making process. Under the laboratory conditions no time was allowed between decision making tasks. The times for individual subjects were then classified into three categories: high (shortest times taken), low (longest times taken) and moderate. The data was prepared in this way to simplify comparison with the previous subjective measure. In addition to this analysis of the time taken, the decisions recorded on the response sheets were individually evaluated by one expert in the problem domain in order to rate the appropriateness of the recommended decision.
difference in the decision makers’ perceived efficiency of the decision making process when using the case-based OMIS compared with the other two decision aids. Three questions were planned to measure the variable perceived efficiency of the decision making process. These questions are shown in Table 1. As described above, the responses to these questions were combined and converted to an ordinal value for analysis.
6. Efficiency of the Decision Making Process
Q4_TIME 6.27 Q13_TIME 5.67 Q19_TIME 5.73
In order to explore the impact of the case based OMIS on the efficiency of the decision making process, two measures were taken. A subjective measure we have termed the perceived efficiency of the decision making process was obtained through the questionnaire. A measure of the actual efficiency of the decision making process was obtained from the time taken for processing the three credit transfer requests included in the analysis. This data was calculated based on the times recorded on the subjects’ response sheets.
6.1. Perceived Efficiency of the Decision Making Process (Subjective Measure) The subjective measure of efficiency reflected the perceptions of the experimental subjects of the time required to perform the decision making task. The hypothesis (H1) was that there is no significant
Table 1. Questions to measure perceived efficiency of the decision making process Question 4. Required information can be easily located 13. I feel there was enough time to perform the tasks 19. I feel that the decision making process was timeconsuming Initial item analysis indicated that the questions were unidimensional, however an Alpha coefficient of 0.5884 indicated low reliability of this measure (see Table 2). Since removing any one question would not increase the reliability of the measure, all three questions were retained. Table 2. Reliability of the measure of perceived efficiency Item-total Statistics Scale Scale Corrected Mean Variance ItemSquared Alpha if Item if Item Total Multiple if Item Deleted Deleted Correlcn Correlcn Deleted 3.82 3.72 3.78
0.38 0.40 0.41
0.14 0.16 0.17
0.52 0.49 0.46
Reliability Coefficients 3 items Alpha =
.59
Standardized item alpha = 0.59
After coding, the responses were crosstabulated to investigate the relationship between efficiency and tool used. The perceptions of the case based OMIS users varied quite strongly. The crosstabulation shows that 38% of the case based OMIS users perceived the decision making process as highly efficient when solving the experimental tasks while 44% of the case based OMIS users perceived the process as having low efficiency. Only 19% of the case based OMIS users had mild views on the efficiency of the case based OMIS. The spreadsheet OMIS users were also strongly divided with 38% in the low category and 44% in the
1060-3425/98 $10.00 (C) 1998 IEEE
high category. The macro spreadsheet users were much less impressed with the efficiency of the decision making process using that tool, 69% felt that their efficiency was low. While the opinions within the experimental groups varied, there appeared to be a relationship between perceived efficiency of the decision making process and tool used as shown in Figure 2.
CB-OMIS users were in the low efficiency category. In another words, the case-based OMIS users spent more time in solving the problems with the decision support tool than the other two groups. None of the case-based OMIS users performed the tasks with high efficiency. These results are shown graphically in Figure 3.
no of subjects
10
no of subjects
12 10
Low efficiency
8 6
Medum efficiency
4
8
Low efficiency
6
Medum efficiency
4
0 Spread sheet
High efficiency
2
High efficiency
2
Macro sprdsht
CBR OMS
0 Spread sheet
Macro sprdsht
CBR OMS
Tool
Figure 3: Objective measure of efficiency of the decision making process by tool used
Tool
Figure 2. Subjective measure of efficiency by tool used Cramer’s V correlation coefficient showed a weak relationship (0.21771), but this was not significant at the 0.05 level, indicating that these results are probably due to sampling error. As the results could not be generalised into the population, the hypothesis H1 (there is no significant difference in the decision makers’ perceived efficiency of the decision making process when using the case-based OMIS compared with the other two decision aids) was accepted.
6.2. Actual Efficiency of the Decision Making Process (Objective Measure)
If tool used had no effect on actual efficiency of the decision making process, the subjects’ responses should be distributed equally in each category. The expected frequencies are shown in Table 3. The expected frequencies shown were expressed under the null hypothesis that there is no difference in actual efficiency of the decision making process using the case-based OMIS compared to the other two decision support tools. Table 3. Hypothetical distribution of efficiency TOOL_USED Count Exp Val TIME_SAV 1
The actual times taken for the three decision making tasks were calculated from times recorded on the response sheets during the laboratory session. This measure was then used to test hypothesis H2, that there is no significant difference in the actual efficiency of the decision making process when using the case-based OMIS compared with the other two decision aids. In order to examine the relationship between actual efficiency of the decision making process and tool used, a crosstabulation analysis was performed. Nine out of 15 (60%) of the spreadsheet users accomplished the tasks with high efficiency, while the macro users were split fairly evenly between moderate and high efficiency. However, 10 of the 16 (63%) of the
Low 2 Moderate 3 High Column Total
Spread sheet 1 2 4.1 4 5.4 9 5.4 15 31.9%
Macro Spread sheet 2 1 4.4 7 5.8 8 5.8 16 34.0%
CaseBased OMIS 3 10 4.4 6 5.8 0 5.8 16 34.0%
Number of Missing Observations:
Row Total 13 27.7% 17 36.2% 17 36.2% 47 100%
0
Table 4 shows a clear relationship between actual efficiency of the decision making process and tool used, as the obtained frequencies vary from the expected frequencies for the three experimental groups. Cramer’s V correlation coefficient (0.46598) showed that a
1060-3425/98 $10.00 (C) 1998 IEEE
moderate relationship exists between tool used and actual efficiency of the decision making process in the sample. The Chi-Square test indicated that this relationship was significant at the 0.05 level, that is, it can be assumed to hold in the population. An analysis of residuals (hypothetical vs observed values for efficiency) was also performed. The residuals are shown in Table 4 which highlights that while macro users (residual = 2.2 in the high category) and spreadsheet users (residual = 3.6 in the high category) overall performed the experimental task efficiently, the CB-OMIS users (residual = 5.6 in the low category) were not as efficient in performing the experimental tasks. Table 4. Residual table: actual efficiency of the decision making process by tool used
TIME_SAV 1 Low 2 Moderate 3 High Column Total
Spread sheet 1 2 4.1 -2.1 4 5.4 -1.4 9 5.4 3.6 15 31.9%
Macro Spread sheet 2 1 4.4 -3.4 7 5.8 1.2 8 5.8 2.2 16 34.0%
TOOL_USED Count Exp Val Residual 1
Spread sheet 1 -1.1
Macro Spread sheet 2 -1.6
CaseBased OMIS 3 2.6
2
-.6
0.5
0.1
3
1.5
0.9
-2.4
15 31.9%
16 34.0%
TIME_SAV Low Moderate High Column Total
16 34.0%
Number of Missing Observations:
TOOL_USED Count Exp Val Residual
Table 5. Standardized residuals for actual efficiency of the decision making process by tool used
CaseBased OMIS 3 10 4.4 -5.6 6 5.8 0.2 0 5.8 -5.8 16 34.0%
Number of Missing Observations:
Row Total 13 27.7% 17 36.2% 17 36.2% 47 100%
0
If the standardized residuals are assumed to be drawn from the standardized normal distribution, a standardized residual value (z value) of 1.96 or greater is significantly different from the mean. This indicates that the observed efficiencies are drawn from a different population from the hypothetical one. The standardized residual for the case-based OMIS group in the low category is a significantly large residual (2.6). It can also be seen from Table 5 that there are substantial differences between the case-based OMIS users and the other two groups. Thus, the hypothesis (H2) that there is no significant difference in the actual efficiency of the decision making process when using the case-based OMIS compared with the other two decision aids was rejected. That is, it has been shown that the actual efficiency of the decision making process when using the case-based OMIS is less than that of the decision making process using either of the other two decision aids.
Row Total 13 27.7% 17 36.2% 17 36.2% 47 100%
0
6.2. Effects of Uncontrolled Variables Results presented in the previous section showed that a significant number of subjects who used the case-based OMIS had a low actual efficiency of the decision making process. This relationship might be caused by variables other than the tool used. This section summarises the tests performed to determine if other variables might have had an impact. One possible confounding variable is the subjects’ level of familiarity with the tool. While none of the subjects had used the specific tool before, the interface for the spreadsheet tool was essentially that of a conventional spreadsheet and prior experience in the use of this software may have had an impact on the ease of use of this tool. The interface for both the macro spreadsheet and the case-based OMIS were both significantly different from any common software product. Lack of familiarity with this style of interface could have had an effect. While the level of familiarity with the interface was not directly measured in the experiment, the introduction of the second spreadsheet (macro spreadsheet) was intended to compensate for differences in the perceptions of the interface. It is apparent that the actual efficiency of the decision making process using the macro spreadsheet is not significantly different from that of the conventional spreadsheet. Thus it can be assumed that the interface had an insignificant effect on the efficiency of the decision making process. There were two other characteristics that we attempted to control: credit transfer experience and gender. An analysis of the relationship of each of these variables with perceived and actual efficiency of the
1060-3425/98 $10.00 (C) 1998 IEEE
decision making process has shown no impact from either of these characteristics. Since none of the variables: level of familiarity with the interface, credit transfer experience or gender, have a significant effect on efficiency, we can be reasonably sure that the tool itself is the cause for the difference in actual efficiency between the use of the three tools.
7. Comparison of Efficiency Measures It can be seen from Table 6 that there was a distinct difference between the users’ perceptions of their efficiency using the case-based OMIS and their actual performance. Both measures of efficiency showed some relationship with tool used. The actual efficiency of the decision making process varied significantly between the tools used while variations in the decision makers’ perceptions of efficiency were not large enough to be significant. Also interesting is that the results indicate that the case-based OMIS is significantly less efficient than the other two tools. The reasons for this low efficiency are not clear but raise a number of questions. Table 6. Comparison of subjective and objective measures of efficiency
Low Mod High
Spreadsheet subjective objective
Macro subjective objective
CB-OMIS subjective objective
37.5% 18.8% 43.8%
68.8% 18.8% 12.5%
43.8% 18.8% 37.5%
13.3% 26.7% 60.0%
6.3% 43.8% 50.0%
62.5% 37.5%
If the reduced efficiency were due to unfamiliarity with the style of interface, we would expect the subjects using the macro spreadsheet would also be less efficient in their decision making process, since the interface of the macro spreadsheet was designed to simulate that of the case-based OMIS. However, as shown above, the subjects using the macro spreadsheet were not significantly inefficient in their decision making. A related issue is that of training. The subjects were familiar with spreadsheets but not with the concepts underlying the case-based tool. The subjects using the case-based tool were more positive about the decision making process but less confident in the outcomes. This indicates that more training was necessary. The reduced efficiency may also be due to the level of abstraction of the data presented. This was a major difference between the tools used. Although the impact of the degree of abstraction of the data available for decision making was not explicitly measured, the difference in performance of subjects using the more detailed (spreadsheet based) information versus the more abstract (case-based) information implies that the more
generalised information did not provide enough evidence on which to justify a decision. As a result, the subjects using the more generalised data (case-based) spent longer on each decision, even though all the relevant cases were retrieved by the tool. Additional evidence of this is provided by the responses of the experimental subjects using the case-based tool in open-ended questions on the questionnaire. Many of them stated that they needed more detail to make a decision. In particular, they asked for more detailed reasons for past decisions. If this difference is the cause of the reduced efficiency, it indicates that higher levels of abstraction may not be useful for decision makers. A certain amount of detail, even if not specifically relevant, may help the decision maker, at least from the point of view of efficiency of the process. It should be noted that the specifics of the software product used to build the case-based OMIS did not support a greater level of detail in that tool. Any further exploration of this issue would require a different design for the case-based OMIS. Another possible factor in the reduced efficiency is the degree of structure imposed by the tool on the decision making process. The software used to build the casebased system (CBR Express ) enforces a particular structure on the search process involving switching between a case description and the actions taken. This structure and the need for greater user training needed as a result may reduce the efficiency of the decision making process. Further analysis is being undertaken on the other dependent variables noted earlier in this paper. Weak relationships were found between the tool used and the appropriateness of the recommendations and confidence in the outcomes of the decision making process.
8. Conclusions Organisational Memory Information Systems have been recognised as means for intra-office communication, supporting the collection, storage and distribution of the knowledge and experience of the organisation. These systems are proposed to improve the quality of organisational decision-making processes by using the power of recent computer technologies, however, the true impact of these technologies is unknown without design approaches which have been adequately tested at the human level. This project has evaluated a case-based design for an OMIS which provides a decision maker with access to past experience in the form of cases using an analogical reasoning mechanism to retrieve the relevant cases. The results of this study gave experimental evidence of the impact of this form of OMIS for intelligent decision
1060-3425/98 $10.00 (C) 1998 IEEE
support. Overall the experiment showed that analogical search to recall previous decisions which are in some way “similar” to the current decision situation does not improve the efficiency of performance of the decision maker. At this time it appears that the specific design of the case-based OMIS has few, if any, benefits and may in fact lead to lower efficiency in the process of decision making. While the specific reasons for this lower efficiency are unknown, the variation in the level of abstraction of data provided and the structure imposed on the decision-making process by the OMIS and level of training needed are issues for further investigation.
[2]
[3]
[4]
[5] [6]
[7]
[9]
[10]
[11] [12]
9. REFERENCES [1]
[8]
E.W. Stein and V. Zwass (1995) "Actualizing Organisational Memory Information Systems", Information Systems Research, 6:2, June. L. Olfman and J. Morrison (1995) "Organisational Memory " in the Proceedings of the Hawaii International Conference on Systems Sciences (HICSS-28 ). V. Symons (1991) "A Review Of Information Systems Evaluation: Content, Context And Process", European Journal of Information Systems, vol.1, n.3, 205-212. C. Avgerou (1995) "Evaluating Information Systems by Consultation and Negotiation", International Journal of Information Management, vol 15, No 6, 427-436. Adelman, L. (1992) Evaluating Decision Support and Expert Systems, John Wiley & Sons, Inc. S. Maynard, F. Burstein and D.R. Arnott (1995) "DSS Evaluation Criteria: A Multiple-Constituency Approach" in the Proceedings of the 6th Australasian Conference on Information Systems, Australia. P. Reagan and J. Rohbaugh (1990) “Group Decision Process Effectiveness: A Competing Value Approach”, Group and Organisational Studies, vol.15, No1, March, 20-43.
[13]
[14]
[15]
[16]
[17]
J.S. Chandler (1982) "A Multiple Criteria Approach for Evaluating Information Systems", MIS Quarterly, 66.6174 (March). F.V. Burstein, H.G. Smith, A. Sowunmi and R. Sharma (1996) "Organisational Memory Information Systems: a Case-based Approach" in the Proceedings of The United Nations University Workshop 'DSS for Sustainable Development', Macau, February, 1996. F.V. Burstein (1995) "IDSS: Incorporating Knowledge into DSS" in F. Burstein, P. O’Donnell, A. Gilbert (ed), First Melbourne Workshop on Intelligent Decision Support Systems Proceedings, Department of Information Systems, Monash University, Melbourne, 93-96. Kolodner, J.L. (1993) Case Based Reasoning. California: Morgan Kaufmann. J.L. Kolodner (1991) "Improving Human Decision Making through Case-Based Decision Aiding", AI Magazine, 12:2. F.V. Burstein, A. Sowunmi; H.G. Smith, A.V. McMillan, and C. Cole (1995) "Building an Intelligent Decision Support System: a Case-Based Approach" in Proceedings of Pan Pacific Conference on Information Systems, Singapore. R. Sharma (1996) “Hierarchical Classification Approach to Knowledge Acquisition for Case-based Decision Support Systems”, unpublished Masters Thesis, Monash University. J. Morrison and M. Weiser (1996) "A Research Framework for Empirical Studies in Organisational Memory" in the Proceedings of the Hawaii International Conference on Systems Sciences (HICSS’29), vol.3, 178187. P. Todd and I. Benbasat (1992) "The Use if Information in Decision Making: an Experimental Investigation of the Impact of Computer-Based Decision Aids", MIS Quarterly, September, 373-393. P. Todd and I. Benbasat (1991) "An Experimental Investigation of the Impact of Computer-Based Decision Aids on Decision Making Process", Information Systems Research (2:2), June, 87-115.
1060-3425/98 $10.00 (C) 1998 IEEE
