the communication and elicitation method - Semantic Scholar

E F H A v t oRr & r \ l - u R \ 4 \ T I O l\ l ( H N ( ) L ( ) C \ . 2 0 0 0\ o. L . 1 9 , N ( ) .5 , . 11 5 l l 7

tär

Knowledge acquisitionin ecologicalproduct design:the effectsof computer-mediated communicationand elicitationmethod JÜRGEN SAUER. SIMONE SCHRAMME ANdBRUNO RÜTTINGER Institute of Psychology,Darmstadt University of Technology,Hochschulstr.1, D-64289 Darmstadt, Germany; e-mail: sauer(a,osvcholoeie.tu-darmstadt.de

.{bstract. This article presentsa study that eraminesmultiple effectsof using differentmeansof computer-mediatedcomlllunicatior.rand knowledge elicitation nrethodsduring a prodr.rct design process. The experinrental task involved a typical scenario in product design. in which a knowledge engineer' consults two experts to generateknowledge about a design issue. Employing a 3x2 between-subjectsdesign, three conference types (face-to-face. computer. multimedia) and two knowledge elicitation methods (structured interview, network technique) were compared. One hundred and eight participantstook part in the study. The1,'wereassignedto 36 groups of three.Quantitativeand qualitativeperformancedata were collectedand the group processesu,ith the IPA method analysed.The results showed that the coüipr-rterconference -qrouprvasgenerallymore productivethan the two other gror.rps durin-ethe conference.However. participants were unable to nraintirintheir higher performancelevelsin a later task where the conferenceresults had to be edited before being fed into an expelt system. As expected. the computer conference group showed the lowest socio-emotional content durir-rginteraction. The network technique was largely more productive than structured interviewing,though it was more time-consuming. Furthermore, the findings suggested that both the network rechnique and computer conferencing achieved their higher productivity in knowledge generation onlv at the cost of inlormation processingdepths,resultingin poorer performance fbr subsequenttransferactivities.The resultsare discussed rvith legard to organizationalchoicein managingconferences ofthis kind.

In order to manage the conplex requirementsof the designprocess,a core designteam is formed to direct the various stagesof the process.In addition to the core team, experts are consulted concerningvarious aspects of the designactivities,as this allows the widening of the knowledge base of the design team (Scott et al. 1991). The experts usually come from very different professional backgrounds, such as marketing, materials science,production processes.control or environmental science.Although the experts are not part of the core desi-enteam and are involved on a non-permanentbasis. they play a very irnportant role in supporting the core group (Ehrlenspiel1995). As designershave to consult expertswho may work at a distant location, the use of computer-mediated communication (CMC) is not uncommon to save costs and time of travelling. While there has been an increasingproliferation of systemssupporting CMC in a rviderangeof areas(e.g.mailing, project management. conf-erencing, diary managel.nent;for an overview see. Johansen 1988), the effects of using CMC are very context specilic and lequire assessmentof its rnultiple effects on important valiables, such as performance parameters, participant satisfaction, socio-emotional content (McCarthy and Monk 1994).

1. Introduction

1.1 Contputer-mediated conurtunication In the context of product design. teams have been widely r.rsed to take advantage of the rvider knowledge base of groups compared to individuals, and the complexity of the process renders this task virtually unmanageable by a single person (Frankenberger 1997).

This article is concernedrvith holding team meetings in a" conf'erencesetting, which is an increasinglyused application for CMC instruments. While face-to-face communication is still of high importance for qonfer-

Beltuviour & I nl rtrntrtt i rtn Tct hnrt I og.t I S S N f ) 1 , + , 1 - 9 1 9pXr i n t I S S N 1 3 6 2 - 3 0 0 1 online r 1000Ta\lor & FrancisLtd h t t p : u n u . t . r i l J f( i , l l k l o u f r l i l l \

316

J. Suueret al

er,ces.with changesin work organization(e.g.telework, outsourcing, globally operating organizations) others forms of communication have shown to be viable alternatives.There are severalalternativesavailable to face-to-facecommunication, such as video conferencing and computer conferencing(Johansen1988).They differ with regard to the fidelity with which they simulate aspectsof face-to-facecommunication.For the purpose of drawing a comparison between meelnsof communication, they may be rated on three dimensions:media richness (or communication bandwidth), geographical scope and time delay in communication (Barua et al. 1997). This helps produce a utility profile for each meansof communication.which also takes into account the overall circumstancesand the task for which it is being used. For exarrple. face-to-facerneetingsenjoy a high level of media richness but suffer from low geographical scope (i.e. groups must not be spatially distributed). Communication media also differ with regard to the underlying social context cues during comrnunication ( S t r a u s a n d M c G r a t h 1 9 9 4 ) .W h i l e m u l t i m e d i a c o n ferences(e.g. with a PC-basedvideo link) model much of the bandu'idth of communication channels used in face-to-face communication (auditory and visual information of different kind, non- and para-verbal cues). in the case of computer conferencesthis is reduced to text-basedinformation. However. reduced bandwidth does not necessarilyhave to be a disadvantage. As some people have argued, computer conferences have a number of benefits. For exarnple. the participation rate is more equally balanced (i.e. the discussion is not controlled by dorninant group men-rbers)than in face-to-facemeetings (Sproull and K i e s l e r 1 9 8 6 .W a l t h e r 1 9 9 5 ,S t r a u s 1 9 9 6 ) .I t h a s b e e n found that a more balancedparticipation rate generally makes better use of group resources(Rüttinger et a/. 1994). Furthermore, it has been argued that in computer conferences.differences in status do not affect communication patterns to the same degree as during face-to-facecommunication (Sproull and Kiesler 1986). However. the findings are not unequivocal since other researchsuggestedthat status did affect the participation and influence of group members even (Weisband et al . 1995). during computer cor.rferencing Multimedia as a more recent CMC tool usesseveral media (e.g.video. audio. text. animation, graphics)that are integrated in a single communication environment. Due to its multiple mode featuresthat produce a higher level of media (or interaction) richness.it resemblesin some ways face-to-fhce communication. though the quality and quantity of non-verbal cues is lower. Furthermore. additional demands accrue because of the multiple task nature of the multimedia environment,

which requires integration of different task elements. Due to its more recent emergence,there has been little researchinto multimedia applications,though one may draw upon work examining video conferencesas the closestanalogue(e.g. Finn et al. 1997). CMC researchhas also pointed at differencesin taskorientation and relationship-orientationin communication patterns. In particular, computer conferencing has been associatedwith higher task-orientation and lower socio-emotional content (Rüttinger et al. 1994). Work by Rice and Love (1987) also con{irmed the dominance of task-orientation over socio-emotional content in CMC, though they argue that socio-emotionalmessages are much more prevalent than commonly assumed. Furthermore. there is evidence for a time-based effect. Socio-emotional orientation increasesin CMC if the group process evolves over a longer period of time, leading to a diminishing difference between face-to-face and computer conferencing (Walther and Burgoor 1 9 9 2 ,W a l t h e r 1 9 9 5 ) .

1.2. Knutrledgct'licitation While the effects of CMC have attracted considerable research interest. few studies have examined the use of elicitation techniquesin the context of product design. There is a wide range of techniques available for knowledge elicitation, such as structured interviews, simulations, sorting tasks and role play (see Cordingley 1989). One may distinguish between different types of knowledge elicitation techniques, based on the source of the information or the degree of formalization of the elicitation technique. A classification system by Konradt (1992) distinguishes between three types: inquiry techniques, observation, and testing. Inquiry refers to direct person-centred data acquisition methods, such as interviews, focus groups and questionnaires.Observation is concerned with techniques such as videotaping and verbal protocols. The category testing refers to techniques that use structured material that is to be administered with standardizedinstructions. Examples of these are repertory grids, network and charting techniques. Each of the techniques naturally has its strengths and weaknesses.which means its effectiveness is largely dependent on the context in which it is employed (see Bainbridge 1979, Cordingley 1989). Network techniques (see Kirwan and Ainsworth 1992) support the organization and structuring of knowledge, with less variation in performance as a result of the abrlities of the knowledge engineer.The elicrtedknowledgeis well structured,which considerably facilitates subsequentdata analysis (see method for a

gn Ktrt,ult'tlgt d(quisitiun itl et'oIogit ul prr.ttluct de,si more detailed discussion). Interviewing, in contrast, requiresconsiderabletime and effort during the process of data analysis.It is a flerible method that is applicable to a wide range of subject areas. Furthermore, it requires less specialized trair.ring for the knowledge engineerthan network techniques.

1.3. The presentstudt' So far CMC research has primarily lbcused on 'process' criteria (i.e. participation ratc. interaction patterns) while the use of performance indicators has usually been limited to rneasuringconferenceduration. Furthermore, the task scenariosusedhad generallylittle in common with typical applicationsof CMC in the real world. Against this backgror.ind,the article airls to present :mpirical data about the effectsof using different forms of CMC in a realistic team-basedtask environment. modelled on the product design process in extended design teams. In this study, a range of performance measuresand processvariableswere emplol'edto gain a fuller picture of the multiple effects of usir-rgdifferent conference types. The multi-level analysis examined various aspects of conferenceeffectivenessas well as group interaction processes(see McCarthy and Monk 1994). A person-centredand a method-centredknowledge elicitation technique was chosen for the experimental rvork to addressthe question of horv the match betrveenconl'erencetype and knowledge elicitation can be improved. It was expectedthat computer conferencingwould be more productive than the other two conf-erencetypes because it is more task-focused due to its small communication bandwidth (see also Rüttinger et al. 1994).This should be advantageousfor the task used in this study, which does not require social context cues. These rvould be needed for tasks with hi-qherambivalence levels(Straus and McGrath 1994).Tbe low media richness of computer conferences should also be reflectedin a lower socio-emotionalcontent of communication patterns (i.e. positive as well as negativeones), rvhich should be dernonstrated by the IPA analysis ( B a l e s1 9 7 0 ) .D u e t o i t s c o m p l e xt e c h n i c a el n v i r o n m e n t , the multimedia conferencewas expectedto show poorer performance than the face-to-face conference. The socio-enotional content was expectedto be highest in face-to-tace conferences since it allowecl the most immediirte tbrm of communication. Finally. it was predicted that a netrvork techniquewould be the more effective elicitation method since it provides a better structure to the problem-solving activity (see Hacker and Jilge 1993).

317

2. Method 2.1. Desicn A 3 x 2 betrveen-subjects design was emplo.ved.rvith 36 teams of three members each. With the unit of analysisbeing the tearn, this meant that there were six casesin each cell. The independentvariable conference type was varied at three levels:face-to-faceconference (FTF), computer conference (CC) and multimedia cont-erence(MM). The second independent variable knowledgeelicitalion techniquervasvaried at two levels: structured interview (SI) versus HSLT (a network techniquewhich is describedbelow). 2.1.1. Confbren('et.t'pe: During FTF participants sat around a table, allorving for direct communicution. During CC. participantswere seatedin separaterooms, using Silicon Graphics Workstations (Indys) for communication. Using different rooms was considered a better simulation of this kind of conferencethan merely separatingparticipantsby screenswhile being situatedin the same room, as it has been fiecluently done iu ltboratory-basederperiments.Being placed in separirte rooms, too, and using the sllrre work statior"rs, participants in lhe MM condition communicated by 'Inperson' (Silicon means of a confelencesystem,called Graphics). There were video pictures of the other perrticipantspresentedon screen while auditory informaticln was receiveclby headphones.Fllrthermore, the system provided an embeddeclrvhiteboard fbl reading and writing messages. 2.1.2. Krtott'ledge elic'itation tet:hnitlue: The effectiveness of two knou,ledge elicitatiou techniques were compared as a tunction of the conference type employed.While in practice interviervirrghas been very widely used for the purpose of knowledge engineering for design(Scott ct ul. l99l), attempts have been made to use a more structured approach to knowledge elicitation (see Hacker and Jilge 1993). A further diliculty is that the successof SI is strongly dependent on the preparation rr-rade by knowledge engineersprior 'right' t o d a t a a c q u i s i t i o na n d o n t h e i r a b i l i t y t o a s k t h e questions.Thus designershave asked for an elicitation technique that produces more structured material. For this purpose! a network techniclue, called HSLT (HeidelbergerStruktur-Lege-Technik)was chosen,that hzrs been widely used in the German psychological end Gröben 1979)A c o m m u n i t y( S c h e e l a . s HSLT is less known in the English researchliterature, it is brief'ly describedbelow. This method aims to create an explicit structure of the knowledge thirt an expert has about a particular

J. Sauer et a\

318

area. In the first phase, the principal concepts are lvritten on cards. In the second phase, the concepts are related to each other by using a number of rules that define the relationship betweentwo concepts(e.g. 'A is 'C a part of B', is an example of D', 'E is a feature of F'). The rules that are used during a sessionmay be selectedon the basis of the kind of knowledge to be elicited. HSLT aims to produce a hierarchicalsemantic network as used in cognitive psychology for the representationof human knowledge (e.g. Eysenck and Keane 1995). These networks distinguish between 'objects' 'relations'. and Objects refer to animate and inanimate things (e.g. bear, hut, tree) while relations refer to the link between objects and their characteristics (e.g. subordinate,an example of, a feature of). As one is more concerned with abstract terms. one will refer to objects as 'concepts'.An example of a network structure createdby HSLT is presentedin figure l. The task in this example was to design an environmentallyfriendly PC. The structure shows a number of components of a PC, which were supplemented by features that participants considered to be relevant for environmental friendlinessof the PC.

2.2. Participunt.s A total of 108 participants took part in this study. They were all male studentsof Darmstadt University of Technology. Their ages ranged from 20 36 years (mean: 24.0 years).All of them were reading technical subjects at the university (electronic and mechanical engineering, computer science,etc.). To fulfil selection requirements,participants neededto have some knowledge in the specific subject areas. This was verified by using a basic selectionprocedure comprising an interview and a short written test measuring knowledge of ecologicalissuesin computer design. The test comprised six multiple-choice items (e.g. 'Which one of the following components of a PC consumesmost energy?').The resultsof the test were the basisfor assigningthe participantsthe experimentalrole of a novice or an expert. While in general a minimum score of three was required for participants to be considered an expert, a subjective assessmentof the candidate based on the interview sessionwas also used to decidewhat role they should play in the experiment. However, if a prospective participant did very poorly on

PC t

P"rt

Monitor

-Fealure

Processor

Feature

Size, resolution, colour depth, refresh rate, recycling, energy consumption, radiation level

+

PVC,heavymetal,

clock frequency, high processing speed 'ti\

Hard disk

Workingmemory

Casing

Farfr

SCS/,E/DE

rra

f*tr*

l

Noise level

Sizein MB

Kind of flame retardent used

Production ofporsonous gases

Figure 1. Example of a network structurecreatedby HSLT: Relationsare presentedin boxes (e.g. is a feature ol); conceptsare presentedin italics (e.g. hard disk); arrows symbolize'to affect';plus-signsymbolises'and'.

Kuou'ledgeacquisitionin ecologicalproduct design the test (only'0 I items correctly answered),he was not allorved to take part at all, as his knou'ledgebase was c o n s i d e r e di n s u f f r c i c n t

2.3. Depcrrdentttariubles The experimental sessions in the FTF and MM conditions u,erevideotaped.For the CC condition. the on-screeninfbrmation was loggedinto a resultsfile at twominute intervals.representingthe experimentalprotocol. Four central performance variables r,',ererneasured, and two qualidistin-euishingbetweentwo clr"rantitative tative perlormance measures. 'Conference duration' was an indicator of the time requirementsof the knowledge elicitation process.The productivity of the knowledge elicitation sesslonwas 'knowledge generation measured by a index', which rdicated the number of -eeneratedconcepts per time unit. Both variables were quantitative performance measures.Their strengthsis the objectivity rvith which aspectsof perlbrmance are evaluatedbut they are less usefbl for the assessment of overall performance. Therefore,two qualitativeperforrnanceindicatorswere picture. taken as a supplementto gain a mol-econ-rplete First. a qualitative'evaluationof the conferenceoutcome' was carried out by two expert ratels.using a scoring 'data system based on a 6-point scale.Second,the for expert system input' were also evaluated by the raters employing the sanrescale.A more detailecldescriptionof each performancerreasureis given in the resultssection. Of no lesser irnportance was the collection of data about the interaction process c'luring the conference sincethis may be indicative of communicationefficiency and group climate. This u,as achieved by using the interaction processanalysis(IPA) method. developedby Bales (1970). which is one of the rnost widely-used nethods for analysing group processes.Basedon a set of 12 categories.the IPA method aims to collect a n u m h e r o f i n d i c c sa b o u t i n l e r i l c t i o ni ) ü t l c fr r si r r g r o u l ' r s . It emphasisesthe processof cor.nmunicationrather than the content of the message.Fol the purpose of this study. rt allowed us to test the hypotheses(seeabove) with regard to the socio-emotionalcontent of the three conference tvpes. The frequency distribution of messagesacrosscategoriesrvasexamined.a central aspectof the many analr-sesoffered by the IPA method.

different tasks of the group members required that novices(i.e. knowledge en-tineers)and experts received in part different training regimes,rvhich are described below. Of the total number of participants. -16 were lrained as knowledge engineers,and l2 as experts.An overview of the training regimesrnay be found in table l. The first part was identical for both groups, in rvhich they were familiarized with a purpose-built software, 'PC called Configuration'. Built for the purpose of the clverall researchproject, this is a computer-supportecl infbrmzrtion system,designedto provide decision support to individuals for purchasing compr,rters.'PC Configuration' provides pricesof hardware components and their compatibility. which allows the decisionrnaker to select a confi-eurationthat fulfils functional requirements(e.g.job den,ands)trnd is cost-effectiveat the same time. However, "PC Configuration' does not contain any information about the ecologicalproperties of hardware components. Once the participants \\'ere 'PC familiar with Configuration'. they moved on to acquiring basic information about the subject area of 'PC configuration'. ecologicalaspectsof The infolmation acquisition processdiffered however in length and coutent between knowledge engineersand expelts. as shown in table 1. All participants received the same training, independentlyof the communication medium 'PC configuration'). used(i.e. FTF were also trained on

Table 1. Structule of trainins and testine session. Lr(,

itt::.

cnlütcct

S 1 ! d \r r r \ r l l r n

I ailrüpn

ü.11\ ttJ

b,pc4

rnlilmrlron

ilbout erinliplr.rli.il ol ecoloeical l'( s (10 n)ilr) lmrli il i\irnn

$ ith l. rn$ i.dr!

f.eiaär*t knowlcds€ insubiect rrca140 I nrin)

2.4. Truinittg ()üline ol ecological crilcria to

Considerable training was necessarybefore taking part in the experimental session.Training took place over two sessionsand lasted for uo to 3.75 hours. The

319

bc 'iltluded ir) (iccrsiur supporl slslenr (cr 40 nin)

320

J. Sauer et al.

2.4.1. Knott'ledgeengineer: With training taking place in groups of two. knowledgeengineersacquired knowledge of the subjectarea (i.e. configuration of ecological PCs) by reading two articles from computer journals. This was to reduce possible knowledge differences betweenparticipants. Furthermore. it provided participants with a suflicientknowledgebasethat would enable them to apply a knorvledge elicitation technique successfully.Following the basic training session,the knowledge engineerwas made familiar with his respective knowledgeelicitation method. For the structured interview technique,the task was to develop the interview schedule. To gain some familiarity with the technique. knowledge engineers were asked to practise it in the context of football. After some practice has been gained, knowledge engineers in groups of two developed the interview schedulefor the testing session.For HSLT, the training sessionswere of a similar structure. After having been given some information about the rationale of the technique,knowledgeengineersalso practisedHSLT in the context of football. 2.4.2. Erperts: With experts also being trained in groups of two. they were askedto carry out a document analysis of relevant material about ecologicalissuesin computer design. They were given 12 articles from computer journals to enhance their knowledge of the subjectarea. Altogether, 120 minutes were allocatedfor this activity. that is, three times longer than the knowledge engineerswere given time to read. In the following session,the experts were given the opportunity to practice the knowledge elicitation process with a knowledge engineer. During task completion in the training sessions,all participants already worked with the technicaltask environment to be used in their experimentalcondition (i.e. CC or MM system)to obtain a satisläctorydegreeof familiarity for the experimentalsessions.

some artefacts in CMC research emerged because of experimentsbeing cut off too early, which did not allow for more relational communication patterns to occur in the generallyslower CMC. When the conference finished, the experts had completed their part of the experiment (see table l). The knowledge engineer was then asked to make suggestions for the enhancement of a computersupported information system by including ecological criteria in 'PC configuration'. The knowledge engineer was told that theserecommendationswere subsequently to be implemented in the next version of 'PC Configuration'. though this step was not part of their task.

3. Results 3.1. Group perJ'ormance 3.1.1. Con/brenceduration'. The duration of the conferencewas an indicator of the time requirements of the knowledge elicitation process (seefigure 2). The results showed that multimedia conferences took longest to complete (8 1.7 minutes), followed by computer conferences(77.8 minutes)and face-to-facesessions(62.3 minutes). While this difference was statistically significant (F - 4.36; df :2,30: p