Proceedings of DETC’04 ASME 2004 Design Engineering Technical Conferences and Computers and Information in Engineering Conference September 28-October 2, 2004, Salt Lake City, Utah, USA Proceedings of DETC2004: International Design Engineering Technical Conference September 2004, Salt Lake City
DETC2004-57600
DETC2004-57600 USING DIGITAL LIBRAR IES TO ENHANCE DISTRIBUTED DESIGN TEAM PERFORMANCE Neal Juster Department of Design Manufacture and Engineering Management University of Strathclyde Glasgow, UK
[email protected]
Hilary Grierson Centre for Academic Practice University of Strathclyde Glasgow, UK
[email protected]
David Nicol Centre for Academic Practice University of Strathclyde Glasgow, UK
[email protected]
Bill Ion Department of Design Manufacture and Engineering Management University of Strathclyde Glasgow, UK w.j.ion@ strath.ac.uk
Angela Stone Department of Design Manufacture and Engineering Management University of Strathclyde Glasgow, UK
[email protected]
Andrew Woodhouse Department of Design Manufacture and Engineering Management University of Strathclyde Glasgow, UK
[email protected]
ABSTRACT This paper describes the purpose and initial results of a 5year project to develop, implement and use a testbed to improve the teaching and learning of students partaking in global team based design projects. It is expected that the project will, when complete, fundamentally change the way design engineering is taught by combining the use of digital libraries with virtual design studios. The paper describes the first stage of the project to inculcate basic digital library skills in undergraduate students. Lessons learnt in this first stage are detailed. The project team will be using these lessons to design the next stage of the project. INTRODUCTION In the first quarter of 2003, The Joint Information Systems Committee (JISC) in the UK and the National Science Foundation (NSF) in the US jointly funded four projects under the Digital Libraries in the Classroom [1] initiative. One of the projects, Digital Libraries for Distributed Innovative Design Education and Teamwork (DIDET) [2] is a collaboration between the University of Strathclyde in Glasgow, UK and Stanford University, CA, USA. This paper outlines the
rationale and purpose of the DIDET project and describes some initial findings of students using digital library resources to aid team based design activity as a first step towards running globally distributed team based design projects supported by digital libraries and associated technologies. The goal of DIDET is to enhance student learning opportunities by enabling them to partake in global, team based design engineering projects, in which they directly experience different cultural contexts and access a variety of digital information sources via a range of appropriate technology. The design and development of new products for the global marketplace requires engineers to perform in internationally situated teams. Modern communication technologies such as virtual environments, digital libraries, shared workspaces, video and audio conferencing and email are increasingly being used to enhance performance by supporting information creation and sharing. Therefore, in higher education, it is necessary for design engineering students to learn to work in distributed teams by utilizing cutting edge information management technologies. The process of design is context dependent and openended, and therefore, does not revolve around a specific body
1
Downloaded From: http://proceedings.asmedigitalcollection.asme.org/ on 10/15/2014 Terms of Use: http://asme.org/terms
Copyright © 2004 by ASME
of information or knowledge. This poses a problem for design education since teachers cannot predict in advance exactly what information and knowledge students will decide they require access too. Coaching , where the course tutors guide and facilitate rather than try to specify what information should be used is a commonly used approach in the project based elements of engineering degree courses. Project based teaching is a common method of teaching the ‘design’ component of an engineering degree programme. A related pedagogy, Problem Based Learning [3] (PBL), is now being used increasingly in the more ‘science’ based parts of engineering degree courses such as mechanics and dynamics. In PBL students may also need access to information that has not been sourced explicitly by the teacher. The educational paradigm shift from teaching to coaching and the increased use of PBL requires students to have access to as wide a range of information as possible. In design it is usual that much of that information lies outside the students' immediate domain. Digital libraries provide an excellent opportunity for extending the range of information available to design students. However, as discussed by Currier et al [4], digital libraries bring their own problems for all stakeholders including library staff, teachers and learners. Previous experiments conducted by Strathclyde [5] have shown that virtual studio environments, discussion fora and synchronous chat facilities can aid communication between design engineers. However the same study also shows that barriers of culture, discipline, distance, network and technology may prevent successful use of ICT. In a separate investigation by Stanford [6] the important role that expert coaches play in facilitating the successful adoption of new technologies by design teams, and the key learning mechanisms within design activity through which knowledge acquisition takes place have been identified. The DIDET project is attempting to develop a testbed to integrate a digital library into the classroom. This testbed will be used to teach students state of the art team based engineering design process skills augmented by information archiving and retrieval skills in a digital library environment. The nature of the state of the art engineering design processes that will be used will require the students to interact with and generate large amounts of information. Being able to quickly ascertain the nature of the relevant design information, to determine if it exists, and to access it and utilize if it does exist should accelerate and improve the design process. Also, for the creation and reuse cycle to continue, it is imperative that, when new information is created, it is converted to appropriate digital formats, indexed, and rapidly made accessible on the World Wide Web. The DIDET project is attempting to inculcate in students the ability to utilize digital library technologies and information handling behaviors. It is hoped that these new skills and behaviors will improve their design performance. This paper describes the first stage of the project to inculcate basic digital library skills in undergraduate students. The work described in this paper provides a base-line from which future teaching experiments can be designed. The next section outlines the tasks set for the students. This is followed by evaluation of the data collected and discussion of the lessons learnt.
PROJECT TASK DESCRIPTION Working in teams of four, forty third-year students studying for a MEng/BEng in Product Design Engineering were given five weeks to design and prototype a ‘can crushing device’ for use in the domestic market. The project was an assessed part of the course (approximately 4% of the total marks available to students over all subjects taken in the third year of the degree program). Teams met face-to-face but the tasks and assessments were designed to encourage online storage and sharing of information and resources. The technology made available to the students was a customised version of open-source groupware, TikiWiki [7]. In addition to enabling online-storage and sharing of information, TikiWiki enables students to communicate with each other via email, chat, forums and its shoutbox. TikiWiki also provides tools for the tutors to monitor its use by students. The can crusher project was designed to focus on information management, team working, use of an online support tools, proof-of-concept modelling and product evaluation. The five weeks of design activit y were preceded by a week of introduction and followed by three weeks of reflection and feedback. Each week students had two hours of formal contact and were expected to work individually and as teams for at least an additional four hours. The teaching programme was structured as follows: Week 0 Introduction & familiarisation Week 1 Accessing & gathering information Week 2 Storing & organising information Week 3 Exploring and selecting concepts Week 4 Developing & testing concepts (prototyping) Week 5 Presenting (crits) Week 6 Reflecting (videos) Weeks 7 & 8 Feedback The above programme consisted of three main phases 1. Information Management. The students had not used TikiWiki prior to the project but they were familiar with the online environment, WebCT [8], having used it for at least one project in the previous academic session. The students were given introductory notes, demonstrations and hands-on experience of TikiWiki in Week 0. Each team was required to investigate a particular aspect of the project (e.g mechanics, safety, recycling, ergonomics). Each team was expected to create an information resource using Wiki pages (web pages within TikiWiki). In order to facilitate the sharing of resources at the later concept generation stage, the teams were required to supply metadata when they uploaded files into their workspace. The metadata comprised a file name, at least three keywords and a description for the resource. A short tutorial was provided on the use of keywords and metadata. These processes covered weeks 1 and 2. 2. Concept Generation. In week 3 each team’s Wiki pages were made available to all of the other teams. Using these resources teams were expected to generate at least three design concepts for a can crusher. 3. Development of can crusher. Teams developed their best concept through to proof-of-concept model stage. This included documenting the process. Each team was required to
2
Downloaded From: http://proceedings.asmedigitalcollection.asme.org/ on 10/15/2014 Terms of Use: http://asme.org/terms
Copyright © 2004 by ASME
make two short presentations; one around the proof-of-concept model made informally to the class tutors and the other a 5 minute video, produced in the University’s TV studio outlining the design process and what they had learnt from the experience. The phase covered weeks 4 to 6. EVALUATION Evaluation was designed to address the following key issues: ?? How do students manage, store, link and use information and resources in a shared workspace (including student use of metadata)? (This part of the evaluation used data from Phase 1 of the student activity) ?? What is the value of existing engineering internet based collection such as EEVL [9], SMETE [10] and NEEDS [11]? (Phase 1) ?? Does the use of structured information and resources support design learning (in groups) and project working? (Phase 2) Evaluation was conducted using a variety of methods. It focused primarily on students’ and teaching staff’s perceptions and experiences. An evaluator observed and met with teams in focus groups for 10 minutes during the weekly class studio sessions. At the beginning of each class, one minute reaction cards (cards with 4 questions to be answered by students) were handed out. The cards were gathered in at the end of the class. In week 3 the evaluator observed the team presentations of their information structures, (linked Wiki pages) and discussed with teams the effectiveness of the structures choosen. At the end of the project, a feedback session for each team with 2 members of staff allowed both staff and students to give feedback on their overall experience and project outcomes. Also, at the end of the project there was a focus group meeting with all the teaching staff. Objective data was provided through weekly analysis of TikiWiki team pages and files. This provided information about how students had interacted with the resources and what metadata they had created. INITIAL FINDINGS – PHASE 1 Phase one of the project required each team to generate an information resource that would be used by all teams in Phase 2. This section identifies the main information searching and archiving strategies use by the teams in Phase 1. Information Gathering. Most of the students relied heavily on the Internet and in some cases on books available in the library for finding information relevant to their projects. Nearly all students ignored other informa tion sources available in the University library. The library contains a comprehensive collection of electronic resources such as electronic journals, online library catalogues, scientific and technical databases, eg. Aluminium Industry Abstracts, Compendex database, Materials Business File, CSA Materials Science. Many of these sources contain references to “can crusher” and related technologies. A a search in the online US Patents database [12] will return over 80 patents whose titles include “can crusher”. Students failed to make use of the patent resource. Use of TikiWiki Digital Resources. The TikiWiki shared workspace had direct links to eleven digital resources (eg
British standards, design guides) selected by the teaching staff for the students to access. The links included three Digital Repositories of engineering resources – SMETE, EEVL and NEEDS. Although students were advised to search through these resources, their use was disappointing. Students commented that they found the current on-line resources, “difficult to use and understand”. The majority of the students found that they could not find useful information relating to the project and that searches did not furnish specific enough information. When asked why they did not make extensive use of the Digital Resources, some students noted that they preferred to use sources they were more familiar with such as the internet (i.e. Google) and libraries. Problems encountered when searching for information. There was concern by some groups that there was too little information to find about their topic. Other students were concerned that there was too much information. Students found it hard to use keywords that sufficiently narrowed the responses from search engines. Storing and organising information. Online storage and organisation of the information was supported by customised TikiWiki groupware. Within TikiWiki each team could store information in private file and image galleries. Once a resource was created (as pdf, word doc, jpg, etc.) it could be uploaded into the team’s private area. Each resource required a filename, with the option of a description a set of keywords prior to archiving. In general the teaching staff thought the gathered information was not being organised or structured as well as might have been expected. Once the students had found the information they did not seem to be thinking about it – ordering it, editing it or guiding others through it. They were less concerned about the way their resources were structured for others to use. However, during the reflection week many students noted that it was useful to have access to all the information and resources and to have that information structured in some manner. Very few teams presented the information in such a way that it would be easy to navigate quickly to the important information. It seems that the students were unaware that they were to do this. Most teams failed to prioritise the information they have found, organising with little hierarchy to the information. Irrelevant information often lay within the Wiki pages as a result of students’ reluctance to edit material they found to the topic they were researching. Use of Metadata. Almost from the beginning of the project it was observed that students were not using keywords and descriptions when uploading their resources. Some noted it took too much time to do this and one student noted that since TikiWiki “…didn’t force you to put in keywords and descriptions they weren’t going to”. Quite a few students said they did not always use keywords and descriptions thinking they could go back and insert them later. (Something which is not possible with TikiWiki.) Information Quality. The quality of the information sourced by the students was of a fairly high standard. Some teams had found very useful information. Staff noted, this information, if harvested properly, would contribute to an
3
Downloaded From: http://proceedings.asmedigitalcollection.asme.org/ on 10/15/2014 Terms of Use: http://asme.org/terms
Copyright © 2004 by ASME
excellent resource over a few years. The resources were rich in media. Formats included – text, images, animations and links to other web sites. However there was a tendency to rely too heavily on text with many students commenting that pdfs and some word documents were too long to read and that the use of summaries would have been useful.
correspond to the day(s) prior to formal project deadlines. The week of 26 October was the week of the final video presentations. This pattern of student activity is consistent with the teaching team’s observations of “traditionally” run design projects.
INITIAL FINDINGS – PHASE 2 Phase two required teams to generate concepts for the cancrusher using appropriate information collated by the other teams in Phase 1. This section identifies the main information searching strategies use by the teams in Phase 1. Searching the created information resource. Students used two different methods to search through the created resources. One method involved navigating through each team’s Wiki pages and the other required searching through the file and image galleries using keywords. The first method was adopted by all of the students interviewed but only half had looked for resources using keywords. The original intention was to have the teams search through the resources using keywords but not enough emphasis was placed by staff at the beginning of the project on the appropriate use of keywords when archiving the information. Students reported a lot of hits due to very general descriptions being used for the resources. Descriptions were also sometimes misleading and filenames totally confusing (there were a large number of files called “can_crusher_n.xxx.” Usefulness of Resources. Students reported that although they found useful resources, they also found irrelevant information and information that was too general on the Wiki pages generated by other teams. Students often got lost in the Wiki pages or found the navigation hard. A common comment was: “At times there were too many ‘clicks’ to get to the information and not much useful information when you got there.” Students reported it took time to find the best and most important information within TikiWiki. Students were critical about the size of some of the resources. Some were too long: some had too much text and not enough graphics. Some of links to external resources did not work. Students also commented on the lack of hierarchy or priority to the structuring of the resources, commenting “…lot of information has been ‘thrown onto’ the Wiki pages with little structure or relevance.” TikiWiki Experience. TikiWiki was regarded very highly by the students. The majority of teams found the first week of using TikiWiki very hard and daunting but by the third week, with continued use, they were more familiar with it. Several teams noted that by the end of the project they were very confident with it and ready to use it on the next project. Students were positive in its use to support teamwork, as they need not all be together to work, commenting, “…good for sharing info with group as it’s difficult to get everyone together” and, “It seems good if we can access it from different places.” TikiWiki’s communication tools made the experience more collaborative allowing for working together at weekends. Figure 1 shows the pattern of student usage. 30 September corresponds to the start of week 0. Most of the peaks
Figure 1: Student Use of TikiWiki Students found TikiWiki useful as a ‘store’ for their created resources noting it was “…convenient…to have all the available information in one place.” However, as a searchable repository the findings were not so positive. By not enforcing the use of keywords and descriptions for the resources and by providing an alternative method of searching via structured Wiki pages, no conclusive findings on TikiWiki as a digital repository could be ma de from the study. Students also commented they had not realised at the time the purpose of the keywords and descriptions. At the beginning of the project students were unable to see the benefits that would arise from the time cost of adding suitable metadata. At the end of the project students wished they had invested more time in metadata creation as they could better analyse the cost/benefit ratio. STAFF VIEWS A brief focus group with the teaching staff was carried out at the end of the project. During this focus group staff made the following comments. Structure of student resources. Staff felt that many high quality resources had been generated as a result of the project. Some teachers maintained that a selection from these resources could be harvested to benefit future cohorts of students. Teaching staff noted, however, that not all teams had structured their Wiki pages as well as might have been expected. Some maintained that students had not sufficiently considered how others might use their resources; ‘they did not seem to be thinking about it – ordering it, editing it or guiding others through it’. Others noted students’ reluctance to review and edit information; ‘irrelevant information often lay within the Wiki pages’. and ‘some teams had failed to prioritise the information they have found’ Searching: Wiki pages and keywords. At the end of the project teaching staff acknowledged that insufficient preparation had been given to students in the creation of keyword and descriptive metadata. Also, in retrospect it was realised that the structured Wiki pages (which could be browsed) had undermined the value of metadata as a tool for
4
Downloaded From: http://proceedings.asmedigitalcollection.asme.org/ on 10/15/2014 Terms of Use: http://asme.org/terms
Copyright © 2004 by ASME
searching. One member of staff commented that ‘ideally students would have used browsing mode [of Wiki pages] for inspiration and keyword searching for targeting’. However, staff gave two reasons for the lack of success with the keyword searching. First, there was insufficient emphasis by teaching staff that keywords were important. Second, because few resources with keywords were created it was not possible to carry out productive searches. Interacting with resources and design learning. Observation of team concepts, and the presentations by students, confirmed to teaching staff that all teams had used resources sourced by other teams to inform their concepts. Analysis of TikiWiki concept pages revealed that 50% of the teams referenced resources. Staff reported that the best concepts were those generated by teams that had interacted with a wider range of resources. This was also evidenced in the assessment reviews; teams that appeared most knowledgeable were those that reported having browsed resources in the early stages of concept generation and/or had having regularly revisited these materials to further develop their concepts. Staff reported that ‘the shared resources had helped improve the concept designs compared to previous years’. However, all staff agreed that, although many high quality resources were generated as a result of the project, students could have made even more use of these to inform their concepts. DISCUSSION The project has shown that students are able to source and structure information that can be used by others. However, both staff and students identified that there was room for improvement. Staff suggested that students could have been more critical in their selection of resources and could have given more thought to how these resources were prioritized and linked together in a hierarchy. This indicates that more training is required for students in how to structure information. Better concepts (i.e. those considered by the teaching team to best match the specification given at the beginning of the project) were generated by teams that interacted more with the stored resources (gathering, editing, analysing and applying) and, those that reflected on the resources regularly during the design process (browsing initially for ideas and returning to target more specific information). Further research is required to determine whether teams who are likely to produce “better” designs are naturally inclined to make use of a wide range of information or whether good use of information results in better performing teams. It is therefore difficult to draw any conclusions as to whether use of structured information and resources support design learning. The project showed that metadata searching was not well used. Three reasons might account for this. First, not many teams supplied metadata (keywords or descriptions) hence some were not fully alert to TikiWiki’s keyword searching facilities. Secondly, the keywords supplied by one group of students uploading information to TikiWiki might not have matched with those of other groups subsequently searching for information. Keywords are only useful when the resources are referenced to an agreed and controlled vocabulary. In this study, staff had provided students with some terms on each of the topics to be investigated and students had used those terms (vocabulary) or related terms as metadata. Howe ver,
insufficient emphasis was placed upon the students’ use of this supplied vocabulary even though the project team had identified some of these issues at the project planning stage. In future projects students need to be supplied with, or work together to develop, an agreed set of keywords. Moreover, if these resources were to be reused by future cohorts of students they might also have to work with the same controlled vocabulary. Thirdly, students may just have preferred browsing the Wiki pages rather than searching using keywords. This might seem strange given that Google is their preferred way of searching the internet. However, Google searches the text of resources and this is different from TikiWiki, which was set up to search only the keywords and descriptions. In future, more emphasis could be placed on students providing rich descriptions of resources as metadata or TikiWiki could be reconfigured to search text within files. LESSONS LEARNED The “experiment” described in this paper was conducted without a control group, however there are a number of general points that emerged from the activity that are important in the context of designing new experiments or when including the teaching of information archiving and retrieval skills in an engineering course: ?? Encouraging students to organise design information and resources using inter-linked Wiki-pages helped support the access and sharing of these resources across design teams. ?? More work is needed to help prepare students to use metadata (e.g. keywords and descriptions) for cataloguing and searching resources. Academic staff need to work with librarians to formulate an agreed (and controlled) keyword vocabulary and students need more training in how to select keywords and formulate resource descriptions. ?? Students require more training in the structuring and editing of resources being archived for future use by others. ?? The resources sourced through nationally available digital repositories such as SMETE, NEEDS and EEVL were not found to be useful by engineering students conducting the can crusher project. FUTURE WORK The project is being continued in the second semester of 2003/04. Students will use TikiWiki to support the redesign of the drive mechanism of a domestic bread maker. Additional lectures will be given by a librarian to reinforce the use of metadata and the use of hierarchies to structure stored information. In parallel the team at Stanford University are experimenting with the use of video to record design tasks and indexing tools to allow future design teams to search for clips relevant to the task in hand. During the academic year 2004/05 new experiments will be designed and implemented to address the research question “does the use of structured information and resources support design learning and project working” During the academic year 2004/05 the work of both the Strathclyde and Stanford teams will be combined and the developed systems used to support global design projects.
5
Downloaded From: http://proceedings.asmedigitalcollection.asme.org/ on 10/15/2014 Terms of Use: http://asme.org/terms
Copyright © 2004 by ASME
5. ACKNOWLEDGMENTS The author’s acknowledge the funding provided by JISC under the Digital Libraries in the Classroom initiative. The authors also thank the students for their participation in the evaluation. REFERENCES 1. http://www.jisc.ac.uk/index.cfm?name=programme_dlitc 2. http://dmem1.ds.strath.ac.uk/didet/introduction.htm 3. Savin-Baden, M., 2000, Problem-based Learning in Higher Education: Untold Stories. SRHE/Open University Press 4. Currier S, Brown S and Ekmekioglu F C, 2001, ‘INSPIRAL: Investigating Portals for Information Resources and Learning’ Final Report to JISC, University of Strathclyde, Glasgow, UK.
Sclater N, Grierson H, Ion W J, MacGregor S P, 2001, ‘Online collaborative design projects: overcoming barriers to communication’. International Journal of Engineering Education, vol. 17, no.2, pp189-196. 6. Eris O, Leifer L, 2002, ‘Facilitating Product Development Knowledge Acquisition: Interaction Between The Expert and The Team’. International Journal of Engineering Education special issue on the Social Dimension of Engineering Design, 2002. 7. http://TikiWiki.org/TikiWiki 8. http://www.webct.com/ 9. http://www.eevl.ac.uk 10. http://www.smete.org/smete/ 11. http://www.needs.org/needs/ 12. http://patft.uspto.gov/netahtml/search-bool.html
6
Downloaded From: http://proceedings.asmedigitalcollection.asme.org/ on 10/15/2014 Terms of Use: http://asme.org/terms
Copyright © 2004 by ASME
