Keywords User studies, Information retrieval, Digital libraries, Electronic media. Abstract Describes key aspects of the methodology and outcomes of the JISC ...
The Emerald Research Register for this journal is available at www.emeraldinsight.com/researchregister
The current issue and full text archive of this journal is available at www.emeraldinsight.com/0022-0418.htm
JD 60,3
The JISC User Behaviour Monitoring and Evaluation Framework
302
Linda Banwell, Kathryn Ray and Graham Coulson
Received 1 November 2003 Revised 20 January 2004 Accepted 25 January 2004
University of Northumbria, Newcastle upon Tyne, UK
Christine Urquhart, Ray Lonsdale, Chris Armstrong, Rhian Thomas, Siaˆn Spink, Alison Yeoman and Roger Fenton University of Wales, Aberystwyth, UK and
Jennifer Rowley University of Wales, Bangor, UK Keywords User studies, Information retrieval, Digital libraries, Electronic media Abstract Describes key aspects of the methodology and outcomes of the JISC User Behaviour Monitoring and Evaluation Framework in its first three annual cycles (1999-2002). The Framework was initiated to assure the JISC that their investment in digital content and network infrastructure facilitates use and learning, and to identify barriers and facilitators to the use of electronic information services (EIS). Key Framework outcomes are: a multi-dimensional across sector methodology for the continued monitoring of user behaviour in respect of EIS and the factors that impact on that behaviour; a profile of user behaviour in respect of EIS over the three annual cycles of the Framework; the EIS Diagnostic Toolkit that can be used to benchmark development in the provision and use of EIS in specific disciplines or at specific institutions; a methodology for monitoring, and a profile of the EIS resources available to higher and further education users; and a summary of some of the key issues in their provision. The challenge for the future is the embedding of EIS in curricula and learning experiences.
Journal of Documentation Vol. 60 No. 3, 2004 pp. 302-320 q Emerald Group Publishing Limited 0022-0418 DOI 10.1108/00220410410534202
1. Introduction 1.1 Context In a further and higher education environment that is characterised by several drivers for change, it is important that information professionals and policy makers are able to make well-informed decisions concerning the development, provision and funding of electronic information services (EIS). To this end, the Joint Information Systems Committee (JISC), established a User Behaviour Monitoring and Evaluation Framework to investigate and profile the use of electronic information services within higher and further education in the UK. The Framework is designed to identify barriers and facilitators to the use of EIS in learning, and other knowledge work in higher and further education. Over the duration of this project there has been considerable change in the use of electronic information services (e-mail, Web-based sources) as well as the growth in use of Virtual or Managed Learning Environments. Brophy (2001), in a review paper, suggests that networked learning (which includes distance learning delivery via the Web, and use of virtual learning environment software) might be expected to produce a different pattern of use of resources by students. He suggests that the most effective
approach to deep learning will require a different pedagogical approach. Academic staff may need to adopt a different “problem-solving” approach to delivery of education. The observations from the Framework projects over the three cycles suggest that changes will also be required at strategic level within higher education to ensure that the proposed benefits of e-learning are to be achieved (and are in line with conclusions from the PriceWaterhouseCoopers report on the e-university (HEFCE, 2000)). During the period of the Framework, there has also been considerable development in policy and awareness regarding the use of information technology (IT) to support learning in further education (FE). Two of the most significant players in this firmament have been the British Educational Communications and Technology’s National Grid for Learning, and the National Learning Network (NLN) which is being developed by the Learning and Skills Council’s Further Education Information and Learning Technology Committee. Becta also leads the ILT Champions program in which selected members from each FE college are designated to lead ILT initiatives, and host the complementary Ferl Web site. JISC’s Regional Support Centres (RSCs) are also a useful source of support for e-learning in further education.
1.2 Structure of the Framework As initially envisaged the Framework had four strands: (1) A. A general survey of end-users of all electronic information services. (2) B. A “real time” survey of the use of JISC services. (3) C. A general survey of EIS provision. (4) D. A qualitative longitudinal monitoring of EIS use. Strands A and C were executed by the JUSTEIS Project Team, based at the University of Aberystwyth. Strand D was executed by the JUBILEE Project Team at the University of Northumbria. Strand B proved difficult to execute for various technological, organisational and political reasons, and very little work was conducted under Strand B. This article therefore focuses on Strands A, C and D. The Framework has operated in three annual cycles, during each of which the Framework methodology has evolved, and further data have been collected: (1) Cycle 1 – 1999/2000. The research design and data collection methodologies for the Framework were established, with particular reference to questionnaire and interview design, and access methods. Preliminary findings were reported. (2) Cycle 2 – 2000/2001. The methodology was further validated through use with different groups. In addition, the methodology was refined on the basis of experience with Cycle 1. Further population of data sets was undertaken. A pilot study was conducted in further education (3) Cycle 3 – 2001/2002. Deliverables were crystallised, with yet further population of datasets for higher education. The EIS Diagnostic Toolkit reached the development stage in which it could be piloted for application in institutions. A full-scale project was conducted in further education that included both survey and action research components.
JISC User Behavior Monitoring 303
JD 60,3
304
1.3 Purpose of this article This article summaries key aspects of the methodology and outcomes of the Framework in its first three annual cycles, 1999-2002. The methodology is described and evaluated, as a basis both for informing the interpretation of the findings and as a platform to enhancing understanding of the study of user information behaviour. Outcomes include the profiles of EIS use and the EIS Diagnostic Toolkit. Key aspects of the profile of information use for students in higher education are identified. Comparisons with student use in FE are made. Outcomes from the study of EIS availability are summarised. The nature and application of the EIS Diagnostic Toolkit is outlined.
2. Literature review The literature review conducted at the beginning of Cycle 1 confirmed that there is a substantial gap in the evidence surrounding student user behaviour concerning electronic information services. Areas that have been investigated have often focused on one or two aspects of user behaviour in small pilot projects or considered changes at one institution only (Lubans, n.d.a, b, c, d). Leaders of other projects discuss the need for a user-centered approach, and consideration of the organisational changes involved in the development of the hybrid (and digital) library in higher education (Pinfield and McKenna, 1998). A review (Chowdhury and Chowdhury, 1999) of the progress in digital library research described how closer analysis and consideration of user needs and how users actually did their work, led to improved interface design of a component of Cypress, a distance learning project at the University of California at Berkeley. Methodological approaches vary, depending on the philosophical stance taken and the particular emphasis of the project. From the information retrieval perspective, Spink (1999) considers information retrieval as a process of successive information seeking episodes, in an attempt to integrate interactive information retrieval research within an information-seeking framework. More emphasis is placed on context in the “rounding” approach adopted by Solomon (1999) for a longitudinal study of the information behaviour of 20 college students as they progressed through their degree programme. Other studies place more emphasis on the perceptions, and attitudes, the motivational factors which affect, for example, use of the Web by a small group of undergraduates in one institution (Dalgleish and Hall, 2000) or the impact of personality on approaches to learning among 500 dissertation students (Heinstrom, 2000), Motivational factors are often considered in studies which examine aspects of information literacy, e.g. the perceptions of a small group of physical science graduates (Brown, 1999), or the effect of new methods of teaching and learning on students’ information behaviour (Eskola, 1999). Some of these studies may be conducted at more than one institution to help validate the methodology (and the findings obtained) but it is rare that more than three institutions will be involved. It is only recently that a larger study commissioned by OCLC (2002) has reported on the Web-based information habits of US college students across many institutions. This was a one-off online survey (December 2001), with 1,050 US college respondents in the sample. The emphasis was on reasons for using Web resources (search engines, Web portals, Library sites and course-specific Web sites). Within the context of previous work, the Framework projects make a considerable contribution to:
.
.
the profiling of user EIS behaviour and the use of EIS in learning in HE across UK higher and further education; and the methodological approaches for understanding and changing user behaviour and learning in networked environments.
Although the study was funded by the JISC, which funds several specialised database and gateway services, the Framework projects were not confined to examination of those JISC-funded services as that might have prevented a fuller understanding of the context in which those services are used, or not used, and the factors which might predict heavier network use in the future. As a model (Savolainen, 1999) of the factors affecting non-work use of electronic networks suggests, experience of use, awareness, lifestyle choices, and routines will all influence actual use of networks, and that model will hold true for academic and leisure use of electronic networks by students in higher education.
JISC User Behavior Monitoring 305
3. Methodology 3.1 Research design issues The methodology comprised the three strands: A, C and D. Strand C has a distinct focus on EIS availability, but both of the other two strands explored aspects of user behaviour. While Strand A focussed on the creation of a profile of EIS use, Strand D sought to build an understanding of user behaviour and the factors that influence the behaviour, as a platform for the development of the EIS Diagnostic Toolkit. Table I summarises the extent of data collection over the three cycles, both in terms of the number of sites involved and the number of respondents. The primary purpose of this summary is to emphasise the scale of the project in terms of the numbers of respondents and sites involved. Further details of the methodology for each strand follow. 3.2 Strand A The objective of strand A was to: Undertake a periodic survey of EIS uptake and use, investigating the quantity and quality of take up with a view to bridging the gap between the perceptions and reality of user behaviour.
Strand A used a multi-stage cluster sampling process. This was designed to ensure that the sample included small, medium and large institutions, as well as different types of institution. Departments were then selected within institutions, distributed
Cycle Cycle Cycle Total Cycle
1 HE 2 HE 3 HE HE 3 FE
Number of sites (Strand A)
Number of respondents (Strand A)
Number of sites (Strand D)
Number of respondents (Strand D)
Number of sites (Strand C)
26 35 29 90 17
653 711 611 1,974 270
6 £ 3 6 £ 3 6 £ 3 54 30
763 506 431 1,700 528
44 48 29 121 22
Table I. Data collection
JD 60,3
306
among five disciplinary clusters. The original intention was to select students within departments randomly. However access and lack of willingness to participate led to very low initial response rates. The methodology was modified during Cycle 1, and the modified approach was used in Cycles 2 and 3. The modified approached retained the principles of the multi-stage cluster sampling but used substitution to cover departments that were unable to participate, and convenience samples at the student, and staff levels. FE institutions for Cycle 3 excluded those in Scotland. The number of respondents and sites is summarized in Table I. Data collection methods comprised: . Critical incident interview/questionnaire, with use of a critical success factors technique and vignette (to assess use of EIS, attitudes towards EIS, awareness of EIS and searching strategies among students). . Critical incident interview/questionnaire, with use of a critical success factors technique and vignette (for the interview), plus some additional questions, for academic staff. . Interviews with other stakeholders to assess awareness and attitudes towards use of EIS in the curriculum, and the status of IT/information skills support. . Small-scale action research projects in FEIs. The critical incident interview questionnaires shared common themes. The questionnaire asked respondents to think back to an occasion during the last week when they needed to find information. A total of 14 questions then probed: the event in terms of the purpose of the information search; how the search was conducted (including assistance sought, urgency and time taken); the resources used for the search; the guidance that led to the electronic resources used for the search, and search outcomes in terms of satisfaction, problems and success; and information skills training. The interview checklist included similar questions, but was also guided by interviewer checklists on the categories of electronic resources and giving prompts in respect of each question. Details of the number of respondents in each cycle are shown in Table 1.Interviews from both face-to-face and telephone were transcribed and entered into a qualitative software package (NUD*IST) for coding and further analysis. Quantitative data were extracted from the interview transcripts manually and entered into SPSS for collation with the data from the questionnaires. 3.3 Strand C The objectives of Strand C were to assess: . the provision of EIS through examination of LIS Web sites; and . senior librarians’ purchasing intentions and problems with respect to EIS. These twin objectives were achieved through two distinct methodological approaches. The first objective was achieved through an analysis of LIS Web sites. A stratified random sample of HEIs was selected in each cycle. The numbers of sites analysed is shown in Table I; this shows that most of the work in this Strand relates to HE sites, with just 22 further education colleges included in Cycle 3. For each of these institutions, all Web pages relating to the LIS site were copied into a Word document, a process of tagging, sorting, and deleting was performed to reduce the database to an authority list from which statistics could be generated. Analysis provided statistics relating to Web
site presence, subject trees and the complexity of the Web site, and, with the aid of a taxonomy of EIS types, links from the Web site to different types of EIS. In order to achieve the second objective within Strand C, interviews were conducted with senior librarians at 28 HEI’s, and four FEI’s over the three cycles. The questions cover general perceptions of benefits to users of EIS, current concerns affecting ability to plan for EIS, special arrangements for remote users, management and staffing issues, Web site development and maintenance, licensing and collaboration issues, budgeting and collection development, service evaluation, and emerging patterns. All the interviews were transcribed and loaded into NUD*IST 6 for qualitative data analysis. 3.4 Strand D The objectives of Strand D were to: . Contextualise and understand user interaction with EIS. . Provide a basis for the longitudinal tracking of users to determine success criteria’s for information seeking from the users points of view. . Develop success criteria, and to use these as a basis for the EIS Maturity Toolkit, for use by HE managers. Six fieldwork HEI’s were recruited for each cycle, taking into account: the type of institution, size, whether the institution is multi-site or single campus, and geographical spread across the UK, Wales and Scotland. One science, one social science, and one arts discipline were included in each cycle. More specifically, fieldwork was conducted in the following disciplines: . Cycle 1 – English, Health and Business. . Cycle 2 – History, Sociology, and Computing. . Cycle 3 – Design, Law and Geography/Environmental Studies, with some revisits to Health and Business, to amplify longitudinal comparison. Cycle 3 disciplines also provided the focus for fieldwork in the FE fieldwork sites, with the addition of Business, to maximise likely responses from sites. Research methods used throughout have comprised: . Questionnaire, both staff and student, in paper and e-mail versions, sent to academic and LIS/IT staff and students in target disciplines to collect background data on information behaviour, especially in relation to EIS. . Face-to-face interviews with key informants in disciplines and LIS service personnel at case study sites were supplemented by the use of e-mail and telephone interviews. . Face-to-face interviews with other academics in target disciplines snowballed by LIS staff and other academics. . Themed snapshots with students, via telephone or e-mail interviews. . Feedback requested on the institutional case study report. . Small-scale action research mini projects in FE. . Re-visits to Cycle 1 HE sites to monitor changes and test the Toolkit. . Development of Web-enabled toolkit prototype.
JISC User Behavior Monitoring 307
JD 60,3
308
Electronic observatories were established in Cycles 1 and 2, but despite several attempts to refine their design, these did not attract sufficient participants to be worthwhile. Both staff and students were reluctant to engage in discussion about the use of information resources. Perhaps they do not have a sufficient level of interest in these processes to want to reflect and discuss them. Respondents certainly engaged more positively in the interview and questionnaire processes, in which they were prompted to respond. Table I summarises the number of sites and respondents in each Cycle. Data have subsequently been prepared, using the NUD*IST style of qualitative categorisation, for entry into the database. Questionnaire responses have been coded and entered into SPSS files. Reports for each case study site were collated to inform the sites themselves, and to enable them to validate the data collected by the JUBILEE researchers. The case study report contents are confidential to the sites. 3.5 Critique of methodology Conducting a cross-sectoral survey using a mixture of qualitative and quantitative approaches, working with two project teams and three strands posed a number of dilemmas both in research design and data collection, and in analysis and drawing out pictures from the rich data sets that would be of value to different stakeholders. A critique of the methodology of the Framework has value both as a basis for understanding and interpreting outcomes, and also as a commentary on research into user behaviour. Rowley et al. (2002) provide a more extended critique of the Framework methodology. The first and most obvious challenge in research design arose in the context of integration. The structure of the Framework was non-negotiable, yet somehow from the various strands and work it was necessary to draw out a coherent and representative message. This involved managing integration at different levels: . Research design and distinctiveness of the objectives of different projects. . Standardisation of terminology and taxonomies of EIS and types of information use as appropriate. . Communication on fieldwork sites used, questionnaires and interview schedules. . Evolution of analytical approaches and conceptual frameworks with both common and distinctive project-based features. . Coherent dissemination. None of these would have been achieved without a sense of common purpose, mutual respect, and a willingness to learn. Integration and its benefits grew gradually through the three cycles of the Framework, as teams grew in confidence with their own contributions, and developed a deeper understanding of the contributions of the other team. Other issues emerged as work with the Framework developed: (1) Representatives, sampling and access. Strand A, in particular set out to conduct a survey based on a representative sample. Ultimately, gathering respondents across a number of institutions provide very difficult, and the samples of users for both strands became “lumpy”. Although a good spread across different user
categories, and institutions was maintained the eventual samples were far from random. While there is reason to be confident that the range of perspectives is well represented in the sample achieved, precise measurements of the significance of specific factors is more difficult to achieve. In addition, even the achievement of this collection of respondents required detailed attention to access arrangements. Respondents were often difficult to activate. (2) Measures. A key aspect of research design is the selection of appropriate indicators or measures of use. This involves consideration of a number of factors, among which are: . Which categories should be used for analysis. For example, what informs the typology of the purposes for which users use EIS? . Which user categories should be used to analyse and report behaviour? . Which terms can be used to report behaviour? A particular problem is that users are extremely naı¨ve about formal descriptions of types of EIS. (3) Interpreting data. Quantitative data analysis in Strand A, generated statements like: “46.9 per cent of undergraduate students cited assignment or lab report as a purpose for using EIS”. What does this mean? Why is this figure higher than for other use purposes? What percentage would be acceptable? Do we have any benchmarks that can be employed to indicate the link between activities or purposes and specific types of information sources? These types of questions emphasise the need to ask what? And why? And how? questions in parallel. “Skimming” to obtain a broad picture, must be accompanied with a considerable amount of “dipping” as an aid to interpretation of any questionnaire based data. The down side is that this leads to complex research designs and challenges for analysis and presentation of results. (4) Understanding user behaviour. Both Strands A and D collected qualitative data. One of the real challenges of the qualitative approach is to lift from a mass of anecdotal evidence some insights, tools and models that are useful to management, academics and library and information professionals. Two approaches have been used to the development of a series of nodes for the analysis and presentation of the qualitative data. Both sets of data were entered into NUD*IST and sets of nodes developed. For Strand D data this led in Cycle 1 to the seven key themes that form the basis of the EIS Diagnostic Toolkit (see section 4.4). Later cycles of Strand D to further develop the toolkit, and as a basis for gap analysis. For Strand A, the emphasis in detailed coding was understanding the context of information seeking and use, thus helping to resolve the reasons for certain trends occurring, and, if possible, predicting future opportunities or problems. 4. Outcomes This section draws out some of the key outcomes from the Framework. The outcomes are presented in four sections: (1) the profile of EIS use in higher education; (2) the profile of EIS use in further education;
JISC User Behavior Monitoring 309
JD 60,3
310
(3) EIS Availability; and (4) the EIS Diagnostic Toolkit. Much of the monitoring and evaluation framework activity can best be viewed as a strategic management exercise in determining how best to satisfy customers using the services JISC provides, and how to develop future services. Management theorists would approach this by considering whether the product, price, place, promotion (the traditional 4 Ps of marketing) plus (for services) people, and process were working correctly. For marketing of services the customer relationship management model with the 5 Cs of customers, company, competitors, collaborators and context can be applied. 4.1 Profiling user behaviour in HE Taking evidence from JUBILEE and JUSTEIS together, general themes that emerge which characterise and influence student approaches to the use of EIS include the following themes that have been arranged using the marketing theory framework outlined above: . Product. Use of JISC gateways, such as SOSIG and OMNI is very limited. Some students do use subject portals such as the NeLH portal for the health sector. In general, the use of portals by students is significantly influenced by advice offered to them by tutors, other students, and any training that they receive. Disciplinary differences are difficult to discern, due to different structuring in different institutions. However there does appear to be a pattern that Humanities and Arts students use databases least among the disciplines. Pure and Applied Sciences, and Clinical Medicine disciplinary clusters make the most use of databases. In Cycle 3 electronic journals were mentioned by 50 per cent of interviewees as a resource that they used, and by over 25 per cent of questionnaire respondents as one of the EIS used in a recent search (Strand A). There is some evidence of an upward trend in the use of electronic journals. . Price. Students are generally positive about EIS, viewing them as both accessible and convenient, and time saving. The demand for off-campus access to EIS appears to be rising, and students often prefer to email files or searches home, in preference to paying upfront for printing on institutional machines. . Place. Students benefit from direction embedded in the learning process, and academic staff, and sometimes lecturers’ and departmental Web sites are important in this. VLE/MLEs have significant potential to impact on EIS use. . Promotion. The main influences on student use of EIS are tutor expectations and explicit assessment requirements, followed closely by expectations from their peer groups. Habit is also an important influence in choice of information service. . People. There are a number of different models of information skills provision, involving variously LIS and academic staff, and all have a contribution to make to the development of EIS use skills. Academic staff need support and opportunity to develop their skills with EIS specifically and networked learning environments, in general. . Process. Undergraduates use EIS mostly for academic purposes connected to assessment, and also for leisure and lifestyle reasons.
Search engine use predominates over all other EIS; search engines are an important area for practising search skills. Over the three cycles there has been a shift in the preferred search engine from Yahoo to Google. Some research postgraduates make more use of EIS, and JISC-negotiated services in particular than undergraduates. For example, Web of Science databases and CHEST deals such as NISS-BIOMED are valued by the clinical and biological sciences community. Drilling down further, a few selected comments from students and academic and LIS staff illustrate the richness of the qualitative data sets that have been collected and analysed under both Strands A and D, and offer insights into attitudes, use and experience. Students appreciated the range of information sources to which they had access: It has helped me improve my studies because I have been able to get lots of information to help with assignments (Art and Design student). Information is accessible without leaving the comfort of your own home (Business student). Given me access to information and graphics that would not otherwise be so easily found, viewed and down loaded (Art and Design student). Access to more and better standard of information (Geography student).
Some students were apprehensive about computer use: I’m not sure how to use some packages so I’m frightened of crashing the computer (Art and Design student).
For some students, however, the main complaint related to the time that it took to locate information: Sometimes too much information making it difficult for me to pinpoint exactly what I need (Business student). Some sites are difficult to define and find (Business student). It takes time to find information (Business student).
The time that it takes for students to find information may be related to the search strategies that they adopt. In general their searching is dominated by the use of the Internet and search engines: OK, last night I’m looking for information about Edinburgh Create Centre.. So I didn’t know the address, but I used Freeserve and Google. I just put in the name and it came up with the address. ABI Inform was recommended by one of the other people on my course, and Yahoo, I just always go to Yahoo, and search for anything.. I’m not that good with the universities’ databases. [. . .] probably go to the pharmacology textbook, then maybe try the Internet. It just gave a list of different things and I tried them all. I could have come into the library but I was too lazy so I just did it from home.
JISC User Behavior Monitoring 311
JD 60,3
When electronic journals are mentioned students tend to be a little unsure and vague about the nature of the source that they are using: It was an Internet-based journal site [Interviewer: Can you tell me what that was?] . . . Medline (Medical student).
312
I’m looking for general articles or something like that I tend to go through the university Web site . . . and locate the database or e-journals through there. They tend to have full journal articles.
Academic staff are aware that students seek the quick solution: Instant gratification. One stop shop.. Which is why Google is so popular, it gives you the illusion that you’re quickly into the relevant sites (Library staff).
Furthermore, they are beginning to recognise and develop their role in supporting the student in learning how to filter and evaluate information: You’ve got some students who know how to use it really well, but other students who have got fairly basic knowledge of it (Law academic). It’s more about giving the skills to acquire and seek information. We’re expecting them more and more to find their own information. You’ve got to be able to filter that information and be critical about it . . .
4.2 Profiling user behavior in FE FE students are a less homogeneous group than students in HE. They are more frequently part-time, and study a wide range of courses that are diverse in terms of both subject and level. Some courses are specifically vocational in nature, while others will attract many students who are pursuing leisure interests. This profile means that students‘ approaches to learning vary considerably across FE, and it is to be expected that their use of digital information resources will be equally diverse. Nevertheless, some general patterns do emerge (using a similar marketing framework to that used for HE): . Product. The information sources of preference are the Internet, book and institutional Web sites (particularly the BBC News site). Students also use Encarta and other resources on CD-ROM. The Internet is seen as accessible, and relevant, and is used for both text and graphics. The majority of FE students are using the Internet as a matter of routine to assist with their coursework. . Price. FE students appear to accept search output with noise as the norm, and expect to engage in browsing to locate relevant items. . Place. Targeting of learning resources for subject, level and course is important. In relation to training sessions on Infotrac, students suggest that they perceived this source to be more complicated to access, and search, and in general to use than “the Internet”. Authentication processes were cited as one barrier. Although students believed that they had learnt from the training sessions, they remained reluctant to add Infotrac to the repertoire of information sources that they used regularly. Institutional or even Web-based e-mail is not widely used by students. The most effective means of communication with students is text messaging. This may have various consequences for e-learning environments.
.
.
.
Promotion. Students acquire skills in the use of digital information resources through their own experiences, tutors and friends. Formal training is not a very significant influencer, although IT key skill sessions are useful at a basic level, and can be very important in establishing initial confidence. Not all students are IT literate, with older students sometimes being particularly lacking in experience of IT use. Support for the development of information skills is often based around classroom teaching or supervised searching sessions, which afford a high level of interaction and peer-to-peer support. People. Staff are concerned that students exhibit poor discrimination and evaluation of Internet resources. Students need help with relevance judgements, and with the application and interpretation of Internet resources, so that they are used to support their learning, rather than as a “cut and paste” add on. Support in EIS use from learning resource, library and teaching staff is limited (see following discussion). Process. Student use of digital information resources is integrated across leisure, work and study. Information skills and approaches perfected in one arena will be translated to other arenas of their lives.
Students use basic search strategies, often relying entirely on simple keyword searching using search engines, recommended text books and Web sites with known URLs. FE, particularly as compared with HE, has a long tradition of relatively limited investment in ICT and learning resources. This has a number of consequences that need a response if governmental agendas concerning e-learning are to be progressed. The “People” element in marketing is critical for EIS uptake in FE: . Professional staffing levels in both ICT and learning resources (LR) are very limited; in the LR context, for example, this may mean that a few (and sometimes only one or two) professional staff have responsibility for all management, service delivery, collection development and liaison with teaching staff. This leaves little time for innovation and keeping up-to-date with ICT developments. Consequently, professional LIS staff are often not well equipped to provide leadership in relation to the development of staff or student information skills, and the use of digital information resources. On the other hand, the more entrepreneurial can be an ILT champion in their college, as they have had to be multi-skilled. . Academic staff typically have most of their time committed to teaching, leaving little time for staff development relating to the use of ICT in teaching and learning, or other innovations. . Academic staff are typically accommodated in multi-occupancy offices where access to IT facilities is shared and on a first come, first served basis. Those staff with limited IT skills may lack the privacy and opportunity to learn, practice and develop their skills. . Recent FEFC funding initiatives have had considerable sector wide impact on the availability of ICT facilities for students, although the tension between class use and open access use that characterises all deployment of ICT resource in
JISC User Behavior Monitoring 313
JD 60,3
.
314
.
education continues to demand careful management. Part-time students still find it difficult to schedule access to on campus ICT facilities; some are able to resolve this problem through home access, but this means that remote access to in-college databases and digital learning resources will become increasingly important. Other initiatives more closely focussed on the link between learning and ICT are achieving slower penetration. Colleges have been proactive in implementing access to JISC resources, such as Infotrac, and some have experimented with course training sessions in the use of such resources. Similarly, colleges are experimenting with the opportunities offered by virtual learning environments (VLEs) and managed learning environments (MLEs), but there is work to do on the full integration of these environments into learning experience across further education. Despite attractive licensing arrangements, tight LR budgets leave librarians with selection dilemmas in respect of digital resources. Resources with a wider subject scope, and better suited to the 16-19 market are the best option, but these are in short supply. There are also issues with licensing arrangements, and long-term uncertainties in respect of access to digital resources.
4.3 EIS availability The survey of Web sites was conducted for both HE, and in the third cycle, FE. As Figure 1 shows almost all sites provided a wide range of types of links, and over the three cycles there is a modest upward drift in links to EIS (although care should be exercised in the interpretation of the data in Figure 1, due to small variations in the methodology between cycles). The following summarise some of the key changes over the three cycles of the Framework: . Over the last three years, BIDS predominant position within the higher education community has weakened. This is probably largely attributable to the move of Web of Science to MIMAS. . Online hosts such as DIALOG and EINS have been all but supplanted by JISCand CHEST-mediated deals for access to made databases such as ERIC, MEDLINE, JUSTIS (Context), ABC-Clio, Ovid, Chadwyck-Healey, CSA and LEXIS-NEXIS. . Subject gateways feature on LIS Web sites. Increasingly non-JISC gateways such as the World Wide Web Virtual Library are offered as frequently as JISC gateways. Links to the Resource Discovery Network (RDN) have increased significantly over the three years and in Cycle 3, for the first time, links are being made to the Distributed National Electronic Resource (DNER). Also visible for the first time is the National Electronic Library for Health (NeLH) with over 50 per cent of the sites examined referencing it. . The union catalogue, COPAC, and the British Library OPAC consistently have proved the most popular remote OPACs with the Library of Congress maintaining a steady position
JISC User Behavior Monitoring 315
Figure 1. Percentages of sites linking to resource types
.
.
An analysis of the percentage of Web sites linking to the various aggregations of e-journals, demonstrates clearly the continuing dominance of IDEAL and Ingenta. Cycle 3 figures show that 71.4 per cent of the sites examined subscribe to more than one aggregator. The following feature strongly in search engine recommendations: Yahoo, Google, Northern Light and AltaVista.
Purchasing intentions and planning over the three cycles have been beset by huge uncertainties for those interviewed. Problems stem from the basic budget infrastructure and year-on-year funding changes which impede long-term planning. The rapidly increasing volume of EIS available is also impacting on staff training and competencies, particularly in view of providing service support to users. It is interesting to note that views and problems have not changed greatly over the three years. There are recurrent problems with purchasing EIS and all institutions are dealing with these in their own way; there is no standard practice. The problems of licensing models and pricing structures head the list of main problems, along with the inflexibility of these models so that institutions cannot tailor the deals and journals offered to suit their user demand. Three years down the line, and much discussion, nothing appears to have changed very much in the market place and the introduction of national initiatives has also proved problematic, as they also come with inflexibilities, at great cost. 4.4 The EIS Diagnostic Toolkit In general terms, the information providers goal is to create a situation where users, academics and students use information seamlessly and with confidence to support their learning and leisure. Evidence from cycle 1 fieldwork in JUBILEE suggests that
JD 60,3
316
the prerequisites for achieving this goal are for users to perceive they are in a situation where: . Access to information, including EIS, is easy for all users. . The resource base (technical and human resources) itself is good to excellent. . Users have the technical and evaluative skills to use information, including from EIS. . EIS are embedded in course design and delivery, and in the research process. . EIS are embedded in student learning. . Quality assurance processes are in place for Internet-based information. . Seamlessness is achieved. These user-based success criteria are the basis for toolkit themes, and provide the framework for driving change. Barriers will be encountered by users, which will create dilemmas for individuals and institutional managers. Enablers will provide solutions and will permit movement along the development path. Institutional and discipline contexts and constraints will determine the exact nature of the development path appropriate in each individual and institutional context. The themes that provide the structure for the Toolkit are: . The wider environment. . Institutional context. . Strategic management within the institution. . Library service issues. . User needs. . Communication. . Quality. . Resources. Cycle 1 was the baseline cycle when the set of toolkit themes was identified. In Cycle 1, three example matrices were developed: (1) Evidence in relation to one discipline (Business Studies) at six sites, showing the variations between sites in that discipline. (2) Evidence of the three Cycle 1 target disciplines at one site, showing the variations between disciplines at the same site. (3) Drilling down into one theme, access, showing Cycle 1 evidence of poor, intermediate, and good practice from all six sites. The toolkit was developed further in Cycle 2 through inclusion of additional ways of synthesising and presenting data by site and by discipline. Cycle 3 has seen further development of the content of the toolkit, and its development as a web enabled prototype, accessible at http://62.31.146.55/Jubilee. The data, on which the prototype is based, is the HE evidence from the Cycle 2 sites, in three disciplines. This prototype will be further developed in the fourth cycle of the Framework in 2002-3, through piloting in both higher and FE.
The Toolkit also proposes and uses a number of generic development stages. These were characterised as baseline, change, congruence, embedding and full integration. Generic characteristics have been described for each development stage, derived from fieldwork evidence. These are presented in Table II. JUBILEE has tested these stages as benchmarks for the use of EIS in HE and FE.
JISC User Behavior Monitoring 317
5.Conclusions and recommendations 5.1 Framework and methodology The Framework approach has successfully developed a methodology that has delivered useful insights into the use of EIS to support learning in HE and FE. The framework is far more ambitious and sophisticated than any work attempted elsewhere. Its particular strength is the broad base of institutions and communities Stage 1
2
3
4
5
Characteristics Baseline Starting point, status quo
No systematic practice, sporadic surveys, historical picture, not a priority Systems uncoordinated/fragmented Understanding of broader issues Introduction of new systems to adapt to/accommodate change Some early decisions made User needs, objectives, service targets Change Point at which there is recognition established On-going methods of satisfying needs etc. of a need to change devised, e.g. working parties, informal strategy document, service evaluation (surveys, staff involvement) Resource use monitored Relationships being strengthened Provide and monitor effectiveness of skills Congruence Stage where the vision is starting training Institution wide policies and standards to be implemented established Collect and optimise statistical and qualitative data User education and monitoring, restructured Embedding framework for needs assessment Appropriate partnerships are developing from congruence and Partnership between students, researchers and are accepted as part of the culture staff Widespread implementation of strategies and policies at all levels Ensure meeting of targets Individual behaviour recognised and satisfied Full integration Generic frameworks adopted and adaptable All the diverse elements are Student/user-centred strategies, involvement in assimilated signifying maturity the curriculum and the ability to fully exploit Appropriate multidimensional systems in place potential available Continuous performance measurement
Table II. EIS Diagnostic Toolkit: generic development stages
JD 60,3
318
with which it has variously engaged. The Framework has reached the reluctant users of EIS as well as the enthusiasts. The first three cycles have demonstrated that: . It is possible to create and manage a multi-dimensional framework. Strands of the framework have merged and become more integrated as work has proceeded. . There is considerable value to be achieved both in fieldwork operations and outcomes in engaging and coordinating more than one research team to undertake this work. . The joined-up approach makes a significant contribution to understanding of user behaviour, and institutional development paths. . The fieldwork has created rich data sets and the JUSTEIS coding of interview data allows the data to be “cut” in different ways. . It is not possible to conduct a purely quantitative study that offers any real insights, due to the changing context (which affects comparability between cycles). In interviews it is possible to reach a shared meaning with the respondent, whereas in questionnaires understanding of EIS terminology cannot be checked. Access to student respondents needs considerable expertise in field work execution and a flexible approach to data collection. . The EIS Diagnostic Toolkit offers a valuable tool for HE managers. The process associated with the development of such a toolkit is lengthy and needs to be on going and dynamic. . It is very much more difficult to analyse library web sites for the EIS that they make available than might be supposed, due different representations of EIS and different Web site structures.
5.2 Recommendations for the future of the framework Proposals have already been submitted and approved for the fourth cycle of the Framework. The fourth cycle will: . integrate FE and HE perspectives more fully; . develop enhanced regional perspectives of both FE and HE activity; . enhance the longitudinal monitoring of institutions embedding of EIS through the collection of further FE data through a second cycle in FE; . continue to develop a bank of action research projects, with strengthened collaborations between the JUSTEIS and JUBILEE project teams; . evaluate the EIS Maturity Toolkit for use in FE; . further data mining of existing data sets to enhance understanding of student searching behaviour; and . develop international networks and perspectives. In considering future long-term developments of the Framework, and any related studies of user behaviour in digital information and learning environments, it is important to take the following points into account: . Comparative work across FE and HE is particularly valuable.
.
.
.
.
.
A multidimensional methodological approach to data collection and data mining is essential in building the rich picture that is necessary to the type of understanding of user behaviour that is appropriate for informing future developments. The Framework needs to be dynamic, with the methodology changing to reflect both developments in user behaviour and also developments in the education sector, and government agendas relating to the sector. Innovation takes time to disseminate and resource injection does not immediately affect academic or student user behaviour. The speed of innovation, and hence the optimal frequency cycle for a monitoring framework is difficult to judge. Monitoring user behaviour and trends in user behaviour requires relatively large projects.
5.3 Changing user behaviour Considerable work towards the successful embedding of EIS in learning across the wider higher and further education community has been undertaken. JISC initiatives have been instrumental both in terms of infrastructure and resource availability. While there is always scope for improvement on both of these dimensions, and advances to maintain a leading edge are essential, the major issue is embedding EIS, and optimising the value that institutions, the education community and society, in general, derive from EIS. The challenge is to make student use of EIS habitual. This will be achieved through: . Recognising the importance of leisure and work use in study learning. . Embedding EIS in the curriculum, and specifically in assignments. . Supporting staff and students in developing their EIS skills, including information retrieval, or data mining skills, and information evaluation skills Both at sectoral level and at institutional level, the HE/FE communities are engaged in a major change project. EIS and IT-based communication have been available across the education community for many years. It is only with the advent of wider availability of electronic journals (and perhaps for the future, electronic books), and software, such as VLEs and MLEs that the opportunity for more seamless integration of information resource access into the learning process has emerged. These circumstances demand new perspectives on pedagogy, knowledge and the relationships between these two. In addition, understanding and innovations in pedagogy should be informed jointly by educational developers, academics, and LIS professionals. References Brophy, P. (2001), “Networked learning”, Journal of Documentation, Vol. 57 No. 1, pp. 130-56. Brown, C.M. (1999), “Information literacy of physical science graduate students in the information age”, College and Research Libraries, Vol. 60 No. 5, pp. 426-38. Chowdhury, G.G. and Chowdhury, S. (1999), “Digital library research: major issues and trends”, Journal of Documentation, Vol. 55 No. 4, pp. 409-48.
JISC User Behavior Monitoring 319
JD 60,3
320
Dalgleish, A. and Hall, R. (2000), “Uses and perceptions of the World Wide Web in an information-seeking environment”, Journal of Librarianship and Information Science, Vol. 32 No. 3, pp. 104-16. Eskola, E. (1999), “University students’ information seeking behaviour in a changing learning environment How are students’ information needs, seeking and use affected by new teaching methods?”, Information Research, Vol. 4 No. 3, available at: www.shef.ac.uk/,is/ publications/infres/isic/eeskola.html Heinstrom, J. (2000), “The impact of personality and approaches to learning on information behaviour”, Information Research, available at: www.shef.ac.uk/, is/publications/infres/ paper78.html Higher Education Funding Council for England (HEFCE) (2000), E-university Project: Business Model, Circular 00/43: summary report, Circular 00/44 full report from PricewaterhouseCoopers, HEFCE, Bristol. Lubans, J. Jr (n.d.a), “How first-year university students use and regard Internet resources”, available at: www.lubans.org. Lubans, J. Jr (n.d.b), “Key findings on Internet use among students”, available at: www.lubans.org Lubans, J. Jr (n.d.c), “Students and the Internet” available at: www.lubans.org Lubans, J. Jr (n.d.d), “Study 4: Internet use among 3rd year students at Duke University”, available at: www.lubans.org. OCLC (2002), “White paper on the information habits of college students (research by Harris Interactive)”, executive summary available at: http://www2.oclc.org/oclc/pdf/ printondemand/informationhabits.pdf (accessed 23 August 2002). Pinfield, S. and McKenna, B. (1998), “The BUILDER project”, Electronic Library, Vol. 16 No. 5, pp. 305-7. Rowley, J., Urquhart, C., Lonsdale, R. and Banwell, L. (2002), “User behaviour in relation to EIS within the UK HE academic community”, Journal of Educational Media, Vol. 27 No. 3, pp. 107-22. Savolainen, R. (1999), “Seeking and using information from the Internet: the context of non-work use”, in Wilson, T.D. and Allen, D.K. (Eds), Exploring the Contexts of Information Behaviour. Proceedings of the Second International Conference on Research in Information Needs, Seeking and Use in Different Contexts, 13/15 August 1998, Sheffield, UK, Taylor Graham, London, pp. 356-70. Solomon, P. (1999), “Information mosaics: patterns of action that structure”, in Wilson, T.D. and Allen, D.K. (Eds), Exploring the Contexts of Information Behaviour. Proceedings of the Second International Conference on Research in Information Needs, Seeking and Use in Different Contexts, 13/15 August 1998, Sheffield, UK, Taylor Graham, London, pp. 150-75. Spink, A. (1999), “Towards a theoretical framework for information retrieval in an information seeking context”, in Wilson, T.D. and Allen, D.K. (Eds), Exploring the Contexts of Information Behaviour. Proceedings of the Second International Conference on Research in Information Needs, Seeking and Use in Different Contexts, 13/15 August 1998, Sheffield, UK, Taylor Graham, London, pp. 21-34.
