Bioinformatics training: selecting an appropriate

B RIEFINGS IN BIOINF ORMATICS . VOL 11. NO 6. 552^562 Advance Access published on 2 July 2010

doi:10.1093/bib/bbq023

Bioinformatics training: selecting an appropriate learning content management systemçan example from the European Bioinformatics Institute Victoria Ann Wright, Brendan W. Vaughan, Thomas Laurent, Rodrigo Lopez, Cath Brooksbank and Maria Victoria Schneider Submitted: 4th May 2010; Received (in revised form) : 5th June 2010

Today’s molecular life scientists are well educated in the emerging experimental tools of their trade, but when it comes to training on the myriad of resources and tools for dealing with biological data, a less ideal situation emerges. Often bioinformatics users receive no formal training on how to make the most of the bioinformatics resources and tools available in the public domain. The European Bioinformatics Institute, which is part of the European Molecular Biology Laboratory (EMBL-EBI), holds the world’s most comprehensive collection of molecular data, and training the research community to exploit this information is embedded in the EBI’s mission. We have evaluated eLearning, in parallel with face-to-face courses, as a means of training users of our data resources and tools. We anticipate that eLearning will become an increasingly important vehicle for delivering training to our growing user base, so we have undertaken an extensive review of Learning Content Management Systems (LCMSs). Here, we describe the process that we used, which considered the requirements of trainees, trainers and systems administrators, as well as taking into account our organizational values and needs. This review describes the literature survey, user discussions and scripted platform testing that we performed to narrow down our choice of platform from 36 to a single platform. We hope that it will serve as guidance for others who are seeking to incorporate eLearning into their bioinformatics training programmes. Keywords: bioinformatics; online training; eLearning platform; databases; learning content management system; European Bioinformatics Institute

Corresponding author. Victoria Ann Wright, EMBL Outstation, European Bioinformatics Institute, Wellcome Trust Genome Campus, Hinxton, Cambridge, CB10 1SD, UK. Tel: þ01223 4494508; Fax: þ01223 492621; E-mail: [email protected] Victoria Ann Wright is the eLearning Content Developer in the Outreach and Training Team at the European Bioinformatics Institute (EMBL-EBI). Her work focuses on developing an eLearning portal to train users on the tools and resources EMBL-EBI provides. BrenVaughan is a Web Developer in the External Services team (ES) at EMBL-EBI. His work focuses on developing and maintaining web infrastructure for the institute and its associated collaborative projects. Thomas Laurent is a Web Developer in the ES team at EMBL-EBI. His work focuses on developing and maintaining EBI-hosted web portals for European Union consortiums. Rodrigo Lopez is the Head of the ES team at the EMBL-EBI. He previously completed a Cand. Scient. in Molecular Toxicology and Informatics in 1987 at the University of Oslo. Cath Brooksbank is the Head of Outreach and Training at EMBL-EBI where she coordinates EMBL-EBI’s public relations and user training programmes. MariaVictoria Schneider is the Training Programme Project Leader in the Outreach and Training Team at EMBL-EBI where she is responsible for developing and coordinating the in-house and external training activities of EMBL-EBI. ß The Author 2010. Published by Oxford University Press. For Permissions, please email: [email protected]

Downloaded from http://bib.oxfordjournals.org/ by guest on January 7, 2016

Abstract

Bioinformatics training: an example

INTRODUCTION

 Support and training to researchers who use EMBL-EBI’s resources via courses and workshops held at EMBL-EBI;  Road shows to train researchers on site at their own institutions;  Support for other bioinformatics trainers, including community building and resource sharing as discussed in Schneider et al. (this issue);  Online training for the potential and current users of EMBL-EBI resources, including other trainers. There is general consensus that, if used correctly, eLearning can be a powerful educational tool. Indeed,

Zhang and colleagues [4] describe eLearning as an ‘indispensable part of academic and professional education’ and they believe that eLearning is ‘a promising alternative to traditional classroom learning, which is especially beneficial to remote and lifelong learning and training’. Kruse [5] also explains the value of eLearning: ‘it is unarguable that eLearning is rapidly growing as a form of training delivery and most are finding that the clear benefits to eLearning will guarantee it a role in their overall learning strategy’. eLearning employs interactive systems and communication technologies to improve learning. It makes possible for a truly ambitious education system for a future learning society [2]. We anticipate that online training will become an increasingly important training vehicle, not only enabling the EMBL-EBI to train more users than we can physically accommodate in our face-to-face training courses, but also as an adjunct to face-to-face training and as a means of sharing training materials among a community of trainers. Having previously run a small pilot project to develop a prototype eLearning portal, the EMBL-EBI is currently in the process of redeveloping its eLearning resources. The first part of this process has been an extensive review of currently available Learning Content Management Systems (LCMSs)—see Box 1.

Box 1. What is a learning content management system? Content management systems (CMSs) are now widely used to manage large volumes of dynamic web-based material. A learning content management system is a specialized type of CMS fine tuned to different types of users who need to interact with online learning resources.When designing and providing a programme of eLearning courses it is common to use an LCMS as the ‘container’. According to Colace De Santo [17], an LCMS ‘includes all the functions enabling creation, description, importation or exportation of contents as well as their reuse and sharing.’ Cohen and Nycz [6] explains that an LCMS ‘makes it possible for technologists, content specialists, instructional specialists and students to work seamlessly to create and refine eLearning’. eLearning is very much a user motivated method for learning and the quality of the learning environment is very important. Bouchard [16] explains that it must be possible to clearly define learning objectives, decide on the content structure, personalize learning and change the pace of the learning activities for learning to be successful, thus the platform needs to support these.

Downloaded from http://bib.oxfordjournals.org/ by guest on January 7, 2016

The huge amount of data generated by new technologies such as next-generation sequencing requires that today’s life scientists not only keep up with their field of study, but also with the rapidly evolving tools and databases used to collect, store and analyse biological data. While the classical academic career path for molecular life scientists provides a thorough education in bench-based research, it tends not to prioritize formal training on the databases and tools that allow the efficient extraction and exploitation of the data. The rapid advancement of eLearning technologies to enhance and complement traditional classroom teaching has enjoyed wide uptake in many fields, including bioinformatics. Bioinformatics is a fast-paced discipline, requiring continuous training of its users in emerging areas and concepts [1]. In view of this, eLearning could provide a firm basis for the continuing professional development of molecular life scientists [2]. EMBL-EBI is one of the few places in the world that has the resources and expertise to maintain and provide public access to a comprehensive collection of large biological data sets, ranging from DNA and protein sequences through to models of complex pathways [3]. Hand-in-hand with our mission to make biological information freely available in ways that promote scientific progress is a need to train our users to exploit the data to their full potential. A core part of EMBL-EBI’s mission is therefore to provide user-orientated bioinformatics training to researchers at all levels, from PhD students to independent investigators. Our user base spans wet-lab experimentalists to computational scientists, basic biologists to clinical researchers and early-stage researchers to experienced investigators, including those who need to train others. The EMBL-EBI offers several opportunities aimed at fulfilling this:

553

554

Wright et al.

WHAT DO WE MEAN BY eLEARNING? There are several definitions for the term eLearning and the perception of what it means is still evolving. The delivery of knowledge, and thus learning, can occur in a variety of ways. These include: the traditional face-to-face method used by most universities and educational institutes; the delivery of courses using only electronic (typically internet-based) media or a combination of both (blended learning). There are lots of different implementations of eLearning and numerous types of media used. Cohen and Nycz [6] describe eLearning in the broader context of using technology to meet society’s needs for learning. Expanding on this they add: ‘eLearning is the current technological solution to the problem of finding the best match between the needs of a given set of learners to learn a given content, using a given set of learning tools’. Koohang and Harman [7] define eLearning as ‘the delivery of education (all activities relevant to instructing, teaching and learning) through various electronic media. The electronic medium could be the Internet, intranets, extranets, satellite TV, video/ audio tape, and/or CD ROM.’ At the EMBL-EBI, we define eLearning quite broadly, as the delivery of education through electronic media. The most important concept in this definition is that of ‘education’ as opposed to ‘information’. Learning is an active process that requires, in order to be successful, the use of a range of different techniques to keep the learner engaged (see below). This may involve

practicing and testing one’s knowledge, as well as absorbing new information through reading, listening or viewing. We aim to build a system that promotes active learning, not just an encyclopaedia of learning materials.

USEFUL DEFINITIONS AND CONCEPTS eLearning can be subdivided into trainer-led courses, which are typically delivered to a specific, preadvertised schedule, and trainee-led (self-directed) courses, which may also be self-paced. eLearning can be delivered on a one-way (asynchronous) or two-way (synchronous) basis. Asynchronous technologies deliver content one way at one point in time and can include: email, video, blogs and audio aids. Synchronous technologies deliver two or more technologies at the same time and can include teleconferencing, chat rooms and web conferencing [8]. It may be safe to say that synchronous technologies are typically used for trainer-led eLearning, and that asynchronous technologies may be typically used for trainee-led eLearning. Asynchronous interaction is also beneficial when students are geographically dispersed and it is difficult to assemble them at the same time [9].

Defining our users from all sides Before we began working on our selection criteria, we felt that it was important to define our main groups of users. Whilst trainees—the end users of our online courses—are clearly an important group of stakeholders, others need to be taken into account. Trainers—the people who will create and modify our online course content—are an equally important user group. Systems administrators—the people who will install, customize and update the LCMS and any associated modules or plugins—are also vital to the success of any eLearning portal. We therefore sought to structure our selection process around the needs of these three ‘stakeholder’ groups.

Defining priorities for selecting an LCMS There are many advantages and disadvantages of eLearning and these can be viewed from different perspectives, for example, from the learner’s perspective or from that of the organization seeking to implement the eLearning project. Kruse [5] suggests that the pros and cons of eLearning vary depending

Downloaded from http://bib.oxfordjournals.org/ by guest on January 7, 2016

In this review we describe the process used to select an appropriate LCMS for EMBL-EBI’s online training programme. We generated a list of requirements for our LCMS by reviewing the current literature, soliciting users’ views and reviewing organizational and technical requirements. The most suitable platform was chosen through completion of scripted platform testing and face-to-face discussion groups from the perspective of the trainers, trainees and systems administrators. This review seeks to aid those who find themselves in a similar situation. We believe that every organization’s requirements are unique and that there are several excellent LCMSs available. Our focus here is to describe, and make available to all, the procedure we used to match the attributes of the LCMS to the requirements of our users and trainers.

Bioinformatics training: an example

555

Table 1: Summary of the advantages and disadvantages of eLearning from the trainees’ perspective Advantages of eLearning from the trainees’ perspective

Disadvantages of eLearning from the trainees’ perspective

Available** Self-pacing** Portable** Accessible** Interactive** Long-lasting* Individual* Skills Modularity** Feedback Costs Private Boundaries Consistency* Subjects Reviews*

Knowledge Lack of skills Incorrect** Discipline* Support** Information* Technology** Styles Culture

Complete training whenever Learn at different speeds Complete training wherever Intuitive and easy to use Engages and motivates users Available as a refresher course Target individual learning styles Increases computer skills Complete the relevant parts Quick and interactive feedback Only internet/electricity required Overcomes issues of shyness Less physical/social boundaries Similar styled courses Learn outside specialism Learners can rate courses

Requires knowledge of computers No social/cultural interaction Content may be out-of-date/wrong Need motivation to work May be no/little support available Too much/little information on web Issues with software/web sites Constrained to chosen web style No access to computers (languages)

Table 2: Summary of the benefits and difficulties of eLearning from a systems administration perspective Benefits of eLearning from the technical perspective

Difficulties of eLearning from the technical perspective

Knowledge Standardization** Single site Repository** Data* Storage Tracking*

Portability Technology Upgrades* Security** Systems Redundancy Maintaining*

Acquisition/storage of data Standardization of website Single site back-ups/updates One place for all materials Monitor website usage Larger storage capacity Able to track course usage

Does not rival that of books Compatibility and bandwidth issues Regular upgrades may be needed Data may not be secure Systems can crash Content may be duplicated Requires maintenance and support

We define a system administrator as a user who will install, customize and update the LCMS and any associated modules or plugins.We prioritized which advantages were important to meet our requirements, and which disadvantages it was important to address, using a star-rating system, where two stars means ‘must have’ (for advantages) or ‘must address’ (for disadvantages) and one star means ‘good to have’ or ‘good to address’.

on programme goals, target audience and organizational infrastructure and culture. As a starting point to our selection process, we wanted to define which of these pros and cons were important to us, so that we could build a prioritized list of requirements for our LCMS. To do this, we conducted a literaturebased review. The literature review included searching Google Scholar [10] and SCOPUS [11] with key terms for hits in the article title, abstract and keywords: ‘eLearning’; ‘eLearning benefits’; ‘eLearning advantages’ and ‘eLearning disadvantages’. Using the term ‘e learning’ as opposed to ‘eLearning’ yielded more results but also includes many unwanted ones on learning alone. The highly cited top hits published within the last 10 years were then reviewed

for relevant information. Key articles used in this literature review include Kruse [5] and Zhang [4]. From the literature review we categorized the main advantages and disadvantages of eLearning, from the trainees’ perspective (Table 1), from a systems administration perspective (Table 2) and from the trainers’ perspective (Table 3). We then prioritized which advantages were important to meet our requirements, and which disadvantages it was important to address, using a star-rating system, where two stars means ‘must have’ (for advantages) or ‘must address’ (for disadvantages) and one star means ‘good to have’ or ‘good to address’. These tables do not seek to provide an exhaustive list of all of the advantages and disadvantages of eLearning but aim to provide an

Downloaded from http://bib.oxfordjournals.org/ by guest on January 7, 2016

We define a trainee as an end user of our online courses. We prioritized which advantages were important to meet our requirements, and which disadvantages it was important to address, using a star-rating system, where two stars means ‘must have’ (for advantages) or ‘must address’ (for disadvantages) and one star means ‘good to have’ or ‘good to address’.

556

Wright et al.

Table 3: Summary of the advantages and disadvantages of eLearning from the trainers’ perspective Advantages of eLearning from the trainers’ perspective

Disadvantages of eLearning from the trainers’ perspective

Low costs* Distribution* Audience* Evaluation* Quality Community Reusable* Promotion*

High costs* Content* Time* Training* Instruction* Support* Updates*

Long-term costs reduced Faster material distribution Maximizes participants Evaluate learners’ progress Reviewing content increases quality Knowledge can be shared Content is easily reused Promotion of other courses

Large development costs Difficult to aim it at the correct level Initial set up requires a lot of time Training on the software is needed Inferior to face-to-face teaching? Leaner support may be required Course content needs to be updated

We define a trainer as someone who will create and modify our online course content.We prioritized which advantages were important to meet our requirements, and which disadvantages it was important to address, using a star-rating system, where two stars means ‘must have’ (for advantages) or ‘must address’ (for disadvantages) and one star means ‘good to have’ or ‘good to address’.

Methods for selecting an LCMS Evaluating an eLearning platform is not a trivial task; there are many aspects to consider and multiple methods, models and frameworks to evaluate them. Most published literature evaluates platforms based on identified required features but this alone is not always adequate. To elicit the best method for evaluating LCMSs another literature review was conducted, this literature review also revealed potential LCMSs to evaluate. This literature review included searching Google Scholar [10] and SCOPUS [11] with the keywords: ‘eLearning platform’; ‘learning content management systems’; ‘evaluating LCMS’; ‘evaluating eLearning platform’; ‘selecting eLearning platform’ and ‘selecting LCMS’. Again, the highly cited top hits published within the past 10 years were viewed for relevant information. Key articles used in this literature review are discussed below. The success of an online course is closely linked to the ability of the chosen LCMS to fulfil the requirements of its stakeholders, so what makes a good LCMS? Box 2 lists some successful examples of eLearning programmes in bioinformatics: University of Manchester online Masters level courses [12], Bioinformatics Information Resource

Box 2. Successful examples of eLearning programmes in bioinformatics eLearning has become an accepted part of education in many fields and is, for example, widely used in medical education [1]. Perhaps due to the difficulty in providing online training in such a fast-paced environment, eLearning is only just emerging as an educational tool for bioinformatics. Examples of successful eLearning programmes for bioinformatics include the following:  The University of Manchester runs online Masters level courses [12] as well as biological data analysis for computer statistics modules and computational modelling courses specifically designed for biologists;  Bioinformatics Information Resource and eLearning Center (BIREC) [13] was created to provide free bioinformatics information resources and an online self-learning platform. Their goal is to provide maximum amount of information in one place and present the information focussed on the audience;  In the Education Department of the Bielefeld Bioinformatics Server (BiBiServ) [14], two online courses are offered, one introductory and one on the advanced level: (i) Sequence Analysis with Distributed Resources (SADR) and (ii) Algebraic Dynamic Programming in Bioinformatics (ADP);  BioManager [15] is a web-based bioinformatics application which integrates a variety of common bioinformatics tools, for teaching, BioManager has been utilized as a training environment.

and eLearning Center (BIREC) [13], the Education Department of the Bielefeld Bioinformatics Server (BiBiServ) [14] and BioManager [15]. Bouchard [16] suggests one of the most important factors in the success of any learning environment is the ability of the environment to support learner self-direction. Bouchard explains that it must be possible to define learning goals, decide on the learning sequence, personalize learning and change the pace of the learning activities for learning to be successful, thus the

Downloaded from http://bib.oxfordjournals.org/ by guest on January 7, 2016

overview of the important requirements to consider when approaching eLearning as a training method. It is important to understand that the advantages and disadvantages from each perspective assume that all of the requirements from the other perspectives have been met: this avoids redundancy in the tables. For example, the features from a trainee’s perspective assume that technical requirements, such as browser compatibility, have been met.

Bioinformatics training: an example

(‘must have’ and ‘nice to have’) instead of six, mainly to keep it simple but also because this was an approach to decide which the important features and requirements to test were, rather than the actual testing of the LCMS. It was useful to implement Graf’s suggestion of splitting categories into sub-categories in our testing, as this allowed us to incorporate more details in the investigation. We also put into effect the idea of using an example course when testing the LCMSs. Kurilovas and Dagiene [20] also use a rating system (but with only five weights) and propose using subcategories of features to evaluate an LCMS properly, as does Graf. They do, however, concentrate only the technical evaluation of the LCMSs. The tool is based on analysis of the level of support for the features and the level of modification needed to reach the desired level of support. They give each feature a rating between 0 and 4 (a rating of 0 implies that the feature has failed or it does not exist and a rating of 4 means that the feature has excellent support and meets the criteria out of the box with minimal effort). The technical criteria used by this tool provided a basis for our organization to generate a list of features to incorporate when evaluating the LCMSs (see below). This rating tool was specifically designed for the technical evaluation of LCMSs and our investigation extended beyond this to include the assessment of the entire platform, communication tools for example; thus, this tool was not directly relevant for us to use, but provided guidance for the technical evaluation of the LCMSs. Graf [18] and Kurilovas and Dagiene [20] both provide a good starting point when generating a list of ranked requirements and features with which to evaluate LCMSs. However, as mentioned previously, the list of features and requirements (and indeed the importance of the requirements and features) is specific to each organization, so after reviewing the sub-categories and categories suggested in the papers, we developed our own weighting system and our own list of features and requirements. There is a substantial amount of heterogeneous but valid information on several previous attempts to compare different LCMSs. We established that our first step was to define a set of features and requirements with which we would evaluate the LCMSs. The results from our second literature are summarized in Tables 4 and 5. Two categories of features have been taken

Downloaded from http://bib.oxfordjournals.org/ by guest on January 7, 2016

LCMS needs to support these. Cohen and Nycz [6] suggest that the teaching of adults (andragogy) should be approached in a different way to the teaching of children (pedagogy) and they advise that problem-based learning (PBL) should be supported by the platform. PBL aims to provide the information and tools needed to solve presented problems in the proper order, with the appropriate level of depth and presented in the correct format. The idea that a successful learner is an active learner (as opposed to a passive learner) is embedded in the constructivist theory of learning. In a constructivist environment, knowledge construction takes place in individual contexts and through social disclosure, collaboration and experience [6]; learning becomes an active process of constructing knowledge in such a way that the learner builds on prior knowledge and experience to draw meaning and construct new knowledge [7]. EMBL-EBI focus on training rather than education and so the teaching of adults is highly important when designing a learning environment. Thus, for the online learning environment to be successful, the leaner should be active, the andragogical approach should be implemented as PBL and the LCMS should support this [17]. Colace and De Santo [17] also explain that a more advanced LCMS should be able to store interactions between the user and each learning object, aiming at gathering detailed information about their utilization and efficacy. Colace and De Santo [17] also suggest that the features of the system as well as how they are integrated to facilitate learning and training should be examined. They suggest that to evaluate the platforms properly, both andragogical and technological aspects must be carefully investigated, thus, the features used to evaluate the platforms are usually separated into areas [18]. Graf [19] tested nine LCMSs in detail. The author used an example course, which simulated a real life teaching situation, to test each system. Graf split the main evaluation categories into subcategories, which were weighted and evaluated based on the experience from the usage of each LCMS when conducting the example course. Graf used a qualitative weight and sum approach which used six levels of importance for the weights, from ‘essential’ to ‘not valuable’. This approach provided a good starting point for developing our own rating system. We based our ranking system, on which features and requirements were important for us to assess in a LCMS, on this idea. However, we used two weights

557

558

Wright et al.

Table 4: This table displays features (from the andragogical and communication perspective) that EMBL-EBI considered in the evaluation of LCMSs

Table 5: Features (from the systems administration perspective) that EMBL-EBI considered in the evaluation of LCMSs

Category

Category

Communication tools File exchange Media and collaboration tools

Administration tools Monitoring progress

Features

Easy file management** Polls** Webcasts** RSS Feeds** Photos** Text chat (real time help)** Forums* External email* Video conferencing* Wiki functionality* Podcasts* Animations*

Communication tools Monitoring participation view stats on:

- The users on the course** - How the course is being used** Personal user profile* Username and password assignment* Productivity tools Navigation

Course delivery tools Course templates

Pedagogy

Study tools

Course management

Course evaluation

Multimedia indexing and searching (of e.g. lecture videos)** Users can view course map** Support to trainers for development of modules and courses. Available course templates** Quizzes** Assessment tools** Glossary and references** Learning objects creation and repository** Continuous feedback (may be automated)** Gradebook and associated tools* Resume reading content at previous point** On-line note taking or add notes to course content* Updates to users for news about the course** Publication of calendar for upcoming events** Course evaluation feature**

We used a star-rating system to define high-priority features: two stars mean ‘must have’ and one star means ‘good to have’.

into consideration: (i) communication and andragogy tools (Table 4) and (ii) technical requirements and features (Table 5). In our search for a suitable LCMS for the EMBLEBI we took into account the following requirements:  easy to use from all perspectives (systems administration, trainers, trainees and the organization);

Adaptability Graphical flexibility Can the organization customize the look of the layout?** Location flexibility Can content be delivered in multiple languages?* Personalization Can the user customize his/her own view of the platform?* Security Permissions Can it assign users permissions to edit by a specific criteria, thus controlling who can visit/edit the site?** Security Is it secure?** Are the vulnerabilities resolved quickly?** Costs Costs Is it open source? Is it free?** Support Community Is it easy to get answers to questions about the system?** Is there enough documentation?** Is the product available in your area?** Are the support teams are local?** System Programming Does it use appropriate programming languages?** Web Based Is it 100% web-based** Compatibility and portability Browser Does the platform refer to the SCORM compatibility standards?** What browsers are compatible? (versions)** Portability Can the software be installed on multiple OS? (versions)** Is it easy to import content?* VLE Standards Is the platform compatible with Mobile devices? (e.g. PDAs, laptops, iPods, cell phones)* Installation, maintenance and scalability Installation Is it easy to find and install add-on modules?** Is it easy to set up the pages for a simple site?** Is it easy to build a more advanced site?** Accessibility Is the navigation scheme simple?** Does it have scalable fonts and graphics?** Maintenance Is it easy to update the siteçadd pages, images, and updates links?** Is it easy to update the system?** Is there a system to monitor technical and content changes and give any errors?** Can you create content online?* Is it easy to download the code and files and install it?*

We used a star-rating system to define high-priority features: two stars mean ‘must have’ and one star means ‘good to have’.

 secure storage of the data, allowing only those users with the correct privileges to change the information (different levels of administration rights);

Downloaded from http://bib.oxfordjournals.org/ by guest on January 7, 2016

User progress tracking** Users can view own grades** Course progress tracking**

Features

Bioinformatics training: an example  easy to maintain and update;  allows our organization to provide a consistent look and feel to the courses that is customisable to EMBL-EBI’s design guidelines;  allows EMBL-EBI to build courses that cater for different learning styles and incorporate different types of materials;  space for users to add their comments and feedback;  compliance with eLearning standards;  repository for materials from face-to-face training courses; Organizational requirements that we took into account include the following:

The selection process To select an LCMS most suited to EMBL-EBI’s requirements from the vast list of the current popular eLearning platforms, we broke down the

selection process into these phases, summarized in Figure 1:  Phase 1: We generated a list of eLearning platforms by reviewing the literature (the same literature review process was used for finding LCMSs as was used to discover the most suitable methods for evaluating LCMSs—see above) to discern which platforms were worth investigating. This generated a list of 36 platforms (Table 6).  Phase 2: In accordance with EMBL-EBI’s philosophy of supporting open-source systems whenever possible, we selected only open-source LCMSs from our phase 1 list, thereby reducing the number of LCMSs to investigate to 13 (Table 6);  Phase 3: We (the EBI’s eLearning Content Developer and the Training Programme Project Leader) then performed a short investigation into each remaining LCMS. This involved: checking the platform’s website, including any online demonstration material; researching the literature produced by the platform author and by other authors; investigating the level of support provided for each platform and briefly evaluating the platform against the criteria summarized in Tables 4 and 5. This reduced our list to only three platforms—Drupal, ATutor and Moodle—to consider for further testing.  Phase 4: We (the EBI’s training team, members of the ES team and the Outreach and Training group leader) used the prioritized lists in Tables 1–5 to define exactly what our evaluators would test during the scripted platform testing.  Phase 5: Scripted testing of each of the three LCMSs was carried out by three sets of evaluators: (i) trainers (12 people), who were asked to created a course using each LCMS; (ii) trainees (nine people), who were asked to complete a course on each LCMS) and (iii) systems administrators (four people). The systems administrators were further divided into two groups: one group was asked to install the LCMS and any modules/ plugins required, and will ultimately be responsible for building and maintaining the EMBLEBI’s LCMS; the second group was asked to investigate the site and its content from an administrative perspective; for example, to find out how many users were enrolled on a course. This group will be responsible for administration of the online training programme and therefore will have access

Downloaded from http://bib.oxfordjournals.org/ by guest on January 7, 2016

 Free/open source—In accord with EMBL-EBI’s philosophy of supporting and contributing to open-source software projects, we are strongly in favour of using an open-source LCMS so long as it meets our requirements. Cohen and Nycz [6] add that open source platforms offer better reusability, reduced costs, and rapid fixes to bugs/ problems.  Comparison with tried and tested systems—Our eLearning pilot project was built using Moodle, so it was important to us that we compared other platforms to Moodle, hence, it had a place as one the final three LCMS lined up for testing. Moodle was previously chosen as the LCMS for the pilot eLearning portal as, at the time of choice, it was a free and popular LCMS which was considered to require minimal installation support and maintenance. The EMBL-EBI has also selected Drupal as a Content Management System (CMS) for its website, so there would be some obvious benefits to using the same system as an LCMS. Of course both of these platforms are successful and popular LCMSs [20, 21].  Compatibility with EMBL-EBI software—The EMBLEBI uses Linux as the main operating system for its servers, PhP to create dynamic web content and MySQL to store data. Compatibility with this software was therefore a prerequisite for our LCMS.

559

560

Wright et al.

rights greater than those of trainers. All three LCMSs were installed on the EMBL-EBI web server and an identical sample course (a method supported by Graf [19]) was set up on each platform. We chose to use a topic unrelated to bioinformatics, because we felt that this would best enable our platform testers to focus on the usability of the LCMS as opposed to the quality of the content. The evaluators were assigned a role (e.g. trainer) and asked to follow a guideline script, complete a questionnaire and partake in discussions about the platforms (Supplementary Data). The feedback provided through this process provided us with guidelines for determining which platform best meets the needs of trainers, trainees and systems administrators. Full guidelines used for the course creation as well as the forms used for the evaluation of the LCMSs are available as Supplementary Data.  Phase 6: After the physical testing of the platform the feedback was collated and analysed and a platform was chosen. The evaluation of the results is presented in the following section.

EVALUATING THE RESULTS To choose the most suitable LCMS for EMBL-EBI out of the three potential platforms (Drupal, Moodle and ATutor), the user feedback from the scripted testing day (Phase 5) was collated and analysed:  The type of user supplying the feedback (systems admin, trainer and trainee) was taken into consideration at all points in the feedback evaluation process;  The platforms were ranked according to which LCMS the user had chosen as their favourite and their least favourite platform;  Qualitative information, provided as comments in the evaluation forms (for example, on what the user liked the most and least about the platform) were consolidated to create a list of the main advantages and disadvantages of each platform;  Individual feature evaluations from all perspectives were examined for each platform, for example, how easy it was to create (trainer) and complete a quiz (trainee);  The users’ feedback after completing testing of all three platforms was discussed;

Downloaded from http://bib.oxfordjournals.org/ by guest on January 7, 2016

Figure 1: Flow diagram of the different phases of the project and the stakeholders at each stage. The inner square boxes describe which stakeholders had input in that phase. OTT is the EMBL-EBI’s Outreach and Training Team, which is responsible for coordinating our communications and training programmes; ES is the External Services Team, which is responsible for maintaining and developing the EMBL-EBI website.

Bioinformatics training: an example Table 6: List of popular LCMSs was generated by reviewing past and present literature to discern which platforms to include on our ‘long list’ Free open source?

A Tutor Adobe Connect Pro Apriori BELTS Blackboard Bodington Claroline Cognitivity Docebo Dokeos Drupal Ecollege EduZope Efront ILIAS iLinc Interact Intralearn Joomla (LMS) LAMS LON-CAPA .LRN Meridian KSI Moodle OpenLMS Plateau Saba Suite Sakai (CMS) SharepointLMS Spiral universe Sumtotal systems SwitchPort Telania Thinking cap TopClass (WBT)

Yes No No No No Yes Yes No Yes Yes Yes No No Yes Yes No No No No No Yes Yes No Yes Yes No No Yes No No No No No No No

is merely an empty vessel, which will then need to be filled with courses. In parallel with work on identifying a suitable LCMS, we have gathered comments from potential users of our eLearning platform, including molecular life scientists attending our hands-on training courses, delegates at our roadshows, and masters’ students at our open days. Their feedback, and our own previous experience of creating course materials, is being used to create a prototype course, which will be tested by trainees and refined in light of their feedback. As we learn from this process, we will apply our learning to the development of further courses, initially focusing on Europe’s core data resources and on our most frequently used bioinformatics tools. An important aim of this project, in line with EMBL-EBI’s philosophy of making the results of its work openly accessible, is to enable others to reuse and build upon our training materials. We have therefore begun to engage with a broader community of trainers from beyond the EMBL-EBI to gather their requirements in the hope that we may be able to support some of their training needs. This work is discussed in more detail in the accompanying paper by Schneider etal. We warmly encourage readers of this paper to contact us with comments and suggestions, by e-mailing us at [email protected]. Should you wish to learn about future eLearning developments at the EMBL-EBI, we encourage you to join our mailing list: http://listserver.ebi.ac .uk/mailman/listinfo/elearning.

The table also describes which platforms are freely available and which are open source (the source code is published and made available to the public, so it can be copied, modified and redistributed by anyone for free).

CONCLUSION

 The feedback and options were reviewed by the authors of this paper and a platform was chosen;  A list of requirements, based on feedback from all three groups, was compiled and used as the basis of a prototype for the EMBL-EBI’s eLearning platform, which is now in development.

(1) define and involve the different types of users (administrators, trainers and end users); (2) prioritize your requirements; (3) find an unbiased way of scoring these requirements against each LCMS being tested and (4) recognize which issues raised during the testing are crucial to a go/no go decision.

Choosing an appropriate LCMS is a complex task that requires you to:

RELATED WORK AND FUTURE DIRECTIONS

SUPPLEMENTARY DATA

We have now begun the task of translating our list of requirements into a portal; of course, the portal itself

Supplementary data are available online at http://bib .oxfordjournals.org/.

Downloaded from http://bib.oxfordjournals.org/ by guest on January 7, 2016

Platform

561

562

Wright et al.

Key Points  Users of bioinformatics data resources and tools require training on how to exploit these services to their full potential; providing such training is a core part of the EMBL-EBI’s mission.  eLearning has the potential to provide a valuable complement to face-to-face training; it is likely to become increasingly important as increasing numbers of bioscientists, from a diversifying range of fields, begin to exploit bioinformatics data resources and tools.  Choosing an appropriate learning content management system (LCMS) is an important first step towards developing an online training programme. This review summarizes the process that the EMBL-EBI went through to make this choice.  The selection of an LCMS is specific to each organization and this review makes no claims for the LCMSs that it shortlisted; we focus instead on the process that we went through, in the hope that our experience may be valuable to other bioinformatics trainers who need to make their own choices.

10. 11. 12.

13.

14.

15.

1.

2. 3.

4. 5.

6.

7. 8.

9.

Kowalewska A. The possibilities of using e-learning in students’ training in the field of health education. Rocz Panstw Zakl Hig 2006;57:29–33. Saravanana V, Shanmughavel P. E-Learning as a new tool in bioinformatics teaching. Bioinformation 2007;2:83–5. Brooksbank C, Cameron G, Thornton J. The European Bioinformatics Institute’s data resources. Nucleic Acids Res 2010;38:D17–25. Zhang D, Zhao L, Nunamaker L. E-learning replace classroom learning? Commun ACM 2004;47:74–9. Kruse K. The benefits and drawbacks of e-Learning. http:// www.e-learningguru.com/articles/art1_3.htm (20 July 2009, date last accessed). Cohen E, Nycz M. Learning objects and e-Learning: an informing science perspective. Interdisciplinary J Knowledge Learn Objects 2006;2:23–34. Koohang A, Harman K. Open source: a metaphor for e-Learning. Inform SciJ 2005;8:75–86. Mircea G. The future of e-Learning. Designing Tomorrow’s Education 2006. Available at SSRN: http://ssrn.com/ abstract=904201 (23 June 2010, date last accessed). Haverila M, Barkhi R. The influence of experience, ability and Interest on e-learning effectiveness. EurJ Open Distance

16.

17.

18.

19.

20.

21.

Downloaded from http://bib.oxfordjournals.org/ by guest on January 7, 2016

References

E-Learning 2009. http://www.eurodl.org/?keyword= collaborative%20learning&article=359 (23 June 2010, date last accessed). Google. Google scholar. http://scholar.google.co.uk/ (22 September 2009, date last accessed). Elsevier B. Scopus. http://www.scopus.com/ (21 September 2009, date last accessed). Manchester U. Bioinformatics online modular degree programme. http://octette.cs.man.ac.uk/bioinformatics/ (16 March 2010, date last accessed). BIREC Bioinformatics information resource and eLearning center. http://www.birec.org/ (16 March 2010, date last accessed). Sczyrba A, Konermann S, Giegerich R. Two interactive bioinformatics courses at the Bielefeld University Bioinformatics Server. Briefings in Bioinformatics 2008;9: 243–9. Cattley S, Arthur JW. BioManager: the use of bioinformatics web application as a teaching tool in undergraduate bioinformatics training. Briefings in Bioinformatics 2007;8:457–65. Bouchard P. Some factors to consider when designing semi-autonomous learning environments. ElectronicJ e-Learn 2009;7:93–100. Colace F, De Santo M. Evaluating on-line learning platforms: a case study. Proceedings of the 36th Hawaii International Conference on System Sciences 2003. Big Island, HI, USA: IEEE Computer Society, 2003. Graf S, List B. An evaluation of open source e-learning platforms stressing adaptation issues. Proceedings of the Fifth IEEE International Conference on Advanced LearningTechnologies. Washington, DC, USA: IEEE Computer Society, 2005; 163–5. Graf S, Kinshuk, Liu T. Supporting teachers in identifying students’ learning styles in learning management systems: an automatic student modelling approach. EducationalTechnology & Society 2009;4:3–14. Kurilovas E, Dagiene V. Learning objects and virtual learning environments technical evaluation criteria. Electronic Journal of e-Learning 2009;7:127–36. Mooney S, Baenziger P. Extensible open source content management systems and frameworks: a solution for many needs of a bioinformatics group. Briefings in Bioinformatics 2008;9:69–74.