IWWUA presentacion ppt - GPLSI

Usability and Accessibility Evaluations along the eLearning Cycle

Ludivine Martin, Emmanuelle Gutiérrez y Restrepo, Carmen Barrera, Alejandro Rodríguez Ascaso, Olga C. Santos, Jesús G. Boticario

aDeNu Research Group http://adenu.ia.uned.es Adaptive Dynamic online Educational systems based oN User modelling Artificial Intelligence Department, Computer Science School, UNED, Madrid

PLAN 1. Introduction 2. E-learning, LMS, Usability 3. Adaptation in the e-learning lifecycle 4. Usability and accessibility in the learning life cycle • Interactions • Methodologies • Results 5. Conclusion and Challenges

aDeNu Research Group http://adenu.ia.uned.es

2

1. Introduction


3

1. Introduction •

aLFanet Development of an adaptive LMS (Learning Management System), which combined design and runtime adaptation along the full life cycle of the learning process with a pervasive use of educational standards from IMS (Instructional Management Systems).

•

EU4ALL European Unified Approach for Accessible Lifelong Learning. Development of a framework to support the delivery of learning services along the life cycle of the learning process to cover the needs of people with disabilities.

•

ALPE Accessible eLearning Platform for Europe: a validation project involving an accessible, standard based e-learning solution for visually impaired, hearing impaired and adult learners.


4

2. E-learning, LMS, usability


5

E-learning • E-learning = learning through electronic media, or computer-enhanced learning • Currently a concept assimilated in educational centres at all stages, especially in higher education institutions trying to support the life-long learning paradigm. • It removes time and space barriers --> benefit for people with disabilities although it is not always the case. • Current research in e-learning strongly focuses on providing adaptation support to users. Personalized learning adapted to individual needs and preferences improves the learning process effectiveness. aDeNu Research Group http://adenu.ia.uned.es

6

E-learning • Disappointment and frustration are common in e-learning, due to – poor features – lack of pedagogical guidance – ineffective evaluation procedures – little support to fulfil administrative tasks. – ….usability issues • A direct correlation has been established between the key aspects in the usability field and their effect on learning. • In the case of e-learning, the main task for the user is learning (P. Zaharias) aDeNu Research Group http://adenu.ia.uned.es

7

E-learning •

Combination of usability and instructional design (L. Lohr): –

Usability (ISO, 1993) • Effectiveness: The user's ability to achieve specific goals in the environment • Efficiency: The resources used when performing a systemsupported task • Satisfaction:The user's comfort level and acceptance of the system overall

– Formative evaluation (Instructional Design) • Effectiveness: The attainment of instructional objectives • Efficiency: How quickly and cost-effectively learning objectives were achieved • Satisfaction: The user's interest in the content and the desire to continue to learn aDeNu Research Group http://adenu.ia.uned.es

8

E-learning • Combination of instructional design and usability (L. Lohr): – Definition for the instructional interface design process • Effectiveness: Learner interprets instructional interface function correctly; instructional interface function performs according to the learner's expectations • Efficiency: Learner experiences minimal frustration interpreting instructional interface function; learner experiences minimal obstacles in using instructional interface element • Satisfaction: Learner seems comfortable in the environment overall aDeNu Research Group http://adenu.ia.uned.es

9

E-learning • Double persona: learner-user (D. Smulders) Using a e-learning site = making functional connections Learning = making cognitive connections • Usability as a precursor of learnability • Usability shouldn’t be a barrier to learning, especially because it can impact motivation. • Usability as a basic condition for usable e-learning systems


10

E-learning • Definition of usability in e-learning includes more components than interface design (S. Shilwant) – usefulness : utilizing knowledge gained during the needs assessments to ensure usefulness – ease of use: employing interface design principles to achieve ease of use – learnability : using instructional design to attain learnability – motivation: focusing on motivational principles (feedback, curiosity, challenge). Success of a training experience largely dependant in student own motivation. Importance of affective dimension.


11

E-learning • What is usable e-learning? (M. Feldstein): Starting point: learnability. Definition of usability in e-learning = the ability of a learning object to support or enable a very particular concrete cognitive goal. --> which specific presentation or interface features have measurable impact on specific cognitive tasks or goal ? example: does the course navigation design teaches the learner about the course structure? Or does it create visual static that interferes with the learner’s ability to process the main points on a page?


12

E-learning • How to measure usability in e-learning? – Expert methods based on heuristics or adapted heuristics (motivation to learn) – Empirical methods based on satisfaction questionnaires. – On-going research on methods addressing both technical and pedagogical issues – No out-of-the box widely accepted usability evaluation in e-learning exists. – Need for usability specialists to move “out of the comfort zone” and get familiar with methods from instructional design and learning theories. (M. Feldstein)


13

Learning Management Systems (LMS) • Large variety of LMS, also called VLE, Virtual Learning Environments • Commercial or open source • Different categories : – general LMS with tools for create and manage courses, – collaborative learning support systems – questionnaire and test authoring systems – people and institutions resources management systems – virtual classrooms One LMS can belong to several categories at the same time. aDeNu Research Group http://adenu.ia.uned.es

14

Learning Management Systems (LMS) • Wide variety of integrated features: – school management – course management – class management – content management – communication tools – student tools – assessment tools – etc.


15

dotLRN • •

• •

•

Before adopting a LMS: identification of the needs and comparative analysis UNED chose ArsDigita Community System (ACS) developed at the MIT. Along time, ACS evolved into dotLRN (officially named .LRN), open source enterprise e-learning platform. dotLRN today is used by over a 1/2 million users in more than 18 countries. complete portal framework along with out-of-the-box capability for course management, online communities, content management, and learning management. Used as a framework in our projects and as a collaborative tool.


16

LMS particularities •

•

•

Out-of-the-box product. Usually an institution will adopt an existing LMS, not build a new one. Therefore the LMS already exists. It is not built for a very specific purpose and audience. High level of customization. LMS usually offer a wide range of configuration options at the admin level, the tutor level and the student level. This flexibility affects the overall usability of the system Variety of needs. LMS have to be flexible enough to address a variety of teaching, learning and collaborative styles.


17

LMS particularities when evaluating usability and accessibility •

Educational standards. To facilitate the reusability of the author work and the adaptability of the course to the learner. Educational standards (IMS, SCORM), usually complex, impact the LMS user interface, therefore the user experience


18

LMS particularities when evaluating usability and accessibility •

Container/contained relationship. – the platform, or “shell” or “receptacle”, including “modules”, where the course materials are stored and delivered – the formal content, or packaged course materials, compliant to educational standards. Here the line between interface design and course design begins to blur – content generated by users, or content produced by the entire community


19

3. Adaptation in the e-learning lifecycle


20

3. Adaptation in the e-learning lifecycle • Learning ideally should be a personalised and adaptive process which from beginning to end should consider the learner’s specific needs. • aDeNu research group (aLFanet project): deliver adapted courses based on – use of educational standards – user-modelling techniques – multi-agent architecture : machine learning techniques to learn the model attributes and provide recommendations to learners. • Objective: adapt the user interface to the learners’ background, interests, special needs, functional abilities and interaction preferences. aDeNu Research Group http://adenu.ia.uned.es

21

3. Adaptation in the e-learning lifecycle


22

4. Usability and accessibility in the e-learning life cycle


23

Interactions


24

Interactions

• • •

•

(a) Authoring / Design phase / LMS development phase integrates the A&U requirements defined via requirements gathering and elicitation methods. integrates the evaluation results from the auditing phase that closes the cycle. Results become requirements or new guidelines for redesign. course design: author needs to take into account A&U guidelines. Before delivering the course, a strict evaluation on accessibility, usability and standards compliance is performed LMS development: A&U evaluations (manual and validators) are performed at this phase, documented in a Bug Tracker.


25

Interactions • (b) Administration or publication phase U&A evaluations to detect persistent failures or desirable improvements that come up when the course design is imported in the LMS. U&A inspection methods (expert reviews): heuristic evaluation and cognitive walk through (task-based scenarios). • (c) Use phase Real life with real users (students and tutors) in real situations of use. Empirical testing or user-based methods: direct observation of a sample of users (thinking aloud and contextual inquiry), with task based scenarios. We also engage users to report failures or suggest improvements. (contact forms) Tracking of the users interactions (web logs) aDeNu Research Group http://adenu.ia.uned.es

26

Interactions • (d) Auditing or feedback phase Results from experts, users evaluations and automatic processing of interactions are analyzed and translated into technical requirements or guidelines for improvements to be integrated in the next release.


27

Accessibility Methodologies • LMS accessibility evaluation based on – Web Content Accessibility Guidelines (WCAG) – but also the other accessibility guidelines in order to cover all LMS processes, components, including any integrated interaction elements or applications (chat for example): • ATAG: Authoring Tool Accessibility Guidelines 1.0 • UAAG: User Agent Accessibility Guidelines 1.0 • GDALA: Guidelines for Developing Accessible Learning Applications 1.0 • Comprehensive accessibility evaluation = technical assessment + evaluation with end users (including users with disabilities). aDeNu Research Group http://adenu.ia.uned.es

28

Accessibility Methodologies Technical Assessment • Evaluation all templates, content, and templates and content combined. • In-house accessibility expert extensive manual review • Audit from external experts (Tenuta) • 2 Web accessibility evaluation tools: HERA , AccMonitor End-users assessment • Continuous conformance assessment with specific users with different disabilities and using different assistive technologies • Upcoming large testing as part of the ALPE project: 300 users over 3 countries. aDeNu Research Group http://adenu.ia.uned.es

29

Usability methodologies Expert Assessment • In house expert: Heuristic evaluation • External experts from Tenuta project: task scenarios, evaluation based HHS guidelines.


30

Usability methodologies Evaluation with users • Short sessions with involved users. Task based scenarios. Direct observation. Iterative design. • ALPE project market validation project --> focus on the user satisfaction / acceptance of the product. 1. Large-scale survey with 300 users over 3 countries (Spain, UK, Greece). Online trial including task based scenarios and a questionnaire (background and feedback questionnaires). 2. Small-scale survey: Real course over 3-month period in real conditions with a real tutors. Online questionnaires. 3. Laboratory session Direct observation. Task based scenarios. Thinking Aloud.


31

Results Accessibility Results • The initial expert accessibility assessment detected 22 problems: 27 % of Priority 1, 64% of Priority 2 and 9% of Priority 3. 75% of those accessibility problems have been fixed. The remaining issues are under study for the next release. • The accessibility evaluation by Tenuta experts was also positive: “The website has been well coded, offers a number of accessibility features and no major accessibility issues were identified.”


32

Results Usability Results •

•

Heuristics study showed that the platform complied particularly well on the following categories: – Aesthetic and Minimalist Design – Pleasurable and Respectful Interaction with the User – Visibility of System Status. Improvements must be made in the following categories: – Match Between System and the Real World – Help and Documentation. Positive conclusions from the Tenuta usability audit : "This website has been thoughtfully produced. There has obviously been a thorough consideration of the user throughout. There are several minor usability issues that could be addressed to improve its usability.“


33

5. Conclusions and challenges


34

Conclusions and Challenges • From the point of view of usability, LMS have specificities that must be taken into account in a user-centred framework. • LMS are complex systems and A&U need to be considered for the entire e-learning life-cycle. • Evaluations along the cycle in a continuous process. • ALPE and EU4ALL projects as a framework to dotLRN development


35

Conclusions and Challenges • Accessibility of the content generated by users or how to support accessible collaborative learning. Need for accessible authoring tools + automatic assessment of created content. The LMS can provide certain support, such as WYSIWYG editor that forces the learners to include alt and longdesc text for every image linked, but there is no guarantee that the text provided fully describes the image. Often, WYSIWYG editors are not accessible themselves. Therefore, we need to increase the accessibility awareness level of all users, by creating some guides for example.


36

Conclusions and Challenges • Usability of the content generated by user --> editorial quality (instructive design) • Ensure usability of a highly flexible system lot of customization options at every level which moreover integrates – components (the platform, the course, the content generated by users) – generated from different sources (developers, administrators, authors, students, tutors) – for different purposes.


37

Conclusions and Challenges • Future work – ALPE: 300 users trials – Integration of wiki and blogs in dotLRN.


38