Conceptual definition of the technical and ...

Conceptual definition of the technical and pedagogical usability criteria for digital learning material Petri Nokelainen University of Tampere and Hame Polytechnic, Finland [email protected] Abstract: The major goal of the multi disciplinary Digital Learning 2 project is to develop a tool to evaluate usability of digital learning materials. This paper concentrates on conceptual definition of the technical and pedagogical usability criteria. We developed following components of the technical usability: 1. accessibility, 2. learnability and memorability, 3. user control, 4. help, 5. graphical layout, 6. reliability, 7. consistency, 8. efficiency, 9. memory load and 10. errors. Components of the pedagogical usability are as follows: 1. learner control, 2. learner activity, 3. cooperative learning, 4. goal orientation, 5. applicability, 6. added value, 7. motivation, 8. valuation of previous knowledge, 9. flexibility and 10. feedback. A PMLQ research instrument containing 92 items was developed on the basis of forementioned technical and pedagogical usability criteria. Preliminary empirical validation of the instrument showed that the data (5th and 6th grade elementary school children, n=66) supported the dimensionality of the criteria.

Introduction The multi disciplinary Digital Learning 2 (DL 2) research project (2001 -) has members from research fields of education, computer science and hypermedia. The major goal of the project is to develop a tool to evaluate usability of digital learning materials and environments. In order to reach that goal, we need to define numerous concepts, for example, technical and pedagogical usability. This paper is about both technical and pedagogical usability criteria from which we derive the questions that are presented to the user when she is evaluating learning material or environments. Purpose of the evaluation tool is not to classify evaluation targets to ”good” or ”bad” ones, but instead, to show in understandable manner some of the design principles and goals behind the evaluation target. We see two possible ways to use the tool: Firstly, to evaluate any given system or learning material by self-report multiplechoice questionnaire, and secondly, to seek for other user’s evaluations of learning systems or materials. The evaluation tool discussed here is just an abstraction. In practice, implementations of the tool will be embedded into various, mainly commercial, systems to provide feedback channel from students and teachers to system developers. The evaluation tool may also act as a peer-to-peer forum of shared experiences as users are able to search for each other’s recommendations of learning materials. Following stages of the DL 2 project have been essential to developing the criteria for technical and pedagogical usability: 1. Investigation of the previous research focusing on usability of learning materials or environments, 2001-2002, 2. Analysis of relevant psychological and educational theories, 2001 – 2002 (Syvänen & Nokelainen, 2002), 3. Development of the first version of pedagogical usability criteria, 2001 – 2002 (Horila, Nokelainen, Syvänen & Överlund, 2003), 4. Development and empirical testing of the technical and pedagogical usability criteria, 2003 - . The technical and pedagogical usability criteria presented here was developed for the needs of 5th and 6th grade elementary school students and teachers. This was due to fact that one of the DL 2 projects main sponsors, eWSOY, has a large scale learning environment, OPIT. The system has over 4.000 learning units covering almost every elementary school subject from grade 3 to 6. The second reason for choosing the OPIT was the fact that eWSOY’s professional development teams produce all the learning units. This controls the quality level of how learning units and their contents are developed and implemented. Third reason for evaluating learning material in the OPIT was the fact that four project members are elementary school teachers in Finnish schools that are actively using the system. First, we give an overall view to the elementary concepts of the two usability criteria, technical and pedagogical, that are behind the evaluation process. Next we give a more detailed view to the theoretical framework of both usability criteria. Finally, we present preliminary empirical results and conclusions.

Elementary concepts of the study

Digital learning material covers in this study all learning material that is published in digital format, for example a CD-ROM about medieval art, or a trigonometry math learning unit in a learning environment. Learning object (LO) is defined as a smallest meaningful part of a learning material, for example a voice sample or animation sequence demonstrating safe operation procedure in factory. Almost all learning objects have some sort of goal, and some of them have even a manuscript. However, in most cases their focus is too limited to be evaluated even by technical usability. A web page teaching how to optimize image for different publication purposes, a JAVA applet to practice for touch-type system, or a single learning module in learning environment or platform, are all examples of individual learning material (LM). Each of them has an individual user interface and contents that we are able to evaluate technically and pedagogically. Thus we may treat some of the PBL (problem based learning) data sources, whether produced by teacher, some third party or learners themselves, as individual learning material. Unit of learning material (ULM) consists of individual LM’s with common goal. ULM is an abstraction that binds parts (LM’s) together to form a greater picture of specific major topic. It is difficult to define optimal size for ULM: if the unit is large and versatile, its use is limited as it has too much overlap with other units, if the unit is small and sharply focused, the number of units needed to cover any major topic is likely to grow too large to handle properly (Hiddink, 2001, 307). As an example of an ULM, we may think of a web page offering instruction how to use digital camera. The major goal (”How to take a good picture with a digital camera?”) might be divided into following sub goals (LM’s): ”What are the central concepts of digital image?” and ”How to shoot a moving target?” Those sub goals have their sub goals such as ”What is a pixel?”, ”What is dpi?” etc. The nature of the latter questions leads developer to choose optimal media for the message, for example a power point slide show is one way to present central concepts of digital image, and FLASH animation might be a proper way to show how to capture motion successfully. A PBL material could also be seen as ULM with LM’s, as all formal learning that takes place in schools is to be planned by teacher. For example, when teacher sets up a PBL environment, she usually prepares stimulation tasks and primary data sources to serve certain goal or sets of goals. As students enter the system, they use the prepared material and bring in some of their own, to serve the learning goals defined by themselves. Students feel that they ”own” their goals, but usually those goals are a subset of major goal defined by the teacher when setting up the environment. This leads us to conclusion that also a PBL environment has features that we may evaluate with technical and pedagogical usability criteria. Virtual learning environment (VLE) is a platform for individual learning material or unit of learning materials. Main divide between VLE’s is commercial vs. non-commercial systems and ready-made contents vs. no readymade contents. Next we elaborate latter of those categorizations in greater detail. The target platform of this study, the OPIT, is an example of first category: a system that contains ready-made contents. Goal-orientation and structuralization are typical features of learning materials in this kind of systems. Teachers are in most cases not expected to produce their own learning materials; Students are usually not making their own goals, as the materials are highly structured. PBL may be present, but usually not in its collaborative forms. The second category, learning platforms without contents, is divided into two main sub categories. First one contains platforms that have special tools like chat and news, but mainly attract teachers to upload structured presentations (WebCT, Black Board). Collaborative work is enabled by these systems, but mostly it depends on the nature of individual learning materials created by teachers an instructors. Second sub category contains three kinds of platforms that all specialize in collaborative knowledge constructing: Firstly, ones that enable file sharing (WorkMate), secondly, ones that support news and email chat or certain special feature like annotation (Dyn3W, OurWeb), and thirdly, ones that offer graphical tools to analyze information (Fle3, Knowledge Forum, Belvedere). According to Keinonen (1998, 62), usability defines products development process, product itself, use of product, and user experiences with product and user expectations with product. In this study, we define usability as attributes of the product itself. These attributes can be measured with self-report questionnaires via end users subjective experiences. In this study, operationalization of usability attributes is based on Nielsen’s (1990, 148) model, where the toplevel concept is system acceptability. System acceptability has two components: 1. social acceptability, and 2. practical acceptability. The main point here is, that every learning platform and material communicates implicitly and explicitly its developers implicit and explicit views about social and practical acceptability.

Nielsen gives two examples of violating social acceptability. The first one is about allowing only teacher, not students, to make additions and modifications to the learning material (1990, 147-148). In the second example he asks, if it is socially acceptable to create a computer program to expose those who have submitted fraudulent applications for unemployment benefits (1993, 24). The tone of social acceptability is biased slightly more towards ethical and moral choices of program developers in the latter example, as in the first one the focus is also in allowance of user control. Practical acceptability is composed of cost, compatibility, reliability and usefulness (Nielsen, 1990, 148). Usefulness is further divided into utility and usability. Entertainment product has high utility for end user if it is fun to use, and learning material has high utility if learners learn from using it (1993, 25). Nielsen (1990, 151) divides usability further into five usability attributes: learnability, efficiency, memorability, errors and satisfaction. Learnability is the time that it takes from novice user to learn to use the system for productive work. Efficiency is the productivity level that experienced user can reach with the system. Memorability is the time that it takes from occasional user to recall essential features of the system. Errors are divided into two sub groups: less severe and severe errors. Satisfaction means end users subjective evaluation of how pleasant it was to use the system. Pedagogical usability is categorized in this study into sub component of utility, as technical usability was a sub component of usability. Thus dialog between user and system, and learning goals set by student and teacher are both aspects of pedagogical usability: Does the learning platform, and/or the learning material embedded into it, enable student and teacher to reach the learning goals? We must bear in mind that operationalized, i.e. the ones that we can ask from respondents via questionnaire, dimensions of technical and pedagogical usability may correlate with each other and thus both represent latent dimensions, i.e. those that we can not directly measure, of usability and utility. User control, memory load and feedback are examples of such common components between user and system, and user and contents of learning material. For example, we may conclude that the system has given control to the user, when the user has no difficulties using the system. However, goals of pedagogical and technical usability are not always in harmony. It is not optimal to minimize memory load when pedagogical usability is concerned, in order to learn, one must be active - in some cases long-term memory is activated more effectively when memory load is increased. On the other hand, minimal memory load is in most cases desirable state when thinking about technical usability. Evaluation is defined in this paper as end users subjective experiences of learning system or material measured with self-report questionnaires. However, criteria described here works also as a ground for heuristic evaluation (Nielsen & Molich, 1990). According to researchers specialized in usability measurement (Rubin, 1994; Nielsen, 1993; Kirakowski, 2003), it is viable to use questionnaires to collect demographic data (pre test), evaluate system and conduct posttest. Kirakowski (2003) sees questionnaire as a relevant tool to measure end user subjective satisfaction. Various test designs are more appropriate to collect factual information, for example time to complete certain task (Rubin, 1994; Nielsen, 1993, 192-200).

The technical usability criteria After analyzing the instrumental concepts, we focused on previous research on usability of digital learning material and environments. We learned that neither seminal usability readings (i.e., Shneiderman, 1988; Nielsen, 1993; Preece, Rogers & Sharp, 2002; Tognazzini, 2003) nor studies (i.e., Chin, Diehl & Norman, 1988; Lin, Choong & Salvendy, 1997; Henninger, 2000; Chalmers, 2003) expanded matters of technical usability to usability issues characteristic to learning material. However, since 90’s, several research groups have focused on pedagogical dimensions of usability (Reeves, 1994; Squires & Preece, 1996; Soloway et al., 1996; Quinn, 1996; Albion, 1999; Squires & Preece, 1999; Horila, Nokelainen, Syvänen & Överlund, 2001). We decided to treat, as described earlier in the previous section, technical and pedagogical usability as two separate, but partly overlapping dimensions. Table 1 presents most widely applied technical usability dimensions (Shneiderman, 1988; Chin, Diehl & Norman, 1988; Nielsen, 1993; 1994; Lin, Choong & Salvendy, 1997; Preece, Rogers & Sharp, 2002; Chalmers, 2003; Tognazzini, 2003). Usability goals, principles, heuristics, rules and questionnaires presented in the table have several common components, such as consistency, user control, and error handling and recovery. Most of the components in the table are still relevant, but for example accessibility (W3C, 1999) is missing. Left hand side of Table 1 is about abstract definition of usability (usability goals), the middle is about practical observation

of usability (principles / heuristics) and the right hand side is about concrete evaluation of usability (usability questionnaires). Practical implication of this study followed phases of Table 1 from abstract (left) to concrete (right). We examined the theoretical definitions of usability and compared them to practical usability goals, principles, heuristics and rules in order to find general trends of usability. We analyzed well-known usability questionnaires for at least three purposes. Firstly, our goal was to examine how the items were operationalized from various theoretical usability aspects. Secondly, we wanted to learn as ”standard” use of terms as possible. Thirdly, we were curious about features that characterize usability questionnaires. Perlman (1998) has implemented a useful tool to analyze various usability questionnaires. We examined for the purposes of this study following instruments: Questionnaire for User Interface Satisfaction ”QUIS” version 1 (Chin, Diehl & Norman, 1988), Purdue Usability Testing Questionnaire ”PUTQ” (Lin, Choong & Salvendy, 1997), ”QUIS” version 7 (Norman, Shneiderman & Harper, 2003), and Software Usability Measurement Inventory ”SUMI” (Kirakowski, 1994). We stress here users subjective experience of usability, as the major goal of this study is to analyze theoretical structure of technical usability in order to produce a questionnaire for the purposes of DL2 projects’ evaluation tool. We did not study instruments measuring users attitudes toward computers (for example TAM, Technology Acceptance Model), as it is two different things to measure subjective satisfaction as an attribute of usability, and as an attitude towards computers (Nielsen, 1993, 33). However, some of the TAM model dimensions, and dimensions in both technical and pedagogical criteria, are closely related. Technical usability criteria applied in this study are as follows: 1. accessibility, 2. learnability and memorability, 3. user control, 4. help, 5. graphical layout, 6. reliability, 7. consistency, 8. efficiency, 9. memory load and 10. errors. (Table 1.)

Table 1. Summary of the previous research of technical usability. Figure 1 presents the measurement model of technical usability. Theory based and thus tentative relationships between usability dimensions are marked with arches that show no direction of effect. We intent build more specific model in the near future, as we are able to test it with numerous empirical samples. The model has two levels, first one with 1. accessibility, 2. user control and 3. learnability and memorability, and the second one with the dimensions 4.-10. The idea behind this categorization is to distinct more latent from more concrete dimensions. The latent dimensions cover such things that we are not able to ask from end user, but instead we are able to ask them from expert. For example, we may ask from a 6th grade elementary school teacher, but not from student, if the learning material is suitable for students with motoric disabilities. The criteria are in order of importance when purpose of use is to evaluate existing digital learning material.

Figure 1. Tentative measurement model of technical usability criteria. Next we walk through the technical usability criteria presented in Figure 1 in the context of evaluation of digital learning material. Dimension

Description

1. Accessibility

The most important point is that the learning material has no value for learner, if she is not able to use it in the first place. Learner should be able to use learning material with different browsers and devices. (W3C, 1999.) The system should provide an option for various media elements (such as narratives and pictures). Needs of real users should have priority compared to needs of designers and programmers of the system (Ambler, 2001). Learnability concerns novice, and memorability concerns casual expert user (Nielsen, 1993, 31). System that is hard to learn is only valuable for those users who are able to spend time to learn it. System that is impossible to learn has no value for any user. Worldviews of the system and user should meet - at least in conceptual level (Norman, 1988). Friendly help and error messages improve learnability and memorability of the system (Lin, Choong & Salvendy, 1997). User has the feeling that the software operates for her, not the opposite way (Shneiderman, 1998; Nielsen, 1993; Tognazzini, 2003; Lin, Choong & Salvendy, 1997). Use of the system should be so intuitive, that no "help" is needed (Squires & Preece, 1996). The system visibility means, that user is all the time aware of what the system is doing (Shneiderman, 1998; Nielsen, 1993.) User should be able to navigate to any available part of the system without unnecessary restrictions. If user has multiple options to choose from, only the relevant ones given the task should be enabled. Help should be available at any time Nielsen (1993, 149) and in all possible situations, meaning that those situations should be mapped before real users start using the system. Help and instruction should be presented in understandable form (Nielsen, 1993, 151-152). Context sensitivity, fading and proactivity, if properly implemented, works with help. Well-designed help scaffolds user to use system in more productive way (Chalmers, 2003). Layout is structured in best possible way to promote users ability to use the system, for example according to Fitt's law "the time to acquire a target is a function of the distance to and size of the target" (Tognazzini, 2003). User interface should be intuitive for most users. Information (subject) and structure (user interface) should be two different things (Leflore, 2000, 103). Simplified graphics should be used when introducing new information (Leflore, 2000, 103-104). Color should not be used as a principal source of information (Marcus, 1992, 87). Signaling (Mautone & Mayer, 2001) is a viable way to highlight essential matters in learning contents, as users tend to treat visually similar text blocks also similar regarding to their contents.

2. Learnability and memorability

3. User control

4. Help

5. Graphical layout

6. Reliability

System should be technically reliable. User should be able to trust that her work is safe with the software. (Nielsen, 1993; Shneiderman, 1998; Tognazzini, 2003). Metacoding should improve technical reliability of learning material, as for example hyperlinks refer more often to right targets (Hiddink, 2001).

7. Consistency

8. Efficiency of use 9. Memory load

10. Errors

Consistent user interface gives the user transferable skills, that are useful in other systems, too (Shneiderman, 1998; Nielsen, 1993; Tognazzini, 2003; Lin, Choong & Salvendy, 1997; Chalmers, 2003). User interface components should be placed in consistent way. Learning object standardizing should promote use of consistent user interface design (Hiddink, 2001). High ability students need less organized user interfaces than low ability students (Chalmers, 2003). User should be able to adopt conceptual structure of the system in order to automatize common routines, for example with shortcuts and recordable macros (Shneiderman, 1998; Nielsen, 1993; Tognazzini, 2003). User is at her best recognizing things, computer is much more effective in remembering things (Nielsen, 1993, 129). Human working memory is often referred to contain 7±2 memory "slots" (Miller, 1956), for example user interface is suitable for most users if only essential features are visible at first level. Less is more - more synchronous information available, longer it takes from the user to process it and make decisions (Nielsen, 1993). Minimizing complexity of the system is a good way to prepare for individual differences in information processing (Norman, 1988). Designers should aim at, from the very first moment of system development process, minimizing possible error situations (Nielsen 1990; 1993; 1994; Preece, Rogers & Sharp, 2002; Tognazzini, 2003; Shneiderman, 1987; 1998). Error messages should tell clearly, what is wrong and what are users next possible steps (if there is any) to correct the problem (Nielsen, 1990; 1993; 1994; Shneiderman, 1987).

Table 2. Dimensions of technical usability.

The pedagogical usability criteria Next we take a look at the pedagogical usability criteria. We first show where the previous study has reached, and after that, shortly sketch the major dimensions of our model of pedagogical usability. Table 2 presents previous research work that is related to the criteria of pedagogical usability (Reeves, 1994; Squires & Preece, 1996; Quinn, 1996; Albion, 1999; Squires & Preece, 1999; Horila, Nokelainen, Syvänen & Överlund, 2001). We consider Reeves’s paper (1994) as the first one to systematically analyze connections between usability and pedagogical issues. All except the study of Horila et al. (2003) concentrate on predictive evaluation, i.e. the evaluation of software prior to its use. In this paper we follow the study setting of Horila et al. (2003) and Jones et al. (1999) thus concentrating to develop usability criteria for interpretive evaluations, i.e. evaluating the users use of the software. Results of most of the earlier research work are not generalizable as empirical validations are missing or study settings are inadequately conducted.

Table 2. Summary of the previous research of pedagogical usability.

Pedagogical usability criteria applied in this study are as follows: 1. learner control, 2. learner activity, 3. cooperative learning, 4. goal orientation, 5. applicability, 6. effectiveness, 7. motivation, 8. valuation of previous knowledge, 9. flexibility and 10. feedback. Overall view to the dimensions of pedagogical usability is presented in Table 3. Dimension

Description

1. Learner control

Minimizing (working) memory load, as users have limited memory capacity, usually 7 +/- 2 units (Miller, 1956; Shneiderman, 1998, 355). Control of the technology should be taken away from the teachers and instructional designers and given to the learners (Jonassen, Myers & McKillop, 1996). Meaningful encoding (chunking), e.g. presenting learning material in meaningful units (Wilson & Meyers, 2000). Teacher role (facilitative/didactic) depends on underlying pedagogic assumptions (Reeves, 1994). Learning material should gain learners attention. Learners should feel that they own the goals of action and thus the results (Jonassen, Peck & Wilson, 1999). Constructivist view is based on social learning and knowledge sharing via collaborative tasks. Learners are able to discuss and negotiate about different approaches to the learning task (Jonassen, 1995). Tools might support asynchronous or synchronous social navigation (Mayes & Fowler, 1999; Kurhila, Miettinen, Nokelainen & Tirri, 2002). Instructivists emphasize few clearly defined goals, constructivist goals should also be clear, but set by the learners themselves (Wilson & Meyers, 2000). Task decomposition could be inductive or deductive. Authentic activities and contexts: examples should be taken from authentic situations results (Jonassen, Peck & Wilson, 1999). Transfer - learned knowledge or skills are useful in other contexts, too. Learning by doing (Wilson & Meyers, 2000). Human development should be considered in a way that the material is relevant for target population’s developmental stage (Wilson & Meyers, 2000). Scaffolding, supporting learners when it is most needed, for example in the zone of proximal development (Chalmers, 2003). Prompting and fading, support that changes due to learner skill level (Hannafin & Peck, 1988, 47). Added value for learning. Relevance of media elements (sound, animation, video). Motivation affects all learning (Ruohotie, 2002; Chalmers, 2003). Intrinsic (need for deep understanding) and extrinsic (need for high grades) motivation (Reeves, 1994; Ruohotie & Nokelainen, 2003). Prerequisites, what is needed to accomplish learning tasks. Meaningful encoding (elaboration), learner is encouraged to make use of her previous knowledge (Wilson & Meyers, 2000). Pretesting and diagnostics help to adapt learning material for different learners (Hannafin & Peck, 1988, 48; Wilson & Meyers, 2000). Task decomposition, small and flexible learning units (Leflore, 2000). Feedback is response sensitive and accurate (Hannafin & Peck, 1988, 47). Learner has feeling that there is a real dialogue between her and computer (Mayes & Fowler, 1999).

2. Learner activity

3. Cooperative learning

4. Goal orientation 5. Applicability

6. Added value 7. Motivation 8. Valuation of previous knowledge 9. Flexibility 10. Feedback

Table 3. Dimensions of pedagogical usability.

Preliminary empirical evaluation of the criteria A PMLQ (Pedagogically Meaningful Learning Questionnaire) research instrument was developed on the basis of forementioned technical and pedagogical usability criteria. The instrument contained 92 multiple-choice items. The 5-point scale ranged from 1 (totally disagree) to 5 (totally agree). The sixth response option was “Not applicable”. The PMLQ instrument has three parts: First part is about technical and pedagogical usability of the learning platform (or system) containing 43 items; the second is about technical usability of the learning material (24 items), and the third part measures pedagogical usability of the learning material (25 items). The propositions are clearly marked when measuring issues about system or contents.

Empirical measurements were carried out in October 2003 with 5th and 6th grade elementary school students (n=66) and their teachers (n=4). 24 of the children were boys and 42 were girls. Three teachers were male and one was female. Participants evaluated the OPIT learning system and four learning modules embedded into the system. Two of the modules were about mathematical topics (decimal numbers and fractions) and remaining two modules were about third foreign language of Finnish school children, English (singular vs. plural and knowledge test about British Isles). The design of this empirical test contained three stages: Firstly, each participant was profiled with a previously developed ACALQ (Abilities for Computer Assisted Learning Questionnaire) instrument that characterizes respondents by their self-rated motivational level, metacognitive preparedness and social abilities. This information is for future purposes in order to control individual differences in evaluations of learning materials. Secondly, participants filled out a PMLQ questionnaire for the OPIT platform. Shortly, after using each of the four modules, they filled out a similar usability questionnaire for each module. Thirdly, for each module, individual score and turnaround time was recorded. Preliminary results of this first version of the PMLQ instrument showed that full scale of 1 to 5 was in use. Empirical research evidence was investigated with Bayesian dependency modeling due to small sample size. The results supported the chosen dimensionality, although small sample size has its limitations with a large number of dimensions. Interview with the children and teachers revealed some deficiencies in wordings. Our next step is to collect more data with revised items.

Conclusions Major purpose of this study was to present both technical and pedagogical usability criteria for evaluation of digital learning material. We first defined elementary concepts of both technical and pedagogical usability criteria. Next we presented a more detailed view to the theoretical framework of the technical usability criteria. The components of technical usability discussed in this study were as follows: 1. accessibility, 2. learnability and memorability, 3. user control, 4. help, 5. graphical layout, 6. reliability, 7. consistency, 8. efficiency, 9. memory load and 10. errors. We also characterized the components of pedagogical usability: 1. learner control, 2. learner activity, 3. cooperative learning, 4. goal orientation, 5. applicability, 6. effectiveness, 7. motivation, 8. valuation of previous knowledge, 9. flexibility and 10. feedback. We developed a PMLQ (Pedagogically Meaningful Learning Questionnaire) research instrument on the basis of forementioned technical and pedagogical usability criteria. The instrument contained 92 multiple-choice items. Preliminary empirical evaluation of the criteria with the instrument showed that the data (5th and 6th grade elementary school children, n=66) supported the chosen dimensionality. A mobile learning criteria (Syvänen, Nokelainen, Ahonen & Turunen, 2003) will be added to the future version of the evaluation tool.

References Albion, P. (1999). Heuristic evaluation of educational multimedia: from theory to practice. 16th Annual conference of the Australasian Society for Computers in Learning in Tertiary Education, ASCILITE. Retrieved from: http://www.usq.edu.au/users/albion/papers/ascilite99.html [8.2.2002] Ambler, S. (2001). User interface Development Throughout the System Development Lifecycle. In Q. Chen (Ed.), HumanComputer Interaction: Issues and Challenges, pp. 11-28. Hershey: Idea Group Publishing. Chalmers, P. (2003). The role of cognitive theory in human -computer interface. Computers in Human Behavior, 19, 593607. Chin, J., Diehl, V., & Norman, K. (1988). Development of an Instrument Measuring User Satisfaction of the HumanComputer Interface. Proceedings of ACM CHI'88 Conference on Human Factors in Computing Systems, pp. 213-218. Hannafin, M., & Peck, K. (1988). The Design, Development and Evaluation of Instructional Software. New York: Macmillan. Henninger, S. (2000). A methodology and tools for applying context-specific usability guidelines to interface design. Interacting with computers, 12(2000), 225-243. Hiddink, G. (2001). ADILE: Architecture of a Database-Supported Learning Environment. Journal of Interactive Learning Research 12(2/3), 301-319.

Horila, M., Nokelainen, P., Syvänen, A., & Överlund, J. (2002). Pedagogical usability criteria, version 1.0. [In Finnish.] Digital Learning project. Häme Polytechnic and University of Tampere. ISBN 951-784-146-9. Jonassen, D. (1995). Supporting Communities of Learners with Technology: a Vision for Integrating Technology with Learning in Schools. Educational Technology July-August 1995, pp. 60-63. Jonassen, D., Myers, J., & McKillop, A. (1996). From Constructivism to Constructionism: Learning with Hypermedia/Multimedia Rather Than from It. In B. Wilson (Ed.), Constructivist Learning Environments, pp. 93-106. Englewood Cliffs, NJ: Educational Technology Publishers. Jonassen, D., Peck, K., & Wilson, B. (1999). Learning with Technology. A Constructivist Perspective. Upper Saddle River, NJ: Merrill. Jones, A., Scanlon, E., Tosunoglu, C., Morris, E., Ross, S., Butcher, P., & Greenberg, J. (1999). Contexts for evaluating educational software. Interacting with Computers, 11 (1999), 499-516. Keinonen, T. (1998). One-dimensional usability – Influence of usability on consumers’ product preference. Saarijärvi: Gummerus. Kirakowski, J. (1994). The Use of Questionnaire Methods for Usability Assessment. Retrieved from: http://www.ucc.ie/hfrg/questionnaires/sumi/sumipapp.html [13.9.2002] Kirakowski, J. (2003). Questionnaires in Usability Engineering. Retrieved from: http://www.ucc.ie/hfrg/resources/qfaq1.html [13.9.2003] Kurhila, J., Miettinen, M., Nokelainen, P., & Tirri, H. (2002). Use of Social Navigation Features in Collaborative ELearning. In Proceedings of the E-Learn 2002 Conference, pp. 1738-1741. Leflore, D. (2000). Theory Supporting Design Guidelines for Web-Based Instruction. In B. Abbey (Ed.), Instructional and Cognitive Impacts of Web-Based Education, pp. 102-117. Hershey: Idea Group Publishing. Lin, H., Choong, Y., & Salvendy, G. (1997). Proposed Index of Usability: A Method for Comparing the Relative Usability of Different Software Systems. Behaviour and Information Technology, 16(4/5), 267-278. Marcus, A. (1992). Graphic Design for Electronic Documents and User Interfaces. New York: ACM Press. Mautone, P., & Mayer, R. (2001). Signaling as a cognitive guide in multimedia learning. Journal of Educational Psychology, 93, 377-389. Mayes, J., & Fowler, C. (1999). Learning technology and usability: a framework for understanding courseware. Interacting with Computers, 11(1999), 485-497. Miller, G. (1956). The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information. The Psychological Review, 63, 81-97. Retrieved from http://www.well.com/user/smalin/miller.html [21.11.2003] Nielsen, J. (1990). Evaluating Hypertext Usability. In D. H. Jonassen & H. Mandl (Eds.), Designing Hypermedia for Learning, pp. 147-168. Berlin: Springer-Verlag. Nielsen, J. (1993). Usability Engineering. Boston: Academic press. Nielsen, J. (1994). Heuristic evaluation. In J. Nielsen and R. L. Mack (Eds.), Usability Inspection Methods. New York, NY: John Wiley & Sons. Nielsen, J., & Molich, R. (1990). Heuristic evaluation of user interfaces. Proceedings of the ACM CHI'90 Conference, (Seattle, 1-5 April), 249-256. Norman, D. (1982). Learning and Memory. San Francisco: Freeman. Norman, D. (1988). The psychology of everyday things. New York: Basic Books. Norman, K., Shneiderman, B., & Harper, B. (2003). QUIS: The Questionnaire for User Interaction Satisfaction, version 7. Retrieved from: http://www.cs.umd.edu/hcil/quis/ [18.11.2002] Perlman, G. (1998). Web-Based User Interface Evaluation with Questionnaires. Retrieved from: http://www.acm.org/~perlman/ [12.10.2001]

Preece, J., Rogers, Y., & Sharp, H. (2002). Interaction Design: Beyond Human-Computer Interaction. New York, NY: John Wiley & Sons. Quinn, C. (1996). Pragmatic Evaluation: Lessons from Usability. Retrieved from: http://www.ascilite.org.au/conferences/adelaide96/papers/18.html [8.2.2002] Reeves, T. C. (1994). Evaluating what really matters in computer-based education. In M. Wild, & D. Kirkpatrick (Eds.), Computer education: New Perspectives, (pp. 219-246). Perth, Australia: MASTEC. Rubin, J. (1994). Handbook of Usability Testing. How to plan, design, and conduct effective tests. New York: John Wiley & Sons. Ruohotie, P. (2002). Motivation and Self-regulation in Learning. In H. Niemi & P. Ruohotie (Eds.), Theoretical Understandings for Learning in the Virtual University, pp. 37-72. Saarijärvi: University of Tampere, Research Centre for Vocational Education. Ruohotie, P., & Nokelainen, P. (2003). Practical Considerations of Motivation and Computer-supported Collaborative Learning. In T. Varis, T. Utsumi, and W.R. Klemm (Eds.), Global Peace Through The Global University System, pp. 226236. Shneiderman, B. (1998). Designing the User Interface: Strategies for Effective Human-Computer Interaction. (3rd ed.). Menlo Park, CA: Addison Wesley. Soloway, E., Jackson, S., Klein, J., Quintana, C., Reed, J., Spitulnik, J., Stratford, S., Studer, S., Jul, S., Eng, J., & Scala, N. (1996). Case Studies of Learner-Centered Design. In R. Bilger, S. Guest & M. Tauber (Eds.), electronic proceedings of CHI 96. Retrieved from http://www.acm.org/sigchi/chi96/proceedings/papers/Soloway/es_txt.htm [13.6.2002] Squires D., & Preece, J. (1996). Usability and Learning: Evaluating the Potential of Educational Software. Computers and Education, 27(1), 15-22. Squires, D., & Preece, J. (1999). Predicting quality in educational software: Evaluating for learning, usability and the synergy between them. Interacting with Computers, 11(1999), 467-483. Syvänen, A., & Nokelainen, P. (2002). Theoretical framework of the evaluation criterion for pedagogically meaningful digital learning material. Häme polytechnic, University of Tampere. Syvänen, A., Nokelainen, P., Ahonen, M., & Turunen, H. (2003, August). Approaches to Assessing Mobile Learning Components. Paper presented at 10th Biennial Conference of the European Association for Research on Learning and Instruction. Padova, Italy. Tognazzini, B. (2003). First principles. Retrieved from: http://www.asktog.com/basics/firstPrinciples.html [4.11.2003] W3C. (1999). Web Content Accessibility Guidelines 1.0. W3C Recommendation 5 May 1999. Retrieved from: http://www.w3.org/TR/1999/WAI-WEBCONTENT-19990505/ [12.5.2003] Wilson, B., & Myers, K. (2000). Situated Cognition in Theoretical and Practical Context. In D: Jonassen & S. Land (Eds.), Theoretical Foundations of Learning Environments, pp. 57-88. Mahwah, NJ: Lawrence Erlbaum Associates.

Acknowledgements This study is part of Digital Learning 2 research project financed by National Technology Agency in Finland (http://www.tekes.fi/eng). The main goal of the project is to develop a tool to evaluate, both technically and pedagogically, digital learning materials. The project is carried out in collaboration with two Finnish research groups in Häme Polytechnic and University of Tampere. For further details visit: http://dll.hamk.fi/dl2/en/.