Peer assessment of language learning resources in ...

Int. J. Technology Enhanced Learning, Vol. 6, No. 4, 2014

Peer assessment of language learning resources in virtual learning environments with e-rubrics Elena Martín-Monje* Filologías Extranjeras y sus Lingüísticas, Universidad Nacional de Educación a Distancia (UNED), Despacho 2, Planta 0, Senda del Rey 7 - 28040 Madrid, Spain Email: [email protected] *Corresponding author

Esteban Vázquez-Cano Didáctica, Organización Escolar y Didácticas Especiales, Universidad Nacional de Educación a Distancia (UNED), Despacho 215., C/Juan del Rosal, 14 (28040) Madrid, Spain Email: [email protected]

Miguel Fernández Bilingual Education Program, Chicago State University, ED 215, 9501 South King Drive, Chicago, IL 60628-1598, USA Email: [email protected] Abstract: The aim of this paper is to analyse the usefulness of e-rubrics in the peer assessment of resources and materials for foreign language teaching and also in the assessment of language skills, taking into account the work produced by 99 students of the master’s degree in information and communication technologies applied to language teaching and processing, at the Spanish National University for Distance Learning (UNED), during the academic years 2012–2013 and 2013–2014. Following a methodological framework based on virtual, collaborative action-research, a qualitative analysis is applied with a twofold approach: firstly, through the use of text frequency techniques and secondly, through a reticular, category-based social network analysis thanks to UCINET and yED Graph Editor. Results indicate that the use of e-rubrics optimises self-assessment and the adequacy of assessment criteria. Rubrics foster usefulness in terms of format and connectivity in the audio-visual elements included in digital online resources. Furthermore, e-rubrics are deemed as significantly effective indicators of progress in oral and written production in foreign language learning. Keywords: e-rubrics; peer assessment; online discussion forums; foreign languages; digital resources; virtual learning environments; learning management systems; e-learning; higher education; distance education.

Copyright © 2014 Inderscience Enterprises Ltd.

321

322

E. Martín-Monje, E. Vázquez-Cano and M. Fernández Reference to this paper should be made as follows: Martín-Monje, E., Vázquez-Cano, E. and Fernández, M. (2014) ‘Peer assessment of language learning resources in virtual learning environments with e-rubrics’, Int. J. Technology Enhanced Learning, Vol. 6, No. 4, pp.321–342. Biographical notes: Elena Martín-Monje is a Lecturer at UNED (Spain), where she teaches mainly in the areas of English for Specific Purposes and Computer-Assisted Language Learning (CALL). She is a member of the ATLAS research group and her current fields of research are mobile learning and MOOCs (Massive, Open, Online Courses). Her publications include papers in indexed journals and book chapters both at national and international levels. Her research and teaching practice have received official recognition: a Prize for Doctoral Excellence for her PhD, First Prize for Educational Experiences at the 5th Educared Conference (Spain) and a University Excellence in Teaching Award. Esteban Vázquez-Cano is also a graduate in Romance Languages and English Studies (Universidad Complutense de Madrid). After several years teaching in secondary schools, he then became inspector for educational centres, moving on to higher education afterwards. Lecturer at the University of Castilla la Mancha and now UNED, he has published several books related to technology enhanced learning and also numerous papers in international journals. Miguel Fernández is an Associate Professor in the Bilingual Education Program at Chicago State University, where he teaches both Bilingual and ESL teacher training courses. He holds a PhD in English Philology from University of Granada, Spain, as well as two Master’s degrees: MA in English Language and Literature from University of Granada, Spain; and MA in Language Testing from Lancaster University, UK. His areas of interest include bilingual education, second language acquisition, and language testing.

1

Introduction

Assessment is probably the most complex and decisive task performed by teachers. It is complex because there are many intervening factors in the teaching and learning process, which are difficult to observe and value accurately (Trujillo, 2012): attitudes, motivation, cognitive and learning styles, communicative abilities, peer-to-peer and peer-teacher relationships, etc. Besides, it is decisive as a key point in learning and a necessary premise for the improvement of teaching and learning. Among the three types in which assessment is normally divided (diagnostic, formative and summative) the second one is probably the most interesting one, since it focuses on tracking students’ continuous progress and allows for the introduction of ad hoc changes in order to make the learning process as effective as possible. In the past few decades assessment has ceased to be exclusively a teacher’s task, thanks to the implementation of novel modes of formative assessment, in which peer assessment is actively encouraged. Faced with such a novelty, teachers and students alike have sometimes felt disoriented, lacking appropriate tools that would assist them in order to perform this formative assessment in a consistent, reliable way. Under such circumstances the introduction of scoring rubrics has proved to be a big advantage, since they provide specific criteria for the assessment of students’ work and help them equally

Peer assessment of language learning resources

323

while working on their task – they allow students to clearly identify the objectives – and also when assessing their peers’ work, enabling a more objective, effective and thorough performance. This paper analyses the role of electronic rubrics or e-rubrics in the context of distance learning, more specifically in the evaluation of digital materials for language learning. After a review of the relevant literature related to this topic (Beyreli and Ari, 2009; Hanrahan and Isaacs, 2010; Al-Jarf, 2011), the research hypothesis is stated as (the use of e-rubric enhances peer assessment and improves the evaluation of online digital materials for language learning). This hypothesis is researched according to the following objectives: 

To assess the potential of e-rubrics in the peer assessment of digital online materials created for foreign language teaching.



To analyse the usefulness of e-rubrics for the assessment of language skills.

2

Related work

Since the end of the last century there has been a trend in higher education directing assessment towards a student-centred approach, making a special effort to achieve effective feedback that allows students to make real progress in their learning paths (Boud and Falchikov, 2007; Fernández March, 2010). This evolution implies a conceptualisation of education completely different to traditional methodologies, clearly disassociating two concepts that have been often mixed up: assessment and grading (Fernández March, 2010; Ibarra and Rodríguez Gómez, 2010). In this context, new assessment tools have emerged, such scoring rubrics, particularly useful when performing formative assessment (Cebrián, 2009; Cebrián, 2011). Gottlieb (2006) uses the term ‘rubric’ to refer to any scoring guide with specific criteria that is used for interpreting students’ work and reporting classroom-based data on language proficiency and academic achievement. One of the main advantages of rubrics is that they make corrections more objective, making the rater’s task easier and helping students identify the aims of the task to be performed (Farr and Trumbull, 1997; Genesee and Upshur, 1996; Navarrete and Gustke, 1996; O’Malley and Pierce, 1996; Parkes, 2010). There are other benefits associated with the use of rubrics: they promote language expectations, help the rater to ascertain the criteria by which the student’s progress will be measured and they also allow for a qualitative description of the different levels of student’s achievement (Griffin, 2009). Furthermore, Andrade et al. (2008, 2009) acknowledge the fact that the use of rubrics allows students to become more involved in their learning, leading to better results: with the aid of a rubric a student can do checks and revise his/her work, improving thus its quality. As regard the teacher, the main advantage is the reliability it adds to the assessment process, both among different raters (inter-rater) and also within one rater throughout the time (intra-rater) (Mansilla and Duraisingh, 2009; Reddy, 2007). In line with this, Jonsson and Svingby (2007, p.141) establish that “the reliable scoring of performance assessments can be enhanced by the use of rubrics, especially if they are analytic, topicspecific, and complemented with exemplars and/or rater training”. The latter, rater training is as important as the design of a solid rubric, since it may have an impact on the results achieved.

324

E. Martín-Monje, E. Vázquez-Cano and M. Fernández

2.1 Electronic rubrics (e-rubrics) The use of Information and Communication Technologies (ICT) in education has become widespread at all levels, and both teachers and students are increasingly making the most of the technological advances in the classrooms, be it for the creation of resources or for daily teaching practice and students’ completion of tasks. Computers and other devices have become key tools in teaching, and assessment is becoming more and more part of this growing trend. As far as rubric use is concerned, ICT has been decisive for the development of e-rubrics. Nowadays, there are many resources available online which can help us design these scoring guides (see Table 1 further on for a listing of the main online tools for rubric creation). E-rubrics help make the rating process more dynamic and are useful when recording students’ results, as they help with the tracking of their progress. Wolfe et al. (2010) insist that rater training is faster and more effective when it is done online with these e-rubrics. Dornisch and Sabatini McLoughlin (2006, p.6) value the use of online tools for the design of e-rubrics, but advise on a careful procedure in order to avoid potential pitfalls. Online rubric resources can help busy educators to find or craft powerful rubrics, but they must be used thoughtfully so that inherent limitations in the examples and frameworks provided by the site are reduced or removed before they prove problematic to the users. Finally, some research evidences that the use of e-rubrics improves holistic assessment of written compositions (Attali et al., 2012), complementing the valuation done manually by the rater (Enright and Quinlan, 2010). Table 1 shows the free online tools that are most widely used for creating e-rubrics. Table 1

Free online tools for e-rubrics

Name

URL

Rubrician

http://www.rubrician.com/general.htm

My Teacher Tools

http://www.rubrics4teachers.com/

Discovery School

http://school.discovery.com/schrockguide/assess.html

Teach-nology

http://www.teach-nology.com/web_tools/rubrics/

Rubistar

http://www.rubistar.4teachers.org/index.php

Rubric Builder

http://www.landmark-project.com/classweb/tools/rubric_builder.php3

Tech4Learning

http://www.myt4l.com/index.php?v=pl&page_ac=view&type=tools&tool=r ubricmaker

2.2 Peer assessment and e-rubrics One of the aspects that encouraged in formative assessment is the one done among students themselves, also called peer-assessment. It is a traditional model, but with the help of virtual tools it allows to achieve professional competences both on the student’s and the teacher’s side, since it is a teaching task that is performed jointly by all those who

Peer assessment of language learning resources

325

involved in the teaching and learning process, achieving a deeper engagement in the assessment process in a collaborative and virtual context (Cebrián, 2011; Hanrahan and Isaacs, 2010; Orsmond et al., 1996; Vickerman, 2009). As far as the practical implementation is concerned, one of the most interesting examples in the context of Spanish Education is ‘Agora Virtual’, an online environment characterised by its flexibility (it can be used in campus-based teaching, online teaching and also blended learning contexts) and also by the use of formative assessment through e-portfolios and e-rubrics, in order to favour student peer assessment (Cebrián and Accino, 2009). Looking at foreign languages specifically, it is relevant to highlight the research performed by Hung (2012), which also includes the use of e-portfolios as an alternative technique for peer assessment and collaborative learning, all within a teachertraining programme. All the literature revised points in the same direction: both rubrics and e-rubrics favour peer assessment and generate a greater student involvement both in the teaching and learning process and in the assessment of the progress made (Falchikov and Goldfinch, 2000; Fernández March, 2010; Reddy and Andrade, 2010).

2.3 Evaluation of digital materials in foreign languages through e-rubrics It should be noted that there is not much literature available specifically related to the use of e-rubrics in foreign language teaching. Apart from Hung (2012), previously mentioned, the two most prominent examples can be found in the work of Erben et al. (2009) and Moya and O’Malley (1994), both referred to English as a second language and focusing on the assessment of language skills and also evaluation of digital materials, being the latter more geared towards the e-portfolios. With regard to the evaluation of digital materials, this process tends to concentrate on the teacher’s production of new resources for students, not on materials produced by students themselves (which is the main concern of this study). However, many of the criteria and guidelines provided can be easily transferable. Some important factors to be taken into account when creating digital materials for language teaching and learning are (Vázquez Cano et al., 2014): 

Pedagogic criteria: Adequacy of content & exercises to level/age of students; adequacy of methodology employed; variety, grade of difficulty, level of participation (students & teachers);



Technical criteria: General functionality of programme; integration of different mediums and materials.



Functional criteria: Whether the design of the programme takes into account flexibility needed in language teaching & learning (e.g. Possibility of recording) and also the possibility of integrating all skills, including listening and speaking.

2.4 Dimensions for the assessment of language skills with e-rubrics Within the four language skills that are traditionally considered – listening, speaking, reading and writing; it is writing that has raised more interest for the implementation of rubrics and e-rubrics in its assessment (Al-Jarf, 2011; Spence, 2010; Wilson, 2006). Holistic rubrics (with a uni-dimensional scale) are normally the ones chosen for assessing

326

E. Martín-Monje, E. Vázquez-Cano and M. Fernández

written production (Attali et al., 2012), and lately a variety of models has been developed for assisting the rater or even substituting them, as experimented with some software: software Criterion (Lim and Kahng, 2012), e-rater (Enright and Quinlan, 2010) or BETSY (Coniam, 2009). All three of them provide feedback to the student on the composition or essay submitted based on the pre-established criteria of a holistic rubric. Some examples of performance assessments are essays, interviews, problem-solving tasks, role-plays or group discussions. Performance assessments comprehend three characteristics: the performance of a sort of task, the task’s authenticity and the qualification of the rater. The questionnaires for measuring performance assessments are validated in order to contribute to measuring students’ abilities to respond to real-life language tasks, value students’ true language abilities, and reflect on how students will perform in future real-life language situations. Brown (2004, p.255) identifies some dimensions of performance assessment, which are common in all questionnaires and taken into account for the development of our study. 1

Students make a constructed response.

2

They engage in higher-order thinking with open-ended tasks.

3

Tasks are meaningful, engaging and authentic.

4

Tasks call for the integration of language skills.

5

Both processes and products are assessed.

6

The depth of a student’s mastery is emphasised over breadth.

Although this automation of the process has been criticised by some authors (see e.g. Han et al., 2008), the truth is that it means a significant development in the application of e-rubrics for the assessment of foreign language skills, providing a more transparent, objective system for the student and greater reliability as regard the performance of the rater.

3

Context and description of the research project

As previously stated, the methodological framework is based on virtual, collaborative action-research, as a means to promote the students’ generic competences in higher education, as stated by the European Higher Education Space (EHES) (Pool-Cibrian and Martínez-Guerrero, 2013; Zimmerman and Schunk, 2011). This methodology is based on positive interdependence, encouraging group responsibility and collaborative work (Kitsantas and Dabbagh, 2010; Linnenbrink and Pintrich, 2003), along with the development of continuous zones of proximal development language competencies (Vygotsky, 1987). The evidence provided by Gómez Lucas and Álvarez Teruel (2011) and Millis and Rhem (2010) suggests that employing collaborative work as a strategy in higher education contributes to the enhancement of learning, allowing the sharing of different opinions and points of view, increasing the value of one’s own perspective and facilitating the exchange with course mates, since it activates and directs learning towards a successful approach to communicative situations among peers.

Peer assessment of language learning resources Figure 1

Glogster created by a student (see online version for colours)

Source: http://www.glogster.com/role/espacios-virtuales-de-aprendizaje-l-garciapardo/g-6l44u04dhrh60ju48jdcua0

327

328 Figure 2

E. Martín-Monje, E. Vázquez-Cano and M. Fernández E-rubric created by students in pairs

CATEGORY 2

1,5

1

0,5

Content

Basic knowledge of the topic is included. The content seems to be good, but it is not dealt in depth.

Essential information about the topic is included, but there are gaps in the facts and important information.

The content is minimum and very simplified. It does not represent a summary of the topic.

Organisation The content is well organised in hierarchy, using titles and lists to group the related material.

Titles and lists are used in the organisation, but it presents a weak hierarchy of topics.

Most of the content is organised in a logical way, but lists and examples are not included.

The organisation is not clear and it lacks logic. It looks like a compilation of isolated data and images.

Audios

The audio files (music, sound and recordings) are original, have been correctly edited and are audible, which improves the quality and the content of the glog.

The audio files (music, sound and recordings) have good quality but they are not original. They provide additional information to the content of the glog.

Although some of the audio files (music, sound and recordings) are original, they do not have good quality and they do not provide anything to the content of the glog.

The audio files (music, sound and recordings) are not original, have no quality or do not work. They do not provide anything to the content of the glog.

Visual Material

Videos, photographs and additional images that support the ideas of the poster are used. They also ensure good comprehension.

The visual material supports the contents very well, although some images are not directly related to the topics, titles or subtitles.

The videos and photographs take up a lot of space and some do not provide anything new to the contents.

It is almost nonexistent or it is treated vaguely. All the content is only related to the text.

Design

The design is excellent. It captures the attention in the key points. Colours and forms help comprehension.

The design is very attractive and original, but it does support neither the contents nor the general outline.

The design is visually acceptable. Some disorganisation is present, as contents are not hierarchised.

The design is disorganised and chaotic. It makes comprehension difficult. Too much information with little visual attraction is given.

The topics are covered in depth. Clear and concise examples are given. The knowledge of the content is excellent.

Source: http://rubistar.4teachers.org/index.php?&screen=ShowRubric&rubric_ id=2273863

Peer assessment of language learning resources Figure 3

329

Results of the analysis and assessment done in pairs

SELF ASSESSMENT CONTENT 2.0 ORGANIZATION 1.5 AUDIOS 1.5 VISUAL MATERIAL 2.0 DESIGN 2.0 TOTAL 9.0/10.0 Example of score following the e-rubric In general, it can be seen as a good example of a glogster, especially because it builds a good framework in terms of contents, supported by the titles which help the student focus on the main ideas. The design is simple and the colors support the hierarchy of the material. It would not be excellent, because there is some confusion in the use of the arrows that indicate direction, development or generation of ideas/hierarchized concepts. Good visual resources, such as videos, comic strips, figures, forms and hyperlinks have been used. The contents of the comic strips, for instance, invite to perceive what can be a complex topic as something entertaining. It is important to remember that students can have different learning styles, and while some can be more visual, others can be auditory learners. For that reason, the materials have to be balanced. From my own perspective as graphic designer, I see the use of animated elements can be confusing, but I understand this will depend on the audience towards they are targeted. Like in my own glogster, I think the use of some resources (personal edited audios and videos) is missing. The best in the creation of rubrics is that they help us with self-assessment and find out what we would have been able to use or add to ours. This way we can see what elements are missing and which are correct and have been well perceived by the students. Now we need to use any resource that is available to catch the students’ attention and overcome the fear of being creative. This will help us become content producers.

All the activities related to action research were organised fully online in the Moodle-like virtual learning environment developed by the Spanish National University for Distance Learning (UNED), aLF, as devised in the course in which this research took place, ‘Creation and edition of printed and audiovisual materials’, within the master’s degree in ICT applied to language teaching and processing offered by this university. The aim of this postgraduate programme is to provide students with advanced competences and skills related to technology enhanced language teaching and processing. The vast majority of those who enrol are foreign language teachers wishing to update their professional skills and familiarise with the latest research in computer-assisted language learning. The course in which this study was conducted aim is to show students how to produce digital materials for language learning. It is common practice for them to try their own teaching jobs and often to conduct a small experiment that will be developed into their final master’s dissertation. This research project was developed in four stages, following the aims and dimensions of the research: Stage 1: Students’ individual production of a digital poster for foreign language teaching using the online tool ‘Glogster.Edu’ (see http://edu.glogster.com/). Stage 2: Students’ pair work. Elaboration of an e-rubric for the assessment of the quality of functionality of the posters created in stage 1, from a double perspective: format and content. Lecturers provided general information and an introduction guide with several examples for the development of e-rubrics as part of the subject’s contents. The tool to be used was ‘Rubistar’ (http://rubistar.4teachers.org/). Stage 3: Analysis and assessment of the quality and functionality of the digital material created (digital poster) using the e-rubric produced in stage 2. This activity was done by the same pairs of students who designed the different e-rubrics in the previous stages, assessing the digital poster created by a randomly chosen student.

330

E. Martín-Monje, E. Vázquez-Cano and M. Fernández

Stage 4: Once the three previous stages had finished, students were asked to reflect on the three dimensions of this research project: usefulness of e-rubrics for peer assessment, for the assessment of online digital materials and also for the assessment of foreign language skills. The purpose of this stage is to assess students’ perceptions about their own work and the project itself in order to provide feedback to both professors and peers. This phase was all done within the virtual learning environment, using online forums for active discussion. The three main questions put forward in the forum match the three research dimensions previously stated (see Figure 4). Figure 4

4

aLF-online environment subject forums and chats

Description of research objectives and cohort

This research is based on the following hypothesis: ‘The use of e-rubrics enhances peer assessment and improves the evaluation of digital online material intended for foreign language teaching’. This hypothesis is researched according to the following research questions: 

Do e-rubrics improve peer assessment of digital online materials (e.g. Glogster) created for foreign language teaching?



Are e-rubrics useful for the assessment of language skills?



Are e-rubrics useful for the assessment of audiovisual learning objects for language learning?

Peer assessment of language learning resources

331

Table 2 shows the cohort that took part in this research project. Table 2

Cohort

Variables Gender

Age

Items

F

%

Male

43

43.4

Female

56

56.5

22–25

22

22.2

25–30

26

26.2

30–35

36

36.3

35–40

10

10.1

+40

5

5.0

For the Social Network Analysis (SNA) study, 38 students participated in forums to discuss about the use of e-rubrics on language teaching and learning.

5

Methodology

The methodological approach has been twofold: on the one hand, a quantitative analysis and on the other hand, a qualitative one, through a reticular, category-based SNA. The authors developed and implemented a questionnaire called Scale of Factors that Foster Innovation with E-Rubrics (SFFI-ER). Apart from a series of descriptive general and academic data, such as age and gender, the questionnaire comprised 29 items, to which participants responded using a 5-point Likert-type assessment scale. The five response options are as follows: 1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, and 5 = totally agree. The students completing the questionnaire were asked to indicate the extent to which they agreed or disagreed with the different statements. The data provided in the questionnaire were analysed using version 19 of the SPSS statistical software package. The questionnaire was designed specifically for this research and was firstly validated by a pilot test technique among ten professors of the foreign language department and for this purpose was taking as reference the scales published in the Common European Framework of Reference for Languages: learning, teaching, assessment (Appendix, Council of Europe, 2011). Next, a factor analysis of principal components was conducted to determine the internal structure of the questionnaire. However, before carrying out the analysis, and as a prior statistical requisite that guarantees correct application, a series of other tests were performed: first, Bartlett’s test of sphericity was used to test the hypothesis that the correlation matrix obtained is not an identity matrix; in other words, that significant interrelations exist between items that justify the factor analysis. Secondly, the KMO (Kaiser–Mayer–Olkin) index, which measures sampling adequacy (data suitability) to carry out the factor analysis, was used. Finally, the reliability of the factors extracted from the questionnaire was analysed, both individually and together. In addition, an inferential analysis was conducted by comparing the means of the questionnaire factors in relation to sex and age. To this end, the student’s t-test was used in all the variables for independent samples.

332

E. Martín-Monje, E. Vázquez-Cano and M. Fernández

Secondly, the messages in forums and chats were analysed from the perspective of SNA, whose methodology provides a relational approach following the reticular morphology of social connections (Wasserman and Faust, 1994; Spandel, 2006; Knoke and Yang, 2008). It enables an understanding of the form and structure of the relationships established as a whole, something which is essential in order to reach an understanding of the underlying mechanisms in the students’ statements in educational forums, facilitating hidden interaction patterns. SNA is based on the premise that the structures formed by the relationships among different elements provide a more thorough explanation of the whole, the social environment and also each of the elements when their different traits are taken unitarily (Barabási, 2002; Borgatti et al., 2002; Castells and Monge, 2011; Caverlee et al., 2010). Therefore, the procedure of using Atlas-Ti and a reticular SNA generates more complete results in order to understand relationships and opinions in virtual learning environment. With that aim this SNA methodology has been applied to identify text units that may justify the reasons declared by students when reflecting on the potential benefits of e-rubrics for peer assessment, assessment of digital materials for foreign languages and language skills. To this end, the software UCINET 6 and the viewer yED Graph Editor 3.11.1 were utilised, with the aim of editing the graph and making it easier to understand. The matrix scheme used to generate the exponential graph has been the following:   1 Pr  Y  y     exp   A g A  y      A 

where (a) the summation is over all configurations A; (b) ηA is the parameter corresponding to the configuration A (and is non-zero only if all pairs of variables in A are assumed to be conditionally dependent); (c) g A  y    yij A yij is the network statistic corresponding to configuration A; gA(y) = 1 if the configuration is observed in the network y, and is 0 otherwise; (d) κ is a normalising quantity which ensures that equation (1) is a proper probability distribution. All exponential random graph models are form of equation (1), which describes a general probability distribution of graphs on n nodes. Dependence assumptions have the consequence of picking out different types of configurations as relevant to the model. Note from point (b) above, parameters are zero whenever variables in a configuration are conditionally independent of each other. In other words, the only configurations that are relevant to the model are those in which all possible ties in the configuration are mutually contingent on each other.

6

Results and discussion

The results obtained in the statistical tests applied to the questionnaire attest to its internal consistency and construct validity. The item-total correlation of the dimension was analysed to eliminate those items with a correlation coefficient of below 0.2. In addition, the reliability of the scale was analysed using the Cronbach’s Alpha Test (.821). Seven questionnaire questions were eliminated after pre-pilot test for low adequacy (KMO less than 0.6) as a result of low discriminatory power or low correlation with the dimension as a whole. Bartlett’s test of sphericity (p = .000) and the KMO sampling adequacy measure (.771) were found to be suitable when analysing the factorial structure of the scale using the Varimax with Kaiser normalisation method for the principal component analysis. A

Peer assessment of language learning resources

333

‘scree plot’ was conducted to determine how many factors to extract. The factors retained were those ones with an eigenvalue larger than 1 (Guttman–Kaiser rule). The rotated solution gives the factor loadings for each individual variable in the data set in order to establish the names of the different factors and their percent of variance. Table 3

Matrix of factors extracted by Varimax rotation and factor loadings of items Item

F1 (PA)

2 (MA)

3 (SA)

1

Do rubrics promote self-assessment?

.853

2

Do rubrics save time in the process of evaluation?

.852

3

Do rubrics promote greater objectivity in assessment criteria?

.851

4

Do rubrics allow the observation of communicative appropriateness?

.684

5

Do rubrics promote interaction in assessment?

.681

6

Do rubrics increase objectivity when assessing production and skills?

.699

7

Do rubrics enhance the assessment of content analysis?

.691

8

Are rubrics useful for the assessment of iconic items?

.863

9

Are rubrics useful for testing the appropriateness of images?

.858

10

Do rubrics favour the establishment of new learning connections?

.699

11

Are rubrics useful for testing the appropriateness of orthography?

.584

12

Are rubrics useful for analysing size, colour and position of digital elements?

.855

13

Are rubrics useful for analysing audiovisual elements?

.876

14

Are rubrics useful for the assessment of oral production?

15

Are rubrics useful for the analysis of coherence and cohesion of foreign language texts?

.819

16

Are rubrics useful for the evaluation of student performance?

.834

17

Are rubrics useful for the analysis of communicative appropriateness of linguistic register?

.765

18

Are rubrics useful for the assessment of syntax, lexis, and grammar?

.806

.

.691

19 Are rubrics useful for the analysis of the prosody?

.864

20

Are rubrics useful for the assessment of written production?

.865

21

Are rubrics useful for the evaluation of the quality of students’ response?

.756

% of variance explained = 75.44

23.07

24.36

28.01

Cronbach’s alpha α

.770

.775

.800

Notes:

Peer Assessment (PA); Material Assessment (MA); Skill Assessment (SA).

The principal component factor analysis identified three underlying factors in the questionnaire, with a total explained variance of 75.44%. The partial correlations (each multiplied by minus 1) are found in the anti-image correlation matrix. On the main

334

E. Martín-Monje, E. Vázquez-Cano and M. Fernández

diagonal of this matrix are the MSAs for the individual variables. All these factors, Peer Assessment (PA), Material Assessment (MA) and Skill Assessment (SA), account for representative eigenvalues. Each of the factors identified is described below.

6.1 Factor 1: e-rubrics and peer assessment for teaching and learning One of the most interesting aspects in the elaboration and use of e-rubrics for the improvement of peer assessment in teaching and learning is its applicability to the development of generic competences within the EHES, mainly those related to selfregulated learning and interpersonal competences (Dublin Descriptors, 2005), as well as those competences which are specific to university degrees and masters programmes at UNED, especially those that connect with competence areas such autonomous and selfregulated management of workload, management of communication and information processes and team work assuming different roles or functions. In all these contexts, peer assessment plays an essential role in the development of students and future professionals. Table 4 shows the frequencies in the most relevant categories referred to peer assessment. Table 4

E-rubrics and peer assessment in teaching and learning SA

D

N

A

TA

1.1

Rubrics promote self-assessment?

Items

4%

8%

8%

15%

65%

1.2

Do rubrics save time in the process of evaluation?

8%

9%

17%

12%

54%

1.3

Do rubrics promote greater objectivity in assessment criteria?

5%

9%

13%

10%

63%

1.4

Do rubrics allow the observation of communicative appropriateness?

7%

12%

33%

25%

23%

1.5

Do rubrics promote interaction in assessment?

10%

11%

23%

26%

30%

1.6

Rubrics increase objectivity when assessing production and skills?

8%

9%

32%

26%

25%

Notes:

SA (Strongly Disagree); D (Disagree); N (Neutral); A (Agree); TA (Totally Agree). Cronbach α = .770; 6 items; 4-point Likert scale (1–5); mean = 4.01; standard deviation = 0.46; N = 99.

The items that obtained a more positive valuation have been the enhancement of selfassessment, a greater objectivity in the application of assessment criteria and time management (1.1. = 80%; 1.2. = 66%; 1.3 = 73%, respectively). E-rubrics help students to develop more autonomy in the assessment process, favouring their awareness of the requirements stated in the assessment criteria and reducing the time devoted to the entire evaluation process. Besides, they are an ideal tool for collaborative work, since their design and application can be developed in pairs or reduced groups, improving thus the results obtained.

Peer assessment of language learning resources

335

6.2 Factor 2: e-rubrics and assessment of digital materials for foreign language teaching The assessment of digital online materials is a key aspect in which e-rubrics can provide substantial support. The number of online tools for the creation of digital materials is growing exponentially and very often there are no specific criteria that can help assess their quality and applicability in Education and foreign language learning. The production of e-rubrics assists in the evaluation of different aspects related to the format and interface of that digital online material, among others: design, interactivity, educational value, competences involved, etc. In addition, their use can lead to projects or documents that provide guidance about the correct selection of digital online materials. An example of this is the American programme QM, ‘Quality Matters’ (see www.qmprogram.org), devoted to evaluating the quality of virtual education through the use of e-rubrics (see http://www.qmprogram.org/rubric). Table 5 shows the frequencies in the most relevant categories referred the assessment of digital materials for foreign language teaching. Table 5

E-rubrics and assessment of digital materials for foreign language teaching Items

SA

D

N

A

TA

2.1

Do rubrics enhance the assessment of content analysis?

7%

10%

22%

35%

26%

2.2

Are rubrics useful for the assessment of iconic items?

7%

10%

26%

23%

34%

2.3

Are rubrics useful for testing the appropriateness of images?

5%

7%

21%

32%

35%

2.4

Do rubrics favour the establishment of new learning connections?

8%

11%

34%

25%

22%

2.5

Are rubrics useful for testing the appropriateness of orthography?

11%

12%

28%

36%

13%

2.6

Are rubrics useful for analysing size, colour and position of digital elements?

8%

10%

18%

37%

27%

2.7

Are rubrics useful for analysing audiovisual elements?

5%

9%

22%

36%

28%

Notes:

SA (Strongly Disagree); D (Disagree); N (Neutral); A (Agree); TA (Totally Agree), Cronbach α = .775; 7 items; 4-point Likert scale (1–5); mean = 3.91; standard deviation = 0.41; N = 99.

In this second factor, the three categories that can be highlighted are usefulness of erubrics for the assessment of audio and video elements, for the analysis of size, colour and position of the elements involved in web design and for the appraisal and appropriateness of the images used (2.7 = 64%; 2.6 = 64%; 2.3 = 67%, respectively). On many occasions, the design of online materials is not evaluated according to the aims with which it was outlined (didactic, business-oriented, etc.). The digital materials designed for foreign languages must be simple, interactive and effective, so that a potential ‘wow effect’ does not distract users and learners from the development of their language skills.

336

E. Martín-Monje, E. Vázquez-Cano and M. Fernández

6.3 Factor 3: e-rubrics and assessment of language skills One of the key elements in the evaluation of linguistic competences is the objectivity used in the evaluation of the skills acquired according to the levels of competence established in the official curricula and the European framework of language. The qualitative aspects related to the linguistic competence have always been difficult to measure and evaluate. In this sense, e-rubrics can be especially useful when categorising and assessing levels of competence attainment in auditory, oral, grammatical and discourse skills, providing greater objectivity in assessment and student feedback to improve performance. Table 6 shows the frequency of meaning of the most significant categories in the incidence of the e-rubrics in the assessment of linguistic skills. Table 6

E-rubrics and the assessment of linguistic skills Items

SA

D

N

A

3.1 Are rubrics useful for the assessment of oral production?

5%

3.2 Are rubrics useful for the analysis of coherence and cohesion?

9%

19%

38%

29%

8%

11%

34%

25%

22%

3.3 Are rubrics useful for the evaluation of student performance?

8%

10%

29%

32%

21%

3.4 Are rubrics useful for the analysis of communicative appropriateness of linguistic register?

6%

13%

33%

26%

22%

3.5 Are rubrics useful for the assessment of syntax, lexis and grammar?

5%

10%

17%

36%

32%

3.6 Are rubrics useful for the analysis of the prosody?

7%

11%

22%

33%

27%

3.7 Are rubrics useful for the assessment of written production?

3%

8%

20%

25%

44%

3.8 Are rubrics useful for the evaluation of the quality of students’ response?

6%

11%

35%

36%

12%

Notes:

TA

SA (Strongly Disagree); D (Disagree); N (Neutral); A (Agree); TA (Totally Agree). Cronbach α = .800; 8 items; 4-point Likert scale (1–5); mean = 4.03; standard deviation = 0.39; N = 99.

In this third factor, students gave their opinions and discussed how the e-rubrics could assist teachers as well as students to better assess the attainment of linguistic competences. The assessment of linguistic competences is often represented in terms of indicators of development and performance usually improves with the observation of the items related to production and linguistic comprehension. E-rubrics have been determined to be useful by students in improved objectivity in the evaluation of the items: written and oral production, error analysis and the correct usage of syntactical and lexical patterns (3.7 = 69%; 3.1 = 67%; 3.5 = 68%, respectively).

Peer assessment of language learning resources

337

6.4 Factor analysis through Social Networking Analysis (SNA) This quantitative study was complemented with the analysis of the perceptions, commentaries and opinions of the students in the three stages of the forums from SNA. We analysed the network of interactions obtained in aLF with the most significant connections. In order to do so, we edited the final UCINET network using the yED Graph Editor 3.11.1 (Figure 5) to make it more visual and comprehensible. We have highlighted in red colour the variables with high incidence into the three factors (variables with the highest intermediation and proximity degree ‘core nodes’). Brown, blue and green colours are used to highlight the following factors, respectively: factor 1: e-rubrics and peer assessment for teaching and learning; factor 2: e-rubrics and assessment of digital materials for foreign language teaching; and factor 3: e-rubrics and assessment of language skills. Figure 5

E-rubrics network (see online version for colours)

We can observe three main nodes in the parallelogram format corresponding to the three central threads of the forum. The average density of the three threads of the forum with the dichotomised matrix was .56 with .23 of standard deviation; which represents a high value for a sample of 38 students and an average rang of the network of 3.813; which indicates that the keyword is interrelated with the average of about 4. This a high value for a total of 28 nodes. Moreover, this result shows that more than two-thirds of the possible connections were present and that a high level of participation was obtained. We have analyse the centrality of the network to identify the most prominent aspects. To this end, we have referred to the analysis of the nodal value of intermediation and proximity (Table 7). In grey colour, we show the variables with the highest incidence in the model.

338 Table 7

E. Martín-Monje, E. Vázquez-Cano and M. Fernández Nodal value of intermediation and proximity in the ‘rubrics’ social network

Core nodes

Skill assessment

Material assessment

Peer assessment

Items

Degree Degr.

nDegr.

Betweeness Betw.

nBetw.

Closeness Cl.

nClos.

Self-assessment

19.0

54.550

65.5

32.5

77.0

41.10

Time management

17.0

52.275

63.0

30.5

71.2

38.96

Assessment criteria

18.0

51.250

61.5

25.5

70.3

34.60

Adequacy

11.0

28.750

35.0

19.3

56.0

35.51

Interaction

12.0

30.225

33.0

20.0

55.0

31.00

Skills

12.0

30.750

32.5

20.5

56.0

31.00

Content

10.5

28.750

32.5

19.0

51.0

29.81

Computer graphics

9.0

27.750

31.5

18.2

48.0

27.43

Images

15.0

35.500

55.0

29.4

74.0

38.45

Web links

17.0

51.500

60.0

31.3

74.0

39.61

Spelling & punctuation

11.0

28.500

32.0

18.0

55.0

31.37

Size, colour & location

16.0

50.500

59.0

30.2

72.0

39.12

Audio/video

18.0

52.500

61.5

32.0

76.0

41.10

Oral production

16.0

50.500

59.0

30.2

72.0

39.12

Coherence & cohesion

11.0

28.500

32.0

18.0

55.0

31.37

Student’s production

11.0

28.500

32.0

18.0

55.0

31.37

Register

11.0

28.500

32.0

18.0

55.0

31.37

Syntax & Lexis

15.0

35.500

55.0

29.4

74.0

38.45

Fluency

7.0

18.500

12.0

8.0

15.0

11.37

Written production

18.0

52.500

61.5

32.0

76.0

41.10

Quality of response

12.0

30.750

32.5

20.5

56.0

31.00

Facilitator

21.0

16.0

50.500

59.0

30.2

72.0

Motivation

22.0

18.0

52.500

61.5

32.0

76.0

Methodology

24.0

19.0

54.550

65.5

32.5

77.0

Reflection

20.0

15.0

35.500

55.0

29.4

74.0

Guidance

21.0

16.0

50.500

59.0

30.2

72.0

Interaction

21.0

16.0

50.500

59.0

30.2

72.0

Time management

23.0

18.0

52.500

61.5

32.0

76.0

The centrality shows the position of the concepts featured in the network (Spencer, 2003) and reveals a rather high result of 67% with a total number of 28 nodes. The maximum value (maximum number of connections of a node in the network) is 24 (methodology) forming nodes 22–28 (facilitator, motivation, methodology, reflection, guidance, interaction and time management), the nucleus of the graph, according to the concept of ‘k-cores’ (Seidman, 1983). The results show that the aspects with the highest normalised degree (Nrmdegree: percentage of connections that have a node above the total of the network) and the greatest nodal degree is found is the shaded items in Table 7. The results of the intermediation 58,131 provides us relevant information regarding the frequency with which a node appears in the shortest circuit (or geodesic), which connects

Peer assessment of language learning resources

339

the other two, in other words, showing when a topic can be an intermediary amongst others. We have reported in the facilitator group those nodes which have a higher degree of intermediation (≥20) and which reoccur in the three analysed dimensions. The results of the degree of proximity indicate that these bigger nodes are concentrated in those aspects which serve to interrelate the three dimensions: co-evaluation, didactic functionality and audiovisual format. Upon summarising the graph, we can conclude by pointing out that all of the secondary nodes are accessible through the connections of the graph and its three main nodes: co-evaluation, didactic functionality and audiovisual format. These results imply that e-rubrics have a great potential for language teaching and learning and foster the students’ participation with positive elements in the three analysed dimensions.

7

Conclusions

The use of e-rubrics is already quite widespread in assessing processes with the objective of obtaining more objective ratings in education. This research has taken a theoretical collaborative approach for the enhancement of assessment (especially peer assessment) of digital materials aimed at foreign language teaching (Cebrián, 2011; Orsmond et al., 1996; Hanrahan and Isaacs, 2010; Vickerman, 2009). With that purpose the authors of this paper created an activity based on the elaboration of e-rubrics to be used in the collaborative assessment of digital materials for foreign language teaching among some of the students of the master’s degree in ICT applied to language teaching and processing at UNED. Looking at the results obtained it can be concluded that the participants appreciated the usefulness and benefits of this collaborative creation of e-rubrics in three main dimensions: peer assessment, assessment of digital materials for foreign language teaching and assessment of language skills. With regard to peer assessment, the aspects most valued by students were its effectiveness in optimising self-evaluation, the improvement of proposed assessment criteria and the economisation of time that is implied in its applicability to peer assessment. As for the assessment of digital materials, students valued the improved observation of the functionality and adequacy of audiovisual elements, their format and connectivity. Finally, e-rubrics are perceived as very effective in the rating of language production and language skills, especially in speaking and writing and also in the learning of grammar and vocabulary. Thanks to SNA seven recurrent aspects have been identified in these three analysed dimensions, which provide a more general overview of the functionality of e-rubrics: they make the rating process easier, enhance student motivation with regard to selfevaluation, improve its application to other methodological processes, support reflection on learning, provide guidance both in the learning and the assessment practices, enable student interaction and collaboration and lead to saving time in assessment procedures. Additionally, the creation and application of e-rubrics by students helps the development of generic competences as stated by the EHES, mainly those related to self-regulated learning and interpersonal competences (Dublin Descriptors, 2005), as well as the specific competences for university degrees and masters programmes, especially those related to competence areas such autonomous and self-regulated management of

340

E. Martín-Monje, E. Vázquez-Cano and M. Fernández

workload, management of communication and information processes and team work assuming different roles or functions. In all these contexts, peer assessment plays an essential role in developing rounded-up students and future professionals.

7.1 Limitations and future research The main limitation of this study was the relatively small sample size. Readers also need to remember that we considered only clearly determined population and sampling settings; reference to other settings may have produced different results. The extent to which our findings can be generalised certainly requires further investigation.

Acknowledgements This research project has been developed as part of the VII Edition of Research Networks for Teaching Innovation, put forward by UNED, with the title ‘Development of e-rubrics for the enhancement of formative assessment in 2.0 foreign language teaching activities’.

References Al-Jarf, R. (2011) ‘Creating and sharing writing iRubrics’, Asian EFL Journal Professional Teaching Articles, Vol. 51, pp.41–62. Andrade, H.G., Du, Y. and Wang, X. (2008) ‘Putting rubrics to the test: the effect of a model, criteria, generation, and rubric referenced self assessment on elementary school students’ writing’, Educational Measurement: Issues and Practice, Vol. 27, No. 2, pp.3–13. Andrade, H.L., Wang, X., Du, Y. and Akawi, R.L. (2009) ‘Rubric-referenced self-assessment and self-efficacy for writing’, The Journal of Educational Research, Vol. 102, No. 4, pp.287–301. Attali, Y., Lewis, W. and Steier, M. (2012) ‘Scoring with the computer: alternative procedures for improving the reliability of holistic essay scoring’, Language Testing, Vol. 30, No. 1, pp.125–141. Barabási, A.L. (2002) Linked – The New Science of Networks, Perseus Publishing, Cambridge, MA. Beyreli, L. and Ari, G. (2009) ‘The use of analytic rubric in the assessment of writing performance: inter-rater concordance study’, Educational Sciences: Theory and Practice, Vol. 9, No. 1, pp.105–125. Borgatti, S., Everett, M. and Freeman, L. (2002) Ucinet for Windows: Software for Social Network Analysis, Analytic Technologies, Harvard, MA. Boud, D. and Falchikov, N. (Eds) (2007) Rethinking Assessment in Higher Education: Learning for the Longer Term, Routledge, London. Brown, D.H. (2004) Language Assessment: Principles and Classroom Practices, Longman, New York, NY. Castells, M. and Monge, P. (2011) ‘Network multidimensionality in the digital age: prologue to the special section’, International Journal of Communication, Vol. 5, pp.788–793. Caverlee, J., Liu, L. and Webb, S. (2010) ‘The SocialTrust framework for trusted social information management: architecture and algorithms’, Information Sciences, Vol. 180, pp.95–112.

Peer assessment of language learning resources

341

Cebrián, M. (2009) ‘Formative and peer-to-peer evaluation using a rubric tool’, in MéndezVilas, A., Solano Martín, A. and Mesa González, J.A. (Eds): Research, Reflections and Innovations in Integrating ICT in Education, Formatex, Badajoz, pp.60–64. Available online at: http:// www.formatex.org/micte2009/book/Index.pdf Cebrián, M. (2011) La evaluación formativa a través de las e-rúbricas y los e-portafolios, Enseñanza y aprendizaje en la Universidad, Universidad de Vigo. V Ciclo de conferencias. Available online at: http://vicadc.uvigo.es/opencms/export/sites/vicadc/vicadc_gl/documentos/ ciclos_conferencias/Material.ePor_eRubric.pdf Cebrián, M. and Accino, J. (2009) ‘Del ePortafolio a las tecnologías de la federación: La experiencia de Ágora Virtual®’, Jornadas Internacionales sobre docencia, investigación e innovación en la Universidad, Santiago de Compostela. Coniam, D. (2009) ‘Experimenting with a computer essay-scoring program based on ESL student writing scripts’, ReCALL: The Journal of EUROCALL, Vol. 21, No. 2, pp.259–279. Council of Europe (2011) Common European Framework of Reference for Learning, Teaching, Assessment, Council of Europe. Dornisch, M.M. and Sabatini McLoughlin, A. (2006) ‘Limitations of web-based rubric resources: addressing the challenges’, Practical Assessment, Research & Evaluation, Vol. 11, No. 3. Dublin Descriptors (2005) Shared “Dublin” descriptors for the Bachelor’s, Master’s and Doctoral awards, Draft 1.31 working document on JQI meeting in Dublin. Erben, T., Ban, R. and Castañeda, M. (2009) Teaching English Language Learners Through Technology, Routledge, New York. Enright, M. and Quinlan, T. (2010) ‘Complementing human judgment of essays written by English language learners with e-rater® scoring’, Language Testing, Vol. 27, No. 3, pp.317–334. Falchikov, N. and Goldfinch, J. (2000) ‘Student peer assessment in higher education: a metaanalysis comparing peer and teacher marks’, Review of Educational Research, Vol. 70, No. 3, pp.287–322. Farr, B.P. and Trumbull, E. (1997) Alternative Assessments for Diverse Classrooms, ChristopherGordon, Norwood. Fernández March, A. (2010) ‘La evaluación orientada al aprendizaje en un modelo de formación por competencias en la educación universitaria’, Revista de Docencia Universitaria, Vol. 8, No. 1, pp.11–34. Fetterman, D. (1984) Ethnography in Educational Evaluation, Sage, Beverly Hills. Genesee, F. and Upshur, J.A. (1996) Classroom-Based Evaluation in Second Language Education, Cambridge University Press, Cambridge. Gómez Lucas, M.C. and Álvarez Teruel, J.D. (eds) (2011) El trabajo colaborativo como indicador de calidad del Espacio Europeo de Educación Superior, Marfil, Alcoy. Gottlieb, M. (2006) Assessing English Language Learners: Bridges from language Proficiency to Academic Achievement, Corwin Press, Thousand Oaks. Griffin, M. (2009) ‘What is a rubric?’, Assessment Update, Vol. 21, No. 6, pp.4–13. Han, N-R., Chodorow, M. and Leacock, C. (2006) ‘Detecting errors in English article usage by non-native speakers’, Natural Language Engineering, Vol. 12, No. 2, pp.115–129. Hanrahan, S. and Isaacs, G. (2010) ‘Assessing self- and peer-assessment: the students’ views’, Higher Education Research & Development, Vol. 20, No. 1, pp.53–70. Hung, S.A. (2012) ‘A washback study on e-portfolio assessment in an English as a foreign langauge teacher preparation program’, Computer Assisted Language Learning, Vol. 25, No. 1, pp.21–36. Ibarra, M.S. and Rodríguez Gómez, G. (2010) ‘Aproximación al discurso dominante sobre la evaluación del aprendizaje en la universidad’, Revista de Educación, Vol. 351, pp.385–407. Jonsson, A. and Svingby, G. (2007) ‘The use of scoring rubrics: reliability, validity and educational consequences’, Educational Research Review, Vol. 2, No. 2, pp.130–144.

342

E. Martín-Monje, E. Vázquez-Cano and M. Fernández

Kitsantas, A. and Dabbagh, N. (2010) Learning to Learn with Integrative Learning Technologies (ILT): A Practical Guide for Academic Success, Information Age Publishing, Estados Unidos. Knoke, D. and Yang, S. (2008) Social Network Analysis, SAGE, USA. Lim, H. and Kahng, J. (2012) ‘Review of Criterion®’, Language Learning & Technology, Vol. 16, No. 2, pp.38–45. Linnenbrink, E. and Pintrich, P. (2003) ‘The role of self-efficacy beliefs in student engagement and learning in the classroom’, Reading & Writing Quarterly, Vol. 19, pp.119–137. Mansilla, V.B. and Duraisingh, E.D. (2009) ‘Targeted assessment rubric: an empirically grounded rubric for interdisciplinary writing’, The Journal of Higher Education, Vol. 80, No. 3, pp.334–353. Millis, B.J. and Rhem, J. (2010) Cooperative Learning in Higher Education: Across the Disciplines, across the Academy, Stylus Publishing, Virginia. Moya, S. and O’Malley, M. (1994) ‘A portfolio assessment model for ESL’, The Journal of Educational Issues of Language Minority Students, Vol. 13, pp.1–16. Navarrete, C. and Gustke, C. (1996) A Guide to Performance Assessment for Linguistically Diverse Students, New Mexico Highlands University, Albuquerque. O’Malley, J.M. and Pierce, L.V. (1996) Authentic Assessment for English Language Learners: Practical Approaches for Teachers, Addison-Wesley, New York. Orsmond, P., Merry, S. and Reiling, K. (1996) ‘The importance of marking criteria in the use of peer assessment’, Assessment & Evaluation in Higher Education, Vol. 21, No. 3, pp.239–250. Parkes, K.A. (2010) ‘Performance assessment: lessons from performers’, International Journal of Teaching and Learning in Higher Education, Vol. 22, No. 1, pp.98–106. Pool-Cibrian, W.J. and Martínez-Guerrero J.I. (2013) ‘Autoeficacia y uso de estrategias para el aprendizaje autorregulado en estudiantes universitarios’, Revista Electrónica de Investigación Educativa, Vol. 15, No. 3, pp.21–37. Available online at: http://redie.uabc.mx/vol15no3/ contenido-pool-mtnez.html Reddy, Y.M. (2007) ‘Effect of rubrics on enhancement of student learning’, Educate, Vol. 7, No. 1, pp.3–17. Reddy, Y.M. and Andrade, H. (2010) ‘A review of rubric use in higher education’, Assessment & Evaluation in Higher Education, Vol. 35, No. 4, pp.435–448. Spandel, V. (2006) ‘In defense of rubrics’, English Journal, Vol. 91, No. 1, pp.19–22. Spence, L.K. (2010) ‘Discerning writing assessment: insights into an analytical rubric’, Language Arts, Vol. 87, No. 5, pp.337–352. Trujillo, F. (2012) Evaluar para aprender. Available online at: http://www.educacontic.es/ blog/evaluar-para-aprender Vázquez Cano, E., Martín Monje, E. and Fernández Álvarez, M. (2014) ‘El rol de las e-rúbricas en la evaluación de materiales digitales para la enseñanza de lenguas en entornos virtuales de aprendizaje’, REDU. Revista de docencia universitaria, Vol. 12, No. 1, pp.135–157. Vickerman, P. (2009) ‘Student perspectives on formative peer assessment: an attempt to deepen learning?’, Assessment & Evaluation in Higher Education, Vol. 34, No. 2, pp.221–230. Vygotsky, L.S. (1987) ‘Thinking and speech (N. Minick, Trans.)’, in Rieber, R.W. and Carton, A.S. (Eds): The Collected Works of L. S. Vygotsky: Vol. 1. Problems of General Psychology, Plenum Press, New York, pp.39–285. (Original work published 1934). Wasserman, S. and Faust, K. (1994) Social Network Analysis: Methods and Applications, Cambridge University Press, Nueva York. Wilson, M. (2006) Rethinking Rubrics in Writing Assessment, Heinemann, Portsmouth, NH. Wolfe, E.W., Matthews, S. and Vickers, D. (2010) ‘The effectiveness and efficiency of distributed online, regional online, and regional face-to-face training for writing assessment raters’, The Journal of Technology, Learning, and Assessment, Vol. 10, No. 1, pp.1–21. Zimmerman, B.J. and Schunk, D. (2011) ‘Self-regulated learning and performance an introduction and an overview’, in Zimmerman, B.J. and Schunk, D. (Eds): Handbook of Self-Regulation of Learning and Performance, Routledge.