vGallery: web spaces for collaboration and assessment ... - CiteSeerX

vGallery: web spaces for collaboration and assessment Rob Woodbury, Ian Roberts and Susan Shannon School of Architecture, Landscape Architecture and Urban Design Adelaide University Adelaide South Australia 5005 Rob Woodbury As an Associate Professor in the School of Architecture, Landscape Architecture and Urban Design at Adelaide University, I currently lead two CUTSD funded projects in design education. I lead the vGallery development team, which comprises the three co-authors of this paper. Ian Roberts I have a long-standing interest in applications of IT to enhance learning. While I am currently providing maintenance and support to Adelaide University Online facilities, I am also contributing to the design and development of a number of e-Learning initiatives in the School of Architecture, Landscape Architecture and Urban Design. Susan Shannon With a background as a practising architect, university teacher, and assessment researcher, I am currently evaluating an Organisational CUTSD Grant (Adelaide/Deakin) which considers how learning changes, particularly in relation to learning from assessment, when computers are introduced.

Key words online assessment, self-assessment, peer assessment, online evaluations, collaboration

Abstract Assessment possibilities at Adelaide University have been expanded for formative and summative feedback, peer assessment and self-assessment through the mechanism of the virtual Gallery (vGallery). The vGallery is a tool that uses the WWW to support collaboration and is available to staff and students at the University. In establishing virtual spaces within vGallery, teachers, acting as Curators, make decisions about the dimensions of assessment and/or feedback provided to students. The Curator can configure the vGallery to accommodate peer and self-assessment or feedback. Options include one or more of the following: simple numeric scoring; keyword grading; quantitative criteria-based scaling and free text feedback -all configurable by the Curator. The criteria-based scores are automatically totalled by the vGallery to provide an assessment score that reflects the Curator's weightings. Irrespective of the level of privacy applied to Exhibits by the Curator, assessment and other critiques remain private to the Exhibitor, Critic and Curator. Furthermore, Curators have the opportunity to reverse the critiqueing role by providing opportunities for feedback and evaluation of vGalleries, and other aspects of teaching and learning in their subjects. While Curators can create and maintain access control lists manually, vGallery also interfaces with the University's Student Information System to allow current class enrolment lists to be applied automatically. In a Semester 2, 2000 pilot study, the vGallery assessment regime has been evaluated by students in comparison with traditional criteria-based assessment. Their requests for more personalised feedback from the teacher has resulted in further enhancements to the vGallery.

Introduction vGallery is an initiative to deliver a web-based tool to Adelaide University staff and students to enhance collaborative learning and directly to support the provision of efficient and timely assessment (Roberts &

Woodbury, 2000). vGallery aids interactions amongst groups of learners (students and staff) based around student work and critiques thereof, by allowing Adelaide University staff to create virtual spaces for collaboration - taking the term ‘collaboration’ to cover a range of meanings from the private presentation of a piece of student work for formal assessment to the posting of work for general viewing and discussion (online or face to face) (Woodbury, Roberts, & Shannon, 2000). At the time of writing vGallery has been used by 15 staff (as Curators) and 250 students (as Exhibitors) across all major Academic areas and contains over 1500 Exhibits in about 50 Galleries. The contents of the vGallery comprises: •

Galleries and related information/resources -representing virtual web spaces in which users may place and view work. Galleries possess functionality that restricts who may submit work and when they may do so. Galleries are the source of Exhibit metadata that, amongst other things, determines who may view, respond to or critique the work.

•

Exhibits -representing the bulk of the content of the vGallery and may include material in any digital form ranging from simple text, WWW sites, animations, etc.

•

Critiques (including evaluation/survey responses)

Irrespective of their roles, all vGallery participants interact using standard web browsers, as, in the development of vGallery, the designers and developers have been motivated by the belief that the use of ubiquitous technology eases of uptake of new tools (Mullins, Hopkins, & Roberts, 1999).

Assessment and vGallery While seeking to support collaboration in its broadest sense, recent vGallery development has focussed on supporting critiqueing and, more specifically, various modes of assessment. Associated as they are with higher quality outcomes for students (Prosser & Trigwell, 1999b, p11), how can assessment strategies contribute to promoting deep approaches to learning? The assessment strategies available through the vGallery have been developed upon this conception of the relationship between learning and teaching (Shannon, 2000). John Biggs identifies ‘teaching and assessing in a way that encourages a positive working atmosphere, so that students can make mistakes and learn from them’, as one of four factors relevant to assessment that encourages a deep approach to learning (Biggs, 1999, pp16-17). The assessment strategies available through the vGallery have been developed upon this principle through the: •

Maintenance of privacy between student/Exhibitor and teacher/Critic

•

Enhancement of peer collaboration

•

Enhancement of self-assessment opportunities

•

Support for formative assessment

vGallery modes of assessment The vGallery assessment modes available provide for many different types of feedback. Assessment options for both formative and summative purposes, include traditional mastery feedback from the teacher to the student, self-assessment and peer-assessment. Provision is also made for evaluation of learning modes as the vGallery can be configured with a student evaluation of teaching (SET) capacity. Currently available of assessment methods are: •

Simple numeric score -trialed variously as a predicted score (as entered by the teacher), a transferred score, a desired score in student self-assessment and a partial score, for example carried over from performance-based assessment.

•

Keyword grading, or Key phrase feedback -trialed as a simple means for the teacher to automatically respond through words to frequently occurring assessment scenarios, and a method of comparing the teacher’s view of the project in its various assessment dimensions, with the student’s self-assessment.

•

Quantitative criteria-based scoring -trialed as the principal means of providing detailed criteria-based formative and summative assessment feedback on complex submissions. The vGallery assists the teacher by compiling weighted scores and displaying them as a single score reflecting multi-variate dimensions of assessment.

•

Free text feedback - trialed as a means for the teacher to provide brief personalised feedback to students, a means for students to comment on their process of self-assessment and recording for student and teacher as yet unmarked areas of assessment.

The design of vGallery does not assume that all feedback and assessment will flow from the teacher. Valuing the role that self-assessment can play in the development of autonomous learners, (Clifford, 1999) v-Gallery provides teachers with the ability to facilitate self-assessment along with teacher-based assessment.

Application of assessment principles through the vGallery In the displayed first year work (from a core Semester 2, 2000, Level 1 Bachelor of Design Studies subject 1 Design and Form 1B) Kwan Jin Tseng shows a fit out for a nightclub. His peer critic is Sally Murphy.

Under Ground by Kwan Jin Tseng Peer critic: Sally Murphy Sample assessment

Keyword assessment

1

Teacher

Self

Peer

Visual Communication:

Visual Communication:

Visual Communication:

The names of students have been changed to preserve the confidentiality of critiques to work in the vGallery.

Free text assessment

Excellent work: images are superb

Excellent work: images are superb

Excellent work: images are superb

Narrative Process Communication: Beautifully resolved and communicated.

Narrative Process Communication: Not enough detail to adequately show what you've done

Narrative Process Communication: Not enough detail to adequately show what you've done

Object qualities of nightclub: superior

Object qualities of nightclub: superior

The nightclub has good experiential qualities, but lacks a lot of detail to show how each space is utilies to the fullest. For example, the dance floor and the stage and screen have not been created.

I admire Jin's ability to take on such a hard task, and furthermore to do it successfully. The strong theme of mystery and deconstruction certainly shine through, I felt very spaced out. The form making ideas werre superb and the cad skins displayed very high. I was very engaged during the presentation by the design, however, the talk could have been planned and to the point. Brilliant work Jin.

Well constructed attempt to integrate narrative and form-making information. Perhaps a further improvement would be to show the wys in which you have engaged the form-making strategies as a list underscoring the very small titles, the underground theme is strong, people exceptionally well modelled, lighting moody. Raconteur role 2.5 /3 Presenter role; presentation too long; 6/7

Likert dimensions Nightclub design (20%) -engaging boring (0) - exciting (7) Form-making strategies (20%) -several form-making strategies: few (0) - many (7) -relationship to night club function: weak (0) - strong -relationship to each other: weak (0) - strong Rich Development of ideas (10%) -process narrative: scant (0) - rich (7) -reinterpretation of ideas elsewhere in composition: weakly (0) - strongly CAD Skills (20%) -FormZ Model, object

6

6

7

6

5

5

5

7

6

5

7

6

5

7

5

5

7

6

6

5

7

location, layer structure, geometric cleverness, wise use of detail: weak (0) - skilled (7) Verbal and visual Presentation (20%) -Engaging, polished, Pointed: unrehearsed (0) polished (7) Total (weighted)

6

5

4

81%

84%

83%

Teacher's assessment By teasing out the dimensions of assessment, and declaring a weighting scheme, the teacher, as a vGallery Curator, sets an assessment frame of reference, thereby making transparent her thinking and the requirements and priorities of the task explicit. In working through selection of key phrases, writing a free text response as a personal correspondence with the student, and applying scoring to the quantitative criteriabased scores she demonstrates the process of mastery judgement over all the dimensions of judgement relevant to the project. Unlike traditional criteria based assessment, vGallery quantitative scoring aims to be sufficiently atomised to value different dimensions of assessment, without losing the essential holistic focus. The holistic is valued and discussed best in the free text response, whereas when students want to know where to direct their effort to improve their assessment outcomes, they can refer easily to their quantitative scoring on all dimensions of assessment. Where self-assessment is required, students cannot view the teacher's or peers' critiques until they have submitted their own. This is beneficial as the teacher can then view student's self-judgement as qualitative information about learning unaffected by the teacher's assessment. Self assessment vGallery provides the teacher with the opportunity to collect self-assessment of student work. This self assessment is performed using the same assessment framework (dimensions and weightings) created for the Teacher's assessment and models a constructive and reflective approach. Where self-assessment is enabled, students are obliged to submit their own assessment before seeing those submitted by teachers or peers. Once the self-assessment is complete, a student is able to view and compare the various critiques. Further vGallery developments would allow students to select their own dimensions for assessment, and judge their work against these criteria. Learning contracts (Anderson, Boud, & Sampson, 1994) could be enhanced and enabled through this mechanism, for students would be obliged to post their criteria for assessment when the vGallery is created. Already students comment that it is helpful to see the criteria for assessment, and their relative weighting on the vGallery during the assignment. Peer assessment In addition to teacher and self-assessment, vGallery supports assessment from the third important source of criticism viz the student's peers. The mechanism of a peer reviewing the piece of assessment work on the same criteria as the self-assessment, and teacher assessment, enables the Exhibitor to place their selfjudgement in a rich environment where they can see self-judgement with mastery judgement and a peer who has also completed the assignment. Once again, if peer and self-assessment is enabled, a student must supply a self-assessment before viewing peer assessments. vGallery greatly enhances the collaboration potentials for these multiple modes of assessment.

Roles in learning In the Nightclub fit out assignment, the four specific roles within the vGallery are differentiated, and their different contributions to Jin’s learning discussed.

Curator Rob Woodbury, and Susan Shannon performed the Curator role within the show-cased vGallery. The Curator role is the most powerful in the vGallery and is restricted to Adelaide University staff. Curators can create and configure Galleries and retain editorial rights over contents of their Galleries. Within their Galleries, Curators are obliged to exercise editorial responsibilities over their contents and can: •

Apply open and closing date times for the posting of Exhibits

•

Indicate who may possess the Curator, Exhibitor and Critic roles with access to current subject enrolment information from the Student Information System

•

Indicate specific requirements of Exhibits and whether Exhibitors may submit more than one Exhibit

•

Specify the type of Critiques that will be used within their Gallery including: Standard terms assessment or feedback terms and assessment criteria and weightings.

•

Determine the default level of privacy of Exhibits

Exhibitor Kwan Jin Tseng, (a pseudonym) who is the Exhibitor, is an Adelaide University student. Exhibitors may be staff or students or other users registered in the system directory. Within the Gallery rules as determined by the Curator, Exhibitors: •

May submit their work for general viewing or private inspection.

•

Are identified as authors of the work

•

Are required to acknowledge the University’s rules on plagiarism as they submit their work.

•

Retain the right to change or remove their work while the Gallery is open unless a Critique or other response to the item exists.

•

May view Critiques to their work.

Critic The Critics in this vGallery are the teacher, Susan Shannon, and the Exhibitor, Kwan Jin Tseng, as well as his class-mate Sally Murphy (a pseudonym). Critics may view content in a Gallery and provide some form of critique, feedback or assessment of that content. Based on the Curator's requirements, these critiques might serve purposes ranging from formal assessment to peer feedback and, in keeping with standard assessment practice, remain confidential to the Exhibitor, Critic and Curator. Reporting facilities exist in vGallery that allow the Curator to view and export sets of critiques associated with particular Galleries. After James submits his critique in the form of self-assessment, he can view Sally’s peer assessment, and the teacher’s assessment. In this project there is no summative component for self, or peer assessment. However, through the mechanism of preference of “viewer access ordering”, other motivation exists to complete self-assessments. As the students are paired with their self-selected critic class-mate, peer pressure to complete peer assessments exists.

Viewer Currently Viewers in the Gallery have no role beyond viewing work to which they have access. Based on the Curator’s requirements for a gallery, anonymous viewers may have access to some or all Exhibits in a Gallery. Alternatively tighter restrictions might reduce access to the Exhibitors, Attributed exhibitors or the Curator alone.

Evaluations The vGallery was also utilised in Semester 2, 2001 as an on-line evaluation tool. Students anonymously responded to questions about their learning and teaching posed by the teacher. The show-cased evaluation is of students enrolled in the first year class utilising vGallery as their: •

Submission tool,

•

Display gallery,

•

Formative assessment and critique feedback mechanism,

•

Informal peer assessment and critique feedback mechanism,

•

Self-evaluation feedback for themselves, and for the teacher,

•

Summative assessment mechanism.

The evaluation questions posed related to encouraging self-reflection about the different information to be derived from looking at the teacher's summative assessment, and the student's own self-assessment. The context for assessment evaluation related to a design for a Playground for 6-10 year old children.

Playground by Sally Murphy

Student & Teacher assessment An example of the assessment feedback from the teacher, and self-assessment is given for this assignment.

Keyword assessment Communication

Animation Writing Essay Textual critique

Teacher assessment

Student self-assessment

Excellent work: images are superb Beautifully resolved and communicated enhances understandings Concise and precise text : well done. The text brings it all together Sally, and until I read it I didn't really follow the themes or the transition space which is weak in form-making terms. Magic

Excellent work, images are superb Beautifully resolved and communicated enhances understandings Concise and precise text : well done; word count? I was overall happy with my design. However, the transition site was weak in form making ideas, and needed more refinement.

modelling and shadows lift your project. 3rd overall image very helpful to understanding. Much better animation in LATE gallery no jerks now. Excellent Form-Z model, Space ship solid and void form is superb. Likert scoring Form-making (20%) Application of ideas: poor (0) - rich (7) Relationship between 2 Form-making ideas -dialogue between 2 ideas: unrelated (0) - related (7) -from a child's perspective: adult (0) - child (7) Essay Explanation of Composition (20%) -titles reflect your compositional strategy: poorly (0) - strongly (7) -compositional work-in-progress thinking: unreflective (0) - reflective (7) -reflection of view of own work expressed in essay writing: (6) |||||| weakly (0) - strongly (7) Rich Development of ideas (20%) -3 intermediate images audit trail: unaudited (0) - audited (7) -relationship to finished composition: undisclosive (0) - disclosive (7) CAD Skills (20%) FormZ Model -object location -layer structure -geometric cleverness: poor (0) - clever (7) Total (weighted)

5

5

5

5

6

7

6

6

6

6

6

6

6

5

6

5

6

6

81%

80%

Student evaluations Question: 1

What information did you gain from the teacher's assessment? Student a

At first it seems like just snother assessment telling you what the teacher liked or didn't like, but really the likert and vaire methods of response are much more in depth that you would get from an 'olden times' (paper) assessment, where a teacher probably doesn't really have time to care about such a responsive response!

Student b

Quite a lot. My main composition was on the plan so it was lost on the viewer. Susan did write more than what fit on the screen so I missed a bit of info that could have been helpful.

Student c

That I should be clearer in my model than in my text about the form-making ideas- very important!!

Question: 2

Question: 3

Student d

It lets you see where your marks have come from, and how well you did in the various assesment areas.

Student e

The information gained from the assessment was very clear. The assessment criteria was outlined and I was able to also give myself an evaluation. The assessment was very useful, particulary for future projects, because it covered so many facets of the design task and the experiences created within each project.

Student f

From the teacher’s assessment, I can gain more information on the weakness that I have and the things taht I am good in.

Student g

Clear information on how I went for each of the outlined assessment criteria. It’s great to get detailed information on where I can improve rather than a single mark which isn’t very helpful.

What understandings did you gain from completing your own self-assessment for the project? Student a

I was able to elaborate on what I thought about my own work in simpler terms and therefore understand what I would need to focus on in further work.

Student b

That I didn't really know how to mark my own work. The things that I liked, Susan did not like and vice versa.

Student c

I picked out the things myself that I could have improved upon. I didn't see those things when I was making the model!

Student d

It let me compare the mark that I thought I should have got to the one allocated to me. This put me in the markers place and a gained a greater understanding of the process that they undertake.

Student e

I now know how it feels to assess work!!! It gives you a true reflection on work by having categories only directly concerned with the task and no 'bonus' areas. by completing the self-assessment, I was able to give more energy into the assessment dimensions, and realise the true purpose of the assignment.

Student f

I know what are the things we are assessed own and try to improve next time.

Student g

I did not complete the self-assessment.

What links can you see between the teacher's assessment, and your self-assessment? Student a

Very close in comments chosen from the selectable comments. I like this because it means that ideas on my work can be compared in a straightforward matter.

Student b

We came up with pretty much the same mark.

Student c

Susan is more indepth than I am, but we more or less thought the same things were lacking.

Student d

Most links were very similar if not the same. This showed me that in many areas I was approaching the assisnment in the correct way.

Student e

Both assessments resulted in similar outcomes. I believe the marking of the form Z model was difficult on our behalf, not knowing how a 'well sructured model' show appear.

Question: 4

Question: 5

Student f

I can’t really see any links between my own assessment and the teacher’s assessment, because different people have different views.

Student g

-

What other types of assessment would be helpful to you? Student a

I found the feedback quite adequate, especially in comparison to other subjects.

Student b

I like it when Susan talks to us and tells us face to face what was good and what could have been done better.

Student c

One on one assessment or dialogue between Susan/Rob and I.

Student d

I believe that oral assesment would be helpful. Feedback on your work or maybe taking examples from the class and discussing it during tutorial times.

Student e

i think more work in progress may be of some assistance, but I think such a broad assessment criteria gives an accurate reflection of the task.

Student f

I think that the assessment we have now is enough.

Student g

I find preliminary assessment very helpful. Perhaps more time spent on this would be good.

Have you any other comments on this on-line assessment process? Student a

efficient and I find it personal, even though some people may not.

Student b

no!

Student c

Nope.

Student d

It gives you easier access to your mark than if it was only on a notice board.

Student e

Reasonably simple to use and navigate through, but access and privacy is a constraint within such a tool.

Student f

–

Student g

One problem with the on-line assessment critique was that comments by the teacher were cut off in mid-sentence- perhaps by a word limit? We don’t get to see how we compare with the rest of the class, which can be seen on lists.

Discussion Responses a-e are quite positive in their reflections on the opportunities offered through on-line assessment, whereas responses f and g are not wholly positive. These and other students’ evaluations raise issues of familiarity, legibility, subjectivity, learning goals and the purpose of self-assessment and requests for competitive ranking. Our prior evaluations of students enrolled in the first year have already led to changes that enhance the flexibility and address of the vGallery. Current evaluations have also indicated further opportunities for vGallery design improvement. It is possible that some of the outlined evaluation issues spring from an inadequate introduction to the vGallery and explanation of the assessment strategies being applied. We propose therefore to have a more formal vGallery induction process next year for students. We also propose to encourage students from the outset to self-assess on the same criteria as the teacher intends to use, thereby increasing reflection (Brew, 1999) about the vGallery assessment process and the link between the assignment, their exhibit and their expected and actual assessment outcomes. Furthermore, as a direct comparison between the available

2

assessment modes is one mechanism for qualitatively discriminating between learning outcomes, familiarity with, and emphasis on, student self-assessment should be fore-grounded. Prosser and Trigwell state that ‘if the assessment is geared to testing understanding, student responses can be used to determine qualitative differences’ (Prosser & Trigwell, 1999a, p128). vGallery performs well as a student self assessment tool designed to elicit qualitative information about approaches to learning. It succeeds through the comparisons available of simple numeric score, keyword grading, and quantitative criteria-based scoring, and reflection on the assessment task in the free text box. This evaluation may further the teacher’s knowledge about the students’ perception of the online learning environment as a precursor to more closely aligning online teaching and assessment with eliciting deep approaches to learning.

Conclusion The evaluation and development of the vGallery as a sophisticated web-based means of collaborating and assessment is on going. vGallery has major potential to harness the power of the computer to enable the learner to access a wide range of assessment feedback, from teacher, peers and self. This feedback remains confidential, is tightly linked to the students work and the parameters within which it is given are in the hands of the Curator. The potential to provide a more personal vGallery feedback style is being explored as are options for meeting demands for more assessment comparison with classmates. While vGallery can be configured to provide extremely detailed feedback, some students believe that it is still deficient in this respect, and their evaluations reveal a request “for more personalised feedback”. The authors expect to explore this matter in the future.

References Anderson, G., Boud, D., & Sampson, J. (1994). The effective use of learning contracts: A guide for teaching staff in higher education . Sydney: School of Adult and Language Education, University of Technology, Sydney Biggs, J. (1999). Teaching for Quality Learning at University. Buckingham: The Society for Research into Higher Education Brew, A. (1999). Towards Autonomous Assessment: Using Self-Assessment and Peer Assessment. In S. Brown & A. Glasner (Eds.), Assessment Matters in Higher Education: Choosing and Using Diverse Approaches (pp. 159, 71). Buckingham, UK: The Society for Research into Higher Education Clifford, V. A. (1999). The Development of Autonomous Learners in a University Setting. Higher Education Research and Development, 18(1), 115-28 Mullins, G., Hopkins, D., & Roberts, I. (1999, 12/11/1999). Adelaide University Online -envisioning a future. Paper presented at the LearnIT Symposium, Adelaide. http://online.adelaide.edu.au/LearnIT.nsf/URLs/Adelaide_University_Online Prosser, M., & Trigwell, K. (1999a). Relational Perspectives on Higher Education Teaching and Learning in the Sciences. Studies in Science Education, 33, 31-60 Prosser, M., & Trigwell, K. (1999b). Understanding Learning and Teaching: The Experience in Higher Education: Open University Press, 325 Chestnut Street, Philadelphia, PA 19106 Roberts, I. W., & Woodbury, R. (2000, 30/10/2000). vGallery, [Lotus Domino database]. Adelaide University. Available: http://online.adelaide.edu.au/vGallery [2000, 30/10/2000]. Shannon, S. (2000). Departmental Learning and Teaching Award - 2000. School of Architecture, Landscape Architecture and Urban Planning, Adelaide University. Available: http://www.arch.adelaide.edu.au/DeptL&TAward2000 [2000, 30/10/2000].

2

Self-assessment, peer assessment and teacher assessment all carried out online and using identical dimensions for assessment

Woodbury, R., Roberts, I. W., & Shannon, S. (2000, 31/10/2000). vGallery -web spaces for collaboration and assessment. Paper presented at the LearnIT 2000 Symposium, Adelaide. http://online.adelaide.edu.au/LearnIT.nsf/URLs/vGallery