c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 4 ( 2 0 0 9 ) 26–38
journal homepage: www.intl.elsevierhealth.com/journals/cmpb
Development of a web database portfolio system with PACS connectivity for undergraduate health education and continuing professional development Curtise K.C. Ng a,∗ , Peter White a,1 , Janice C. McKay b,2 a
Department of Health Technology & Informatics, The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong b Discipline of Medical Imaging, Department of Imaging and Applied Physics, Curtin University of Technology, PO Box U1987, Perth, WA 6845, Australia
a r t i c l e
i n f o
a b s t r a c t
Article history:
Increasingly, the use of web database portfolio systems is noted in medical and health educa-
Received 11 April 2007
tion, and for continuing professional development (CPD). However, the functions of existing
Received in revised form
systems are not always aligned with the corresponding pedagogy and hence reflection is
31 October 2007
often lost. This paper presents the development of a tailored web database portfolio system
Accepted 6 October 2008
with Picture Archiving and Communication System (PACS) connectivity, which is based on the portfolio pedagogy.
Keywords:
Following a pre-determined portfolio framework, a system model with the components
Electronic portfolio
of web, database and mail servers, server side scripts, and a Query/Retrieve (Q/R) broker for
Picture archiving and
conversion between Hypertext Transfer Protocol (HTTP) requests and Q/R service class of
communication system
Digital Imaging and Communication in Medicine (DICOM) standard, is proposed. The system
Education
was piloted with seventy-seven volunteers.
Continuing professional
A tailored web database portfolio system (http://radep.hti.polyu.edu.hk) was developed.
development
Technological arrangements for reinforcing portfolio pedagogy include popup windows
Database
(reminders) with guidelines and probing questions of ‘collect’, ‘select’ and ‘reflect’ on
Reflection
evidence of development/experience, limitation in the number of files (evidence) to be uploaded, the ‘Evidence Insertion’ functionality to link the individual uploaded artifacts with reflective writing, capability to accommodate diversity of contents and convenient interfaces for reviewing portfolios and communication. Evidence to date suggests the system supports users to build their portfolios with sound hypertext reflection under a facilitator’s guidance, and with reviewers to monitor students’ progress providing feedback and comments online in a programme-wide situation. © 2008 Elsevier Ireland Ltd. All rights reserved.
∗
Corresponding author. Tel.: +852 3400 8587; fax: +852 2362 4365. E-mail addresses: curtise
[email protected], curtise
[email protected] (C.K.C. Ng),
[email protected] (P. White),
[email protected] (J.C. McKay). 1 Tel.: +852 3400 8568; fax: +852 2362 4365. 2 Tel.: +61 8 9266 7151; fax: +61 8 9266 4344. 0169-2607/$ – see front matter © 2008 Elsevier Ireland Ltd. All rights reserved. doi:10.1016/j.cmpb.2008.10.004
c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 4 ( 2 0 0 9 ) 26–38
1.
Introduction
Increasingly, the use of a web database portfolio system is noted in areas of medical and health education, and for continuing professional development (CPD) [1–3]. Within health care, for either taught programmes or CPD purposes, both academic and professional development should be noted, and students or practitioners should be encouraged to make the connection between the two, to develop the ability to adapt and adjust learning informed by experience and practice. A portfolio is an instrument which can support both the outcomes-based educational approach and requirements for meeting competence in health care education and professional accreditation processes. The purpose of both is usually to foster reflection on previous learning and practice experiences, leading to applied learning and continuous development in a cyclic manner [4–7]. Electronic portfolios create increased opportunities, when compared with the traditional paper-based formats, to support and facilitate professional development from pre-registration education to later CPD activities. Based on the enormous capability of storage and management, they provide a link for the illustration of professional development to different parties such as academic institutions, registering bodies and employers [8–11].
2.
Background
2.1.
Analysis of related work
Web database portfolio systems create the possibility to promote a large scale portfolio practice and to provide learning and assessment opportunities, and tracking capability of development for pre-registration education and CPD. The existing commercial web database portfolio systems, however, do not fit with these purposes probably due to the consequences of the electronic portfolio boom [9,12]. The great financial potential has attracted corporate and academic institutions such as McGraw-Hill and Wadsworth, and prestigious universities such as Stanford University and the University of California, Los Angeles (UCLA) to share in this huge market. However, financial profitability depends on efficiency leading to distortion of the portfolio pedagogy. The functions of these systems are not aligned with the pedagogy, i.e. the ability to ‘collect’, ‘select’ and ‘reflect’ on evidence of development/experience. Instead, there is a branching effect to provide functions other than those normally required of a portfolio, including online resumes, course management tools, advising portfolio tools, institutional portals and content management. From a pedagogical perspective, the widening foci create a threat to the effectiveness of the portfolio pedagogy by users using the related functions provided rather than building their own portfolios. Together with the standardization of the portfolio building process to increase efficiency, the core component of the portfolio, i.e. reflection, is lost [9].
2.2.
Prior work leading to current design decisions
An emphasis on reflective learning and professional development can be captured using electronic portfolio systems.
27
What is important is that the system can facilitate authors to meet the content requirements of a portfolio rather than the technical process of portfolio building. This paper presents the development of a tailored web database portfolio system, built for use by radiographers, which is strongly based on the portfolio pedagogy, so as to capture their reflective learning and professional development. Flexible tools are granted to users as authors of the portfolio rather than as content providers or uploaders of artifacts, with some standardization to ensure efficiency. Picture Archiving and Communication System (PACS) connectivity is also injected into the system to provide a means of facilitating CPD and evidence of life long learning for radiographers by capturing, for example, their workplace outcomes, such as medical images produced. Since CPD is developing among all health professions, it could also be extended to other professional areas so as to provide evidence of practitioners’ competences throughout their professional lives.
3.
Design considerations
To develop a web database portfolio system based on the portfolio pedagogy, a portfolio framework was determined in advance. The following discussion describes the setting up of a portfolio system used on a voluntary basis by students on an undergraduate radiography programme. Students expect to receive strong support in their professional development, including progress towards and ability to meet required graduating professional competences. A reflection workshop to teach the students how to collect and select relevant artifacts was arranged after appropriate standards of progress were delivered at corresponding stages. Students prepared evidence to demonstrate meeting these stages from, for example, assessment instruments. This was placed into the portfolio along with their own reflective writing to link up the individual pieces of evidence and to illustrate the level of competence attainment relevant to the standard. This incorporates the mechanism of the portfolio in tying up evidence from multi-dimensional assessment strategies [13]. Throughout the academic year, a facilitator was available to provide additional support and guidance to students who had difficulties with the portfolio pedagogy, such as assisting them to recognize their own development. Evaluation of the portfolio by faculty was conducted by reading the reflective writing and following the links to the corresponding individual artifacts. They provided ratings of attainment of each competence statement noted. Faculty then provided feedback to each participant to inform them of their progress of competence development along with suggested remedies where necessary. Through this process, faculty obtained a more in-depth view of each student’s learning and development progress for programme evaluation. This process was repeated over two years. As a formal component of a programme, this practice can be repeated throughout the students’ study until graduation as a means of monitoring their progress towards, and ability to meet, graduating professional competences. Although the portfolio was developed based on an undergraduate radiography programme, the framework is suitable for other health care programmes or CPD purposes by substituting corresponding competences for different professions [14].
28
c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 4 ( 2 0 0 9 ) 26–38
Fig. 1 – A model of a web database portfolio system with PACS connectivity.
4.
System description
4.1. Model of web database portfolio system with PACS connectivity A model of a web database portfolio system is proposed (Fig. 1) taking into consideration the situation of the portfolio framework and the possibility of providing a means of facilitating CPD and providing evidence of life long learning by capturing artifacts such as medical images produced. Web clients such as undergraduate students, clinical practitioners, facilitators and reviewers can use browsers to communicate with a web server through internet, to access the system. The web server runs appropriate Active Server Pages (ASP) scripts (ASP-VBScript) to provide requested services such as user authentication, portfolio building and viewing, feedback and grading, internal message communication, memorandum and changes of user preferences. The ASP scripts provide functions of file upload and connectivity with mail, database and PACS servers apart from general web page rendering. The web server in this study is a Dell PowerEdge 2850 using Internet Information Server (IIS) 6.0 from the Microsoft Server 2003. The mail server uses the default Simple Mail Transfer Protocol (SMTP) virtual server inside the IIS 6.0 to send out automated Electronic Mail (e-mail), so as to alert users. The database server is another Dell PowerEdge 2850 machine running Microsoft SQL Server 2005 in a Microsoft Server 2003 platform. This is responsible for archiving and retrieval of data for system contents rendering and users’ input through Object Linking and Embedding Database (OLEDB) connection. Capturing of medical images produced by clinical radiographers is achieved by the Query/Retrieve (Q/R) service class of Digital Imaging and Communication in Medicine (DICOM) standard. The Q/R broker converts Hypertext Transfer Protocol (HTTP) requests from clients to Q/R service class including C-Find, C-Move and CStore for a PACS server, and retrieved DICOM images/data to HTTP responses for browser display and archiving in a web database [15].
4.2.
Implementation and evaluation
The system was piloted with seventy-seven volunteers (seventy-four authors and three reviewers) from an under-
graduate radiography programme. There were thirty-five students (authors) from the first year group, twenty from the second year and nineteen from the final year groups. An evaluation questionnaire of the system called the ePortfolio questionnaire was delivered to them after they had experienced two cycles of electronic portfolio practice. This aimed to gather users’ views on the electronic portfolio practice from their experience of using it, including conceptual and technical aspects. Feedback then informed the revision/modification/improvement of the concept and system. For the questionnaire, closed ended questions using a five point scale (strongly disagree, disagree, neutral, agree and strongly agree) were used to collect specific feedback on three main aspects, including features, benefits, and technical performance of the electronic portfolio system. The items in the first two groupings were developed using secondary data from the literature on (electronic) portfolios [4–6,9,12,13,16–18]. Those in the last category were adapted from the user-perceived web quality instrument developed by Aladwani and Palvia [19]. Open questions were developed to gather general comments on the electronic portfolio practice and the think-aloud technique was conducted with five volunteers from the user group as a pilot test. They were required to verbalize their thoughts and perceptions when completing the questionnaire, so as to identify any unclear or ambiguous questions. This further ensured its validity [20]. The questionnaires collected from students and staff were analyzed separately. Although the same questionnaire was used for both students and staff, they were part of two cohorts. The mean was used to determine the central tendency and variability was measured using standard deviation (SD) of each closed-ended question. Content analysis was used for the open questions of the student set with quasi-statistics as an accounting system. The proportions of the frequency were then put into tables for data presentation. The responses from open questions by staff have been used as quotations in the discussion as necessary [20,21].
5.
Status report
5.1.
System overview
Using the proposed model in Fig. 1, a tailored web database portfolio system (http://radep.hti.polyu.edu.hk) was developed. There is a common login page to direct users to author (Fig. 2), reviewer or administrator mode based on their credentials, and to request passwords if forgotten. The functionalities of the author and reviewer modes are similar. However, a ‘Build Portfolio’ area is provided in the author mode and it becomes ‘Grade Portfolio’ in the reviewer mode. One additional function, ‘Memo’, is also included in the reviewer view. It is an electronic space for reviewers to put down their own experiences and feelings from their teaching, for example, and to accumulate such to provide an evaluation of their teaching performance over a period of time. The administrator interface is for granting special permission to users and maintaining contents of the system such as competence statements archived in the database.
c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 4 ( 2 0 0 9 ) 26–38
29
Fig. 2 – A snapshot of the ‘Build Portfolio’ function of the author view.
5.2.
Build Portfolio
‘Build Portfolio’ is one of the most important functions of this web database portfolio system. This allows users to conveniently develop their portfolio. In Fig. 2, the first rectangle highlights the main menu. Users can switch to other functions by clicking the appropriate links. The second rectangle shows the sub-menu (menu of the ‘Build Portfolio’ function) that allows users to select which stage and competence category they would like to work on. The bottom section is the action menu generated by contents from the database and the ASP script provides a framework for its rendering. There is one heading and four columns. The ‘Competence’ column shows the corresponding competences required for that stage under the selected category. ‘Criteria’ further defines the expected levels of development. The ‘Portfolio Covering Letter’ and ‘Reflect’ column provide links to reflection input forms and the ‘Upload Evidence’ directs users to the file upload page. When a user wants to build a portfolio, the appropriate stage and competence categories are selected in the main menu. Evidence of development against the competence category/statements should be provided first through accessing the links under the ‘Upload Evidence’ column. A popup window with probing questions for collection and selection of evidence is shown together with the file upload page. This reinforces users to review their experience for the selected
area, to collect evidence that can represent it, and choose the best evidence out of the collection. Normally, a maximum of two files can be uploaded for each area, so as to support the evidence selection process. If the previous uploaded files occupy the quota, accommodation of new files can be achieved through replacing the previous ones. Also, permission for increasing the quota to a maximum of three can be granted through the administrator interface if the user has a strong reason. The file upload page also has a link to the Q/R interface for DICOM images from PACS. This interface uses a patient root Q/R information model. Clients can input patient level keys such as patient name, patient ID, date of birth and study date to query images [22]. These HTTP inputs are then converted to the DICOM standard through a third party Component Object Model (COM) object, i.e. the Q/R broker called by the ASP page. The Q/R broker becomes the Service Class User (SCU) of the C-Find service and the Service Class Provider (SCP), i.e. the PACS server sends back the query result. Again, it is translated back to HTTP responses. The subsequent retrieval procedure is translated to C-Move and C-Store services by the Q/R broker. Eventually, the required DICOM images are sent to the web server. Another function of the Q/R broker is to convert the retrieved DICOM files to Joint Photo-graphics Experts Group (JPEG) format for browser display and archiving in the web database [15].
30
c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 4 ( 2 0 0 9 ) 26–38
Fig. 3 – A snapshot of the retrieved images from PACS in JPEG format for display and further archiving.
Fig. 3 shows a snapshot of the Q/R interface displaying the retrieved Magnetic Resonance (MR) images of a patient called PACS Interface Demo taken on 10/07/2004 in JPEG format. The first rectangle shows the competence area that the images are for. Following is a hyperlink to the file upload page. The third rectangle highlights the information about the images shown underneath it. The images will be archived as evidence for the competence area once the thumbnail is clicked. The last rectangle lists the image file archived in this way. The last step in the portfolio building is reflective writing, using the reflection input form. Again, a browser window with guiding questions for reflection pops up together with the form when the ‘reflect’ hyperlink is clicked. It explains to authors that reflection is the written account/justification of the evidence provided and it should include responses to the following questions:
1. How do you think the experience selected shows your competence development for this area? 2. Do you see any area requiring further development based on the experience? 3. How can you improve your competences next time? 4. From this evidence what do you see as your particular strengths?
In Fig. 4, a snapshot of a reflection input form is shown. The heading is the area for reflection. On the left hand side is a text area for users’ reflection input. Opposite is a column headed ‘Evidence Insertion’ listing out the evidence archived. These are buttons to insert hyperlinks of corresponding evidence files into the text area using JavaScript. At the bottom, there are buttons for saving and submitting the reflection. When a user finishes building a portfolio, it is necessary to click the ‘submit’ button in the left lower corner of the ‘Build Portfolio’ page to declare it is ready for review. The electronic portfolio is then locked. The hyperlinks of the file upload page and reflection input forms are hidden, and access to them is denied to block any subsequent modification. Also, automated e-mails are generated for notifying the responsible reviewer and informing the user of the submission status.
5.3.
Grade Portfolio
The ‘Grade Portfolio’ is the most important function of the reviewer interface. As shown in Fig. 5, there are filter and search utilities for locating authors’ portfolios. The query result is a list of hyperlinks to corresponding authors’ portfolios. The bottom browser is an author’s portfolio called ‘Grade Portfolio Form’ as it has utilities for providing levels of attainment and feedback. The layout is similar to the authors’ ‘Build Portfolio’ page. The major difference is an absence of the
c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 4 ( 2 0 0 9 ) 26–38
31
Fig. 4 – A snapshot of a reflection input form.
criteria column which is substituted by two new columns, i.e. grade and comments. Again, the portfolio is divided into different categories. There is a checkbox called ‘Benchmark Guide’ to aid the reviewer in providing levels of development. After checking this box, the expected levels of development, as shown in the criteria column of the author mode, are filled in the grade column. This provides a reference for the reviewer to determine the attainment level and saves reviewing time as it is only necessary to make the necessary adjustment. The √ ‘ ’ symbols under the reflection and evidence columns signify that there is reflection and evidence for the area as hyperlinks. The utilities under the grade and comments columns enable the reviewer to provide the levels of attainment and feedback after reviewing the materials. Eventually, the checkbox ‘Finished Review?’ is checked to send out automated e-mails notifying authors of the availability of feedback.
folios, i.e. ‘View Portfolio Form’. The general layout is also similar to the ‘Grade Portfolio Form’. However, grade and comments cannot be modified in this interface. If the relevant contents such as reflection, evidence, grade and comments are present in the portfolio, corresponding information or hyperlinks are shown. For example, there is an overall comment for the portfolio shown in Fig. 6 but not elsewhere. There is no hyperlink under the comments column. Also, the font colour of the grades change according to their values. Red is used to highlight an attainment below standard while blue indicates outstanding achievement. Grey represents appropriate development progress. In author mode, the ‘View Portfolio Form’ is named ‘View Portfolio’ and the interface is more or less the same.
5.4.
Other utilities of the system include an introduction, user guide, internal message, settings changing and logout. In the introduction, the portfolio pedagogy and its implementation settings are presented. The user guide serves as a quick technical reference. The internal message utility provides a channel for text communication between authors and reviewers. Its advantage over e-mail is to centralize the communication, so as to make it more organized and holistic over time. It seems that e-mail is not an appropriate choice because messages for
View Portfolio
The ‘View Portfolio’ function aims to increase the readability of portfolios for authors and reviewers. In reviewer mode, there are filter and search utilities to locate relevant portfolios (Fig. 6). Filter parameters include year of study, completion, overall grade and problems with portfolios. This is similar to the ‘Grade Portfolio’ interface. The search results, i.e. a list of authors’ names, serve as hyperlinks to corresponding port-
5.5.
Other utilities
32
c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 4 ( 2 0 0 9 ) 26–38
Fig. 5 – A snapshot of the ‘Grade Portfolio’ function.
this purpose may scatter over the whole mailbox and mix with other e-mails or even be deleted after a period of time. Automated e-mail notification will fire once there is any message for a user. Changes of password and an option of receiving automated e-mail can be made inside the settings changing utility and the logout function is to foster the system security.
5.6.
Findings of the ePortfolio questionnaire
Sixty-two questionnaires were collected from the student participants, yielding a response rate of 83.8% (62 out of 74). The three reviewers returned their responses to their questionnaires, yielding a 100% response rate. Table 1 shows only two questions (Q.13 and Q.24) that have mean values below three (neutral) while the remaining questions have positive responses as noted in the ‘students’ column. However, when a more stringent approach is used by rounding up the mean to integers, Q.11, Q.12, Q.13, Q.16, Q.18 and Q.24 should be considered as neutral responses. The students have provided a positive response to most of the issues raised in the questions. The first four questions relate to the ‘Benefits of the ePortfolio’ (Q.11, Q.12, Q.13 and Q.16) including development of personal and professional attributes and higher order skills, assurance of education quality, strengthening of teacher–student relationships and
facilitation of collaborative learning. The other two questions (Q.18 and Q.24) relate to the ‘Technical Performance of the ePortfolio’ including ease of use and attractive layout. Using the same rounding up approach, there were four questions (Q.4, Q.13, Q.16 and Q.24) which have a neutral response from the reviewers’ perspective while a positive response is noted for the remaining issues.
6.
Lessons learned
6.1.
Alignment with the portfolio pedagogy
A good web database portfolio should align with the central underlining values of portfolio pedagogy, which are ‘collect’, ‘select’ and ‘reflect’. They are the three most important features of portfolio pedagogy, as set out below [9]. 1. A collection of materials from an individual’s experiences. 2. A selection of materials from the collection with annotations to justify the selection action which is the outcome of reflective thinking. 3. Reflection to narrate an individual’s development process and attainment through the contents of portfolio for evaluation. 4. Presumption of development.
c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 4 ( 2 0 0 9 ) 26–38
33
Fig. 6 – A snapshot of the ‘View Portfolio’ function.
5. Diversity of contents along and across portfolios. 6. A means of communication to readers of an individual’s progress and attainment. 7. An evaluation tool ([16], p. 130). In this system, a popup window with guidelines and probing questions of ‘collect’, ‘select’ and ‘reflect’ are shown when authors access file upload pages and reflection input forms. Although covered in the reflection training workshop, this technical arrangement reminds and reinforces these important aspects. The limitation in the number of files to be uploaded is another measure to foster authors to exercise their reflective thinking when making their selection. The ‘Evidence Insertion’ functionality in the reflection input form links the individual uploaded artifacts with the reflective writing, so as to increase the coherence of the portfolio as a whole. This arrangement removes the major criticism of web database portfolio systems, i.e. the standardization of the portfolio building process shifts the authors’ role to that of content provider or uploader of artifacts, leading to a loss of reflection [9]. The authors are required to state how they think the experience selected demonstrates their competence development for this area followed by the JavaScript generated hyperlinks to corresponding evidence, and achieves the real spirit of the selection process as noted above [9,10,16]. Together with reflective comment to suggest attainment and further improvement, this forms the presumption of development. This realizes the hypertext reflection in a web database
portfolio environment which helps to contextualize and support issues to be justified, which is difficult to accomplish in a paper-based portfolio [23]. Accommodation of diversity of contents along and across portfolios is achieved through the system tolerance to different uploaded file formats from traditional JPEG, Portable Document File (PDF) to DICOM. Various reflection input forms such as portfolio covering letter, reflection form for competence category and for individual statements, also provide flexibility for different users. The portfolio covering letter serves to present an organized, overall picture of development, which is an efficient channel for both authors and reviewers. However, detailed reflection is best noted using individual reflection forms. Combined use of the three channels accommodates a great variety of contents. The ‘Grade Portfolio’ function provides a convenient interface for portfolio evaluation by use of filter and search utilities to locate authors’ portfolios, automated generation of hyperlinks to highlight submitted contents, and ‘Benchmark Guide’ to facilitate the grading process. Communication of attainment to readers is the major use of the ‘View Portfolio’ page, enhancing the readability of portfolios through functions such as filter and search utilities, automated hyperlinks generation, and colour highlighting of development. Other communication functionalities include internal messages and automated e-mail generation after portfolio submission, evaluation and receipt of messages. However, functionalities other than for portfolio practice are absent in the system, to eliminate any branching out effect.
34
c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 4 ( 2 0 0 9 ) 26–38
Table 1 – A summary of findings of the closed-ended questions. Questions
Student (n = 62) Meana
Features of the ePortfolio 1. It serves as a collection of materials (referring to the ‘collect’ procedure) from an individual’s experiences 2. It involves a selection of materials from the collection (referring to the ‘select’ procedure) 3. It contains annotations (written accounts) to justify the selection action, based on the outcome of the individual’s reflective thinking 4. It contains reflections indicating an individual’s development process and attainment, following evaluation of these contents 5. There is a presumption (self-evaluation) of development 6. It is capable of accommodating diversity of contents, such as different kinds of evidence over time and covering different development issues 7. It provides a means of communication with readers on an individual’s progress and attainment 8. It serves as an evaluation tool Benefits of the ePortfolio 9. It promotes reflective learning 10. It facilitates recognition of strengths for mapping of career pathways 11. It facilitates development of personal and professional attributes and various higher order skills, such as self-motivation and critical thinking 12. It assures education quality 13. It strengthens teacher–student relationships 14. It illustrates competence development 15. It supports resume writing and continuing professional development (CPD) 16. It facilitates collaborative learning (involving sharing of portfolios with peers, and providing comments and feedback on others’ works) Technical Performance of the ePortfolio 17. It is a secure platform 18. It is easy to use 19. It is always reliable and available 20. Its speed of page loading is fast 21. It is interactive and generates dynamic contents 22. It is flexible for accommodation of your work 23. Its organization of contents is clear 24. Its layout is attractive a
SD
Reviewer (n = 3) Meana
SD
3.87 (4)
0.61
4.67 (5)
0.58
3.82 (4)
0.59
3.67 (4)
1.53
3.69 (4)
0.53
4.33 (4)
0.58
3.81 (4)
0.70
3.33 (3)
1.15
3.98 (4) 3.73 (4)
0.61 0.61
4.00 (4) 4.67 (5)
0.00 0.58
3.52 (4)
0.67
4.00 (4)
1.00
3.85 (4)
0.72
4.33 (4)
0.58
3.63 (4) 3.50 (4) 3.48 (3)
0.79 0.72 0.83
4.00 (4) 4.33 (4) 4.00 (4)
1.00 1.15 1.00
3.39 (3) 2.92 (3) 3.51 (4) 3.65 (4) 3.42 (3)
0.86 0.77 0.65 0.66 0.74
4.67 (5) 3.33 (3) 4.00 (4) 4.00 (4) 3.33 (3)
0.58 0.58 1.00 1.00 0.58
3.87 (4) 3.40 (3) 3.50 (4) 3.82 (4) 3.50 (4) 3.52 (4) 3.56 (4) 2.84 (3)
0.71 0.98 0.94 0.78 0.72 0.76 0.80 0.71
4.00 (4) 4.33 (4) 4.33 (4) 4.00 (4) 3.67 (4) 3.67 (4) 3.67 (4) 2.67 (3)
0.00 0.58 0.58 1.00 0.58 0.58 0.58 1.53
Legend: 1 = Strongly disagree; 2 = Disagree; 3 = Neutral; 4 = Agree; 5 = Strongly agree (mean rounding up to integers).
6.2. Discussion of the current electronic portfolio practice Positive mean scores were noted with respect to ‘Features of the ePortfolio’ in the ePortfolio Questionnaire (Q.s 1–8, except Q.4 from the reviewer perspective, Table 1). This not only indicated good alignment of the system functionalities with portfolio pedagogy, as suggested in Section 6.1, but also ensured that quality portfolio processes were in place in the current electronic portfolio practice. This is because a successful portfolio process should posses the features of portfolio pedagogy, such as ‘collect’, ‘select’ and ‘reflect’ of evidence of development [9,11,24]. The only neutral response (Q.4) from the staff findings could be explained by one reviewer’s suggestions for improvement: There should be on-going (embedded in their learning, in the form of reflective diary) documentation of their learning experiences rather than an overall retrospective summary after the semester. In this way, it may provide quality data. (Reviewer of First Year Portfolios)
From that reviewer’s perspective, the reflections contained in the electronic portfolios cannot indicate an individual’s development process since, it is suggested, this is only an overall retrospective summary after the completion of the academic year. The reviewer’s comments are understandable. The most important issue is that the accumulated reflections throughout the three years of study should be able to demonstrate an individual’s development process. It is useful to request students to note down their important experiences throughout the year as is common practice with keeping a reflective journal, as they may forget some issues over a longer period of time [8]. However, the point is whether it is necessary to expose all experiences without a selection process. If this is the case, the reviewers may be overloaded with reviewing quantities of students’ reflections. It is also not good for the students’ development of portfolio building skills since the ability to select is one of the essential elements of portfolio building, not only evidence but also reflection. The reflection which the reviewer works on is actually the final masterpiece of the students’ selection effort [4,9]. However, it may be useful to provide an electronic reflective journal facility
c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 4 ( 2 0 0 9 ) 26–38
in the system where students are encouraged to put down thoughts during the year, and from which they select their final masterpiece. Although only four benefits have been perceived by student participants, as noted in the ‘Benefits of the ePortfolio’ (Table 1), the most important ones ‘promotion of reflective learning’ and ‘illustration of competence development’ were identified in the responses. The capability of electronic portfolio practice for competence diagnosis is the core component of the study aim, while the reflection element is essential for this practice to succeed [9,13,24,25]. Identification of the two elements as benefits further confirms the success of the current practice. The importance of reflection is also noted in a reviewer’s response to the open question, ‘the Most Successful Aspects of this Exercise/System’: . . . ability to encourage students to be reflective learners. (Reviewer of Third Year Portfolios) Students and reviewers responded positively to ‘Technical Performance of the ePortfolio’ (Table 1), and positive findings were noted in the items: ‘security of platform’, ‘reliability and availability’, ‘fast page loading’, ‘interactivity and generation of dynamic contents’, ‘flexibility for accommodation of work’, ‘clear organization of contents’. One more positive response was shown in the reviewers’ findings for this category, that it was easy to use (Q.18, Table 1). Further positive responses to the open questions from the reviewer of the third year portfolios are noted below: ‘Benefits Gained from the ePortfolio Exercise’: . . . it is easy to use. (Reviewer of Third Year Portfolios) ‘the Most Successful Aspects of this Exercise/System’: . . . ease of use. (Reviewer of Third Year Portfolios) ‘Suggestions for Improvement’: Use of the portfolio does require a good introduction, although once this has been done, it is easy to use. (Reviewer of Third Year Portfolios) The reviewer of the third year portfolios suggested that ease of use depended on a good introduction. The introduction for reviewers took place on a face-to-face, individual basis in contrast to a group briefing session for student participants. This may account for the students’ neutral response to the issue of ease of use (Q.18). Reinforcement of technical issues may be necessary for students in the training workshop.
6.3.
PACS connectivity
The electronic portfolio system established in this study also provides PACS connectivity for retrieval of medical images from any PACS. It seems its importance is not significant for pre-registration education as students’ assessment outcomes can serve as readily available evidence to demonstrate their professional attainment. However, when extending the use of the electronic portfolio system from pre-registration education to later CPD, it seems the workplace outcomes, for example, medical images produced in the radiography dis-
35
cipline, should also be considered as an indicator of their professional development. The use of this is a derivative of the competence assessment, medical record review. The medical records, as the readily available evidence of clinical skills, can be used to infer individuals’ holistic competence [26,27]. It is noted that the mechanism of the portfolio is to tie up evidence from multi-dimensional assessment strategies [13]. The current system provides a one-stop solution for radiographers to retrieve their evidence of professional development in the workplace, i.e. medical images produced. These images can be saved into the system as evidence of development for portfolio building. This is the collection and selection of evidence of development. Reflective writing can then be provided to justify their professional development supported by the retrieved images, thus linking the evidence together. Although this can be achieved by obtaining images from the web based PACS and uploading them to the ePortfolio system, the importance of this integration is to provide automation or ease of use, so as to promote the use of workplace outcomes as evidence for portfolio building in the CPD situation. New features have now been added to the DICOM standard, such as integration with inputs other than traditional radiological images, for example, visible light images and waveforms. The use of the PACS interface is not limited to radiography/radiology but can also serve other related health care disciplines for which these objects can provide evidence of workplace outcomes. Using PACS connectivity may create possible breaches of patient privacy, so user-authentication of the system and conversion of DICOM images received to JPEG are the measures in place to safeguard patient confidentiality and prevent abuse by users. User-authentication can validate the identity of users and limit accessibility to certain images archived in the PACS based on their profiles. The entire conversion procedure is carried out in the system server and this prevents users from obtaining the original DICOM files. In effect, users can only receive those JPEG images from which all patient identifiers have been removed. The conversion of DICOM to JPEG files also facilitates image distribution through the internet in terms of transmission speed and compatibility [14,15].
6.4.
Study advancement
This tailored web database portfolio system addresses the major criticism of existing products, which is overstandardization leading to a shifting role of those building the portfolio from authors to content providers or uploaders and hence losing reflection capabilities [9]. A balance between standardization and flexibility is achieved in several ways. The standardized components include the fixed interfaces for portfolio building, viewing, grading and internal communication. These serve as the framework to render contents such as competence standards and users’ data into a meaningful sense. It is crucial to standardize these elements to ensure efficiency with little interference to authors’ reflection activities. Its effect on efficiency over the existing web portfolio practice, i.e. portfolios housed in a personal homepage style, includes time saving to learn homepage creation and determination of its layout [9,11,28]. Maintenance and administration of portfolio practice is also another benefit of standardization, so as to retain its original aim, i.e. to serve a large population
36
c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 4 ( 2 0 0 9 ) 26–38
Fig. 7 – A snapshot of the electronic portfolio built through the use of ‘CPD Now’.
[9,12]. As the standardized components are those provided by the stakeholders, such as education institutions and registering authorities, this approach should not hinder reflection practices. However, the most important function of the standardized components is to trigger and facilitate self-directed reflective thinking. The remaining functionalities should be flexible enough to facilitate this. It is suggested that the portfolio is an effective and efficient way to measure and record individuals’ personal achievements and professional accomplishments in terms of occupational standards and competences. Through critical reflection on learning experiences, individual, separate evidence of professional and competence development are integrated and organized into a meaningful sense, i.e. a simple, clear and manageable display of attainment which is useful for stakeholders and students/practitioners [13,17]. This forms the foundation for designing flexible tools to facilitate reflection. The ‘Evidence Insertion’ utility of the reflection input forms is developed based on this agenda. It enables authors to use their reflective writing to link up with individual artifacts into a holistic masterpiece through cyberspace. This technical arrangement inside a web database portfolio system is an innovative way to address the criticisms of web database portfolios including role shifting, hypertext reflection of web portfolios, and the contribution of portfolio pedagogy in the competence movement [9,13,17,23]. Other
flexible tools include capability to accommodate a variety of uploaded file formats and choices of various reflection input forms for different purposes. Together with the innovative development of PACS connectivity in a web database portfolio system, an advanced portfolio practice for undergraduate medical and health education, and CPD, should be achieved. For reinforcement of the claim of innovation, the ‘CPD Now’ system developed by the influential body of the radiography profession, the Society and College of Radiographers, UK, can be used to illustrate this issue. To use the UK system for portfolio building, the user needs to go to ‘My CPD’ (portfolio building) area and create his/her own CPD Framework. Under this framework, the user submits the learning plans and evidence by putting down relevant information for the different fields inside the standard templates. The major technical difference between the templates for ‘learning plan’ and ‘evidence’, is the file upload functionality provided in the evidence template. After completion of this part, the portfolio building process is completed and the user can go to the ‘My Portfolio’ (portfolio viewing) area to view his/her portfolio, as noted in Fig. 7. The electronic portfolio contains three main parts, My Personal Details, My Progress and Download My Careers. Under the My Progress section, there are three items including My CPD Evidence, My Learning Plans and My Certificate. The My CPD Evidence and My Learning Plans are the core contents of the portfolio which are lists of inputs from the templates
c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 4 ( 2 0 0 9 ) 26–38
of ‘learning plans’ and ‘evidence’. In Fig. 7, My CPD Evidence is shown. It lists the previous evidence input in a sequential order and without any access to the supporting evidence files (attachments) [29,30]. This is a typical example of the existing web database portfolio systems. The portfolio building process is greatly restricted by the template, i.e. over-standardization leading to role shifting [9,12]. Also, hypertext reflection cannot be achieved [23]. Furthermore, PACS connectivity is not available inside this system [29,30].
7.
Conclusion
Evidence to date suggests the system supports users to build their portfolios with sound hypertext reflection under a facilitator’s guidance, and with reviewers to monitor students’ (authors’) progress and provide feedback and comments online in a programme-wide situation. Positive mean scores were noted with respect to ‘Features of the ePortfolio’ in the ePortfolio questionnaire. These responses indicated good alignment of system functionalities with portfolio pedagogy and quality portfolio processes. The web database portfolio system’s format and use seem to have been well accepted by students and reviewers. On face value it may appear that the use of the electronic portfolio system is not closely related to retrieval of clinical data from PACS, but the clinical data is an integral part of any practitioner’s workplace outcomes and development. Ways must be provided to enable such evidence to form part of their portfolios, for example, by using the file upload facility. The provision of PACS connectivity in the current electronic portfolio system has illustrated the possibility of providing a more convenient way to address this issue. Subsequently, this becomes an integral part of the development of an advanced electronic portfolio system for practitioners, and goes further than merely addressing the major criticisms of existing products.
8.
Future plans
The outcomes of the electronic portfolio system with PACS connectivity once fully established could be extended to monitor CPD of clinical radiographers and other professional areas, so as to provide evidence of practitioners’ competences throughout their professional lives. Although this system is established for the radiography discipline, the standardized framework provides ease of substituting relevant competence standards of other disciplines into the database through the administrator interface. In this way, a large population of medical and health care students and workers can be served, so as to achieve the ultimate goal of web database portfolio technology without sacrificing the most important value of portfolios, i.e. reflection. Finally, it is the functionalities of reflection facilitation rather than powerful or attractive technology that are the most crucial factors.
references
[1] P. Bridge, D. Eddy, The virtual portfolio: from conception to reality, J. Radiother. Pract. 5 (2006) 1–7.
37
[2] M. Lawson, D. Nestel, B. Jolly, An e-portfolio in health professional education, Med. Educ. 38 (2004) 569– 570. [3] G. Duque, A. Finkelstein, L. Winer, S. Gold, Immediacy and efficacy of giving feedback and evaluating medical students during their rotations in geriatric medicine: the McGill electronic portfolio, J. Am. Geriatr. Soc. 51 (2003) S197. [4] J. Gordon, Assessing students’ personal and professional development using portfolios and interviews, Med. Educ. 37 (2003) 335–340. [5] A. Tiwari, C. Tang, From process to outcome: the effect of portfolio assessment on student learning, Nurs. Educ. Today 23 (2003) 269–277. [6] J.G. Alexander, S.W. Craft, M.S. Baldwin, G.W. Beers, G.S. McDaniel, The nursing portfolio: a reflection of a professional, J. Contin. Educ. Nurs. 33 (2002) 55–59. [7] Open University School of Health & Social Welfare, Building portfolios, Nurs. Manag. 9 (2002) 30–33. [8] V. Dagley, B. Berrington, Learning from an evaluation of an electronic portfolio to support general practitioners’ personal development planning, appraisal and revalidation, Educ. Primary Care 16 (2005) 567–574. [9] M. Kimball, Database e-portfolio systems: a critical appraisal, Comput. Composit. 22 (2005) 434–458. [10] S.L. Ramey, M.L. Hay, Using electronic portfolios to measure student achievement and assess curricular integrity, Nurs. Educ. 28 (2003) 31–36. [11] G. Pullman, Electronic portfolios revisited: the efolios project, Comput. Composit. 19 (2002) 151–169. [12] T. Batson, The electronic portfolio boom: what’s it all about? Campus Technol. (2002) (http://www.campus-technology. com/article.asp?id=6984 Accessed 29 November 2005). [13] M. McMullan, R. Endacott, M.A. Gray, M. Jasper, C.M.L. Miller, J. Scholes, C. Webb, Portfolios and assessment of competence: a review of the literature, J. Adv. Nurs. 41 (2003) 283–294. [14] C.K.C. Ng, P. White, J.C. McKay, Establishing a method to support academic and professional competence throughout an undergraduate radiography programme, Radiography (2007) (available online at http://www.sciencedirect.com). [15] H.K. Huang, PACS and Imaging Informatics: Basic Principles and Applications, John Wiley & Sons, Inc., New Jersey, 2004. [16] K.B. Yancey, Portfolio, electronic, and the links between, Comput. Composit. 13 (1996) 129–133. [17] M. Ryan, K.H. Carlton, Portfolio applications in a school of nursing, Nurs. Educ. 22 (1997) 35–39. [18] D.O. Weddle, S.P. Himburg, N. Collins, R. Lewis, The professional development portfolio process: setting goals for credentialing, J. Am. Diet. Assoc. 102 (2002) 1439–1444. [19] A.M. Aladwani, P.C. Palvia, Developing and validating an instrument for measuring user-perceived web quality, Inform. Manag. 39 (2002) 467–476. [20] B. Johnson, L. Christensen, Educational Research: Quantitative and Qualitative Approaches, Allyn and Bacon, Boston, 2000. [21] D.F. Polit, C.T. Beck, Nursing Research: Principles and Methods, seventh ed., Lippincott Williams & Wilkins, Philadelphia, 2004. [22] M.V. Herk, L. Zijp, Conquest DICOM Server 1.4.11 Documentation/MicroPACS Conformance Statement, The Netherlands Cancer Institute, Amsterdam, 2005. [23] S. Watkins, World wide web authoring in the portfolio-assessed, (inter)networked composition course, Comput. Composit. 13 (1996) 219–230. [24] R. Mason, C. Pegler, M. Weller, E-portfolios: an assessment tool for online courses, Br. J. Educ. Technol. 35 (2004) 717– 727.
38
c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 4 ( 2 0 0 9 ) 26–38
[25] K.A. Karlowicz, The value of student portfolios to evaluate undergraduate nursing programs, Nurs. Educ. 25 (2000) 82–87. [26] J.A. Sliwa, K.J. Kowalske, Assessing resident clinical competence, Am. J. Phys. Med. Rehabil. 79 (2000) 468–473. [27] E.S. Holmboe, R.E. Hawkins, Methods for evaluating the clinical competence of residents in internal medicine: a review, Ann. Intern. Med. 129 (1998) 42–48.
[28] P.E. McShane, Electronic portfolio: a paperless portfolio for interns to track accomplishments and market themselves, J. Am. Diet. Assoc. 105 (2005) 37. [29] Axia Interactive Media Limited, CPD Now: SOR Professional Development, Axia Interactive Media Limited, West Yorkshire, 2007 (http://sor.cpdnow.net/Logon.aspx Accessed 9 May 2007). [30] S. Kelly, CPD Now, Synergy (March 2005) 21–23.
