CLINICAL TRIALS
DATA MANAGEMENT AND TRIAL CONDUCT
Clinical Trials 2014; 11: 344–354
Development and implementation of PROgmatic: A clinical trial management system for pragmatic multi-centre trials, optimised for electronic data capture and patient-reported outcomes Per Cramona, A˚se Krogh Rasmussena, Steen Joop Bonnemab, Jakob Bue Bjornerc,d, Ulla Feldt-Rasmussena, Mogens Groenvoldd, Laszlo Hegedu¨sb and Torquil Watta
Background Many clinical trials are conducted as explanatory trials, but the applicability of results from explanatory trials to clinical practice may be questioned. Pragmatic trials elucidate both benefits and harms of an intervention under conditions close to daily clinical practice. We have planned a pragmatic multi-centre trial in patients with Graves’ hyperthyroidism. However, trial management is a complicated task in pragmatic trials, due to limited interaction between participants and trial personnel. Purpose The aim of this project was to develop and implement PROgmatic, a fully integrated trial management system for pragmatic multi-centre trials, optimised for electronic data capture and patient-reported outcomes (PROs). Methods Necessary tasks and logistical challenges that should be handled by PROgmatic were identified, and the system was designed and developed to handle these tasks. A combination of generic applications and custom coding was applied to develop an integrated system that met the required needs. PROgmatic features include secure web-based data entry; electronic case report forms (eCRFs); central participant registration and randomisation; automated emails linking to electronic PROs; automated reminders to participants; automated notifications to trial personnel regarding booking of trial visits, safety and compliance alerts; and monitoring of trial progress. PROgmatic underwent rigorous pilot testing, including data verification and validation, before it was released for trial management. Results PROgmatic was successfully implemented in the GRAves’ Selenium Supplementation (GRASS) trial (ClinicalTrials.gov: NCT01611896) December 2012. The feedback from trial personnel on usability and utility has been positive, and PROgmatic has handled all intended tasks properly. Limitations Implementation of PROgmatic in future studies requires adaptation of the custom coding. Not all email systems accept emails with active links, and participants who use these systems therefore need to complete paper surveys.
a Department of Medical Endocrinology, Copenhagen University Hospital, Rigshospitalet, Copenhagen, Denmark, bDepartment of Endocrinology and Metabolism, Odense University Hospital, Odense, Denmark, cNational Research Centre for the Working Environment, Copenhagen, Denmark, dInstitute of Public Health, University of Copenhagen, Copenhagen, Denmark Author for correspondence: Per Cramon, Department of Medical Endocrinology, Section 2131, Copenhagen University Hospital, Rigshospitalet, Blegdamsvej 9, 2100 Copenhagen, Denmark. Email:
[email protected]
Ó The Author(s), 2014 Reprints and permissions: http://www.sagepub.co.uk/journalsPermissions.nav Downloaded from ctj.sagepub.com at Copenhagen University Library on March 24, 2015
10.1177/1740774513517778
PROgmatic: a trial management system
345
Conclusions PROgmatic facilitated the complex task of conducting a pragmatic multi-centre trial. The automated electronic capture of PRO data is time saving and reduces the risk of erroneous data entry. Email notifications to trial personnel combined with serially activated eCRFs that logically lead patient flow through the trial have helped making the pragmatic trial feasible. PROgmatic provides a template for other pragmatic multi-centre trials with patient-reported measures as high-priority outcomes. Clinical Trials 2014; 11: 344–354. http://ctj.sagepub.com
Background Clinical trials can be divided into two broad categories: explanatory and pragmatic. Many clinical trials are conducted as explanatory trials, with a highly selected patient population, rigorously defined treatment regimens, high follow-up intensity, highly controlled settings, and specially trained personnel. Explanatory trials are efficacy trials that examine whether an intervention works under optimal or selected conditions. The applicability of results from explanatory trials to clinical practice may thus be questioned. Pragmatic trials mimic routine clinical practice, with broad selection criteria, simple trial designs, few follow-up visits, routine settings, and routine personnel. Pragmatic trials investigate whether an intervention works under conditions close to clinical practice [1–3]. Our research group has planned the GRAves’ Selenium Supplementation (GRASS) trial, a multi-centre clinical trial of selenium supplementation versus placebo in patients with Graves’ hyperthyroidism [4]. The purpose of the GRASS trial is to investigate whether patients with Graves’ hyperthyroidism will benefit from selenium supplementation under conditions close to clinical practice. Hence, a pragmatic trial design was chosen. The GRASS trial aims at including 492 participants from seven hospitals, with estimated trial duration of 48 months. Recruiting a large number of patients from multiple trialsites and collecting large amounts of data over a long study period create management challenges. The pragmatic design makes trial management and data quality control further complicated for two reasons: with few follow-up visits, there is only limited interaction between trial personnel and participants, and second, relying on routine personnel implies limited resources allocated to local trial management. Patient-reported outcomes (PROs) play a central role in GRASS. However, missing PRO data is a common problem [5,6] compromising power and precision and, more seriously, missing data may introduce bias [5]. A trial management system that can facilitate management of a pragmatic trial and, at the same time, secure high-quality PRO data was therefore developed.
The purpose of this article is to report development and implementation of PROgmatic: a web-based trial management system designed for pragmatic multi-centre trials, optimised for electronic data capture and PROs.
Methods Overview of the GRASS trial Graves’ hyperthyroidism is an autoimmune disease causing hyperfunction of the thyroid gland. The primary treatment is usually anti-thyroid drugs for 12–18 months with gradual tapering until withdrawal [7]. Each trial participant will remain in the trial until 12 months after cessation of treatment, that is, 24–30 months. Participants receive the trial intervention (selenium or placebo) in addition to the usual treatment. The primary outcome is the proportion of participants in need of additional treatment after initial anti-thyroid drug treatment. Thyroid-related quality of life, hyperthyroid, and eye symptoms are high-priority secondary outcomes measured by the Thyroid Patient-Reported Outcome (ThyPRO) questionnaire, which has been thoroughly validated for use in clinical research [8–10]. Participants receive the ThyPRO questionnaire eight times during the trial, according to time points specified in Table 1. Clinical data and other trial relevant information will be collected at three trial visits. System development The first step in the development process was to identify tasks and logistical challenges that should be handled by PROgmatic, and thereby establish system requirements. PROgmatic should provide web-based data entry via electronic case report forms (eCRFs). Nearly 4000 PROs will be administered during the trial period. A survey module was needed to facilitate collection and data entry of this large amount of PRO data. A module consisting of a
http://ctj.sagepub.com
Clinical Trials 2014; 11: 344–354 Downloaded from ctj.sagepub.com at Copenhagen University Library on March 24, 2015
346
P Cramon et al.
Table 1. Schedule for trial assessments. Specification of time points for follow-up visits and variables assessed at each time point, including baseline Baseline
Follow-up 6 weeks
Visit Urine sample Blood samples ThyPRO survey
x x x x
12 weeks
6 months
12 months
18 months
24 months
x
x
x
x
x
x x
12 months after anti-thyroid drug treatment withdrawal x
x
x x
ThyPRO: Thyroid Patient-Reported Outcome.
in collaboration with SAS Institute A/S. The third step in the development process was to model data flow through PROgmatic in flow diagrams. The diagrams specified variable names and PROgmatic actions, and every component of PROgmatic was modelled in diagrams before being programmed. An example of a flow diagram is shown in Figure 2. The development process of each component is described below. An overview of PROgmatic features is shown in Table 2. Figure 1. Simplified structure and data flow in the trial management system PROgmatic. CRF: case report form. The flow of paper surveys is not shown in the diagram.
central trial database, a controller program, and an email server should be able to handle the following tasks: booking of trial visits; timing of PRO collection; monitoring of events requiring discontinuation of the intervention, compliance, and adverse reactions. The controller program should monitor the database and activate the email server to send email notifications, for example, when trial personnel need to take action. The same module should send emails to participants with built-in links to electronic surveys. The first step of the development process resulted in a simple model of the system structure and data flow (Figure 1). The second step was to investigate whether available management systems could meet the trial-specific needs. We could not identify an available system with features and flexibility needed to manage the GRASS trial. We then examined whether some of the tasks could be handled by generic applications. SurveyXact (http://www.surveyxact.com), a web questionnaire system hosted by Ramboll Management (http:// www.ramboll.com), was chosen to facilitate the collection and entry of survey data. We selected a dynamic web application developed by SAS Institute A/S to provide data entry via eCRFs. The central database and the controller program were developed
eCRFs Trial personnel use the web-based eCRFs to collect trial data. Only trial personnel enter data into eCRFs. Data collected at trial visits are entered into the corresponding eCRFs. Trial personnel also enter data into eCRFs when involved in the management of a trial participant, for example, when booking a trial visit. The eCRFs were based on a dynamic web application developed by SAS Institute A/S. They are dynamic web pages written in Python Programming Language (http://www.python.org/), using the Django web framework (https://www.djangoproject. com/). The eCRFs are hosted on an Apache web server and connect to a MySQL database. The layout of the eCRFs is simple and user-friendly (Figure 3). The main web page shows the participants together with the participants’ active eCRFs. Only the next eCRF to be completed is flagged as active for each participant. Submission of one eCRF activates the next eCRF, but if necessary, it is possible to return to an eCRF and modify data. Clicking the identification (ID) number for a specific participant opens an overview page that shows completed eCRFs; the active eCRF; the global eCRF; and survey completion status. In this way, local trial personnel can monitor local participant progress and local trial-site progress in the dynamic web application. The coordinating trial-site can access this information for all trial-sites. The global eCRF is used to record any contact to participants between trial visits and to manage
Clinical Trials 2014; 11: 344–354
http://ctj.sagepub.com Downloaded from ctj.sagepub.com at Copenhagen University Library on March 24, 2015
PROgmatic: a trial management system
347
Figure 2. Flow diagram: administration of patient-reported outcomes (PROs). eCRF: electronic case report form. Every component of the trial management system PROgmatic was modelled in diagrams before being programmed.
Table 2. Overview of PROgmatic features Feature
Description
eCRFs Central randomisationa Electronic questionnairesa Reminder emailsa
Secure data entry into eCRFs via web interface Stratified block randomisation with randomly varying block sizes Participants receive automated emails linking to electronic ThyPRO questionnaires Participants receive automated reminders if they do not complete the ThyPRO survey within a specified time frame Participants who do not want to answer electronic surveys can answer paper questionnaires. A secretary at the coordinating trial-site manages the flow of paper surveys via this module Trial personnel receive automated email notifications prior to visit 2 and 3, where the trial personnel are instructed to book the trial visits Trial personnel receive an email notification if a participant has severe symptoms that may be caused by the trial intervention If a participant becomes pregnant, receives radioiodine therapy, or surgery, the trial personnel receive an email alert notifying that the patient shall discontinue the intervention Trial personnel receive an email alert if the participant’s self-reported compliance is too low (below a prespecified value) Enables monitoring of participant progress, trial-site progress, and overall trial progress
Administration module for paper questionnaires Notification of trial visitsa Safety alerts (symptoms)a Safety alerts (stop intervention)a Compliance alerta Monitoring of trial progress ThyPRO: Thyroid Patient-Reported Outcome. a Automated tasks.
http://ctj.sagepub.com
Clinical Trials 2014; 11: 344–354 Downloaded from ctj.sagepub.com at Copenhagen University Library on March 24, 2015
348
P Cramon et al.
Figure 3. Electronic Case Report Form 1A (eCRF1A): patient data. Trial personnel use the web-based eCRFs to collect trial data. Only trial personnel enter data into eCRFs. The eCRFs are behind the firewall of the Capital Region healthcare system, and access to the web application requires valid user ID and password. Data entered into the eCRFs are stored on a secure server. The CPR number is a personal ID number administered by The Danish Civil Registration System.
Figure 4. An extract of the electronic ThyPRO survey (US version). In addition to the ThyPRO survey, the electronic and paper questionnaires also contain questions on known adverse reactions to the trial intervention, events requiring discontinuation of the trial intervention (e.g. pregnancy), and self-reported compliance. ThyPRO: Thyroid Patient-Reported Outcome.
ID: identification; CPR: central person register.
participants fully or partly discontinued in the trial. Based on variables from the flow diagrams, a SAS database program was written to generate SAS database tables. Each variable in the eCRFs has a matching variable in the SAS database, and data entered into eCRFs are stored real-time in the appropriate SAS database tables. Central participant registration and randomisation A SAS macro performs block randomisation with randomly varying block sizes. Randomisation is stratified by clinical trial-site and disease status (incident or relapse), and the allocation ratio of participants assigned to ‘experimental intervention’ or ‘control intervention’ is 1:1. Randomisation numbers are integrated in PROgmatic, and participants are assigned an individual randomisation number, based on the entered stratification variables, immediately after submission of the inclusion eCRF. Trial intervention with the corresponding randomisation number is then provided to the participant. The participant receives trial intervention for the entire trial at the baseline visit. Electronic and paper surveys The electronic surveys are provided by Ramboll’s web questionnaire system, SurveyXact. The layout of the electronic ThyPRO questionnaire (Figure 4) is coded in the Cascading Style Sheets (CSS) language and is very similar to the layout of the original paper version of the ThyPRO questionnaire [8–10].
Variable names, formats, and precoded response choices are specified in the survey builder module. The participant chooses mode of administration at the baseline visit: electronic or paper. The chosen mode can be changed after the baseline visit, but a change requires submission of ad hoc programming code. In case of electronic surveys, the participant’s personal email address is entered into the inclusion eCRF, and at relevant time points, the participant receives emails with a built-in link to the electronic survey. Clicking the link automatically opens a new browser window with the electronic survey. When the participant clicks the submit button on the last survey page, the data are transferred to the SurveyXact server via an encrypted hyper text transfer protocol secure (HTTPS) connection, and the built-in link immediately becomes deactivated. Deactivation of the link implies that participants cannot see or edit their answers, and more importantly, no third person can access the survey data. The administration of paper surveys is also managed by PROgmatic. At specified time points, trial personnel at the coordinating trial-site receive an email with instructions to send a paper survey and a stamped envelope (addressed to the coordinating trial-site) to the participant. Completed returned paper surveys are immediately entered into the electronic survey system, SurveyXact. In addition to the ThyPRO survey, the electronic and paper questionnaires also contain questions on known adverse reactions to the trial intervention; events requiring discontinuation of the trial intervention (e.g. pregnancy); and selfreported compliance.
Clinical Trials 2014; 11: 344–354
http://ctj.sagepub.com Downloaded from ctj.sagepub.com at Copenhagen University Library on March 24, 2015
PROgmatic: a trial management system Controller program and email notifications The controller program is a collection of SAS programs located on the SAS database server. The program controls data flow and trial management and is thereby the core of PROgmatic. The program monitors email tasks and, when necessary, activates the email server. The email server sends emails to trial personnel when action needs to be taken, for example, when it is time to book a trial visit. Developers from SAS Institute A/S and SurveyXact have created an integrated solution, enabling the controller program to manage user accounts in SurveyXact. The program automatically creates users in SurveyXact, and the participant’s unique trial ID is used to create a SurveyXact user ID. At relevant time points, the program sends emails to the participant’s personal email address with a built-in link to the electronic survey. The program creates built-in links based on the participant’s SurveyXact user ID and the IDs of the electronic SurveyXact questionnaires. The program automatically imports survey data from SurveyXact to the SAS database server via an encrypted HTTPS connection. Since completed returned paper surveys are entered into the electronic survey system, the controller program can react to data from both electronic and paper surveys. This implies that participants have the same flow through the trial regardless of the mode of survey administration, and the outcomes evaluated are the same for the two cohorts (i.e., paper cohort and electronic cohort). An email alert is sent to local trial personnel in case of severe symptoms indicative of adverse reactions; events requiring discontinuation of the trial intervention; and low self-reported compliance. The program also reacts to missing data. For participants answering electronic surveys, automated reminder mails are sent to the participant after 5 and 10 days if the participant has not completed the survey. After 15 days, an email notification is sent to the local trial personnel, instructing the personnel to contact the participant and kindly ask why the survey has not been completed (Figure 2). In case of missing paper surveys, an email notification is sent to trial personnel at the coordinating trial-site after 10 days, instructing to send a postal reminder, and after 17 days, the personnel are instructed to contact the participant (Figure 2). However, the controller program is flexible in case of missed data collection, that is, the program continues to handle trial management and survey data collection if visits are skipped or surveys missed. In order to conduct intention-to-treat analyses with as little missing data as possible, it is in the interest of the trial to collect as much data from each participant as possible. Therefore, if a participant withdraws the informed consent, local trial personnel may ask which aspects of the trial the participant
349
wishes to withdraw from, that is, trial intervention, trial visits, or surveys. The controller program can manage partly discontinued participants via information from the global eCRF. All email alerts regarding the trial intervention (adverse reactions, self-reported compliance, etc.) are cancelled if trial intervention is stopped. Email notifications regarding booking of trial visits are cancelled if the participant no longer wishes to attend them. Finally, all emails concerning surveys (electronic and paper) are cancelled if the participant no longer wishes to complete them. Table 2 shows many of the automated tasks managed by the controller program. Data validation and verification Data quality is protected at two levels: data entry interfaces and data validation rules. Data entry interfaces used by trial personnel have data type control and use a skip-logic pattern, that is, data fields not relevant for the specific participant are greyed out. Predefined validation rules secure that eCRFs cannot be submitted before all necessary data are entered into PROgmatic. Submission of an incomplete eCRF will be rejected, and a visual alert will notify the trial personnel of incomplete data entry. Electronic questionnaires administered to participants have data type and data range controls. Visual alerts inform participants if one or more items are not completed. A validation test of the electronic questionnaires ensures that the survey system captures valid responses. Pilot testing Functional testing was performed systematically and included identification of functions PROgmatic was expected to perform, creation of test patients (i.e., fictive patients for whom we generate names, birth dates, disease status, etc.), and comparison of actual and expected actions. One investigator (who also functions as trial personnel) and two programmers participated in the testing procedures. Testing of the eCRFs included activation rules; valid data capture; graphical display; and ease of use. Numerous test patients were created via the inclusion eCRF, and each patient was tracked through PROgmatic to ensure that all eCRFs were activated correctly and all scheduled emails were generated and sent. The scheduled dates for generation of emails to patients and trial personnel were manipulated in order to test patient and data flow through the entire trial. Activation and deactivation rules for the reminder email function were tested. Assignments of IDs and randomisation numbers were tested. Email accounts for the test patients were created on various hosts and email systems to test the active email links.
http://ctj.sagepub.com
Clinical Trials 2014; 11: 344–354 Downloaded from ctj.sagepub.com at Copenhagen University Library on March 24, 2015
350
P Cramon et al.
Data import from the MySQL database and SurveyXact to the SAS database was tested, and validation and verification checks were performed on the imported data. Finally, the ability of the controller program to react on survey responses, including safety and compliance matters, was evaluated. All errors detected during the test phase were corrected, and PROgmatic was subsequently re-tested. This iterative process was continued until no system errors occurred. Backups and security Backup procedures ensure safe data storage. SurveyXact data are stored on a server hosted by Ramboll, and data are imported every night to the in-house SAS database server. Ramboll has implemented a running backup procedure to secure the survey data, and only technical staff from Ramboll have physical access to the servers. SurveyXact is an encrypted survey system, and the survey responses are anonymised. Nightly backup of the MySQL database and SAS database secures the remaining trial data, and the backup procedures have been tested. The web application (eCRFs) is protected by the Capital Regions firewall, and access is restricted to trial personnel with valid user ID and password. Personnel from hospitals located within the Capital Region can access the web application directly via hospital computers, while personnel from hospitals outside the Capital Region connect via the Citrix remote desktop system using an authentication token. Access to the SAS database is restricted to a few programmers from SAS Institute A/S and trial personnel from the coordinating trial-site, and valid user ID and password are required. All access to databases is logged. Running backups combined with data access logging enables tracking of trial data changes. However, the current version of the trial management system does not feature a real audit trail, that is, a database file containing all modifications to trial data (date, person who made the change, previous data value, and new data value). PROgmatic is approved by the Danish Data Protection Agency (Capital Region ID: 2007-58-0015; project ID: 30-0770).
Results Development timeline and costs The system was developed in 21 months, with three phases: initial preparation (12 months), development (6 months), and pilot testing (3 months). Initial preparation included establishment of system requirements; meetings with software companies and information technology (IT) employees from
the Capital Region healthcare system; and development of ad hoc solutions to facilitate integration of the various system components. Once the system architecture was in place, the trial-specific components (electronic surveys, database, controller program, and eCRFs) were developed, and the system was pilot tested. Capital Region had a license agreement with SurveyXact, so the only costs (~US$5000) associated with SurveyXact were due to development of ad hoc solutions. Programmers from SurveyXact spent 5 working days on layout and SAS integration. SAS programmers spent 55 working days on system development, costing approximately US$55,000. Two medical doctors with programming experience spent 140 working days on system development. Pilot testing PROgmatic has undergone rigorous pilot testing before it was released for trial management. The pilot testing revealed programming errors, as well as a system limitation. Issues detected during pilot testing are shown in Table 3. Minor adjustments made during the pilot-testing phase, for example, rewording of questions in eCRFs, are not included. A few of the detected errors are described below. Resubmission of the inclusion eCRF led to an increment of the previously assigned randomisation number. This was a crucial error, and additional coding secured that trial IDs and randomisation numbers became locked immediately after assignment. The test procedure identified two email systems that do not accept emails with active links. The built-in link was deactivated in email systems associated with http:// www.jubii.dk and http://www.facebook.com. However, the Facebook message system is not a conventional email system, and until now, no participants have provided a Facebook.com email address. Participants who use these systems need to complete paper surveys, which is a system limitation. The pilot testing showed that the SAS database, electronic surveys, and eCRFs can easily be changed while PROgmatic is running. There have been a few changes in the SAS database and eCRFs after trial launch. The changes were tested on a system replica running on an offline PC prior to implementation on the server, and the implementation was done in a time window where the controller program was inactive and no trial personnel entered data into the eCRFs. Deployment PROgmatic has been successfully implemented in the GRASS trial. The first patient was included in December 2012, and patients are currently recruited
Clinical Trials 2014; 11: 344–354
http://ctj.sagepub.com Downloaded from ctj.sagepub.com at Copenhagen University Library on March 24, 2015
PROgmatic: a trial management system
351
Table 3. Issues detected during the pilot-testing phase System componenta eCRFs Layout (1) Content (no data) Functionality (3)
Email notifications Layout (1) Content (1) Functionality (1) Electronic surveys Content (1) Functionality (1)
SAS (central) database Content (no data)
Controller program Timing of actions (1) Enrolment (1) Safety alerts (2)
Randomisation (2)
Program activation (1)
Management of eCRFs (1)
Description of the detected issue
The initial layout of the overview page in the web application was too complex The pilot testing lead to a minor revision of the content of the eCRFs. Some variables were omitted and new variables were added to the eCRFs 1) The skip-logic functionality did not work when personnel from outside the Capital Region connected to PROgmatic via the Citrix remote desktop system 2) For some variables, the value 0 was transferred to the SAS database as a missing value 3) An error occurred when personnel returned to an eCRF to modify previously entered data The layout of the initial email notifications was not formatted correctly, for example, line breaks were ignored and some words were split apart Two incorrect templates had been implemented, for example, when the self-reported compliance was too low, the local personnel received an adverse reaction alert The built-in link was deactivated in email systems associated with http://www.jubii.dk and http://www.facebook.com The baseline survey lacked questions on smoking status The electronic surveys could be accessed and edited multiple times via the built-in links. SurveyXact has a feature that immediately deactivates the built-in link when the electronic survey is completed, but this feature had not been activated The pilot testing lead to a minor revision of the database. Some variables were omitted and new variables were added to the database. Furthermore, the pilot testing identified a few variables with incorrect formats The program miscalculated dates, for example, dates of email notifications, and dates of visit 2 and 3. The error was caused by incorrect date formats Some patients that should have been enrolled were excluded by the program 1) The program did not react on the imported safety data, for example, notifications were not sent to local personnel in case of symptoms indicative of adverse reactions. It turned out that the defined critical level of adverse reactions was incorrect 2) Safety alerts were sent even though the corresponding surveys had not been completed 1) The program did not provide randomisation numbers and unique trial IDs from the correct allocation sequences 2) Resubmission of the inclusion eCRF led to an increment of the previously assigned randomisation number The controller program is activated every night to perform its intended tasks. Activation of the program failed two times during the pilot testing. The failures were due to program code variables which were not contained in the SAS database Serial activation of eCRFs is managed by the controller program. Activation of some of the eCRFs failed during pilot testing
eCRF: electronic case report form; ID: identification. a Number of detected component issues are shown in parentheses.
from the trial-sites. The trial has only minimal interference with daily clinical practice. Participants follow their usual treatment at their usual hospitals by whichever physician is involved in their treatment. PROgmatic controls data flow and trial management, and trial personnel are only involved in the management of included patients when they are prompted to take action by PROgmatic. The trial personnel have been able to use PROgmatic after a
short introduction accompanied by a brief manual. The feedback from personnel on usability and utility has been positive, and PROgmatic has handled all intended tasks properly. However, PROgmatic is monitored closely to secure that it continuously performs as intended. As of 1 November 2013, a total of 54 patients have been enrolled in the GRASS trial. Only six participants have chosen to answer paper surveys, and the
http://ctj.sagepub.com
Clinical Trials 2014; 11: 344–354 Downloaded from ctj.sagepub.com at Copenhagen University Library on March 24, 2015
352
P Cramon et al.
remaining 48 participants electronic surveys. Participants found completion of electronic surveys easy, and only a very brief instruction was needed. The importance of regularly checking the email inbox and completing surveys was underlined. A total of 132 surveys (paper or electronic) have been administered so far, and 123 surveys have been completed, yielding a survey response rate of 93%. For every 12 completed electronic surveys, only one notification regarding missing data was sent to instruct local personnel to contact the participant. Therefore, the majority (92%) of electronic surveys were collected automatically without involvement from trial personnel. The flow of paper surveys was managed by the coordinating trial-site.
Discussion Trial management and data quality control can be challenging in large multi-centre trials. We identified a need for a fully integrated trial management system with automated system features. The system should facilitate trial management and collection of high-quality data in a pragmatic trial with limited resources allocated to local trial management and limited contact between personnel and participants. The system should be able to manage the collection of both electronic and paper surveys, and it should be able to react on the incoming survey data. Furthermore, the trial management system must be feasible to use on hospital computers within the firewall of the regional healthcare information system, in order to integrate the trial management system with clinical systems. This implies high demands on security. Web-based management systems are increasingly used to support data collection and trial management in clinical trials [11–14]. However, we could not identify an available system satisfying all our needs, that is, a system with the required PRO features; facilitation of automated trial management and data collection; meeting security demands; and flexible in terms of handling skipped visits, missed surveys, and partially discontinued participants. We therefore chose to develop PROgmatic. Generic applications that could handle trial-specific tasks were identified. A generic survey module and a generic web application were implemented. We had already gained experience with the SurveyXact web questionnaire system from previous studies, and therefore chose this system to facilitate collection and entry of survey data. The effectiveness of trial data collection can be improved by replacing paper CRFs by eCRFs [15,16]. We had experience with a dynamic web application developed by SAS Institute A/S, and we chose this application to provide data entry via eCRFs. The central database and the controller program were also developed in collaboration
with SAS Institute A/S, which ensured that custom programming and system development depended on few people, rendering the development process more efficient. Designing PROgmatic in detailed flow diagrams, programming the controller program, and pilot testing PROgmatic have been major tasks in the development process. PROgmatic provides both data management and automated trial management. PROgmatic guides participant flow through the trial via serially activated eCRFs combined with automated email notifications to trial personnel. This has proven to be an effective way to manage participant flow in a trial with limited interaction between personnel and participants. The survey module automatically collects PRO data, which significantly reduces the work load of the trial personnel and reduces the risk of erroneous data entry. The majority (92%) of electronic surveys were collected automatically without involvement from trial personnel. The transfer of data from the survey module to the central database is automated. The automated reminder function improves completion rates. The design of PROgmatic allows monitoring of trial progress at several levels: participant, trial-site, and overall. The coordinating trial-site has access to the SAS database where all data on trial progress are summarised. Hence, researchers at the coordinating trial-site have immediate access to information on overall recruitment status and overall trial progress. This information is crucial for the coordination and management of a multi-centre clinical trial [11]. Safety is always a concern in clinical trials, but the monitoring and reporting of safety is a particular challenge in pragmatic trials, due to the limited contact with participants. Incorporating questions on known adverse reactions in the surveys allows PROgmatic to monitor safety, and if necessary, alert the local trial personnel. PROs are increasingly used to assess adverse events in clinical studies, and studies on safety surveillance in vaccination programmes have shown that web-based self-reporting of adverse events is a feasible approach [17,18]. However, participants in the GRASS trial were instructed to contact local trial personnel in case of symptoms indicative of adverse reactions, and the safety monitoring performed by PROgmatic was therefore an additional safety precaution. To ensure we capture serious adverse events, data on hospital admissions and mortality will be obtained through national databases at the end of the trial. PROgmatic modules can be adapted to forthcoming studies. However, a combination of generic applications and custom coding was applied to develop an integrated trial management system, meaning that adaptation to future studies is dependent on programming expertise. The custom coding was necessary to create trial-specific solutions.
Clinical Trials 2014; 11: 344–354
http://ctj.sagepub.com Downloaded from ctj.sagepub.com at Copenhagen University Library on March 24, 2015
PROgmatic: a trial management system Custom coding increases system maintenance and makes it harder to adapt PROgmatic to future studies. However, custom coding is not an entirely negative concept because it allows tailoring the system perfectly to the clinical trial. Our research group expects that PROgmatic can be used in other studies where electronic data capture and PROs are of high priority. PROgmatic is currently being adapted to two forthcoming studies, including another pragmatic multi-centre clinical trial. PROgmatic in comparison Numerous clinical trial management systems have been developed to manage data collection, including entry, storage, and processing of data. Some of these systems have been described in the scientific literature. None of the systems have the exact same features as PROgmatic, and no systems are therefore directly comparable with PROgmatic. Geyer et al. [13] developed a web-based clinical trial management system by utilising publicly available web applications, including a web questionnaire system. This strategy made it possible to develop a flexible low-cost system, which only required little professional programmer assistance. However, the system components were not integrated, which caused reduced automation of data processes, creating some inefficiency. Development of PROgmatic has relied on extensive programming expertise. In return, PROgmatic system components are fully integrated, and data transfer between components is fully automated leading to more efficient data management. Durkalski et al. [11] developed a trialspecific system that shares many features with PROgmatic, although this system does not feature a web questionnaire module. These authors determined that custom coding was more time-efficient to develop trial-specific solutions compared with the effort needed to develop generic code applications [11]. They also concluded that this strategy increased system maintenance and reduced the value of using their system for other trials [11]. Custom coding was also applied to create trial-specific solutions during the development of PROgmatic, and we agree that this strategy reduces the system’s immediate applicability to other trials. However, we believe that the custom-coded solutions can relatively easily be adapted to other studies. The current work of adapting PROgmatic to two forthcoming studies will reveal the costs and time-consumption associated with this process. Another clinical study data management system, the TrialDB, was developed at Yale University and has been applied in a large number of clinical studies (http://ycmi.med. yale.edu/trialdb/) [14]. It is one of the most
353
comprehensive trial management systems hosted at an academic institution, and it has several features currently not available in PROgmatic. TrialDB facilitates patient-entered data differently from PROgmatic. In TrialDB, the study designer can designate eCRFs as patient-enterable. Patients receive a username and password, and a web address. The patients then log in and add or edit data in individual eCRFs, and save the data [14]. PROgmatic facilitates patient-entered data via the web questionnaire module. In PROgmatic, patients receive an email with a built-in link to an electronic questionnaire at specific time points, and no user-name and password are needed. The fully automated and highly user-friendly questionnaire module makes PROgmatic perfectly suited for collection of PROs in a prospective study. Compared to other clinical trial management systems, the main distinctions of PROgmatic are numerous PRO functionalities; high number of automated features; and flexibility of PROgmatic in terms of handling skipped visits, missed surveys, and participants partially discontinued in the trial.
Conclusion We have developed an integrated trial management system that can provide high-quality data without compromising the pragmatic nature of the trial. Participants follow their usual treatment at their usual hospitals while PROgmatic automatically captures PRO data and notifies trial personnel when trialrelated actions need to be taken. The automated electronic capture of PRO data is time saving and reduces the risk of erroneous data entry. The combination of email notifications to trial personnel and serially activated eCRFs that lead patient flow through the trial has proven to be a promising way to manage a pragmatic clinical trial. PROgmatic provides a template for other pragmatic multi-centre trials with patient-reported measures as high-priority outcomes.
Acknowledgements The authors would like to thank Søren Holbech Nielsen, PhD, and Andreas Nauta Pedersen, MSc, for their dedicated efforts and commitment to develop the controller program and for integrating the different system components. Trial registration: The GRASS trial is registered at ClinicalTrials.gov (Identifier: NCT01611896).
Conflict of interest The authors declare no conflict of interest.
http://ctj.sagepub.com
Clinical Trials 2014; 11: 344–354 Downloaded from ctj.sagepub.com at Copenhagen University Library on March 24, 2015
354
P Cramon et al.
Funding This work was supported by the Danish Agency for Science, Technology and Innovation (grant 271-090143) and the Danish Council for Independent Research (grant 09-066886). L.H. is supported by an unrestricted research grant from the Novo Nordisk Foundation, and U.F.-R. from Arvid Nilsson’s Fund. S.J.B. is supported by an unrestricted research grant from The Research Council at Odense University Hospital.
10.
References
12.
1. Ware JH, Hamel MB. Pragmatic trials – Guides to better patient care? N Engl J Med 2011; 364: 1685–87. 2. Patsopoulos NA. A pragmatic view on pragmatic trials. Dialogues Clin Neurosci 2011; 13: 217–24. 3. Treweek S, Zwarenstein M. Making trials matter: Pragmatic and explanatory trials and the problem of applicability. Trials 2009; 10: 37. 4. Watt T, Cramon P, Bjorner JB, et al. Selenium supplementation for patients with Graves’ hyperthyroidism (the GRASS trial): Study protocol for a randomized controlled trial. Trials 2013; 14: 119. 5. Bell ML, Fairclough DL. Practical and statistical issues in missing data for longitudinal patient reported outcomes. Stat Methods Med Res 2013; doi: 10.1177/ 0962280213476378 (Epub ahead of print). 6. Hardouin JB, Conroy R, Sebille V. Imputation by the mean score should be avoided when validating a Patient Reported Outcomes questionnaire by a Rasch model in presence of informative missing data. BMC Med Res Methodol 2011; 11: 105. ¨ s L. Treatment of Graves’ hyperthyroidism: Evi7. Hegedu dence-based and emerging modalities. Endocrinol Metab Clin North Am 2009; 38: 355–71. 8. Watt T, Rasmussen AK, Groenvold M, et al. Improving a newly developed patient-reported outcome for thyroid
9.
11.
13.
14.
15.
16.
17.
18.
patients, using cognitive interviewing. Qual Life Res 2008; 17: 1009–17. Watt T, Bjorner JB, Groenvold M, et al. Establishing construct validity for the thyroid-specific patient reported outcome measure (ThyPRO): An initial examination. Qual Life Res 2009; 18: 483–96. ¨ s L, Groenvold M, et al. Validity and Watt T, Hegedu reliability of the novel thyroid-specific quality of life questionnaire, ThyPRO. Eur J Endocrinol 2010; 162: 161–67. Durkalski V, Wenle Z, Dillon C, Kim J. A web-based clinical trial management system for a sham-controlled multicenter clinical trial in depression. Clin Trials 2010; 7: 174–82. Musick BS, Robb SL, Burns DS, et al. Development and use of a web-based data management system for a randomized clinical trial of adolescents and young adults. Comput Inform Nurs 2011; 29: 337–43. Geyer J, Myers K, Vander SA, et al. Implementing a lowcost web-based clinical trial management system for community studies: A case study. Clin Trials 2011; 8: 634–44. Brandt CA, Deshpande AM, Lu C, et al. TrialDB: A webbased Clinical Study Data Management System. AMIA Annu Symp Proc 2003; 2003: 794. Eisenstein EL, Collins R, Cracknell BS, et al. Sensible approaches for reducing clinical trial costs. Clin Trials 2008; 5: 75–84. Lopez-Carrero C, Arriaza E, Bolanos E, et al. Internet in clinical research based on a pilot experience. Contemp Clin Trials 2005; 26: 234–43. Wade AG, Crawford GM, Pumford N, McConnachie A. Patient reported outcome data following influenza A (H1N1p) vaccination in the 2009–2010 season: Webbased and telephone evaluation. Ther Clin Risk Manag 2011; 7: 409–20. Mackenzie IS, MacDonald TM, Shakir S, et al. Influenza H1N1 (swine flu) vaccination: A safety surveillance feasibility study using self-reporting of serious adverse events and pregnancy outcomes. Br J Clin Pharmacol 2012; 73: 801–11.
Clinical Trials 2014; 11: 344–354
http://ctj.sagepub.com Downloaded from ctj.sagepub.com at Copenhagen University Library on March 24, 2015
