SIGNAL SOLUTIONS ... Paper-based assessments also require manual transcription of the data for entry into an electronic
Virgil Investigative Study Platform: Improving Signal Detection in Psychiatry Clinical Trials Williams, JBW1,2, Negash, S1, Webber-Lind, B1, Stein, L1, Randolph, C1,3 1
MedAvante, Inc., 2 Dept. Of Psychiatry, Columbia University College of Physicians and Surgeons,
3 Loyola
University Medical Center
BACKGROUND • The Hamilton Depression Rating Scale (HAM-D) is one of the most widely used endpoints in clinical trials of major depression4. While the Structured Interview Guide for HAM-D (SIGH-D) has helped standardize the administration and scoring of the scale, paper-based data collection is still prone to errors which contribute to poor interrater reliability.
• Clinical trials of treatments for major depressive disorder (MDD) suffer from high placebo response rates that lead to inconclusive results1, 2. • The very high failure rate in these trials is in part due to inconsistent and inefficient psychiatric assessments.
• The use of a tablet device-based electronic clinical outcome assessments (eCOA) system for digital data collection has obvious advantages: eliminating calculation errors, flagging missing data, eliminating source data verification and reducing site burden associated with central oversight.
• Traditional paper-based clinical outcome assessments (COA) are prone to high error rates. • Paper-based assessments also require manual transcription of the data for entry into an electronic data capture (EDC) system, introducing additional prospects for errors. Monitoring the accuracy of transcription (source data verification) is both costly and ineffective3.
®
• The Virgil Investigative Study Platform enables eCOA collection of digital electronic source (eSource) data and provides real-time clinical guidance to standardize measurements and improve data quality.
• Attempts to implement central oversight typically involve scanning a large volume of paper source and uploading audio or video files, further adding to the administrative burden at trial sites.
• The purpose of this study was to examine error rates in psychiatry trials by comparing traditional paper-and-pencil administration of SIGH-D with administration of the same scale using the Virgil tablet device.
METHODS • Paper-based assessments of the seventeen-item SIGH-D administered in a phase II MDD trial were compared against eCOA administrations of the same scale in a separate phase III MDD trial.
• The first 250 administrations of SIGH-D centrally reviewed for the pencil-and-paper trial were compared to the first 250 administrations completed with the Virgil tablet.
• Assessments in both modes of administration were reviewed by the same cohort of expert calibrated clinicians who listened to audio recordings with copies of worksheets to identify scoring discrepancies.
• The percentage of reviews with two or more discrepancies were compared between paper-based and Virgil administrations. Item-level discrepancies were also examined.
RESULTS • Figure 1 shows the percentages of reviews with discrepancies for paper-based and Virgil administrations for SIGH-D. As the figure indicates, a substantially higher number of Virgil administrations had no discrepancies, and the percentage of reviews with two or more discrepancies was lower in Virgil administrations compared to paper-based.
Work & Activities NO ≥2 DISCREPANCIES DISCREPANCIES
n=250
Reviews with no discrepancies
ITEM
Depressed Mood
Figure 1: Percentage of SIGH-D Reviews with Discrepancies
Paper/Pencil Administration
Figure 2: SIGH-D Item Discrepancies
#
%
#
%
PAPER-PENCIL
37
15%
168
67%
VIRGIL
96
38%
105
42%
Virgil Administration n=250
Insomnia Early Insomnia Middle Insomnia Late Genital Symptoms Somatic Symptoms Gastrointestinal Loss Of Weight Somatic Symptoms General
PAPERPENCIL
VIRGIL
36% 30% 9% 12% 12% 6% 11% 8% 16%
18% 19% 9% 10% 13% 2% 3% 4% 7%
ITEM
Feelings Of Guilt Suicide Anxiety Psychic Anxiety Somatic Hypochondriasis Insight Agitation Retardation
PAPERPENCIL
VIRGIL
32% 10% 34% 20% 25% 1% 0% 4%
21% 4% 12% 6% 17% 3% 2% 4%
Reviews with discrepancies
CONCLUSION
• Analysis of variance (ANOVA) comparing the number of errors per administration revealed significantly lower error rates on Virgil versus paper-based administration of the SIGH-D (F (1,498) = 45.9 p
