Interobserver variability in the interpretation of computed tomography ...

J Neurosurg 115:1191–1196, 2011

Interobserver variability in the interpretation of computed tomography following aneurysmal subarachnoid hemorrhage Clinical article George M. Ibrahim, M.D.,1,2 Stefan Weidauer, M.D., 3 and R. Loch Macdonald, M.D., Ph.D.1,2 Division of Neurosurgery, St. Michael’s Hospital, Labatt Family Centre of Excellence in Brain Injury and Trauma Research, Keenan Research Centre of the Li Ka Shing Knowledge Institute of St. Michael’s Hospital, Toronto; 2Department of Surgery, University of Toronto, Ontario, Canada; and 3Institute of Neuroradiology, Johann Wolfgang Goethe-Universität Frankfurt, Germany

1

Object. Numerous abnormal findings may be evident on CT scans after aneurysmal subarachnoid hemorrhage (SAH). Here, the authors assess the interobserver variability in the radiological interpretation of the initial CT scan following SAH. Methods. Two experienced reviewers, a neurosurgeon and a neuroradiologist, independently prospectively reviewed the initial CT scans of 413 patients enrolled in the CONSCIOUS-1 trial. Measured variables included SAH, intraventricular hemorrhage, intracerebral hemorrhage, subdural hematoma, chronic infarction, midline shift, and hydrocephalus. To assess interobserver variability, weighted kappa values and intraclass correlation coefficients (ICCs) were calculated and Bland-Altman analysis was performed. Results. Moderate to substantial agreement was found for most of the CT scanning findings. There was fair to moderate interobserver agreement between reviewers when determining the extent of SAH based on a descriptive categorical classification (kappa 0.41; 95% CI 0.33–0.49), and better agreement when a semiquantitative scale was used (ICC 0.56; 95% CI 0.49–0.62). There was poor agreement between reviewers for the presence of hydrocephalus (kappa 0.34; 95% CI 0.20–0.48), but substantial to near perfect agreement on ventriculocranial ratio measurements (ICC 0.77; 95% CI 0.72–0.81). Conclusions. The authors’ findings suggest that there is considerable interobserver variability in the interpretation of CT scans after SAH. Quantitative measures may reduce interobserver variability in comparison with qualitative or categorical scales. Variability in interpretation of CT scans has implications for patient care and conduct of clinical trials. It may be beneficial to develop standardized assessments to ensure consistent evaluation of measured variables. (DOI: 10.3171/2011.7.JNS11725)

Key Words • interobserver variability • subarachnoid hemorrhage • computed tomography • vascular disorders

T

initial CT study following aneurysmal SAH is a valuable study. Numerous abnormal findings may be detected, which influence the clinician’s choice and timing of intervention. The presence and extent of hydrocephalus, intra- or extraaxial hematomas, and midline shift may prompt surgeons to perform interventions ranging from insertion of a ventriculostomy to decompressive craniectomy. Furthermore, the extent of subarachnoid blood predicts angiographic vasospasm, delayed cerebral ischemia, and patient outcome.1,6,10 The CT findings are, however, subject to interobservhe

Abbreviations used in this paper: EDH = epidural hematoma; ICC = intraclass correlation coefficient; ICH = intracerebral hemorrhage; IVH = intraventricular hemorrhage; SAH = subarachnoid hemorrhage; SDH = subdural hematoma; VCR = ventriculocranial ratio.

J Neurosurg / Volume 115 / December 2011

er variability. The literature contains 3 studies evaluating interobserver agreement on extent of SAH and ventricular size after SAH.21,23,26 In each, concerns were raised regarding considerable interobserver variability. Here, we present the largest series evaluating interobserver variability of numerous CT findings following aneurysm rupture. Study Population

Methods

We conducted a post hoc analysis of 413 patients enrolled between January 2005 and March 2006 in CONSCIOUS-1 (Clazosentan to Overcome Neurological iSChemia and Infarction OccUrring after Subarachnoid hemorrhage), a randomized, double-blinded, placebocontrolled phase 2 dose-finding trial of clazosentan for 1191

G. M. Ibrahim, S. Weidauer, and R. L. Macdonald prevention of angiographic vasospasm after aneurysmal SAH.14 The methods and results have been published.14 Eligible patients were 18–70 years of age with confirmed SAH. Two hundred eighty-nine (70%) were women, and the mean age was 51 ± 11 years. Computed Tomography Scanning Analysis

All patients underwent CT scanning performed at baseline, which showed SAH. Patients then underwent catheter angiography, which confirmed the presence of a ruptured intracranial saccular aneurysm. Subarachnoid blood on the CT scans was graded using a descriptive categorical scale as diffuse (long axis ≥ 20 mm) or localized (long axis < 20 mm) and thick (short axis ≥ 4 mm) or thin (short axis < 4 mm). The amount of subarachnoid blood in fissures and cisterns was also measured using the semiquantitative Hijdra scale.9 This calculates a score for subarachnoid clot burden visualized in 10 fissures and cisterns: interhemispheric fissure, basal aspect of sylvian fissure (right and left), lateral aspect of sylvian fissure (right and left), suprasellar cistern (right and left), ambient cistern (right and left), and quadrigeminal cistern. Each is assigned a score of 1 (no blood), 2 (small amount of blood), or 3 (completely filled with blood) for a cumulative score out of 30. Intraventricular hemorrhage was assessed using a modification of the Graeb score, in which a score of 0 (no blood), 1 (sedimentation, less than 25% filled), 2 (moderately filled), or 3 (completely filled) was determined for each ventricle for a maximum possible score of 12.8,13 Hydrocephalus was assessed by subjective assessment of ventricular size and calculation of the VCR.25 The presence and extent of SDH, EDH, and ICH were assessed, as well as midline shift and infarction. Volumes of ICH and infarctions were approximated by measuring the length (A), width (B), and height (C) and using the formula (A+B+C)/2. All images were prospectively and centrally reviewed by 2 independent reviewers, a neurosurgeon (R.L.M) and a neuroradiologist (S.W.). Reviewers were able to manipulate the display window settings to identify pertinent abnormalities. A third adjudicator independently assessed clot size when discrepancies arose between the 2 reviewers. Statistical Analysis

Interobserver variability expresses degree of agreement among raters or the reproducibility of a measurement. To determine the interobserver variability in the presence of IVH, ICH, SDH, infarction, midline shift, and hydrocephalus, as well as the categorical classification of SAH, weighted kappa values were calculated.11 Interobserver variability in the Hijdra and modified Graeb scores and VCR were assessed using the ICC.19 Interobserver variability was interpreted as poor (< 0.20), fair (0.21–0.40), moderate (0.41–0.60), substantial (0.61– 0.80), and almost perfect (0.81–1.00) agreement based on the respective kappa and ICC scores. Bland-Altman plotting was performed for Hijdra scores, modified Graeb scores, and VCR values.3 The mean difference between reviewers and a 1.96 SD were calculated. Analysis was performed using SAS 9.1 (SAS, Inc.). 1192

Results

Both reviewers agreed that none of the scans were normal and that no acute infarctions or epidural hematomas were present (kappa 1.0). Frequency of abnormal findings and interobserver agreement are summarized in Tables 1 and 2, respectively. Subarachnoid Hemorrhage

When the qualitative, categorical classification criteria were used (by classifying SAH as thick or thin and diffuse or localized), fair to moderate agreement was present (kappa 0.41; 95% CI 0.33–0.49). When discrepancies in subarachnoid clot size classification arose between reviewers, an adjudicator was involved. The adjudicator agreed with Reviewer 1 in 172 cases (65%) and with Reviewer 2 in 94 cases (35%). The use of the Hijdra scale improved reliability in the assessment of SAH (ICC 0.56 [95% CI 0.49–0.62]). The Pearson coefficient for the correlation between the 2 reviewers was 0.73 (Fig. 1A). Bland-Altman analysis (Fig. 1B) demonstrated a mean difference of 3.9 with 1.96 and -1.96 SDs of 13.5 and -5.0, respectively. Only 3.6% of data were outside 2 SDs of the mean. Interestingly, agreement between reviewers on the cumulative score was higher than agreement on the amount of subarachnoid blood within the individual cisterns or fissures (kappa = 0.30–0.53). Better agreement was elicited for certain locations of SAH. For example, there was better agreement on the lateral aspects of the sylvian fissure (kappa 0.53 [95% CI 0.48–0.60] and 0.51 [95% CI 0.45–0.58], left and right, respectively) compared with the basal aspects (kappa 0.36 [95% CI 0.30–0.43] and 0.37 [95% CI 0.31–0.43], left and right, respectively). Within the basal cisterns, there was moderate agreement between reviewers for the quadrigeminal and ambient cisterns (kappa 0.41–0.52), but poor agreement on the amount of subarachnoid blood in the suprasellar cisterns (kappa 0.30 and 0.35, left and right, respectively).

Intraventricular Hemorrhage

There was substantial agreement between reviewers on the presence of IVH (kappa 0.70 [95% CI 0.61–0.79]). Better agreement was elicited when the semiquantitative modified Graeb score was used (ICC 0.77 [95% CI 0.72– 0.81]). The Pearson correlation coefficient between the 2 reviewers was 0.80 and Bland-Altman analysis showed 6.1% of data greater than 2 SDs from the mean (Fig. 1B and C). Again, agreement on the cumulative score was higher than agreement on the amount of IVH within the individual ventricles (kappa 0.52–0.65). There was better agreement between reviewers on the amount of blood within the third (kappa 0.65 [95% CI 0.59–0.71]) and fourth (kappa 0.61 [95% CI 0.54–0.69]) ventricles compared with the lateral ventricles (kappa 0.52 [95% CI 0.44–0.61] and 0.53 [95% CI 0.45–0.61], left and right, respectively).

Intracranial Hemorrhage

There was substantial agreement between reviewers J Neurosurg / Volume 115 / December 2011

Interobserver variability of CT findings after SAH TABLE 1: Frequency of abnormal findings on CT scans in 413 patients with aneurysmal SAH* CT Finding SAH diffuse/thick diffuse/thin local/thick local/thin mean Hijdra score IVH present mean Graeb score ICH present mean no. (when present) mean vol (ml) SDH present mean max thickness (mm) chronic infarction present mean no. (when present) mean vol (ml) midline shift septal shift pineal shift hydrocephalus present mean VCR

Reviewer 1

Reviewer 2

280 (68) 84 (20) 34 (8) 4 (1) 18.3 ± 5.9

223 (54) 173 (42) 6 (1) 1 (0.2) 14.2 ± 6.3

313 (76) 3.9 ± 2.3

332 (80) 3.6 ± 2.0

50 (12) 1.0 ± 0.2 14.3 ± 21.7

79 (19) 1.0 ± 0.2 14.5 ± 25.0

13 (3) 5.8 ± 2.5

10 (2) 5.7 ± 2.4

32 (8) 1.2 ± 0.4 1.7 ± 3.0 16 (4) 12 (3) 16 (4)

39 (9) 1.1 ± 0.4 2.5 ± 6.1 26 (6) 26 (6) 15 (4)

375 (91) 0.17 ± 0.04

346 (84) 0.19 ± 0.04

* Numbers in parentheses denote percentages; error ranges represent standard deviation.

for the presence of ICH (kappa 0.72 [95% CI 0.63–0.82]). There was also substantial agreement on the number of distinct ICHs (kappa 0.71 [95% CI 0.62–0.80]). The occurrence of ICH was too low to make meaningful estimates of agreement on ICH size. Moderate agreement was found for the presence of SDH (kappa 0.44 [95% CI 0.17–0.70]). The low incidence of these also precluded meaningful estimates of agreement on SDH size. Infarction and Midline Shift

There was moderate agreement between reviewers on the presence of chronic infarctions on CT scanning (kappa 0.60 [95% CI 0.46–0.75]). Moderate agreement was also associated with the number of distinct chronic infarctions (kappa 0.50 [95% CI 0.36–0.65]). Substantial agreement on midline shift was found between reviewers (kappa 0.70 [95% CI 0.54–0.87]). This was consistent for shift of both the pineal gland and septum pellucidum (kappa 0.68 [95% CI 0.47–0.89] and 0.70 [95% CI 0.54– 0.87], respectively). Due to infrequent occurrence, meaningful analysis of size of chronic infarcts and extent of midline shift was not possible.


TABLE 2: Observer agreement in the interpretation of CT scanning findings after SAH CT Scanning Finding

Binary or Categorical Outcome Measure, Kappa (95% CI)

SAH

0.41 (0.33–0.49)

IVH

0.70 (0.61–0.79)

ICH SDH chronic infarction midline shift septal midline shift pineal midline shift hydrocephalus

0.72 (0.63–0.82) 0.44 (0.17–0.70) 0.60 (0.46–0.75) 0.70 (0.54–0.87) 0.70 (0.54–0.87) 0.68 (0.47–0.89) 0.34 (0.20–0.48)

Quantitative Outcome Measure, ICC (95% CI) Hijdra Score: 0.56 (0.49–0.62) Graeb Score: 0.77 (0.72–0.81)

VCC: 0.77 (0.72–0.81)

Hydrocephalus

There was poor agreement between reviewers on the presence of hydrocephalus (kappa 0.34 [95% CI 0.20–0.48]). However, substantial agreement was found for the VCR (ICC 0.77 [95% CI 0.72–0.81]). The Pearson correlation coefficient between the 2 reviewers was 0.88 with 2.5% of data deviating over 2 SDs from the mean on Bland-Altman analysis (Fig. 1D and E). Additionally, when individual components of the VCR were compared, there was near-perfect agreement on measurements of ventricular width and brain width (0.85 [95% CI 0.82– 0.88] and 0.99 [95% CI 0.98–0.99], respectively).

Discussion

Here, we assess the interobserver variability in the interpretation of the initial CT scanning study obtained after SAH. Moderate to substantial agreement was found for most of the CT scanning findings. Considerable variability was obtained for certain outcome measures, including hydrocephalus. Our findings furthermore suggest that semiquantitative scores for SAH, IVH, and hydrocephalus improve interobserver agreement. Furthermore, often the cumulative score is more reliable than the individual cisternal or ventricular scores from which it is composed. It is often impractical to rely on an absolute threshold such as the VCR for interventions such as insertion of a shunt, as such decisions are often based on other clinical information. However, the severity of radiological findings play an important role in guiding the decisionmaking process. For example, van Gijn et al.25 defined hydrocephalus after SAH as ventriculomegaly in excess of the 95th percentile, and this threshold was used in determining whether a shunt was inserted. Similarly, other radiographic features, such as the evacuation of SDHs for clot size greater than 1 cm or midline shift in excess of 0.5 cm, often lead clinicians to intervene.4 Our findings suggest that quantitative outcomes may be a more robust guide for clinical decision-making than categorical scales. This must be interpreted with caution as this study 1193

G. M. Ibrahim, S. Weidauer, and R. L. Macdonald

Fig. 1. A: Scatter diagram of Hijdra score for 2 reviewers in 413 patients. B: Bland-Altman plot of interobserver variability in interpretation of subarachnoid clot burden using the Hijdra score. Upper and lower bars represent 95% limits of agreement, and the center bar represents the mean. C and D: Scatter diagram (C) and Bland-Altman analysis (D) for intraventricular clot burden measured using the modified Graeb score. E and F: Scatter diagram (E) and Bland-Altman analysis (F) for hydrocephalus measured using the VCR.

does not attempt to evaluate the effect of interobserver variability on therapeutic decision-making. In addition, clinical trials require standardization in methodology and assessment of measured outcomes. Meta-analyses also often combine data from multiple trials, and therefore uniformity in interpretation of imaging findings is necessary. We suggest that quantitative outcomes may facilitate consistent evaluation of measured outcomes by decreasing interobserver variability. This trend has been observed previously in other disease conditions. One study assessed postresection imaging of gliomas, where better agreement was present when changes in tumor size were measured directly compared with a categorical classification system.28 In their assessment of interobserver agreement of 1194

classification of hydrocephalus and SAH using the Fisher grade, Svensson et al.21 found moderate to substantial agreement between reviewers (kappa 0.50–0.63). Reasons for variability were primarily disagreement between reviewers on the most clinically relevant components of the staging system (that is, extent of subarachnoid clot or intraventricular blood) and too few categories.6 Van Norden et al.26 noted higher interobserver agreement on Hij dra scores (kappa 0.67–0.75) than Fisher grades (kappa 0.37–0.55). The Fisher grading system has been subject to criticisms and instead, we chose a categorical, descriptive system, classifying SAH as thin/thick and local/ diffuse.5,7,18,20 However, we observed similar trends and suggest that better reliability is obtained by minimizing subjectivity in the interpretation of CT findings. J Neurosurg / Volume 115 / December 2011

Interobserver variability of CT findings after SAH It is perhaps not surprising that subjective interpretation of the presence of hydrocephalus was associated with poor agreement. Other studies have also found relatively high variability in the assessment of the presence or absence of hydrocephalus.2 Significant ventriculomegaly with transependymal edema is easily detected; however, signs such as prominent temporal horns, altered morphology of the frontal horns, and compression or displacement of the thalami may be more subtle.16 Numerous quantitative methods aim to assess hydrocephalus.15,17,29 While our findings suggest that quantitative assessments are more reliable than subjective binary or categorical outcomes, reviewers must be cognizant that these summarize complex 3D structures with a 2D relationship determined on a single slice. Slice selection and asymmetry in the ventricular system may lead to reduced reliability, and contributions of the third and fourth ventricles are ignored altogether.22,29 Volumetric techniques may be an attractive alternative; however, at present, the time and specialized software required for such calculations preclude their use in patients with SAH. Numerous studies have examined methods to increase interobserver agreement. In the assessment of cerebral infarct volumes, van der Worp et al.24 suggested that manual tracing of the infarct perimeter is the most reproducible method. This is consistent with our findings that reviewer calculations for outcome variables such as VCR are reliable. We further add that these are superior to subjective assessments of abnormal findings. In their assessment of brain atrophy, van Zagten and colleagues27 found similarly high interobserver agreement for the VCR (ICC 0.82; 95% CI 0.75–0.94). Interestingly, a similar study examining brain atrophy found substantially less agreement for subjective estimation of brain atrophy based on qualitative categorical scales (kappa 0.11–0.59).12 The main limitation of this study is the use of only 2 reviewers in the interpretation of the CT scan. Although increasing the number of reviewers would render the data more robust and improve the validity of the study, both reviewers have extensive experience in the assessment of SAH, highlighting the existence of interobserver variability even among experts. An additional study to test interobserver agreement between reviewers of various expertise may further identify the best scale to measure and generalize the findings to a broader clinical setting. Furthermore, images were obtained from multiple centers with equipment from many manufacturers, which may contribute to discrepancies identified. Strengths include the large patient population, the systematic documentation of CT findings, as well as confirmation of aneurysmal etiology of SAH with conventional angiography.

Conclusions

The findings on initial CT scans after aneurysmal SAH may affect clinicians’ decisions regarding choice and timing of intervention. In this study, the initial CT scans of 413 patients with SAH were assessed by 2 independent, experienced reviewers. We found that semiquantitative assessment measures improve interobserver agreement compared with categorical or descriptive J Neurosurg / Volume 115 / December 2011

scales. In addition to possible implications on patient care, this has important applications for design of clinical trials, highlighting the requirement for standardized assessments and processes to ensure consistent evaluation of measured outcomes. Disclosure Actelion Pharmaceuticals, Ltd., was the sponsor of the CONSCIOUS-1 trial, and they provided financial support for the exploratory analysis described in this article. The data analysis and writing are the work of the authors. R. Loch Macdonald is the chief scientific officer of Edge Therapeutics, Inc., and is a consultant for Actelion Pharmaceuticals, Ltd. Author contributions to the study and manuscript preparation include the following. Conception and design: Macdonald. Acquisition of data: Macdonald, Weidauer. Analysis and interpretation of data: Ibrahim. Drafting the article: Ibrahim. Critically revising the article: all authors. Reviewed submitted version of manuscript: all authors. Approved the final version of the manuscript on behalf of all authors: Macdonald. Statistical analysis: Ibrahim. Study supervision: Macdonald. References 1. Adams HP Jr, Kassell NF, Torner JC, Haley EC Jr: Predicting cerebral ischemia after aneurysmal subarachnoid hemorrhage: influences of clinical condition, CT results, and antifibrinolytic therapy. A report of the Cooperative Aneurysm Study. Neurology 37:1586–1591, 1987 2. Bhattathiri PS, Gregson B, Prasad KS, Mitchell P, Soh C, Mitra D, et al: Reliability assessment of computerized tomography scanning measurements in intracerebral hematoma. Neurosurg Focus 15(4):E6, 2003 3. Bland JM, Altman DG: Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1:307–310, 1986 4. Bullock MR, Chesnut R, Ghajar J, Gordon D, Hartl R, Newell DW, et al: Surgical management of acute subdural hematomas. Neurosurgery 58 (3 Suppl):S16–S24, 2006 5. Claassen J, Bernardini GL, Kreiter K, Bates J, Du YE, Copeland D, et al: Effect of cisternal and ventricular blood on risk of delayed cerebral ischemia after subarachnoid hemorrhage: the Fisher scale revisited. Stroke 32:2012–2020, 2001 6. Fisher CM, Kistler JP, Davis JM: Relation of cerebral vasospasm to subarachnoid hemorrhage visualized by computerized tomographic scanning. Neurosurgery 6:1–9, 1980 7. Frontera JA, Claassen J, Schmidt JM, Wartenberg KE, Temes R, Connolly ES Jr, et al: Prediction of symptomatic vasospasm after subarachnoid hemorrhage: the modified Fisher scale. Neurosurgery 59:21–27, 2006 8. Graeb DA, Robertson WD, Lapointe JS, Nugent RA, Harrison PB: Computed tomographic diagnosis of intraventricular hemorrhage. Etiology and prognosis. Radiology 143:91–96, 1982 9. Hijdra A, Brouwers PJ, Vermeulen M, van Gijn J: Grading the amount of blood on computed tomograms after subarachnoid hemorrhage. Stroke 21:1156–1161, 1990 10. Kistler JP, Crowell RM, Davis KR, Heros R, Ojemann RG, Zervas T, et al: The relation of cerebral vasospasm to the extent and location of subarachnoid blood visualized by CT scan: a prospective study. Neurology 33:424–436, 1983 11. Landis JR, Koch GG: The measurement of observer agreement for categorical data. Biometrics 33:159–174, 1977 12. Leonardi M, Ferro S, Agati R, Fiorani L, Righini A, Cristina E, et al: Interobserver variability in CT assessment of brain atrophy. Neuroradiology 36:17–19, 1994 13. LeRoux PD, Haglund MM, Newell DW, Grady MS, Winn HR:

1195

G. M. Ibrahim, S. Weidauer, and R. L. Macdonald Intraventricular hemorrhage in blunt head trauma: an analysis of 43 cases. Neurosurgery 31:678–685, 1992 14. Macdonald RL, Kassell NF, Mayer S, Ruefenacht D, Schmiedek P, Weidauer S, et al: Clazosentan to overcome neurological ischemia and infarction occurring after subarachnoid hemorrhage (CONSCIOUS-1): randomized, double-blind, placebocontrolled phase 2 dose-finding trial. Stroke 39:3015–3021, 2008 15. Mesiwala AH, Avellino AM, Ellenbogen RG: The diagonal ventricular dimension: a method for predicting shunt malfunction on the basis of changes in ventricular size. Neurosurgery 50:1246–1252, 2002 16. Naidich TP, Schott LH, Baron RL: Computed tomography in evaluation of hydrocephalus. Radiol Clin North Am 20:143– 167, 1982 17. O’Hayon BB, Drake JM, Ossip MG, Tuli S, Clarke M: Frontal and occipital horn ratio: a linear estimate of ventricular size for multiple imaging modalities in pediatric hydrocephalus. Pediatr Neurosurg 29:245–249, 1998 18. Shimizu H: Re: Subarachnoid hemorrhage on computed tomography scanning and the development of cerebral vasospasm: the Fisher grade revisited (Smith ML et al. Surg Neurol 2005;63:229-35). Surg Neurol 66:551–552, 2006 (Letter) 19. Shrout PE, Fleiss JL: Intraclass correlations: uses in assessing rater reliability. Psychol Bull 86:420–428, 1979 20. Smith ML, Abrahams JM, Chandela S, Smith MJ, Hurst RW, Le Roux PD: Subarachnoid hemorrhage on computed tomography scanning and the development of cerebral vasospasm: the Fisher grade revisited. Surg Neurol 63:229–235, 2005 21. Svensson E, Starmark JE, Ekholm S, von Essen C, Johansson A: Analysis of interobserver disagreement in the assessment of subarachnoid blood and acute hydrocephalus on CT scans. Neurol Res 18:487–494, 1996 22. Sze RW, Ghioni V, Weinberger E, Seidel KD, Ellenbogen RG: Rapid computed tomography technique to measure ventricular volumes in the child with suspected ventriculoperitoneal shunt failure II. Clinical application. J Comput Assist Tomogr 27:668–673, 2003

1196

23. van der Jagt M, Hasan D, Bijvoet HW, Pieterman H, Koudstaal PJ, Avezaat CJ: Interobserver variability of cisternal blood on CT after aneurysmal subarachnoid hemorrhage. Neurology 54:2156–2158, 2000 24. van der Worp HB, Claus SP, Bär PR, Ramos LM, Algra A, van Gijn J, et al: Reproducibility of measurements of cerebral infarct volume on CT scans. Stroke 32:424–430, 2001 25. van Gijn J, Hijdra A, Wijdicks EF, Vermeulen M, van Crevel H: Acute hydrocephalus after aneurysmal subarachnoid hemorrhage. J Neurosurg 63:355–362, 1985 26. van Norden AG, van Dijk GW, van Huizen MD, Algra A, Rinkel GJ: Interobserver agreement and predictive value for outcome of two rating scales for the amount of extravasated blood after aneurysmal subarachnoid haemorrhage. J Neurol 253:1217–1220, 2006 27. van Zagten M, Kessels F, Boiten J, Lodder J: Interobserver agreement in the assessment of cerebral atrophy on CT using bicaudate and sylvian-fissure ratios. Neuroradiology 41:261– 265, 1999 28. Vos MJ, Uitdehaag BM, Barkhof F, Heimans JJ, Baayen HC, Boogerd W, et al: Interobserver variability in the radiological assessment of response to chemotherapy in glioma. Neurology 60:826–830, 2003 29. Woods BT, Douglass A, Gescuk B: Is the VBR still a useful measure of changes in the cerebral ventricles? Psychiatry Res 40:1–10, 1991

Manuscript submitted April 24, 2011. Accepted July 12, 2011. Please include this information when citing this paper: published online August 5, 2011; DOI: 10.3171/2011.7.JNS11725. Address correspondence to: R. Loch Macdonald, M.D., Ph.D., St. Michael’s Hospital, 30 Bond Street, Toronto, Ontario M5B 1W8, Canada. email: [email protected].
