Biomarkers for diagnosis of Pediatric Acute

Journal of Neuroimmunology xxx (xxxx) xxx–xxx

Contents lists available at ScienceDirect

Journal of Neuroimmunology journal homepage: www.elsevier.com/locate/jneuroim

Biomarkers for diagnosis of Pediatric Acute Neuropsychiatric Syndrome (PANS) – Sensitivity and specificity of the Cunningham Panel Eva Hesselmarka,⁎, Susanne Bejerota,b,c a b c

Center for Psychiatry Research, Department of clinical neuroscience, Karolinska Institutet, CAP Research Centre, Gävlegatan 22 B 8tr, 113 30, Stockholm, Sweden School of Medical Sciences, Örebro University, Örebro, Sweden University Health Care Research Center, Faculty of Medicine and Health, Örebro University, Örebro, Sweden

A R T I C L E I N F O

A B S T R A C T

Keywords: PANDAS PANS Obsessive-compulsive disorder Sensitivity and specificity Biomarkers Antibodies Calcium/calmodulin kinase II

Objective: Pediatric Acute Neuropsychiatric Syndrome (PANS) and pediatric autoimmune neuropsychiatric disorders associated with streptococcal infections (PANDAS) are conditions marked by sudden onset of obsessive-compulsive disorder (OCD), tics, or avoidant/restrictive food intake in combination with multiple psychiatric symptoms. A diagnosis of PANS or PANDAS may be supported by the Cunningham Panel, a commercially available set of immunologic assays currently in clinical use. However, the relationship between Cunningham Panel results and patient symptoms remains unclear. This study was done to assess the diagnostic accuracy of the Cunningham Panel in patients with suspected PANS or PANDAS. Method: All Swedish patients who had taken the Cunningham Panel prior to June 2014 (n = 154) were invited and 53 patients participated in the study. Based on comprehensive psychiatric assessment (the reference standard of diagnosis), subjects were classified as PANS, PANDAS, or neither. Prior Cunningham Panel test results were collected from patient records, and new blood samples were similarly analyzed within the scope of this study. In addition, results were compared to healthy controls (n = 21) and a test-retest reliability analysis was performed. Results: Sensitivities of individual biomarkers in the Cunningham Panel ranged from 15 to 60%, and specificities from 28 to 92%. Positive predictive values ranged from 17 to 40%, and negative predictive values from 44 to 74%. A majority of the healthy controls had pathological Cunningham Panel results and test-retest reliability proved insufficient. Conclusion: Clinical use of the Cunningham Panel in diagnosing PANS or PANDAS is not supported by this study.

1. Introduction There is growing clinical and scientific evidence to support the concept of Pediatric Acute Neuropsychiatric Syndrome (PANS) in children undergoing assessment, diagnosis, and treatment of sudden onset obsessive-compulsive disorder (OCD) or severely restricted food intake (Murphy et al., 2015). Proposed diagnostic criteria for PANS are as follows: 1) sudden onset (< 72 h) of OCD or eating restriction; 2) at least two qualifying attributes (anxiety; mood or behavior disturbances; irritability or aggression; developmental regression; deterioration in school performance; sensory or motor abnormalities; and somatic symptoms); and 3) lack of a known medical or neurologic disorder to better explain symptoms (Swedo et al., 2012). Use of the term PANS has evolved from literature on Pediatric Autoimmune Neuropsychiatric Disorders Associated with Streptococcal infections (PANDAS; Swedo et al., 1998). PANDAS is proposed to have

⁎

a pathophysiology similar to that of Sydenham's chorea, which in turn is a neurological manifestation of rheumatic fever – an autoimmune disease triggered by streptococcal infections (Swedo, 1994; Swedo et al., 1998). The PANDAS criteria are similar to the PANS criteria, but include tics or tic-disorder as a possible primary symptom, and require a confirmed streptococcal infection before symptom onset. The requirement of a streptococcal infection may be too narrow a criterion for effective clinical use, and hence the later defined term PANS requires no specific infection for diagnosis (Swedo et al., 2012). See Table 1 for PANS and PANDAS criteria used in this study. The suggested autoimmune etiology of the conditions has led to suggestion of alternative treatment options (Chang et al., 2015), and the development of possible diagnostic biomarkers. Autoantibodies to dopamine receptors D1 and D2, β-tubulin, and lysoganglioside-GM1 (lyso-GM1) and calcium calmodulin dependent kinase II activity (CaMKII-activity) previously linked to Sydenham's chorea are proposed

Corresponding author. E-mail addresses: [email protected] (E. Hesselmark), [email protected] (S. Bejerot).

http://dx.doi.org/10.1016/j.jneuroim.2017.09.002 Received 22 February 2017; Received in revised form 10 July 2017; Accepted 5 September 2017 0165-5728/ © 2017 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/BY-NC-ND/4.0/).

Please cite this article as: Hesselmark, E., Journal of Neuroimmunology (2017), http://dx.doi.org/10.1016/j.jneuroim.2017.09.002


E. Hesselmark, S. Bejerot

Table 1 Diagnostic criteria for PANS and PANDAS. PANDAS Pediatric autoimmune Neuropsychiatric disorders associated with streptococcal infections Swedo et al., 1998

PANS Pediatric acute-onset neuropsychiatric syndrome Swedo et al., 2012

Criteria I, II and III must be met I. Abrupt, dramatic onset of obsessive-compulsive disorder or severely restricted food intake II. Concurrent presence of additional neuropsychiatric symptoms, with similarly severe and acute onset, from at least two of the following seven categories: 1. Anxiety 2. Emotional lability and/or depression 3. Irritability, aggression and/or severely oppositional behaviors 4. Behavioral (developmental) regression 5. Deterioration in school performance 6. Sensory or motor abnormalities 7. Somatic signs and symptoms, including sleep disturbances, enuresis or urinary frequency III. Symptoms are not better explained by a known neurologic or medical disorder, such as Sydenham chorea, systemic lupus erythematosus, Tourette disorder or others.

All 5 criteria must be met 1) Presence of obsessive-compulsive disorder (OCD) or a tic disorder. 2) Prepubertal symptom onset. 3) Acute symptom onset and episodic (relapsing-remitting) course. 4) Temporal association between Group A streptococcal infection and symptom onset/exacerbations. 5) Associated with neurological abnormalities, (particularly motoric hyperactivity and choreiform movements)

also examined, test-retest reliability is investigated, and panel results in patients with PANS or PANDAS are compared with those of healthy controls. In order to recruit patients with PANS or PANDAS, and gain access to their pretreatment Cunningham Panel results, we recruited a nationwide sample of patients who had already been tested with the Cunningham Panel. Each patients' first recorded Cunningham Panel results were used as the index test in the diagnostic accuracy analysis, thereby avoiding influence of treatment given to patients in response to Cunningham Panel results. In the analysis of change of the Cunningham Panel results, the baseline test prior to study inclusion is labeled time point 1 and a second test taken after the psychiatric assessment in this study is labeled time point 2. See Supplement 1 for details of study design and data collection. This investigation complied with Standards for the Reporting of Diagnostic Accuracy Studies (STARD) guidelines. All study participants or respective guardians granted informed consent. Our protocol was approved by the Regional Ethics Review Board of Stockholm (2014/ 551-31/2; 2014/1711-32; 2015/964-31, 2016/2121-32). This study has been registered pre enrollment at clinicaltrials.gov as: PANS - A detailed study of the patients, their symptoms, biomarkers, and treatment offered in a Scandinavian cohort; http://clinicaltrials. gov; NCT02190292.

biomarkers for PANDAS and acute-onset OCD (Cox et al., 2013; Kirvan et al., 2003, 2006, 2007; Singer et al., 2015). The above are analytes of the Cunningham Panel (Moleculera, 2016), a commercially available set of assays intended for diagnosing PANS or PANDAS and for monitoring symptom severity currently performed by Moleculera Labs, Oklahoma, City, OK, USA. CaMKII activity has been shown to be elevated by monoclonal antibodies from patients with chorea, and in addition these antibodies bind to both Lysoganglioside and to an epitope in the GABHS cell wall, implying molecular mimicry as a cause for the autoimmune process (Kirvan et al., 2003). This process has also been shown in serum from children with PANDAS, thus strengthening both the link between Sydenham's chorea and PANDAS, as well as supporting the diagnostic use of these analytes (Kirvan et al., 2006). Despite multiple studies (Cox et al., 2013, 2015; Kirvan et al., 2006; Morris-Berry et al., 2013; Singer et al., 2008, 2015), it remains unclear whether these biomarkers are sensitive and specific markers of PANS or PANDAS. In previous studies of the Cunningham Panel, mean and median relationships in patient populations and comparison samples were analyzed, without addressing diagnostic accuracy (Cox et al., 2013, 2015; Kirvan et al., 2006; Singer et al., 2015). No association between symptom exacerbations and biomarker levels has been shown in prospective studies (Morris-Berry et al., 2013, Singer et al., 2008, Singer et al., 2015); and in a recent therapeutic trial involving children with PANDAS (Williams et al., 2016), treatment response in both intervention and placebo groups was jointly predicted by CaMKII activation and antinuclear antibody (ANA) titers. Thus, the clinical utility of the Cunningham Panel is subject to question. Objectives of this study were as follows: 1) evaluate the Cunningham Panel as a diagnostic tool for PANS and PANDAS, 2) determine if clinical improvement coincides with decreased CaMKII activity, and 3) compare Cunningham Panel results in patients who meet criteria for PANS or PANDAS with results from healthy controls.

2.2. Participants and assessment procedure All 154 Swedish patients tested by Cunningham Panel prior to June 2014 were eligible for inclusion. Age > 40 years and inability to complete the assessment in Swedish were exclusion criteria. We chose to include adults as well as children as the Cunningham Panel is used regardless of age, even though PANS and PANDAS are both defined as pediatric disorders. Candidates were invited by mail to participate in the study, and 53 individuals consented to participate. Wieslab, the facility administrating the Cunningham Panel in Sweden, provided assistance by sending out invitations. Details of the recruitment process are found in Supplement 1, and Supplementary Fig. S1. Each confirmed participant underwent a one-time examination to determine PANS or PANDAS caseness, documenting current and past symptoms, psychiatric health, neurologic and motor symptoms, and cognitive functioning. All assessments were done locally (at outpatient clinics or as house calls) between January 2015 and June 2016. Prior Cunningham Panel results (time point 1) were supplied by Wieslab. Blood was collected for Cunningham Panel retesting after each interview (time point 2), at a clinic local to the patient. A flowchart of study participants is shown in Fig. 1.

2. Method 2.1. Study design This study is a diagnostic accuracy study with the aim to evaluate the Cunningham Panel as a diagnostic tool for PANS and PANDAS. Reference standard for diagnosing PANS or PANDAS is a comprehensive psychiatric interview performed by an experienced physician (Chang et al., 2015). The test evaluated in this study, i.e. the index test, is the Cunningham Panel (Moleculera, 2016). The relationship between symptom severity changes and shifts in Cunningham Panel results is

2



administered cognitive assessments. Each psychiatric interview took 3–5 h. In 45 cases, both the participant and a parent were interviewed. However, 7 patients had unavailable parents, and 1 patient was assessed only with parental interview. A core feature of PANS and PANDAS is a dramatic and acute symptom onset. Indication of < 72 h from onset to maximal symptoms was regarded as “acute onset” in this study. Symptoms or previous diagnosis of Tourette Syndrome was not used as an exclusion criterion for PANS in this study. A diagnostic criterion of PANDAS is that there should be a temporal association between GABHS and the onset of symptoms. This was assessed during the clinical interview, where all participants were asked if there was a preceding illness or other event to the onset. After the full interview a clinical summary was made by a senior psychiatrist (SB), who considered all available evidence for any GABHS infection preceding symptom onset when applying the PANDAS criteria to each case. For a complete description of our psychiatric assessment protocol, see Supplement 1. Current and past symptoms specifically related to PANS and PANDAS were rated using both standardized instruments and a structured interview to determine the presence or absence of relevant psychiatric and motor symptoms. The structured interview was partly based on communication with Professor James Leckman from Yale Child Study Center and on the instrument Pediatric Acute Neuropsychiatric Symptom Scale developed by Dr. Leckman and colleagues (Leckman, 2014, personal communication). All recorded symptoms were also recorded by disease onset (before, during, or after appearance of suspected PANS or PANDAS) and manner of progression (significant flares, fluctuating nature, or chronic course).

Fig. 1. Flowchart of participant recruitment and assessment: 53 subjects involved in analyzing diagnostic properties of Cunningham Panel and 46 in correlating psychiatric health and Cunningham Panel fluctuations. No participant had current or previous Sydenham's chorea. Note: Cunningham Panel positivity indicated by a composite score (CaMKII and any antibody) and diagnostic properties in PANS is illustrated. More data on PANDAS and diagnostic accuracy shown in Tables 3 and 4.

2.4.2. Index test of diagnosis: Cunningham Panel of PANS biomarkers The Cunningham Panel provides a measure of CaMKII activity and autoantibodies to dopamine receptors D1 and D2, β-tubulin, and lysoGM1. ELISA assays of these autoantibodies are described comprehensively by Moleculera Labs (Moleculera, 2016) and are detailed elsewhere in the literature (Kirvan et al., 2007; Singer et al., 2015). Assay of serum CamKII activity is described by Kirvan et al. (2003) and Moleculera (2016). In Sweden, serum obtained for a Cunningham Panel is first sent to Wieslab, where it is frozen. It is then forwarded to Moleculera to undergo standardized procedural analysis. Results are returned to Wieslab, and the referring physician receives a copy of the printed report.

2.2.1. Healthy controls Twenty-seven healthy controls, age and gender-matched to study subjects, were recruited in August 2016. Exclusion criteria for healthy controls were psychiatric care 1 year before inclusion and lifetime duration of any psychiatric, autoimmune, or movement disorder. Six of these participants did not provide a blood sample, resulting in 21 healthy control participants being included in the study. 2.3. Blinding and study integrity Data collection was planned in advance of patient enrollment. The research team was blinded to prior Cunningham Panel results (time point 1) at each psychiatric assessment. Wieslab and Moleculera were blinded to participant identities at time point 2. Participants were blinded to their own Cunningham Panel results (time point 2) when reporting changes in psychiatric health. Wieslab and Moleculera were blinded to participant IDs during test-retest reliability analysis and to which samples were provided by healthy controls. Wieslab provided a discounted price (~ USD 700) for each Cunningham Panel analysis. Dr. Anna Lauren, a Wieslab employee, served as co-applicant in seeking permission of the Ethical Committee to conduct this study. However, neither Wieslab nor Moleculera Labs influenced this study in terms of design, participant selection, psychiatric assessments, or outcomes presented.

2.5. Cunningham Panel results relative to change in psychiatric health Because CaMKII activity is suggested to be used for monitoring treatment outcome, we expected changes would follow severity of psychiatric symptoms. Given the one-time nature of interviews, the measure of change in psychiatric symptoms was done in retrospect, using self- or parent-rated Clinical Global Impression-Improvement (CGI-I) scores (Guy, 1976). Each participant or caretaker was asked: “Compared to when you took the first Cunningham Panel test, how do you feel today?” Mental health changes were ranked on a 7-point Likert scale, where 1 is “very much improved” and 7 is “very much worse”. A rating of 4 represented no change in status. Caretakers' scores were primarily used, but if no caretaker score was available, the participant score was used. The CGI-I ratings and time point 2 blood samples were obtained a mean of 74 weeks after the initial sample at time point 1.

2.4. Diagnostic accuracy of Cunningham Panel 2.6. Test-retest reliability of Cunningham Panel 2.4.1. Reference standard of diagnosis: psychiatric interview The goal of the psychiatric interview was to make valid PANS and PANDAS diagnoses. Interview data included valid protocols, such as M.I.N.I.-KID (Mini International Neuropsychiatric Interview for Children and Adolescents; Sheehan et al., 1998) and CY-BOCS (Children's Yale-Brown Obsessive Compulsive Scale; Scahill et al., 1997). A senior psychiatrist (SB) carried out the psychiatric interview and motor assessment. A PhD student (EH) trained in clinical psychology

Ten participants who provided blood samples at time point 2 contributed to this arm of the study. A total of seven 4.5-ml tubes of whole blood were collected (at one sitting) from each subject. One serum sampling tube (BD Vacutainer, yellow top) was sent to Wieslab for CaMKII analysis at the time of sampling, and one identical serum tube was sent to a biobank at Karolinska Institutet, Stockholm, Sweden (KI Biobank) for serum separation and storage at −80 °C. After a mean of 3



3.2. Diagnostic accuracy of Cunningham Panel in PANDAS

396 days in storage, 10 serum samples at KI Biobank were selected by one of the authors (EH) and sent to Wieslab for second, blinded analysis of CaMKII activity. Results of initial and subsequent CaMKII assays were compared.

All 13 PANDAS cases had at least one positive value on the Cunningham Panel (sensitivity 100%), however 34 out of 38 nonPANDAS cases also had at least one positive value on the panel (specificity 11%). See Table 2 for details. None of the individual scores, nor any composite score, had a diagnostic odds ratio > 1, indicating that the Cunningham Panel was no better than chance in detecting PANDAS in this sample. Positive predictive values were in the range 17–28% and negative predictive values were in the range 61–100%. ROC curves for Cunningham Panel analytes are shown in Supplementary Fig. S2. All calculated areas under the ROC curve were non-significant, i.e. the 50% line was within all calculated confidence intervals of the Area under the curve, which further strengthens the findings that the Cunningham Panel could not accurately diagnose PANDAS in this sample. See Supplementary Table S3 for cross-tabulation data.

2.7. Data analysis and statistics 2.7.1. Diagnostic accuracy of Cunningham Panel in PANS or PANDAS All results were designated either “pathologic” or “normal” according to cut-off scores defined by Moleculera Labs. Because the Cunningham Panel measures five different analytes independently, three composite scores were generated for Cunningham Panel results as follows: 1) any antibody positive, 2) any antibody and CaMKII positive, and 3) any analyte (antibody or CaMKII) positive. Sensitivity, specificity, predictive value, accuracy, and diagnostic odds ratio were calculated using cross tabulation for PANS and PANDAS separately.

3.3. Diagnostic accuracy of Cunningham Panel in PANS 2.7.2. Cunningham Panel results relative to change in psychiatric health Difference in CaMKII results were calculated by subtracting CaMKII at time point 1, from CaMKII results acquired at time point 2. A negative difference should be associated with improved health measured with CGI-I. Correlation between difference in Cunningham Panel scores and CGI-I were calculated using Pearson correlation.

Eighteen out of 20 PANS cases (sensitivity 90%) and 29 out of 31 non-PANS cases (specificity 6%) had at least one positive value on the Cunningham Panel. None of the individual scores or any composite score corresponded with a diagnostic odds ratio > 1, indicating that the Cunningham Panel was no better than chance in detecting PANS in this sample. Positive predictive values ranged from 30 to 44% and negative predictive values ranged from 44 to 64%. All calculated areas under the ROC curve, except for Lyso-GM1 were non-significant, i.e. the 50% line was within the calculated confidence intervals of the Area under the curve. Area under the curve for Lyso-GM1 was 65.5% (95% CI 50.5%–80.6%). In contrast, the diagnostic odds ratio for Lyso-GM1 was > 1 and it had an accuracy of 58%. See Table 3 for details on diagnostic properties and Supplementary Table S4 for cross-tabulation data.

2.7.3. Test-retest reliability of Cunningham Panel All data are presented below and in Supplementary Table S1. 2.7.4. Comparison with healthy controls Twenty-one healthy controls provided blood samples that were analyzed with the Cunningham Panel. Samples were sent to Wieslab for analysis according to standard procedure. For this analysis, we used the baseline test from time point 1, as the patient group result. CaMKII activity was considered normally distributed and was compared by t-test. Titers of autoantibodies were not normally distributed, so MannWhitney U test was used for comparisons.

3.4. Cunningham Panel results relative to change in psychiatric health Seven participants did not provide a second blood sample and 3 participants did not provide a CGI-I score, resulting in 43 patients being included in this analysis. There was a moderate correlation between difference in CaMKII and in CGI-I in patients who met diagnostic criteria for PANS or PANDAS (n = 18, r = 0.522; p = 0.04), but not in the non-PANS or PANDAS group (n = 25, r = − 0.263; p = 0.20). A scatterplot of the correlation is presented as Supplementary Fig. S3.

3. Results 3.1. Participant characteristics and PANS and PANDAS status Of the 53 participants, 40 were children and adolescents (ages 5–19, mean age 11.6 years) and 13 were adults (mean age 24.5 years). Twenty participants were female and 33 were male. Mean age of onset of severe psychiatric symptoms was 7.8 years. Mean age of participants at time of psychiatric assessment was 15.5 years, resulting in a 8 year duration between symptom onset and time of assessment within this study. All participants had present or past psychiatric disorders at time of our assessment, with 83% reporting lifetime OCD and 77% reporting anxiety, see Supplementary Table S2, for details. Furthermore, 49% had a history of suicidal ideation, and 43% had a history of violent behavior toward other people. Hallucinations were also common is our sample, with 25% reporting visual hallucinations and 23% reporting auditory hallucinations. At time of assessment the mean CY-BOCS score was 12.1 (n = 26) and mean Y-BOCS score was 18.86 (n = 14). No participant had current or previously diagnosed Sydenham's chorea. Overall, 24 participants met diagnostic criteria for PANS or PANDAS (4 subjects met diagnostic criteria for PANDAS only, 11 for PANS only, and 9 for both PANS and PANDAS), and 29 participants did not fulfil criteria for PANS or PANDAS. Patients without PANS or PANDAS presented more severe psychiatric symptoms during our psychiatric assessments (PANS or PANDAS: CGI-S mean, 4.4 ± 1.4; no PANS or PANDAS: CGI-S mean, 5.3 ± 1.7; t = 2.198, df = 48; p = 0.03).

3.5. Test-retest reliability of Cunningham Panel In 2 out of 10 samples, the CaMKII-results were clinically different, i.e. on different sides of the diagnostic cutoff set by Moleculera, between the first and second analysis. A third sample differed more than two standard deviations between the two analyses. Seven of the 10 samples had small differences between the two analyses sampled on the same occasion. The second analysis results were both higher and lower than the first analysis results. Full results are presented in Supplementary Table S1. 3.6. Cunningham Panel results and comparison to healthy controls Cunningham Panel results for PANS-cases, PANDAS-cases and noncases and healthy controls are presented in Table 4. We compared Cunningham Panel results of cases to non-cases of PANS, PANDAS, and PANS or PANDAS and to healthy controls. There were no differences between groups on any measure. Ten out of 21 (48%) healthy controls had positive CaMKII results as compared to 35 out of 53 (66%) in the study sample. Seventeen had at least one positive autoantibody titer, resulting in a total of 18 (86%) healthy controls with at least one positive value on the Cunningham Panel, compared to 92% in the study sample. 4



Table 2 Diagnostic properties of Cunningham Panel in diagnosing PANDAS (N = 53). Sensitivity

CaMKII D1 D2 (n = 33) lyso-GM1 β-tubulin Any antibody positive CaMKII + any antibody Any positive value

Specificity

Positive predictive value

Negative predictive value

Accuracy

Positive likelihood ratio

Negative likelihood ratio

Diagnostic odds ratio

Diagnostic odds ratio 95% CI Lower bound

Upper bound

62% 46% 0% 15% 46% 69%

33% 50% 92% 80% 28% 25%

23% 23% 0% 20% 17% 23%

72% 74% 70% 74% 61% 71%

40% 49% 66% 64% 32% 36%

0.9 0.9 0.0 0.8 0.6 0.9

1.2 1.1 1.1 1.1 2.0 1.2

0.8 0.9 0.0 0.7 0.3 0.8

0.2 0.2

2.8 3.0

0.1 0.1 0.2

4.0 1.2 3.0

31%

48%

16%

68%

43%

0.6

1.5

0.4

0.1

1.5

100%

10%

27%

100%

32%

1.1

0.0

n/a

PANDAS cases (n = 13). Clinical Cut-off values provided by Moleculera Labs. CaMKII = calcium/calmodulin dependent kinase II activation, D1 = dopamine receptor D1 antibody, D2 = dopamine receptor D2 antibody, lyso-GM1 = lysoganglioside-GM1 antibody, β-tubulin = β-tubulin antibody. 95% CI = 95% confidence interval.

Table 3 Diagnostic properties of Cunningham Panel in diagnosing PANS (N = 53). Sensitivity

CaMKII D1 D2 (N = 33) lyso-GM1 β-tubulin Any antibody CaMKII + any antibody Any positive value

Specificity

Positive predictive value

Negative predictive value

Accuracy

Positive likelihood ratio

Negative likelihood ratio

Diagnostic odds ratio

Diagnostic odds ratio 95% CI lower bound

upper bound

50% 50% 0% 20% 60% 75% 35%

24% 52% 90% 82% 30% 27% 45%

29% 38% 0% 40% 34% 38% 28%

44% 63% 58% 63% 56% 64% 54%

34% 51% 54% 58% 42% 45% 42%

0.7 1.0 0.0 1.1 0.9 1.0 0.6

2.1 1.0 1.1 1.0 1.3 0.9 1.4

0.3 1.1 0.0 1.1 0.7 1.1 0.4

0.1 0.4

1.0 3.2

0.3 0.2 0.3 0.1

4.6 2.1 4.0 1.4

90%

6%

37%

50%

38%

1.0

1.7

0.6

0.1

4.5

PANS cases (n = 20). Clinical Cut-off values provided by Moleculera Labs. CaMKII = calcium/calmodulin dependent kinase II activation, D1 = dopamine receptor D1 antibody, D2 = dopamine receptor D2 antibody, lyso-GM1 = lysoganglioside-GM1 antibody, β-tubulin = β-tubulin antibody. 95% CI = 95% confidence interval.

Table 4 Cunningham Panel results of total sample. Results are presented as median and 5th and 95th percentiles. There were no significant difference between cases and non cases. Cut off scores presented are defined by Moleculera and Wieslab. Cutoff score

CaMKII D1 D2 (n = 33) Lyso-G β-tubulin

130 4000 16,000 640 2000

Total sample

PANS

PANDAS

no PANS or PANDAS

Healthy controls

n = 53

n = 20

N = 13

n = 29

n = 21

Median

5; 95

Median

5; 95

Median

5; 95

Median

5; 95

Median

5; 95

139 2000 4000 160 2000

106; 248 500; 32,000 1000; 19,200 40; 1280 250; 8000

132 3000 4000 320 2000

106; 245 288; 30,800 1000; 8000 42; 1248 250; 8000

162 2000 4000 160 1000

113; 229 700; 8000 1000; 8000 128; 640 400; 8000

139 4000 4000 160 2000

103; 231 500; 32,000 1000; 18,400 25; 1280 375; 8000

124 2000 4000 160 2000

102; 197 1000; 15,200 2000; 16,000 40; 1280 550; 8000

Cut off scores provided by Moleculera Labs and Wieslab. Note: no significant differences in patient subsets by any measure. CaMKII = calcium/calmodulin dependent kinase II activation, D1 = dopamine receptor D1 antibody, D2 = dopamine receptor D2 antibody, lyso-GM1 = lysoganglioside-GM1 antibody, β-tubulin = β-tubulin antibody.

5



4.1.2. Sample size and recruitment The sample size of this study is small, especially the size of the group meeting diagnostic criteria for PANDAS (n = 13). Nevertheless, the sample is highly representative of the group of patients likely undergo clinical testing with the Cunningham Panel. Using the known proportion of positive and negative CaMKII results in the full Wieslab sample (73%), we have calculated that in order to reach a sensitivity of 90%, we would need to add 50 more participants, all classified correctly as cases by both reference- and index test. In such a hypothetical model, the specificity of CaMKII for diagnosing PANDAS would be 49%, meaning that half of the identified cases would be false positives. In the light of this, we argue that the sample size is large enough to conclude that the diagnostic accuracy of the Cunningham Panel is poor. The sample size of the test-retest reliability sample is too small for a definite statistical analysis, which is a limitation. The cost of the Cunningham Panel did not allow us to reanalyze more than 10 samples. However, the finding that three of 10 results may be unreliable after storage in − 80 °C is surprising. In 2015 a study reporting longitudinal CaMKII activity data from 8 children with PANDAS, authored by researchers associated with Moleculera Labs, stated that “samples were collected aliquoted, frozen and stored” (Singer et al., 2015). It is unclear for how long the samples were stored in the study, but the authors state that the sera was prospectively collected within a larger study, from which results were presented already in 2008 (Kurlan et al., 2008). There in an expected variability in antibody levels in sera that has been frozen and thawed (Hendriks et al., 2014) and this should be taken into account when interpreting these results, as well as when conducting future studies. The sample size of the healthy control group was also small. However, results were surprising, with 18 out of 21 of the healthy control samples having a pathological result on at least one of the analytes included in the Cunningham Panel, which indicates poor diagnostic accuracy even in this small sample. Our results indicate a similar variability as a previous study of the Cunningham Panel, which presented data from 4 independent healthy samples (Singer et al., 2015). When interpreting our results, it should also be noted that the healthy controls were not tested for streptococcal infections.

3.7. Representativeness of study sample Out of the 154 eligible participants, 53 were finally assessed in this study. To investigate if the recruitment procedure had introduced bias in the study, we asked Wieslab for an anonymous summary of data on all 101 individuals assessed with the Cunningham Panel prior to June 2014, who had not consented to be part of the study. Because the individuals in this non-participant sample had not consented to participate in research, individual results were not forwarded to the research team. The non-participant sample included 124 tests, due to some individuals being tested multiple times. The non-participant sample did not differ significantly in proportion of females (43%), mean age (14.2 ± 8.2 years) or on proportion of positive results on any of the measures in the Cunningham Panel compared to the sample who participated in this study, indicating low risk of bias regarding gender, age of Cunningham Panel results in the study sample. See Supplementary Tables S5 and S6 for details. 4. Discussion This study evaluated the diagnostic accuracy of the Cunningham Panel, which is a set of laboratory biomarkers used for diagnosing PANS and PANDAS. We studied a representative sample of 53 Swedish patients previously assessed with the Cunningham Panel in a clinical setting. A senior psychiatrist examined each patient with a clinical assessment comprising multiple psychiatric, motor and cognitive measures in order to make valid PANS or PANDAS diagnoses. In addition, all patients were re-tested with the Cunningham panel. In our sample the diagnostic properties of the Cunningham Panel were unsatisfactory. Furthermore, improvement or deterioration of symptoms was only slightly related to change in CaMKII activity, and the test-retest reliability analysis indicated that CaMKII results differed in 3 out of 10 samples. Moreover, pathological Cunningham Panel results were present in 18 of 21 healthy controls. Clinical use of the Cunningham Panel for diagnosing PANS or PANDAS, or for monitoring symptom severity, is not supported by this study. 4.1. Limitations

4.1.3. Changes in CaMKII values and correspondence to symptom severity The analysis of change in CaMKII and change in symptoms suffer from several limitations. First, the time between the two CaMKII results differed between subjects and the expected fluctuation over time of CaMKII is unknown. Second, the use of the parent rated CGI-I is not an ideal measure of change in symptom severity. There is an obvious risk for recall bias influencing the results, and the measure is probably insensitive. By taking the time point 2 blood sample after the collection of CGI-I data, thereby blinding participants to change in CaMKII results, the risk of recall bias was possibly mitigated. Still, the results from this specific analysis should be interpreted with caution. Serum sampling was not done in relation to symptom exacerbations in this study, and this may have caused an underestimation of Cunningham Panel correspondence to clinical exacerbations. However, a previous study using a design where samples were collected at clinical exacerbations presented no such association (Singer et al., 2015).

Results of our study reveal a mismatch between PANS or PANDAS diagnosis by psychiatric interview and that implied by Cunningham Panel. Whether our psychiatric interview, Cunningham Panel, or both are invalid is unclear. Diagnostic criteria for PANS and PANDAS are based on research (Swedo et al., 1998; 2012) and clinical consensus (Chang et al., 2015) but are not recognized by for instance the American Psychiatric Association, casting doubt on their validity. The use of a psychiatric interview, cognitive and motor skill testing and use of the proposed diagnostic criteria as the reference standard of diagnosis in this study may therefore be questioned. However, the proposed criteria are currently in clinical use, and the mismatch between the results of the psychiatric interview and Cunningham Panel raise concern about the clinical value of the panel. 4.1.1. Reference standard of diagnosis: psychiatric interview The main limitation of our psychiatric assessment is that it relied on retrospective reports for diagnosis of PANS and PANDAS. Patients were seen in one single sitting, in most cases several years after the first psychiatric symptoms emerged. Preferably, the diagnostic workup should be done at symptom onset. However, psychiatric symptoms are often reported in retrospect, and most patients had ongoing symptoms during the psychiatric interview. The validity of our psychiatric interview is strengthened by the use of well validated instruments. We also defined interview content and diagnostic criteria prior to study start and in accordance with PANS and PANDAS literature, which further strengthens the validity of the psychiatric interview.

4.2. Conclusions Although our findings identified a moderate correlation between change in CaMKII and change in symptom severity in individuals with PANS or PANDAS, there was no indication that the Cunningham Panel can be used to diagnose PANS or PANDAS. Our results also suggest that test-retest reliability of CaMKII may be insufficient, and that Cunningham Panel results are commonly elevated in healthy controls. Supplementary data to this article can be found online at http://dx. doi.org/10.1016/j.jneuroim.2017.09.002. 6



infection and exacerbations of childhood tics and obsessive-compulsive symptoms: a prospective blinded cohort study. Pediatrics 121, 1188–1197. http://dx.doi.org/10. 1542/peds.2007-2657. Moleculera, 2016. http://www.moleculeralabs.com/pandas-pans-diagnosis-andtreatment/. Morris-Berry, C.M., Pollard, M., Gao, S., Thompson, C., Tourette Syndrome Study, G., Singer, H.S., 2013. Anti-streptococcal, tubulin, and dopamine receptor 2 antibodies in children with pandas and tourette syndrome: single-point and longitudinal assessments. J. Neuroimmunol. 264, 106–113. http://dx.doi.org/10.1016/j.jneuroim. 2013.09.010. Murphy, T.K., Patel, P.D., McGuire, J.F., Kennel, A., Mutch, P.J., Parker-Athill, E.C., et al., 2015. Characterization of the pediatric acute-onset neuropsychiatric syndrome phenotype. J. Child Adolesc. Psychopharmacol. 25, 14–25. http://dx.doi.org/10.1089/ cap.2014.0062. Scahill, L., Riddle, M.A., McSwiggin-Hardin, M., Ort, S.I., King, R.A., Goodman, W.K., et al., 1997. Children's yale-brown obsessive compulsive scale: reliability and validity. J. Am. Acad. Child Adolesc. Psychiatry 36, 844–852. http://dx.doi.org/10. 1097/00004583-199706000-00023. Sheehan, D.V., Lecrubier, Y., Sheehan, K.H., Amorim, P., Janavs, J., Weiller, E., et al., 1998. The mini-international neuropsychiatric interview (m.I.N.I.): the development and validation of a structured diagnostic psychiatric interview for dsm-iv and icd-10. J. Clin. Psychiatry 59 (Suppl. 20), 22–33 (quiz 4-57). Singer, H.S., Gause, C., Morris, C., Lopez, P., Tourette Syndrome Study, G., 2008. Serial immune markers do not correlate with clinical exacerbations in pediatric autoimmune neuropsychiatric disorders associated with streptococcal infections. Pediatrics 121, 1198–1205. http://dx.doi.org/10.1542/peds.2007-2658. Singer, H.S., Mascaro-Blanco, A., Alvarez, K., Morris-Berry, C., Kawikova, I., Ben-Pazi, H., et al., 2015. Neuronal antibody biomarkers for sydenham's chorea identify a new group of children with chronic recurrent episodic acute exacerbations of tic and obsessive compulsive symptoms following a streptococcal infection. PLoS One 10, e0120499. http://dx.doi.org/10.1371/journal.pone.0120499. Swedo, S.E., 1994. Sydenham's chorea. A model for childhood autoimmune neuropsychiatric disorders. JAMA 272, 1788–1791. Swedo, S.E., Leonard, H.L., Garvey, M., Mittleman, B., Allen, A.J., Perlmutter, S., et al., 1998. Pediatric autoimmune neuropsychiatric disorders associated with streptococcal infections: clinical description of the first 50 cases. Am. J. Psychiatry 155, 264–271. http://dx.doi.org/10.1176/ajp.155.2.264. Swedo, S.E., Leckman, J.F., Rose, N.R., 2012. From research subgroup to clinical syndrome: modifying the pandas criteria to describe pans (pediatric acute-onset neuropsychiatric syndrome). Pediatr. Ther. 2. http://dx.doi.org/10.4172/2161-0665. 1000113. Williams, K.A., Swedo, S.E., Farmer, C.A., Grantz, H., Grant, P.J., D'Souza, P., et al., 2016. Randomized, controlled trial of intravenous immunoglobulin for pediatric autoimmune neuropsychiatric disorders associated with streptococcal infections. J. Am. Acad. Child Adolesc. Psychiatry 55 860-7 e2. http://dx.doi.org/10.1016/j.jaac.2016. 06.017.

Acknowledgements We wish to warmly thank the participants in this study, and their families. We also thank Sara Ekman and Machi Cleantous for assisting with data management and data collection and Wieslab for assistance with patient recruitment. This research was funded by grants from the Swedish Research Council (523-2011-3646) and by grants provided by the Stockholm County Council (PPG projects 20130671 and 20150150). The funding sources had no influence over the study design, collection or interpretation of data or any other part of the research process. We have no conflicts of interest to disclose. References Chang, K., Frankovich, J., Cooperstock, M., Cunningham, M.W., Latimer, M.E., Murphy, T.K., et al., 2015. Clinical evaluation of youth with pediatric acute-onset neuropsychiatric syndrome (pans): recommendations from the 2013 pans consensus conference. J. Child Adolesc. Psychopharmacol. 25, 3–13. http://dx.doi.org/10. 1089/cap.2014.0084. Cox, C.J., Sharma, M., Leckman, J.F., Zuccolo, J., Zuccolo, A., Kovoor, A., et al., 2013. Brain human monoclonal autoantibody from sydenham chorea targets dopaminergic neurons in transgenic mice and signals dopamine d2 receptor: implications in human disease. J. Immunol. 191, 5524–5541. http://dx.doi.org/10.4049/jimmunol. 1102592. Cox, C.J., Zuccolo, A.J., Edwards, E.V., Mascaro-Blanco, A., Alvarez, K., Stoner, J., et al., 2015. Antineuronal antibodies in a heterogeneous group of youth and young adults with tics and obsessive-compulsive disorder. J. Child Adolesc. Psychopharmacol. 25, 76–85. http://dx.doi.org/10.1089/cap.2014.0048. Guy, W., 1976. Ecdeu Assessment Manual for Psychopharmacology —Revised (dhew publ no adm 76–338). U.S. Department of Health, Education, and Welfare, Rockville, MD. Hendriks, J., Stals, C., Versteilen, A., Mommaas, B., Verhoeven, M., Tirion, F., et al., 2014. Stability studies of binding and functional anti-vaccine antibodies. Bioanalysis 6, 1385–1393. http://dx.doi.org/10.4155/bio.14.96. Kirvan, C.A., Swedo, S.E., Heuser, J.S., Cunningham, M.W., 2003. Mimicry and autoantibody-mediated neuronal cell signaling in sydenham chorea. Nat. Med. 9, 914–920. http://dx.doi.org/10.1038/nm892. Kirvan, C.A., Swedo, S.E., Snider, L.A., Cunningham, M.W., 2006. Antibody-mediated neuronal cell signaling in behavior and movement disorders. J. Neuroimmunol. 179, 173–179. http://dx.doi.org/10.1016/j.jneuroim.2006.06.017. Kirvan, C.A., Cox, C.J., Swedo, S.E., Cunningham, M.W., 2007. Tubulin is a neuronal target of autoantibodies in sydenham's chorea. J. Immunol. 178, 7412–7421. Kurlan, R., Johnson, D., Kaplan, E.L., Tourette Syndrome Study, G., 2008. Streptococcal

7