High False-Positive Rate of a Putative Biomarker Test to Aid in the Diagnosis of Schizophrenia
High False-Positive Rate of a Putative Biomarker Test to Aid in the Diagnosis of Schizophrenia
Objective: The current study determined the ability of a 51-analyte immunoassay panel to discriminate between subjects with chronic schizophrenia and healthy control subjects in an American population.
Methods: Serum samples were collected from 25 subjects with a DSM-IV-TR diagnosis of schizophrenia and 50 healthy control subjects. Blinded samples were sent to the RulesBaseMedicine (RBM) laboratory, which analyzed the 51 biomarkers and converted the results into the VeriPsych score by the application of RBM-determined decision rules and returned these scores to the investigators. The VeriPsych score yields a conditional probability ranging from strongly positive to strongly negative that the sample was from someone who had schizophrenia. Sensitivity and specificity were calculated for these data. The study was conducted between February 27, 2010, and August 31, 2011.
Results: On the basis of this test, the conditional probability of having schizophrenia ranged from 35% to 98% in the subjects previously diagnosed with schizophrenia and ranged from < 12% to 99% in the healthy control subjects. The sensitivity of this 51-plex biomarker was 89% in this study, while the specificity was 34%.
Conclusions: The current study confirms that the 51-plex test performs as expected in individuals with chronic schizophrenia (sensitivity = 89%), indicating that the abnormalities in this multiple biomarker test persist and are not affected by the number of years this illness has been present or by its treatment. However, there was a high false-positive rate in healthy control subjects in our sample, leading to a low specificity rate of 34%. Due to the high false-positive rate in our normal controls, this biomarker test was not able to discriminate between healthy control subjects and subjects with chronic schizophrenia in our sample.
J Clin Psychiatry 2016;77(4):e451-e456
© Copyright 2016 Physicians Postgraduate Press, Inc.
aDepartment of Psychiatry and Behavioral Sciences, The University of Kansas School of Medicine-Wichita
*Corresponding author: Cheryl A. Wehler, MD, Department of Psychiatry and Behavioral Sciences, The University of Kansas School of Medicine-Wichita, 1010 N Kansas, Wichita, KS 67214 .
Quantifiable biological measures that directly correlate with the presence of a particular pathological process (ie, a disease) have been used in general medicine for years. Such molecular biomarkers are used to diagnose disease, assess risk, guide treatment decisions, and measure efficacy. The example of low-density lipoprotein levels illustrates the way in which biomarkers are used in medicine. By measuring this biomarker, physicians can identify if a patient is at increased risk for developing a specific disease (in this case, atherosclerosis) and its sequelae (ie, heart disease, stroke, or peripheral vascular disease).1,2
While attempts to develop such tests for psychiatric disorders have been ongoing for a number of years, minimal progress has been made to date. The development of such measures should help in a number of ways: (1) permit risk assessment for developing a disease, (2) aid in the ability to diagnose a specific mental illness earlier than is currently possible, and (3) provide objective quantification of the severity of disease and measurement of response to therapeutic intervention.
The need for biomarkers in psychiatry is especially important for serious and persistent mental illnesses such as schizophrenia. The lifetime prevalence of schizophrenia appears to be approximately 0.3%-0.7%.3,4 While the prevalence of schizophrenia is low, its disease burden is disproportionately high owing to its onset in early adulthood and persistent disability.5 In addition to the negative impact of the illness itself, comorbidities (eg, atherosclerosis, smoking, and substance abuse) are prevalent and costly.
The majority of biomarkers used in medicine were developed based on the pathophysiology of a disease. Therein lies the conundrum in developing a biomarker for schizophrenia. The current symptom-based diagnosis of schizophrenia most likely defines a cluster of disorders that have diverse pathophysiologies.6 These pathophysiologic mechanisms have yet to be elucidated, but it is widely accepted that multiple susceptibility genes are thought to confer risk of schizophrenia.7-14 Given this complexity, it is unlikely that a single biomarker will suffice to identify schizophrenia. Therefore, a clinical test for this syndrome most likely requires multiple biomarkers.
In October 2010, a 51-analyte blood-based signature of schizophrenia was marketed to aid in the confirmation of the diagnosis of schizophrenia in individuals with a first- or recent-onset psychotic episode.15-17 The test had a more than 15-year incubation period, beginning with work at Cambridge Centre for Neuropsychiatric Research led by Sabine Bahn. The initial work was conducted using advanced molecular profiling techniques (microarrays, proteomics, lipidomics, and metabolomics) to globally investigate abnormalities in gene/protein/metabolite/lipid "expression" in postmortem human brain tissue and in blood derived from patients and matched controls, to establish evidence-based hypotheses. In 2005, a spin-off company was created, which in turn partnered with RulesBaseMedicine (RBM) of Austin, Texas, to further refine and commercialize the test. Subsequently, RBM became a wholly owned subsidiary of Myriad Genetics under the name Myriad RBM, Inc.
The sensitivity and specificity work for the commercial test was based principally on studies of populations of individuals who had a first or recent onset of schizophrenia, paranoid type, were predominantly antipsychotic naive, and were evaluated at 1 of 4 European universities. Controls were matched for age, gender, and social demographics. Exclusion criteria included a family history of mental illness and medical conditions such as type II diabetes, hypertension, and cardiovascular or autoimmune diseases. Initially 181 analytes were tested for their ability to distinguish the ill individuals from the controls and then refined into a 51-plex immunoassay panel by RBM. During the validation phase of this biomarker (VeriPsych) test, a linear support vector machine algorithm was used to discriminate schizophrenia patients from controls. This yielded a cross-validation classification accuracy of 83% (sensitivity, 83%; specificity, 83%).15
The initial goal of the current study was to determine whether individuals with long-standing schizophrenia continued to have the same results on this 51-plex biomarker test as did individuals with first- or recent-onset schizophrenia. Originally, healthy control subjects age matched to the older individuals with schizophrenia were included only for assay validity. However, these subjects subsequently became the focus of the results due to the finding of a high false-positive rate in healthy controls. In the process, a younger sample of healthy controls was added to match the age of the populations on which the test was originally developed.
- Diagnostic tests may be developed in refined research samples, but their sensitivity and specificity need to be confirmed in real-world clinical samples before they are adopted.
- Until such replication has been done, the clinician should exercise caution in interpreting the results of such tests when they are first introduced to the market, as they are not regulated by any government agency.
Subjects with a diagnosis of schizophrenia (295.1-295.3, 295.6, 295.7, 295.9 of the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, Text Revision [DSM-IV-TR])18 were recruited from local community mental health center and private psychiatric clinic populations. These individuals were not transient but instead had been treated for many years in this community and were well known to their usual health care providers and often to the research site as well. Healthy, ambulatory volunteers with no personal or family history of schizophrenia and no acute or active medical comorbidities, including morbid obesity, were recruited as controls. These individuals were well known to the research site, as many either worked part-time at the site or had participated in normal control phase 1 studies conducted at the site. As mentioned above, controls were overenrolled to bridge the age range from the individuals with first-onset schizophrenia for whom the test was developed and individuals with chronic schizophrenia who were initially the focus of this study. No matching other than for age was done in the current study.
The research protocol was approved by the Human Subjects Committee 2 of the University of Kansas School of Medicine-Wichita. The study was conducted in accordance with current US Food and Drug Administration regulations, International Conference on Harmonisation guidelines, Good Clinical Practice standards, and the Declaration of Helsinki in addition to local ethical and legal requirements. Written informed consent was obtained from all study participants prior to the sample blood collections, which were obtained and processed in accordance with the sponsor’s (RBM) specifications (available on request). The study was conducted between February 27, 2010, and August 31, 2011.
Multiplex Immunoassay and Analysis
Multiplex immunoassay analysis was carried out using the DiscoveryMAP immunoassays in a Clinical Laboratory Improved Amendments-certified laboratory at RBM. The VeriPsych test measured 51 analytes (Table 1) in a single serum sample and applied a decision rule to convert those results into a single number, the VeriPsych score. Each VeriPsych score was associated with a conditional probability of a patient’s likelihood of having schizophrenia. These conditional probability ranges and the associated classification were developed from the range of signals seen in a retrospective validation study15 of 806 subjects. (That earlier validation study15 included 577 samples from subjects diagnosed with schizophrenia and 229 demographically matched healthy controls.) From that study,15 5 classes of VeriPsych scores (Table 2), associated with differing conditional probabilities for a schizophrenia diagnosis, were derived. In this study, blinded samples were sent to the RBM laboratory for analysis and VeriPsych scores were calculated there and returned to the study investigators.
Serum samples were collected from 25 subjects with a previous diagnosis of schizophrenia and 50 healthy control subjects. The subjects with schizophrenia included 18 men and 7 women ranging from 23 to 40 years old. The overall mean age was 33.6 years; the mean age of women was 33.9 and of men was 33.6 years.
The healthy control sample consisted of 25 men and 25 women ranging from 18 to 40 years old. The overall mean age was 27.0 years; the mean age of women was 28.0 and of men was 26.0 years. The control group was divided into 2 samples: those who were "older" (age > 27 years) (mean ± SD age = 33.2 ± 3.1 years) and those who were "younger" (age ≤ 27 years) (mean ± SD age = 20.8 ± 2.7 years). The older healthy controls did not differ in age from the group with chronic schizophrenia (P = .162). The younger controls were comparable in age to the sample used to originally develop the test.
Multiplex Immunoassay Analyses
The probability of having schizophrenia based on the results of this 51-plex immunoassay is shown in Table 2 for both the 25 subjects previously diagnosed with schizophrenia and the 50 healthy control subjects. Results of subanalyses based on age and sex are also presented. On the basis of this test, the conditional probability of having schizophrenia ranged from 35% to 98% in the subjects diagnosed with schizophrenia. As seen in Table 2, of these 25 subjects 48% (12) were classified by this biomarker as strongly positive for having schizophrenia, 20% (5) were classified as positive, 24% (6) had an indeterminate probability of having the syndrome, 8% (2) were negative, and 0% were strongly negative for having schizophrenia.
On the basis of this biomarker assay, the conditional probability of having schizophrenia ranged from < 12% to 99% in the healthy control subjects. As seen in Table 2, of the 50 normal healthy subjects, 42% (21) were classified by this biomarker as strongly positive for having schizophrenia, 12% (6) were classified as positive, 18% (9) had an indeterminate probability of having the syndrome, 22% (11) were negative, and 6% (3) were strongly negative for having schizophrenia.
Sensitivity and specificity were calculated for these data, excluding all samples with an indeterminate (ie, the test was unable to classify sample as positive or negative) indication, including 24% of subjects with a previous diagnosis of schizophrenia and 18% of healthy controls. In addition, samples that were in conditional probability ranges indicating strongly positive or positive were combined, as were those indicating strongly negative or negative. Sensitivity is the probability that a test will indicate "disease" among those with the disease; in this study, the sensitivity of this 51-plex biomarker was 89%. Specificity is the fraction of those without the disease who will have a negative test result; in this study, the specificity was 34%.
To assess whether older age might contribute to the high false-positive rate in healthy controls, we compared the sensitivity and specificity in young versus the older healthy controls, and age did not make a difference. The results were similar regardless of age. Of the sample of 25 subjects with a previous diagnosis of schizophrenia, 92% (23) were older and 8% (2) were younger than 27 years. In the sample of 50 healthy controls, 4% (2) had a mean age of 27 years, 50% (25) were older than 27, and 46% (23) were younger than 27.
In the older subjects with schizophrenia, the conditional probability of having schizophrenia, based on this test, ranged from 35% to 97%. As seen in Table 2, of the 23 older subjects with schizophrenia, 48% (11) were classified by this biomarker as strongly positive for having schizophrenia, 17% (4) were classified as positive, 26% (6) had an indeterminate probability of having the syndrome, 9% (2) were negative, and 0% (0) were strongly negative for having schizophrenia. In the younger subjects with schizophrenia, the conditional probability of having schizophrenia, based on this test, ranged from 72% to 98%. As seen in Table 2, of the 2 younger diseased subjects, 50% (1) was classified by this biomarker as strongly positive for having schizophrenia, 50% (1) was classified as positive, and 0% had an indeterminate probability of having the syndrome or were classified as negative or strongly negative for having schizophrenia.
In the older healthy control subjects, the conditional probability of being identified as having schizophrenia, based on this biomarker test, ranged from < 12% to 97%. As seen in Table 2, of the 25 older healthy controls, 36% (9) were classified by this test as strongly positive for having schizophrenia, 8% (2) were classified as positive, and 20% (5) had an indeterminate probability for having schizophrenia, while 24% (6) were negative and 12% (3) were strongly negative for having schizophrenia. In the younger healthy control subjects, the conditional probability of having schizophrenia, based on this test, ranged from 25% to 99%. As seen in Table 2, of the 25 younger healthy controls, 48% (12) were classified by this test as strongly positive for having schizophrenia, 16% (4) were classified as positive, 16% (4) had an indeterminate probability of having the syndrome, 20% (5) were negative, and 0% (0) were strongly negative for having schizophrenia.
Sensitivity and specificity were calculated for each age category separately. In the older subjects in this study, the sensitivity of this 51-plex biomarker was 88%, while the specificity was 45%; in younger subjects, the sensitivity of this 51-plex biomarker was 100%, while the specificity was 24%.
The results in female and male subjects yielded similar results as the calculations of sensitivity and specificity for the overall sample. Of the sample of 25 subjects with schizophrenia, 28% (7) were women and 72% (18) were men. Of the sample of 50 healthy controls, 50% (25) were women and 50% (25) were men.
In the female subjects with schizophrenia, the conditional probability of having schizophrenia, based on this test, ranged from 47% to 95%. As seen in Table 2, of the 7 women, 43% (3) were classified by this biomarker as strongly positive for having schizophrenia, 28.5% (2) were classified as positive, 28.5% (2) had an indeterminate probability of having the syndrome, and 0% (0) were negative or strongly negative for having schizophrenia. In the male subjects with schizophrenia, the conditional probability of having schizophrenia, based on this test, ranged from 35% to 98%. As seen in Table 2, of the 18 men, 50% (9) were classified by this biomarker as strongly positive for having schizophrenia, 17% (3) were classified as positive, 22% (4) had an indeterminate probability of having the syndrome, 11% (2) were negative, and 0% (0) were strongly negative for having schizophrenia.
In the healthy female control subjects, the conditional probability of testing positive for schizophrenia, based on this test, ranged from < 12% to 89%. As seen in Table 2, of the 25 healthy female controls, 24% (6) were classified by this test as strongly positive for having schizophrenia, 16% (4) were classified as positive, and 20% (5) had an indeterminate probability of having the syndrome, while 32% (8) were negative and 8% (2) were strongly negative for having schizophrenia. In the healthy male control subjects, the conditional probability of being classified as having schizophrenia, based on this test, ranged from 25% to 99%. As seen in Table 2, of the 25 healthy male controls, 60% (15) were classified by this test as strongly positive for having schizophrenia, 8% (2) were classified as positive, 16% (4) had an indeterminate probability of having the syndrome, 12% (3) were negative, and 4% (1) were strongly negative for having schizophrenia.
Sensitivity and specificity were calculated for each sex separately. In the female subjects in this study, the sensitivity of this 51-plex biomarker was 100%, while the specificity was 50%; in male subjects, the sensitivity of this 51-plex biomarker was 86%, while the specificity was 19%.
The present study was initially conducted to determine whether abnormalities found using a putative biomarker assay to aid in the confirmation of acute-onset schizophrenia persisted in subjects with a chronic history of this disorder. In this study, participants with schizophrenia had had their disease for some 5-15 years, and all had been treated with multiple antipsychotic medications. The test performed comparably in the individuals with chronic schizophrenia (ie, 68% being either strongly positive or positive and only 8% negative) as it had in individuals with first-onset psychosis.
Initially, a small number of healthy controls were included for comparison; however, analyses of the initial set of normal controls caused concern because of the high rate of false-positive test results. Hence, the study was extended and more subjects and controls recruited. While matching for socioeconomic status was not employed, an effort was made initially to include healthy controls approximating the age of the subjects with schizophrenia in this study. Since the sample of subjects with chronic schizophrenia was, by definition, older than the sample of first-onset subjects in whom the test was developed and validated, we questioned whether older age might be contributing to the disproportionate number of false positives. A concerted effort was, therefore, made to recruit younger healthy controls. In the end, we had 2 healthy control populations: those above the age of 27 years, whose mean age was similar to the mean age of our subjects with schizophrenia, and those under 27 years (25), whose mean age was 20.8, which was comparable to the mean age of the European sample upon whom the test was originally based.
Although age did not account for the high false-positive rate in healthy controls, it is an important variable to consider when developing such tests, as extensive work has shown that gene expression and messenger RNA (mRNA) varies considerably over the human lifespan, including genes associated with psychiatric illnesses such as schizophrenia.19,20 The interested reader is referred to the following website for additional information on how gene expression varies over the human lifespan ().
Another variable to consider when developing such tests is the fact that psychiatric and medical illnesses are frequently comorbid. These conditions may be pathophysiologically or pathoetiologically related or not. There has been considerable interest in whether inflammatory processes are causally involved in psychiatric illness, such as bipolar disorder, and the common comorbidities associated with this illness (eg, atherosclerosis).21 Of note, more than one-third of the analytes in this test (Table 1) were related to either inflammatory processes or immunologic response. If the comorbidities are etiologically unrelated to the pathophysiology underlying the psychiatric illness, they nevertheless could produce positive test results on such a test. That is unlikely to be a problem with this study for 2 reasons: (1) the test was developed primarily using individuals with first-episode psychosis without significant comorbidity, and (2) the high false-positive rate occurred in both young and older healthy controls.
This study confirms that the 51-plex test performs in individuals with chronic schizophrenia comparably to how it performed in studies of subjects with first-onset of the syndrome, thus indicating that the abnormalities in this multiple biomarker test persist and are not affected by the number of years this illness has been present or by its treatment.
Our data do not, however, show marked differences between subjects with schizophrenia and healthy controls. In the development and validation studies15-17 of the 51-plex test, good separation was reported in the mean VeriPsych score of those subjects with schizophrenia versus healthy control subjects. Sensitivity and specificity calculations were consistently in the 80% + range. In contrast, the overall specificity in the current study was 34%. Although it is marginally higher in women than in men, it appears that this 51-analyte immunoassay was not able to discriminate subjects with chronic schizophrenia versus healthy control subjects in our sample.
There are likely multiple explanations for the discrepancy in the performance of this test. A potential limitation of the current study is that the exclusion criteria were limited compared with that used in the development and validation of this 51-plex immunoassay. Conceivably, that difference might have contributed to the high false-positive rate in this sample. While a family history of mental illness was exclusionary for the healthy control group in both this study and the developmental studies, controls in the developmental studies were also excluded if they had a family history of general medical conditions such as type II diabetes, hypertension, and cardiovascular or autoimmune diseases.15 Perhaps, the inflammatory biomarkers used in this assay measure preclinical comorbidities in the control subjects in this sample. Additionally, although the healthy control subjects in this study were not obese, it is conceivable that their diet and lifestyle were sufficiently different from those in earlier studies, thereby increasing numerous immuno-inflammatory markers and consequently rendering this 51-plex panel invalid in this population. Further research is necessary in the US population to identify a biological signature of schizophrenia from which a robust diagnostic test could be developed and used in routine clinical practice. Such a test would have to maximize specificity to be useful as a diagnostic or confirmation test.
When presented with these results, RBM emphasized that the test was marketed as an aid to the confirmation of the diagnosis of schizophrenia of recent onset and thus not marketed as a screening test. They thought the focus on the false-positive rate was therefore not particularly relevant. The authors respectfully disagree with the company’s position.
Development of biomarkers for psychiatric illness is in its infancy. It is important to remember that such advances take much time and great effort. There was a 275-year span between the earliest medical biomarker—the taste of sweet urine by Thomas Willis in 1674—and the delineation of 2 disorders in diabetes by Himsworth in 1949.6 Acknowledging that the current symptom-based diagnosis of schizophrenia defines a cluster of disorders that are likely to have diverse pathophysiologies, developing a biological signature may require teasing out the biomarkers present in various subgroups of this disease. Once signatures of this disease are developed, validation of such tests in multiple populations may be prudent in order to maximize their utility.
The development of this 51-plex immunoassay is an example of the continued effort to develop a clinically usable biological diagnostic test for schizophrenia. While this specific test is no longer available commercially, there is a growing interest in such tests, whether biomarkers or genetic, in psychiatry. Hence, it behooves psychiatrists to increase their familiarity with the appropriate application of such tests in clinical practice and to understand the limitations of these tools. The test was marketed in October 2010 and withdrawn from the market in January 2013. Reasons that have been given for its withdrawal included (1) low market acceptance because it was a diagnostic test and clinicians did not feel a need for such a test and (2) its cost.22,23 This article provides another potential reason, a high false-positive rate in healthy controls.
Submitted: November 26, 2014; accepted May 5, 2015.
Potential conflicts of interest: None reported.
Funding/support: RulesBaseMedicine (RBM), Inc, provided a grant that funded the collection and shipment of the samples. Their laboratory analyzed the 51 biomarkers in the samples and calculated the VeriPsych scores.
Role of the sponsor: The sponsor had no role in the calculation of the sensitivity and specificity results or in the development of the manuscript. The authors did both of these activities without sponsor support or involvement.
Acknowledgment: The authors gratefully acknowledge the statistical support of Rosalee Zackula, MS, from the Research Office of the University of Kansas School of Medicine, Wichita. Ms Zackula has no conflicts of interest to report.
7. O’ Donovan MC, Craddock N, Norton N, et al; Molecular Genetics of Schizophrenia Collaboration. Identification of loci associated with schizophrenia by genome-wide association and follow-up. Nat Genet. 2008;40(9):1053-1055. PubMed doi:10.1038/ng.201
8. Purcell SM, Wray NR, Stone JL, et al; International Schizophrenia Consortium. Common polygenic variation contributes to risk of schizophrenia and bipolar disorder. Nature. 2009;460(7256):748-752. PubMed
9. Stefansson H, Ophoff RA, Steinberg S, et al; Genetic Risk and Outcome in Psychosis (GROUP). Common variants conferring risk of schizophrenia. Nature. 2009;460(7256):744-747. PubMed
10. Schizophrenia Psychiatric Genome-Wide Association Study (GWAS) Consortium. Genome-wide association study identifies five new schizophrenia loci. Nat Genet. 2011;43(10):969-976. PubMed doi:10.1038/ng.940
14. Hall J, Trent S, Thomas KL, et al. Genetic risk for schizophrenia: convergence on synaptic pathways involved in plasticity. Biol Psychiatry. 2015;77(1):52-58. PubMed doi:10.1016/j.biopsych.2014.07.011
15. Schwarz E, Izmailov R, Spain M, et al. Validation of a blood-based laboratory test to aid in the confirmation of a diagnosis of schizophrenia. Biomark Insights. 2010;5:39-47. PubMed
16. Schwarz E, Guest PC, Rahmoune H, et al. Identification of a biological signature for schizophrenia in serum. Mol Psychiatry. 2012;17(5):494-502. PubMed
17. Schwarz E, Guest PC, Rahmoune H, et al. Identification of a blood-based biological signature in subjects with psychiatric disorders prior to clinical manifestation. World J Biol Psychiatry. 2012;13(8):627-632. PubMed doi:10.3109/15622975.2011.599861
19. Colantuoni C, Hyde TM, Mitkus S, et al. Age-related changes in the expression of schizophrenia susceptibility genes in the human prefrontal cortex. Brain Struct Funct. 2008;213(1-2):255-271. PubMed doi:10.1007/s00429-008-0181-5
20. Tao R, Cousijn H, Jaffe AE, et al. Expression of ZNF804A in human brain and alterations in schizophrenia, bipolar disorder, and major depressive disorder: a novel transcript fetally regulated by the psychosis risk variant rs1344706. JAMA Psychiatry. 2014;71(10):1112-1120. PubMed doi:10.1001/jamapsychiatry.2014.1079
21. Savitz J, Preskorn S, Teague TK, et al. Minocycline and aspirin in the treatment of bipolar depression: a protocol for a proof-of-concept, randomised, double-blind, placebo-controlled, 2×2 clinical trial. BMJ Open. 2012;2(1):e000643. PubMed doi:10.1136/bmjopen-2011-000643
22. Bahn S, Noll R, Barnes A, et al. Challenges of introducing new biomarker products for neuropsychiatric disorders into the market. Int Rev Neurobiol. 2011;101:299-327. PubMed doi:10.1016/B978-0-12-387718-5.00012-2
Buy this Article as a PDF