Screening Tests Are Not Diagnostic: A Commentary on a Study of Electroconvulsive Therapy for Depressed Patients With and Without Borderline Personality Disorder

See article by Yip et al.

^aDepartment of Psychiatry and Human Behavior, Brown Medical School, and the Department of Psychiatry, Rhode Island Hospital, Providence, Rhode Island
*Corresponding author: Mark Zimmerman, MD, 146 West River St, Providence, RI 02904 (mzimmerman@lifespan.org).

J Clin Psychiatry 2021;82(2):21com13909

To cite: Zimmerman M. Screening tests are not diagnostic: a commentary on a study of electroconvulsive therapy for depressed patients with and without borderline personality disorder. J Clin Psychiatry. 2021;82(2):21com13909.
To share: https://doi.org/10.4088/JCP.21com13909

© Copyright 2021 Physicians Postgraduate Press, Inc.

The treatment of patients with major depressive disorder (MDD) and comorbid borderline personality disorder (BPD) is greatly understudied. Many placebo-controlled studies of the efficacy of antidepressants for MDD explicitly exclude patients with BPD.¹ Despite the lack of a single placebo-controlled study demonstrating the efficacy of an antidepressant in treating patients with comorbid MDD and BPD,² antidepressants are widely used in the treatment of these patients³ and even recommended in some official treatment guidelines.⁴

When MDD is severe, persisting, and unresponsive to pharmacologic interventions, electroconvulsive therapy (ECT) is sometimes recommended. Over the past 40 years, only a small number of small-scale studies examined whether patients with MDD and BPD respond to ECT as well as patients with MDD without BPD.⁵ The report by Yip and colleagues⁶ in this issue is the largest study to have examined this topic.⁶ Yip et al found no difference between patients with and without BPD in their response to ECT and concluded that their data help provide the clinician with “a rationale for proceeding with ECT among depressed patients, notwithstanding comorbid BPD.”^6(p8) However, confidence in this conclusion is limited because it is likely that most of the patients considered to have BPD in their study did not, in fact, have BPD. As I will describe, the improper use of a screening measure as an indicator of a diagnosis severely limits the conclusions that can be drawn from the study by Yip et al.

Yip and colleagues “diagnosed” BPD with the McLean Screening Inventory for BPD (MSI-BPD).⁷ A screening measure is not a substitute for a diagnostic evaluation. The purpose of a screening test is to cast a broad net to ensure that most patients with the disorder are captured in that net. The screening test is followed by the more definitive diagnostic assessment, an evaluation that is generally more expensive and/or invasive than the screening procedure. In psychiatry, the self-administered screening questionnaire is followed by a diagnostic interview. In research, a semistructured interview is the usual diagnostic procedure.

The 2 statistics most commonly reported in describing the performance of a screening measure are sensitivity and specificity. Sensitivity refers to how well the test identifies individuals with the disorder. With regard to the MSI-BPD, sensitivity refers to how many patients with BPD score at or above the cutoff used to indicate that the patient has screened positive. Specificity refers to how well the screening test identifies individuals without the disorder, that is, how many individuals without BPD score below the cutoff used to indicate that the patient has screened positive for BPD. Yip and colleagues indicated that the MSI-BPD had a sensitivity of 81% and a specificity of 85% in detecting BPD. These numbers were drawn from the initial report of the MSI-BPD’s performance.⁷

Two other statistics important in understanding a screening test’s utility are positive and negative predictive value. Positive predictive value indicates the probability a person who screens positive on the test actually has the disorder, that is, how many individuals who screen positive for BPD actually have BPD. Negative predictive value refers to the probability a person who screens negative on the test does not have the disorder, that is, how many individuals who screen negative for BPD do not have BPD. Positive and negative predictive values are less commonly used to describe a screening test’s performance because these statistics are influenced by the prevalence of the disorder in the sample studied. At constant sensitivity and specificity, the positive predictive value of a test is higher when the prevalence of the disorder is higher. For example, using the sensitivity and specificity values cited by Yip et al⁶ for the MSI-BPD, in a sample with a 10% prevalence of BPD the positive predictive value of the scale would be 37%. If the prevalence of BPD were 40%, then the positive predictive value of the MSI-BPD would be about twice as high (78%).

Let’s now turn to the report by Yip et al.⁶ They did not discuss the issue of the positive predictive value of the MSI-BPD. They acknowledged that the MSI-BPD is a screening measure, but they did not take this a step further and describe how this fact might have impacted the composition of the BPD group. In their study, the prevalence of BPD was 20.9%. One must keep in mind that the purpose of a screening scale is to identify all (or almost all) possible cases and therefore be overinclusive. Thus, the actual prevalence of BPD in their sample is likely lower than 20%, but for my initial calculations I will use Yip and colleagues’ prevalence estimate. Based on a prevalence of 20.9%, sensitivity of 81%, and specificity of 85%, the positive predictive value of the MSI-BPD in their sample was 58.8%. That is, more than 40% of the patients in the BPD group would not have been diagnosed with BPD had they been interviewed.

However, this estimate of positive predictive value is likely an overestimate. The sensitivity and specificity figures of the MSI-BPD used by Yip et al were based on the original study of the measure⁷ in which the cutoff was selected to optimize sensitivity and specificity. Selecting a cutoff score that optimizes agreement with a diagnostic standard essentially means that all cutoffs were examined and the best one chosen. That is, multiple statistical tests were conducted, thereby inflating type I error. It is therefore not surprising that studies attempting to replicate the initial publication will find the performance of the scale is inferior to its performance in the original report. A meta-analysis⁸ of 9 studies of the MSI-BPD in adults found that at the cutoff of 7 used by Yip et al the sensitivity of the MSI-BPD was 81.7% and the specificity of the scale was 63.2%. Based on these values of sensitivity and specificity, and a prevalence of BPD of 20.9%, the positive predictive value of the scale is only 37.0%. That is, about two-thirds of patients in Yip and colleagues’ BPD group would not have been diagnosed with BPD had they been interviewed.

To be sure, this analysis is imperfect. Yip et al did not examine the performance of the MSI-BPD in patients with MDD who are referred for ECT. In fact, I am unaware of any study of the performance of the MSI-BPD in a sample of patients with MDD. It is certainly possible that in a sample of patients with MDD the MSI-BPD performs better than in other patient samples, though there is no reason to believe this to be true.

Not only is it important to know a scale’s sensitivity and specificity to calculate positive predictive value, it is also necessary to know the prevalence of the disorder. It has been my experience that clinicians are hesitant to refer patients with MDD with comorbid BPD for ECT because of skepticism regarding ECT’s efficacy. Thus, I would have expected a relatively low prevalence of BPD in a sample of patients referred for ECT.

In the Discussion section of their article, Yip et al briefly reviewed the findings of the 2 studies^9,10 examining the prognostic significance of BPD in patients with MDD receiving ECT that they considered to be “methodologically sound.” The study by Feske et al⁹ found a robustly lower remission rate in patients with BPD compared to patients with no personality disorder (22.2% vs 71.7%, respectively). In that study, the diagnosis of BPD was based on a semistructured diagnostic interview for personality disorders. The study by Lee et al¹⁰ used the MSI-BPD to identify the BPD cohort and found no difference in outcome between the patients who did and did not screen positive for BPD. Of note, in both the study by Yip et al and the study by Lee et al, the prevalence of BPD per the MSI-BPD was 21%, whereas in the study by Feske et al, the prevalence of BPD was 14%. Thus, the true prevalence of BPD in patients referred for ECT is likely lower than the 21% reported by Yip et al and Lee et al. Based on a BPD prevalence of 14%, and the values from the meta-analysis⁸ of the MSI-BPD’s sensitivity (81.7%) and specificity (63.2%), the positive predictive value of the MSI-BPD would be 26.5% in Yip and colleagues’ study. That is, nearly three-quarters of the patients in the BPD screen–positive cohort would not be diagnosed with BPD when interviewed.

A screening measure is not a diagnostic test. The MSI-BPD does not identify a group of patients who have BPD. The likelihood that patients who screen positive on the MSI-BPD have BPD (ie, positive predictive value) depends on the prevalence of the disorder in the sample. When the prevalence is low, it is more likely than not that the patient who screens positive does not have BPD. Rather, a positive screen on the MSI-BPD simply identifies patients who are more likely to have BPD than patients who screen negative. A positive screen should be followed by a diagnostic interview.

In their Discussion, Yip et al⁶ mention in passing that the MSI-BPD is a screening tool, but the title of their article and the rest of their Discussion refer to diagnosable BPD. Nowhere do they suggest that it is possible if not likely that the majority of the patients whom them considered to have BPD likely would not have been so diagnosed if interviewed. Future literature reviews and meta-analyses of the impact of BPD on the efficacy of ECT in patients with MDD need to carefully consider the procedures used to diagnose BPD. I would recommend the exclusion of studies using a screening measure to “diagnose” BPD.

While one can raise other questions about the use of a self-report questionnaire to identify patients with BPD while patients are depressed, a discussion of issues concerning personality assessment such as state effects might distract from the core concern of this commentary—the misuse of screening measures to identify a diagnostic group. Such a misuse of screening scales for this purpose is not limited to BPD but rather applies broadly to any study relying on a screening test to identify the index diagnostic group.¹¹ The use of descriptors such as probable or suspected to describe the index group identified by the screening test is insufficient. Rather, studies using screening tests as diagnostic proxies need to discuss the positive predictive value of the screening test.

In conclusion, meaningful conclusions about the efficacy of ECT, or any treatment for that matter, in patients with MDD with comorbid BPD cannot be drawn from studies that use a screening measure such as the MSI-BPD to compose the BPD cohort.

Published online: March 30, 2021.
Potential conflicts of interest: None.
Funding/support: None.