J Clin Psychiatry 2022;83(2):21m14274

To cite: Liebmann EP, Resnick SG, Stacy MA. Measurement-based care for psychotic disorders in the Veterans Health Administration: current practices and future directions. J Clin Psychiatry. 2022;83(2):21m14274.
To share: https://doi.org/10.4088/JCP.21m14274

© Copyright 2022 Physicians Postgraduate Press, Inc.

^aDepartment of Psychology, VA Connecticut Healthcare System, West Haven, Connecticut
^bYale School of Medicine, Department of Psychiatry, New Haven, Connecticut
^cOffice of Mental Health and Suicide Prevention, VA Central Office, Washington, DC
*Corresponding author: Meaghan Stacy, PhD, Department of Psychology, VA Connecticut Healthcare System, 950 Campbell Ave – 116B, West Haven, CT 06516 (meaghan.stacy@yale.edu).

CME Background

Articles are selected for credit designation based on an assessment of the educational needs of CME participants, with the purpose of providing readers with a curriculum of CME articles on a variety of topics throughout each volume. Activities are planned using a process that links identified needs with desired results.

To obtain credit, read the article, correctly answer the questions in the Posttest, and complete the Evaluation. This activity is free.

CME Objective

After studying this article, you should be able to:

• Incorporate measurement-based care into treatment for psychotic disorders

Accreditation Statement

The CME Institute of Physicians Postgraduate Press, Inc., is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians.

Credit Designation

The CME Institute of Physicians Postgraduate Press, Inc., designates this journal-based CME activity for a maximum of 1 AMA PRA Category 1 Credit™. Physicians should claim only the credit commensurate with the extent of their participation in the activity.

Note: The American Nurses Credentialing Center (ANCC) and the American Academy of Physician Assistants (AAPA) accept certificates of participation for educational activities certified for AMA PRA Category 1 Credit™ from organizations accredited by the ACCME.

Release, Expiration, and Review Dates

This educational activity was published in March 2022 and is eligible for AMA PRA Category 1 Credit™ through April 30, 2024. The latest review of this material was February 2022.

Financial Disclosure

All individuals in a position to influence the content of this activity were asked to complete a statement regarding all relevant personal financial relationships between themselves or their spouse/partner and any commercial interest. The CME Institute has resolved any conflicts of interest that were identified. In the past 3 years, Marlene P. Freeman, MD, Editor in Chief, has received research funding from JayMac and Sage; has been a member of the Independent Data Safety and Monitoring Committee for Janssen (Johnson & Johnson), Novartis, and Neurocrine; and has served on advisory boards for Eliem and Sage. As an employee of Massachusetts General Hospital (MGH), Dr Freeman works with the MGH National Pregnancy Registry, which receives funding from Alkermes, Aurobindo, AuroMedics, Johnson & Johnson/Janssen, Otsuka, Sage, Sunovion, Supernus, and Teva, and works with the MGH Clinical Trials Network and Institute, which receives research funding from multiple pharmaceutical companies and the National Institute of Mental Health. Dr Freeman has also received royalties through MGH for the Massachusetts General Hospital Female Reproductive Lifecycle and Hormones Questionnaire. No member of the CME Institute staff reported any relevant personal financial relationships. Faculty financial disclosure appears at the end of the article.

“When one is dealing with human lives and life opportunities, it is immoral to adopt a mode of decision-making which has been demonstrated repeatedly to be either inferior in success rate or, when equal, costlier to the client or the taxpayer.”^{1(p. viii)}

Providers regularly collect and interpret a range of clinical data to make treatment decisions, but their ability to predict outcome based on clinical judgment alone is inferior to methods based on structured quantitative assessment.^1–4 One such approach is measurement-based care (MBC), a multistep process that includes frequent assessment and timely feedback of outcomes during the clinical encounter, which is then used to guide clinical care.^5–8 Within Veterans Health Administration (VHA) mental health services, MBC is defined as a clinical process in which a Veteran independently completes patient-reported outcome measures (PROMs) immediately prior to the clinical encounter, the results of which are discussed in the appointment and used to inform shared decision-making.⁸ MBC is acceptable to clients and providers⁹ and has numerous benefits,^10–19 including improved treatment response and outcomes.^{13,15,16,18,20} Several organizations have recommended or required the implementation of MBC,^21–24 and the VHA began the national MBC in Mental Health Initiative in 2016 with the goal of establishing MBC as the standard of care in mental health services.⁸

To date, the MBC literature has largely focused on depression, anxiety, and posttraumatic stress disorder (PTSD).^25,26 Psychotic disorders, including schizophrenia and bipolar disorders, have significant consequences for those with the diagnosis and for society.^27–32 Incorporating MBC into treatment for psychotic disorders may be beneficial given that it has the potential to reduce variability in mental health care and improve patient outcomes¹⁴ and may facilitate symptom remission³³ and increase mental health literacy.³⁴ Adding MBC to usual care can improve symptoms, interpersonal problems, social role functioning, and quality of life, especially for clients likely to experience treatment nonresponse.¹⁷ This is especially important given that people with psychotic disorders are often underserved and experience health disparities.³⁵ In this way, MBC can be viewed as evidence-based and recovery-oriented, a priority in systems serving individuals with psychotic disorders.^26,35,36

Despite the benefits of MBC and the fact that providers view it favorably,³⁷ it has not been widely adopted.^2,7,38 Beyond the obstacles that impede the adoption of MBC generally,^7,8,39 implementing MBC for psychotic disorders is further complicated by a lack of well-established recommendations for psychosis-specific PROMs,^25,40 challenges in using self-report measures,⁴¹ and a lack of research on measures of functioning or quality of life specifically as part of MBC for psychotic disorders. When research is limited, current clinical practices can be informative.

Understanding the use of PROMs with Veterans with psychotic disorders in VHA mental health programs may be illustrative and informative to current practitioners. This article fills a gap in the literature by examining the frequencies and proportional use of PROMs to identify which are commonly used by VHA providers when working with Veterans with psychotic disorders. We also compare the relative frequency and proportional use of the same measures among all other Veterans receiving PROMs in VHA. Characterizing use patterns will serve as an initial step to identifying potential candidate measures for psychosis-related MBC practice and studies.

METHODS

Data Acquisition and Sample

The study was approved by the Institutional Review Board at VA Connecticut Healthcare System. Data on Veterans with at least 1 mental health encounter and at least 1 completed PROM administered during fiscal years (FYs) 2016–2019 (FY16–FY19) were extracted from the VHA Corporate Data Warehouse (CDW), including variables describing age, sex, race, mental health diagnosis, and presence of standard categories of service utilization within each FY. Veterans were considered to have a diagnosis of a psychotic disorder if 2 or more outpatient mental health visits or 1 bed day of care indicated an ICD-10 diagnosis of schizophrenia, schizoaffective disorder, bipolar disorder, or other psychotic disorders.

The initial dataset included 4,106,267 PROMs from 1,214,347 Veterans. Of these, the psychotic disorder status of 21,450 (1.77%) unique Veterans changed at least once, so they were excluded to preclude counting these Veterans twice in aggregate analyses. The final analytic sample included 3,935,504 PROM administrations and 1,192,897 Veterans.

Patient Reported Outcome Measures (PROMs)

We examined a wide range of PROMs available in Mental Health Assistant (MHA) and/or Behavioral Health Laboratory (BHL), software systems that integrate with the VA’s electronic health record and store data in CDW.⁸ Measures recommended by a US federal interagency task force for use in MBC have been promoted as core measures in VHA^24,25: the Patient Health Questionnaire-9 (PHQ-9)⁴² the General Anxiety Disorder-7 Scale (GAD-7)⁴³; the PTSD Checklist-5 (PCL-5),⁴⁴ and a measure of substance use created for MBC in VHA, the Brief Addiction Monitor (BAM).⁴⁵ Three versions of the BAM were combined: BAM, BAM-Revised (BAM-R), and the BAM for intensive outpatient programs (BAM-IOP).

Additional PROMs spanned multiple constructs. Depression measures included the Beck Depression Inventory-II (BDI-II),⁴⁶ the Center for Epidemiologic Studies Depression Scale (CES-D),⁴⁷ and the Zung Depression Scale (ZUNG).⁴⁸ Those measuring anxiety/stress included the Beck Anxiety Inventory (BAI)⁴⁹ and the Perceived Stress Scale (PSS).⁵⁰ Multisymptom measures included the Symptom Checklist-90-Revised (SCL-90-R)⁵¹ and the Behavior and Symptom Identification Scale-24 item (BASIS-24)⁵²; given low usage, we combined the BASIS-24 scale and BASIS-4 Psychosis subscale. Insomnia-related PROMs included the Sleep Need Questionnaire (SNQ)⁵³ and Insomnia Severity Inventory (ISI).⁵⁴ Quality of life/well-being scales were the full and short versions of the World Health Organization Disability Assessment Schedule (WHODAS),⁵⁵ the Quality of Life Inventory (QOLI),⁵⁶ the Veterans RAND 12-Item Health Survey (VR12),⁵⁷ the full and short versions of the Warwick-Edinburgh Mental Well-being Scale (WEMWBS),⁵⁸ the Medical Outcomes Study Short Form 36 Health Survey (SF-36),⁵⁹ the Quality of Life and Satisfaction Questionnaire–Short Form (Q-LES-Q-SF),⁶⁰ and the World Health Organization Quality of Life abbreviated version (WHOQOL-BREF).⁶¹ Other measures were the Pain Outcomes Questionnaire (POQ),⁶² the Working Alliance Inventory–Short Revised (WAI-SR),⁶³ the Internalized Stigma of Mental Illness Inventory (ISMI),⁶⁴ and the full and short versions of the Mental Health Recovery Measure (MHRM).⁶⁵

All PROMs were available for the full study period except the ISI and PCL-5. The ISI was added in June 2018. The PCL-5, which replaced the civilian (PCL-C) and military (PCL-M) versions of the PCL, was added in November 2016. In this dataset, the earliest observed administrations of the ISI and PCL-5 for both groups occurred in the second quarter of FY18 and the first quarter of FY16, respectively.

Analysis

Percentages of total PROM administrations were calculated by aggregating across time and Veterans and calculated for Veterans with and without psychotic disorders. The number and percentage of facilities administering any PROMs were calculated by aggregating the data pooled across time and diagnosis to the facility level (total = 141 facilities) and determining which sites had a total count greater than zero. Facility means, standard deviations, and minimums and maximums of the percentage total for each PROM were calculated to characterize variability in PROM utilization at the systems level.

Trends in the percentage of unique Veterans administered a particular PROM over time were examined by determining the percentage of the total number of unique Veterans per quarter who had received a particular PROM. Absolute change (%Unique_FY19Q4 – %Unique_FY16Q1) was calculated for each PROM to characterize the direction and magnitude of change over time. Trends in the repeated administration of PROMs over time were also examined. Analyses of repeated administration were limited to Veterans who had > 1 visit within a quarter (n = 675,989, 56.67%). This constraint was imposed to include in the denominator for only those Veterans for whom it would have been possible to have a repeated PROM administration (ie, had multiple visits). Percentages for each quarter were created by calculating the proportion of Veterans with > 1 PROM administration to those with > 0 PROM administrations. Absolute change in the percentage of repeated PROMs (%Repeated_FY19Q4 – %Repeated_FY16Q1) was also calculated.

RESULTS

Sample characteristics are presented in Table 1. Veterans with psychotic disorder diagnoses had lower rates of being married (30.10% vs 48.09%) and an Operation Enduring Freedom/Operation Iraqi Freedom/Operation New Dawn (OEF/OIF/OND) Veteran (17.33% vs 32.24%) compared to Veterans without psychotic disorders. A greater proportion of Veterans with psychotic disorders had alcohol use disorders (35.71% vs 21.30%) and drug use disorders (35.50% vs 14.17%) compared to Veterans without psychotic disorders. A larger percentage of Veterans with psychotic disorders had histories of outpatient substance use disorder treatment (25.12% vs 15.11%), inpatient treatment (27.51% vs 6.28%), residential treatment (12.69% vs 3.93%), and acute care (22.27% vs 3.65%).

Pooling data across time and diagnosis, all 141 VA facilities administered the 4 core MBC measures during FY16–FY19 at least once, but with varied percentages of administration (Table 2). The PHQ-9 accounted for 44% of PROMs across facilities and nearly 85% of PROMs at maximum. In comparison, the PCL-5, GAD-7, and BAM accounted for 20.27%, 16.42% and 10.28%, respectively. The 4 core PROMs accounted for 88.18% of all PROMs administered to Veterans with psychotic disorders and 92.56% of PROMs administered to Veterans without psychotic disorders.

Among the non-core measures, only the BDI-II and the BAI were administered at all facilities and accounted for the largest percentage of PROMs at the facility and diagnostic levels compared to other non-core PROMs. The quality of life/well-being PROMs accounted for the greatest number of PROMs for a single category of non-core PROMs and exhibited wide variation in systems-level use across measures. The WHODAS was administered by 99% of facilities, whereas the WHOQOL was administered at < 3% of facilities. Even among quality of life/well-being measures with nearly 100% of facilities administering them, the mean percentage of total PROMs administered across facilities did not exceed 1% for any of these measures. At the patient level, while the WHODAS, QOLI, VR-12, and WEMWBS accounted for a larger percentage of PROM administrations among Veterans with psychotic disorders compared to those without, the contribution of these measures across both diagnostic groups was low.

Among the remaining non-core PROMs, rates of use by facilities varied from 99% for the ISI to 11% for the MHRM. The mean percentage total for the measures across facilities did not exceed 0.72% for any of these measures. At the patient level, the percentages of total administration were higher among those with psychotic disorders, except for the BRS and SNQ; however, these PROMs accounted for less than 1% of all administrations for both groups.

Trends Over Time

Figure 1 depicts trajectories by group of the quarterly percentage of unique Veterans administered a core PROM and the top 6 non-core PROMs based on the percentage of total PROMs administered. The PHQ-9 had an absolute change of 4.62% and 1.47% for Veterans with and without psychotic disorders, respectively. The absolute change for Veterans with and without psychotic disorders for the PCL-5 (17.9% and 29.00%) and the GAD-7 (23.99% and 26.69%) exceeded that for the PHQ-9. The BAM had a slightly negative trajectory over time for Veterans with and without psychotic disorders (−4.00% vs −1.77%). Trajectories for the BDI-II and BAI indicate decreasing administration; the absolute change for Veterans with and without psychotic disorders were negative for the BDI-II (−8.90% vs −9.33%) and the BAI (−2.79% vs −3.64%), respectively. The absolute change for Veterans with and without psychotic disorders was 3.85% versus −0.46% for the WHODAS, 2.62% versus 0.90% for the QOLI, 2.64% versus 0.44% for the BASIS-24, and 1.09% versus 3.46% for the ISI.

The quarterly percentages of repeated PROM administrations by diagnostic group are presented in Figure 2. The percentage of repeated core PROM administrations increased during FY16–FY19. The absolute change in percentage of repeated PROM administration for Veterans with and without psychotic disorders was 16.16% versus 11.15% for the PHQ-9, 29.63% versus 25.96% for the PCL-5, 12.99% versus 9.18% for the BAM, and 9.38% versus 11.35% for the GAD-7. The absolute change for the BDI-II and BAI among Veterans with psychotic disorders was 4.11% and 8.66%, respectively, and −1.09% and 1.63%, respectively, for Veterans without psychotic disorders. The WHODAS exhibited positive absolute change for Veterans with and without psychotic disorders (9.44% vs 10.39%), while the QOLI exhibited negative absolute change for both groups (−0.86% vs −4.67%).

DISCUSSION

MBC is a low-cost, potentially low-burden, and transdiagnostic intervention that enhances the speed with which a patient receives appropriate interventions and obtains optimal results. It can improve the quality of care by influencing patient-physician interactions and encouraging shared decision-making.⁵ Despite its promise, MBC has not been fully adopted, and minimal literature has focused on MBC for psychotic disorders. Although there has recently been effort to highlight potential PROMs for use with psychotic disorders,⁴⁰ to our knowledge, there are no research studies of self-report measures being used for MBC with psychotic disorders. This study bridges that gap by assessing what measures VHA providers are using in the treatment of psychotic disorders.

The use of PROMs in VHA mental health care for Veterans has increased in recent years, consistent with the MBC in MH Initiative and mandates from health care organizations and accrediting bodies.⁸ Notably, the measures used by providers did not differ based on diagnostic category; most PROMs administered to Veterans with and without psychotic disorders were 4 core measures (PHQ-9, BAM, PCL-5, and GAD-7) identified by VHA for MBC, particularly the PHQ-9. Further, the proportion of unique Veterans who completed repeated measures did not differ between those with and without psychotic disorders. While the MBC in MH Initiative required that at least one core measure be adopted, it also strongly encouraged the use of quality of life, functioning, and other non-symptom measures. However, our results suggest that, when working with Veterans with or without psychotic disorders, little experimentation occurred beyond the 4 core PROMs, and no clear consensus emerged outside of the core measures. This is encouraging in that it suggests Veterans with psychotic disorders receive MBC at similar rates to those without psychotic disorders; however, it highlights the need for psychosis-specific measures to tailor MBC for Veterans with these diagnoses.

While the pattern of PROMs usage for Veterans with psychotic disorders largely mirrored that for Veterans without psychotic disorders, it is noteworthy that the BAM was the second most common measure for individuals with psychotic disorders, but ranked fourth for those with non-psychotic disorders. This potentially reflects the higher prevalence of substance use among those with psychotic disorders^66,67 and suggests providers are tailoring assessments to individual Veterans.

Given that MBC is, in part, based on the premise of using repeated measures to inform treatment, understanding the use of repeated measures within this sample is critical. For both diagnostic groups, the use of repeated measures increased over time, reflecting the increasing implementation of MBC throughout VHA. The repeated measures with the greatest increases over time were also the 4 core measures, again highlighting the importance of clear guidelines and need for psychosis-specific PROMs to inform implementation of MBC for psychotic disorders.

Psychiatric rehabilitation promotes the pursuit of personal goals despite the existence of symptoms.^68,69 As such, MBC in the context of psychotic disorders is an opportunity for measures of quality of life, well-being, and recovery to guide treatment. In this study, the proportion of Veterans with psychotic disorders who completed the QOLI, WHODAS, VR-12, WEMWBS, Q-LES-Q-SF, and MHRM were larger than the proportion of Veterans without; however, the proportions who were administered these measures were small. While this finding again suggests providers are tailoring assessments to each individual patient, it also highlights the opportunity for providers to incorporate recovery-oriented measures, especially when considering the importance individuals with psychosis place on personal recovery as an outcome.⁴⁰ It also suggests a potential need for additional training or specific organizational commitment to incorporating recovery-oriented measures more broadly for individuals with psychosis. Although such measures exist, they have not been systematically examined in an MBC paradigm (eg, sensitivity to change as part of treatment), and anecdotal reports suggest that providers may not know how to integrate these PROMs into clinical practice. PROMs used for MBC should measure what is targeted in treatment. If practitioners are largely focusing on symptom reduction, tracking measures that are unlikely to change over time without direct intervention, such as quality of life, may actually be detrimental to treatment outcomes and the working alliance.

For MBC to be successful, measures must be self-report, current, and actionable during the encounter^5,70 and used to guide care. Ideal PROMs are brief, low-cost, publicly available, easily administered, acceptable to clients and clinicians, psychometrically sound, able to be repeated frequently, sensitive to change, and able to differentiate between desired and undesired outcomes.^14,71–74 While there are candidates for measures for psychotic disorders, few meet criteria for MBC. Recently, McKenzie et al⁴⁰ proposed a standardized set of 9 PROMs to support MBC for psychotic disorders, though the selection criteria did not require thresholds to depict meaningful change or severity, so their utility in making treatment decisions may be limited. It is promising that there is some overlap between this set of 9 measures and the measures identified by the current study, specifically the PHQ-9; however, it is noteworthy that the remaining 8 measures recommended by McKenzie et al⁴⁰ are largely absent in current VHA practice and suggest potential measures for future investigation and implementation. Future work is needed to evaluate existing measures for MBC in psychotic disorders, create and validate brief measures for MBC in psychotic disorders, and assess the effectiveness of incorporating MBC into self-management interventions to maintain gains and recognize signs of relapse.⁷⁵

As with all studies, there are several limitations to note. This study uses VHA data, which may not be representative of other settings. PROMs in VHA administrative data are limited to the assessments in MHA and BHL software. There may be other candidate measures not included in MHA/BHL that VA clinicians would use during treatment of psychotic disorders or are using but not able to document in MHA/BHL. We also cannot confirm that these measures were utilized according to the principles of measurement-based care. The data convey only that measures were administered; we do not know what transpired within the clinical encounter. Providers may not have provided feedback or used the information to make shared decisions about treatment. Future qualitative research or natural language processing studies of clinical documentation might inform our ability to understand the frequency of “share and act” within clinical encounters. Despite these limitations, this study is the first, to our knowledge, to evaluate the relative use of measures in the setting of MBC for those with and without psychotic disorders, a critical step for MBC to be widely and effectively implemented throughout mental health care systems.

Submitted: September 30, 2021; accepted December 22, 2021.
Published online: March 1, 2022.
Disclosure of off-label usage: Dr Stacy has determined that, to the best of her knowledge, no investigational information about pharmaceutical agents or device therapies that is outside US Food and Drug Administration–approved labeling has been presented in this article.
Financial disclosure: Drs Liebmann, Resnick, and Stacy have no conflicts to disclose.
Funding/support: The authors are employees of the Veterans Health Administration (VHA). This material is based upon work supported in part by the Department of Veteran Affairs Office of Academic Affiliations, Inter-Professional Advanced Fellowship in Addiction. The authors have no conflicts of interest.
Role of the sponsor: The VHA had no role in the conduct or reporting of this study.
Previous presentation: None.
Additional information: Data from the VHA Corporate Data Warehouse are stored within VHA servers and are not on a URL or accessible by the public.

Clinical Points

• Measurement-based care (MBC) is an evidence-based intervention; however, there is little existing guidance for its use among individuals with psychotic disorders.
• The Patient Health Questionnaire-9 (PHQ-9), General Anxiety Disorder-7 (GAD-7), PTSD Checklist-5 (PCL-5), Brief Addiction Monitor (BAM)–Revised (BAM-R), and BAM for intensive outpatient programs (BAM-IOP) are appropriate for Veterans with and without psychotic disorders.
• Further work in developing MBC for individuals with psychotic disorders is necessary to make it consistent with recovery-oriented and person-centered care.