Predictive Modeling of Suicidal Ideation in US Veterans: A Head-to-Head Comparison of the Patient Health Questionnaire-2 Versus Complex Machine Learning Approaches

Levin E. Waechli, MMed; Lea S. Tschaggelar, BMed; Philipp Homan, PhD; Antonia Lueoend, MD; Ian C. Fischer, PhD; Tobias R. Spiller, MD; Robert H. Pietrzak, PhD, MPH

J Clin Psychiatry 2026;87(3):25m15965

Abstract

Objective: Suicide among veterans remains a public health concern, underscoring the need for accurate and clinically feasible approaches to predicting suicide risk. This study evaluated whether increasingly complex machine learning models offer meaningful improvements in predicting suicidal ideation (SI) in veterans over a simple Patient Health Questionnaire-2 (PHQ-2) sum score.

Methods: Data were analyzed from 2 waves of the National Health and Resilience in Veterans Study, a nationally representative, longitudinal survey of US veterans (collected in winter 2019/2020 and winter 2020/2021). We compared the performance of a simple PHQ-2 sum score model with a “clinically available variables” set including 9 predictors and a “comprehensive variables” set containing 32 predictors. Models utilizing logistic regression, random forest, gradient boosting machine, support vector machine, and a neural network were used to predict past-year SI. Performance was assessed using several metrics, including area under the curve (AUC).

Results: The PHQ-2 sum score model achieved an AUC of 0.80, while the AUCs of the “clinically available variables” model ranged from 0.78 to 0.83. The AUCs of the “comprehensive variables” model set ranged from 0.85 to 0.88.

Conclusions: Machine learning models in both model sets provided a statistically significant, yet clinically modest, performance improvement over the PHQ-2 sum score, suggesting that more complex approaches do not necessarily translate into clinically meaningful advantages in predicting SI. These findings highlight the importance of balancing data demands, model interpretability, and real-world feasibility when developing suicide risk prediction tools, and caution against complex models when simpler alternatives perform comparably.

J Clin Psychiatry 2026;87(3):25m15965

Author affiliations are listed at the end of this article.

Members Only Content

This full article is available exclusively to Professional tier members. Subscribe now to unlock the HTML version and gain unlimited access to our entire library plus all PDFs. If you're already a subscriber, please log in below to continue reading.

Subscribe Now

Already a member? Log in

References (51)

Suicide - National Institute of Mental Health (NIMH). Accessed November 11, 2024. https://www.nimh.nih.gov/health/statistics/suicide
World Health Organization. Factsheet Suicide; 2025. Accessed November 16, 2025. https://www.who.int/news-room/fact-sheets/detail/suicide
US Department of Veterans Affairs. 2024 National Veteran Suicide Prevention Annual Report. 2024. Accessed November 16, 2025. https://www.mentalhealth.va.gov/docs/data-sheets/2024/2024-Annual-Report-Part-1-of-2_508.pdf
Nichter B, Hill ML, Na PJ, et al. Prevalence and trends in suicidal behavior among US military veterans during the COVID-19 pandemic. JAMA Psychiatry. 2021;78(11):1218–1227. PubMed
Fischer IC, Nichter B, Aunon FM, et al. Suicidal thoughts and behaviors in older US military veterans: results from the National Health and Resilience in Veterans Study. Am J Geriatr Psychiatry. 2023;31(10):844–852. PubMed
Fortney JC, Curran GM, Hunt JB, et al. Prevalence of probable mental disorders and help-seeking behaviors among veteran and non-veteran community college students. Gen Hosp Psychiatry. 2016;38:99–104. PubMed
Kuwert P, Knaevelsrud C, Pietrzak RH. Loneliness among older veterans in the United States: results from the National Health and Resilience in Veterans Study. Am J Geriatr Psychiatry. 2014;22(6):564–569. PubMed
Kessler RC, Bossarte RM, Luedtke A, et al. Suicide prediction models: a critical review of recent research with recommendations for the way forward. Mol Psychiatry. 2020;25(1):168–179. PubMed
Pigoni A, Delvecchio G, Turtulici N, et al. Machine learning and the prediction of suicide in psychiatric populations: a systematic review. Transl Psychiatry. 2024;14(1):140. PubMed
Andreotti ET, Ipuchima JR, Cazella SC, et al. Instruments to assess suicide risk: a systematic review. Trends Psychiatry Psychother. 2020;42(3):276–281. PubMed
Carter G, Milner A, McGill K, et al. Predicting suicidal behaviours using clinical instruments: systematic review and meta-analysis of positive predictive values for risk scales. Br J Psychiatry. 2017;210(6):387–395. PubMed
Atmakuru A, Shahini A, Chakraborty S, et al. Artificial intelligence-based suicide prevention and prediction: a systematic review (2019–2023). Inf Fusion. 2025;114:102673.
Kessler RC, Hwang I, Hoffmire CA, et al. Developing a practical suicide risk prediction model for targeting high-risk patients in the Veterans Health Administration. Int J Methods Psychiatr Res. 2017;26(3):e1575. PubMed
Kessler RC, Bernecker SL, Bossarte RM, et al. The role of big data analytics in predicting suicide. In: Passos IC, Mwangi B, Kapczinski F, eds. Personalized Psychiatry. Springer International Publishing; 2019:77–98.
Beede E, Baylor E, Hersch F, et al. A human-centered evaluation of a deep learning system deployed in clinics for the detection of diabetic retinopathy. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. In: CHI ’20. Association for Computing Machinery; 2020:1–12.
Rousson V, Trächsel B, Iglesias K, et al. Evaluating the cost of simplicity in score building: an example from alcohol research. PLOS ONE. 2023;18(11):e0294671 PubMed
Kim M, Jung S, Park JE, et al. Validation of the Patient Health Questionnaire–9 and Patient Health Questionnaire–2 in the general Korean population. Psychiatry Investig. 2023;20(9):853–860. PubMed
Hopkins D, Rickwood DJ, Hallford DJ, et al. Structured data vs. unstructured data in machine learning prediction models for suicidal behaviors: a systematic review and meta-analysis. Front Digit Health. 2022;4:945006. doi:10.3389/fdgth.2022.945006. PubMed CrossRef
Simon GE, Johnson E, Shortreed SM, et al. Predicting suicide death after emergency department visits with mental health or self-harm diagnoses. Gen Hosp Psychiatry. 2024;87:13–19. PubMed
Zheng L, Wang O, Hao S, et al. Development of an early-warning system for high-risk patients for suicide attempt using deep learning and electronic health records. Transl Psychiatry. 2020;10(1):1–10. PubMed
Jankowsky K, Steger D, Schroeders U. Predicting lifetime suicide attempts in a community sample of adolescents using machine learning algorithms. Assessment. 2024;31(3):557–573.
Lemaire CM, Graham DP. Factors associated with suicidal ideation in OEF/OIF veterans. J Affect Disord. 2011;130(1):231–238. PubMed
Nichter B, Stein MB, Monteith LL, et al. Risk factors for suicide attempts among US military veterans: a 7-year population-based, longitudinal cohort study. Suicide Life Threat Behav. 2022;52(2):303–316.
Tsai J, Schick V, Hernandez B, et al. Is homelessness a traumatic event? Results from the 2019–2020 National Health and Resilience in Veterans Study. Depress Anxiety. 2020;37(11):1137–1145. PubMed
Kroenke K, Spitzer RL, Williams JBW, et al. An ultra-brief screening scale for anxiety and depression: the PHQ–4. Psychosomatics. 2009;50(6):613–621. PubMed
Osman A, Bagge CL, Gutierrez PM, et al. The Suicidal Behaviors Questionnaire-Revised (SBQ-R): validation with clinical and nonclinical samples. Assessment. 2001;8(4):443–454. PubMed
Spiller TR, Na PJ, Merians AN, et al. Changes in mental health among US military veterans during the COVID-19 pandemic: a network analysis. J Psychiatr Res. 2023;165:352–359. PubMed
Arroll B, Goodyear-Smith F, Crengle S, et al. Validation of PHQ-2 and PHQ-9 to screen for major depression in the primary care population. Ann Fam Med. 2010;8(4):348–353. PubMed
Fowler JC. Suicide risk assessment in clinical practice: pragmatic guidelines for imperfect assessments. Psychotherapy. 2012;49(1):81–90. PubMed
Wong A, Young AT, Liang AS, et al. Development and validation of an electronic health record–based machine learning model to estimate delirium risk in newly hospitalized patients without known cognitive impairment. JAMA Netw Open. 2018;1(4):e181018. PubMed
Spiller TR, Tufan E, Petry H, et al. Delirium screening in an acute care setting with a machine learning classifier based on routinely collected nursing data: a model development study. J Psychiatr Res. 2022;156:194–199. PubMed
Eloranta S, Boman M. Predictive models for clinical decision making: deep dives in practical machine learning. J Intern Med. 2022;292(2):278–295. PubMed
DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44(3):837–845. PubMed
Collins GS, Moons KGM, Dhiman P, et al. TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ. 2024;385:e078378. PubMed
Corson K, Gerrity MS, Dobscha SK. Screening for depression and suicidality in a VA primary care setting: 2 items are better than 1 item. Am J Manag Care. 2004;10(11 Pt 2):839–845. PubMed
Kessler RC, Bauer MS, Bishop TM, et al. Using administrative data to predict suicide after psychiatric hospitalization in the Veterans Health Administration system. Front Psychiatry. 2020;11:390. doi:10.3389/fpsyt.2020. 00390. CrossRef
Jiang T, Rosellini AJ, Horváth-Puhó E, et al. Using machine learning to predict suicide in the 30 days after discharge from psychiatric hospital in Denmark. Br J Psychiatry J Ment Sci. 2021;219(2):440–447. PubMed
Passos IC, Mwangi B, Cao B, et al. Identifying a clinical signature of suicidality among patients with mood disorders: a pilot study using a machine learning approach. J Affect Disord. 2016;193:109–116. PubMed
Walsh CG, Ribeiro JD, Franklin JC. Predicting risk of suicide attempts over time through machine learning. Clin Psychol Sci. 2017;5(3):457–469.
Colic S, He JC, Richardson JD, et al. A machine learning approach to identification of self-harm and suicidal ideation among military and police Veterans. J Mil Veteran Fam Health. 2022;8(1):56–67.
Thornton J, Tandon R. Does machine-learning-based prediction of suicide risk actually reduce rates of suicide: a critical examination. Asian J Psychiatry. 2023;88:e103769.
Roza TH, Salgado TA, Machado CS, et al. Prediction of suicide risk using machine learning and big data. In: Passos IC, Rabelo-da-Ponte FD, Kapczinski F, eds. Digital Mental Health: A Practitioner’s Guide. Springer International Publishing; 2023:173–188.
Ehtemam H, Sadeghi Esfahlani S, Sanaei A, et al. Role of machine learning algorithms in suicide risk prediction: a systematic review-meta analysis of clinical studies. BMC Med Inf Decis Mak. 2024;24(1):138.
Kirtley OJ, Mens K, Hoogendoorn M, et al. Translating promise into practice: a review of machine learning in suicide research and prevention. Lancet Psychiatry. 2022;9(3):243–252. PubMed
Yarborough BJH, Stumbo SP, Schneider J, et al. Clinical implementation of suicide risk prediction models in healthcare: a qualitative study. BMC Psychiatry. 2022;22(1):789. PubMed
Maurer DM, Raymond TJ, Davis BN. Depression: screening and diagnosis. Am Fam Physician. 2018;98(8):508–515. PubMed
Chen S, Zhang X, Lin S, et al. Suicide risk stratification among major depressed patients based on a machine learning approach and whole-brain functional connectivity. J Affect Disord. 2023;322:173–179. PubMed
Xu X, Yao B, Dong Y, et al. Mental-LLM: leveraging large language models for mental health prediction via online text data. Proc ACM Interact Mob Wearable Ubiquitous Technol. 2024;8(1):1–32.
Fischer IC, Nichter B, Na PJ, et al. Longitudinal trends in suicidal thoughts and behaviors among US military veterans during the COVID-19 pandemic. JAMA Psychiatry. 2023;80(6):577–584. PubMed
Nock MK, Borges G, Bromet EJ, et al. Cross-national prevalence and risk factors for suicidal ideation, plans and attempts. Br J Psychiatry. 2008;192(2):98–105.
Glavin D, Grua EM, Nakamura CA, et al. Patient Health Questionnaire-9 item pairing predictiveness for prescreening depressive symptomatology: machine learning analysis. JMIR Ment Health. 2023;10(1):e48444. PubMed