Interpreting Exploratory Analyses: Reply to Lambrichts et al

J Clin Psychiatry 2022;83(1):21lr14269a

To cite: Andrade C. Interpreting exploratory analyses: reply to Lambrichts et al. J Clin Psychiatry. 2022;83(1):21lr14269a.
To share: https://doi.org/10.4088/JCP.21lr14269a

© Copyright 2021 Physicians Postgraduate Press, Inc.

^aDepartment of Clinical Psychopharmacology and Neurotoxicology, National Institute of Mental Health and Neurosciences, Bangalore, India
*Corresponding author: Chittaranjan Andrade, MD, Department of Clinical Psychopharmacology and Neurotoxicology, National Institute of Mental Health and Neurosciences, Bangalore 560 029, India (candrade@psychiatrist.com).

See letter by Lambrichts et al and article by Andrade

To the Editor: Dr Lambrichts et al¹ examined an important subject: predictors of relapse after discontinuation of maintenance electroconvulsive therapy (M-ECT). Their study was special because it emerged out of a natural experiment: M-ECT was perforce abruptly discontinued in all patients because of service delivery constraints during the COVID pandemic.

In their response² to my commentary³ on their study,¹ Lambrichts and colleagues state that the P values for the test of overall significance of their Cox regression models were both below the Bonferroni-corrected threshold for statistical significance. I do not contest this, but what if this was so because of overfitting, the risk of which they acknowledge²? This is food for thought. But my concern with false-positive findings was also more general. No primary hypothesis was stated in the study,¹ and the statistical significance of many independent variables was tested in exploratory analyses in regressions that were run in parallel. Furthermore, as they clarify,² in Table 2 of their paper¹ they reported statistical test results of “all possible pairwise comparisons.” In other words, there were many exploratory statistical analyses conducted with no primary hypothesis stated.

It is well known that the larger the number of statistical tests conducted, the greater the likelihood that some P values will fall below the .05 threshold for declaration of statistical significance. In exploratory analyses, such statistically significant values do not necessarily represent findings that are true in the population; they could be chance findings, and hence false-positive results for hypotheses related to these findings when the hypotheses are stated after the findings are known.

This concern, of course, applies to all exploratory analyses, and so it is not an error; it is merely a limitation. Hypotheses generated in exploratory analyses can certainly be important but need to be confirmed in subsequent studies.

Published online: December 7, 2021.
Potential conflicts of interest: Please see https://www.psychiatrist.com/jcp/about-dr-andrade.
Funding/support: None.