Risk Prediction Models in Psychiatry: Toward a New Frontier for the Prevention of Mental Illnesses
Toward a New Frontier for the Prevention of Mental Illnesses
Objective: We conducted a systematic, qualitative review of risk prediction models designed and tested for depression, bipolar disorder, generalized anxiety disorder, posttraumatic stress disorder, and psychotic disorders. Our aim was to understand the current state of research on risk prediction models for these 5 disorders and thus future directions as our field moves toward embracing prediction and prevention.
Data Sources: Systematic searches of the entire MEDLINE electronic database were conducted independently by 2 of the authors (from 1960 through 2013) in July 2014 using defined search criteria. Search terms included risk prediction, predictive model, or prediction model combined with depression, bipolar, manic depressive, generalized anxiety, posttraumatic, PTSD, schizophrenia, or psychosis.
Study Selection: We identified 268 articles based on the search terms and 3 criteria: published in English, provided empirical data (as opposed to review articles), and presented results pertaining to developing or validating a risk prediction model in which the outcome was the diagnosis of 1 of the 5 aforementioned mental illnesses. We selected 43 original research reports as a final set of articles to be qualitatively reviewed.
Data Extraction: The 2 independent reviewers abstracted 3 types of data (sample characteristics, variables included in the model, and reported model statistics) and reached consensus regarding any discrepant abstracted information.
Results: Twelve reports described models developed for prediction of major depressive disorder, 1 for bipolar disorder, 2 for generalized anxiety disorder, 4 for posttraumatic stress disorder, and 24 for psychotic disorders. Most studies reported on sensitivity, specificity, positive predictive value, negative predictive value, and area under the (receiver operating characteristic) curve.
Conclusions: Recent studies demonstrate the feasibility of developing risk prediction models for psychiatric disorders (especially psychotic disorders). The field must now advance by (1) conducting more large-scale, longitudinal studies pertaining to depression, bipolar disorder, anxiety disorders, and other psychiatric illnesses; (2) replicating and carrying out external validations of proposed models; (3) further testing potential selective and indicated preventive interventions; and (4) evaluating effectiveness of such interventions in the context of risk stratification using risk prediction models.
J Clin Psychiatry 2017;78(5):572-583
© Copyright 2016 Physicians Postgraduate Press, Inc.
aSchool of Psychiatry, University of Perugia, Perugia, Italy
bDepartment of Epidemiology and Biostatistics, The George Washington University Milken Institute School of Public Health, Washington, DC
cDepartment of Psychiatry, Southwest General Health Center, University Hospitals, Middleburg Heights, Ohio
dDepartment of Psychiatry, Lenox Hill Hospital, New York, New York
eHofstra Northwell School of Medicine, Hempstead, New York
fDepartment of Medicine, Division of Psychiatry, Clinical Psychology and Psychiatric Rehabilitation, University of Perugia, Santa Maria della Misericordia Hospital, Perugia, Italy
*Corresponding author: Michael T. Compton, MD, MPH, Lenox Hill Hospital, Department of Psychiatry, 111 E. 77th St, New York, NY 10075 .
Risk prediction models are statistical models used to estimate the probability that a currently unaffected individual will develop a condition in the future.1 Specifically, such models, derived from large studies, estimate the risk of individuals developing future outcomes based on 1 or more underlying characteristics (risk factors or predictors).2 Prognostic prediction models use combinations of predictors to estimate the absolute probability that a certain outcome will occur within a specific time period in an individual with a particular predictor profile.3
Although work toward developing risk prediction models is relatively nascent in psychiatry, other fields of medicine have carried out extensive research on such models. For example, to assess the risk of breast cancer in healthy women, one of the most widely used tools is the Gail model (also known as the Breast Cancer Risk Assessment Tool).4 Initially developed in 1989, it was first validated in Caucasian women and then modified for different races and ethnicities (though even with the modifications, it underestimates risk in African American women and overestimates risk in Asian women).5 It is used to estimate the 5-year risk of invasive breast cancer.4 The original Gail model was based on follow-up of Caucasian women undergoing annual mammographic screening through the Breast Cancer Detection Demonstration Project.6 It relied on 5 risk factors: age, age at menarche, the number of previous breast biopsies, age at first live birth, and the number of first-degree relatives with breast cancer. The National Surgical Adjuvant Breast and Bowel Project modified this model by precluding its use among women with a history of ductal carcinoma in situ and including the risk factor of atypical ductal hyperplasia on breast biopsy.6
Another commonly known risk prediction model produces the Framingham risk score (FRS) for the prediction of 10-year coronary event risk among individuals without previously diagnosed coronary heart disease (CHD).7 Widely regarded as the gold standard in CHD risk assessment tools, the FRS was developed from the Framingham Heart Study8 and has been validated in multiple populations.9 The Framingham Heart Study is a long-term and ongoing cardiovascular study; it began in 1948 and has provided substantial insight into the epidemiology of and risk factors for cardiovascular disease.10 Two versions of the FRS are currently in use. The first version8 was published in 1998 and includes age, sex, smoking history, blood pressure, cholesterol, high-density lipoprotein cholesterol, and blood glucose level or history of diabetes7; it evaluates CHD risk among individuals who are free of CHD at presentation.9 The second version was published within the National Cholesterol Education Program Adult Treatment Panel III guideline and includes hypertension medication status among its inputs. Furthermore, its use is precluded among those with a diabetes diagnosis (because type 2 diabetes was considered to be a CHD "risk equivalent," having the same 10-year risk as individuals with prior CHD), thereby making the second version of the FRS applicable only to individuals without CHD or diabetes.9 Several studies have reported factors that offer additional predictive value beyond what the FRS achieves; however, limited evidence on these modified models remains.11
Risk prediction has 2 different objectives. One is to assign to each individual in the population sampled a score based on some combination of baseline factors, which can be translated into an estimate of that individual’s probability of having onset of a specific disorder in a subsequent period of time. The FRS is such a score, applicable (in its original form) to a community population of those without previously diagnosed CHD. The outcome is onset of CHD in the subsequent 10 years (eg, a score of 12% means that 12 of 100 people with the particular level of risk based on the predictor values will have a heart attack in the next 10 years). The predictors, all measured at the time of entry into the study, included age, sex, smoking history, etc, and the model assigned weights to those predictors to derive a score. The other objective is to classify each individual as "high risk" or "low risk," a binary classification, based on a certain cut point, usually done to identify those high-risk individuals who warrant a particular preventive intervention (eg, therapeutic lifestyle changes and possibly a low-density lipoprotein (LDL)-lowering drug for those with a 10-year risk > 20% who have an LDL level above the target).
Assessing a risk prediction model’s validity consists of 2 main phases: model development (including internal validation using the same data or data source) and external validation (using new data from a different source). Validation requires demonstrating that the model is accurate in the population for whom it is intended. This process must ascertain the model’s ability to distinguish between individuals with different outcomes ("discrimination") and show the level of agreement between predicted and observed risks in groups of individuals with similar risk predictions ("calibration").2 Several statistical approaches and machine learning algorithms are used to develop predictive models. When the number of predictors being examined is relatively small relative to the sample size, the typically used statistical approaches to developing predictive models include Cox regression, logistic regression, and classification and regression trees (CART). When the number of predictors is large relative to the sample size, a commonly used method is the support vector machine (SVM).12 The SVM is a supervised machine classifier, widely used in many domains because it performs well on high dimensional data (ie, those with more variables included)12; it is able to classify individuals into predefined groups. In recent years, the SVM approach has been successfully applied to disease diagnosis, prediction of transition from latent to overt disease, and treatment prognosis.13
The choice of statistical methods has implications for the accuracy and interpretation of the model. For a thorough discussion of each, including strengths and weaknesses, see Ahmed et al,2 Strobl et al,12 and Wu and Lee.14 CART-derived risk prediction models are fit with bootstrap aggregating (bagging) and boosting (bias reduction for a single tree) techniques.15 Using these methods allows for more flexible regression model fitting by permitting selection of covariates without predefined (parametric or nonparametric) relationships with the outcomes. In some instances, these models have been shown to be superior to other regression models, particularly with complex datasets.16,17
- The development and validation of risk prediction models, which are crucial to the goal of prevention, have been largely neglected to date for many psychiatric illnesses.
- Risk prediction models for psychiatric disorders are feasible and could ultimately provide clinicians with valuable information regarding the level of risk for onset of a specific condition within a particular time frame. Proven preventive interventions could then be provided in a targeted fashion.
- Although much more research is clearly needed, risk prediction models hold promise for accurately stratifying risk and targeting interventions to avert the onset of psychiatric disorders.
The traditional statistical approach is to quantify how close predictions are to the actual outcome using measures such as explained variation and the Brier score. Performance can be quantified further in terms of calibration (ie, Do approximately X of 100 patients with a risk prediction of X% have the outcome?) using a goodness-of-fit test. Furthermore, discrimination is essential (ie, Do patients who have the outcome have higher risk predictions than those who do not?), which can be quantified with measures such as sensitivity, specificity, and the area under the curve (AUC), or concordance statistic, the latter indicating the discriminative ability of generalized linear regression models. For a binary outcome, the concordance is identical to the AUC, which plots the sensitivity against the false-positive rate (or 1 – specificity), for consecutive cutoffs for the probability of an outcome.18 Thus, AUC is defined as the probability that the predicted probability of a randomly selected diseased subject will exceed that of a randomly selected nondiseased subject plus half the probability of a tie. The AUC is a value between 0.5 and 1.0, with a higher value indicating better prediction performance. A prediction model with an AUC value of 0.5 is no better than tossing a coin, and at the other extreme, a model with an AUC value of 1.0 is a perfect model, with 100% accurate predictions.14 For systematic reviews, the success rate difference (SRD), a generalized version of the AUC, is commonly reported. The SRD is defined as the difference between the probability that a diseased person has a higher predicted probability of occurrence than a nondiseased person and the probability that a nondiseased person has a higher predicted probability of disease occurrence than a diseased person.19,20 The SRD corresponds to an effect size defined by Cohen d, and for binary outcomes and ordinal predictors, is equivalent to Kendall Ï„. Traditional measures for binary and survival outcomes include the Brier score to indicate overall model performance, the concordance statistic for discriminative ability (AUC), and goodness-of-fit statistics for calibration.18
For binary classification, where the classes are often called positive and negative classes, different classification outcomes include true positive, true negative, false positive, and false negative, according to what the predicted class and true underlying class were. The simplest and often most relevant performance measure in a classification task is the percentage of correctly made classifications. This classification performance is called accuracy and is defined simply as the ratio of correct classifications to all classifications. While classification performance measures the overall accuracy of the classifier, it loses information about the distribution of true and false positives and negatives. Different performance measures have been developed for measuring different types of errors. Four basic measures are sensitivity, or true-positive rate; specificity, or true-negative rate; positive predictive value (PPV), or precision; and negative predictive value (NPV). Sensitivity and specificity are often used together to assess the distribution of correctly made positive and negative classifications.21
The accuracy of the classification is indicated by the PPV (the probability that a "high-risk" individual subsequently has onset in the specified time period) and NPV (the probability that a "low-risk" individual remains free of onset in the specified time period). PPV and NPV have meaning only in relation to each other, or in relation to the incidence of onset during the specified time period. PPV + NPV – 1 must be greater than 0 for nonrandom decision making.
In recent years, there have been increased research efforts focused on reducing the incidence of mental disorders, and there is now substantial cause for optimism regarding the opportunities for prevention.22 Risk prediction models are key to advancing the goals of prevention. Despite the evidence for the effectiveness and cost-effectiveness of a number of preventive interventions within the mental health field, research focused on the prediction of mental disorders remains insufficient. Improving prevention will require better predictive capabilities as well as effective interventions to prevent mental illnesses.
Given that accurate prediction of risk is a first step toward successful prevention within psychiatry and allied disciplines, we conducted a systematic review of risk prediction models designed and tested for a number of major psychiatric disorders: depression, bipolar disorder, generalized anxiety disorder, posttraumatic stress disorder, and schizophrenia and other primary psychotic disorders. Our goal was to summarize the literature to date on risk prediction models for these 5 disorders and give recommendations for future research directions as our field moves toward embracing prediction and prevention.
Systematic searches of the MEDLINE electronic database (from 1960 through 2013) were conducted independently by the first 2 authors in July 2014. The year 1960 was selected as the start date for the search partly because it was only in the second half of the twentieth century that the science of risk prediction models was developed. Search terms included risk prediction, predictive model, or prediction model combined with depression, bipolar, manic depressive, generalized anxiety, posttraumatic, PTSD, schizophrenia, or psychosis. As depicted in Figure 1, we compiled articles that (1) were published in English, (2) provided empirical data (as opposed to reviews or commentaries), and (3) presented results pertaining to developing or validating a risk prediction model in which the outcome was the diagnosis of 1 of the 5 aforementioned mental illnesses. The titles and abstracts of all identified citations were screened based on the defined inclusion criteria. All references within included studies and those of any previous pertinent reviews were carefully reviewed to identify additional relevant studies. Any articles published after the search date that became known to the investigative team were also included. Consensus was then derived, resulting in a final set of articles that would be reviewed and summarized. Based on the above search terms and criteria, 268 articles were initially identified. From this list, we selected 30 original research reports. Additionally, 13 original research articles were identified from 5 review articles.12,23-26 Four non-English articles,27-30 though potentially relevant to the review, were excluded. The 2 independent reviewers then abstracted 3 types of data elements (sample characteristics, variables included in the model, and reported model statistics) and again came together to reach consensus regarding any discrepant abstracted information. Table 1 summarizes findings, including only the 21 articles developing a prognostic score (risk prediction) or binary classification (classification for preventive intervention). For risk prediction models, when data were available, the calculation SRD = (2 ×— AUC) – 1 was included; effect sizes equivalent to Cohen d "small," "medium," and "large" are SRDs of 11%, 28%, and 43%, respectively. For binary classification models, we included 2 indicators: PPV + NPV – 1 and sensitivity + specificity – 1. Due to dichotomization, prediction indicators will almost always be higher than classification indicators. Select studies are described below. Otherwise informative articles are summarized in Table 2.
With regard to models developed for the prediction of major depressive disorder, we identified 12 relevant reports.31-35,52-58 Among them, 5 focused on samples of patients with medical disorders (specifically, rheumatoid arthritis,52 traumatic brain injury,31 intracerebral hemorrhage,34 prior history of depression,56 and stroke35), only 2 involved individuals in the general population,33,58 and 5 focused on specific demographic groups (eg, adolescents,32 new mothers,53 older adults,55,57 spouses of women with breast cancer54). Regarding the former type of study, the best-performing predictive model was developed among 382 patients recently hospitalized for stroke; it demonstrated PPV, NPV, AUC, and SRD of 94%, 33%, 78%, and 56%, respectively.35 Regarding the other studies pertaining to depression, the most informative study32 described a model developed using data from 4,791 adolescents, which demonstrated PPV, NPV, AUC, and SRD of 64%, 89%, 80%, and 60%, respectively.
In terms of models for the prediction of bipolar disorder, we found only 1 relevant report.36 It was developed using data from 268 Chinese patients with a major depressive episode and demonstrated PPV, NPV, AUC, and SRD of 87%, 67%, 85%, and 70%, respectively, with significant predictors (P < .001) including age at first onset, maximum duration of depressive episodes, somatalgia, hypersomnia, diurnal variation of mood, and irritability.36
We identified 2 relevant reports on the development of prediction models for generalized anxiety disorder.37,59 The best-performing predictive model was derived using data from 271 participants (60 with lupus, 50 with gout, 100 with rheumatoid arthritis, and 61 healthy controls) and demonstrated PPV, NPV, AUC, and SRD of 81%, 85%, 90%, and 80%, respectively.37
Our search revealed 4 reports38-40,60 on risk prediction models for PTSD. The largest-sample study included 25,478 people, ages 16 years and older, affected by the 1998 floods in Hunan, China, and demonstrated PPV, NPV, AUC, and SRD of 23%, 98%, 85%, and 71%, respectively.38 The study conducted by Boscarino et al39 among 2,368 New York City residents after the September 11, 2001, World Trade Center attacks demonstrated PPV, NPV, AUC, and SRD of 34%, 99%, 94%, and 89%, respectively. Russo et al40 conducted a study analyzing electronic medical record data of 878 hospitalized injury survivors, demonstrating PPV, NPV, AUC, and SRD of 58%, 78%, 72%, and 44%, respectively.
With regard to models developed for the prediction of schizophrenia and other primary psychotic disorders, we identified 24 relevant reports,41-51,61-73 primarily focused on samples of "at-risk," "ultra-high risk," or presumably "prodromal" young people. We will briefly review only 2 of the studies48,51 that appear to be most impactful in this rapidly evolving field. Perhaps the best-performing predictive model related to 179 subjects and demonstrated PPV, NPV, AUC, and SRD of 83%, 87%, 81%, and 62%, respectively.48 Significant predictors (P < .001) included psychopathology and life functioning. Another study51 included 58 clinical high-risk subjects and demonstrated PPV, NPV, AUC, and SRD of 70%, 94%, 91%, and 82%, respectively.
For all models in the systematic review, assumption of nonrandomness was never violated based on our calculation. In addition, AUC and SRD values were all high, indicating that the risk prediction models reviewed have acceptable levels of discrimination. Several other studies74-77 were identified pertaining to the prediction of subdiagnostic symptomatology (eg, depressive symptoms following stroke, depressive symptoms in children), which are not reviewed herein.
Although the field of risk prediction in mental health lags behind other areas of medicine (eg, breast cancer, cardiovascular disease), a number of promising studies have been conducted to begin to ascertain the operative combinations of risk factors for a number of psychiatric disorders. These models must be successfully replicated and validated in multiple samples external to the one used for the model development phase. This often takes many years to achieve. The use of risk prediction models must be thoroughly evidence based, with research demonstrating that the model is reliable and applicable to the intended populations of individuals.78
Concerning major depressive disorder, bipolar disorder, and generalized anxiety disorder, our review found that there is not a well-established group of predictive variables. Future investigations should try to increase prediction accuracy using sociodemographic, environmental, neuropsychological, and biologic variables in different samples of at-risk individuals.
Among the psychiatric disorders we examined, the greatest progress appears to be in the area of psychotic disorders. This is probably because several clinical prodromal states have been defined (eg, attenuated psychotic symptoms syndrome, brief and intermittent psychotic symptoms) and there are now specific research diagnostic criteria for these prodromal syndromes. Efforts to establish a prevention approach to schizophrenia have focused on developing and validating criteria for identifying individuals at risk for imminent onset of psychosis (ie, clinical high-risk or prodromal adolescents and young adults) and following them over time.63 The identification and refinement of prodromal states (and prodrome research consortia) for other psychiatric conditions (eg, generalized anxiety disorder) would bring greater attention to prediction and prevention of such psychiatric conditions. It is likely that the method with greatest success will employ, depending on the proportion of the outcome in the population (see Table 2), a 2-step approach: one step aimed at the identification of individuals who appear to be prodromal or at elevated risk (who could be offered "selective preventive interventions") and another step that seeks to further stratify risk so that "indicated preventive interventions" can be given to those in the highest risk stratum in an even more targeted and intensive fashion.
Validation is extremely important in developing risk prediction models. To be useful for clinicians, a prediction model must provide accurate estimates of the risks, and the use of those estimates should improve management and therapeutic decision making and, consequently, (relevant) individuals’ outcomes and cost-effectiveness of care. Validation studies are crucial because the performance of most developed and internally validated prediction models, when applied to new individuals, is poorer than the performance seen in the sample in which it was developed.3 The lack of internal and external validation of the models given in Table 1 indicates that the science of risk prediction models in psychiatry is nascent.
As noted by Paulus,79 psychiatry needs to pragmatically address prediction as an alternative focus for testing hypotheses about clinically relevant issues in the field. One possible obstacle to developing risk prediction models in psychiatry is the prominent lack of availability of biological markers of disease. Another obstacle may be the misperception among clinicians and journal reviewers that a particular discrimination value (eg, an AUC threshold of 0.80) is required before clinical adoption. In most prediction algorithms, including those pertaining to the FRS, the AUC often ranges from 0.75 to 0.80.80
However, risk prediction is only as helpful as the preventive interventions available. That is, risk prediction models are useful only if those identified as high risk can be provided an effective preventive intervention. For example, in the field of cardiovascular disease, the FRS can be easily used by primary care physicians to help in deciding on the best approach to risk reduction (eg, recommending and facilitating lifestyle modifications such as smoking cessation and blood pressure and lipid monitoring and control), thereby ultimately reducing morbidity and mortality. Despite the availability of several validated risk prediction algorithms, their use has lagged in primary care.81 If valid predictive models are developed, then just as much work needs to be done on figuring out how to make them easily applicable and useful for clinicians. In fact, the final step toward implementation of a validated prediction model is the quantification of its impact in clinical care. Although this final step is important to improve health care, literature shows that impact studies are even less frequently performed than validation studies.82
As research on risk prediction models advances in our field, so must the discovery of preventive interventions. In the area of psychosis, for example, in addition to tertiary prevention83—from pharmacotherapy to prevent relapse, to psychosocial and family-based strategies,84 which prevent some of the worst consequences of the illness, such as suicide85—true primary preventive strategies must be developed. One such potential preventive intervention (an "indicated preventive intervention" for those deemed to be at ultra-high risk) that is being studied for preventing the transition from a high-risk state to frank psychosis is the use of omega-3 fatty acids that are thought to be neuroprotective.86,87 All such preventive interventions will ultimately need to be studied in randomized controlled trials, with replications being performed in independent samples. The 2 areas of research on risk prediction and preventive intervention development must work hand in hand to make the prevention of mental disorders a realistic goal.
Several methodological limitations of our systematic, qualitative review must be acknowledged. First, to make the review feasible, we chose to focus on risk prediction models for only 5 types of psychiatric illnesses, excluding other behavioral disorders (eg, Alzheimer’s disease, obsessive-compulsive disorder, substance use disorders). Second, we excluded articles published in languages other than English. Third, given the relative scarcity of research on this topic to date, and the variability across studies, we were not able to conduct a quantitative systematic review or meta-analysis. Meta-analytic results would be useful to provide important information regarding common predictors and the predictive power of existing models, but these are infeasible at present given the very limited state of research in this neglected area of clinical psychiatry.
In summary, risk prediction models can be feasibly developed for diverse psychiatric disorders just as they have been for chronic medical conditions. Measures of discrimination and calibration of risk prediction models reported to date have been reasonable, and our calculations revealed that the assumption of nonrandomness was never violated among the models reported in Table 1 and that SRD values indicate that the models reviewed have acceptable levels of discrimination. The initial identification of a potentially accurate model must be followed by external validation and further refinements, which has not yet been done for these models. Validation research is needed. Ultimately, for such models to be useful in clinical settings, the predictor variables must be readily available and easy to assess, and those individuals deemed to be at high risk must be offered preventive interventions that are effective and present a low risk of adverse events. As such, additional research on effective selective and indicated preventive interventions is necessary. The initial steps toward risk prediction and risk stratification have been taken, especially for psychotic disorders. Now our field must advance further so that effective prediction and prevention will be feasible.
Submitted: March 30, 2015; accepted March 15, 2016.
Online first: June 21, 2016.
Potential conflicts of interest: None reported.
Funding/support: None reported.
1. Meads C, Ahmed I, Riley RD. A systematic review of breast cancer incidence risk prediction models with meta-analysis of their performance. Breast Cancer Res Treat. 2012;132(2):365-377. PubMed doi:10.1007/s10549-011-1818-2
2. Ahmed I, Debray TP, Moons KG, et al. Developing and validating risk prediction models in an individual participant data meta-analysis. BMC Med Res Methodol. 2014;14(1):3. PubMed doi:10.1186/1471-2288-14-3
3. Moons KG, Kengne AP, Woodward M, et al. Risk prediction models, I: development, internal validation, and assessing the incremental value of a new (bio)marker. Heart. 2012;98(9):683-690. PubMed doi:10.1136/heartjnl-2011-301246
4. Corbelli J, Borrero S, Bonnema R, et al. Use of the Gail model and breast cancer preventive therapy among three primary care specialties. J Womens Health (Larchmt). 2014;23(9):746-752. PubMed doi:10.1089/jwh.2014.4742
6. Min JW, Chang MC, Lee HK, et al; Korean Breast Cancer Society. Validation of risk assessment models for predicting the incidence of breast cancer in Korean women. J Breast Cancer. 2014;17(3):226-235. PubMed doi:10.4048/jbc.2014.17.3.226
7. Greenland P, LaBree L, Azen SP, et al. Coronary artery calcium score combined with Framingham score for risk prediction in asymptomatic individuals. JAMA. 2004;291(2):210-215. PubMed doi:10.1001/jama.291.2.210
9. Hu G, Root M, Duncan AW. Adding multiple risk factors improves Framingham coronary heart disease risk scores. Vasc Health Risk Manag. 2014;10:557-562. PubMed
10. Mahmood SS, Levy D, Vasan RS, et al. The Framingham Heart Study and the epidemiology of cardiovascular disease: a historical perspective. Lancet. 2014;383(9921):999-1008. PubMed doi:10.1016/S0140-6736(13)61752-3
13. Orr×¹ G, Pettersson-Yeo W, Marquand AF, et al. Using Support Vector Machine to identify imaging biomarkers of neurological and psychiatric disease: a critical review. Neurosci Biobehav Rev. 2012;36(4):1140-1152. PubMed doi:10.1016/j.neubiorev.2012.01.004
16. Bauer E, Kohavi R. An empirical comparison of voting classification algorithms: bagging, boosting, and variants. Mach Learn. 1999;36(1-2):105-139. doi:10.1023/A:1007515423169
17. Friedman J, Hastie T, Tibshirani R. Additive logistic regression: a statistical view of boosting. Ann Stat. 2000;28(2):337-407. doi:10.1214/aos/1016218223
18. Steyerberg EW, Vickers AJ, Cook NR, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010;21(1):128-138. PubMed doi:10.1097/EDE.0b013e3181c30fb2
26. Brietzke E, Mansur RB, Soczynska JK, et al. Towards a multifactorial approach for prediction of bipolar disorder in at risk populations. J Affect Disord. 2012;140(1):82-91. PubMed doi:10.1016/j.jad.2012.02.016
28. Huang P, Tan HZ, Zhou LB, et al. The application of support vector machine for prediction of posttraumatic stress disorder on adults in flood district [in Chinese]. Zhonghua Liu Xing Bing Xue Za Zhi. 2009;30(1):78-81. PubMed
29. Han SS, Kim YH. Factors predicting depression in hemodialysis patients [in Korean]. Taehan Kanho Hakhoe Chi. 2005;35(7):1353-1361. PubMed
30. Meyer C, Dittrich U, Küster S, et al. Psychoreactive disorders after motor vehicle accidents: is it possible to predict the development of psychoreactive disorders after motor vehicle accidents [in German]? Unfallchirurg. 2005;108(12):1065-1071. PubMed doi:10.1007/s00113-005-0994-0
32. Van Voorhees BW, Paunesku D, Gollan J, et al. Predicting future risk of depressive episode in adolescents: the Chicago Adolescent Depression Risk Assessment (CADRA). Ann Fam Med. 2008;6(6):503-511. PubMed doi:10.1370/afm.887
33. King M, Walker C, Levy G, et al. Development and validation of an international risk prediction algorithm for episodes of major depression in general practice attendees: the PredictD study. Arch Gen Psychiatry. 2008;65(12):1368-1376. PubMed doi:10.1001/archpsyc.65.12.1368
35. de Man-van Ginkel JM, Hafsteinsdóttir TB, Lindeman E, et al. In-hospital risk prediction for post-stroke depression: development and validation of the Post-stroke Depression Prediction Scale. Stroke. 2013;44(9):2441-2445. PubMed doi:10.1161/STROKEAHA.111.000304
36. Gan Z, Diao F, Wei Q, et al. A predictive model for diagnosing bipolar disorder based on the clinical characteristics of major depressive episodes in Chinese population. J Affect Disord. 2011;134(1-3):119-125. PubMed doi:10.1016/j.jad.2011.05.054
37. Mak A, Tang CS, Chan MF, et al. Damage accrual, cumulative glucocorticoid dose and depression predict anxiety in patients with systemic lupus erythematosus. Clin Rheumatol. 2011;30(6):795-803. PubMed doi:10.1007/s10067-010-1651-8
39. Boscarino JA, Kirchner HL, Hoffman SN, et al. A brief screening tool for assessing psychological trauma in clinical practice: development and validation of the New York PTSD Risk Score. Gen Hosp Psychiatry. 2011;33(5):489-500. PubMed doi:10.1016/j.genhosppsych.2011.06.001
40. Russo J, Katon W, Zatzick D. The development of a population-based automated screening procedure for PTSD in acutely injured hospitalized trauma survivors. Gen Hosp Psychiatry. 2013;35(5):485-491. PubMed doi:10.1016/j.genhosppsych.2013.04.016
43. Yung AR, Phillips LJ, Yuen HP, et al. Risk factors for psychosis in an ultra high-risk group: psychopathology and clinical features. Schizophr Res. 2004;67(2-3):131-142. PubMed doi:10.1016/S0920-9964(03)00192-0
46. Eack SM, Prasad KM, Montrose DM, et al. An integrated psychobiological predictive model of emergent psychopathology among young relatives at risk for schizophrenia. Prog Neuropsychopharmacol Biol Psychiatry. 2008;32(8):1873-1878. PubMed doi:10.1016/j.pnpbp.2008.08.024
47. Koutsouleris N, Meisenzahl EM, Davatzikos C, et al. Use of neuroanatomical pattern classification to identify subjects in at-risk mental states of psychosis and predict disease transition. Arch Gen Psychiatry. 2009;66(7):700-712. PubMed doi:10.1001/archgenpsychiatry.2009.62
48. Ruhrmann S, Schultze-Lutter F, Salokangas RK, et al. Prediction of psychosis in adolescents and young adults at high risk: results from the prospective European prediction of psychosis study. Arch Gen Psychiatry. 2010;67(3):241-251. PubMed doi:10.1001/archgenpsychiatry.2009.206
49. Koutsouleris N, Borgwardt S, Meisenzahl EM, et al. Disease prediction in the at-risk mental state for psychosis using neuroanatomical biomarkers: results from the FePsy study. Schizophr Bull. 2012;38(6):1234-1246. PubMed doi:10.1093/schbul/sbr145
50. Koutsouleris N, Davatzikos C, Bottlender R, et al. Early recognition and disease prediction in the at-risk mental states for psychosis using neurocognitive pattern classification. Schizophr Bull. 2012;38(6):1200-1215. PubMed doi:10.1093/schbul/sbr037
51. Nieman DH, Ruhrmann S, Dragt S, et al. Psychosis prediction: stratification of risk estimation with information-processing and premorbid functioning variables. Schizophr Bull. 2014;40(6):1482-1490. PubMed doi:10.1093/schbul/sbt145
52. Covic T, Adamson B, Spencer D, et al. A biopsychosocial model of pain and depression in rheumatoid arthritis: a 12-month longitudinal study. Rheumatology (Oxford). 2003;42(11):1287-1294. PubMed doi:10.1093/rheumatology/keg369
53. Dennis CL, Janssen PA, Singer J. Identifying women at-risk for postpartum depression in the immediate postpartum period. Acta Psychiatr Scand. 2004;110(5):338-346. PubMed doi:10.1111/j.1600-0447.2004.00337.x
55. Shin KR, Jung D, Jo I, et al. Depression among community-dwelling older adults in Korea: a prediction model of depression. Arch Psychiatr Nurs. 2009;23(1):50-57. PubMed doi:10.1016/j.apnu.2008.03.001
56. ten Doesschate MC, Bockting CL, Koeter MW, et al; DELTA Study Group. Prediction of recurrence in recurrent depression: a 5.5-year prospective study. J Clin Psychiatry. 2010;71(8):984-991. PubMed doi:10.4088/JCP.08m04858blu
57. Okamoto K, Harasawa Y. Prediction of symptomatic depression by discriminant analysis in Japanese community-dwelling elderly. Arch Gerontol Geriatr. 2011;52(2):177-180. PubMed doi:10.1016/j.archger.2010.03.012
58. Bellón J×, de Dios Luna J, King M, et al. Predicting the onset of major depression in primary care: international validation of a risk prediction algorithm from Spain. Psychol Med. 2011;41(10):2075-2088. PubMed doi:10.1017/S0033291711000468
59. King M, Bottomley C, Bellón-Saame×±o JA, et al. An international risk prediction algorithm for the onset of generalized anxiety and panic syndromes in general practice attendees: predictA. Psychol Med. 2011;41(8):1625-1639. PubMed doi:10.1017/S0033291710002400
60. Roberge MA, Dupuis G, Marchand A. Post-traumatic stress disorder following myocardial infarction: prevalence and risk factors. Can J Cardiol. 2010;26(5):e170-e175. PubMed doi:10.1016/S0828-282X(10)70386-X
61. Davidson M, Reichenberg A, Rabinowitz J, et al. Behavioral and intellectual markers for schizophrenia in apparently healthy male adolescents. Am J Psychiatry. 1999;156(9):1328-1335. PubMed
62. Amminger GP, Leicester S, Yung AR, et al. Early-onset of symptoms predicts conversion to non-affective psychosis in ultra-high risk individuals. Schizophr Res. 2006;84(1):67-76. PubMed doi:10.1016/j.schres.2006.02.018
63. Cannon TD, Cadenhead K, Cornblatt B, et al. Prediction of psychosis in youth at high clinical risk: a multisite longitudinal study in North America. Arch Gen Psychiatry. 2008;65(1):28-37. PubMed doi:10.1001/archgenpsychiatry.2007.3
64. Riecher-Rössler A, Pflueger MO, Aston J, et al. Efficacy of using cognitive status in predicting psychosis: a 7-year follow-up. Biol Psychiatry. 2009;66(11):1023-1030. PubMed doi:10.1016/j.biopsych.2009.07.020
65. Seidman LJ, Giuliano AJ, Meyer EC, et al; North American Prodrome Longitudinal Study (NAPLS) Group. Neuropsychology of the prodrome to psychosis in the NAPLS consortium: relationship to family history and conversion to psychosis. Arch Gen Psychiatry. 2010;67(6):578-588. PubMed doi:10.1001/archgenpsychiatry.2010.66
66. Walker EF, Brennan PA, Esterberg M, et al. Longitudinal changes in cortisol secretion and conversion to psychosis in at-risk youth. J Abnorm Psychol. 2010;119(2):401-408. PubMed doi:10.1037/a0018399
67. Dragt S, Nieman DH, Veltman D, et al. Environmental factors and social adjustment as predictors of a first psychosis in subjects at ultra high risk. Schizophr Res. 2011;125(1):69-76. PubMed doi:10.1016/j.schres.2010.09.007
68. Mechelli A, Riecher-Rössler A, Meisenzahl EM, et al. Neuroanatomical abnormalities that predate the onset of psychosis: a multicenter study. Arch Gen Psychiatry. 2011;68(5):489-495. PubMed doi:10.1001/archgenpsychiatry.2011.42
69. Ayalew M, Le-Niculescu H, Levey DF, et al. Convergent functional genomics of schizophrenia: from comprehensive understanding to genetic risk prediction. Mol Psychiatry. 2012;17(9):887-905. PubMed doi:10.1038/mp.2012.37
70. Lin YT, Liu CM, Chiu MJ, et al. Differentiation of schizophrenia patients from healthy subjects by mismatch negativity and neuropsychological tests. PLoS ONE. 2012;7(4):e34454. PubMed doi:10.1371/journal.pone.0034454
71. Bicikova M, Hill M, Ripova D, et al. Determination of steroid metabolome as a possible tool for laboratory diagnosis of schizophrenia. J Steroid Biochem Mol Biol. 2013;133:77-83. PubMed doi:10.1016/j.jsbmb.2012.08.009
72. Schultze-Lutter F, Klosterkötter J, Ruhrmann S. Improving the clinical prediction of psychosis by combining ultra-high risk criteria and cognitive basic symptoms. Schizophr Res. 2014;154(1-3):100-106. PubMed doi:10.1016/j.schres.2014.02.010
74. Hackett ML, Anderson CS; Auckland Regional Community Stroke (ARCOS) Study Group. Frequency, management, and predictors of abnormal mood after stroke: the Auckland Regional Community Stroke (ARCOS) study, 2002 to 2003. Stroke. 2006;37(8):2123-2128. PubMed doi:10.1161/01.STR.0000231387.58943.1f
75. Bureau JF, Easterbrooks MA, Lyons-Ruth K. Maternal depressive symptoms in infancy: unique contribution to children’s depressive symptoms in childhood and adolescence? Dev Psychopathol. 2009;21(2):519-537. PubMed doi:10.1017/S0954579409000285
77. Maarsingh OR, Dros J, van der Windt DA, et al. Diagnostic indicators of anxiety and depression in older dizzy patients in primary care. J Geriatr Psychiatry Neurol. 2011;24(2):98-107. PubMed doi:10.1177/0891988711405332
78. Steyerberg EW, Moons KG, van der Windt DA, et al; PROGRESS Group. Prognosis Research Strategy (PROGRESS) 3: prognostic model research. PLoS Med. 2013;10(2):e1001381. PubMed doi:10.1371/journal.pmed.1001381
84. Falloon IR, Boyd JL, McGill CW, et al. Family management in the prevention of exacerbations of schizophrenia: a controlled study. N Engl J Med. 1982;306(24):1437-1440. PubMed doi:10.1056/NEJM198206173062401
85. Meltzer HY, Alphs L, Green AI, et al; International Suicide Prevention Trial Study Group. Clozapine treatment for suicidality in schizophrenia: International Suicide Prevention Trial (InterSePT). Arch Gen Psychiatry. 2003;60(1):82-91. PubMed doi:10.1001/archpsyc.60.1.82
86. Amminger GP, Sch×¤fer MR, Papageorgiou K, et al. Long-chain omega-3 fatty acids for indicated prevention of psychotic disorders: a randomized, placebo-controlled trial. Arch Gen Psychiatry. 2010;67(2):146-154. PubMed doi:10.1001/archgenpsychiatry.2009.192
87. Amminger GP, McGorry PD. Update on Ï‰-3 polyunsaturated fatty acids in early-stage psychotic disorders. Neuropsychopharmacology. 2012;37(1):309-310. doi:10.1038/npp.2011.187
Buy this Article as a PDF