Can Classification Tree Analyses Help Improve Decision Making About Treatments for Depression and Anxiety Disorders? A Preliminary Investigation

Can Classification Tree Analyses Help Improve Decision Making About Treatments for Depression and Anxiety Disorders? A Preliminary Investigation

Louisa Rhodes, BA (Hons); Ulrike M. Naumann, MSc; and June S. L. Brown, PhD

Objective: To identify how decisions about treatment are being made in secondary services for anxiety disorders and depression and, specifically, whether it was possible to predict the decisions to refer for evidence-based treatments.

Method: Post hoc classification tree analysis was performed using a sample from an audit on implementation of the National Institute for Health and Clinical Excellence Guidelines for Depression and Anxiety Disorders. The audit was of 5 teams offering secondary care services; they included psychiatrists, psychologists, community psychiatric nurses, social workers, dual-diagnosis workers, and vocational workers. The patient sample included all of those with a primary problem of depression (n = 56) or an anxiety disorder (n = 16) who were offered treatment from February 16 to April 3, 2009. The outcome variable was whether or not evidence-based treatments were offered, and the predictor variables were presenting problem, risk, comorbid problem, social problems, and previous psychiatric history.

Results: Treatment decisions could be more accurately predicted for anxiety disorders (93% correct) than for depression (55%). For anxiety disorders, the presence or absence of social problems was a good predictor for whether evidence-based or non-evidence-based treatments were offered; 44% (4/9) of those with social problems vs 100% (6/6) of those without social problems were offered evidence-based treatments. For depression, patients’ risk rating had the largest impact on treatment decisions, although no one variable could be identified as individually predictive of all treatment decisions.

Conclusions: Treatment decisions were generally consistent for anxiety disorders but more idiosyncratic for depression, making the development of a decision-making model very difficult for depression. The lack of clarity of some terms in the clinical guidelines and the more complex nature of depression could be factors contributing to this difficulty. Further research is needed to understand the complex nature of decision making with depressed patients.

Prim Care Companion CNS Disord 2011;13(5):doi:10.4088/PCC.10m01124

© Copyright 2011 Physicians Postgraduate Press, Inc.

Submitted: December 5, 2010; accepted April 13, 2011.

Published online: September 1, 2011 (doi:10.4088/PCC.10m01124).

Corresponding author: June S. L. Brown, PhD, Department of Psychology, Institute of Psychiatry, De Crespigny Park, London, SE5 8AF (june.brown@kcl.ac.uk).

Decision making is a key task of all mental health professionals. Ridley and Shaw-Ridley state that when making clinical decisions “accurate judgment forms the basis for establishing reasonable goals and selecting appropriate treatment, which in turn are essential in achieving positive outcomes.”1(p400)

A number of research studies emphasize the importance of accurate clinical judgment and attempt to identify the environmental factors that may influence decision making.2 However, very few studies have attempted to identify how patient information is used when making clinical decisions.

In a qualitative study, Martin3 used grounded theory to identify the influences on clinical judgment in mental health nursing and found that decisions were rather idiosyncratic, these being very dependent on the time and situation.

In their study on the delivery of evidence-based treatment for multiple anxiety disorders, Roy-Byrne et al4 demonstrate that evidence-based treatments result in greater improvement in anxiety symptoms compared to “usual care.” Roy-Byrne et al4 highlight the importance of improving mental health care by using evidence-based treatments in real-world practice settings where patient characteristics and clinician skills are quite variable. Similarly for depression, the Texas Medication Algorithm Project showed that the evidence-based clinical interventions were more successful in the treatment of major depression than treatment as usual.5

Mental health services in the United Kingdom are expected to use clinical guidelines when offering treatment. For depression and anxiety disorders, this involves implementing the recommendations set out by the National Institute for Health and Clinical Excellence (NICE),6-9 which were based on evidence-based treatments.

As part of the clinical guidelines, NICE also recommends a series of key factors that should be used to inform clinical judgment when making treatment decisions for depression and anxiety disorders. For depression, the important factors include risk, comorbid mental health problems, complex psychosocial problems, and treatment history. The Guidelines recommend that this information is used to determine the appropriate step in the stepped care model at which the person should be treated and the interventions that may be most suitable, although this may also be influenced by patient choice.

One method of quantitatively assessing the relative importance of different factors involved in decision making is to use recursive partitioning, specifically classification tree analysis. Mann et al10 successfully used this form of analysis to distinguish suicide attempters in major psychiatric disorders and found that current suicidal ideation is the best indicator of a recent suicide attempt in psychiatric patients. Mann et al10 emphasize that a major advantage of this analysis is that it seeks to resemble the clinical decision-making process and may therefore be easy to interpret in practice.

This study was conducted as a post hoc analysis to a project using an audit cycle investigating the implementation of the Guidelines for Depression and Anxiety Disorders. A key finding was that, after several implementation initiatives, the use of stepped care increased significantly, but the use of evidence-based treatments did not, even though there was some increase in the use of psychological treatments at the second audit.

A question that arises is how decisions about treatment are made and, specifically, whether it was possible to predict the decision to refer for evidence-based treatments.

METHOD

The audit project was conducted in the London Borough of Southwark (United Kingdom) with 5 Assessment and Brief Treatment (ABT) Teams who offer secondary care services as part of the Community Mental Health Teams.

The multidisciplinary ABT Teams include psychiatrists, psychologists, community psychiatric nurses, social workers, dual-diagnosis workers, and vocational workers. The teams treat adults of working age (18-65 years) who have a range of severe or complex mental health problems.

For people with anxiety disorders and depression, a range of treatments are offered, including pharmacotherapy, evidence-based psychological therapy (predominantly cognitive-behavioral therapy [CBT]), allocation of a care coordinator, social support, and referral for non-evidence-based psychological therapy (eg, psychodynamic psychotherapy).

The sample for this analysis includes all patients with a primary problem of depression or an anxiety disorder who were offered treatment from the ABT Teams from February 16 to April 3, 2009. Seventy-two patients are included in the sample, 56 with a primary problem of depression and 16 with a primary problem of an anxiety disorder (including obsessive-compulsive disorder, posttraumatic stress disorder, generalized anxiety disorder, and panic disorder).

Measures

Information was taken from electronic patients’ notes. The outcome variable was whether or not evidence-based interventions were offered. Interventions were as follows.

• Pharmacotherapy and psychiatric medical review only (outpatient appointment with psychiatrist for review of medication and assessment of mental state), to be referred to as pharmacotherapy/medical review
• Evidence-based psychological therapy only (CBT, behavioral activation)
• Combined treatments (pharmacotherapy/medical review plus evidence-based psychological therapy)
• Non-evidence-based treatments (allocation of a care coordinator; social support for such problems as housing, finances, benefits, and immigration support; and psychological therapies not recommended by NICE, such as referral for cognitive analytic therapy and long-term psychodynamic therapy)

The input variables were presenting problem, risk, comorbid problem, social problems, and previous psychiatric history.

Analysis

A recursive partitioning method was used to build classification trees to predict the dependent variable using continuous and categorical predictor variables.11 Separate classification trees were generated for depression and for anxiety disorders using SPSS version 15.0 (SPSS Inc; Chicago, Illinois). The classification trees were used to determine the relative importance of the predictors when making treatment decisions and to assess how consistently decisions were made.

When interpreting the classification tree data, the following information was important:

1. The risk estimate, which indicates the risk of incorrectly predicting a treatment category for a patient;
2. The classification table data on percentage accuracy, which gives a percentage for how accurate the model is in predicting the treatment a client will be offered using the input variable information;
3. The order of the predictor variables, which gave an indication of which variables may have had more of an influence over treatment decisions; and
4. The cross-validation risk estimate for the final tree, which is then calculated by averaging the risk for all trees and indicates the reliability of the tree classifications.

To validate both classification trees, cross-validation with 5 sample folds was used. For this process, SPSS creates a series of classification trees, each time excluding 5 cases (a subsample) from the data. A misclassification risk is then generated for each subsample by applying the tree to the excluded subsample and identifying the number of cases that are incorrectly classified.

RESULTS

The classification tree analysis more accurately predicted the treatment decisions for patients with anxiety disorders (93% correct) compared to patients with depression (55% correct).

Anxiety Disorders

The anxiety disorders classification tree shows that the presence or absence of social problems was a good predictor for whether evidence-based or non-evidence-based treatments were offered (Figure 1). Patients with social problems were offered a wider variety of treatments than those without social problems, and they were more likely to be offered non-evidence-based treatments, for example, ABT support or non-evidence-based psychological therapy (56%). However, the anxious patients who had no social problems were all offered some form of evidence-based treatment (50% combined treatment, 33% evidence-based psychological therapy, 17% pharmacotherapy/medical review).

The risk estimate for the anxiety disorders classification tree was 0.07, indicating that the treatment category predicted by the model was incorrect for only 7% of cases. This was supported by the classification table, which indicated that the model correctly classified the treatment category for 93% of patients. The cross-validation estimate for risk of misclassification was low at 0.27.

Depression

The depression classification tree shows that, relative to the other variables, risk status had the largest impact on treatment decisions, although no one variable could be identified as individually predictive of all treatment decisions. Examination of the depression classification tree (Figure 2) shows that depressed patients were offered a wider range of treatment categories, and it is therefore more difficult to identify consistent patterns. Over half of patients who were judged to have moderate/high risk received pharmacotherapy/medical reviews only, and the remaining patients were equally likely to receive any of the other 3 evidence-based and non-evidence-based treatments. However, the low-risk patients tended to receive evidence-based treatments, that is, either pharmacotherapy/medical reviews only or the combined treatment.

The risk estimate for the overall depression classification tree was 0.446, indicating that the treatment category predicted by the model was incorrect for 45% of cases. This was supported by the classification table, which revealed that the model correctly classified the treatment category for only 55% of patients. The cross-validation estimate for risk of misclassification was high at 0.696.

DISCUSSION

The classification tree results show very clear differences between anxiety disorders and depression. For anxiety disorders, the treatment decisions appear to be very consistent, which is reflected in the 93% predictive capability and fairly low cross-validation estimate for risk of misclassification (0.27). For depression, the treatment decisions appear to be much more idiosyncratic, and the low predictive capability of the depression classification tree (55%) and the high cross-validation estimate for risk of misclassification (0.696) support this.

There are a number of possible explanations for these differences. First, the audit results showed that cases of depression are often less clearly stated than anxiety disorders in the notes in terms of severity and diagnoses, which may be reducing the consistency of clinicians’ treatment decisions. Second, the lack of clarity of some diagnostic terms in the Guidelines can also affect treatment decision making. In the United Kingdom, the NICE recommendations for depression9 define complex depression as including “depression that shows an inadequate response to multiple treatments, is complicated by psychotic symptoms, and/or is associated with significant psychiatric comorbidity or psychosocial factors.” These non-specifically defined terms allow more room for clinical judgment with regard to depression, which may result in more idiosyncratic decisions. In contrast, the diagnostic information about the different anxiety disorders is much clearer.

Third, risk was identified as the variable that had the largest impact on treatment decisions for depressed patients, relative to the other variables. When risk issues are present, these will often take greater priority in terms of treatment choices. In some cases, an increase in medication may be the first-line treatment. For others, the complex psychosocial problems of patients may be causing them to be judged high risk, in which case non-evidence-based ABT support may be considered more appropriate. Some patients may later be offered psychological therapies, whereas others may be discharged without being offered any further treatment. Thus, the impact of social problems on depression, for which there are few evidence-based treatments,12 can be seen as adding a layer of complexity to the management of depressed patients judged to be at high risk. The reactive and varied way in handling risk may be contributing to the lack of consistency and lower predictive capability of the depression classification tree. Finally, the treatment choices for anxiety are much more limited, as evidence-based treatments are pharmacotherapy or evidence-based psychological therapies, particularly CBT.

Some limitations need to be mentioned. The sample size for anxiety disorders is considerably smaller than that for depression, which may have overinflated the difference in prediction capability of the classification trees. However, the cross-validation estimate for risk of misclassification of data does support the conclusion that the decision making for anxious patients is more consistent and uniform than for depressed patients.

It is possible that decision making could be influenced by a range of other factors that were not investigated during the audit. It is also possible that the depressed patients generally have more complex problems that require tailor-made treatments. For example, patients may have depression and secondary personality disorder. How evidence-based treatments fit into clinical practice is also of importance. Some clinicians may be more keen to base clinical decisions on clinical judgment than on research findings.

However, the biggest concern with the lack of consistency in treatment decisions for depressed patients is that they may not be getting equal access to the treatments that are recommended by the Clinical Guidelines. It may also be possible that treatments that are not evidence-based at the moment may become evidence-based in the future. In addition, the data for whether evidence-based treatment was offered do not indicate whether the intervention was actually taken up. For example, medication may have been offered and prescribed, but may not have been taken by the patient.

The study demonstrates the relative importance of factors used to make treatment decisions and the difficulty of developing a decision-making model for depression compared to anxiety disorders. Further research is needed to understand the complex nature of decision making affecting the idiosyncratic patterns of treatment offered to patients with depression. We think this analysis has led to a useful understanding of how decisions about evidence-based treatments are made and hope that it will assist clinicians in their thinking about how they make decisions.

Author affiliations: Departments of Psychology (Ms Rhodes and Dr Brown) and Biostatistics and Computing (Ms Naumann), Institute of Psychiatry, King’s College London, London, United Kingdom.

Potential conflicts of interest: None reported.

Funding/support: Dr Brown is supported by the National Institute for Health Research Specialist Biomedical Research Centre for Mental Health at the South London and Maudsley NHS Foundation Trust and Institute of Psychiatry, King’s College London.

REFERENCES

1. Ridley CR, Shaw-Ridley M. Clinical judgement accuracy: from meta-analysis to meta theory. Couns Psychol. 2009;37(3):400-409. doi:10.1177/0011000008330830

2. Tanner CA. Thinking like a nurse: a research-based model of clinical judgment in nursing. J Nurs Educ. 2006;45(6):204-211. PubMed

3. Martin PJ. Influences on clinical judgement in mental health nursing. NT Research. 1999;4(4):273-281. PubMed

4. Roy-Byrne P, Craske MG, Sullivan G, et al. Delivery of evidence-based treatment for multiple anxiety disorders in primary care: a randomized controlled trial. JAMA. 2010;303(19):1921-1928. PubMed doi:10.1001/jama.2010.608

5. Trivedi MH, Rush AJ, Crismon ML, et al. Clinical results for patients with major depressive disorder in the Texas Medication Algorithm Project. Arch Gen Psychiatry. 2004;61(7):669-680. PubMed doi:10.1001/archpsyc.61.7.669

6. National Institute for Health and Clinical Excellence. CG22: Anxiety: Management of anxiety in adults in primary, secondary and community care. http://guidance.nice.org.uk/CG22. Published 2004.

7. National Institute for Health and Clinical Excellence. CG31: Core interventions in the treatment of obsessive compulsive disorder (OCD) and body dysmorphic disorder (BDD). http://guidance.nice.org.uk/CG31. Published 2005.

8. National Institute for Health and Clinical Excellence. CG26: Post-traumatic stress disorder (PTSD): The management of PTSD in adults and children in primary and secondary care. http://guidance.nice.org.uk/CG26. Published 2005.

9. National Institute for Health and Clinical Excellence. CG90: Depression in adults (update). http://guidance.nice.org.uk/CG90. Published 2009.

10. Mann JJ, Ellis SP, Waternaux CM, et al. Classification trees distinguish suicide attempters in major psychiatric disorders: a model of clinical decision making. J Clin Psychiatry. 2008;69(1):23-31. PubMed doi:10.4088/JCP.v69n0104

11. Breiman L, Friedman J, Ohlsen CJ, et al. Classification and Regression Trees. London, UK: Chapman and Hall; 1984.

12. Bonnefoy J, Morgan A, Kelly MP, et al. Constructing the evidence base on the social determinants of health: a guide. World Health Organisation. 2007; http://www.who.int/social_determinants/knowledge_networks/add_documents/mekn_final_guide_112007.pdf.