The evidence for evidence-based therapy is not as clear as we thought
Scientists often find that they cannot replicate prior findings.
Over the past decade, many scholars have questioned the credibility of research across a variety of scientific fields.
Some of these concerns arise from cases of outright fraud or other misconduct. More troubling are difficulties in replicating previous research findings. Replication is cast as a cornerstone of science: we can trust the results originating in one lab only if other labs can follow similar procedures and get similar results. But in many areas of research – including psychology – scientists have found that too often they cannot replicate prior findings.
As psychologists specialising in clinical work (Alexander Williams) and methodology (John Sakaluk), we wondered what these concerns mean for psychotherapy. Over the past 50 years, therapy researchers have increasingly embraced the evidence-based practice movement. Just as medicines are pitted against placebos in research studies, psychologists have used randomised clinical trials to test whether certain therapies (eg, 'exposure therapy', or systematically confronting what one fears) benefit people with certain mental-health conditions (eg, a phobia of spiders). The treatment-for-diagnosis combinations that have amassed evidence from these trials are known as empirically supported treatments (ESTs).
We wondered, though: is the credibility of the evidence for ESTs as strong as that designation suggests? Or does the evidence-base for ESTs suffer from the same problems as published research in other areas of science? This is what we (with our coauthors, the US psychologists Robyn Kilshaw and Kathleen T Rhyner) explored in our study published recently in the Journal of Abnormal Psychology.
The Society of Clinical Psychology – or Division 12 of the American Psychological Association – has done the arduous work since the 1990s of establishing a list of more than 70 ESTs. They have continued to update the ESTs listed, and the evidence cited for them, to the present day. We conducted a 'meta-scientific review' of these ESTs. Across a variety of statistical metrics, we assessed the credibility of the evidence cited by the Society for every EST on their list. We examined measures related to statistical power, which indicates plausibility of the reported data given the sample sizes of the experiments. We computed Bayesian indices of evidence that shows how probable the results were, assuming the therapies actually helped those receiving them. We even looked at rates of misreported statistics – if a study reports, say, '2 + 2 = 5', we know that there must be a problem with at least some of the numbers. All told, we analysed more than 450 research articles. What we found is a study in contrasts.
Around 20 per cent of ESTs performed well across a majority of our metrics (eg, problem-solving therapy for depression, interpersonal psychotherapy for bulimia nervosa, the aforementioned exposure therapy for specific phobias). This means not only that the therapies have been subjected to clinical trials, but that the evidence produced from these clinical trials seems credible and supports the claim that the EST will help people. We also found a 'murky middle': 30 per cent of ESTs had mixed results across metrics, performing neither consistently well nor poorly (eg, cognitive therapy for depression, interpersonal psychotherapy for binge-eating disorder).
That leaves 50 per cent of ESTs with subpar outcomes across most of our metrics (eg, eye-movement desensitisation and reprocessing for PTSD, interpersonal psychotherapy for depression). In other words, although these ESTs seemed to work based on the claims of the clinical trials cited by the Society of Clinical Psychology, we found the evidence from these trials lacked statistical credibility. For these ESTs, the relevant research results are sufficiently ambiguous that we cannot be sure that they really do work better than other forms of therapy.
There is a large, dense body of literature showing that psychotherapy usually helps those who seek it out. Our results don't challenge that conclusion. What does it mean, though, if the evidence behind the therapies thought to be best supported by research is not as strong as one would hope?
One conclusion we draw is that we might be in need of what we're calling 'psychological reversal'. The term, a version of what the US medical scholars Vinay Prasad and Adam Cifu called medical reversal, argues for desisting from the use of psychological practices if they are found to be ineffective, inadvertently harmful or more expensive to employ than equally effective alternatives. If some ESTs lack credible evidence that they are superior to simpler, less costly and time-consuming forms of therapy, shifting resources towards the latter group of treatments will benefit therapy clients and all those bearing the costs of mental-health care.
The other conclusion is a lesson in humility for those who provide therapy (one of the authors of this article among them). For close to a century, psychologists have debated the 'dodo bird hypothesis'. Deriving its name from the proclamation of the Dodo Bird in Alice in Wonderland ('Everybody has won and all must have prizes!'), the dodo bird hypothesis suggests that different forms of psychotherapy perform equally well, and that this is because of the common factors of all therapies (eg, they all provide clients with a rationale for the therapy). The existence of ESTs seems to refute the hypothesis, demonstrating that some therapies do work better than others for certain mental-health conditions. We put forward a different possibility: the 'do not know' bird hypothesis. Given the problems with credibility we found across many clinical trials, we contend that we currently do not know in many cases if some therapies perform better than others. Of course, this also means we do not know if the majority of therapies are equally effective, and, if such equality exists, we do not know if it owes to common factors. When it comes to comparing psychotherapies, therapists could do worse than to channel every philosophy undergrad: when someone purports one therapy works better than another, wonder aloud: 'How do we know?'
Psychotherapy could be on the verge of a renaissance. Research on mental-illness treatment can benefit greatly from the lessons psychology has learned about credibility. For example, investigators can ensure that their studies have sufficient power; that is, enough participants in a clinical trial to reliably detect if a psychotherapy works. They can also practise open science by making their datasets publicly available so that other researchers can verify that a trial's statistics are reported accurately; and/or preregister their therapy trials, specifying in advance their methods and hypotheses, which makes the research process transparent and helps prevent the burying of negative findings.
Ethical therapists can continue to engage in practice that is evidence-based, not eminence-based, rooting their therapies in scientific evidence rather than their own conjecture or that of senior colleagues. They can also continue the routine outcome measurement many already employ: solicit therapy clients' feedback early and often, be open to surprise about what's working and what's not, and adjust accordingly. Clients can ask their therapists upfront if they will offer the opportunity for such mutual assessment of their progress.
Therapy helps the vast majority of those who receive it. Happily – if the discipline embraces reform in research, and cultivates a humble, flexible approach to therapy – it could help even more.
This article was originally published at Aeon and has been republished under Creative Commons. Read the original article.
- Psychotherapy is not harmless: on the side effects of CBT - Big Think ›
- Freud is renowned, but his ideas are ill-substantiated - Big Think ›
To create wiser adults, add empathy to the school curriculum.
- Stories are at the heart of learning, writes Cleary Vaughan-Lee, Executive Director for the Global Oneness Project. They have always challenged us to think beyond ourselves, expanding our experience and revealing deep truths.
- Vaughan-Lee explains 6 ways that storytelling can foster empathy and deliver powerful learning experiences.
- Global Oneness Project is a free library of stories—containing short documentaries, photo essays, and essays—that each contain a companion lesson plan and learning activities for students so they can expand their experience of the world.
Philosophers like to present their works as if everything before it was wrong. Sometimes, they even say they have ended the need for more philosophy. So, what happens when somebody realizes they were mistaken?
Sometimes philosophers are wrong and admitting that you could be wrong is a big part of being a real philosopher. While most philosophers make minor adjustments to their arguments to correct for mistakes, others make large shifts in their thinking. Here, we have four philosophers who went back on what they said earlier in often radical ways.
Just before I turned 60, I discovered that sharing my story by drawing could be an effective way to both alleviate my symptoms and combat that stigma.
I've lived much of my life with anxiety and depression, including the negative feelings – shame and self-doubt – that seduced me into believing the stigma around mental illness: that people knew I wasn't good enough; that they would avoid me because I was different or unstable; and that I had to find a way to make them like me.
A joint study by two England universities explores the link between sex and cognitive function with some surprising differences in male and female outcomes in old age.
- A joint study by the universities of Coventry and Oxford in England has linked sexual activity with higher cognitive abilities in older age.
- The results of this study suggest there are significant associations between sexual activity and number sequencing/word recall in men. In women, however, there was a significant association between sexual activity in word recall alone - number sequencing was not impacted.
- The differences in testosterone (the male sex hormone) and oxytocin (a predominantly female hormone) may factor into why the male cognitive level changes much more during sexual activity in older age.
Mathematicians studied 100 billion tweets to help computer algorithms better understand our colloquial digital communication.