Once a week.
Subscribe to our weekly newsletter.
The evidence for evidence-based therapy is not as clear as we thought
Scientists often find that they cannot replicate prior findings.
Over the past decade, many scholars have questioned the credibility of research across a variety of scientific fields.
Some of these concerns arise from cases of outright fraud or other misconduct. More troubling are difficulties in replicating previous research findings. Replication is cast as a cornerstone of science: we can trust the results originating in one lab only if other labs can follow similar procedures and get similar results. But in many areas of research – including psychology – scientists have found that too often they cannot replicate prior findings.
As psychologists specialising in clinical work (Alexander Williams) and methodology (John Sakaluk), we wondered what these concerns mean for psychotherapy. Over the past 50 years, therapy researchers have increasingly embraced the evidence-based practice movement. Just as medicines are pitted against placebos in research studies, psychologists have used randomised clinical trials to test whether certain therapies (eg, 'exposure therapy', or systematically confronting what one fears) benefit people with certain mental-health conditions (eg, a phobia of spiders). The treatment-for-diagnosis combinations that have amassed evidence from these trials are known as empirically supported treatments (ESTs).
We wondered, though: is the credibility of the evidence for ESTs as strong as that designation suggests? Or does the evidence-base for ESTs suffer from the same problems as published research in other areas of science? This is what we (with our coauthors, the US psychologists Robyn Kilshaw and Kathleen T Rhyner) explored in our study published recently in the Journal of Abnormal Psychology.
The Society of Clinical Psychology – or Division 12 of the American Psychological Association – has done the arduous work since the 1990s of establishing a list of more than 70 ESTs. They have continued to update the ESTs listed, and the evidence cited for them, to the present day. We conducted a 'meta-scientific review' of these ESTs. Across a variety of statistical metrics, we assessed the credibility of the evidence cited by the Society for every EST on their list. We examined measures related to statistical power, which indicates plausibility of the reported data given the sample sizes of the experiments. We computed Bayesian indices of evidence that shows how probable the results were, assuming the therapies actually helped those receiving them. We even looked at rates of misreported statistics – if a study reports, say, '2 + 2 = 5', we know that there must be a problem with at least some of the numbers. All told, we analysed more than 450 research articles. What we found is a study in contrasts.
Around 20 per cent of ESTs performed well across a majority of our metrics (eg, problem-solving therapy for depression, interpersonal psychotherapy for bulimia nervosa, the aforementioned exposure therapy for specific phobias). This means not only that the therapies have been subjected to clinical trials, but that the evidence produced from these clinical trials seems credible and supports the claim that the EST will help people. We also found a 'murky middle': 30 per cent of ESTs had mixed results across metrics, performing neither consistently well nor poorly (eg, cognitive therapy for depression, interpersonal psychotherapy for binge-eating disorder).
That leaves 50 per cent of ESTs with subpar outcomes across most of our metrics (eg, eye-movement desensitisation and reprocessing for PTSD, interpersonal psychotherapy for depression). In other words, although these ESTs seemed to work based on the claims of the clinical trials cited by the Society of Clinical Psychology, we found the evidence from these trials lacked statistical credibility. For these ESTs, the relevant research results are sufficiently ambiguous that we cannot be sure that they really do work better than other forms of therapy.
There is a large, dense body of literature showing that psychotherapy usually helps those who seek it out. Our results don't challenge that conclusion. What does it mean, though, if the evidence behind the therapies thought to be best supported by research is not as strong as one would hope?
One conclusion we draw is that we might be in need of what we're calling 'psychological reversal'. The term, a version of what the US medical scholars Vinay Prasad and Adam Cifu called medical reversal, argues for desisting from the use of psychological practices if they are found to be ineffective, inadvertently harmful or more expensive to employ than equally effective alternatives. If some ESTs lack credible evidence that they are superior to simpler, less costly and time-consuming forms of therapy, shifting resources towards the latter group of treatments will benefit therapy clients and all those bearing the costs of mental-health care.
The other conclusion is a lesson in humility for those who provide therapy (one of the authors of this article among them). For close to a century, psychologists have debated the 'dodo bird hypothesis'. Deriving its name from the proclamation of the Dodo Bird in Alice in Wonderland ('Everybody has won and all must have prizes!'), the dodo bird hypothesis suggests that different forms of psychotherapy perform equally well, and that this is because of the common factors of all therapies (eg, they all provide clients with a rationale for the therapy). The existence of ESTs seems to refute the hypothesis, demonstrating that some therapies do work better than others for certain mental-health conditions. We put forward a different possibility: the 'do not know' bird hypothesis. Given the problems with credibility we found across many clinical trials, we contend that we currently do not know in many cases if some therapies perform better than others. Of course, this also means we do not know if the majority of therapies are equally effective, and, if such equality exists, we do not know if it owes to common factors. When it comes to comparing psychotherapies, therapists could do worse than to channel every philosophy undergrad: when someone purports one therapy works better than another, wonder aloud: 'How do we know?'
Psychotherapy could be on the verge of a renaissance. Research on mental-illness treatment can benefit greatly from the lessons psychology has learned about credibility. For example, investigators can ensure that their studies have sufficient power; that is, enough participants in a clinical trial to reliably detect if a psychotherapy works. They can also practise open science by making their datasets publicly available so that other researchers can verify that a trial's statistics are reported accurately; and/or preregister their therapy trials, specifying in advance their methods and hypotheses, which makes the research process transparent and helps prevent the burying of negative findings.
Ethical therapists can continue to engage in practice that is evidence-based, not eminence-based, rooting their therapies in scientific evidence rather than their own conjecture or that of senior colleagues. They can also continue the routine outcome measurement many already employ: solicit therapy clients' feedback early and often, be open to surprise about what's working and what's not, and adjust accordingly. Clients can ask their therapists upfront if they will offer the opportunity for such mutual assessment of their progress.
Therapy helps the vast majority of those who receive it. Happily – if the discipline embraces reform in research, and cultivates a humble, flexible approach to therapy – it could help even more.
This article was originally published at Aeon and has been republished under Creative Commons. Read the original article.
- Psychotherapy is not harmless: on the side effects of CBT - Big Think ›
- Freud is renowned, but his ideas are ill-substantiated - Big Think ›
These alien-like creatures are virtually invisible in the deep sea.
- A team of marine biologists used nets to catch 16 species of deep-sea fish that have evolved the ability to be virtually invisible to prey and predators.
- "Ultra-black" skin seems to be an evolutionary adaptation that helps fish camouflage themselves in the deep sea, which is illuminated by bioluminescent organisms.
- There are likely more, and potentially much darker, ultra-black fish lurking deep in the ocean.
The Pacific blackdragon
Credit: Karen Osborn/Smithsonian<p>When researchers first saw the deep-sea species, it wasn't immediately obvious that their skin was ultra-black. Then, marine biologist Karen Osborn, a co-author on the new paper, noticed something strange about the photos she took of the fish.</p><p style="margin-left: 20px;">"I had tried to take pictures of deep-sea fish before and got nothing but these really horrible pictures, where you can't see any detail," Osborn told <em><a href="https://www.wired.com/story/meet-the-ultra-black-vantafish/" target="_blank">Wired</a></em>. "How is it that I can shine two strobe lights at them and all that light just disappears?"</p><p>After examining samples of fish skin under the microscope, the researchers discovered that the fish skin contains a layer of organelles called melanosomes, which contain melanin, the same pigment that gives color to human skin and hair. This layer of melanosomes absorbs most of the light that hits them.</p>
A crested bigscale
Credit: Karen Osborn/Smithsonian<p style="margin-left: 20px;">"But what isn't absorbed side-scatters into the layer, and it's absorbed by the neighboring pigments that are all packed right up close to it," Osborn told <em>Wired</em>. "And so what they've done is create this super-efficient, very-little-material system where they can basically build a light trap with just the pigment particles and nothing else."</p><p>The result? Strange and terrifying deep-sea species, like the crested bigscale, fangtooth, and Pacific blackdragon, all of which appear in the deep sea as barely more than faint silhouettes.</p>
David Csepp, NMFS/AKFSC/ABL<p>But interestingly, this unique disappearing trick wasn't passed on to these species by a common ancestor. Rather, they each developed it independently. As such, the different species use their ultra-blackness for different purposes. For example, the threadfin dragonfish only has ultra-black skin during its adolescent years, when it's rather defenseless, as <em>Wired</em> <a href="https://www.wired.com/story/meet-the-ultra-black-vantafish/" target="_blank">notes</a>.</p><p>Other fish—like the <a href="http://onebugaday.blogspot.com/2016/06/a-new-anglerfish-oneirodes-amaokai.html" target="_blank">oneirodes species</a>, which use bioluminescent lures to bait prey—probably evolved ultra-black skin to avoid reflecting the light their own bodies produce. Meanwhile, species like <em>C. acclinidens</em> only have ultra-black skin around their gut, possibly to hide light of bioluminescent fish they've eaten.</p><p>Given that these newly described species are just ones that this team found off the coast of California, there are likely many more, and possibly much darker, ultra-black fish swimming in the deep ocean. </p>
Using machine-learning technology, the genealogy company My Heritage enables users to animate static images of their relatives.
- Deep Nostalgia uses machine learning to animate static images.
- The AI can animate images by "looking" at a single facial image, and the animations include movements such as blinking, smiling and head tilting.
- As deepfake technology becomes increasingly sophisticated, some are concerned about how bad actors might abuse the technology to manipulate the pubic.
My Heritage/Deep Nostalgia<p>But that's not to say the animations are perfect. As with most deep-fake technology, there's still an uncanny air to the images, with some of the facial movements appearing slightly unnatural. What's more, Deep Nostalgia is only able to create deepfakes of one person's face from the neck up, so you couldn't use it to animate group photos, or photos of people doing any sort of physical activity.</p>
My Heritage/Deep Nostalgia<p>But for a free deep-fake service, Deep Nostalgia is pretty impressive, especially considering you can use it to create deepfakes of <em>any </em>face, human or not. </p>
How long should one wait until an idea like string theory, seductive as it may be, is deemed unrealistic?
- How far should we defend an idea in the face of contrarian evidence?
- Who decides when it's time to abandon an idea and deem it wrong?
- Science carries within it its seeds from ancient Greece, including certain prejudices of how reality should or shouldn't be.
Plato used the allegory of the cave to explain that what humans see and experience is not the true reality.
Credit: Gothika via Wikimedia Commons CC 4.0<p>When scientists and mathematicians use the term <em>Platonic worldview</em>, that's what they mean in general: The unbound capacity of reason to unlock the secrets of creation, one by one. Einstein, for one, was a believer, preaching the fundamental reasonableness of nature; no weird unexplainable stuff, like a god that plays dice—his tongue-in-cheek critique of the belief that the unpredictability of the quantum world was truly fundamental to nature and not just a shortcoming of our current understanding. Despite his strong belief in such underlying order, Einstein recognized the imperfection of human knowledge: "What I see of Nature is a magnificent structure that we can comprehend only very imperfectly, and that must fill a thinking person with a feeling of humility." (Quoted by Dukas and Hoffmann in <em>Albert Einstein, The Human Side: Glimpses from His Archives</em> (1979), 39.)</p> <p>Einstein embodies the tension between these two clashing worldviews, a tension that is still very much with us today: On the one hand, the Platonic ideology that the fundamental stuff of reality is logical and understandable to the human mind, and, on the other, the acknowledgment that our reasoning has limitations, that our tools have limitations and thus that to reach some sort of final or complete understanding of the material world is nothing but an impossible, <a href="https://www.amazon.com/dp/B01K2JTGIA?tag=bigthink00-20&linkCode=ogi&th=1&psc=1" target="_blank" rel="noopener noreferrer">semi-religious dream</a>.</p>