The Devil is in the Details: How Selection Bias can Affect Psychedelic Research Outcomes

Psychedelic science has been receiving a lot of attention in the past 10 years, and it's not hard to see why. Researchers are exploring the possibility of using substances like psilocybin, LSD, and DMT to treat a range of mental health conditions, such as depression and addiction (for recent reviews, see Sarris et al., 2022, and Sharma et al., 2023). But there's a catch; while psychedelic medicine has shown some promise, there are some concerns about the validity and reliability of the science being conducted. Many of my blogs over the coming months will delve into these concerns, and why they are so important to acknowledge. This blog, in particular, will cover a sneaky little issue called selection bias, and why it’s an underappreciated caveat of psychedelic science.

‍

As with many seemingly revolutionary advances in science and medicine, things are never as perfect as they seem. Selection bias is a crucial factor to consider when reading about the ostensibly remarkable results of psychedelic trials. In short, selection bias is when the people chosen to participate in a study are not a random group, but instead are chosen in a way that skews the results.

‍

For example, imagine you are trying to find out what sort of music people like. You attend your local venue in which a rock concert is taking place and ask the attendees about their music preferences. Naturally, your results show that people overwhelmingly prefer rock music. Obviously, that is because you have only asked people who attended this rock concert. Can these results be applied to the general population? Certainly not.

‍

There are many similar instances of selection bias in the psychedelic space. One example is a study published by Williams et al. (2021). The study is titled, “People of Color in North America report improvements in racial trauma and mental health symptoms following psychedelic experiences.” This title sounds promising and intriguing. But closer inspection of the methods reveals that one of the inclusion criteria participants had to meet was “…after taking the substance they had an experience they believed contributed to “relief from the challenging effects of ethnic discrimination,” …”.

‍

In other words, only participants who experienced relief from racial trauma after taking a psychedelic were included in the study, resulting in the flawed conclusion that psychedelics can help people relieve racial trauma. It is like a scientist looking at whether a medicine treats a disease and concluding that it does, but only basing this on the 1% of patients who experienced a reduction in symptoms after taking the medicine. What about the 99% of patients who took the medicine but did not experience a reduction in symptoms?

‍

This logic is problematic in psychedelic science (and research in general) because it leads to false conclusions and prevents us from truly understanding the potential of psychedelic medicine. On one hand, the study above arguably has value in that it shows that using psychedelics to relieve racial trauma may be an option for some people, thereby acting as a ‘proof-of-concept’ study. However, it simply was not labelled as such. Such embellished reporting of results may create false hope and misinform the public about the true benefits and limitations of psychedelic medicine.

‍

The example above is particularly extreme. However, selection bias remains a crucial problem throughout all psychedelic research. For example, many studies in the field have been undertaken on psychedelic-experienced individuals who have volunteered to take part. Naturally, the vast majority of these samples will end up consisting of people who have previously responded well to psychedelics, thereby excluding both those who have never taken a psychedelic and also those who have responded negatively to them. When these studies end up having results that portray psychedelics in a good light, one must question whether they actually reflect what would happen in the general population.

‍

There are two primary reasons for these inclusion criteria: safety and scientific rigour. Given the unpredictability of psychedelic experiences, to ensure that participants remain safe, they must be carefully screened. Many will be excluded based on factors such as mental health, drug use, medication, and medical conditions. Secondly, regarding scientific rigour, it is important for studies to be well-controlled and structured to ensure that the results are reliable and accurate. To do this, scientists strive to minimise factors that may change the results by recruiting participants from similar backgrounds. However, this makes the results more reductive, and means they are less applicable to the wider population.

‍

All research in the field of life sciences faces this dilemma; the correct balance must be struck between internal validity (how well the results reflect what the study intends to measure) and external validity (how well the study’s results can be applied to the general population). Ultimately, there is no perfect balance to strike; there is no such thing as the perfect study, particularly when it comes to research involving psychological factors. Such studies will always have flaws. Researchers must carefully consider the inclusion and exclusion criteria for their studies, and ideally recruit a diverse sample of participants.

‍

I do not claim that psychedelic studies should be done any differently at this stage. My takeaway message is simply that selection bias is an important caveat to be aware of when reviewing the encouraging results of psychedelic research, and that it should perhaps be expressed more strongly in the writing of some of these papers. After all, it would be foolish to conclude, after reading these studies, that psychedelics are a miracle cure for all of life’s problems.

‍

By acknowledging the limitations in the research, we can take steps towards ensuring that the results are accurate and applicable to a wider range of people. Ultimately, this will help to advance our understanding of the therapeutic potential of psychedelics and pave the way for future research and treatments that can benefit everyone.

‍