Some months back, there was a publication that ended up being highly hyped by the press. Controversy and science by press release ensued. Public relations people had field days hitting all the target demographic periodical outlets. There was a convenient lack of PR for a concurrently published paper with results that could not be more negative. Great situation for people who like to get all fired up about things, useless hype in terms of evidence-based discovery efforts.
The publication in question? A follow-up study by Mithoefer et al describing the long term effects of their MDMA therapy for PTSD, based upon which the press releases make the bold* claim that MDMA augmented therapy has an 89% success rate in treating PTSD.
Herein, I describe the reasons this is an ostentatious conclusion at best.
First, we go back a couple of years
The contested claim is from a story that really gets going in the first phase of their study, which was published back in 2011. So to understand the recent publication, we must start at the beginning.
Briefly, this study used a double-blind design to investigate the effects of MDMA as a psychotherapy adjunct. They recruited 20 subjects, and decided to complicate their study off the bat by using an unbalanced design (unequal sample sizes). Their active drug group was 12 subjects, while the inactive placebo group was 8 subjects. They used common evaluation tools such as the CAPS (clinician-adminstered PTSD scale).
They set out to compare MDMA vs placebo. For the active drug, they administered 125 mg of MDMA at the start of the therapy session, and offered an optional supplemental dose of 62.5 mg MDMA 2-2.5 hours later. (I assume they also offered a supplemental placebo “dose” to the inactive group.) Not all subjects took them up on this, which complicates things further. Pharmacologically speaking, it gets even harder to interpret when you also consider that patients were asked to discontinue use of any psychotherapeutics for the duration of the study. (Withdrawal effects? Rebound effects? How long were they off the meds? We don’t know.) And some patients were given one or two medications to assist recovery after some MDMA+therapy sessions. (What are the effects of these sporadic and inconsistent treatments on consolidating progress made in therapy?)
When you wish to make a big claim, yes, your work is under a level of scrutiny that directly correlates with the size of the claim you intend to make. Their entire study design seems to treat subjects and groups inconsistently, which is a big problem when you want to harp on the strengths of the prospective, double-blind design of the study. Well folks, you have to treat all subjects the same way, too. To the greatest extent possible.
They found plenty of effects, which may or may not be directly related to the primary finding: that people are pretty effin’ good at determining whether or not you gave them MDMA. (As are the study observers.) So much for placebo, so much for the double-blind study.
So, in what reads like a grand gesture of shoulder-shrugging, they offered MDMA +therapy to their placebo group members in an open-label “crossover” phase. Except one placebo subject, whose CAPS score had decreased from 67 to 15, felt satisfied with his or her experience and declined further participation in the study. (The authors note that a second placebo subject experienced a decrease in CAPS score from 54 to 15 after therapy, which later increased to 64. This makes me ponder the effect of the therapy protocol as well as the drug treatment.) They present the data from the crossover phase in a table, arguing this data further demonstrates that MDMA+ therapy improves PTSD symptoms. We can’t tell this claim from a simple effect of more time passing after placebo+therapy sessions, since there is no control group anymore! Also, the way they say these things makes me wonder how exactly they ran their statistics. Well, the way they say things and their inconsistent use of descriptive statistics. (Yes, I lean toward the stats douchery. One kinda has to, in my field.)
These results, as presented, must be considered with a deluge of caveats. Here are a few important ones:
- Unbalanced design compromises a lot of things, particularly your ability to evaluate the effects of the therapy protocol alone. They seemed much more interested in comparing MDMA vs placebo, not placebo+therapy vs MDMA+therapy. This is one way to look at the problem, but I argue it is the less valid way to look at it.
- It’s pretty hard to argue for an effect when your control group is not functioning as a control, and your blinding is nonexistent.
- The non-standardized dosing (the optional additional dose) presents a big confound.
- The variability of therapy sessions and post-therapy-session drug treatment (with sleep drugs and benzodiazepines, which I view as a considerable blunder) again reduces the validity of the results. We now have a number of confounding factors that distract from the main effect they attempt to demonstrate.
- This is a tiny study, including just 20 people (12 MDMA, 8 placebo followed by crossover MDMA). Their population was primarily female and primarily survivors of child abuse or sexual trauma, so it’s also a big jump to assume that all populations and all causes of PTSD are going to apply.
That said, with all the limitations and caveats and skepticism, at the end of my first reading of this paper I still thought the data were interesting. Overall, I had the impression that it was worth a further look if they could improve the study design to clear up as many of these initial flaws as possible.
And then the follow-up
So the follow-up is where Science By Press Release took place just recently. Recall, at the end of the study we just analyzed, all participating subjects had received MDMA+therapy. The only acknowledged difference between groups at this point is whether or not the group received placebo+therapy a few months earlier. (Of course, there are many un-acknowledged differences within groups and between groups due to the inconsistent treatment design.)
Now the authors conduct a long term follow-up on their subjects. They received responses from all but three subjects. This gives them a total of 16 subjects (20 to start – 1 declined the open label second phase in the original study – 3 did not complete follow up). They take the data from these 16 individuals and see what can be learned from a more distant timepoint.
Some subjects appear to maintain similar CAPS scores, others seem to show further improvement between the last study timepoint and the long-term follow up, and still others seem to relapse. We see that some subjects completed a third MDMA+therapy session, as well. Importantly, the study authors reveal the full dataset in table format (rather than the graphs presented in the first paper, which were really uninformative without error bars). There is a far wider range of CAPS scores at study entry in the MDMA+therapy group (compared to the group that was originally assigned placebo+therapy and then put into a crossover study).
The authors boast that 8/19 subjects are in psychotherapy at follow-up (a 50% decrease from the 16/19 at study entry). However, they downplay the fact that of the three subjects NOT in therapy, one is in therapy at follow-up. (Though unbalanced, if they wish to compare these things, one could argue this is more than a 50% increase.)
The changes in medication prescriptions seems to be a wash. Some started new prescriptions, some went off prescriptions. I don’t see anything outstanding, and neither did the authors.
Still, they become so bold as to assume that the three subjects who did not complete the CAPS assessment showed improvement without the hard data to back it up and conclude, “Therefore, it may be the case that up to 89% (17/19) of those who received MDMA had long-term improvement in their PTSD symptoms.”
Wow. They quickly get their heads on straight and go back to the data, but that’s really amazing. Even so, they really have shifted to treating these subjects as a set of case studies rather than a treatment group to analyze. They kind of had to, given the lack of control group and not much else to go on. Still, when you shift from a prospective study to a series of case reports, you’re looking at far less powerful results. The press releases certainly don’t reflect that.
Again, it’s an interesting result, but at this point the power of the data is severely reduced- and quite a few of the losses of potential information gained are due to blunders on the part of the authors. That said- do we really need to go around selling early (VERY early) results in a civilian cohort (primarily women, primarily survivors of sexual violence) as the answer to a warfighter/combat trauma audience? That’s cheap and awful.
But lest we forget the conveniently ignored study…
One of the conclusions of this long-term study is clear: people know when you’ve given them MDMA. The subjective effects of the drug create a major problem in doing any kind of blinded study. So the authors of this study used a low dose of MDMA (25 mg + 12.5 mg) as an active placebo, and the same dose of MDMA that was used in the previous study (125 mg + 62.5 mg) as their therapeutic dose. Again they had a low number of subjects split into uneven groups (4 active placebo, 8 full dose MDMA). They claim it was to test safety of the full dose (didn’t the last study do this well enough?) and to enhance recruiting efforts. Huh. They added a third therapy session to their regimen, which differed from the first study.
They effectively solved the blinding problem. Subject and investigator guesses didn’t look a lot better than a random (50/50) chance of being right. So, that’s a plus to the active placebo!
However, there is a massive downside here. Magically, when your subjects can’t tell which experimental group they’re in… your clinical effect disappears. Placebo effect for the win!
Where were the press releases for this study? It was published in the very same issue of J Psychopharmacol as the Science By Press Release (long-term follow-up) study- in fact, it’s the next article in that issue- yet it’s ignored. I only stumbled upon the paper by accident, and was stunned to find the exact opposite of what all the press releases were screaming about.
This is why Science By Press Release sucks. It’s also why double-blind studies are useful, why holding extraordinary claims to high standards is important, and why replicability (especially for small sample size studies) can make or break a finding.
It’s whether you choose to incorporate that new knowledge in your worldview that makes you a Scientist rather than a True Believer in a romanticized idea that sounds great but might just not be right.
Notes:
* this is a politically correct term for "overreaching" or "not really correct"
I really don't intend to sound as uncharitable as I come off. However, I think that if you're going to invest the time and effort, it really should be done right and not halfassed. I see many lost opportunities to learn things here and it's disappointing. It's my opinion that, at this time, there's really not much more to go after on this therapeutic front.
Further reading:

