Thinking critically about ME research

The laziest kind of statistical research is that which relies solely on presenting the results of statistical tests, without making any meaningfull effort to critically interpret the results. Prima facie acceptance, without critical inquiry, leads to noise being misidentified as meaningful, to bad science and bad policy.

With that in mind, it is worth taking a look at an article by Crawley et al. (2017) published on 20 September 2017 in Archives of Disease in Childhood , investigating the effectiveness of the Lightning Process for children with myalgic encephalitis (ME, also known as Chronic Fatigue Syndrome or CFS). Esther Crawley is something of a bugbear for many patients with ME/CFS, being a staunch proponent of the controversial PACE trial and related research,¹ but I am less interested in her reputation or that of any other authors; the content of the study itself is the most important thing.

The Lightning Process (LP) is a three day personal development course that is fully grounded in some of the finest pseudo-science available - namely neuro-lingustic programming and osteopathy - plus some life coaching. Crawley et al optimistically report that “LP is effective and is probably cost-effective” for treating children with mild/moderate ME, on the basis that they found participants receiving LP and standard medical care (SMC) reported slightly higher physical function scores than participants receiving only SMC, as well as slightly better results on a variety of secondary measures. These results are interesting, but they are nowhere near good enough to say that “LP is effective” for treating ME.

Statistical significance or non-overlapping confidence intervals are not by themselves enough to attribute effectiveness to a particular intervention, they merely indicate potential. Anyone who has taken an introductory statistics module knows that correlation does not equal causation and, in the case of medical interventions, the placebo effect cannot be ignored. As this was not a blinded study, participants knew what intervention they were (or were not receiving). Participants were openly recruited, so the study may have attracted participants interested in the opportunity to receive the LP free of charge (it typically costs £620 and is not covered by the NHS), and who believed that the LP would help their ME. Nowhere do Crawley et al meaningfully consider the possibility of the placebo effect, or the effect of self selection by study participants, a strange oversight given that only 100 of the 310 eligible agreed to participate in the study.

Furthermore, a proper declaration of intervention effectiveness requires a strong explanatory underpinning. While it is true that LP has not been subject to a randomised control trial prior to now, its various component elements are, to put it politely, bullshit. Multiple flavours of pseudo-scientific nonsense can’t be combined into an effective medical intervention. Neuro-linguistic programming has been thoroughly discredited (the Wikipedia page for NLP has a good summary), and Osteopathy is, to quote Edzard Ernst, ‘little more than wishful thinking’. Life coaching, while not necessarily bullshit, is not going to be any use in treating a physical illness, and does not make sense for the treatment of psychological disorders compared to conventional forms of therapy.²

In short, Crawley et al claim that LP is effective for treating ME, despite providing no meaningful explanation for why this may be the case, completely ignoring the placebo effect and failing to critically consider their findings. The studies authors really should know better than to endorse LP – a commercial treatment based on pseudo-science – as an effective treatment for adolescent ME on the basis of one small trial. While Crawley admits being suprised at the positive results of the trial, the decision to uncritically report that LP ‘works’, with no urging of caution or qualifying statements, seems at best unusual, at worst irresponsible and inappropriate. The extensive media campaign accompanying publication, including an appearance on the Today Programme and numerous newspaper articles (see this roundup from the ME Association), is extraordinary given that the article itself is, at best, mildly interesting. If I was feeling conspiratorial I would say it was an attempt to maintain the appearance of scientific respectability in psychological approaches to ME, as the scientific community abandons the psychological approach in favour of biomarker research and drug trials. It is quite beyond me why this study received any attention, given recent developments like the Center for Disease Control in the US dropping all references to CBT-based interventions for ME, or the identification of possible biomarkers for ME by Montoya et al. (2017).

The greatest contribution to science made by Crawley et al. (2017) is how the paper serves as an important reminder to always critically consider statistical results, and not get carried away by declaring a commercial product based on pseudo-science an effective treatment for any condition on the basis of a single non-blinded trial.