It's amazing how fast six months can pass, isn't it? Well, almost six months, anyway, as it was five and a half months ago that I wrote about a particularly execrable example of quackademic medicine in the form of a study that actually looked at an "energy healing" modality known as "energy chelation" as a treatment for cancer chemotherapy-induced fatigue. Actually, the study design itself wasn't so bad, leaving aside the utter ludicrousness of the concept of "energy chelation." Rather, it was how the authors spun interpreted their results that set my head spinning. Surprisingly, a letter to the editor was accepted for publication describing exactly why this study was a pair of fetid dingo's kidneys. The letter itself and the authors' response to it are quite instructive, which is why I decided to revisit this study.
Before I do, let's just briefly provide the CliffsNotes version why I wasn't impressed with the original study. (If you want the gory details, feel free to go back to my original post to refresh your memory.) First, none of the primary outcomes showed statistically significant differences between placebo control and treatment groups. Unfortunately, that didn't stop the authors from mining their dataset for secondary outcomes. Not surprisingly, they found barely statistically significant differences between the control and treatment arms in a couple of these secondary endpoints plus a surrogate endpoint that they proposed as a biomarker (cortisol slope) even though it's never been validated as a measure of chemotherapy-induced fatigue. These deficiencies in the study interpretation led me to ask this question six months ago: What do we normally call it when there is no difference between the real treatment and a sham treatment in a clinical trial testing a drug or device? That's right. We say there's no effect greater than that of a placebo and that the trial is negative; i.e., the tested experimental intervention doesn't work.
Let me remind you that that is not what the authors concluded. Here's a snippet from the conclusion of the paper:
This RCT examined whether biofield healing, compared with both active (mock healing) and waitlist control groups, positively affected fatigue as well as cortisol slope, depression, and QOL in breast cancer survivors with persistent fatigue. In addition, this study explored the role of belief in receiving healing as a potential predictor of responses. Findings indicate that both touch-based interventions reduce fatigue in fatigued breast cancer survivors, with considerable effect sizes. Previous research by our group on a separate sample of breast cancer patients indicated that the mean Multidimensional Fatigue Symptom Inventory-short form total scores was 5.99 immediately before the start of anthracycline-based chemotherapy, and rose to 19.9 immediately before the fourth cycle.38 Our fatigued survivors in the mock healing group (mean postintervention score ¼ 10.9) dropped to fatigue scores lower than those found for breast cancer patients toward the end of chemotherapy, and the biofield healing group (mean postintervention score ¼ 4.2) fell to fatigue scores that were below prechemotherapy scores, as well as below previously published means noted for breast cancer patients overall.28 This drop in fatigue appears to have clinical as well as statistical significance.
In other words, according to the authors, because both the "real" biofield healing and the "mock" biofield healing resulted in a decrease in fatigue scores and a barely statistically significant difference in a questionable surrogate marker, biofield healing "works." This is the same sort of dubious rationale frequently used to claim that acupuncture "works" when researchers find that sham acupuncture results in the same apparent measured effects as "real" acupuncture. Again, the correct interpretation of this study is that it's a negative study, and "energy chelation' does not work. It is placebo. Stick a fork in it; it's done.
Surprisingly, the editors of the journal that published the original study, Cancer, accepted a, that's essentially the point of a letter to the editor that says, in essence, exactly the same thing. One of the authors of the letter is known to many of my readers, but that's all I'll say about that matter. What's far more important is the message:
The registered primary outcomes of the trial were self-reported fatigue, depressive symptoms, and quality of life. No significant differences were obtained between TT and the mock treatment, whereas both conditions were superior to waitlist control. Essentially, the authors examined 3 primary outcomes, with secondary analyses of 5 subscales of the fatigue measure, and a secondary outcome, cortisol, with all pairwise differences explored between the TT, mock treatment, and waitlist control conditions. With any control for multiple comparisons, the modest difference between TT and mock treatment in cortisol is no longer significant.
And:
There is no known therapeutic benefit to changed cortisol slopes. To justify cortisol as a secondary outcome, the authors selectively cite findings that flatter cortisol slopes are modestly related to metastatic disease and predict mortality in breast cancer patients. These limited correlational data alone do nothing to establish that cortisol is a suitable surrogate endpoint.
The letter even explicitly pointed out that this trial was negative, something I believe to be even more true when one takes into account prior probability and the lack of correcting for multiple comparisons:
We believe that publication of this TT trial encourages more pseudoscientific studies of energy fields or auras and gives the wrong message to clinicians and patients.
Hear, hear!
Not surprisingly, the authors of the original study were not too happy about the letter. Even less surprisingly, in their response they retreated to common tropes used by apologists for reiki and "energy healing." In fact, their response is a veritable template for defending tooth fairy science. First, the authors tried to disabuse their critics of their "misconceptions" about the study by pointing out how very wrong they were to lump "energy chelation" in with healing touch (HT), therapeutic touch (TT), and reiki as an "energy healing" modality:
We wish to clarify some misconceptions put forth by Coyne, Johansen, and Gorski regarding our reported randomized controlled trial.1 First, the intervention used was not therapeutic touch but a specific hands-on technique commonly used in many types of biofield therapies for ameliorating fatigue.
Which matters not at all. It was the authors, after all, who said that energy chelation is a "biofield therapy," which is another name for "energy healing." In any case, their complaint reminds me of arguing that reiki is different from TT because reiki masters use different hand motions to channel the "healing energy" or that in reiki the energy comes from the "universal source" while energy chelation removes "energy blocks" in the patient himself. Until you can convincingly demonstrate that this "universal source" exists and can be manipulated by reiki masters and/or that there are "energy blockages" that "energy chelation" practitioners can remove, all you're doing is comparing two different forms of magic. Alternatively, you can demonstrate with overwhelming indisputable evidence so powerful as to make us question previously understood laws of physics indicating that these techniques cause objective responses, but unfortunately for the authors of this paper this study does nothing of the sort. What this study found were effects due to energy chelation that were no greater than placebo on primary endpoints, all of which are subjective responses, and an unvalidated surrogate endpoint that demonstrates a barely statistically significant effect (p=0.04), after no correction for multiple comparisons. In other words, the authors' response completely misses the point.
Here's their next objection:
Second, there is an evidence base for biofield therapies.2,3
The authors cite this Cochrane Review and one of their own reviews. The problem with this argument, of course, is at the very core of the difference between "science-based medicine" (EBM) and "evidence-based medicine" (SBM). In EBM is no consideration of prior plausibility based on basic science in the studies examined, which were all over the place as far as quality goes. Add to what the Cochrane Review characterizes as poor quality, equivocal data the fact that the clinical trials involved testing a class of healing modalities whose explanation if effective would require, as homeopathy would, that huge swaths of well-established physics, chemistry, and biology (particularly neurobiology) to be overthrown, and the reasonable conclusion is that "biofield therapies" do not work. In particular, this is a case where "statistically significant" doesn't mean "clinically significant," not by a long shot, given that the decrease in pain reported in the Cochrane review was less than 1 unit on a typical pain scale that goes from 1 to 10. One unit has commonly been viewed as the smallest decrease in pain that a patient can perceive.
Objection number three follows:
Third, the study was designed to examine nonspecific and placebo elements that may drive responses: This is why we used the mock healing group as a comparison along with the waitlist control group. We also examined patient expectancy, belief, and patient ratings of practitioner attributes all elements of placebo as potential predictors.
So what? It was the authors who concluded against the evidence in their very own study that, in essence, their "energy chelation therapy" works to relieve symptoms of chemotherapy-induced fatigue even though the "real" energy chelation and the "mock" energy chelation were indistinguishable. The correct conclusion should have been that energy chelation performed no better than placebo and therefore did not work. Again, if energy chelation were a drug therapy, would the authors conclude from a result in which the drug does no better than placebo for the primary outcome measures that the drug worked? Why the double standard?
Up next is this objection:
Fourth, despite Coyne et al.'s seemingly contradictory statements (stating that the study is underpowered while also suggesting cortisol slope results should have been Bonferroni corrected), the power analysis and statistics are correct and clearly described.
"Contradictory"? I fail to see what's "contradictory" in pointing out that the study was underpowered and that the cortisol slope results should have been Bonferroni-corrected; i.e., corrected for multiple comparisons. They are separate criticisms. Even if the study were adequately powered, it would still be flawed because of the lack of correction for multiple comparisons. At least if the authors had properly corrected for multiple comparisons then they could have blamed their negative result on the inadequate statistical power of the study!
Finally, the authors write something that both amused and depressed me at the same time:
The larger issue is what constitutes "pseudoscience" and what information is worthy of dissemination to the public. Should the data from our well conducted, rigorous, randomized controlled trial be dismissed because the mechanisms are unknown or because some scientists do not believe in the specific therapy? We make no claims surrounding mechanisms. We do note that this intervention has significant promise for reducing fatigue, which is the most common complaint among cancer patients, and the therapy produces no harm. Therefore, it merits further investigation. Premature rejection of findings from rigorous randomized controlled trials are as big a threat to science as the continuation of falsehoods based on belief. Thus, as clinicians and scientists, our highest duty to patients should be to investigate promising solutions with high benefit/risk ratios, not to act as gatekeepers of information based on personal opinion.
This paragraph is so wrong that it's not even wrong. Note the wounded cry about "dismissing" results based on dogma rather than science. Note the straw man argument that supporters of SBM reject the results of this study because they "do not believe in the specific therapy" or because the "mechanisms are unknown." The first trope is a massive misstatement of SBM objections to studies such as this. A more accurate characterization would be to say that the results of this trial do not mean what the authors think they mean. The authors conclude that the trial indicates that energy chelation shows "significant promise." (They even repeated that assertion in their response!) An SBM-derived analysis would have concluded that the study's own results indicate that energy chelation functions no better than placebo and therefore does not work. It would have been nice if the authors had addressed the actual criticism than such an easily revealed straw man version of it.
The second trope is commonly used in defense of pseudoscience because it is difficult for many to understand that there's a huge difference between a mechanism that is "unknown" and a mechanism that is physically impossible based on current scientific understanding. An example of the former is a drug whose mechanism of action is as yet unknown but whose effects are easily documented. We know that the drug must function through some sort of biochemical interaction with a receptor, enzyme, or other macromolecule within the cell that we can discover and that we do not need to invoke mechanisms that break the laws of physics and chemistry to explain the drug's effects. Again, to illustrate the difference between such a drug and impossible mechanisms, I like to use the example of homeopathy, which, if it worked would necessitate the overthrow of huge amounts of exceedingly well-established science in multiple disciplines, including physics, chemistry, and biology. Let's just put it this way. Energy chelation is the same. For it to "work," the same sorts of vast quantities of well-established science would need to be overthrown.
As a skeptic, I have to admit that it's certainly possible, albeit infinitessimally so, that so much of what we understand about science is not just wrong and/or incomplete, but so incredibly wrong and/or incomplete that there might be an as yet undiscovered physical mechanism by which energy chelation and "biofield therapies" could work. However, if you're going to convince me that something like energy chelation can truly work as a treatment modality, you'd better have evidence far more compelling than a small, equivocal clinical trial like this in which the effect on primary outcomes was no greater than placebo and whose analysis of secondary outcomes didn't even bother to correct for multiple comparisons. Bringing a study like this "energy chelation" study to argue that "biofield therapies" work (or even that they might work) is akin not just to bringing a knife to a gun fight. It's more akin to bringing fists to an M2 Browning machine gun and grenade fight. Or maybe bringing cavalry to a tank battle.
And that's not just our "personal opinion," either. Of course, it's far easier to dismiss criticisms that one can somehow label "personal opinion" than it is to address the actual criticisms. That's probably why the authors try to characterize the objections to their study as nothing more than a disagreement of opinion, before floating off into the ether of a self-righteous and condescending lecture the "highest duty" as clinicians and scientists. I retort that, as clinicians and scientists, it is our highest duty not to engage in magical thinking that subverts science-based medicine. It's our highest duty not to waste precious resources investigating therapies so utterly implausible that their efficacy requires that the laws of physics be overturned in favor of magic. It's our highest duty to base our treatments in science, not in prescientific vitalism and religion, which is all that most "biofield" therapies are: faith healing, the laying on of hands. I would also retort that our tax dollars should not be funding magic like this.
Yes, you guessed it; the original energy chelation study was funded in part by NCCAM.
REFERENCES:
- Jain S, D Pavlik, J Distefan, RR Bruyere, J Acer, R Garcia, I Coulter, J Ives, SC Roesch, W Jonas, and PJ Mills (2012). Complementary medicine for fatigue and cortisol variability in breast cancer survivors: A randomized controlled trial. Cancer 118: 777-787. DOI: 10.1002/cncr.26345
- Coyne JC, C Johansen, and DH Gorski (2012). Letter re: Complementary medicine for fatigue and cortisol variability in breast cancer survivors: A randomized controlled trial. Cancer, E-pub ahead of print. DOI: 10.1002/cncr.27415.
- Jain S, D Pavlik, J Distefan, RR Bruyere, J Acer, R Garcia, I Coulter, J Ives, SC Roesch, W Jonas, and PJ Mills (2012). Response re: Complementary medicine for fatigue and cortisol variability in breast cancer survivors: A randomized controlled trial. Cancer. E-pub ahead of print. DOI: 10.1002/cncr.27421.
- So PS, Y Jiang, and Y Qin (2008). Touch therapies for pain relief in adults. Cochrane Database Syst Rev CD006535. DOI: 10.1002/14651858.CD006535.pub2
- Log in to post comments
Wow - pseudoscience at its best/worst.....
Sooo faith healing is now 'biofield therapy', in the same way CAM is now 'integrated medicine'. They do so love to play with with, to give the naïve the impression it's something totally new and absolutely not debunked at all.
Bunch of [insert preferred derogatory epithet here]
I often teach physics to premeds.
Many of my students are wonderful -- future Oracs! -- and all of them are remarkably diligent, but many are reluctant to actually think about the meaning of science. They want to get through the course with an "A", and learn just enough to fill in the right bubbles on the MCAT. Physics seems especially irrelevant to them.
This is unfortunate, since a halfway-decent education in physics is very helpful in understanding just why "biofield therapy", various Chopra-babble stuff, and the like, are, well ... bullshit.
I wonder how those of us involved in pre-medical and medical training can get these folks to realize that it's not good enough to be a trained monkey?
Incidentally, premeds are now known as "pre-health" students, which I suppose does not mean "still sick".
Biofield Therapy sounds ever so much like the auras I read about in my youth or the Kirlian photography made famous most recently in the movie Ghostbusters II. What a waste of time and energy, no pun intended.
But, but we were rigorous in testing our magic, we compared our real magic to fake magic and it was clearly effi . . . what?
Uh huh.
It did?
Nevermind . . . you guys are big meany-heads, you know.
What the heck is the etymology of "energy chelation" anyway? Double-checking on Wiki (because I have the IQ of a nose booger at the moment, dratted hell-plague) chelation is the formation of an organic molecule around an atom, usually a metal... I just don't see how energy, at least in the "biofields/The Force" sense, comes into chelation it at all.
The term alone should ring alarm bells about the plausibility.
-- Steve
@Steve
I think that's the crux of what Orac's saying about this being the equivalent of Homeopathy on the pseudoscience implausibility chart. If it were to work as the proponents describe, there would be molecules of heavy metals *poofing* out of existence. It's really embarrassing for the study conductors that they would defend this level of nonsense. Particularly their defense that it's different from TT or Reiki. (I guess because the word chelation is attached? Seems pretty damn identical).
@ Anton P. Nym
As you found out, chelation is a chemical process in which one or more organic molecules attach themselves strongly to a metal atom.
Hence chelation being touted as the cure-all to remove "heavy metal poisoning" in certain circles.
(trivia: the cyanid ion is a chelator. Wonder why no-one wants to use this one)
Energy chelation looks like the same, except it's fairy dust instead of nasty chemicals, I mean organic molecules.
The proponents of this protocols forgot in the first place to prove that fairy dust exists and can chelate.
Shouldn't the same principles that remove Ten Commandments artwork from courthouses prevent tax dollars from funding magical research?
Jain et al whine:
No, the data should not be dismissed. Unfortunately, that's exactly what Jain did. They acknowledge that the data says their magic technique is no better than placebo. Then they turn around and dismiss that very data to claim that their "therapy" has "significant promise for reducing fatigue."
Presumably the differences between energy chelation, healing touch, therapeutic touch, and reiki are like the differences between the Judean People's Front, the People's Front of Judea, the Judean Popular People's Front and the Popular Front of Judea?
Yes, they should: http://www.sciencebasedmedicine.org/index.php/what-is-science/ No doubt these clowns would whine even harder but they should be told to take their cargo cult science nonsense away and not come back until they've gotten some interesting (and reproducible) results by waving their healing hands over a torsion balance in a vacuum jar.
palindrom: "a halfway-decent education in physics is very helpful in understanding just why "biofield therapy", various Chopra-babble stuff, and the like, are, well ... bullshit."
True, but it still doesn't seem worthwhile for pre-meds to have to undergo a year of physics for that limited benefit.
As an alternative, the medical profession could maintain several physicists on retainer* for the purpose of refuting "energy chelation", "quantum homeopathy" and the like.
*we'd need one on call at all times. :)
" ... it still doesn't seem worthwhile for pre-meds to have to undergo a year of physics for that limited benefit."
Ah, but Mr. Bacon, the physics also serves the purpose of giving premeds insight into how medical devices actually work. For the good ones, physics also serves as a fine example of the value of understanding principles and working forward from those, rather than mindlessly committing factoids to memory (which so many other science courses seem to encourage). Some of my diligent students would much rather memorize every equation in the book than sit for a half hour puzzling over a conceptual conundrum. They feel that they are wasting their time if their pencil isn't moving. For many if these students, physics should help reinforce the skills needed for real intellectual growth, such as critical evaluation of one's own understanding (sort of inverse Dunning-Kruger). I do my darndest to teach them these skills.
On a more brutal level, premed physics also serves as part of the safety net that, one hopes, prevents the unqualified and the unmotivated from proceeding careers as physicians, where they could really hurt people. There is ample evidence in the archives of this wonderful blog that this safety net is not foolproof.
I agree with palindrom; medical students should have some basic physics. Honestly, I would expect them to have AT LEAST the amount of physics I got in 12th grade, which would be enough for them to dismiss "energy chelation" as nonsense right off the bat. And they definitely should have a decent grounding in chemistry.
Physics is increasingly important in medicine. Really, it's always been important, but it's about more than just debunking crazy stuff like magnetic bracelets. They need to understand what an MRI machine is actually doing, what risks are really present with an X-ray machine or a CAT scanner, how a radiotherapy machine works, and more.
Believe it or not but there is a push by woo proponents to get various forms of woo accepted, even demanded, by chaplains as part of the normal work of chaplaincy in health care facilities. The trials they will not support, though, are those that compare results of woo to results of a calm nice person trained in active listening who comes into the room, makes sure everyone has a chair, and offers sane emotional support and even prayer if folks want it. I submit there's some merit to the latter, and that the woo adds nothing at all. I also submit that suggesting that folks try woo instead of appropriate medical treatment for things like cancer pain is inhumanly cruel.
Medical students aren't the only ones that should have basic physics. I'm sure I asking too much for high school students to learn the basics, but anyone who graduates from college should learn basic physics. This nonsense spreads because our society is science illiterate.
@phayes,
No, the data should not be dismissed.
I agree with your comment on SBM: when clinical data contradict well-established fundamental knowledge, then the correct conclusion is that the clinical data are almost certainly in error. But in this case the clinical data agree perfectly with fundamental science: energy chelation did nothing other than what you'd expect from a placebo. As Orac describes, the data aren't in error, the authors' interpretations are in error.
And even when clinical data does conflict with fundamental science, I wouldn't say we should dismiss it. Merely that we should evaluate it in the full context of what we know. Concluding with strong justification that the data are erroneous is not the same as simply dismissing it.
Now, what should have been dismissed here was the authors' proposal to conduct this study in the first place!
Basic physics was required for my biology degree (in fact was required for any science degree where I went to university), is required for all US and Canadian four year medical schools http://www.bestpremed.com/preMDreq.php, and is either part of the six year medical school curriculum or a high school requirement everywhere else.
US and Canadian medical schools currently require a bachelor's degree (any major as long as the pre-med requirements are also completed) before one can enter medical school. However, the traditional minimum length of an undergraduate plus medical education was six years. In the US and Canada, it was traditional to do two years of undergraduate study and then enter medical school, but as medical school became much more competitive in the sixties and seventies, it became unusual and eventually not allowed for students to enter without finishing a bachelor's degree. Elsewhere in the world students often enter a six year medical program after high school, but that essentially includes two years of undergraduate level science.
As for this paper, it's amazing that the authors initially did the right thing and set up a controlled study.
However, part of the scientific approach is having the emotional capacity to admit that you were wrong when the evidence says so. They apparently weren't psychologically capable of that final step.
Right - because it was known in advance (or should've been) that the data would be irrelevant to the strongly justified conclusion! It is - logically - exactly the same as simply dismissing it but there's nothing wrong with that.
Respectfully, for once, I disagree with Orac.
This study should have been funded, because it's not just any new woo, it's an old and bizarrely popular one.
The 'laying on of hands' has been recognised for its placebo response for even longer than chicken soup. A bit of TLC does wonders, always has done.
The nonsense about energy and auras notwithstanding (ask anyone who can 'see your aura' to stand the other side of a door and say when your hand is next to the edge - they will refuse the test without exception), when an idea is so widespread that it is affecting society; get it tested. Properly.
They asked for proper tests, they got them, they even had 'pro-CAM' people doing them.
The tests said no. Subject closed.
Re. Pareidolius @ #5: LOL, top comments of the month nomination for that one. I was eating a cookie when I read it and nearly got crumbs in my keyboard.
--
The misuse of the word "energy" in all things woo really ticks me off. Usually it's a poor substitute for something along the lines of "mood" or "emotion," and aside from demonstrating a complete ignorance of physics, it betrays a complete lack of personal self-insight into one's own subjective phenomena.
However, if we translate "energy" to "mood," we might get to a falsifiable hypothesis: that "interventions having elements in common with expressions of affection" improve the mood of patients who are in treatment for life-threatening illnesses. For this we might develop a set of survey questions designed to assess mood, and another set of questions to assess beliefs, and then test a range of these "affection-resembling interventions" on a statistically relevant number of patients.
I would be willing to guess that patients' moods are improved by affection-resembling interventions that they believe are plausible, and this shouldn't be surprising.
However that does not get us an open door for Licensed Clinical Magicians.
What it should get us is psychologists teaching patients' families and friends how to communicate affection and other supportive emotions more effectively to patients. IMHO that would be a good thing, and worthy of taxpayer support.