The media had a feeding frenzy when a data-mining report from Yale researchers published in JAMA Oncology suggested a causal relationship between use of “complementary medicine” and shortened life span among cancer patients. A New York Times subheading was representative: “People who used herbs, acupuncture and other complementary treatments tended to die earlier than those who didn’t.”

The appearance of this apparent death knell due to the “nostrums” as the Times writer called them – “herbs, vitamins, traditional Chinese medicine, homeopathy, naturopathy, yoga, acupuncture and others” – had a fascinating timeliness. The publication of “Complementary Medicine, Refusal of Cancer Therapy, and Survival among Cancer Patients with Curable Cancers” arrived within months of news that the uber-mainstream American Society for Clinical Oncology endorsed integrative breast cancer guidelines that feature many of the same “complementary” approaches.

Given the estimates of life lost among the complementary medicine users reported by the study’s lead authors Skyler Johnson, MD, James Yu, MD, MHS and colleagues, ASCO would seem to be aiding and abetting a rather nefarious process.

What gives? I reached out to leading integrative oncologists and then, on the suggestion of one, to statisticians – and then to the lead author. What I found raises questions as to whether JAMA Oncology should retract the paper – and if the Times and other media outlet follow with their own mea culpas. An undeniable conclusion is that the poorly vetted research has provoked a damaging sort of fake news if one assumes such a study should help the public and clinicians know whether to recommend and use complementary and integrative practices.

Two Integrative oncologist perspectives

University of California at San Francisco professor of clinical medicine and integrative oncologist Donald Abrams, MD was familiar with the study when I reached him. He knew that the author team had published a 2017 report mining the same USA National Cancer Database that concluded lifespan shortened in those who say no to standard care and instead turn to “alternative” methods.

Abrams replied to an electronic query via a long paragraph attached to an email. These were the main points.

• The definition of “complementary” (practices used to complement usual oncologic care such as chemo and radiation) was confounded with “alternative” (therapies used instead of).
• He noted that even the authors note that if one removes those who have delayed treatment or refused treatment – thus who would seem to view their CM as “alternative” rather than purely “complementary” – that “all of the differences go away.”
• He questioned the statistical basis: “Really, out of 1,901,557 people in the cohort [of cancer patients from the USA National Cancer Database], they only find 258 using these oddly defined treatments?” He knew that multiple surveys have established that at least 30%-50% of people with cancer use complementary medicine.
• Abrams called “retro and uninformed” the assumption that complementary services are provided by “non-medical providers.” Many licensed practitioners are providing the services, including many medical doctors and nurses in scores of top academic medical centers.

In short, fundamental definitions in this article were wrong and the statistical work overlaid on them were out of whack with all prior research. Abrams concluded with perhaps more gentleness than this combined perspective would seem to deserve: “A poor paper in my opinion.”

Lynda Balneaves, RN, PhD, the current president of the Society for Integrative Oncology (SIO) – the organization that produced the original guidelines that led to the ASCO-endorsed breast cancer guidelines noted above – began her response generously. She shared that the SIO “always encourages people to conduct research in the area of complementary medicine, It’s Important to take advantage of databases, to get more population-based perception.” She added: “We only wish that we had more databases with better designed assessments and measurement of complementary therapies and outcomes for these therapies.”

Then, in the course of a 45-minute interview published here, Balneaves entered more deeply into the multiple troughs Abrams’ critique had begun to carve in the Yale researchers’ article. Her questions followed the same pattern. She focused on the major “mis-classification bias” of the authors. She adds: “Why would they use a definition of ‘complementary’ from 1998?” She noted that SIO has in fact published a formal “Comprehensive Definition of Integrative Oncology” that was readily available.  She also wondered “why didn’t they bring onto their research team someone with expertise in the field?”

These perspectives from the SIO president were echoed in a document provided by Heather Wright, ND, FABNO, president of the Oncology Association of Naturopathic Physicians (OncANP) which was responsible for its contents. Three naturopathic physicians specializing in oncology were on the team that produced the SIO breast cancer guidelines, including the lead author, Heather Greenlee, ND, PhD.

The OncANP team called out the authors for “trying but failing” to distinguish between “alternative” and “complementary.” They conclude: “The present study found that patients who choose to pursue alternative therapies (not in coordination with oncology care) demonstrate consistently poorer outcomes. The finding is not representative of those patients who choose integrative and complementary cancer care.”

Analysis from a Statistician

Balneaves suggested I get the view of a statistician. She led me to contact Scott Mist, PhD, MAcOM, MS, MA, LAc. Mist is an assistant professor for anesthesiology and perioperative medicine at Oregon Health Sciences University School of Medicine. He also serves as the associate editor for statistics at JACM – Paradigm, Practice and Policy Advancing Integrative Health (The Journal of Alternative and Complementary Medicine). I contribute to JACM as editor-in-chief.

Mist e-mailed a bulleted response within hours of the request. Given the importance of clarifying whether complementary therapies are in fact-life life-threatening, I asked Mist to edit the e-mail for broader consumption, which he did. I include it in full below.  I focus here on the two point to which Mist most called attention.

First, Mist raises a bright red flag over the decision of Johnson and his team to focus on just 258 of 1,901,815 patient that the authors put in the CM group. Mist points out that, elsewhere in the paper, the authors note that multiple surveys have found that between 48% and 88% of people with various types of cancer use some complementary methods. He does the math: “There should have been between 836,799 and 1,673,597 CM users. But they found [just 258] less than 0.01%.  That should have stopped the study right there.” He adds for emphasis: “There is no way that less than 1/100^th of the population is representative of the population of CM users.”

Mist’s analysis raises additional questions regarding other extrapolations that appear to be based on questionable assumptions. He suggests a frequently used method for establishing a set of comparators was inappropriately applied. He concludes with what he considered the other most significant issue: “They say that when controlling for refusal rates and delays, there is no difference in the survival rates. Therefore, it can’t be the CM treatments that are the difference.” Game over.

Response Following Request to the Authors

I sought an interview with Johnson on Friday August 10, 2018, by emailing him Mist’s memo. I shared my plans for a special set of articles to be published August 15, 2018. Then a column in JACM. JJohnson responded promptly on Monday: “Thanks for reaching out. I’m a huge proponent of critical analysis of scientific work. The co-authors and I believe that these results strengthen the position of alternative practitioners and integrative oncologists, who recommend evidence based practices for cancer patients, by encouraging treatment adherence to proven therapies while seeking to improve patient quality of life. We would love to work with members of the integrative oncology community to test these findings prospectively.” He did not directly address any of the issues raised other than to refer my to some paragraphs in the paper. (See the entire dialogue with Johnson/Yu, including responses from Mist and Balneaves, here.)

Should JAMA Oncology be asked to retract the paper?

Bad papers get published. But rarely does a single paper as poor as this one have the impact that this has had, in terms of media play and wrong-headed public education. Should JAMA Oncology be asked to retract the paper – especially given the damaging message to the public?

I put the question to Balneaves, who noted that usually it is either ethical issues or major fatal methodological flaws. She wasn’t ready to make such a call. Mist shared his reflections by email: “Honestly I don’t know. I personally wouldn’t have accepted it simply because there are two missing analyses and the database is incomplete given that around half [of people] are using CM but only 0.01% were registered as using CM.” He continued: “If it were me, I would have made them do the analyses, make alternative hypotheses of why it might be so, and say that this is a preliminary study that needs a prospective study to see if the relationship holds true. I would place serious money that it wouldn’t hold true.”

Then in an email that immediately followed Mist added: “And I don’t like wagering money.”

Chad Aschtgen, ND, FABNO co-founded Seattle Integrative Oncology through which he and his colleagues have agreements with Swedish Medical Center and other leading Seattle area medical establishments to help manage their patient interested in integrative care. After ticking off the familiar litany of challenges to the JAMA Oncology paper, Aschtgen he drove home the negative effect of its publication: “It gives the wrong message to practitioners who are beginning to see the value to their patients in complementary therapies, and to the patients whose lives can benefit from them.”

Coda: I have bias here. I have had cancer – tonsillar cancer – and for it, took 33 radiation treatments and three cisplatin blasts, a treatment plan that University of Washington endowed professor and head and neck surgery chair Neal Futran, MD, DMD, once transparently described to me as “one of the most brutal things we do.” His colleagues successfully burned the cancer away – and concurrently caused and left me with multiple adverse effects. So I also am among the 48%-88% who use complementary therapies and practitioners. (In fact, Dr. Aschtgen is my naturopathic oncology expert with whom I presently stil consult from time to time.) It would take many moons to share all the ways I experience as beneficial the multitude of practices and practitioners from which I have benefitted over the 9 years since my diagnosis, relative to the damage and sequellae of that brutality.

I asked many who I interviewed if they believed that bias informed the development, the review process, and decision to publish this article. I can just say that the consensus was that it is a shame that good professionals, with separate expertise, were not talking to each other when this study was developed. That can be remedied, Dr. Johnson, and Dr. Yu. But the damage done to conventional physician and patient perspectives by the publication will not be undone unless JAMA Oncology sets itself to making amends for the adverse effects of its publication decision.

________________________________________________

The Statistical Issues in the JAMA Oncology Paper: Memo from Scott Mist, PhD, MAcOM, MS, MA, LAc

Sent to me via e-mail, August 9, 2018, by Scott Mist, PhD, MAcOM, MS, MA, LAc and then edited by him for a broader audience. Mist is an assistant professor for anesthesiology and perioperative medicine and the Oregon Health Sciences University School of Medicine. He also serves as the associate editor for statistics at JACM – Paradigm, Practice and Policy Advancing Integrative Health (The Journal of Alternative and Complementary Medicine). His comments follow.

MIST: Of these arguments, the first and the last are the two strongest.

• If the estimate is that between 48% and 88% of the population are using CM (their term) there should have been between 836,799 and 1,673,597 CM users. But they found less than 0.01%.  That should have stopped the study right there.  There is no way that the less than 1/100^thof the population is representative of the population of CM users.  Further this means that at least half or more of their control group is also doing CM but it wasn’t captured.
• Next, you have a problem of known population sizes and sample size. For a population of greater than a million, you need a sample size of 384 to have a 5% margin of error. For a 2.5% margin of error you need 1534. So, by doing matched propensity scores and finding 4 closest neighbors, they are effectively changing the population size from greater than a million to 1290 and assuming that because they are matched and that they represent the greater population. This is the only way that they are able to get significant findings. This is commonly and rightly done in cases of rare occurrences but if the group of interest is not representative, and given the critique from the first point, then one is effectively having issues with outliers. Given that the authors were drilling down into subpopulations (28 cases having died within 5 years – 0.0015% of the sample) within the types of cancer (186 cases (0.0098% of the sample), I feel it is safe to say it is a case of outliers driving the analysis. Should the statement that between 48% and 88% of the population are using CM be true, their analysis suggests that they should have seen 99000 and 198000 such incidents within their database.
• There are several issues beyond the poor quality of the data collected in the database. First, there are several reasons why there are only a few reporting CM treatments and all of the reasons indicate that the database isn’t suited to this analysis. The first reason is that this information is poorly recorded. Other research indicates that MDs are very bad at recording CM treatments. Patients commonly are using other modalities and are afraid to discuss it with their practitioner. Further, EMRs are not well set up to record this kind of data and it is not in the training of most MDs who are already pressed with shorter visits.  Second, the patients that are in the database may be better communicators and are relatively unique. As such, they may not be representative of the greater population of CM users.
• Next, the authors state that they excluded patients with “incurable disease”. I would (sarcastically) argue that 1) they clearly didn’t or there wouldn’t have been anyone dead in their study and 2) that their database is probably flawed in that assumption as well. The variability in what is “incurable disease” is large and not well captured.
• The authors report that it was borderline significant for colorectal cancer. By their own criteria, it shouldn’t have even been included in the manuscript. P=0.02 is not borderline when the criteria is 0.01. That is like saying that 0.1 is borderline when 0.05 is your criteria. Given the multiple comparisons, this should not have been included.
• I would ask if there was a difference between the refusals. If you refused treatment and were on CM what was your survival rate? If you refused treatment and were not on CM, what was your survival rate? If these were similar, then I would say that this could easily be a problem of bias among medical doctors. They may be more likely to take no treatment as the patient’s answer if they are on CM treatments than if they are not.
• Separately, I would wonder if the refusal rates vary by geographic location.  If refusal rates are higher in the intermountain West and the Pacific areas, then of course this is going to be the case. Because there are higher CM usage rates in the West. It is unclear from the authors’ paper that they controlled for this in their treatment delay and refusal paragraph. In fact, in the next paragraph they do not control for this and this could be driving all the results. They control for cancer type, age, sex, income, educational level, clinical stage, and Charlson-Deyo comorbidity score. However, they do not control for location. Given that breast cancer is more prevalent on the east coast and upper Midwest, what they could have just as easily said is that those on the West coast aren’t getting the same treatments and are having higher refusals. And they also happen to be doing more CM treatments so that is what they are blaming it on.
• Lastly, they say that when controlling for refusal rates and delays, there is no difference in the survival rates. Therefore, it isn’t likely that the CM treatments that are the difference. Because that is the only thing that they haven’t controlled for. This leads me to once again say that it is likely MD bias against those using CM treatments that is leading to this problem. If you take out refusals and delay of treatment times and the survivability is the same, then the difference is on the MD side not the CM side. So they are giving poorer treatments, giving up too quickly on this population, letting them choose non-treatment quicker/more frequently.   [I strongly feel that the issues with the database are the issue but if the authors are willing to stand by the statement that while in the statistics in the general population indicate that there should be over 800,000 cases that should in their database but they got a unique and random sample and only got 258, then this is a reasonable argument explaining the differences.]