Why Anecdotes aren't Data
A reader pointed me to a wonderful example of how people's impression of reality does not always match with reality itself - and a perfect example of why scientists spend so much time and effort trying to make sure that they aren't fooling themselves.
This reader pointed me towards the "Parent Ratings of Behavioral Effects of Biomedical Treatments" on the Autism Research Institute's website. For those of you who are not familiar with the work of the Autism Research Institute (ARI), it is - at least at present - primarily involved in finding and promoting "biomedical" treatments for autism, which apparently means the same as "alternative medicine". I say this because they reject - quite vigorously - several "biomedical" treatments that are well inside the mainstream of medical therapy.
Anyway, my helpful reader was trying to convince me that chelation has a significant positive effect on autism, as evidenced by the reports of parents who filled out the questionnaire (available on-line). Leaving until later any discussion about the validity of opinion polls used to judge medical therapies, let's look at what they have found.
On the list, chelation is placed under the heading of "Biomedical/Non-Drug/Supplements" and is included in the group "Detox. (Chelation)". Now, this is a very interesting place to put chelation - and shows the bias of the website quite nicely - since all chelating agents recommended by DAN! doctors (the practitioners recommended by ARI) are, sad to say, drugs (e.g. EDTA, DMSA, DMPS).
But, leaving that aside, the ARI opinion poll shows the following for "Detox. (Chelation)":
Got Worse - 2%
No Effect - 22%
Got Better - 76%
At this point, the pollster adds an odd mathematical twist - they divide the percent who "Got Better" by the percent who "Got Worse" and comes up with a 35:1 "Better:Worse" ratio. This is an interesting trick, since it ignores the 22% who experienced no effect and - it could be argued - are "worse" because they were exposed to the cost, effort and risk of chelation (or Detox. - whatever that might be) without experiencing any benefit. Including these people gives a less impressive - but still positive-sounding - 3:1 "Improvement:No Improvement" ratio.
Note also that the original survey data has six "effectiveness" levels (Definitely Helped; Moderate Improvement; Possibly Helped; No Definite Effect; Little Worse; Much Worse) with three "improved" ratings and only two "worsened".
But how likely is this to represent reality?
Taking a look further up the list (in the "Drugs" section, appropriately), we come to "Secretin" (both IV and Transdermal). The "scores" for these treatments are as follows:
Secretin IV:
Got Worse - 7%
No Effect - 44%
Got Better - 48%
Secretin Transdermal:
Got Worse - 10%
No Effect - 49%
Got Better - 41%
So how does this compare with double-blinded studies? According to the ARI parent survey, IV secretin was 48% effective (a little less than half). The studies done by Repligen (which had a strong motive to find a positive effect) showed that IV secretin was no better than placebo.
So, does this mean that chelation is likely to be better, since it has a better win:loss ration?
Sadly, no.
Chelation is still "topical", it has a lot of people "talking it up" at the moment and, as a result, will rate higher. As parents experience the inevitable "treatment failures" and see the cycles of the disorder coming around again, they will become disenchanted with chelation and it will gradually fall from favor.
If we could go back and see the poll numbers for secretin during its heyday, we would see - I suspect - much the same result. High "effectiveness" numbers when everyone was talking about how secretin was going to "cure" autistic children, followed by a slide from grace as the studies after study showed no effect.
In a way, the chelation promoters have learned a valuable lesson from secretin - you'll notice that they are not encouraging real scientists to study chelation for autism in a blinded fashion. I suspect that they are none too happy with Dr James Adams, who is about to show just how effective chelation is for treating autism.
As they say...stay tuned!
Now, what about the idea - so popular with "alternative" medicine promoters - that the patient (or parent, in this case) is the best judge of whether a treatment works? And the corollary of that hypothesis, that opinion polls are an acceptable means of collecting efficacy "data"? Well, decades of scientific study have shown us that people are not very good judges of whether or not a treatment is working. Else, why would practitioners - and patients! - have been so willing to pursue purging, bleeding, cupping (making a comeback in "traditional" medicine), amulets, potions and general shamanism? Because - in their opinion - it appeared to work!
If you want to know how this could happen, go back to my "Field Guide to Pseudoscience and Quackery: Part 4" and read the last part:
"...the quackeries appear to “work” for the following reasons:
[1] The natural course of the disease: Fully 95% of the ailments for which people seek medical attention are self-limited – meaning that they will get better without treatment. The classic example is the “common cold”, for which there have been quack remedies since the dawn of human history. Left untreated, the average “cold” will resolve in about seven days. Vigorously treated by either quack remedies or real medicine (antibiotics, steroids, etc.), the average “cold” will resolve in about a week.
Also, some disorders are more severe at the outset than they are later - a good example being stroke. Quack practitioners have been making a living for years by treating recent stroke victims with vitamins, herbs, hyperbaric oxygen and the like and then taking credit for their improvement. Of course, even if you don't do anything, most stroke victims are much better a month or two after their stroke than they are the day it happens. The same is true of certain childhood developmental disorders, where quacks eagerly take credit for the natural progression of the disorder as the child gets older.
It's a wonder that someone hasn't promoted giving typical kids mega-vitamins, minerals or chelation in order to improve their verbal skills, coordination and social interaction. After all, if you give high-dose vitamin B6 to one year-olds, they'll have better language, social and physical skills when they're five. Of course, so will the kids that don't get the treatment. It's the natural progression of childhood - even in developmentally delayed children. Developmental delay does not mean developmental stasis.
[2] Regression to the mean: Most chronic or long-term ailments have a cyclic or fluctuating course – getting worse and then getting better and then repeating the cycle. Generally, people will seek medical attention for these ailments when they are at or near their worst. As a result, any treatment given – even an ineffective one – will usually be followed by improvement as the natural course of the disease takes it toward milder symptoms, potentially fooling the patient into believing that the remedy “worked”.
[3] Self-fulfilling Prophesy: Often called the “placebo effect”, which is a gross misnomer. The placebo is not having an effect – it is, in fact, completely inert. What is happening is that the patient, having been examined, given a diagnosis and prescribed a treatment, is expecting to get better. This expectation will cause them to unconsciously emphasize any feeling that their symptoms are improving and ignore or minimize any feeling that the symptoms are staying the same or worsening. In fact, since worry and feeling helpless have repeatedly been shown to increase pain, it is not surprising that a sense of hope – even false hope – will improve bothersome symptoms.
[4] Sense of Empowerment: The one thing that quackeries do that real medicine would do well to emulate is the way they involve the patient in the treatment. Modern medicine has largely tried to cut the patient “out of the loop” as much as possible – with implanted devices, long-acting medications, transdermal patches, etc. – because patient compliance is a huge variable in any treatment regimen. By eliminating this variable as much as possible, real medicine manages to attain more predictable and uniform results. This is all well and good, but it has the undesired effect of making the patient feel like a passive recipient of treatment, rather than an active participant. By giving the patient complex, arcane rituals to perform, the quack gives their patient a sense of control over this aspect of their life."
Bottom line - opinion polls are not a substitute for data. They aren't even a good first approximation of data. In fact, they have only a 50:50 chance of pointing in the same general direction that real data might point. And before someone says it, let me point out that none of this involves "lying" - just fooling ourselves.
Prometheus
This reader pointed me towards the "Parent Ratings of Behavioral Effects of Biomedical Treatments" on the Autism Research Institute's website. For those of you who are not familiar with the work of the Autism Research Institute (ARI), it is - at least at present - primarily involved in finding and promoting "biomedical" treatments for autism, which apparently means the same as "alternative medicine". I say this because they reject - quite vigorously - several "biomedical" treatments that are well inside the mainstream of medical therapy.
Anyway, my helpful reader was trying to convince me that chelation has a significant positive effect on autism, as evidenced by the reports of parents who filled out the questionnaire (available on-line). Leaving until later any discussion about the validity of opinion polls used to judge medical therapies, let's look at what they have found.
On the list, chelation is placed under the heading of "Biomedical/Non-Drug/Supplements" and is included in the group "Detox. (Chelation)". Now, this is a very interesting place to put chelation - and shows the bias of the website quite nicely - since all chelating agents recommended by DAN! doctors (the practitioners recommended by ARI) are, sad to say, drugs (e.g. EDTA, DMSA, DMPS).
But, leaving that aside, the ARI opinion poll shows the following for "Detox. (Chelation)":
Got Worse - 2%
No Effect - 22%
Got Better - 76%
At this point, the pollster adds an odd mathematical twist - they divide the percent who "Got Better" by the percent who "Got Worse" and comes up with a 35:1 "Better:Worse" ratio. This is an interesting trick, since it ignores the 22% who experienced no effect and - it could be argued - are "worse" because they were exposed to the cost, effort and risk of chelation (or Detox. - whatever that might be) without experiencing any benefit. Including these people gives a less impressive - but still positive-sounding - 3:1 "Improvement:No Improvement" ratio.
Note also that the original survey data has six "effectiveness" levels (Definitely Helped; Moderate Improvement; Possibly Helped; No Definite Effect; Little Worse; Much Worse) with three "improved" ratings and only two "worsened".
But how likely is this to represent reality?
Taking a look further up the list (in the "Drugs" section, appropriately), we come to "Secretin" (both IV and Transdermal). The "scores" for these treatments are as follows:
Secretin IV:
Got Worse - 7%
No Effect - 44%
Got Better - 48%
Secretin Transdermal:
Got Worse - 10%
No Effect - 49%
Got Better - 41%
So how does this compare with double-blinded studies? According to the ARI parent survey, IV secretin was 48% effective (a little less than half). The studies done by Repligen (which had a strong motive to find a positive effect) showed that IV secretin was no better than placebo.
So, does this mean that chelation is likely to be better, since it has a better win:loss ration?
Sadly, no.
Chelation is still "topical", it has a lot of people "talking it up" at the moment and, as a result, will rate higher. As parents experience the inevitable "treatment failures" and see the cycles of the disorder coming around again, they will become disenchanted with chelation and it will gradually fall from favor.
If we could go back and see the poll numbers for secretin during its heyday, we would see - I suspect - much the same result. High "effectiveness" numbers when everyone was talking about how secretin was going to "cure" autistic children, followed by a slide from grace as the studies after study showed no effect.
In a way, the chelation promoters have learned a valuable lesson from secretin - you'll notice that they are not encouraging real scientists to study chelation for autism in a blinded fashion. I suspect that they are none too happy with Dr James Adams, who is about to show just how effective chelation is for treating autism.
As they say...stay tuned!
Now, what about the idea - so popular with "alternative" medicine promoters - that the patient (or parent, in this case) is the best judge of whether a treatment works? And the corollary of that hypothesis, that opinion polls are an acceptable means of collecting efficacy "data"? Well, decades of scientific study have shown us that people are not very good judges of whether or not a treatment is working. Else, why would practitioners - and patients! - have been so willing to pursue purging, bleeding, cupping (making a comeback in "traditional" medicine), amulets, potions and general shamanism? Because - in their opinion - it appeared to work!
If you want to know how this could happen, go back to my "Field Guide to Pseudoscience and Quackery: Part 4" and read the last part:
"...the quackeries appear to “work” for the following reasons:
[1] The natural course of the disease: Fully 95% of the ailments for which people seek medical attention are self-limited – meaning that they will get better without treatment. The classic example is the “common cold”, for which there have been quack remedies since the dawn of human history. Left untreated, the average “cold” will resolve in about seven days. Vigorously treated by either quack remedies or real medicine (antibiotics, steroids, etc.), the average “cold” will resolve in about a week.
Also, some disorders are more severe at the outset than they are later - a good example being stroke. Quack practitioners have been making a living for years by treating recent stroke victims with vitamins, herbs, hyperbaric oxygen and the like and then taking credit for their improvement. Of course, even if you don't do anything, most stroke victims are much better a month or two after their stroke than they are the day it happens. The same is true of certain childhood developmental disorders, where quacks eagerly take credit for the natural progression of the disorder as the child gets older.
It's a wonder that someone hasn't promoted giving typical kids mega-vitamins, minerals or chelation in order to improve their verbal skills, coordination and social interaction. After all, if you give high-dose vitamin B6 to one year-olds, they'll have better language, social and physical skills when they're five. Of course, so will the kids that don't get the treatment. It's the natural progression of childhood - even in developmentally delayed children. Developmental delay does not mean developmental stasis.
[2] Regression to the mean: Most chronic or long-term ailments have a cyclic or fluctuating course – getting worse and then getting better and then repeating the cycle. Generally, people will seek medical attention for these ailments when they are at or near their worst. As a result, any treatment given – even an ineffective one – will usually be followed by improvement as the natural course of the disease takes it toward milder symptoms, potentially fooling the patient into believing that the remedy “worked”.
[3] Self-fulfilling Prophesy: Often called the “placebo effect”, which is a gross misnomer. The placebo is not having an effect – it is, in fact, completely inert. What is happening is that the patient, having been examined, given a diagnosis and prescribed a treatment, is expecting to get better. This expectation will cause them to unconsciously emphasize any feeling that their symptoms are improving and ignore or minimize any feeling that the symptoms are staying the same or worsening. In fact, since worry and feeling helpless have repeatedly been shown to increase pain, it is not surprising that a sense of hope – even false hope – will improve bothersome symptoms.
[4] Sense of Empowerment: The one thing that quackeries do that real medicine would do well to emulate is the way they involve the patient in the treatment. Modern medicine has largely tried to cut the patient “out of the loop” as much as possible – with implanted devices, long-acting medications, transdermal patches, etc. – because patient compliance is a huge variable in any treatment regimen. By eliminating this variable as much as possible, real medicine manages to attain more predictable and uniform results. This is all well and good, but it has the undesired effect of making the patient feel like a passive recipient of treatment, rather than an active participant. By giving the patient complex, arcane rituals to perform, the quack gives their patient a sense of control over this aspect of their life."
Bottom line - opinion polls are not a substitute for data. They aren't even a good first approximation of data. In fact, they have only a 50:50 chance of pointing in the same general direction that real data might point. And before someone says it, let me point out that none of this involves "lying" - just fooling ourselves.
Prometheus
posted by Prometheus at 11:00 AM
11 Comments:
I'm going to try to remember this. Doubt the people I wind up dealing with would read it, though.
Parents who are burned out by a certain treatment or not impressed are not as likely to report that to the ARI website. Also, they know that it's St. Bernie who is asking for the data and everyone in this crowd wants to make Bernie Rimland happy.
Demand characteristics.
Then there's " Cognitive bolstering
An aspect of cognitive dissonance theory and decision-making. Cognitive bolstering is where we consciously and unconsciously look for things that support our chosen alternative or downgrade the merits of the rejected alternative."
I know you've written about those before, maybe not using those terms.
Social psychologists would have a field day with this whole DAN! thing.
I was really surprised to see Dr. Hendren try to give this "data" from ARI some kind of gloss of respectibility in his talk at the "summer institute." He didn't say it was really good data, but he said that it was showing them directions to look or something.
Bosh.
Hi Prometheus,
I know you are already aware of all that is non-scientific about the particular AZ chelation study you mentioned (your comments at my blog were most insightful, especially when it came to Doctor's Anecdotes, scratch that, I mean Doctor's Data Labs).
From one of my last e-mails in the extensive conversation I had with him, Jim said,
"If only a fraction of children were helped, wouldn't it be worth it?"
I responded with,
"It would be worth it, if they were actually helped and the science proves it, as determined by the scientific community through the process of real peer-review. It would not be worth it if they were led to ‘believe’ they were helped, and they were led to ‘believe’ that science proves it, if in fact it did not, or the real scientific community was never given an opportunity to review it."
He did not respond to this.
It would be speculative to assume he prefers the opinion polls over science, but since he didn't respond, not even to defend his version of the science any further, it looks like the polls will do.
Good stuff as always. I can't tell you how many times extremists have tried to pat me on the head and tell me to head over to the ARI site to see the real story with real numbers. I did look at it the first time and laughed. Then I realized that people were actually using this as a guide to apply controversial and extreme medical procedures on their children and that was rather sobering.
Thank you for this Prometheus,
Having tried several of these biomedical treatments with my child I can confirm many of the observations reported to the ARI website. Some seemed to help, some had no effect, and others made it worse, depending on when they were given.
More than once we planned to introduce a new supplement or other treatment only to see a big improvement or decline the day before, and we said "It's a good thing we didn't start the ----- or we would have attributed this behavior to it."
Say that a few times and it starts to sink in that autistic behaviors can be cyclical and supplements and other external events may not be the cause.
And I am much more objective than most parents -- just my unbiased opinion ;-)
My troll sense tells me that right now, some altie is copying and pasting that last bit and removing the "I'm Obviously Being a Little Sarcastic" smiley and using it as an example of skeptical arrogance.
You must have a finely tuned altie troll radar system BD. Now you have to coin a name for it:
Alt-ar? No..
Troll-dar?
Woo-dar?
Echinadar?
Trudeau-ar?
Skeptar?
I'm sure you'll think of something better.
Alt-ar? No..
I say yes. Alter powers would be nifty.
These people are called "Alter users" because their power gives them the ability to grab, and reshape matter at a molecular level into what their specific Alter dictates, using only thought and willpower. The powers that these people possess give them superiority over normal humans.
Fits rather well actually, for the alties anyway. Need to adjust my altimeter...
Especially since alties can grab negative studies and reshape specific mined quotes into a conspiracy as their specific ideology dictates, using only a little thought and a lot of paranoia.
I work with special needs kids and I can believe this. I have seen parents fill out behaviour forms completely opposite to what has been filled out by the school- even though the behaviours happen in front of them at the school. This is told to the powers that be that IT ONLY EVER HAPPENS AT SCHOOL. People believe what they want no matter the evidence.
Organizations also misuse information and disregard those who deviate from their established norms. Unless you crawl in the muck and investigate everything you cannot be sure of the helath or effectiveness of your bank, insurance company, oil company, government, etc... Think Enron.
I think there must be a reason- a biological reason- we lie to ourselves so much as a society. Does anyone know what it is?
<< Home