Why Anecdotes aren't Data
This reader pointed me towards the "Parent Ratings of Behavioral Effects of Biomedical Treatments" on the Autism Research Institute's website. For those of you who are not familiar with the work of the Autism Research Institute (ARI), it is - at least at present - primarily involved in finding and promoting "biomedical" treatments for autism, which apparently means the same as "alternative medicine". I say this because they reject - quite vigorously - several "biomedical" treatments that are well inside the mainstream of medical therapy.
Anyway, my helpful reader was trying to convince me that chelation has a significant positive effect on autism, as evidenced by the reports of parents who filled out the questionnaire (available on-line). Leaving until later any discussion about the validity of opinion polls used to judge medical therapies, let's look at what they have found.
On the list, chelation is placed under the heading of "Biomedical/Non-Drug/Supplements" and is included in the group "Detox. (Chelation)". Now, this is a very interesting place to put chelation - and shows the bias of the website quite nicely - since all chelating agents recommended by DAN! doctors (the practitioners recommended by ARI) are, sad to say, drugs (e.g. EDTA, DMSA, DMPS).
But, leaving that aside, the ARI opinion poll shows the following for "Detox. (Chelation)":
Got Worse - 2%
No Effect - 22%
Got Better - 76%
At this point, the pollster adds an odd mathematical twist - they divide the percent who "Got Better" by the percent who "Got Worse" and comes up with a 35:1 "Better:Worse" ratio. This is an interesting trick, since it ignores the 22% who experienced no effect and - it could be argued - are "worse" because they were exposed to the cost, effort and risk of chelation (or Detox. - whatever that might be) without experiencing any benefit. Including these people gives a less impressive - but still positive-sounding - 3:1 "Improvement:No Improvement" ratio.
Note also that the original survey data has six "effectiveness" levels (Definitely Helped; Moderate Improvement; Possibly Helped; No Definite Effect; Little Worse; Much Worse) with three "improved" ratings and only two "worsened".
But how likely is this to represent reality?
Taking a look further up the list (in the "Drugs" section, appropriately), we come to "Secretin" (both IV and Transdermal). The "scores" for these treatments are as follows:
Got Worse - 7%
No Effect - 44%
Got Better - 48%
Got Worse - 10%
No Effect - 49%
Got Better - 41%
So how does this compare with double-blinded studies? According to the ARI parent survey, IV secretin was 48% effective (a little less than half). The studies done by Repligen (which had a strong motive to find a positive effect) showed that IV secretin was no better than placebo.
So, does this mean that chelation is likely to be better, since it has a better win:loss ration?
Chelation is still "topical", it has a lot of people "talking it up" at the moment and, as a result, will rate higher. As parents experience the inevitable "treatment failures" and see the cycles of the disorder coming around again, they will become disenchanted with chelation and it will gradually fall from favor.
If we could go back and see the poll numbers for secretin during its heyday, we would see - I suspect - much the same result. High "effectiveness" numbers when everyone was talking about how secretin was going to "cure" autistic children, followed by a slide from grace as the studies after study showed no effect.
In a way, the chelation promoters have learned a valuable lesson from secretin - you'll notice that they are not encouraging real scientists to study chelation for autism in a blinded fashion. I suspect that they are none too happy with Dr James Adams, who is about to show just how effective chelation is for treating autism.
As they say...stay tuned!
Now, what about the idea - so popular with "alternative" medicine promoters - that the patient (or parent, in this case) is the best judge of whether a treatment works? And the corollary of that hypothesis, that opinion polls are an acceptable means of collecting efficacy "data"? Well, decades of scientific study have shown us that people are not very good judges of whether or not a treatment is working. Else, why would practitioners - and patients! - have been so willing to pursue purging, bleeding, cupping (making a comeback in "traditional" medicine), amulets, potions and general shamanism? Because - in their opinion - it appeared to work!
If you want to know how this could happen, go back to my "Field Guide to Pseudoscience and Quackery: Part 4" and read the last part:
"...the quackeries appear to “work” for the following reasons:
 The natural course of the disease: Fully 95% of the ailments for which people seek medical attention are self-limited – meaning that they will get better without treatment. The classic example is the “common cold”, for which there have been quack remedies since the dawn of human history. Left untreated, the average “cold” will resolve in about seven days. Vigorously treated by either quack remedies or real medicine (antibiotics, steroids, etc.), the average “cold” will resolve in about a week.
Also, some disorders are more severe at the outset than they are later - a good example being stroke. Quack practitioners have been making a living for years by treating recent stroke victims with vitamins, herbs, hyperbaric oxygen and the like and then taking credit for their improvement. Of course, even if you don't do anything, most stroke victims are much better a month or two after their stroke than they are the day it happens. The same is true of certain childhood developmental disorders, where quacks eagerly take credit for the natural progression of the disorder as the child gets older.
It's a wonder that someone hasn't promoted giving typical kids mega-vitamins, minerals or chelation in order to improve their verbal skills, coordination and social interaction. After all, if you give high-dose vitamin B6 to one year-olds, they'll have better language, social and physical skills when they're five. Of course, so will the kids that don't get the treatment. It's the natural progression of childhood - even in developmentally delayed children. Developmental delay does not mean developmental stasis.
 Regression to the mean: Most chronic or long-term ailments have a cyclic or fluctuating course – getting worse and then getting better and then repeating the cycle. Generally, people will seek medical attention for these ailments when they are at or near their worst. As a result, any treatment given – even an ineffective one – will usually be followed by improvement as the natural course of the disease takes it toward milder symptoms, potentially fooling the patient into believing that the remedy “worked”.
 Self-fulfilling Prophesy: Often called the “placebo effect”, which is a gross misnomer. The placebo is not having an effect – it is, in fact, completely inert. What is happening is that the patient, having been examined, given a diagnosis and prescribed a treatment, is expecting to get better. This expectation will cause them to unconsciously emphasize any feeling that their symptoms are improving and ignore or minimize any feeling that the symptoms are staying the same or worsening. In fact, since worry and feeling helpless have repeatedly been shown to increase pain, it is not surprising that a sense of hope – even false hope – will improve bothersome symptoms.
 Sense of Empowerment: The one thing that quackeries do that real medicine would do well to emulate is the way they involve the patient in the treatment. Modern medicine has largely tried to cut the patient “out of the loop” as much as possible – with implanted devices, long-acting medications, transdermal patches, etc. – because patient compliance is a huge variable in any treatment regimen. By eliminating this variable as much as possible, real medicine manages to attain more predictable and uniform results. This is all well and good, but it has the undesired effect of making the patient feel like a passive recipient of treatment, rather than an active participant. By giving the patient complex, arcane rituals to perform, the quack gives their patient a sense of control over this aspect of their life."
Bottom line - opinion polls are not a substitute for data. They aren't even a good first approximation of data. In fact, they have only a 50:50 chance of pointing in the same general direction that real data might point. And before someone says it, let me point out that none of this involves "lying" - just fooling ourselves.